Refactor where response parsing happens

This is an attempt to make model specific response parsing easier and
ensure that parsing doesn't happen more than once for a response and we
don't have weird parsing of sub responses.