Fix tf pytorch test in auto
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove some tests in `tests/auto/test_modeling_tf_pytorch.py`:

  - when we load a model using `from_pretrained` from another framework, it returns a single model object instead of a tuple of (model, loading_info) even if we specify `output_loading_info=True`. Some tests expect to have `loading_info`, and cause errors like `ypeError: cannot unpack non-iterable TFGPT2LMHeadModel object`.

## Remark

It might be better to document this behavior clearly for the users :-)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16044). All of your documentation changes will be reflected on that endpoint.

> Thanks for looking into this. I would be in favor of trying o return loading_info and test that it is not None maybe

Do you mean we should return (model, None) in `from_pretrained` if `output_loading_info=True` + `load from a different framework`?

I think @patrickvonplaten is suggesting having the `load_pytorch_checkpoint_in_tf2_model` that is called in the `from_pt` block return the `loading_info` the same way the regular `from_pretrained` does.

> I think @patrickvonplaten is suggesting having the `load_pytorch_checkpoint_in_tf2_model` that is called in the `from_pt` block return the `loading_info` the same way the regular `from_pretrained` does.

Yeah, this makes more sense! @patrickvonplaten could you confirm?

Kindly cc @patrickvonplaten and @sgugger (low priority PR)

I tried to return `loading_info` when the loading is across frameworks. You can see the attempt in [this commit](https://github.com/huggingface/transformers/pull/16044/commits/09284cbdca9c099c52e7c6f973c06074dd6f9631).

Before going further, I would like to have some feedbacks:

- For the methods in `src/transformers/modeling_tf_pytorch_utils.py`, in particular [load_tf2_weights_in_pytorch_model](https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_tf_pytorch_utils.py#L466), should I:
  - `return model, loading_info` with a condition `if output_loading_info`
     (this has fewer impacts)
  - or return a tuple like `return pt_model, missing_keys, unexpected_keys` and let the caller to decide what to use 
     (more places need to be changed in this case)
    
    This is related to the change in 
    https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_utils.py#L1512-L1514
    A few line below, the PyTorch version is 
        `model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_state_dict_into_model(...)`

- We will also have to change the PT/Flax loading methods to take `from_flax` into account.
- [This line](https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_utils.py#L1506) will be exceptional though. I don't have a clear idea how to deal with it if we insist to have `loading_info` in the final outputs.

> Kindly cc @patrickvonplaten and @sgugger (low priority PR)
> 
> I tried to return loading_info when the loading is across frameworks. You can see the attempt in [this commit](https://github.com/huggingface/transformers/pull/16044/commits/09284cbdca9c099c52e7c6f973c06074dd6f9631).
> 
> Before going further, I would like to have some feedbacks:
> 
> For the methods in src/transformers/modeling_tf_pytorch_utils.py, in particular [load_tf2_weights_in_pytorch_model](https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_tf_pytorch_utils.py#L466), should I:
> 
> return model, loading_info with a condition if output_loading_info
> (this has fewer impacts)

Yes this sounds good to me! Default it to False for backward compatibility and I think that's a good approach :-)

> 
> or return a tuple like return pt_model, missing_keys, unexpected_keys and let the caller to decide what to use
> (more places need to be changed in this case)
> 
> This is related to the change in
> [transformers/src/transformers/modeling_utils.py](https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_utils.py#L1512-L1514)
> 
> Lines 1512 to 1514 in [09284cb](https://github.com/huggingface/transformers/commit/09284cbdca9c099c52e7c6f973c06074dd6f9631)
> 
>  model, loading_info = load_tf2_checkpoint_in_pytorch_model( 
>      model, resolved_archive_file, allow_missing_keys=True, output_loading_info=True 
>  ) 
> 
> A few line below, the PyTorch version is
> model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_state_dict_into_model(...)
> We will also have to change the PT/Flax loading methods to take from_flax into account.
> 
> [This line](https://github.com/huggingface/transformers/blob/09284cbdca9c099c52e7c6f973c06074dd6f9631/src/transformers/modeling_utils.py#L1506) will be exceptional though. I don't have a clear idea how to deal with it if we insist to have loading_info in the final outputs.

Think this last line is really an edge case and I would be fine with always leaving the loading_info `None` in this case