Composite models (encoder-decoder) behave differently across PyTorch and TensorFlow
Author: ydshieh
Pull request: no
Labels: name: WIP

name: bug

### System Info

- `transformers` version: 4.21.0.dev0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.5
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.10.1+cpu (False)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@gante @patrick

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

For composite models (Encoder-Decoder like), the behavior of some config attributes (e.g. `output_hidden_states`) is not consistent.

- PT Encoder-Decoder leaves the components to use their own configs
- TF Encoder-Decoder uses `unpack_inputs`, which uses its own config attributes (and some attributes of its component configs have no effect) 

### Case 1
```python
import torch
import tensorflow as tf
from transformers import BertConfig
from transformers import EncoderDecoderConfig, EncoderDecoderModel, TFEncoderDecoderModel

bert_config = BertConfig()
config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config=bert_config, decoder_config=bert_config)

# Set `output_hidden_states` in top models
config.output_hidden_states = True
config.output_hidden_states = True

pt_model = EncoderDecoderModel(config)
tf_model = TFEncoderDecoderModel(config)

input_ids = torch.tensor([[1, 1]], dtype=torch.int32)
pt_output = pt_model(input_ids, decoder_input_ids=input_ids)

input_ids = tf.constant([[1, 1]], dtype=tf.int32)
tf_output = tf_model(input_ids, decoder_input_ids=input_ids)

print(pt_output.keys())
print(tf_output.keys())
```

### Output (case 1)
```bash
odict_keys(['logits', 'past_key_values', 'encoder_last_hidden_state'])
odict_keys(['logits', 'decoder_hidden_states', 'encoder_last_hidden_state', 'encoder_hidden_states'])
```

### Case 2
```python
import torch
import tensorflow as tf
from transformers import BertConfig
from transformers import EncoderDecoderConfig, EncoderDecoderModel, TFEncoderDecoderModel

bert_config = BertConfig()
config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config=bert_config, decoder_config=bert_config)

# Set `output_hidden_states` in sub models
config.encoder.output_hidden_states = True
config.decoder.output_hidden_states = True

pt_model = EncoderDecoderModel(config)
tf_model = TFEncoderDecoderModel(config)

input_ids = torch.tensor([[1, 1]], dtype=torch.int32)
pt_output = pt_model(input_ids, decoder_input_ids=input_ids)

input_ids = tf.constant([[1, 1]], dtype=tf.int32)
tf_output = tf_model(input_ids, decoder_input_ids=input_ids)

print(pt_output.keys())
print(tf_output.keys())
```


### Output (case 2)
```bash
odict_keys(['logits', 'past_key_values', 'decoder_hidden_states', 'encoder_last_hidden_state', 'encoder_hidden_states'])
odict_keys(['logits', 'encoder_last_hidden_state'])
```

### Expected behavior

Ideally, all of them should return hidden states. Or at the least, PT/TF should return the same results.

Comments:
Seems related to the `@unpack_inputs` decorator - I remember it being much trickier on composite models when I first added it.

Will be taking a look today 👍