Custom Seq2Seq translation model training exits with error
Author: krajasek
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.9.0.dev0
- Platform: Linux-4.14.238-182.422.amzn2.x86_64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.9.0+cu102 (True)
- Tensorflow version (GPU?): 2.5.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes, using deepspeed

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten 
@sgugger 

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

I am training a custom language translation model with pre-trained Roberta as the source model and custom-trained GPT-Neo as the target model.  The training process quickly exits with an error and the error stack trace is pasted below. The custom language translation model has been developed based on the HF example: https://huggingface.co/patrickvonplaten/roberta2roberta-cnn_dailymail-fp16

## To reproduce

Steps to reproduce the behavior:

1. Run an HF Trainer with deepspeed (see below deepspeed script)
2. The Trainer process exits with error (see below stack trace)
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
### Error Stack Trace
```
Traceback (most recent call last):
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 303, in <module>
    tr_model.train(model_dir=model_output_dir, epochs=epochs)
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 194, in train
Traceback (most recent call last):
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 303, in <module>
    train_results = trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1267, in train
Traceback (most recent call last):
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 303, in <module>
    tr_model.train(model_dir=model_output_dir, epochs=epochs)
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 194, in train
    train_results = trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1267, in train
Traceback (most recent call last):
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 303, in <module>
    tr_model.train(model_dir=model_output_dir, epochs=epochs)
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 194, in train
    tr_loss += self.training_step(model, inputs)
    train_results = trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1760, in training_step
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1267, in train
    tr_model.train(model_dir=model_output_dir, epochs=epochs)
  File "src/phase3/language_model/luna_llayla_translator_model_trainer.py", line 194, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1760, in training_step
    train_results = trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1267, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1760, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1760, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1794, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 799, in forward
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 799, in forward
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    output = self.module(*inputs[0], **kwargs[0])
    output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 799, in forward
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 799, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 450, in forward
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 450, in forward
    **kwargs_decoder,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    **kwargs_decoder,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 450, in forward
    **kwargs_decoder,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 450, in forward
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
    **kwargs_decoder,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```

### deepspeed script
```bash
export TOKENIZERS_PARALLELISM=false
export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=eth0
export NCCL_DEBUG=INFO
deepspeed --include localhost:0,1,2,3 src/phase3/language_model/luna_llayla_translator_model_trainer.py
```

### Trainer snippet
```python
    def train(self, model_dir, epochs=10, learning_rate=5e-05,
              clear_cuda_cache=True, metric_name='rouge'):

        if clear_cuda_cache and torch.cuda.is_available():
            torch.cuda.empty_cache()
        print('Loading dataset ...')
        if not self.train_raw_dataset:
            self.load_raw_dataset(self.dataset_path)
        print('Pre-processing dataset ...')
        self.preprocess_data()
        print('Loading metric data ...')
        if not self.metric:
            self.load_metric(metric_name=metric_name)

        train_config = LunaLlaylaTranslatorTrainingConfig.config
        if epochs:
            train_config['num_train_epochs'] = epochs
        if learning_rate:
            train_config['learning_rate'] = learning_rate
        if model_dir:
            train_config['output_dir'] = model_dir
        if self.deepspeed_config:
            train_config['deepspeed'] = self.deepspeed_config

        train_arguments = Seq2SeqTrainingArguments(**train_config)
        print('Training ...')
        trainer = Seq2SeqTrainer(
            model=self.translator_model,
            args=train_arguments,
            data_collator=default_data_collator,
            train_dataset=self.train_processed_dataset,
            eval_dataset=self.validation_processed_dataset,
            compute_metrics=self.compute_metrics,
        )
        train_results = trainer.train()
        print('Training completed.')
        print('Evaluating model ...')
        train_metrics = train_results.metrics
        trainer.log_metrics("train", train_metrics)
        trainer.save_metrics("train", train_metrics)
        print('*** Train metrics ***')
        print(train_metrics)
        eval_metrics = trainer.evaluate()
        try:
            perplexity_score = math.exp(eval_metrics['eval_loss'])
        except OverflowError:
            perplexity_score = float('inf')
        eval_metrics['perplexity_score'] = perplexity_score
        trainer.log_metrics("eval", eval_metrics)
        trainer.save_metrics("eval", eval_metrics)
        print('*** Eval metrics ***')
        print(eval_metrics)

        print('Saving trained model ...')
        trainer.save_state()
        trainer.save_model(output_dir=model_dir)
        print('Evaluating model ...')
        self.evaluate()

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I expect the training process to be complete without errors.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for help to debug your scripts, and provide all relevant code. The error indicates you are attempting to pass a `encoder_hidden_states` to a model that don't accept that but we don't see our your dataset is or how your model is, so no one can help you understand why.

I am not explicitly passing any state to the model at all. I can't share the dataset as it could be proprietary to our organization, but can I at-least know if Roberta to GPT-Neo language translation is possible? I glanced at the code base https://github.com/huggingface/transformers/blob/master/src/transformers/models/gpt_neo/modeling_gpt_neo.py and it looks like the forward() method does not accept encoder_hidden_states as a parameter?

The trainer script works fine when a pre-trained Roberta model is used as the source and a custom-trained Roberta model is used as the target model. It only fails when a pre-trained Roberta model is the source and custom-trained GPT-Neo model is the target.

Here is the list of combinations tried out for seq2seq translation:

1) Roberta to Roberta: No issues faced with the seq2seq model training
2) Roberta to GPT-2: No issues faced with the seq2seq model training
3) Roberta to GPT-Neo: Non-recoverable errors during the seq2seq model training

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.