unexpected keyword argument 'encoder_hidden_states' when using PreTrainedEncoderDecoder
Author: mmsamiei
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

After I have defined my seq2seq class using Encoder Decoder Architecture in the following way:
```
from transformers import PreTrainedEncoderDecoder
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased','gpt2')
```
I try to forward tensor through the model in this following way:
```
model(question_batch, answer_batch)
```
 but I got this error:

```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-16-bb34e9576e8b> in <module>()
----> 1 model(test_history, test_knowledge)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```

can anyone help me?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I am facing the same issue. Probably `PreTrainedEncoderDecoder` supports only `Bert` to `Bert` models.

I'm getting this error too
```
Converting vae...
Converting text encoder...
Downloading100% 1.71G/1.71G [00:24<00:00, 69.1MB/s]
Downloading100% 4.55k/4.55k [00:00<00:00, 746kB/s]
Downloading100% 1.22G/1.22G [00:18<00:00, 66.3MB/s]
Downloading100% 342/342 [00:00<00:00, 57.0kB/s]
Saving diffusion model...
Restored system models.
Checkpoint successfully extracted to /content/gdrive/MyDrive/sd/stable-diffusion-webui/models/dreambooth/TTFM/working
Returning ['default', True, True, 1, '', '', 0, 0, 1, True, True, 50, False, False, 5e-06, 1e-06, 0.0001, '', 5e-05, 1.0, 1.0, 1, 0.5, 1.0, 0.5, 'constant_with_warmup', 0, 75, 'fp16', 100, True, '', 1.0, 512, 1, '', 420420, True, False, True, 25, True, False, True, 5, False, False, False, False, 1, False, 1.0, True, False, True, False, False, '', 7.5, 60, '', '', '', '', '', '', 1, 0, 0, -1, 7.5, 60, '', '', '', '', 7.5, 60, '', '', '', '', '', '', 1, 0, 0, -1, 7.5, 60, '', '', '', '', 7.5, 60, '', '', '', '', '', '', 1, 0, 0, -1, 7.5, 60, '', '', '', 'Loaded config.']
Saved settings.
Custom model name is TTFM
Starting Dreambooth training...
Initializing dreambooth training...
Replace CrossAttention.forward to use default
Instance Bucket 0: Resolution (512, 512), Count: 723
Target Bucket 0: Resolution (512, 512), Count: 0
We need a total of 0 images.
Nothing to generate.
Exception importing 8bit adam: No module named 'bitsandbytes'
WARNING:extensions.sd_dreambooth_extension.dreambooth.train_dreambooth:Exception importing 8bit adam: No module named 'bitsandbytes'
Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 304, in inner_loop
    import bitsandbytes as bnb
ModuleNotFoundError: No module named 'bitsandbytes'
Preparing dataset
Preparing dataset
Preparing Dataset (With Caching)
100% 723/723 [00:46<00:00, 15.47it/s]
Train Bucket 1: Resolution (512, 512), Count: 723
Total images: 241
Total dataset length (steps): 241
Sched breakpoint is 108450
  ***** Running training *****
  Instance Images: 723
  Class Images: 0
  Total Examples: 723
  Num batches each epoch = 241
  Num Epochs = 300
  Batch Size Per Device = 3
  Gradient Accumulation steps = 3
  Total train batch size (w. parallel, distributed & accumulation) = 9
  Text Encoder Epochs: 210
  Total optimization steps = 216900
  Total training steps = 216900
  Resuming from checkpoint: False
  First resume epoch: 0
  First resume step: 0
  Lora: False, Adam: False, Prec: bf16
  Gradient Checkpointing: True
  EMA: True
  LR: 4.5e-05)
Steps:   0% 0/216900 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/scripts/dreambooth.py", line 561, in start_training
    result = main(config, use_txt2img=use_txt2img)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 973, in main
    return inner_loop()
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/memory.py", line 116, in decorator
    return function(batch_size, grad_size, prof, *args, **kwargs)
  File "/content/gdrive/MyDrive/sd/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 829, in inner_loop
    noise_pred = unet(noisy_latents, timesteps, encoder_hidden_states).sample
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/accelerate/utils/operations.py", line 490, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/usr/local/lib/python3.8/dist-packages/torch/amp/autocast_mode.py", line 14, in decorate_autocast
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/unet_2d_condition.py", line 481, in forward
    sample, res_samples = downsample_block(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/unet_2d_blocks.py", line 781, in forward
    hidden_states = torch.utils.checkpoint.checkpoint(
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/checkpoint.py", line 249, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/checkpoint.py", line 107, in forward
    outputs = run_function(*args)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/unet_2d_blocks.py", line 774, in custom_forward
    return module(*inputs, return_dict=return_dict)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/transformer_2d.py", line 265, in forward
    hidden_states = block(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/diffusers/models/attention.py", line 285, in forward
    attn_output = self.attn1(
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward_default() got an unexpected keyword argument 'encoder_hidden_states'
Steps:   0% 0/216900 [00:01<?, ?it/s]
Training completed, reloading SD Model.
Restored system models.
Returning result: Exception training model: 'forward_default() got an unexpected keyword argument 'encoder_hidden_states''.
```