ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
Author: Ki6an
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: **4.0.0**
- Platform: **google colab**
- Python version: 3
- PyTorch version (GPU?): 1.7.0+cu101


### Who can help
@patrickvonplaten 
@patil-suraj


## Information

Model I am using (T5):

The problem arises when using:
```python
from transformers import T5Tokenizer, T5ForConditionalGeneration
import sentencepiece

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small', torchscript = True)

input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
labels = tokenizer('<extra_id_0> cute dog <extra_id_1> the <extra_id_2> 
', return_tensors='pt').input_ids
outputs = model(input_ids=input_ids, labels=labels)
```
```input_ids = tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
outputs = model.generate(input_ids)
```
```import torch
traced_model = torch.jit.trace(model, input_ids )
torch.jit.save(traced_model, "traced_t5.pt")
```

as mentioned in the [article ](https://huggingface.co/transformers/torchscript.html#saving-a-model) I tried to convert the model to `torchscript `

`T5ForConditionalGeneration` model is not supporting `trace`  function for converting the model to `torchscript`

 the output produced :
```---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-e37c13fee7bc> in <module>()
      1 import torch
----> 2 traced_model = torch.jit.trace(model, input_ids )
      3 torch.jit.save(traced_model, "traced_t5.pt")

7 frames
/usr/local/lib/python3.6/dist-packages/transformers/models/t5/modeling_t5.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    774         else:
    775             err_msg_prefix = "decoder_" if self.is_decoder else ""
--> 776             raise ValueError(f"You have to specify either {err_msg_prefix}inputs or {err_msg_prefix}inputs_embeds")
    777 
    778         if inputs_embeds is None:

ValueError: You have to specify either decoder_inputs or decoder_inputs_embeds
```
I got the same issue when converting a question-generation T5 model to `torchscript`, and the issue is [here](https://github.com/patil-suraj/question_generation/issues/52)


Comments:
@patrickvonplaten might be able to help

Seq2Seq models are a bit special - they also need `decoder_input_ids` as the error message states. Since torchscript however does not allow keyword arguments we need to provide positional arguments and therefore it's mandatory to also provide the 2nd argument being the `attention_mask` (for the encoder).

The following is what you are looking for (I think):

```python
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5ForConditionalGeneration.from_pretrained('t5-small', torchscript = True)
input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
attention_mask = input_ids.ne(model.config.pad_token_id).long()
decoder_input_ids = tokenizer('<pad> <extra_id_0> cute dog <extra_id_1> the <extra_id_2>', return_tensors='pt').input_ids

traced_model = torch.jit.trace(model, (input_ids, attention_mask, decoder_input_ids))
torch.jit.save(traced_model, "traced_t5.pt")
```

Thank you for the solution, your mentioned code works perfectly fine for creating a `torchscript `model. 
But I have one more question, the generated `traced_t5.pt` model doesn't seem to have the `model.generate()` method. 
how to get the `token ids` output from this newly created model? (only from model()). 
Also, In General, how can we get the output (token ids) from a t5 model without using the `generate()` method?

Yeah, I don't think our `generate()` method is torchscriptable yet :-/ You should take a look at the `greedy_search` method to see how the `generate()` method can be implemented by hand :-) 

Greedy search: https://github.com/huggingface/transformers/blob/df311a5ccf50be3031474e289b43b1be43111144/src/transformers/generation_utils.py#L622

Generate: 
https://github.com/huggingface/transformers/blob/df311a5ccf50be3031474e289b43b1be43111144/src/transformers/generation_utils.py#L296



thank you, I'll look into it. 

@Ki6an : Were you able to figure out how to make use of `greedy_search` to do the work which `generate` does? If so can I request you to share that as a gist?

@karrtikiyerkcm have a look at [FastT5](https://github.com/Ki6an/fastT5) library, it implements both greedy and beam search for T5. 

Thanks @Ki6an , I was trying something similar for Pegasus Models for the summarisation task.

@Ki6an  Hello, the input_id I  inputed is 64*100 (Batch_size,max_sequence), why the size of 
     T5ForConditionalGeneration.generate result is 100?where is the batch_size



anyone know if it's still not possible to use torchscript with generate?

@patrickvonplaten @Ki6an Hi, what should ```decoder_input_ids``` be if my input text is ```translate English to German: Thank you!```? I'm using this for inference. For decoder models like BERT and GPT, all I need to do is use Tokenizer to get the ```input_ids``` which will be passed into the models. But I'm not sure how that works for encoder-decoder models like T5 here.

> @patrickvonplaten @Ki6an Hi, what should `decoder_input_ids` be if my input text is `translate English to German: Thank you!`? I'm using this for inference. For decoder models like BERT and GPT, all I need to do is use Tokenizer to get the `input_ids` which will be passed into the models. But I'm not sure how that works for encoder-decoder models like T5 here.

Hi, I wanted to follow up on this. I have the same question.

> Seq2Seq models are a bit special - they also need `decoder_input_ids` as the error message states. Since torchscript however does not allow keyword arguments we need to provide positional arguments and therefore it's mandatory to also provide the 2nd argument being the `attention_mask` (for the encoder).
> 
> The following is what you are looking for (I think):
> 
> ```python
> from transformers import T5Tokenizer, T5ForConditionalGeneration
> import torch
> 
> tokenizer = T5Tokenizer.from_pretrained('t5-small')
> model = T5ForConditionalGeneration.from_pretrained('t5-small', torchscript = True)
> input_ids = tokenizer('The <extra_id_0> walks in <extra_id_1> park', return_tensors='pt').input_ids
> attention_mask = input_ids.ne(model.config.pad_token_id).long()
> decoder_input_ids = tokenizer('<pad> <extra_id_0> cute dog <extra_id_1> the <extra_id_2>', return_tensors='pt').input_ids
> 
> traced_model = torch.jit.trace(model, (input_ids, attention_mask, decoder_input_ids))
> torch.jit.save(traced_model, "traced_t5.pt")
> ```

Hi can you please provide example of how to use this t5 jit traced model for inference..
I tried using it but it requires decoder_input_ids.. is there any way of doing inference without the decoder_input_ids?