T5 TorchScript (Trace) Conversion 
Author: gyin94
Pull request: no

# ❓ Questions & Help

How can we correctly set inputs for t5 TorchScript? 

## Details
<!-- Description of your issue -->

```python
from transformers import T5Model
import torch

tokens_tensor = torch.ones(1, 10, dtype=torch.long)
model = T5Model.from_pretrained("t5-small", torchscript=True)
model.eval()
scripted_model = torch.jit.trace(model, (tokens_tensor, tokens_tensor))
```

Error:
```
ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

Comments:
Hey @gyin-ai - can you specify your version?

I cannot reproduce the error on master.

In master, the above example works for me but it doesn't work for T5ForConditionalGeneration

```
from transformers import T5ForConditionalGeneration
import torch

tokens_tensor = torch.ones(1, 10, dtype=torch.long)
model = T5ForConditionalGeneration.from_pretrained("t5-small", torchscript=True)
model.eval()
scripted_model = torch.jit.trace(model, (tokens_tensor, tokens_tensor))
```
It fails with the same error
```ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds```

Sadly, I don't have a good answer here :-/ 

The problem is that `decoder_input_ids` is not the second argument -> so that's why your function does not work. 
This PR would make it possible to run your code:  #6268 , but it does not really solve the problem because one might want to use `input_embeds` instead of `input_ids` and she/he would run into the same problem. It would allow for torchtrace for the most general case though...

i guess since usually one passes `input_ids` and `decoder_input_ids`, we could merge the PR...What do you think? @LysandreJik 

```
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-large", torchscript=True)
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-large", torchscript=True)

tokenized_dict = tokenizer(
    ["please answer the following question: what is the boiling point of nitrogen",], ["-320.4F",], 
    return_tensors="pt"
)
input_tuple = (tokenized_dict['input_ids'], tokenized_dict['attention_mask'])

traced_model = torch.jit.trace(model, input_tuple)
torch.jit.save(traced_model, "flan-t5-large.pt")
```

I was trying to trace `google/flan-t5-large` model in torchScript. But I'm facing following exception:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [29], in <cell line: 13>()
      7 tokenized_dict = tokenizer(
      8     ["please answer the following question: what is the boiling point of nitrogen",], ["-320.4F",], 
      9     return_tensors="pt"
     10 )
     11 input_tuple = (tokenized_dict['input_ids'], tokenized_dict['attention_mask'])
---> 13 traced_model = torch.jit.trace(model, input_tuple)
     14 torch.jit.save(traced_model, "flan-t5-large.pt")

File ~/Library/Python/3.9/lib/python/site-packages/torch/jit/_trace.py:759, in trace(func, example_inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    756     return func
    758 if isinstance(func, torch.nn.Module):
--> 759     return trace_module(
    760         func,
    761         {"forward": example_inputs},
    762         None,
    763         check_trace,
    764         wrap_check_inputs(check_inputs),
    765         check_tolerance,
    766         strict,
    767         _force_outplace,
    768         _module_class,
    769     )
    771 if (
    772     hasattr(func, "__self__")
    773     and isinstance(func.__self__, torch.nn.Module)
    774     and func.__name__ == "forward"
    775 ):
    776     return trace_module(
    777         func.__self__,
    778         {"forward": example_inputs},
   (...)
    785         _module_class,
    786     )

File ~/Library/Python/3.9/lib/python/site-packages/torch/jit/_trace.py:976, in trace_module(mod, inputs, optimize, check_trace, check_inputs, check_tolerance, strict, _force_outplace, _module_class, _compilation_unit)
    972     argument_names = get_callable_argument_names(func)
    974 example_inputs = make_tuple(example_inputs)
--> 976 module._c._create_method_from_trace(
    977     method_name,
    978     func,
    979     example_inputs,
    980     var_lookup_fn,
    981     strict,
    982     _force_outplace,
    983     argument_names,
    984 )
    985 check_trace_method = module._c._get_method(method_name)
    987 # Check the trace against new traces created from user-specified inputs

File ~/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py:1182, in Module._slow_forward(self, *input, **kwargs)
   1180         recording_scopes = False
   1181 try:
-> 1182     result = self.forward(*input, **kwargs)
   1183 finally:
   1184     if recording_scopes:

File ~/Library/Python/3.9/lib/python/site-packages/transformers/models/t5/modeling_t5.py:1660, in T5ForConditionalGeneration.forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, cross_attn_head_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
   1657         decoder_attention_mask = decoder_attention_mask.to(self.decoder.first_device)
   1659 # Decode
-> 1660 decoder_outputs = self.decoder(
   1661     input_ids=decoder_input_ids,
   1662     attention_mask=decoder_attention_mask,
   1663     inputs_embeds=decoder_inputs_embeds,
   1664     past_key_values=past_key_values,
   1665     encoder_hidden_states=hidden_states,
   1666     encoder_attention_mask=attention_mask,
   1667     head_mask=decoder_head_mask,
   1668     cross_attn_head_mask=cross_attn_head_mask,
   1669     use_cache=use_cache,
   1670     output_attentions=output_attentions,
   1671     output_hidden_states=output_hidden_states,
   1672     return_dict=return_dict,
   1673 )
   1675 sequence_output = decoder_outputs[0]
   1677 # Set device for model parallelism

File ~/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
   1190 # If we don't have any hooks, we want to skip the rest of the logic in
   1191 # this function, and just call forward.
   1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
   1195 # Do not call functions when jit is used
   1196 full_backward_hooks, non_full_backward_hooks = [], []

File ~/Library/Python/3.9/lib/python/site-packages/torch/nn/modules/module.py:1182, in Module._slow_forward(self, *input, **kwargs)
   1180         recording_scopes = False
   1181 try:
-> 1182     result = self.forward(*input, **kwargs)
   1183 finally:
   1184     if recording_scopes:

File ~/Library/Python/3.9/lib/python/site-packages/transformers/models/t5/modeling_t5.py:949, in T5Stack.forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, cross_attn_head_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
    947 else:
    948     err_msg_prefix = "decoder_" if self.is_decoder else ""
--> 949     raise ValueError(f"You have to specify either {err_msg_prefix}input_ids or {err_msg_prefix}inputs_embeds")
    951 if inputs_embeds is None:
    952     assert self.embed_tokens is not None, "You have to initialize the model with valid token embeddings"

ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds



How should I trace t5 model? Can you provide any example? Thanks
```

@dhrubo-os  have fixed it, am also seeing the same issue

@dhrubo-os  can be fixed  we just need to pass as below 
```
traced_token_predictor = torch.jit.trace(model,
                                         [
                                          input_ids["input_ids"],
                                          input_ids["attention_mask"],
                                          decoder_input_ids["input_ids"]
                                         ])
```
since model second argument is attention_mask its taking decoder_input_ids as None