Invalid Argument for Onnxruntime Inference on GPT2
Author: mihail911
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
I've been following the ipython notebook provided [here](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)

1. Take an off-the-shelf pretrained `gpt` model and export to onnx format using the following invocation: 
```
python convert_graph_to_onnx.py --framework pt --model gpt2 gpt2.onnx
```
2. Run inference on the exported onnx model, following the steps [here](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb). After invoking the appropriate provider, run inference using something like the following
```
model.run(None, {"input_ids": np.array([blah]), "token_type_ids": np.array([blah]), "attention_mask": np.array([blah])
```
Note above, `blah` is replaced with actual data. 

After invoking the above, I get the error:
```
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids
```

## Expected behavior

I would expect this to work successfully. 

My hypothesis is that the `convert_graph_to_onnx.py` is not exporting all the inputs from the `gpt2` model. In particular in line 43-48: 
```
for arg_name in model_args_name[1:]:  # start at index 1 to skip "self" argument
        if arg_name in input_names:
            ordered_input_names.append(arg_name)
            model_args.append(tokens[arg_name])
        else:
            break
```

`model_args` is only populated with `input_ids` because the order of arguments in the `forward` method of `gpt2` is `input_ids, past, attention_mask, token_type_ids` so the for loop breaks early. 

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: Commit 0e1869cc286d607f1598506be7bd1312b76ca82c
- Onnxruntime: 1.3.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0+cu101
- Using GPU in script?: Yes

Thanks for your help! @mfuntowicz @tianleiwu 


Comments:
Assigning @mfuntowicz, the king of the onnx conversion!

@mihail911, do you need attention_mask and token_type_ids in input? If not, you can inference the exported model like the following:
      model.run(None, {"input_ids": np.array([blah])})

GPT-2 attention is unidirectional (right attends to left). User need not provide attention mask (at least for batch_size=1) and token_type_ids (Assume that all words have token type id=0).

For GPT-2, it is recommended to export model with past to get better performance. Currently, convert_graph_to_onnx.py cannot export past. You can use a custom script to do that. Here is an [example]( https://github.com/microsoft/onnxruntime/blob/7c8e1580a13ce333e47a41146bccfc90b3a70db5/onnxruntime/python/tools/transformers/benchmark_gpt2.py#L246). Note that optimization for past state is ongoing, and it will be available in onnxruntime nightly build sometime next week.

thanks for the prompt response @tianleiwu! 

I agree that gpt2 doesn't strictly require the other parameters, but if I have a model that was trained using the token_type_id params because of having particularly formatted inputs, then not providing them at inference time may lead to decreased performance. 

Is there a way to provide them anyway?

@mihail911, here is example script to export model with token_type_ids (but without past input):
```
import torch
from transformers import (GPT2Config, GPT2Model, GPT2Tokenizer)

# use_cache is True by default in GPT2Model. Here we wrap a class to disable past state output.
class GPT2ModelNoPastState(GPT2Model):
    def __init__(self, config):
        super().__init__(config)

    def forward(self, input_ids, attention_mask, token_type_ids):
        return super().forward(input_ids, past=None, attention_mask=attention_mask, token_type_ids=token_type_ids, use_cache=False)

model_name="gpt2"
config = GPT2Config.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)
model = GPT2ModelNoPastState.from_pretrained(model_name)

example_inputs = tokenizer.encode_plus("This is a sample input", return_tensors="pt")
example_outputs = model(**example_inputs)

input_names = ['input_ids',  'attention_mask',  'token_type_ids']
output_names=["output_1"]
dynamic_axes={'input_ids': {0: 'batch_size', 1: 'seq_len'}, 'attention_mask': {0: 'batch_size', 1: 'seq_len'}, 'token_type_ids': {0: 'batch_size', 1: 'seq_len'}, 'output_1': {0: 'batch_size', 1: 'seq_len'}}
output_path="gpt2.onnx"
torch.onnx.export(model=model,
                  args=(example_inputs[input_names[0]], example_inputs[input_names[1]], example_inputs[input_names[2]]),
                  f=output_path,
                  input_names=input_names,
                  output_names=output_names,
                  example_outputs=example_outputs,
                  dynamic_axes=dynamic_axes,
                  do_constant_folding=True,
                  opset_version=11,
                  use_external_data_format=False)
```

BTW, I noticed that the token type use same embedding table as word embedding:
https://github.com/huggingface/transformers/blob/c58e6c129a153ca1a5021e5d7e642d00bf011e20/src/transformers/modeling_gpt2.py#L465-L469
This looks like a bug. You might try to fix this if you want to get benefit from token_type_ids input.

Thanks for the detailed follow-up @tianleiwu! 

I tried executing your code and I found that the dimensions of the output seemed incorrect. The output ended up being `(batch_size, seq_length, hidden_dim)` rather than the dimensions of the prediction scores when you run the forward pass of the GPT2 model (`(batch_size, seq_length, config.vocab_size)`).

This was the case even if I didn't explicitly provide the `output_names` or the dimensions in the `dynamic_axes` (i.e. I set `output_names=None`)

Do you happen to know why that's the case?

@mihail911, it is expected that last dimension of first output (last_hidden_state) is hidden size as documented in code:
https://github.com/huggingface/transformers/blob/a139d1a1602ee72ca98d5e0412efbd68f746d2c8/src/transformers/modeling_gpt2.py#L383

If you want prediction scores, you can try export GPT2LMHeadModel instead of GPT2Model.


@tianleiwu You are absolutely right. I accidentally missed that. 

This works now -- thanks for all your help!