Inference using ONNX model exported by transformers.onnx raises [ONNXRuntimeError] INVALID_ARGUMENT : Unexpected input data type
Author: Khreas
Pull request: no

Hello !

I am currently encountering a strange behaviour when using a ONNX model generated by the `transformers.onnx` package on an actual documentation example, which can be found here: https://huggingface.co/docs/transformers/serialization#exporting-a-model-to-onnx

I have attached all relevant info I could think of, but don't hesitate to tell me if you need anything else!

## Environment info

- `transformers` version: 4.16.0
- Platform: Windows-10-10.0.19042-SP0
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.0+cpu (False)
- ONNX version: 1.10.2
- ONNX Runtime version: 1.10.0
- ONNX Runtime tools version: 1.7.0
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

From the suggested list, I think that @sgugger, @patil-suraj are the correct persons to tag.

## Information

When exporting a model to onnx using the CLI command `python -m transformers.onnx --model=distilbert-base-uncased onnx/`, the export works fine but the snippet included in the doc regarding actual inference results in the following error:

``[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(int32)) , expected: (tensor(int64))``

Here is the snippet that produces the error:

```python

from transformers import AutoTokenizer
from onnxruntime import InferenceSession

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
session = InferenceSession("onnx/model.onnx")
# ONNX Runtime expects NumPy arrays as input
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

I guess that this is solely a question of tensor format, but since this is an example [from the documentation](https://huggingface.co/docs/transformers/serialization#exporting-a-model-to-onnx) I thought it would be valuable to add this here.

---

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
* [X] N/A


## To reproduce

1. Install `transformers[onnx]` through pip
2. Export the distilbert-base-uncased model to the ONNX format using the following CLI command: 

`python -m transformers.onnx --model=distilbert-base-uncased onnx/`

3. Execute the following snippet to perform inference:

```python

from transformers import AutoTokenizer
from onnxruntime import InferenceSession

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
session = InferenceSession("onnx/model.onnx")
# ONNX Runtime expects NumPy arrays as input
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

3. Witness the following error:

```python
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Unexpected input data type. Actual: (tensor(int32)) , expected: (tensor(int64))
```

## Expected behavior

Normally, the last line of code above should output the actual last hidden state of the model rather than raising an error.

Thanks for your help!

Comments:
For the record, I forgot to mention that the issue can be easily fixed by converting inputs to numpy int64 as the error states. I opened this issue since the snippet is a direct example from the documentation and it could be a good idea to modify the snippet there.

Here's a quick and dirty fix:

```python
import numpy as np
from transformers import AutoTokenizer
from onnxruntime import InferenceSession

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
session = InferenceSession("onnx/model.onnx")
# ONNX Runtime expects NumPy arrays as input
inputs = tokenizer("Using DistilBERT with ONNX Runtime!", return_tensors="np")
inputs = {k: v.astype(np.int64) for k, v in inputs.items()}
outputs = session.run(output_names=["last_hidden_state"], input_feed=dict(inputs))
```

cc @lewtun 

Hi @Khreas thank you for raising this issue and writing a detailed report with a potential solution!

As far as I can tell, this might be a problem with Windows rather than the code snippet itself. I've tested it on macOS and Ubuntu, but cannot reproduce your error - see this [Colab](https://colab.research.google.com/drive/1hGpYj08yK0SteBYLptWwq9e2E1qRwOUf?usp=sharing) for a working example.

In particular, the data type for both the `input_ids` and `attention_mask` returned by the tokenizer is `int64` on these OS':

```python
# Returns (dtype('int64'), dtype('int64'))
inputs.input_ids.dtype, inputs.attention_mask.dtype
```

Could you please check the data types of `input_ids` and `attention_mask` on your system? If these are `int32` by default on Windows, then we'll have to debug the problem at the tokenizer level :)

Edit: I just found this [SO](https://stackoverflow.com/questions/36278590/numpy-array-dtype-is-coming-as-int32-by-default-in-a-windows-10-64-bit-machine) question which seems relevant:

> Default integer type np.int_ is C long:
>
> http://docs.scipy.org/doc/numpy-1.10.1/user/basics.types.html
> But C long is int32 in win64.
>
> https://msdn.microsoft.com/en-us/library/9c3yd98k.aspx
>
> This is kind of a weirdness of the win64 platform.

Are you running on a 64-bit architecture?

Hi @lewtun! Thanks for the quick reply. :)

I ran the following simple code to assess the dtypes:

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
inputs = tokenizer("Is this real life?")

print(f"Type of input_ids and attention masks: {type(inputs['input_ids'])} {type(inputs['attention_mask'])}")
>>> 'Type of input_ids and attention masks: <class 'list'> <class 'list'>'
print(f"dtype of converted numpy arrays: {np.array(inputs['input_ids']).dtype} {np.array(inputs['attention_mask']).dtype}")
>>> 'dtype of converted numpy arrays: int32 int32'
```

I can confirm that the `dtype` of `input_ids` and `attention_mask` are `int32` on my side, but the strange thing is that the tokenizer seems to return lists instead of numpy arrays to begin with.

As you mentionned, I am currently under a Windows machine with a x64 architecture, so the SO post you linked could explain the issue!

Hey @Khreas 

> but the strange thing is that the tokenizer seems to return lists instead of numpy arrays to begin with.

I think you just need to include the `return_tensors="np"` argument when you use the tokenizer :)

> As you mentionned, I am currently under a Windows machine with a x64 architecture, so the SO post you linked could explain the issue!

Thanks for the information. I think in this case, we can't really do much about the issue as it's a low-level OS quirk with this specific version of Windows 😞. 

Hey @lewtun, I guess I didn't have enough sleep haha, thanks for pointing that out. :)

Since the issue can be resolved by a simple cast in one line of code, this isn't a big problem here. At least this issue will give an explanation for people encountering this situation in the future.

I'll close the issue since nothing can be done about it, feel free to reopen it if needed.

Thanks for investigating the issue and have a nice day!