ONNX causal-lm-with-past conversion: attention_mask dtype changed
Author: arampacha
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: Fasle

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
Not sure who should I tag for ONNX related issues @mfuntowicz ? tagging  @patil-suraj as contact for GPT models  

## Information

Model I am using : GPT2, GPTNeo

The problem arises when using the official conversion script (see below).

The tasks I am working on is pre-trained model conversion to ONNX

## To reproduce

Steps to reproduce the behavior:

1. Convert GPT model to ONNX for `causal-lm-with-past` using
```bash
python -m transformers.onnx --model=gpt2 --feature=causal-lm-with-past --atol=5e-4 ./onnx/
```
2. Load the model and check expected input types
```py
import onnx

model = onnx.load("onnx/model.onnx")
inp = model.graph.input
print(f"{inp[0].name}: element type {inp[0].type.tensor_type.elem_type}")
print(f"{inp[-1].name}: element type {inp[-1].type.tensor_type.elem_type}")
```
```
input_ids: element type 7
attention_mask: element type 1
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
The entire process can be reproduced using this colab: https://colab.research.google.com/gist/arampacha/2831d9f6812d2eb4d6d11dc13f76ca49/hf-onnx-attn-mask.ipynb
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The `attention_mask` input should be of integer type (element type 7) not float (element type 1). Because of this `attention_mask` returned by tokenizer should be converted to `float` for inference.

Looks like the unexpected conversion happens because `torch.ones` returns `torch.float32` by default. See this for example https://github.com/huggingface/transformers/blob/9de70f213eb234522095cc9af7b2fac53afc2d87/src/transformers/models/gpt2/configuration_gpt2.py#L266

The problem can be fixed by propagating `attention_mask` dtype:
```py
mask_dtype = ordered_inputs["attention_mask"].dtype
ordered_inputs["attention_mask"] = torch.cat( 
    [ordered_inputs["attention_mask"], torch.ones(batch, past_key_values_length, dtype=mask_dtype)], dim=1
)
```
I can submit PR with a fix but more model classes can be impacted.

Comments:
Thank for this excellent bug report and reproducible Colab @arampacha ! I agree with your analysis and think your solution to propagate the `mask_dtype` to `torch.ones()` should fix the issue.

Before opening a PR, I'd like to get @michaelbenayoun's input as he may have had a specific reason to use the default float values when implementing this 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @arampacha if you're still happy to open a PR with this change, I think that's the best way to proceed on this issue :)

Ok, cool. I'll go on with PR