Fix longformer onnx broken export
Author: fxmarty
Pull request: yes

This PR fixes the ONNX export of longformer, that was **silently** broken for several cases:

* the export registers `padding_len > 0` as a constant equal to `True`, hence during inference in the dynamic case `padding_len == 0`, we would still go through the path `padding_len > 0` that would then contain negative indexing making some ONNX nodes fail (gather). This PR fixes the negative indexes.
* the export registers `hidden_states.size(1) == window_overlap * 2:` as a constant equal `True` during the export, hence using the converted ONNX model was failing when the `input_ids` length was strictly greater than `attention_window` (case where the `else` path should be taken). This PR removes the path `hidden_states.size(1) == window_overlap * 2:`, since the other path can handle this case as well.

Had to run `make fix-copies` than modified led model as well.

@michaelbenayoun @lewisbails Where should I add tests for this? Optimum?

## Before submitting
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, I would add the support for Longformer in `optimum`, and the tests there as well.

Thanks for iterating @fxmarty ! 

Since @ydshieh has also approved, gently pinging @sgugger for final approval :)

Hello @sgugger @lewtun  @ ydshieh @fxmarty  : I am stuck with the same issue. Thanks for the fix. This may be a noob question. But wanted to check when would this change be reflected in the PyPi package ? Is it during the next release ? If so do we know when would that be happening ?

@adithya1111 Somewhere in the next week if I remember correctly.

Hello @adithya1111 yes this will be in the next release, feel free to try the main branch meanwhile. In any case, I would advise you to be very careful with the exported ONNX model, and to check that the outputs are on par with PyTorch for your target sequence length. You can possibly edit the ONNX export code if you want to use the exported ONNX with a different sequence length, as explained in my messages above.

For reference: https://github.com/huggingface/optimum/issues/503

Thanks a lot for the comments @fxmarty @ydshieh  . Another question. I used the default parameters when training. So would that mean the Global Attention Mask is None ? I see that we are now setting the global_attention_mask[:, ::2] = 1 . I assume here we are making every second token global. Could this lead to a discrepancy ?

My original models results are `logits=tensor([[[ 0.6209,  0.0719,  0.1107, -0.5316],
         [ 3.0321, -0.2787, -0.6460, -2.5359],
         [ 2.6904,  0.1169, -0.7495, -2.8346],
         [ 0.6474,  0.0761,  0.1041, -0.5438]]]`

And my ONNX converted predictions are `[array([[[ 0.49600145,  0.08062335,  0.12902021, -0.4010917 ],
         [ 3.0400352 , -0.34643874, -0.6276542 , -2.444679  ],
         [ 2.158992  ,  0.02124629, -0.5462518 , -2.094074  ],
         [ 0.6290194 ,  0.06919068,  0.10753635, -0.5197539 ]]],
       dtype=float32)]`

Its close but there are some discrepancies . PFB my config file for my model 

`{
  "_name_or_path": "/opt/ml/input/data/model-base",
  "architectures": [
    "LongformerForTokenClassification"
  ],
  "attention_mode": "longformer",
  "attention_probs_dropout_prob": 0.1,
  "attention_window": [
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2",
    "3": "LABEL_3"
  },
  "ignore_attention_mask": false,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2,
    "LABEL_3": 3
  },
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 4098,
  "model_type": "longformer",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "position_embedding_type": "absolute",
  "sep_token_id": 2,
  "torch_dtype": "float32",
  "transformers_version": "4.9.1",
  "type_vocab_size": 1,
  "use_cache": true,
  "vocab_size": 50265
}
`

@adithya1111 Could you open an issue in https://github.com/huggingface/optimum/issues with a reproducible code? ONNX export through `transformers.onnx` will eventually depend on `optimum.exporters` so we can track the issue there.

Thanks!

Created a new issue. Thanks

@sgugger @fxmarty @ydshieh @lewtun : Does the latest release fix this issue ? 

With the release you will have no error at the export & running the ONNX model. However, following the discussion above (see the closed comments), and as well in https://github.com/huggingface/transformers/issues/20275 , https://github.com/huggingface/optimum/issues/503 , https://github.com/huggingface/optimum/issues/505 , you can expect to have non-meaningful output running the ONNX model with sensibly different sequence length than the example provided to `torch.onnx.export` during the conversion.

This is WIP to add options to customize more the export, refer to https://github.com/huggingface/optimum/pull/522