add onnx support for deberta and debertav2
Author: sam-h-bean
Pull request: yes

# What does this PR do?

Details: Add ONNX Support for DeBERTa and DeBERTaV2.

Issue: https://github.com/huggingface/optimum/issues/207

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@JingyaHuang @ChainYo 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hello @sam-h-bean, thanks for the PR; it looks nice! 

> Hi @sam-h-bean, thanks for contributing! The `OnnxConfig` of DeBERTa-V2 looks good to me. Can you also add it to the test [tests/onnx/test_onnx_v2.py](https://github.com/huggingface/transformers/blob/main/tests/onnx/test_onnx_v2.py#L176) and run
> 
> ```
> RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -s -k "deberta_v2"
> ```
> 
> to confirm that the export goes well?

@JingyaHuang I am seeing a segmentation fault locally but tests pass in CI/CD?

```
10461 ± RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -s -k "deberta_v2"
WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0
rootdir: /Users/marklar/workspace/transformers, configfile: setup.cfg
plugins: xdist-2.5.0, forked-1.4.0, timeout-2.1.0, hypothesis-6.47.0, dash-2.5.0
collected 341 items / 329 deselected / 12 selected                                                                                                                                                                                         

Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 633/633 [00:00<00:00, 130kB/s]
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52.0/52.0 [00:00<00:00, 53.1kB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.33M/2.33M [00:00<00:00, 9.58MB/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
FSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
FSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Fatal Python error: Segmentation fault

Thread 0x000070000d4e3000 (most recent call first):
```

UPDATE:

I realized if you follow the documentation to the letter it only has you install the dev deps which does not include onnxruntime which feels a bit weird. I am now getting a non-segfault error

```
======================================================================================================================================= short test summary info ========================================================================================================================================
FAILED tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_041_deberta_v2_default - AttributeError: 'torch._C.Value' object has no attribute 'dtype'
FAILED tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_041_deberta_v2_default - AttributeError: 'torch._C.Value' object has no attribute 'dtype'
====================================================================================================================== 2 failed, 340 deselected, 35 warnings in 89.38s (0:01:29) =======================================================================================================================
```

SECOND UPDATE:

Still getting some seg faults and some of this dtype error. I was able to run the distilbert tests OK. I see now that CI/CD does not run the slow tests so this just seems to be a serialization issue. I was able to attach a debugger and get into the core torch.export function but the failure is in there so I'm not sure if it makes sense to go deeper in the debugger or if this is just something obvious I'm missing.

I am seeing that [deberta v2 is leveraging some symbolic library using onnx already](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L124-L137). Line 135 is where the crash is happening now

UPDATE:

I tried deleting the symbolic API for the XSoftmax class and rerunning the test, however I think the code is coming from the hub which unfortunately still has that method in it and is crashing the onnx export with
```
E   AssertionError: deberta-v2, default -> Unsupported: ONNX export of Slice with dynamic inputs. DynamicSlice is a deprecated experimental op. Please use statically allocated variables or export to a higher opset version.
```
This is coming directly from the opset9 library. I also tried setting the default opset to 9 in my onnx config object and got the same error. Does it make sense to delete that method from the code in the model hub since the method is unused?

Hi @sam-h-bean, the symbolic function was added intentionally(check #14013) to support ONNX export as XSoftmax is not a natively supported onnx op. Can you try to export Deberta-V2 with an upper opset, eg.  with `opset=15(onnx>=1.10.0)`
```
onnx_inputs, onnx_outputs = export(preprocessor, model, onnx_config, 15, Path(output.name), device=device)
```
And maybe find the minimal opset if possible(currently we set [default it to 11](https://github.com/huggingface/transformers/blob/main/src/transformers/onnx/config.py#L40)).

Keep me posted if it works, then we can make the necessary change on the export.

@sam-h-bean To confirm, can you try this snippet:
```python
from collections import OrderedDict
from typing import Mapping
from pathlib import Path
from transformers.onnx import export
from transformers.onnx import OnnxConfig
from transformers import AutoTokenizer, AutoModel, AutoConfig

class DebertaV2OnnxConfig(OnnxConfig):
    @property
    def inputs(self) -> Mapping[str, Mapping[int, str]]:
        if self.task == "multiple-choice":
            dynamic_axis = {0: "batch", 1: "choice", 2: "sequence"}
        else:
            dynamic_axis = {0: "batch", 1: "sequence"}
        return OrderedDict(
            [
                ("input_ids", dynamic_axis),
                ("attention_mask", dynamic_axis),
            ]
        )
    @property
    def default_onnx_opset(self) -> int:
        return 15

config = AutoConfig.from_pretrained("microsoft/deberta-v3-large")
base_model = AutoModel.from_pretrained("microsoft/deberta-v3-large")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
onnx_config = DebertaV2OnnxConfig(config)
onnx_path = Path("deberta.onnx")
onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)
print(onnx_config.default_onnx_opset)
```

@JingyaHuang I tried the export in a try/except as follows

```python
try:
     onnx_inputs, onnx_outputs = export(tokenizer, base_model, onnx_config, onnx_config.default_onnx_opset, onnx_path)
 except (RuntimeError, ValueError) as e:
     print(onnx_config.default_onnx_opset)
```

And it prints 15 with no onnx model exported 😢 

The exception is
```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/marklar/workspace/transformers/src/transformers/onnx/convert.py", line 335, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
  File "/Users/marklar/workspace/transformers/src/transformers/onnx/convert.py", line 190, in export_pytorch
    onnx_export(
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/__init__.py", line 305, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 118, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 738, in _export
    proto, export_map, val_use_external_data_format = graph._export_onnx(
RuntimeError: ONNX export failed: Couldn't export Python operator XSoftmax
```

UPDATE: I had made some modification of the XSoftmax, once I reverted them I got this exception running your snippet

```bash
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/marklar/workspace/transformers/src/transformers/onnx/convert.py", line 335, in export
    return export_pytorch(preprocessor, model, config, opset, output, tokenizer=tokenizer, device=device)
  File "/Users/marklar/workspace/transformers/src/transformers/onnx/convert.py", line 190, in export_pytorch
    onnx_export(
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/__init__.py", line 305, in export
    return utils.export(model, args, f, export_params, verbose, training,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 118, in export
    _export(model, args, f, export_params, verbose, training, input_names, output_names,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 719, in _export
    _model_to_graph(model, args, verbose, input_names,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 503, in _model_to_graph
    graph = _optimize_graph(graph, operator_export_type,
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 232, in _optimize_graph
    graph = torch._C._jit_pass_onnx(graph, operator_export_type)
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/__init__.py", line 359, in _run_symbolic_method
    return utils._run_symbolic_method(*args, **kwargs)
  File "/Users/marklar/workspace/transformers/venv/lib/python3.9/site-packages/torch/onnx/utils.py", line 846, in _run_symbolic_method
    return symbolic_fn(g, *args)
  File "/Users/marklar/workspace/transformers/src/transformers/models/deberta_v2/modeling_deberta_v2.py", line 135, in symbolic
    output = masked_fill(g, self, r_mask, g.op("Constant", value_t=torch.tensor(torch.finfo(self.dtype).min)))
AttributeError: 'torch._C.Value' object has no attribute 'dtype'
```

Hi @sam-h-bean, 

I merged a PR a few days ago changing deberta symbolic function #17539 (to follow what is planned to be done [here](https://github.com/huggingface/transformers/pull/17306#issuecomment-1144846653)), that seems to break the ONNX export.

I think that the culprit here is the  `torch.finfo(some_tensor.dtype)`  part. Let me try to find a workaround.

Pinging @ydshieh to make him aware of the issue. 

> Hi @sam-h-bean,
> 
> I merged a PR a few days ago changing deberta symbolic function #17539 (to follow what is planned to be done [here](https://github.com/huggingface/transformers/pull/17306#issuecomment-1144846653)), that seems to break the ONNX export.
> 
> I think that the culprit here is the `torch.finfo(some_tensor.dtype)` part. Let me try to find a workaround.
> 
> Pinging @ydshieh to make him aware of the issue.

@michaelbenayoun Yeah it definitely does break. I have changed that line to 
```python
output = masked_fill(g, self, r_mask, g.op("Constant", value_t=torch.tensor(torch.finfo(torch.float32).min)))
```

and it seems to work. I am now facing 
```bash
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids
```

@michaelbenayoun @ydshieh this PR now contains a fix for the symbolic function

I have very little knowledge on this onnx thing. But what I can add here is: we need to understand what `self` actually is in
```python
def symbolic(g, self, mask, dim):
```
My quick search (in a previous PR) shows it should be some input to the model. And if it is a `torch.Tensor`, it should have `dtype`. 

- Maybe it is not input to model?
- Maybe it is, but for some reason, it is not `torch.Tensor` (for example some symbolic tensor etc - I have no idea)

I suggest to set some break points, launch a code that will call this method, and investigate what `self` is.
Please don't merge this PR with `float32.dtype` without any further investigate first, thank you 🙏  
 




> I have very little knowledge on this onnx thing. But what I can add here is: we need to understand what `self` actually is in
> 
> ```python
> def symbolic(g, self, mask, dim):
> ```
> 
> My quick search (in a previous PR) shows it should be some input to the model. And if it is a `torch.Tensor`, it should have `dtype`.
> 
>     * Maybe it is not input to model?
> 
>     * Maybe it is, but for some reason, it is not `torch.Tensor` (for example some symbolic tensor etc - I have no idea)
> 
> 
> I suggest to set some break points, launch a code that will call this method, and investigate what `self` is. Please don't merge this PR with `float32.dtype` without any further investigate first, thank you 🙏

This code previously was 

```python
output = masked_fill(g, self, r_mask, g.op("Constant", value_t=torch.tensor(float("-inf"))))
```

@ydshieh  Then it was changed to this self reference which broke the code. This is now using the built-in torch -inf so the type will be the same. I can also just change it back to `torch.tensor(float("-inf"))` if you would prefer but I imagine this won't make a great deal of difference in practice

See screenshot of the inspection: 
![Screen Shot 2022-06-10 at 5 42 27 PM](https://user-images.githubusercontent.com/43734688/173154926-51813209-b2c4-4998-848e-ed63e8ee5f90.png)

It is some object that is clearly not a tensor.

Here is the value of the value as it is now:
![Screen Shot 2022-06-10 at 5 43 32 PM](https://user-images.githubusercontent.com/43734688/173155020-77f82ecf-66a9-475c-b231-a3f4bef58718.png)

And the value it was before the broken code was merged:
![Screen Shot 2022-06-10 at 5 44 08 PM](https://user-images.githubusercontent.com/43734688/173155067-a1bbad7d-8870-4457-8a79-e0c1ffdb3ceb.png)



We would like to have the same logic applies to the usual model as well as in ONNX here. For the usual PyTorch models, we use the `dtype` of the inputs (which are torch tensors) to determine the large negative values.

I understand we also need to make ONNX work. Let's dive deeper next week.

In the meantime, could you provide the code snippet you used to investigate these values in your above comment?
(I didn't check all your previous comments , so probably it is there already)

If you have some more time for now, would you mind to check what type `self` is, say `type(self)`. You can also check `dir(self)` to see what kind of attributes it has.

It looks like some kind of symbolic tensor (in some sense).

 

![Screen Shot 2022-06-10 at 6 08 57 PM](https://user-images.githubusercontent.com/43734688/173157094-f096352d-0615-44cb-b63d-9a8fcdeea2a7.png)

It seems like some compiled C tensor. 

You can see it has a type but no dtype below
![Screen Shot 2022-06-10 at 6 09 47 PM](https://user-images.githubusercontent.com/43734688/173157160-040977d3-7e6d-4900-81ad-7fddd713b6be.png)



@ydshieh Sorry I did not see your request to get the debugging steps. I hacked the onnx test [here](https://github.com/huggingface/transformers/blob/main/tests/onnx/test_onnx_v2.py#L304) to look like the following
```python
    # @parameterized.expand(_get_models_to_test(PYTORCH_EXPORT_MODELS))
    # @slow
    @require_torch
    @require_vision
    @require_rjieba
    def test_pytorch_export(self):
        self._onnx_export("test_name", "name", "microsoft/deberta-v2-xlarge", "default", DebertaV2OnnxConfig)
```

Then I imported the requisite `DebertaV2OnnxConfig` and set a breakpoint in the  PyCharm debugger [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/deberta_v2/modeling_deberta_v2.py#L135). Then I inspect in the debugger. 

@sam-h-bean 

Thank you for the code snippet. After some inspection, could you try
```python
self.type().dtype()
```
<img width="247" alt="Screenshot 2022-06-11 120746" src="https://user-images.githubusercontent.com/2521628/173183487-feff0617-aa06-4645-b275-93e78e4dfb38.png">




> @sam-h-bean
> 
> Thank you for the code snippet. After some inspection, could you try
> 
> ```python
> self.type().dtype()
> ```
> 
> <img alt="Screenshot 2022-06-11 120746" width="247" src="https://user-images.githubusercontent.com/2521628/173183487-feff0617-aa06-4645-b275-93e78e4dfb38.png">

That works! Thank you for the help, I agree deriving the type directly from the output is much more clean and robust

Pinging @michaelbenayoun for a review, especially for the modifications on the symbolic function.

> @sam-h-bean, thanks for the modifications! Can you do the last check by running the slow tests?

@JingyaHuang The test will not pass as-is. We either need to modify the tokenizer to return only `input_ids` and `attention_mask` when `type_vocab_size` = 0, or we need to run the test to modify the config we initialize the ONNX config to have a non-zero `type_vocab_size`. This is what I was trying to get across in my comment. By default the ONNX graph will be made expecting 2 inputs since the behavior is to ignore `type_token_ids` in DeBERTa. This is contrary to how Electra works which by default has `type_token_ids`

> RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -s -k "deberta_v2"

Hi @sam-h-bean, my mistake, for sure it won't pass. To follow the default setting of the tokenizer, and to have the test passed without adding lines specific to DeBERTa, I would prefer to tailor the inputs according to the `type_vocab_size` to satisfy both scenarios. 

Here is the full test suite passing. @lewtun @sgugger if I could get a review on this it would unblock getting the support into optimum which I sorely need for my production DeBERTa microservice at you.com 

```bash
10571 ± RUN_SLOW=1 pytest tests/onnx/test_onnx_v2.py -k "deberta" -v                                                                                                                          ⏎
===================================================================================== test session starts ======================================================================================
platform darwin -- Python 3.9.10, pytest-7.1.2, pluggy-1.0.0 -- /Users/marklar/workspace/transformers/venv/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/marklar/workspace/transformers/.hypothesis/examples')
rootdir: /Users/marklar/workspace/transformers, configfile: setup.cfg
plugins: xdist-2.5.0, forked-1.4.0, timeout-2.1.0, hypothesis-6.47.0, dash-2.5.0
collected 367 items / 345 deselected / 22 selected                                                                                                                                             

tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_043_deberta_v2_default PASSED                                                                                      [  4%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_044_deberta_v2_masked_lm PASSED                                                                                    [  9%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_045_deberta_v2_multiple_choice PASSED                                                                              [ 13%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_046_deberta_v2_question_answering PASSED                                                                           [ 18%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_047_deberta_v2_sequence_classification PASSED                                                                      [ 22%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_048_deberta_v2_token_classification PASSED                                                                         [ 27%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_049_deberta_default PASSED                                                                                         [ 31%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_050_deberta_masked_lm PASSED                                                                                       [ 36%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_051_deberta_question_answering PASSED                                                                              [ 40%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_052_deberta_sequence_classification PASSED                                                                         [ 45%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_053_deberta_token_classification PASSED                                                                            [ 50%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_043_deberta_v2_default PASSED                                                                              [ 54%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_044_deberta_v2_masked_lm PASSED                                                                            [ 59%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_045_deberta_v2_multiple_choice PASSED                                                                      [ 63%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_046_deberta_v2_question_answering PASSED                                                                   [ 68%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_047_deberta_v2_sequence_classification PASSED                                                              [ 72%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_048_deberta_v2_token_classification PASSED                                                                 [ 77%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_049_deberta_default PASSED                                                                                 [ 81%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_050_deberta_masked_lm PASSED                                                                               [ 86%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_051_deberta_question_answering PASSED                                                                      [ 90%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_052_deberta_sequence_classification PASSED                                                                 [ 95%]
tests/onnx/test_onnx_v2.py::OnnxExportTestCaseV2::test_pytorch_export_on_cuda_053_deberta_token_classification PASSED                                                                    [100%]
```

> Looks great, thanks for iterating it!
> 
> It probably makes sense to filter out the invalid inputs and I opened a PR on Optimum. But for the moment please remove `token_type_ids` from the inputs of InferenceSession if it doesn't exist in the exported ONNX.

@JingyaHuang I'm not sure what you mean by this. Do you want something beyond removing the inputs in generate dummy inputs? Or is this comment strictly about my personal use of this functionality?

> > Looks great, thanks for iterating it!
> > It probably makes sense to filter out the invalid inputs and I opened a PR on Optimum. But for the moment please remove `token_type_ids` from the inputs of InferenceSession if it doesn't exist in the exported ONNX.
> 
> @JingyaHuang I'm not sure what you mean by this. Do you want something beyond removing the inputs in generate dummy inputs? Or is this comment strictly about my personal use of this functionality?

Hi @sam-h-bean, nothing that you should worry about. Here I mention it for another API([`ORTModelForXXX`](https://github.com/huggingface/optimum/blob/main/optimum/onnxruntime/modeling_ort.py)) in Optimum. I will merge this PR, and then by building transformers from source, you shall be able to leverage Quantization and Graph Optimization features in Optimum. Thank you again for the contribution.

Congratz @sam-h-bean, excellent work. Thanks for adding these configs!!

Can I get a t-shirt 😏 ?