Fail when using pipeline for the inference of DeBERTa-Vx ORTModels
Author: JingyaHuang
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.21.0.dev0
- Platform: Linux-5.4.0-1080-aws-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0+cu102 (True)
```


### Who can help?

@LysandreJik @Narsil

### Reproduction

### Context

The DeBERTa tokenizers output `token_type_ids` by default. However, in the use case of using `transformers.pipeline` for inference of `ORTModelForXXX` in Optimum. Depending on `config.type_vocab_size` the exported IR doesn't always take `token_type_ids` as input, and this will lead to failure as `onnxruntime.InferenceSession` is not tolerant of invalid inputs.

* PR #17617 - Support of DeBERTa onnx
* PR [#225](https://github.com/huggingface/optimum/pull/225) - Discussion in Optimum

```python
from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForSequenceClassification
model = ORTModelForSequenceClassification.from_pretrained("microsoft/deberta-base",from_transformers=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base")
onnx_classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)
text = "Hello, my dog is cute"
pred = onnx_classifier(text)
```

Error Message:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/transformers/pipelines/text_classification.py", line 138, in __call__
    result = super().__call__(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1043, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1050, in run_single
    model_outputs = self.forward(model_inputs, **forward_params)
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/transformers/pipelines/base.py", line 959, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/transformers/pipelines/text_classification.py", line 163, in _forward
    return self.model(**model_inputs)
  File "/home/ubuntu/optimum/optimum/modeling_base.py", line 31, in __call__
    return self.forward(*args, **kwargs)
  File "/home/ubuntu/optimum/optimum/onnxruntime/modeling_ort.py", line 520, in forward
    outputs = self.model.run(None, onnx_inputs)
  File "/home/ubuntu/anaconda3/envs/venv_opt_test/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids
```



### Ideas


To enable Pipeline for DeBERTa-Vxxx ONNX model, I am thinking of configuring `return_token_type_ids` argument in the [`preprocess` method](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/token_classification.py#L195-L200) of `XXXPipeline.preprocess` depending on `self.model.config.type_vocab_size`, which can remove `token_type_ids` from inputs when it is unused. WDYT?



Comments:
Hi @JingyaHuang ,

The use case seems solid. Ideally we should be able to run the code as-is. But as far as I understand it's not really doable because `token_type_ids` **are** used in some models but not others, so finding a all covering solution is tricky.

Am I correct ?

Then, for adding the argument instead of adding `return_token_type_ids` I suggest adding `tokenizer_args` as a dict where you could pass `tokenizer_args={"return_token_type_ids": False}` . We can always think about promoting this particular argument to first class, but it seems that going more explicit is better here WDYT ?



Hi @Narsil,

Exactly, the pipelines work well for many other ort models except for DeBERTa(s). As the exported DeBERTa ONNX model with `token_type_ids` can only be traced by `torch.jit.trace` when `model.config.type_vocab_size`>0. And `token_type_ids` are not traced thus not a valid input when `model.config.type_vocab_size`=0(default), it is definitely tricky.

`tokenizer_kwargs` sounds good to me! We might want to enable users to do something like this:
```python
tokenizer = AutoTokenizer.from_pretrained("{checkpoint}")
model = ORTModelForSequenceClassification.from_pretrained("{checkpoint}")
onnx_classifier = pipeline("text-classification", model=model, tokenizer=tokenizer, return_token_type_ids=False)
text = "Hello, my dog is cute"
pred = onnx_classifier(text)
```
This snippet works already as [`TextClassificationPipeline.preprocess` takes `tokenizer_kwargs` as input](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/text_classification.py#L147), however it is not yet supported for other tasks(e.g. [token-classification](https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/token_classification.py#L193), `FeatureExtractionPipeline`...)

Is it something that we want to apply to other pipelines or there might be some other considerations?


Pinging @mfuntowicz to get his input on wether it's a tracing issue or a pipeline issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.