Tensorflow to Onnx change batch and sequence size
Author: nyoungstudios
Pull request: no

### Feature request

Add the ability to set the batch size and sequence length when converting a model to Onnx rather than it always defaulting to a batch size of 2 and sequence length of 8. Here is the example code that I have by importing `OnnxConfig` and setting it's `default_fixed_batch` and `default_fixed_sequence` before exporting the model.

```python
import tensorflow as tf

from transformers import (
    DistilBertTokenizerFast,
    TFAutoModelForSequenceClassification,
)
from transformers.onnx import export, OnnxConfig
from transformers.models.distilbert import DistilBertOnnxConfig

import onnxruntime
import numpy as np
from pathlib import Path
import tempfile

checkpoint = "distilbert-base-uncased"
tokenizer = DistilBertTokenizerFast.from_pretrained(checkpoint)

# you would load your trained model from the folder path here, but for demo purposes, we will load the transformer checkpoint
model = TFAutoModelForSequenceClassification.from_pretrained(checkpoint)

onnx_config = DistilBertOnnxConfig(model.config, task="sequence-classification")

# current solution for setting the Onnx model's batch size and sequence length
OnnxConfig.default_fixed_batch = 1
OnnxConfig.default_fixed_sequence = 128

# create output folder/filename
save_folder = tempfile.mkdtemp()
onnx_model_path = Path(save_folder).joinpath("model.onnx")

# convert to onnx
onnx_input_names, onnx_output_names = export(tokenizer, model, onnx_config, onnx_config.default_onnx_opset, onnx_model_path)

# predict
session = onnxruntime.InferenceSession(str(onnx_model_path), providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'])
onnx_inputs = tokenizer(["test string"],
                        max_length=128,
                        truncation=True,
                        padding="max_length",
                        return_tensors="np")
onnx_inputs = {k: v.astype(np.int32) for k, v in onnx_inputs.items()}
outputs = session.run(None, input_feed=onnx_inputs)
```

### Motivation

To make it easier to set the batch size and sequence length when converting a model to Onnx. Rather than importing `OnnxConfig` and setting it's `default_fixed_batch` and `default_fixed_sequence` before exporting the model, I think the OnnxConfig class could have kwargs for batch size and sequence length.

### Your contribution

I could create a PR to change the `OnnxConfig` class signature to something like this:

```python
def __init__(self, config: "PretrainedConfig", task: str = "default", patching_specs: List[PatchingSpec] = None, batch_size: int = 2, sequence_length: int = 8):
```

And update the `default_sequence_length` and `default_batch_size` property functions to return the variable set from the `__init__` function

Comments:
Hi @nyoungstudios , I don't get the goal of this feature because when you convert a model to onnx you want to set the `batch_size` to dynamic and the `sequence_length` needs to fit the model config. This way people can use the model with any `batch_size` and the right model `sequence_length`. 

The `batch_size=2` and `sequence_length=8` are here for creating a dummy input which is required by ONNX for conversion. It allows onnx to understand the flow though the model layers.

The conversion is also automated with the `python -m transformers.onnx --model=distilbert-base-uncased feature=sequence-classification onnx/`.

@ChainYo maybe I am missing something, but the code sample I have calls the same functions as in the transformers.onnx package cli command. And trying to run an inference without matching the dimensions will return an error like this:

with this input
```python
onnx_inputs = tokenizer(["test string"],
                        truncation=True,
                        return_tensors="np")
onnx_inputs = {k: v.astype(np.int32) for k, v in onnx_inputs.items()}
print(onnx_inputs)
# {'attention_mask': array([[1, 1, 1, 1]], dtype=int32), 'input_ids': array([[ 101, 3231, 5164,  102]], dtype=int32)}
```
Returns this traceback
```python-traceback
InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: input_ids for the following indices
 index: 0 Got: 1 Expected: 2
 index: 1 Got: 4 Expected: 8
 Please fix either the inputs or the model.
```

> @ChainYo maybe I am missing something, but the code sample I have calls the same functions as in the transformers.onnx package cli command.

It doesn't run the onnx checker function and the validating function to validate that the outputs are the same as the pytorch model.

Oh and another thing I just noticed, you are using a `base` model but with `sequence-classification` task. You need to finetune it or get an already fine-tuned model to be able to use it for sequence-classification.



yes, I did also run this Onnx output model checker
```python
from transformers.utils import logging
logger = logging.get_logger("transformers.onnx")  # pylint: disable=invalid-name
logger.setLevel(logging.INFO)
validate_model_outputs(onnx_config, tokenizer, model, onnx_model_path, onnx_output_names, onnx_config.atol_for_validation)
# Validating ONNX model...
#	-[✓] ONNX model output names match reference model ({'logits'})
#	- Validating ONNX Model output "logits":
#		-[✓] (2, 2) matches (2, 2)
#		-[✓] all values close (atol: 1e-05)
```

But that isn't the problem. Also, I am using Tensorflow not PyTorch. And I am just using the base model here as an example. I have fine tuned the model to make predictions, but that doesn't seem to be relevant to the problem here.

The traceback shows that it is expecting a batch size of 2 and a sequence size of 8. Here I provided a batch size of 1 and a sequence size of 4.

> The traceback shows that it is expecting a batch size of 2 and a sequence size of 8. Here I provided a batch size of 1 and a sequence size of 4.

Oh man, I didn't see it was a feature request, I'm so sorry! I though you were facing a bug while converting your model to onnx :weary:. 

That could be nice to add this yes! Even if doing it via the command line is easier and works like a breeze.

No problem, I can work on a pr for this in my free time. I did try running this example but with PyTorch and was able to use the Onnx model without a fixed sequence length or batch size. Curious to know if you know if Tensorflow to Onnx supports dynamic sequence length and batch size like PyTorch to Onnx? If not, I think I might try to explore seeing that that is any option before writing code to have Tensorflow to Onnx support different fixed sizes.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.