Onnx converted model output shape not matching with the finetuned model (BUG)
Author: user06039
Pull request: no

I have trained a transformer 3 class classification model, the model used is distilbert-base-uncased.

Now, after training I tried to convert the model to onnx for faster inference using below script.

`
!python convert_graph_to_onnx.py --framework pt --model pt_line-distilbert --tokenizer distilbert-base-uncased --quantize onnx/line-distilbert.onnx`



```
====== Converting model to ONNX ======
ONNX opset version set to: 11
Loading pipeline (model: pt_line-distilbert, tokenizer: distilbert-base-uncased)
Creating folder /home/segments/onnx/linetype
Using framework PyTorch: 1.6.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Ensuring inputs are in correct order
head_mask is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/home/miniconda3/envs/reas/lib/python3.8/site-packages/transformers/modeling_utils.py:1645: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape == tensor_shape for input_tensor in input_tensors

====== Optimizing ONNX model ======
2020-10-30 02:50:55.673526328 [W:onnxruntime:, inference_session.cc:1143 Initialize] Serializing optimized model with Graph Optimization level greater than ORT_ENABLE_EXTENDED. The generated model may contain hardware and execution provider specific optimizations, and should only be used in the same environment the model was optimized for.
Optimized model has been written at /home/segments/onnx/line/line-distilbert-optimized.onnx: ✔
/!\ Optimized model contains hardware specific operators which might not be portable. /!\
As of onnxruntime 1.4.0, models larger than 2GB will fail to quantize due to protobuf constraint.
This limitation will be removed in the next release of onnxruntime.
Warning: onnxruntime.quantization.quantize is deprecated.
         Please use quantize_static for static quantization, quantize_dynamic for dynamic quantization.
Quantized model has been written at /home/segments/onnx/line/line-distilbert-optimized-quantized.onnx: ✔
```

Now, when trying to do inference,

```
options = SessionOptions()
options.intra_op_num_threads = 1
options.execution_mode = ExecutionMode.ORT_SEQUENTIAL

model_path = "onnx/line/line-distilbert-optimized-quantized.onnx"
session = InferenceSession(model_path, options)

tokens = tokenizer.encode_plus("did you get it?", max_length=256, truncation=True, padding='max_length')
tokens = {name: np.atleast_2d(value) for name, value in tokens.items()}
sequence, = session.run(None, tokens)
sequence.shape (1, 256, 768)
```

but my model output should be (1, 3) # 3 class classification model

Any way to fix it? I have gone through this issue :- https://github.com/huggingface/transformers/issues/4825  but there's no proper solution mentioned there.

Comments:
It was working for me when using --pipeline sentiment-analysis

Hi @user06039 did you find a solution for this? Cause I am also facing the same issue.