Failed in predict function after converting xlnet model to onnx format
Author: EricYangCn
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:ubuntu 18.04
- Python version:3.7.9
- PyTorch version (1.7.0):
- Using GPU in script:No
- Using distributed or parallel set-up in script?:No
- onnx                1.8.0
- onnxruntime         1.5.2
- transformers        3.5.1


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 TransfoXL/XLNet: @TevenLeScao

 -->
TransfoXL/XLNet: 
@TevenLeScao
## Information

Model I am using (XLNet):

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.train a pytorch model by XLNetForSequenceClassification successfully
2.convert to onnx format by convert_graph_to_onnx.py with some warning:
python ../convert_graph_to_onnx.py --pipeline sentiment-analysis --framework pt --opset 12 ....

====== Converting model to ONNX ======
ONNX opset version set to: 12
Loading pipeline (model: /home/yhq/onnx/icd_model/9-4, tokenizer: /home/yhq/onnx/icd_model/9-4)
Using framework PyTorch: 1.7.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch'}
Found output output_1 with shape: {1: 'batch', 0: 'sequence'}
Found output output_2 with shape: {1: 'batch', 0: 'sequence'}
Found output output_3 with shape: {1: 'batch', 0: 'sequence'}
Found output output_4 with shape: {1: 'batch', 0: 'sequence'}
Found output output_5 with shape: {1: 'batch', 0: 'sequence'}
Found output output_6 with shape: {1: 'batch', 0: 'sequence'}
Found output output_7 with shape: {1: 'batch', 0: 'sequence'}
Found output output_8 with shape: {1: 'batch', 0: 'sequence'}
Found output output_9 with shape: {1: 'batch', 0: 'sequence'}
Found output output_10 with shape: {1: 'batch', 0: 'sequence'}
Found output output_11 with shape: {1: 'batch', 0: 'sequence'}
Found output output_12 with shape: {1: 'batch', 0: 'sequence'}
Ensuring inputs are in correct order
mems is not present in the generated input list.
Generated inputs order: ['input_ids', 'attention_mask']
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/torch/onnx/utils.py:1109: UserWarning: Provided key token_type_ids for dynamic axes is not a valid input/output name
  warnings.warn("Provided key {} for dynamic axes is not a valid input/output name".format(key))
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_xlnet.py:1160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  non_tgt_mask = -torch.eye(qlen).to(attn_mask)
/home/yhq/anaconda3/envs/convert/lib/python3.7/site-packages/transformers/modeling_utils.py:1673: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape == tensor_shape for input_tensor in input_tensors

3.the model I trained works well while predict by loading the model in pytorch format, however failed while loading the model in onnx format,errors as follow:
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'Add_26' Status Message: /onnxruntime_src/onnxruntime/core/providers/cpu/math/element_wise_ops.h:475 void onnxruntime::BroadcastIterator::Init(int64_t, int64_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 6 by 48

my codes:

import onnxruntime,os,torch
from tqdm.auto import tqdm
from torch.utils.data import TensorDataset,SequentialSampler,DataLoader
from transformers import XLNetTokenizer

sess_options = onnxruntime.SessionOptions()
sess_options.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL
sess_options.intra_op_num_threads = 1
session = onnxruntime.InferenceSession(os.path.join(onnx_path,'onnx_model.onnx'), sess_options)
tokenizer = XLNetTokenizer.from_pretrained(onnx_path)
input_ids = []
attention_masks = []
for name in name_list:
    encoded_dict = tokenizer.encode_plus(
        name,
        add_special_tokens=True,
        max_length=max_len,
        pad_to_max_length=True,
        return_attention_mask=True,
        return_tensors='pt',
    )
    input_ids.append(encoded_dict['input_ids'])
    attention_masks.append(encoded_dict['attention_mask'])
input_ids = torch.cat(input_ids, dim=0)
attention_mask = torch.cat(attention_masks, dim=0)

prediction_data = TensorDataset(input_ids,attention_mask)
prediction_sampler = SequentialSampler(prediction_data)
prediction_dataloader = DataLoader(prediction_data, sampler=prediction_sampler, batch_size=1)
for step in tqdm(prediction_dataloader, desc="Predicting"):
    step = tuple(t.detach().cpu().numpy() for t in step)
    ort_inputs = {'input_ids': step[0],
                  'attention_mask': step[1]
                  }
    logits = session.run(None, ort_inputs) #failed


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
I am facing a similar issue. Oddly, it seems to work when input length is 6. Could anyone help out here please? @EricYangCn, were you able to resolve this? Thank you.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.