TorchScript warning
Author: LemonMi
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (**Bert**, XLNet ...):

Language I am using the model on (English, **Chinese** ...):

The problem arises when using:
* [yes] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [yes] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## Details
when i use official script to create serializable and optimizable models from PyTorch, i meed these warnings.
https://huggingface.co/transformers/torchscript.html

I set torchscript=True, it seems doesn't work. And when i use torchscript in my model, the results seems different from the results before i serialize my model.


> /opt/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:955: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  buffered_token_type_ids = self.embeddings.token_type_ids[:, :seq_length]
/opt/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py:201: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  position_ids = self.position_ids[:, past_key_values_length : seq_length + past_key_values_length]
/opt/anaconda3/envs/py37/lib/python3.7/site-packages/transformers/modeling_utils.py:2169: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  input_tensor.shape[chunk_dim] == tensor_shape for input_tensor in input_tensors




## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version:4.10.0
- Platform:linux
- Python version:3.7.11
- PyTorch version (GPU?):1.7.1(cpu)
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):

## Checklist

- [ yes] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ yes] I checked if a related official extension example runs on my machine.




Comments:
Hello! Do you have a reproducer showing different results when using torchscript?

> Hello! Do you have a reproducer showing different results when using torchscript?

Sorry It's my mistake. This warning doesn't affect the results. Thank you for your answer. 

But i meet another problem. When i load a torchscript model, and feed same input for 100 times, the time cost is different. It cost more at beginning. 

Code like this:
```
model.to(device)
input_ids = input_ids.to(device)
attention_mask = attention_mask.to(device)
token_type_ids = token_type_ids.to(device)

for i in range(100):
    begin = time.time()
    model(input_ids, attention_mask, token_type_ids)
    print(time.time()-begin)

```

The time cost like that:

> 0.18257904052734375
0.1774275302886963
0.24630260467529297
1.8313817977905273
0.008061647415161133
0.007055997848510742
0.006993532180786133
0.007005929946899414
0.00698542594909668
0.006965160369873047
0.0069713592529296875
0.006967782974243164
...

Is it the feature of torchscript?  It seems weired..

I suspect this is because torchscript is tracing your model the first times, hence why it takes longer. Once the tracing is done, the times should stabilize.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.