DistilBert PyTorch to TensorFlow conversion - input sequence length is max 5 tokens for tensorflow
Author: vikul-gupta
Pull request: no

## Environment info
     
- `transformers` version: 3.3.1
- Platform: Linux-5.4.0-1028-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik 
@jplu

## Information

Model I am using (Bert, XLNet ...): DistilBert

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

I load the PyTorch model as a TensorFlow model and then save it to the TensorFlow SavedModel format:
`tf_model = TFDistilBertModel.from_pretrained(model_name, from_pt=True)
tf.saved_model.save(tf_model, path)`

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am attempting to take a Sentence Transformer (ST) model I trained in PyTorch and use it in TensorFlow.js. The code above is specifically converting the DistilBert model ST uses to a TensorFlow SavedModel format. Then, I load the SavedModel format into a TFJS Graph Model format and write the pooling layer. When implementing a forward pass in JS, I noticed that the input sequence length must be 5 tokens (instead of 128). I checked the SavedModel format (.pb file) to rule out an issue from TF to TFJS and noticed that the shapes all have 5 where 128 should be.

## To reproduce

Steps to reproduce the behavior:

Run the above code on the DistilBert model we trained (not very reproducible). These are the contents of the folder:
* config.json
* pytorch_model.bin
* sentence_bert_config.json (this file has the max_seq_length=128 parameter - I tried adding it to config.json, but it doesn't work)
* special_tokens_map.json
* tokenizer_config.json
* vocab.txt


This is the output of the script.
2020-12-08 23:32:19.317202: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-08 23:32:20.677265: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-12-08 23:32:20.677408: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.678158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-12-08 23:32:20.678182: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-08 23:32:20.680072: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-08 23:32:20.681791: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-12-08 23:32:20.682130: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-12-08 23:32:20.684002: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-12-08 23:32:20.685065: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-12-08 23:32:20.688448: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-08 23:32:20.688575: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.689406: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.690132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-12-08 23:32:20.690351: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-08 23:32:20.713215: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2499995000 Hz
2020-12-08 23:32:20.713451: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5572187ea750 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-08 23:32:20.713477: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-08 23:32:20.878776: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.879646: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557218b0f200 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-08 23:32:20.879675: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2020-12-08 23:32:20.879895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.880628: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:00:1e.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2020-12-08 23:32:20.880665: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-08 23:32:20.880697: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-08 23:32:20.880712: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-12-08 23:32:20.880726: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-12-08 23:32:20.880744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-12-08 23:32:20.880761: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-12-08 23:32:20.880780: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-08 23:32:20.880853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.881644: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:20.882347: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-12-08 23:32:20.882390: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-08 23:32:21.437267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-08 23:32:21.437314: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-12-08 23:32:21.437328: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-12-08 23:32:21.437554: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:21.438350: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-12-08 23:32:21.439124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 12367 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5)
2020-12-08 23:32:21.678399: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
2020-12-08 23:32:21.874045: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
All PyTorch model weights were used when initializing TFDistilBertModel.

All the weights of TFDistilBertModel were initialized from the PyTorch model.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFDistilBertModel for predictions without further training.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafd1a97fd0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafd0056e10>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafc051ea10>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafc0535690>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafc04cb3d0>, because it is not built.
WARNING:tensorflow:Skipping full serialization of Keras layer <tensorflow.python.keras.layers.core.Dropout object at 0x7fafc04d9f50>, because it is not built.
WARNING:tensorflow:From /home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.
WARNING:tensorflow:From /home/ubuntu/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
This property should not be used in TensorFlow 2.0, as updates are applied automatically.

## Expected behavior
The TensorFlow model should have a maximum input sequence length of 128, not 5.

Comments:
Hello!

For now, as the TF models are implemented right now, this is the normal behavior, if you want to have a different size you have to set it manually yourself before to create the saved model. It is planned to make the size of the sequence length variable when creating a saved model but we don't know when. If you don't know how to do it, I can show you how :)

Hi Julien! Thanks for your response. Could you please show me how to manually set that?

To do that you can run the following lines:
```
from transformers import TFDistilBertModel, DistilBertTokenizer
tf_model = TFDistilBertModel.from_pretrained(model_name, from_pt=True)
tokenizer = DistilBertTokenizer.from_pretrained(model_name)
inputs = tokenizer("My test sentence", padding="max_length", max_length=128, return_tensors="tf")
tf_model._saved_model_inputs_spec = None
model._set_save_spec(inputs)
tf.saved_model.save(tf_model, path)
```

Thanks!