Tensorflow language modeling example doesn't work with TPU in Colab 
Author: graycatr1
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyTorch version (GPU?): 1.10.0+cu111 (False)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: TPU with Colab

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- Longformer, BigBird: @ydshieh
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@Rocketknight1  @gante


## Information

Model I am using (Bert, XLNet ...): distilbert-base-cased

The problem arises when using:
* [x] the official example scripts: https://github.com/huggingface/transformers/raw/v4.17.0/examples/tensorflow/language-modeling/run_mlm.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] the official example dataset: wikitext 
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Open new Colab notebook session with TPU accelerator. (The example script is able to run normally on CPU in Colab following the same steps below.)
2. Run example language modeling script that matches installed transformer version with
```
!pip install transformers datasets
!wget https://github.com/huggingface/transformers/raw/v4.17.0/examples/tensorflow/language-modeling/run_mlm.py
!python run_mlm.py \
--model_name_or_path distilbert-base-cased \
--output_dir output \
--dataset_name wikitext \
--dataset_config_name wikitext-103-raw-v1
```
4. Model training fails with error message:
```
Traceback (most recent call last):
  File "run_mlm.py", line 562, in <module>
    main()
  File "run_mlm.py", line 537, in main
    callbacks=[SavePretrainedCallback(output_dir=training_args.output_dir)],
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 1191, in _numpy
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InternalError: 6 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
```
5. Full log:
```
2022-03-31 03:54:06.522866: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
WARNING:__main__:We are training on TPU - forcing pad_to_max_length
Downloading builder script: 8.48kB [00:00, 5.75MB/s]       
Downloading metadata: 6.84kB [00:00, 4.88MB/s]       
Downloading and preparing dataset wikitext/wikitext-103-raw-v1 (download: 183.09 MiB, generated: 523.53 MiB, post-processed: Unknown size, total: 706.63 MiB) to /root/.cache/huggingface/datasets/wikitext/wikitext-103-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126...
Downloading data: 100% 192M/192M [00:04<00:00, 44.6MB/s]
Dataset wikitext downloaded and prepared to /root/.cache/huggingface/datasets/wikitext/wikitext-103-raw-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126. Subsequent calls will reuse this data.
100% 3/3 [00:00<00:00, 195.25it/s]
https://huggingface.co/distilbert-base-cased/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpsvndtibf
Downloading: 100% 411/411 [00:00<00:00, 354kB/s]
storing https://huggingface.co/distilbert-base-cased/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/ebe1ea24d11aa664488b8de5b21e33989008ca78f207d4e30ec6350b693f073f.302bfd1b5e031cc1b17796e0b6e5b242ba2045d31d00f97589e12b458ebff27a
creating metadata file for /root/.cache/huggingface/transformers/ebe1ea24d11aa664488b8de5b21e33989008ca78f207d4e30ec6350b693f073f.302bfd1b5e031cc1b17796e0b6e5b242ba2045d31d00f97589e12b458ebff27a
loading configuration file https://huggingface.co/distilbert-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/ebe1ea24d11aa664488b8de5b21e33989008ca78f207d4e30ec6350b693f073f.302bfd1b5e031cc1b17796e0b6e5b242ba2045d31d00f97589e12b458ebff27a
Model config DistilBertConfig {
  "_name_or_path": "distilbert-base-cased",
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.17.0",
  "vocab_size": 28996
}

https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer_config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpq4ymtf69
Downloading: 100% 29.0/29.0 [00:00<00:00, 18.1kB/s]
storing https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer_config.json in cache at /root/.cache/huggingface/transformers/81e970e5e6ec68be12da0f8f3b2f2469c78d579282299a2ea65b4b7441719107.ec5c189f89475aac7d8cbd243960a0655cfadc3d0474da8ff2ed0bf1699c2a5f
creating metadata file for /root/.cache/huggingface/transformers/81e970e5e6ec68be12da0f8f3b2f2469c78d579282299a2ea65b4b7441719107.ec5c189f89475aac7d8cbd243960a0655cfadc3d0474da8ff2ed0bf1699c2a5f
loading configuration file https://huggingface.co/distilbert-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/ebe1ea24d11aa664488b8de5b21e33989008ca78f207d4e30ec6350b693f073f.302bfd1b5e031cc1b17796e0b6e5b242ba2045d31d00f97589e12b458ebff27a
Model config DistilBertConfig {
  "_name_or_path": "distilbert-base-cased",
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.17.0",
  "vocab_size": 28996
}

https://huggingface.co/distilbert-base-cased/resolve/main/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmps9iefq03
Downloading: 100% 208k/208k [00:00<00:00, 1.82MB/s]
storing https://huggingface.co/distilbert-base-cased/resolve/main/vocab.txt in cache at /root/.cache/huggingface/transformers/ba377304984dc63e3ede0e23a938bbbf04d5c3835b66d5bb48343aecca188429.437aa611e89f6fc6675a049d2b5545390adbc617e7d655286421c191d2be2791
creating metadata file for /root/.cache/huggingface/transformers/ba377304984dc63e3ede0e23a938bbbf04d5c3835b66d5bb48343aecca188429.437aa611e89f6fc6675a049d2b5545390adbc617e7d655286421c191d2be2791
https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp5rvetabd
Downloading: 100% 426k/426k [00:00<00:00, 3.12MB/s]
storing https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer.json in cache at /root/.cache/huggingface/transformers/acb5c2138c1f8c84f074b86dafce3631667fccd6efcb1a7ea1320cf75c386a36.3dab63143af66769bbb35e3811f75f7e16b2320e12b7935e216bd6159ce6d9a6
creating metadata file for /root/.cache/huggingface/transformers/acb5c2138c1f8c84f074b86dafce3631667fccd6efcb1a7ea1320cf75c386a36.3dab63143af66769bbb35e3811f75f7e16b2320e12b7935e216bd6159ce6d9a6
loading file https://huggingface.co/distilbert-base-cased/resolve/main/vocab.txt from cache at /root/.cache/huggingface/transformers/ba377304984dc63e3ede0e23a938bbbf04d5c3835b66d5bb48343aecca188429.437aa611e89f6fc6675a049d2b5545390adbc617e7d655286421c191d2be2791
loading file https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/acb5c2138c1f8c84f074b86dafce3631667fccd6efcb1a7ea1320cf75c386a36.3dab63143af66769bbb35e3811f75f7e16b2320e12b7935e216bd6159ce6d9a6
loading file https://huggingface.co/distilbert-base-cased/resolve/main/added_tokens.json from cache at None
loading file https://huggingface.co/distilbert-base-cased/resolve/main/special_tokens_map.json from cache at None
loading file https://huggingface.co/distilbert-base-cased/resolve/main/tokenizer_config.json from cache at /root/.cache/huggingface/transformers/81e970e5e6ec68be12da0f8f3b2f2469c78d579282299a2ea65b4b7441719107.ec5c189f89475aac7d8cbd243960a0655cfadc3d0474da8ff2ed0bf1699c2a5f
loading configuration file https://huggingface.co/distilbert-base-cased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/ebe1ea24d11aa664488b8de5b21e33989008ca78f207d4e30ec6350b693f073f.302bfd1b5e031cc1b17796e0b6e5b242ba2045d31d00f97589e12b458ebff27a
Model config DistilBertConfig {
  "_name_or_path": "distilbert-base-cased",
  "activation": "gelu",
  "attention_dropout": 0.1,
  "dim": 768,
  "dropout": 0.1,
  "hidden_dim": 3072,
  "initializer_range": 0.02,
  "max_position_embeddings": 512,
  "model_type": "distilbert",
  "n_heads": 12,
  "n_layers": 6,
  "output_past": true,
  "pad_token_id": 0,
  "qa_dropout": 0.1,
  "seq_classif_dropout": 0.2,
  "sinusoidal_pos_embds": false,
  "tie_weights_": true,
  "transformers_version": "4.17.0",
  "vocab_size": 28996
}

Running tokenizer on every text in dataset:  60% 3/5 [00:00<00:00,  5.09ba/s]Token indices sequence length is longer than the specified maximum sequence length for this model (546 > 512). Running this sequence through the model will result in indexing errors
Running tokenizer on every text in dataset: 100% 5/5 [00:00<00:00,  6.13ba/s]
Running tokenizer on every text in dataset: 100% 1802/1802 [05:23<00:00,  5.57ba/s]
Running tokenizer on every text in dataset: 100% 4/4 [00:00<00:00,  6.25ba/s]
Grouping texts in chunks of 512: 100% 5/5 [00:00<00:00,  8.99ba/s]
Grouping texts in chunks of 512: 100% 1802/1802 [03:46<00:00,  7.95ba/s]
Grouping texts in chunks of 512: 100% 4/4 [00:00<00:00,  7.70ba/s]
INFO:__main__:Validation file not found: using 5% of the dataset as validation as provided in data_args
INFO:__main__:Sample 167621 of the training set: {'input_ids': [23782, 1118, 170, 10268, 1210, 1553, 1768, 2273, 1118, 26835, 1389, 1107, 1103, 1509, 5251, 119, 102, 101, 102, 101, 134, 134, 134, 1305, 1264, 134, 134, 134, 102, 101, 102, 101, 1153, 1189, 1123, 1569, 1264, 1963, 1107, 1384, 1165, 1131, 3042, 1107, 1103, 146, 2924, 26447, 26298, 2348, 119, 1153, 1108, 2700, 1106, 4248, 1754, 1120, 1103, 1371, 3396, 3854, 2348, 1107, 1803, 117, 1141, 1104, 1565, 2139, 1150, 1307, 1111, 1103, 4317, 2883, 4553, 6838, 1107, 1103, 160, 2249, 2924, 13360, 119, 1130, 1351, 1333, 117, 1131, 1307, 1107, 170, 1210, 137, 118, 137, 1342, 2774, 1326, 1222, 1860, 119, 1130, 1333, 117, 1131, 1108, 170, 1420, 1104, 1103, 1264, 1115, 1307, 1107, 1103, 13586, 1635, 119, 1153, 2533, 1754, 1120, 1103, 1333, 1291, 2708, 1187, 1123, 1264, 1845, 2223, 119, 102, 101, 102, 101, 134, 134, 134, 134, 16541, 134, 134, 134, 134, 102, 101, 102, 101, 1153, 1108, 1226, 1104, 1103, 4394, 3086, 137, 118, 137, 2183, 1754, 1535, 112, 188, 1569, 14980, 3163, 1264, 117, 1227, 1112, 1103, 144, 18498, 1733, 117, 1120, 1103, 1369, 2659, 16541, 119, 1430, 1264, 2378, 1803, 4389, 137, 118, 137, 3862, 1107, 6957, 1147, 3086, 119, 1153, 1163, 1104, 1123, 1264, 112, 188, 1369, 2099, 117, 107, 1284, 1589, 1487, 1112, 170, 1264, 1541, 1218, 1105, 1412, 3086, 1110, 170, 4755, 1106, 170, 1974, 1104, 1662, 1250, 1105, 13314, 119, 107, 102, 101, 1130, 1357, 1349, 117, 1131, 1108, 1417, 1112, 1226, 1104, 1103, 2682, 1569, 4322, 1115, 1156, 4845, 1120, 1103, 19099, 6045, 2348, 1111, 1103, 1368, 2659, 16541, 1107, 1498, 119, 1153, 1108, 1103, 3495, 1104, 1103, 144, 18498, 1733, 1120, 1103, 1368, 2659, 16541, 119, 1130, 1103, 2284, 3086, 1342, 1222, 1860, 117, 1131, 1307, 1492, 131, 5507, 1904, 119, 1430, 1264, 1575, 3140, 137, 118, 137, 4650, 117, 1133, 2829, 170, 2878, 3086, 119, 1153, 2297, 122, 1553, 1105, 1125, 1300, 11174, 1107, 1103, 1342, 119, 102, 101, 102, 101, 134, 134, 2825, 7745, 1158, 134, 134, 102, 101, 102, 101, 1109, 144, 18498, 1733, 2604, 1106, 7044, 1111, 1103, 1446, 2659, 16541, 1107, 5470, 1260, 11502, 119, 26835, 1389, 1261, 1146, 21291, 1158, 117, 8692, 1118, 10906, 1513, 1318, 1279, 117, 1150, 2533, 1754, 1120, 1103, 1924, 2659, 2932, 1107, 7120, 119, 1556, 1123, 1302, 21551, 6439, 1264, 137, 118, 137, 16195, 1121, 12556, 12805, 5326, 2822, 117, 4548, 117, 1131, 1281, 2284, 1107, 1103, 23994, 13362, 159, 11964, 2260, 1306, 1105, 1103, 23994, 13362, 159, 1545, 6087, 1306, 4278, 1107, 1120, 1103, 4191, 2271, 159, 1161, 112, 170, 1291, 13995, 1105, 1998, 15782, 1116, 2708, 1120, 2161, 14812, 22839, 1113, 1103, 13122, 3331, 119, 102, 101, 102, 101, 102, 101, 134, 141, 28200, 11936, 12253, 25021, 8127, 2145, 14967, 134, 102, 101, 102, 101, 141, 28200, 11936, 12253, 25021, 8127, 2145, 14967, 1110, 170, 2865, 3695, 1388, 1120, 144, 1197, 17176, 2511, 1233, 17945, 19610, 1107, 9773, 117, 4323, 119, 1135, 2923, 1104, 1126, 8246, 19681, 1709, 6158, 117, 170, 1526, 1402, 1105, 1126, 9287, 2854, 23444, 783, 144, 1197, 17176, 2511, 8897, 1424], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}.
INFO:__main__:Sample 29184 of the training set: {'input_ids': [1104, 1300, 1904, 1105, 7423, 3071, 119, 1109, 1461, 112, 188, 4017, 1127, 1637, 1118, 13921, 138, 1197, 19977, 1161, 1105, 1103, 1390, 1108, 2766, 1118, 6242, 13428, 117, 1150, 1145, 1589, 1114, 138, 1197, 19977, 1161, 1113, 1117, 8281, 2362, 1312, 117, 17533, 8005, 119, 1109, 1461, 1108, 1666, 1118, 2499, 22104, 117, 4317, 6049, 1105, 13428, 1114, 2509, 1494, 1121, 138, 1197, 19977, 1161, 119, 6049, 1145, 1307, 1103, 3651, 1105, 6659, 7789, 1105, 1103, 2753, 132, 22104, 8630, 1103, 7505, 1105, 6316, 5349, 117, 1105, 3895, 8782, 1116, 1307, 1103, 3267, 119, 107, 154, 6592, 12640, 107, 1110, 1982, 1118, 138, 1197, 19977, 1161, 1114, 2509, 3582, 2172, 1121, 6242, 13428, 119, 1109, 1461, 1108, 1802, 1206, 1421, 8522, 1107, 4916, 131, 1109, 157, 12635, 7043, 117, 20984, 1116, 111, 14473, 1116, 117, 13784, 3982, 1324, 6125, 117, 1109, 15375, 9977, 140, 10587, 3464, 1105, 15375, 6935, 132, 1105, 1107, 8125, 21596, 2528, 5406, 1107, 2470, 1392, 119, 107, 154, 6592, 12640, 107, 1108, 3216, 1107, 6523, 4419, 4157, 5406, 1107, 1203, 1365, 1392, 1118, 10829, 144, 21536, 24493, 1182, 117, 1105, 20881, 1118, 20868, 1186, 11637, 2879, 1120, 1103, 8028, 1953, 5406, 1113, 1115, 1331, 119, 138, 1197, 19977, 1161, 6454, 1115, 107, 112, 154, 6592, 12640, 112, 1110, 1103, 1362, 1149, 1104, 1103, 2487, 1105, 1103, 3315, 1434, 1118, 9655, 119, 1135, 112, 188, 1103, 4438, 1106, 4835, 1103, 3507, 1137, 1106, 9353, 24921, 1112, 1126, 6171, 1104, 1185, 19760, 9037, 119, 112, 154, 6592, 12640, 112, 1110, 170, 1642, 1114, 1103, 10275, 1104, 1103, 7127, 117, 1110, 1103, 27836, 1104, 1343, 1150, 1322, 1146, 9207, 2041, 119, 107, 102, 101, 102, 101, 134, 134, 19697, 134, 134, 102, 101, 102, 101, 102, 101, 134, 134, 134, 1953, 1888, 134, 134, 134, 102, 101, 102, 101, 1109, 1390, 1888, 1111, 107, 154, 6592, 12640, 107, 1108, 5819, 1107, 5976, 6554, 117, 7263, 119, 1135, 1108, 2002, 1118, 3274, 12381, 117, 1150, 1145, 2002, 1103, 1390, 1888, 1111, 138, 1197, 19977, 1161, 112, 188, 1763, 1423, 107, 12556, 3174, 2572, 107, 117, 1121, 1117, 1312, 17533, 8005, 119, 1130, 1103, 1888, 117, 138, 1197, 19977, 1161, 1110, 1562, 3179, 1103, 4324, 1104, 1103, 1331, 117, 1112, 1218, 1112, 14210, 1104, 1103, 1331, 112, 188, 2275, 1105, 4204, 2041, 117, 1229, 4241, 1103, 4017, 1104, 1103, 1461, 119, 4112, 3265, 19810, 22429, 1103, 1888, 1112, 170, 107, 23160, 1104, 8468, 18886, 2093, 1105, 20203, 1104, 1103, 11835, 1179, 2483, 1107, 1155, 1157, 6228, 12887, 119, 107, 1249, 1104, 130, 1382, 1368, 117, 1103, 1888, 1144, 1680, 122, 137, 119, 137, 124, 1550, 4696, 1113, 7673, 119, 102, 101, 102, 101, 134, 134, 11444, 7276, 1158, 134, 134, 102, 101, 102, 101, 6082, 9133, 102, 101, 107, 154, 6592, 1424, 107, 782, 125, 131, 1479, 102, 101, 102, 101, 134, 134, 25085, 134, 134, 102, 101, 102, 101, 102, 101, 134, 134, 19821, 134, 134, 102, 101, 102, 101, 5055, 1179, 1121, 1103, 1312, 21757, 119, 102, 101, 102, 101, 134, 134, 17443, 1607, 134, 134, 102, 101, 102, 101], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1]}.
INFO:__main__:Sample 6556 of the training set: {'input_ids': [101, 134, 134, 14789, 3420, 1988, 113, 10428, 3823, 782, 1765, 3823, 114, 134, 134, 102, 101, 102, 101, 1130, 170, 1965, 1227, 1112, 1103, 14789, 8931, 117, 2264, 17004, 21305, 21915, 2446, 1149, 170, 4644, 1104, 5851, 1104, 1103, 2264, 1764, 119, 1130, 10428, 3823, 117, 1155, 4037, 117, 8334, 1104, 1147, 6968, 1137, 1934, 1705, 117, 1127, 1189, 7408, 1111, 3990, 1154, 1103, 2264, 2306, 119, 1188, 1815, 4698, 3673, 1105, 4803, 170, 18258, 1965, 1115, 1125, 1151, 2898, 1111, 3944, 117, 1104, 9305, 2400, 5420, 1111, 1764, 1555, 119, 1109, 7762, 1206, 1144, 19756, 1182, 117, 185, 4854, 6617, 6633, 1105, 189, 3464, 2047, 1182, 117, 1134, 1125, 1640, 1561, 20611, 117, 1108, 3184, 2856, 117, 1105, 1103, 3420, 1988, 3113, 6404, 1104, 1927, 10405, 1108, 1687, 119, 8890, 3113, 6404, 1824, 170, 16358, 28008, 2049, 1104, 2302, 6404, 119, 1636, 3420, 1988, 5927, 1127, 3795, 1121, 7888, 4482, 132, 1118, 1142, 1159, 117, 2264, 1137, 2911, 9709, 1125, 1151, 2918, 1193, 3631, 1166, 1277, 1104, 2890, 2413, 1105, 140, 15630, 1233, 18351, 144, 18318, 119, 3935, 1200, 7888, 6404, 117, 1216, 1112, 1103, 1396, 21998, 1116, 1105, 174, 18276, 3052, 117, 1127, 2125, 1118, 1664, 137, 118, 137, 7888, 24544, 26502, 1115, 1180, 8296, 1104, 2880, 22222, 119, 4187, 1106, 1103, 6256, 1104, 1103, 7888, 3420, 5266, 1154, 170, 2049, 1104, 2302, 6404, 3352, 112, 188, 9099, 18520, 1113, 13817, 7915, 15708, 1116, 1111, 1619, 119, 1249, 170, 12394, 14176, 117, 3420, 5266, 1127, 1593, 1579, 4977, 1118, 1126, 4463, 1137, 3407, 1295, 1104, 9310, 13817, 2830, 117, 1134, 1127, 3795, 1121, 1103, 1664, 137, 118, 137, 4037, 1104, 1103, 2813, 112, 188, 6835, 119, 1448, 1227, 5856, 1104, 3420, 5266, 1217, 1824, 1121, 1664, 137, 118, 137, 7888, 7112, 1219, 1142, 1669, 1108, 1103, 3420, 1988, 1115, 1108, 2120, 1107, 1103, 3199, 1104, 26376, 10691, 119, 102, 101, 1258, 21915, 117, 1103, 3420, 5266, 1127, 3795, 3494, 1121, 8676, 4037, 1897, 1190, 4037, 14255, 1116, 13590, 1174, 1111, 4019, 119, 19340, 1338, 1977, 1105, 1127, 3134, 1136, 1121, 4037, 1104, 1103, 1331, 1104, 3352, 2111, 1133, 1121, 1103, 3376, 11408, 1105, 2964, 4281, 4058, 1223, 2264, 1654, 119, 21614, 1199, 1263, 137, 118, 137, 1858, 1764, 8799, 1127, 1705, 1174, 1112, 11461, 117, 1152, 1127, 25346, 1118, 9112, 1114, 2609, 1764, 2541, 1150, 1127, 1107, 2327, 1555, 3229, 1178, 1111, 170, 1374, 7827, 119, 1109, 3420, 5266, 1104, 1103, 1523, 2250, 1915, 117, 6199, 1103, 3420, 5266, 1104, 1103, 1224, 2813, 117, 8941, 2264, 1107, 4247, 117, 1780, 1199, 1353, 1295, 1104, 4252, 137, 118, 137, 13817, 2830, 1127, 1930, 4572, 119, 1109, 2306, 112, 188, 2299, 137, 118, 137, 1634, 3099, 1105, 12392, 1127, 1253, 3795, 7097, 1121, 1103, 2264, 26076, 119, 102, 101, 5472, 2206, 1107, 1103, 2250, 117, 3420, 1988, 5927, 1127, 1185, 2039, 2935, 1113, 170, 13286, 3142, 1106, 3244, 1147, 1657, 119, 3743, 117, 1152, 1460, 2530, 2653, 117, 1105, 1127, 4071, 1118, 1103, 1352, 1113, 170, 4275, 137, 118, 137, 1858, 3142, 119, 1249, 170, 9547, 117, 1764, 4019, 1310, 1106], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'special_tokens_mask': [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}.
https://huggingface.co/distilbert-base-cased/resolve/main/tf_model.h5 not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp_n559bms
Downloading: 100% 338M/338M [00:12<00:00, 29.1MB/s]
storing https://huggingface.co/distilbert-base-cased/resolve/main/tf_model.h5 in cache at /root/.cache/huggingface/transformers/fe773335fbb46b412a9093627b6c3235a69c55bad3bd1deee40813cd0a8d0a82.33c483181ffc4c7cbdd0b733245bcc9b479f14f3b2e892f635fe03f4f3a41495.h5
creating metadata file for /root/.cache/huggingface/transformers/fe773335fbb46b412a9093627b6c3235a69c55bad3bd1deee40813cd0a8d0a82.33c483181ffc4c7cbdd0b733245bcc9b479f14f3b2e892f635fe03f4f3a41495.h5
loading weights file https://huggingface.co/distilbert-base-cased/resolve/main/tf_model.h5 from cache at /root/.cache/huggingface/transformers/fe773335fbb46b412a9093627b6c3235a69c55bad3bd1deee40813cd0a8d0a82.33c483181ffc4c7cbdd0b733245bcc9b479f14f3b2e892f635fe03f4f3a41495.h5
2022-03-31 04:04:45.813436: W tensorflow/python/util/util.cc:368] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
Some layers from the model checkpoint at distilbert-base-cased were not used when initializing TFDistilBertForMaskedLM: ['activation_13']
- This IS expected if you are initializing TFDistilBertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFDistilBertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFDistilBertForMaskedLM were initialized from the model checkpoint at distilbert-base-cased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertForMaskedLM for predictions without further training.
No loss specified in compile() - the model's internal loss computation will be used as the loss. Don't panic - this is a common way to train TensorFlow models in Transformers! Please ensure your labels are passed as keys in the input dict so that they are accessible to the model during the forward pass. To disable this behaviour, please pass a loss argument, or explicitly pass loss=None if you do not want your model to compute a loss.
INFO:__main__:***** Running training *****
INFO:__main__:  Num examples = 223820
INFO:__main__:  Num Epochs = 3.0
INFO:__main__:  Instantaneous batch size per device = 8
INFO:__main__:  Total train batch size = 64
Epoch 1/3
Traceback (most recent call last):
  File "run_mlm.py", line 562, in <module>
    main()
  File "run_mlm.py", line 537, in main
    callbacks=[SavePretrainedCallback(output_dir=training_args.output_dir)],
  File "/usr/local/lib/python3.7/dist-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py", line 1191, in _numpy
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
tensorflow.python.framework.errors_impl.InternalError: 6 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[strided_slice_9/_270]]
  (1) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[strided_slice_24/_294]]
  (2) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[cond/pivot_t/_4/_91]]
  (3) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"F ... [truncated]
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/context.py", line 2611, in async_wait
    context().sync_executors()
  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/eager/context.py", line 694, in sync_executors
    pywrap_tfe.TFE_ContextSyncExecutors(self._context_handle)
tensorflow.python.framework.errors_impl.InternalError: 6 root error(s) found.
  (0) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[strided_slice_9/_270]]
  (1) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[strided_slice_24/_294]]
  (2) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"Failed to pick subchannel","file":"third_party/grpc/src/core/ext/filters/client_channel/client_channel.cc","file_line":3124,"referenced_errors":[{"created":"@1648699546.241659165","description":"failed to connect to all addresses","file":"third_party/grpc/src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}
	 [[{{node StatefulPartitionedCall}}]]
	 [[MultiDeviceIteratorGetNextFromShard]]
Executing non-communication op <MultiDeviceIteratorGetNextFromShard> originally returned UnavailableError, and was replaced by InternalError to avoid invoking TF network error handling logic.
	 [[RemoteCall]]
	 [[IteratorGetNextAsOptional]]
	 [[cond/pivot_t/_4/_91]]
  (3) INTERNAL: {{function_node __inference_train_function_33804}} failed to connect to all addresses
Additional GRPC error information from remote target /job:localhost/replica:0/task:0/device:CPU:0:
:{"created":"@1648699546.241659886","description":"F ... [truncated]
2022-03-31 04:05:46.781395: W ./tensorflow/core/distributed_runtime/eager/destroy_tensor_handle_node.h:57] Ignoring an error encountered when deleting remote tensors handles: INVALID_ARGUMENT: Unable to find the relevant tensor remote_handle: Op ID: 16279, Output num: 0
Additional GRPC error information from remote target /job:worker/replica:0/task:0:
:{"created":"@1648699546.777954132","description":"Error received from peer ipv4:10.44.170.98:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Unable to find the relevant tensor remote_handle: Op ID: 16279, Output num: 0","grpc_status":3}
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Model could start training.
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Hi, we've noted this issue! We'll aim to investigate this next week - making sure we have stable and well-tested TPU support for TF, including for `generate()`, is on our near-term roadmap. I'll keep you updated!

Hi @ryanzlu . We've investigated the issue. This is caused by our `to_tf_dataset()` method streaming data from the underlying dataset in a way that does not work when connecting to remote TPU instances (which is what happens in Colab). We've tested and the script you're using works fine on a "TPU VM" instance on GCS, which allows a direct connection to a TPU.

Our expectation (or at least our hope) is that Google will move to TPU VMs as the main method for accessing TPUs, as the remote TPU node approach is very restrictive. As such, we're probably not going to implement specific methods for remote TPU nodes, as they may become obsolete. We may look into changing our example/notebook scripts to avoid errors on Colab, though, but in the meantime you could try either using a Colab GPU instance or a paid TPU VM instance on GCS.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.