Running SQuAD 1.0 sample command raises `IndexError`
Author: jaywonchung
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.0-91-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.10.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (automatically through accelerate)
- Using distributed or parallel set-up in script?: No (only one GPU)

### Who can help

@sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): `bert-base-uncased`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Running the example BERT fine-tuning for SQuAD command as-is raises an `IndexError` during the 'postprocessing' stage after evaluation. I also tried the same task with `run_qa_no_trainer.py` and the same error occurs.

## To reproduce

Steps to reproduce the behavior:

1. `conda install -c pytorch pytorch==1.10.1 torchvision==0.11.2 cudatoolkit==11.3.1`
2. `git clone https://github.com/huggingface/transformers` (commit hash: 16d4acbfdb547cb922361ba07a13de12e1503fb8)
3. `cd transformers`
4. `pip install .`
5. `pip install -r examples/pytorch/question-answering/requirements.txt`
6. `cd examples/pytorch/question-answering `
7. Copy-pasted the BERT SQuAD 1.0 fine-tuning command.

Continued in console output:

```console
57add2499a61# cd examples/pytorch/question-answering 
57add2499a61# python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
01/28/2022 23:44:37 - WARNING - __main__ - Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False
01/28/2022 23:44:37 - INFO - __main__ - Training/evaluation parameters TrainingArguments(
_n_gpu=1,
adafactor=False,
adam_beta1=0.9,
adam_beta2=0.999,
adam_epsilon=1e-08,
bf16=False,
bf16_full_eval=False,
dataloader_drop_last=False,
dataloader_num_workers=0,
dataloader_pin_memory=True,
ddp_bucket_cap_mb=None,
ddp_find_unused_parameters=None,
debug=[],
deepspeed=None,
disable_tqdm=False,
do_eval=True,
do_predict=False,
do_train=True,
eval_accumulation_steps=None,
eval_steps=None,
evaluation_strategy=IntervalStrategy.NO,
fp16=False,
fp16_backend=auto,
fp16_full_eval=False,
fp16_opt_level=O1,
gradient_accumulation_steps=1,
gradient_checkpointing=False,
greater_is_better=None,
group_by_length=False,
half_precision_backend=auto,
hub_model_id=None,
hub_strategy=HubStrategy.EVERY_SAVE,
hub_token=<HUB_TOKEN>,
ignore_data_skip=False,
label_names=None,
label_smoothing_factor=0.0,
learning_rate=3e-05,
length_column_name=length,
load_best_model_at_end=False,
local_rank=-1,
log_level=-1,
log_level_replica=-1,
log_on_each_node=True,
logging_dir=/tmp/debug_squad/runs/Jan28_23-44-36_57add2499a61,
logging_first_step=False,
logging_nan_inf_filter=True,
logging_steps=500,
logging_strategy=IntervalStrategy.STEPS,
lr_scheduler_type=SchedulerType.LINEAR,
max_grad_norm=1.0,
max_steps=-1,
metric_for_best_model=None,
mp_parameters=,
no_cuda=False,
num_train_epochs=2.0,
optim=OptimizerNames.ADAMW_HF,
output_dir=/tmp/debug_squad/,
overwrite_output_dir=False,
past_index=-1,
per_device_eval_batch_size=8,
per_device_train_batch_size=12,
prediction_loss_only=False,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
remove_unused_columns=True,
report_to=[],
resume_from_checkpoint=None,
run_name=/tmp/debug_squad/,
save_on_each_node=False,
save_steps=500,
save_strategy=IntervalStrategy.STEPS,
save_total_limit=None,
seed=42,
sharded_ddp=[],
skip_memory_metrics=True,
tf32=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
use_legacy_prediction_loop=False,
warmup_ratio=0.0,
warmup_steps=0,
weight_decay=0.0,
xpu_backend=None,
)
01/28/2022 23:44:37 - INFO - datasets.utils.file_utils - https://raw.githubusercontent.com/huggingface/datasets/1.18.2/datasets/squad/squad.py not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmp13outgcl
Downloading: 5.27kB [00:00, 5.91MB/s]                                                                                                                                                                    
01/28/2022 23:44:37 - INFO - datasets.utils.file_utils - storing https://raw.githubusercontent.com/huggingface/datasets/1.18.2/datasets/squad/squad.py in cache at /root/.cache/huggingface/datasets/downloads/757bbb86e029fd1bb99d893e0eed445a1d920648c31be7fa62f6911432d7d04f.88910a81ad509b864eb2728ed18e25076f86eaa3cd11c5587ab5ceea8903a4bc.py
01/28/2022 23:44:37 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/757bbb86e029fd1bb99d893e0eed445a1d920648c31be7fa62f6911432d7d04f.88910a81ad509b864eb2728ed18e25076f86eaa3cd11c5587ab5ceea8903a4bc.py
01/28/2022 23:44:38 - INFO - datasets.utils.file_utils - https://raw.githubusercontent.com/huggingface/datasets/1.18.2/datasets/squad/dataset_infos.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmp_ol5ulua
Downloading: 2.36kB [00:00, 3.52MB/s]                                                                                                                                                                    
01/28/2022 23:44:38 - INFO - datasets.utils.file_utils - storing https://raw.githubusercontent.com/huggingface/datasets/1.18.2/datasets/squad/dataset_infos.json in cache at /root/.cache/huggingface/datasets/downloads/1966b23d025b3d578359815bf6c1e28cdc39773d03785fbc131ada8108c985d9.36bd0df82ceb24eeafc05394b25c534952fd7b2eaacf2b1f49933a8330f5800b
01/28/2022 23:44:38 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/1966b23d025b3d578359815bf6c1e28cdc39773d03785fbc131ada8108c985d9.36bd0df82ceb24eeafc05394b25c534952fd7b2eaacf2b1f49933a8330f5800b
01/28/2022 23:44:38 - INFO - datasets.builder - No config specified, defaulting to first: squad/plain_text
01/28/2022 23:44:38 - INFO - datasets.info - Loading Dataset Infos from /root/.cache/huggingface/modules/datasets_modules/datasets/squad/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453
01/28/2022 23:44:38 - INFO - datasets.builder - Generating dataset squad (/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)
Downloading and preparing dataset squad/plain_text (download: 33.51 MiB, generated: 85.63 MiB, post-processed: Unknown size, total: 119.14 MiB) to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453...
01/28/2022 23:44:38 - INFO - datasets.builder - Dataset not on Hf google storage. Downloading and preparing it from source
  0%|                                                                                                                                                                              | 0/2 [00:00<?, ?it/s]01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmpeh7ajej8
Downloading: 30.3MB [00:00, 103MB/s]
01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - storing https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json in cache at /root/.cache/huggingface/datasets/downloads/b8bb19735e1bb591510a01cc032f4c9f969bc0eeb081ae1b328cd306f3b24008
01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/b8bb19735e1bb591510a01cc032f4c9f969bc0eeb081ae1b328cd306f3b24008
 50%|███████████████████████████████████████████████████████████████████████████████████                                                                                   | 1/2 [00:01<00:01,  1.08s/it]01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmpfyt6869a
Downloading: 4.85MB [00:00, 92.3MB/s]                                                                                                                                                                    
01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - storing https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json in cache at /root/.cache/huggingface/datasets/downloads/9d5462987ef5f814fe15a369c1724f6ec39a2018b3b6271a9d7d2598686ca2ff
01/28/2022 23:44:39 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/9d5462987ef5f814fe15a369c1724f6ec39a2018b3b6271a9d7d2598686ca2ff
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.52it/s]
01/28/2022 23:44:39 - INFO - datasets.utils.download_manager - Downloading took 0.0 min
01/28/2022 23:44:39 - INFO - datasets.utils.download_manager - Checksum Computation took 0.0 min
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 2341.88it/s]
01/28/2022 23:44:39 - INFO - datasets.utils.info_utils - All the checksums matched successfully for dataset source files
01/28/2022 23:44:39 - INFO - datasets.builder - Generating split train
01/28/2022 23:44:45 - INFO - datasets.builder - Generating split validation
01/28/2022 23:44:46 - INFO - datasets.utils.info_utils - All the splits matched successfully.
Dataset squad downloaded and prepared to /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 728.49it/s]
[INFO|file_utils.py:2140] 2022-01-28 23:44:46,444 >> https://huggingface.co/bert-base-uncased/resolve/main/config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpzb4s_2o4
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 570/570 [00:00<00:00, 1.25MB/s]
[INFO|file_utils.py:2144] 2022-01-28 23:44:46,529 >> storing https://huggingface.co/bert-base-uncased/resolve/main/config.json in cache at /root/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.37395cee442ab11005bcd270f3c34464dc1704b715b5d7d52b1a461abe3b9e4e
[INFO|file_utils.py:2152] 2022-01-28 23:44:46,529 >> creating metadata file for /root/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.37395cee442ab11005bcd270f3c34464dc1704b715b5d7d52b1a461abe3b9e4e
[INFO|configuration_utils.py:644] 2022-01-28 23:44:46,529 >> loading configuration file https://huggingface.co/bert-base-uncased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.37395cee442ab11005bcd270f3c34464dc1704b715b5d7d52b1a461abe3b9e4e
[INFO|configuration_utils.py:680] 2022-01-28 23:44:46,530 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.17.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|file_utils.py:2140] 2022-01-28 23:44:46,615 >> https://huggingface.co/bert-base-uncased/resolve/main/tokenizer_config.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp7u1nz0yk
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28.0/28.0 [00:00<00:00, 63.9kB/s]
[INFO|file_utils.py:2144] 2022-01-28 23:44:46,700 >> storing https://huggingface.co/bert-base-uncased/resolve/main/tokenizer_config.json in cache at /root/.cache/huggingface/transformers/c1d7f0a763fb63861cc08553866f1fc3e5a6f4f07621be277452d26d71303b7e.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79
[INFO|file_utils.py:2152] 2022-01-28 23:44:46,700 >> creating metadata file for /root/.cache/huggingface/transformers/c1d7f0a763fb63861cc08553866f1fc3e5a6f4f07621be277452d26d71303b7e.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79
[INFO|configuration_utils.py:644] 2022-01-28 23:44:46,786 >> loading configuration file https://huggingface.co/bert-base-uncased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.37395cee442ab11005bcd270f3c34464dc1704b715b5d7d52b1a461abe3b9e4e
[INFO|configuration_utils.py:680] 2022-01-28 23:44:46,786 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.17.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|file_utils.py:2140] 2022-01-28 23:44:46,966 >> https://huggingface.co/bert-base-uncased/resolve/main/vocab.txt not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp_ua9i1wf
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 226k/226k [00:00<00:00, 3.43MB/s]
[INFO|file_utils.py:2144] 2022-01-28 23:44:47,123 >> storing https://huggingface.co/bert-base-uncased/resolve/main/vocab.txt in cache at /root/.cache/huggingface/transformers/45c3f7a79a80e1cf0a489e5c62b43f173c15db47864303a55d623bb3c96f72a5.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
[INFO|file_utils.py:2152] 2022-01-28 23:44:47,123 >> creating metadata file for /root/.cache/huggingface/transformers/45c3f7a79a80e1cf0a489e5c62b43f173c15db47864303a55d623bb3c96f72a5.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
[INFO|file_utils.py:2140] 2022-01-28 23:44:47,210 >> https://huggingface.co/bert-base-uncased/resolve/main/tokenizer.json not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmp9xjs8r7s
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 455k/455k [00:00<00:00, 5.41MB/s]
[INFO|file_utils.py:2144] 2022-01-28 23:44:47,386 >> storing https://huggingface.co/bert-base-uncased/resolve/main/tokenizer.json in cache at /root/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4
[INFO|file_utils.py:2152] 2022-01-28 23:44:47,386 >> creating metadata file for /root/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4
[INFO|tokenization_utils_base.py:1771] 2022-01-28 23:44:47,648 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/vocab.txt from cache at /root/.cache/huggingface/transformers/45c3f7a79a80e1cf0a489e5c62b43f173c15db47864303a55d623bb3c96f72a5.d789d64ebfe299b0e416afc4a169632f903f693095b4629a7ea271d5a0cf2c99
[INFO|tokenization_utils_base.py:1771] 2022-01-28 23:44:47,648 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/tokenizer.json from cache at /root/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4
[INFO|tokenization_utils_base.py:1771] 2022-01-28 23:44:47,648 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/added_tokens.json from cache at None
[INFO|tokenization_utils_base.py:1771] 2022-01-28 23:44:47,648 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/special_tokens_map.json from cache at None
[INFO|tokenization_utils_base.py:1771] 2022-01-28 23:44:47,648 >> loading file https://huggingface.co/bert-base-uncased/resolve/main/tokenizer_config.json from cache at /root/.cache/huggingface/transformers/c1d7f0a763fb63861cc08553866f1fc3e5a6f4f07621be277452d26d71303b7e.20430bd8e10ef77a7d2977accefe796051e01bc2fc4aa146bc862997a1a15e79
[INFO|configuration_utils.py:644] 2022-01-28 23:44:47,735 >> loading configuration file https://huggingface.co/bert-base-uncased/resolve/main/config.json from cache at /root/.cache/huggingface/transformers/3c61d016573b14f7f008c02c4e51a366c67ab274726fe2910691e2a761acf43e.37395cee442ab11005bcd270f3c34464dc1704b715b5d7d52b1a461abe3b9e4e
[INFO|configuration_utils.py:680] 2022-01-28 23:44:47,736 >> Model config BertConfig {
  "_name_or_path": "bert-base-uncased",
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "classifier_dropout": null,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "position_embedding_type": "absolute",
  "transformers_version": "4.17.0.dev0",
  "type_vocab_size": 2,
  "use_cache": true,
  "vocab_size": 30522
}

[INFO|file_utils.py:2140] 2022-01-28 23:44:47,914 >> https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin not found in cache or force_download set to True, downloading to /root/.cache/huggingface/transformers/tmpc8j01jc4
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 420M/420M [00:27<00:00, 16.2MB/s]
[INFO|file_utils.py:2144] 2022-01-28 23:45:15,556 >> storing https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin in cache at /root/.cache/huggingface/transformers/a8041bf617d7f94ea26d15e218abd04afc2004805632abc0ed2066aa16d50d04.faf6ea826ae9c5867d12b22257f9877e6b8367890837bd60f7c54a29633f7f2f
[INFO|file_utils.py:2152] 2022-01-28 23:45:15,556 >> creating metadata file for /root/.cache/huggingface/transformers/a8041bf617d7f94ea26d15e218abd04afc2004805632abc0ed2066aa16d50d04.faf6ea826ae9c5867d12b22257f9877e6b8367890837bd60f7c54a29633f7f2f
[INFO|modeling_utils.py:1427] 2022-01-28 23:45:15,557 >> loading weights file https://huggingface.co/bert-base-uncased/resolve/main/pytorch_model.bin from cache at /root/.cache/huggingface/transformers/a8041bf617d7f94ea26d15e218abd04afc2004805632abc0ed2066aa16d50d04.faf6ea826ae9c5867d12b22257f9877e6b8367890837bd60f7c54a29633f7f2f
[WARNING|modeling_utils.py:1685] 2022-01-28 23:45:16,572 >> Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight']
- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[WARNING|modeling_utils.py:1696] 2022-01-28 23:45:16,572 >> Some weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Running tokenizer on train dataset:   0%|                                                                                                                                         | 0/88 [00:00<?, ?ba/s]01/28/2022 23:45:16 - INFO - datasets.arrow_dataset - Caching processed dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-2d1c8b33ff64daaf.arrow
Running tokenizer on train dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.24ba/s]
Running tokenizer on validation dataset:   0%|                                                                                                                                    | 0/11 [00:00<?, ?ba/s]01/28/2022 23:45:44 - INFO - datasets.arrow_dataset - Caching processed dataset at /root/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453/cache-401990ded98120b9.arrow
Running tokenizer on validation dataset: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:27<00:00,  2.46s/ba]
01/28/2022 23:46:11 - INFO - datasets.utils.file_utils - https://raw.githubusercontent.com/huggingface/datasets/1.18.2/metrics/squad/squad.py not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmphozvyc0y
Downloading: 4.51kB [00:00, 6.12MB/s]                                                                                                                                                                    
01/28/2022 23:46:11 - INFO - datasets.utils.file_utils - storing https://raw.githubusercontent.com/huggingface/datasets/1.18.2/metrics/squad/squad.py in cache at /root/.cache/huggingface/datasets/downloads/045393e52f825a7b706c615959251e42aec541d1b37158fb7be61cbbbc20d719.ab3a5db6a587c35cfd241275240e52547dd1e093c74b3ee4f7798d9f6c6304ec.py
01/28/2022 23:46:11 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/045393e52f825a7b706c615959251e42aec541d1b37158fb7be61cbbbc20d719.ab3a5db6a587c35cfd241275240e52547dd1e093c74b3ee4f7798d9f6c6304ec.py
01/28/2022 23:46:11 - INFO - datasets.utils.file_utils - https://raw.githubusercontent.com/huggingface/datasets/1.18.2/metrics/squad/evaluate.py not found in cache or force_download set to True, downloading to /root/.cache/huggingface/datasets/downloads/tmp6fnoaik5
Downloading: 3.31kB [00:00, 5.13MB/s]                                                                                                                                                                    
01/28/2022 23:46:12 - INFO - datasets.utils.file_utils - storing https://raw.githubusercontent.com/huggingface/datasets/1.18.2/metrics/squad/evaluate.py in cache at /root/.cache/huggingface/datasets/downloads/0d7b02f7f3a5192b3530b14825397b3f988ff4b4062efb68456732908a35a909.6f69c3ff9e10aa1cbdc6e91d27e158ea86a785f54a36a9e964ef8b3b78cf3cd6.py
01/28/2022 23:46:12 - INFO - datasets.utils.file_utils - creating metadata file for /root/.cache/huggingface/datasets/downloads/0d7b02f7f3a5192b3530b14825397b3f988ff4b4062efb68456732908a35a909.6f69c3ff9e10aa1cbdc6e91d27e158ea86a785f54a36a9e964ef8b3b78cf3cd6.py
/root/.local/miniconda3/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use thePyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning
  warnings.warn(
[INFO|trainer.py:1252] 2022-01-28 23:46:13,989 >> ***** Running training *****
[INFO|trainer.py:1253] 2022-01-28 23:46:13,989 >>   Num examples = 88524
[INFO|trainer.py:1254] 2022-01-28 23:46:13,989 >>   Num Epochs = 2
[INFO|trainer.py:1255] 2022-01-28 23:46:13,989 >>   Instantaneous batch size per device = 12
[INFO|trainer.py:1256] 2022-01-28 23:46:13,989 >>   Total train batch size (w. parallel, distributed & accumulation) = 12
[INFO|trainer.py:1257] 2022-01-28 23:46:13,989 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1258] 2022-01-28 23:46:13,989 >>   Total optimization steps = 14754
{'loss': 2.483, 'learning_rate': 2.8983326555510372e-05, 'epoch': 0.07}                                                                                                                                  
  3%|█████▎                                                                                                                                                        | 500/14754 [02:21<1:07:16,  3.53it/s][INFO|trainer.py:2103] 2022-01-28 23:48:35,144 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-500
[INFO|configuration_utils.py:430] 2022-01-28 23:48:35,144 >> Configuration saved in /tmp/debug_squad/checkpoint-500/config.json
[INFO|modeling_utils.py:1074] 2022-01-28 23:48:35,626 >> Model weights saved in /tmp/debug_squad/checkpoint-500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-28 23:48:35,627 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-28 23:48:35,627 >> Special tokens file saved in /tmp/debug_squad/checkpoint-500/special_tokens_map.json
{'loss': 1.498, 'learning_rate': 2.796665311102074e-05, 'epoch': 0.14}                                                                                                                                   
  7%|██████████▋                                                                                                                                                  | 1000/14754 [04:44<1:04:54,  3.53it/s][INFO|trainer.py:2103] 2022-01-28 23:50:58,276 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-1000
[INFO|configuration_utils.py:430] 2022-01-28 23:50:58,277 >> Configuration saved in /tmp/debug_squad/checkpoint-1000/config.json
[INFO|modeling_utils.py:1074] 2022-01-28 23:50:58,757 >> Model weights saved in /tmp/debug_squad/checkpoint-1000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-28 23:50:58,757 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-1000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-28 23:50:58,757 >> Special tokens file saved in /tmp/debug_squad/checkpoint-1000/special_tokens_map.json
{'loss': 1.3703, 'learning_rate': 2.694997966653111e-05, 'epoch': 0.2}                                                                                                                                   
 10%|███████████████▉                                                                                                                                             | 1500/14754 [07:07<1:02:41,  3.52it/s][INFO|trainer.py:2103] 2022-01-28 23:53:21,517 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-1500
[INFO|configuration_utils.py:430] 2022-01-28 23:53:21,518 >> Configuration saved in /tmp/debug_squad/checkpoint-1500/config.json
[INFO|modeling_utils.py:1074] 2022-01-28 23:53:22,005 >> Model weights saved in /tmp/debug_squad/checkpoint-1500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-28 23:53:22,006 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-1500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-28 23:53:22,006 >> Special tokens file saved in /tmp/debug_squad/checkpoint-1500/special_tokens_map.json
{'loss': 1.3288, 'learning_rate': 2.593330622204148e-05, 'epoch': 0.27}                                                                                                                                  
 14%|█████████████████████▎                                                                                                                                       | 2000/14754 [09:30<1:00:17,  3.53it/s][INFO|trainer.py:2103] 2022-01-28 23:55:44,792 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-2000
[INFO|configuration_utils.py:430] 2022-01-28 23:55:44,793 >> Configuration saved in /tmp/debug_squad/checkpoint-2000/config.json
[INFO|modeling_utils.py:1074] 2022-01-28 23:55:45,272 >> Model weights saved in /tmp/debug_squad/checkpoint-2000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-28 23:55:45,272 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-2000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-28 23:55:45,273 >> Special tokens file saved in /tmp/debug_squad/checkpoint-2000/special_tokens_map.json
{'loss': 1.2396, 'learning_rate': 2.491663277755185e-05, 'epoch': 0.34}                                                                                                                                  
 17%|██████████████████████████▉                                                                                                                                    | 2500/14754 [11:54<58:01,  3.52it/s][INFO|trainer.py:2103] 2022-01-28 23:58:08,029 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-2500
[INFO|configuration_utils.py:430] 2022-01-28 23:58:08,030 >> Configuration saved in /tmp/debug_squad/checkpoint-2500/config.json
[INFO|modeling_utils.py:1074] 2022-01-28 23:58:08,509 >> Model weights saved in /tmp/debug_squad/checkpoint-2500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-28 23:58:08,509 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-2500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-28 23:58:08,509 >> Special tokens file saved in /tmp/debug_squad/checkpoint-2500/special_tokens_map.json
{'loss': 1.1863, 'learning_rate': 2.389995933306222e-05, 'epoch': 0.41}                                                                                                                                  
 20%|████████████████████████████████▎                                                                                                                              | 3000/14754 [14:17<55:36,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:00:31,193 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-3000
[INFO|configuration_utils.py:430] 2022-01-29 00:00:31,194 >> Configuration saved in /tmp/debug_squad/checkpoint-3000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:00:31,673 >> Model weights saved in /tmp/debug_squad/checkpoint-3000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:00:31,673 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-3000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:00:31,673 >> Special tokens file saved in /tmp/debug_squad/checkpoint-3000/special_tokens_map.json
{'loss': 1.1847, 'learning_rate': 2.288328588857259e-05, 'epoch': 0.47}                                                                                                                                  
 24%|█████████████████████████████████████▋                                                                                                                         | 3500/14754 [16:40<53:18,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:02:54,398 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-3500
[INFO|configuration_utils.py:430] 2022-01-29 00:02:54,400 >> Configuration saved in /tmp/debug_squad/checkpoint-3500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:02:54,877 >> Model weights saved in /tmp/debug_squad/checkpoint-3500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:02:54,878 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-3500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:02:54,878 >> Special tokens file saved in /tmp/debug_squad/checkpoint-3500/special_tokens_map.json
{'loss': 1.1146, 'learning_rate': 2.1866612444082963e-05, 'epoch': 0.54}                                                                                                                                 
 27%|███████████████████████████████████████████                                                                                                                    | 4000/14754 [19:03<50:55,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:05:17,606 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-4000
[INFO|configuration_utils.py:430] 2022-01-29 00:05:17,607 >> Configuration saved in /tmp/debug_squad/checkpoint-4000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:05:18,087 >> Model weights saved in /tmp/debug_squad/checkpoint-4000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:05:18,087 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-4000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:05:18,087 >> Special tokens file saved in /tmp/debug_squad/checkpoint-4000/special_tokens_map.json
{'loss': 1.0815, 'learning_rate': 2.084993899959333e-05, 'epoch': 0.61}                                                                                                                                  
 31%|████████████████████████████████████████████████▍                                                                                                              | 4500/14754 [21:26<48:32,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:07:40,784 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-4500
[INFO|configuration_utils.py:430] 2022-01-29 00:07:40,785 >> Configuration saved in /tmp/debug_squad/checkpoint-4500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:07:41,262 >> Model weights saved in /tmp/debug_squad/checkpoint-4500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:07:41,263 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-4500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:07:41,263 >> Special tokens file saved in /tmp/debug_squad/checkpoint-4500/special_tokens_map.json
{'loss': 1.0743, 'learning_rate': 1.9833265555103702e-05, 'epoch': 0.68}                                                                                                                                 
 34%|█████████████████████████████████████████████████████▉                                                                                                         | 5000/14754 [23:49<46:05,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:10:03,934 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-5000
[INFO|configuration_utils.py:430] 2022-01-29 00:10:03,935 >> Configuration saved in /tmp/debug_squad/checkpoint-5000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:10:04,412 >> Model weights saved in /tmp/debug_squad/checkpoint-5000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:10:04,413 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-5000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:10:04,413 >> Special tokens file saved in /tmp/debug_squad/checkpoint-5000/special_tokens_map.json
{'loss': 1.0973, 'learning_rate': 1.8816592110614073e-05, 'epoch': 0.75}                                                                                                                                 
 37%|███████████████████████████████████████████████████████████▎                                                                                                   | 5500/14754 [26:13<43:42,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:12:27,110 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-5500
[INFO|configuration_utils.py:430] 2022-01-29 00:12:27,111 >> Configuration saved in /tmp/debug_squad/checkpoint-5500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:12:27,588 >> Model weights saved in /tmp/debug_squad/checkpoint-5500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:12:27,589 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-5500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:12:27,589 >> Special tokens file saved in /tmp/debug_squad/checkpoint-5500/special_tokens_map.json
{'loss': 1.0606, 'learning_rate': 1.779991866612444e-05, 'epoch': 0.81}                                                                                                                                  
 41%|████████████████████████████████████████████████████████████████▋                                                                                              | 6000/14754 [28:36<41:31,  3.51it/s][INFO|trainer.py:2103] 2022-01-29 00:14:50,289 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-6000
[INFO|configuration_utils.py:430] 2022-01-29 00:14:50,290 >> Configuration saved in /tmp/debug_squad/checkpoint-6000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:14:50,768 >> Model weights saved in /tmp/debug_squad/checkpoint-6000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:14:50,769 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-6000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:14:50,769 >> Special tokens file saved in /tmp/debug_squad/checkpoint-6000/special_tokens_map.json
{'loss': 1.0263, 'learning_rate': 1.6783245221634812e-05, 'epoch': 0.88}                                                                                                                                 
 44%|██████████████████████████████████████████████████████████████████████                                                                                         | 6500/14754 [30:59<39:02,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:17:13,439 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-6500
[INFO|configuration_utils.py:430] 2022-01-29 00:17:13,440 >> Configuration saved in /tmp/debug_squad/checkpoint-6500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:17:13,919 >> Model weights saved in /tmp/debug_squad/checkpoint-6500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:17:13,919 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-6500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:17:13,919 >> Special tokens file saved in /tmp/debug_squad/checkpoint-6500/special_tokens_map.json
{'loss': 1.0332, 'learning_rate': 1.576657177714518e-05, 'epoch': 0.95}                                                                                                                                  
 47%|███████████████████████████████████████████████████████████████████████████▍                                                                                   | 7000/14754 [33:22<36:36,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:19:36,630 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-7000
[INFO|configuration_utils.py:430] 2022-01-29 00:19:36,631 >> Configuration saved in /tmp/debug_squad/checkpoint-7000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:19:37,108 >> Model weights saved in /tmp/debug_squad/checkpoint-7000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:19:37,108 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-7000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:19:37,108 >> Special tokens file saved in /tmp/debug_squad/checkpoint-7000/special_tokens_map.json
{'loss': 0.9457, 'learning_rate': 1.4749898332655551e-05, 'epoch': 1.02}                                                                                                                                 
 51%|████████████████████████████████████████████████████████████████████████████████▊                                                                              | 7500/14754 [35:45<34:15,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:21:59,797 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-7500
[INFO|configuration_utils.py:430] 2022-01-29 00:21:59,798 >> Configuration saved in /tmp/debug_squad/checkpoint-7500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:22:00,278 >> Model weights saved in /tmp/debug_squad/checkpoint-7500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:22:00,279 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-7500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:22:00,279 >> Special tokens file saved in /tmp/debug_squad/checkpoint-7500/special_tokens_map.json
{'loss': 0.7377, 'learning_rate': 1.373322488816592e-05, 'epoch': 1.08}                                                                                                                                  
 54%|██████████████████████████████████████████████████████████████████████████████████████▏                                                                        | 8000/14754 [38:08<31:47,  3.54it/s][INFO|trainer.py:2103] 2022-01-29 00:24:22,740 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-8000
[INFO|configuration_utils.py:430] 2022-01-29 00:24:22,741 >> Configuration saved in /tmp/debug_squad/checkpoint-8000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:24:23,220 >> Model weights saved in /tmp/debug_squad/checkpoint-8000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:24:23,221 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-8000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:24:23,221 >> Special tokens file saved in /tmp/debug_squad/checkpoint-8000/special_tokens_map.json
{'loss': 0.7174, 'learning_rate': 1.271655144367629e-05, 'epoch': 1.15}                                                                                                                                  
 58%|███████████████████████████████████████████████████████████████████████████████████████████▌                                                                   | 8500/14754 [40:31<29:29,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:26:45,683 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-8500
[INFO|configuration_utils.py:430] 2022-01-29 00:26:45,683 >> Configuration saved in /tmp/debug_squad/checkpoint-8500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:26:46,162 >> Model weights saved in /tmp/debug_squad/checkpoint-8500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:26:46,162 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-8500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:26:46,162 >> Special tokens file saved in /tmp/debug_squad/checkpoint-8500/special_tokens_map.json
{'loss': 0.7188, 'learning_rate': 1.1699877999186661e-05, 'epoch': 1.22}                                                                                                                                 
 61%|████████████████████████████████████████████████████████████████████████████████████████████████▉                                                              | 9000/14754 [42:54<27:08,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:29:08,648 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-9000
[INFO|configuration_utils.py:430] 2022-01-29 00:29:08,649 >> Configuration saved in /tmp/debug_squad/checkpoint-9000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:29:09,127 >> Model weights saved in /tmp/debug_squad/checkpoint-9000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:29:09,128 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-9000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:29:09,128 >> Special tokens file saved in /tmp/debug_squad/checkpoint-9000/special_tokens_map.json
{'loss': 0.7347, 'learning_rate': 1.0683204554697033e-05, 'epoch': 1.29}                                                                                                                                 
 64%|██████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                        | 9500/14754 [45:17<24:48,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:31:31,545 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-9500
[INFO|configuration_utils.py:430] 2022-01-29 00:31:31,546 >> Configuration saved in /tmp/debug_squad/checkpoint-9500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:31:32,024 >> Model weights saved in /tmp/debug_squad/checkpoint-9500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:31:32,024 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-9500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:31:32,025 >> Special tokens file saved in /tmp/debug_squad/checkpoint-9500/special_tokens_map.json
{'loss': 0.7144, 'learning_rate': 9.666531110207402e-06, 'epoch': 1.36}                                                                                                                                  
 68%|███████████████████████████████████████████████████████████████████████████████████████████████████████████                                                   | 10000/14754 [47:40<22:27,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:33:54,470 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-10000
[INFO|configuration_utils.py:430] 2022-01-29 00:33:54,471 >> Configuration saved in /tmp/debug_squad/checkpoint-10000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:33:54,949 >> Model weights saved in /tmp/debug_squad/checkpoint-10000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:33:54,950 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-10000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:33:54,950 >> Special tokens file saved in /tmp/debug_squad/checkpoint-10000/special_tokens_map.json
{'loss': 0.7472, 'learning_rate': 8.649857665717772e-06, 'epoch': 1.42}                                                                                                                                  
 71%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                             | 10500/14754 [50:03<20:04,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:36:17,422 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-10500
[INFO|configuration_utils.py:430] 2022-01-29 00:36:17,423 >> Configuration saved in /tmp/debug_squad/checkpoint-10500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:36:17,901 >> Model weights saved in /tmp/debug_squad/checkpoint-10500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:36:17,902 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-10500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:36:17,902 >> Special tokens file saved in /tmp/debug_squad/checkpoint-10500/special_tokens_map.json
{'loss': 0.6929, 'learning_rate': 7.633184221228141e-06, 'epoch': 1.49}                                                                                                                                  
 75%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                        | 11000/14754 [52:26<17:40,  3.54it/s][INFO|trainer.py:2103] 2022-01-29 00:38:40,348 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-11000
[INFO|configuration_utils.py:430] 2022-01-29 00:38:40,349 >> Configuration saved in /tmp/debug_squad/checkpoint-11000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:38:40,828 >> Model weights saved in /tmp/debug_squad/checkpoint-11000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:38:40,829 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-11000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:38:40,829 >> Special tokens file saved in /tmp/debug_squad/checkpoint-11000/special_tokens_map.json
{'loss': 0.7103, 'learning_rate': 6.616510776738511e-06, 'epoch': 1.56}                                                                                                                                  
 78%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                  | 11500/14754 [54:49<15:22,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:41:03,359 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-11500
[INFO|configuration_utils.py:430] 2022-01-29 00:41:03,360 >> Configuration saved in /tmp/debug_squad/checkpoint-11500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:41:03,839 >> Model weights saved in /tmp/debug_squad/checkpoint-11500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:41:03,840 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-11500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:41:03,840 >> Special tokens file saved in /tmp/debug_squad/checkpoint-11500/special_tokens_map.json
{'loss': 0.7036, 'learning_rate': 5.5998373322488825e-06, 'epoch': 1.63}                                                                                                                                 
 81%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                             | 12000/14754 [57:12<12:59,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:43:26,420 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-12000
[INFO|configuration_utils.py:430] 2022-01-29 00:43:26,421 >> Configuration saved in /tmp/debug_squad/checkpoint-12000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:43:26,902 >> Model weights saved in /tmp/debug_squad/checkpoint-12000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:43:26,902 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-12000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:43:26,902 >> Special tokens file saved in /tmp/debug_squad/checkpoint-12000/special_tokens_map.json
{'loss': 0.6791, 'learning_rate': 4.583163887759252e-06, 'epoch': 1.69}                                                                                                                                  
 85%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                        | 12500/14754 [59:35<10:38,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:45:49,585 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-12500
[INFO|configuration_utils.py:430] 2022-01-29 00:45:49,586 >> Configuration saved in /tmp/debug_squad/checkpoint-12500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:45:50,065 >> Model weights saved in /tmp/debug_squad/checkpoint-12500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:45:50,065 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-12500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:45:50,065 >> Special tokens file saved in /tmp/debug_squad/checkpoint-12500/special_tokens_map.json
{'loss': 0.6995, 'learning_rate': 3.566490443269622e-06, 'epoch': 1.76}                                                                                                                                  
 88%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                  | 13000/14754 [1:01:58<08:15,  3.54it/s][INFO|trainer.py:2103] 2022-01-29 00:48:12,721 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-13000
[INFO|configuration_utils.py:430] 2022-01-29 00:48:12,722 >> Configuration saved in /tmp/debug_squad/checkpoint-13000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:48:13,201 >> Model weights saved in /tmp/debug_squad/checkpoint-13000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:48:13,202 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-13000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:48:13,202 >> Special tokens file saved in /tmp/debug_squad/checkpoint-13000/special_tokens_map.json
{'loss': 0.698, 'learning_rate': 2.549816998779992e-06, 'epoch': 1.83}                                                                                                                                   
 92%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋             | 13500/14754 [1:04:21<05:56,  3.52it/s][INFO|trainer.py:2103] 2022-01-29 00:50:35,826 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-13500
[INFO|configuration_utils.py:430] 2022-01-29 00:50:35,827 >> Configuration saved in /tmp/debug_squad/checkpoint-13500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:50:36,305 >> Model weights saved in /tmp/debug_squad/checkpoint-13500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:50:36,306 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-13500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:50:36,306 >> Special tokens file saved in /tmp/debug_squad/checkpoint-13500/special_tokens_map.json
{'loss': 0.6899, 'learning_rate': 1.533143554290362e-06, 'epoch': 1.9}                                                                                                                                   
 95%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████        | 14000/14754 [1:06:44<03:33,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:52:58,855 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-14000
[INFO|configuration_utils.py:430] 2022-01-29 00:52:58,856 >> Configuration saved in /tmp/debug_squad/checkpoint-14000/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:52:59,337 >> Model weights saved in /tmp/debug_squad/checkpoint-14000/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:52:59,337 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-14000/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:52:59,337 >> Special tokens file saved in /tmp/debug_squad/checkpoint-14000/special_tokens_map.json
{'loss': 0.6963, 'learning_rate': 5.164701098007319e-07, 'epoch': 1.97}                                                                                                                                  
 98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎  | 14500/14754 [1:09:07<01:11,  3.53it/s][INFO|trainer.py:2103] 2022-01-29 00:55:21,815 >> Saving model checkpoint to /tmp/debug_squad/checkpoint-14500
[INFO|configuration_utils.py:430] 2022-01-29 00:55:21,816 >> Configuration saved in /tmp/debug_squad/checkpoint-14500/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:55:22,295 >> Model weights saved in /tmp/debug_squad/checkpoint-14500/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:55:22,295 >> tokenizer config file saved in /tmp/debug_squad/checkpoint-14500/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:55:22,295 >> Special tokens file saved in /tmp/debug_squad/checkpoint-14500/special_tokens_map.json
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14754/14754 [1:10:21<00:00,  3.53it/s][INFO|trainer.py:1481] 2022-01-29 00:56:35,107 >> 

Training completed. Do not forget to share your model on huggingface.co/models =)


{'train_runtime': 4221.1187, 'train_samples_per_second': 41.943, 'train_steps_per_second': 3.495, 'train_loss': 0.9828958572112595, 'epoch': 2.0}                                                        
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14754/14754 [1:10:21<00:00,  3.50it/s]
[INFO|trainer.py:2103] 2022-01-29 00:56:35,108 >> Saving model checkpoint to /tmp/debug_squad/
[INFO|configuration_utils.py:430] 2022-01-29 00:56:35,109 >> Configuration saved in /tmp/debug_squad/config.json
[INFO|modeling_utils.py:1074] 2022-01-29 00:56:35,588 >> Model weights saved in /tmp/debug_squad/pytorch_model.bin
[INFO|tokenization_utils_base.py:2074] 2022-01-29 00:56:35,588 >> tokenizer config file saved in /tmp/debug_squad/tokenizer_config.json
[INFO|tokenization_utils_base.py:2080] 2022-01-29 00:56:35,589 >> Special tokens file saved in /tmp/debug_squad/special_tokens_map.json
***** train metrics *****
  epoch                    =        2.0
  train_loss               =     0.9829
  train_runtime            = 1:10:21.11
  train_samples            =      88524
  train_samples_per_second =     41.943
  train_steps_per_second   =      3.495
01/29/2022 00:56:35 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:553] 2022-01-29 00:56:35,623 >> The following columns in the evaluation set  don't have a corresponding argument in `BertForQuestionAnswering.forward` and have been ignored: offset_mapping, example_id.
[INFO|trainer.py:2353] 2022-01-29 00:56:35,625 >> ***** Running Evaluation *****
[INFO|trainer.py:2355] 2022-01-29 00:56:35,625 >>   Num examples = 10784
[INFO|trainer.py:2358] 2022-01-29 00:56:35,625 >>   Batch size = 8
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1348/1348 [01:38<00:00, 15.27it/s]01/29/2022 00:58:16 - INFO - utils_qa - Post-processing 10570 example predictions split into 10784 features.
 10%|███████████████                                                                                                                                               | 1005/10570 [00:02<00:22, 431.61it/s]
Traceback (most recent call last):                                                                                                                                  | 977/10570 [00:02<00:22, 424.83it/s]
  File "/workspace/transformers/examples/pytorch/question-answering/run_qa.py", line 647, in <module>
    main()
  File "/workspace/transformers/examples/pytorch/question-answering/run_qa.py", line 604, in main
    metrics = trainer.evaluate()
  File "/workspace/transformers/examples/pytorch/question-answering/trainer_qa.py", line 56, in evaluate
    eval_preds = self.post_process_function(eval_examples, eval_dataset, output.predictions)
  File "/workspace/transformers/examples/pytorch/question-answering/run_qa.py", line 540, in post_processing_function
    predictions = postprocess_qa_predictions(
  File "/workspace/transformers/examples/pytorch/question-answering/utils_qa.py", line 152, in postprocess_qa_predictions
    "offsets": (offset_mapping[start_index][0], offset_mapping[end_index][1]),
IndexError: list index out of range
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1348/1348 [01:43<00:00, 13.04it/s]
```

## Expected behavior

Evaluation ends without an `IndexError`.


Comments:
I was able to reproduce the same issue on another machine with Python 3.9.7 and a different type of GPU.

I am not able to reproduce the error, so I have made a blanket fix in the PR linked above. If you have a wait to debug a little bit more and print the values of:
- `len(offset_mapping)`
- `start_index`
- `end_index`
- `offset_mapping[start_index]`
- `offset_mapping[end_index]`

That would help us find the potential cause of the error.

Thanks for looking into this issue! I added this to `utils_qa.py` and re-ran the script. This time I only ran one epoch to save time. I suppose that won't make a difference.


```python
# We shouldn't get index errors now because of all the cehcks above, but for some reason, some
# are still there, see https://github.com/huggingface/transformers/issues/15401#issuecomment-1024845936
try:
    example_offsets = (offset_mapping[start_index][0], offset_mapping[end_index][1])
except IndexError:
    indexerror_count += 1
    print("\n===========================")
    print(f"{indexerror_count=}")
    print(f"{len(offset_mapping)=}")
    print(f"{start_index=}")
    print(f"{end_index=}")
    print(f"{offset_mapping[start_index]=}")
    print(f"{offset_mapping[end_index]=}")
    print("===========================")
    continue
```

The generated log file is very long so I've attached it as a file: [indexerror.log](https://github.com/huggingface/transformers/files/7973386/indexerror.log)


BTW this is the dockerfile for the container I use:

```Dockerfile
# PyTorch 1.10.0 uses cudatoolkit 11.3.1
FROM nvidia/cuda:11.3.1-cudnn8-devel-ubuntu20.04

# Basic installs
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ='America/Detroit'
RUN apt-get update -qq \
    && apt-get -y --no-install-recommends install \
       build-essential software-properties-common \
       zsh wget curl git tar rsync ssh \
    && apt-get clean all \
    && rm -r /var/lib/apt/lists/*

# Install latest cmake
RUN wget https://github.com/Kitware/CMake/releases/download/v3.22.0/cmake-3.22.0-linux-x86_64.tar.gz \
    && tar xzf cmake-* \
    && rsync -a cmake-*/bin /usr/local \
    && rsync -a cmake-*/share /usr/local \
    && rm -r cmake-*

# Install python
ENV PATH="/root/.local/miniconda3/bin:$PATH"
RUN mkdir -p /root/.local \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b -p /root/.local/miniconda3 \
    && rm -f Miniconda3-latest-Linux-x86_64.sh \
    && ln -sf /root/.local/miniconda3/etc/profile.d/conda.sh /etc/profile.d/conda.sh

# Install frequently used python packages and pytorch build dependencies
RUN pip install matplotlib seaborn pandas pyyaml numpy typing_extensions

WORKDIR /workspace

RUN chsh -s /usr/bin/zsh
ENTRYPOINT /usr/bin/zsh
```

Thanks, that's very helpful. Could you give me you version of the `datasets` library as a last step? It looks like you have `[]`s where you should have `None`s

I'm using `datasets` 1.18.2. More info:

```console
$ pip freeze
accelerate==0.5.1
aiohttp==3.8.1
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.4.0
brotlipy==0.7.0
certifi==2021.10.8
cffi @ file:///tmp/build/80754af9/cffi_1625814692085/work
chardet @ file:///tmp/build/80754af9/chardet_1607706775000/work
charset-normalizer==2.0.11
click==8.0.3
conda==4.11.0
conda-package-handling @ file:///tmp/build/80754af9/conda-package-handling_1618262147379/work
cryptography @ file:///tmp/build/80754af9/cryptography_1616767007030/work
cycler==0.11.0
datasets==1.18.2
dill==0.3.4
filelock==3.4.2
fonttools==4.29.0
frozenlist==1.3.0
fsspec==2022.1.0
huggingface-hub==0.4.0
idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work
joblib==1.1.0
kiwisolver==1.3.2
matplotlib==3.5.1
mkl-fft==1.3.1
mkl-random @ file:///tmp/build/80754af9/mkl_random_1626186066731/work
mkl-service==2.4.0
multidict==6.0.2
multiprocess==0.70.12.2
numpy==1.22.1
olefile @ file:///Users/ktietz/demo/mc3/conda-bld/olefile_1629805411829/work
packaging==21.3
pandas==1.4.0
Pillow==9.0.0
pyarrow==6.0.1
pycosat==0.6.3
pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work
pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work
pyparsing==3.0.7
PySocks @ file:///tmp/build/80754af9/pysocks_1605305812635/work
python-dateutil==2.8.2
pytz==2021.3
PyYAML==6.0
regex==2022.1.18
requests @ file:///tmp/build/80754af9/requests_1608241421344/work
ruamel-yaml-conda @ file:///tmp/build/80754af9/ruamel_yaml_1616016711199/work
sacremoses==0.0.47
scipy==1.7.3
seaborn==0.11.2
six @ file:///tmp/build/80754af9/six_1623709665295/work
tokenizers==0.11.4
torch==1.10.1
torchvision==0.11.2
tqdm==4.62.3
transformers @ file:///workspace/transformers-fix
typing-extensions==4.0.1
urllib3 @ file:///tmp/build/80754af9/urllib3_1625084269274/work
xxhash==2.0.2
yarl==1.7.2
```

`transformers @ file:///workspace/transformers-fix`: `git checkout fix_qa_utils` and then the logging modifications mentioned above.

The PR linked above should fix the error. There is something going wrong with Datasets here, but I can't fully reproduce, so made a better check in the function that caused the error :-)

Hi @sgugger this error now re-appears in `postprocess_qa_predictions_with_beam_search`. What I can see is that `postprocess_qa_predictions` is more defensive in checking if the answer is in scope, but `postprocess_qa_predictions_with_beam_search` lacks **some** of the checks.

I am preparing a PR for a cleaned-up version of examples. It will likely fix this too (so don't rush to fix **anything right now**). I guess xlnet is much longer to run compared to bert-base, so not many people test this use case (and don't know it will fail).


PyArrow 10.0.0 fixed the bug that introduces `[]` instead of keeping the `None`. A PR in open in `datasets` to integrate the fix: https://github.com/huggingface/datasets/pull/5174