Bug when finetuning model on Squad
Author: a-maci
Pull request: no
Labels: name: wontfix

## 🐛 Bug

Model: Bert (bert-large-uncased-whole-word-masking)

The problem arises when using:
The official example script for finetuning on squad data:
```
python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
      --model_type bert \
      --model_name_or_path bert-large-uncased-whole-word-masking \
      --do_train \
      --do_eval \
      --do_lower_case \
      --train_file $SQUAD_DIR/train-v1.1.json \
      --predict_file $SQUAD_DIR/dev-v1.1.json \
      --learning_rate 3e-5 \
      --num_train_epochs 2 \
      --max_seq_length 384 \
      --doc_stride 128 \
      --output_dir ./models/wwm_uncased_finetuned_squad/ \
      --per_gpu_eval_batch_size 3 \
      --per_gpu_train_batch_size 3 \
      --save_steps 1500 \
      --logging_steps 250 \
      --fp16 
```

The tasks I am working on is:
* [x ] an official GLUE/SQUaD task: SQUaD

Here is the error log:
```
...
10/09/2019 17:03:29 - INFO - utils_squad -   Writing predictions to: ./models/wwm_uncased_finetuned_squad/predictions_.json
10/09/2019 17:03:29 - INFO - utils_squad -   Writing nbest to: ./models/wwm_uncased_finetuned_squad/nbest_predictions_.json
Traceback (most recent call last):
  File "run_squad.py", line 537, in <module>
    main()
  File "run_squad.py", line 526, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 268, in evaluate
    args.version_2_with_negative, args.null_score_diff_threshold)
  File "/dl/huggingface-bert/transformers/examples/SQuAD_runs/rundir/utils_squad.py", line 511, in write_predictions
    result = unique_id_to_result[feature.unique_id]
KeyError: 1000000000
```

## Additional context
When running on multiple gpus the above problem shows up.


Comments:
https://github.com/huggingface/transformers/issues/940

@ahotrod you have any fix for this bug?

> @ahotrod you have any fix for this bug?

@a-maci no unfortunately not, still searching.  I'm considering rolling back to Transformers 2.0.0 or even pytorch-transformers 1.2.0, one or both of which didn't spawn this error in my earlier SQuAD replications.

> @ahotrod you have any fix for this bug?

@a-maci  I needed XLNet fine-tuned on SQuAD 2.0 with 512 max_seq_length.  I found "**A**" solution:  went back to the original XLNet paper's github for the "native" code.  I could fit 1 batch on each of (2) 1080Ti GPUs,  85,000 steps, ~14.5 hr of fine-tuning with results EM / F1:  84.5 / 87.1.

`INFO:tensorflow:Result | best_exact 84.52792049187232 | best_exact_thresh -2.716632127761841 | best_f1 87.12844471348052 | best_f1_thresh -2.447098970413208 | has_ans_exact 0.8733130904183536 | has_ans_f1 0.9327569452896122 | `

Possibly try the BERT paper's "native" code?

I've described the bug here: https://github.com/huggingface/transformers/issues/940#issuecomment-547686206

Workaround is either to use DataParallel (remove `-m torch.distributed.launch --nproc_per_node=8`) or don't eval in the same run (remove `--do_eval`). You can evaluate the model after training with:

```
python examples/run_squad.py \
--model_type bert \
--model_name_or_path bert-base-cased \
--do_eval \
--do_lower_case \
--predict_file $SQUAD_DIR/dev-v1.1.json \
--train_file $SQUAD_DIR/train-v1.1.json \
--output_dir ./models/wwm_uncased_finetuned_squad/
```

As mentioned in #940, happy to welcome a PR to fix this case if someone from the community wants to contribute (I don't have the bandwidth for this issue at the moment).

Maybe try changing `args.local_rank == -1` to `args.local_rank in [-1, 0]` at this line? https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/examples/run_squad.py#L216

I think evaluate is only used in the main process (local_rank==0) if you're using multiple gpus
(reference: https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/examples/run_squad.py#L543)

It makes more sense to just remove the `DistributedSampler` case entirely. The problem is that `all_results` doesn't get gathered from all GPUs. Unless you also implement a gather you shouldn't use `DistributedSampler` at all.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Is there a fix for this ? Im seeing the same issue for running only evaluation on CPU too. 

Are you trying to do multiprocess evaluation? A single CPU process should work, my WAR above is to run eval seperately as a single process.