text: 
  
  TITLE
  [s2s] distributed eval gets stuck on error w/ multigpu
  
  COMMENTS
  6
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  `examples/seq2seq/distillation.py` and probably others remain hanging on internal error when run w/ multiple gpus (2 here):
  
  ```
  rm -r /tmp/tmpqajqhzwo; PYTHONPATH="src" python examples/seq2seq/distillation.py --supervise_forward --normalize_hidden --label_smoothing=0.0 --eval_beams=1 --val_metric=loss --save_top_k=1 --adafactor --early_stopping_patience=-1 --logger_name=default --length_penalty=0.5 --cache_dir= --task=summarization --num_workers=2 --alpha_hid=0 --freeze_embeds --sortish_sampler --student_decoder_layers=1 --val_check_interval=0.5 --output_dir=/tmp/tmpqajqhzwo --no_teacher --fp16_opt_level=O1 --gpus=2 --max_grad_norm=1.0 --do_train --do_predict --accumulate_grad_batches=1 --seed=42 --model_name_or_path=sshleifer/tinier_bart --config_name= --tokenizer_name=facebook/bart-large --learning_rate=0.3 --lr_scheduler=linear --weight_decay=0.0 --adam_epsilon=1e-08 --warmup_steps=0 --max_epochs=2 --train_batch_size=1 --eval_batch_size=2 --max_source_length=12 --max_target_length=12 --val_max_target_length=12 --test_max_target_length=12 --n_train=-1 --n_val=-1 --n_test=-1 --student_encoder_layers=1 --freeze_encoder --data_dir=examples/seq2seq/test_data/wmt_en_ro --alpha_mlm=0.2 --alpha_ce=0.8 --teacher=sshleifer/bart-tiny-random
  ```
  
  last output:
  ```
  initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
  Traceback (most recent call last):
    File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 281, in <module>
      distill_main(args)
    File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 269, in distill_main
      check_output_dir(args, expected_items=3)
    File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 641, in check_output_dir
      raise ValueError(
  ValueError: Output directory (/tmp/tmpqajqhzwo) already exists and has 7 items in it (expected 3 items). Use --overwrite_output_dir to overcome.
  ```
  and now it hangs, holding onto the gpu. Can't even Ctrl-C the process - needed to suspend+kill manually.
  
  I know that adding `--overwrite_output_dir` will remove the error, but this is not the issue. It shouldn't hang on error (e.g. the test suite needs to continue running in such event).
  
  @sshleifer 