[s2s] distributed eval gets stuck on error w/ multigpu
Author: stas00
Pull request: no
Labels: name: wontfix

`examples/seq2seq/distillation.py` and probably others remain hanging on internal error when run w/ multiple gpus (2 here):

```
rm -r /tmp/tmpqajqhzwo; PYTHONPATH="src" python examples/seq2seq/distillation.py --supervise_forward --normalize_hidden --label_smoothing=0.0 --eval_beams=1 --val_metric=loss --save_top_k=1 --adafactor --early_stopping_patience=-1 --logger_name=default --length_penalty=0.5 --cache_dir= --task=summarization --num_workers=2 --alpha_hid=0 --freeze_embeds --sortish_sampler --student_decoder_layers=1 --val_check_interval=0.5 --output_dir=/tmp/tmpqajqhzwo --no_teacher --fp16_opt_level=O1 --gpus=2 --max_grad_norm=1.0 --do_train --do_predict --accumulate_grad_batches=1 --seed=42 --model_name_or_path=sshleifer/tinier_bart --config_name= --tokenizer_name=facebook/bart-large --learning_rate=0.3 --lr_scheduler=linear --weight_decay=0.0 --adam_epsilon=1e-08 --warmup_steps=0 --max_epochs=2 --train_batch_size=1 --eval_batch_size=2 --max_source_length=12 --max_target_length=12 --val_max_target_length=12 --test_max_target_length=12 --n_train=-1 --n_val=-1 --n_test=-1 --student_encoder_layers=1 --freeze_encoder --data_dir=examples/seq2seq/test_data/wmt_en_ro --alpha_mlm=0.2 --alpha_ce=0.8 --teacher=sshleifer/bart-tiny-random
```

last output:
```
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Traceback (most recent call last):
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 281, in <module>
    distill_main(args)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/distillation.py", line 269, in distill_main
    check_output_dir(args, expected_items=3)
  File "/mnt/nvme1/code/huggingface/transformers-master/examples/seq2seq/utils.py", line 641, in check_output_dir
    raise ValueError(
ValueError: Output directory (/tmp/tmpqajqhzwo) already exists and has 7 items in it (expected 3 items). Use --overwrite_output_dir to overcome.
```
and now it hangs, holding onto the gpu. Can't even Ctrl-C the process - needed to suspend+kill manually.

I know that adding `--overwrite_output_dir` will remove the error, but this is not the issue. It shouldn't hang on error (e.g. the test suite needs to continue running in such event).

@sshleifer 

Comments:
same happens with `finetune.py` - happened in another run when it hit OOM. So basically any error.

@williamFalcon @SeanNaren (lightning friends)

Do you guys have a clever way to collect failures in your multigpu tests?
When something breaks, our multigpu test hangs.


yes... good questions haha. 

So, some things we know:

1. multi gpu tests should run one per test (ie: don’t parametrize via pytest). Seems that the way pytest starts an experiment does not play well with pytorch distributed. 

2. ddp in lightning needs to use subprocess inside a test and call an external file. 
https://github.com/PyTorchLightning/pytorch-lightning/blob/master/tests/backends/test_ddp.py#L57

3. ddp spawn tests need to adhere to that single test per function call i mentioned in 1. pytest parametrized ddp testss WILL freeze the build. 

Thank you for the insights, @williamFalcon 

That is the case already - I discovered the subprocess idea by looking at your distributed ddp test ;) And none of these are parametrized.

So it must be something else.

p.s. btw, have you tried the `parameterized` module https://pypi.org/project/parameterized/? It's more flexible than `pytest`'s `parametrize` - perhaps it won't have the same impact (but that's unrelated to this issue).

oh, and to clarify, this has nothing to do with testing. The hanging happens in the standalone scripts. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.