DDP NCCL freezes in docker AWS Jupyter
❓ Questions and Help
Problem
Jupyter terminal freezes, and connection to AWS node closes.
The problem is reproducible with any Lightning example.
What have you tried?
python pl_examples/basic_examples/image_classifier.py --gpus 4 --accelerator ddp
What's your environment?
Linux using docker image
Lightning 1.0.4
pytorch 1.6
cuda 10.2
The solution was to use
export NCCL_SOCKET_IFNAME=lo