Using DDP with num_workers > 0 hangs before entering the first training epoch loop
Labels: oncall: distributed, module: dataloader

### 🐛 Describe the bug

I am trying to use DDP on a 8 GPU cluster with a data loader with num workers > 0. My code seemed to be running fine yesterday but today I have been unable to get it to work. Each time, the code hangs right before the training loop begins (it gets through the `running_epoch_loss=0` line below. The loop itself is simple and the data loading seems to work properly beforehand.

    # stuff related to device
    nb = run_d["non_blocking"]
    loss_func = torch.nn.CrossEntropyLoss()
    loss_func.cuda(gpu)

    # train
    torch.enable_grad()
    model.train()
    loss_arr = [0, 0]
    running_epoch_loss = 0

    for i, b in tqdm(enumerate(dl_d), desc=f"> training - epoch:{epoch}", total=len(dl_d)):
        batch = generate_pretraining_inputs(b, nb=nb)
        outputs = model(**batch)
        loss = outputs.loss.mean()
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

The data loading script is here:


    data_path = Path(data_d["data_path"])
    assert data_path.is_file()
    train_data = NotesforPT(data_path)
    print(f"Train Data Loaded: {len(train_data)}")

    tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
    model_dict = {"tokenizer": tokenizer, "max_length": model_d["max_length"], 'config_file': model_d['config_file']}
    model = BERTforPreTraining(**model_dict)
    collate = model.collate_fn
    model.cuda(gpu)
    if rank == 0:
        print(f"Model Parameters: {count_parameters(model)}")
    model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])

    train_sampler = torch.utils.data.distributed.DistributedSampler(train_data, num_replicas=args.world_size, rank=rank, drop_last=True)
    train_loader = torch.utils.data.DataLoader(
        dataset=train_data,
        batch_size=run_d["batch_size"],
        shuffle=False,
        num_workers=4,
        pin_memory=True,
        collate_fn=collate,
        sampler=train_sampler,
    )

    return train_data, train_loader, model

When I set num_workers=0 the code runs, but it is too slow for my needs. I've seen various comments on this issue in the forums, but no definitive answer and the suggestions have not worked for me, including ensuring that the data can be evenly split with the batch size and num_workers values as well as setting NCCL_P2P_DISABLE=1. Additionally, running the same script with 1 GPU also does not fix the problem, so it seems to definitely be a num_workers issue. Any help is appreciated!



### Versions

PyTorch version: 1.13.0.dev20220724+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Amazon Linux release 2 (Karoo) (x86_64)
GCC version: (GCC) 7.3.1 20180712 (Red Hat 7.3.1-15)
Clang version: Could not collect
CMake version: version 2.8.12.2
Libc version: glibc-2.26

Python version: 3.8.5 (default, Feb 18 2021, 01:24:20)  [GCC 7.3.1 20180712 (Red Hat 7.3.1-12)] (64-bit runtime)
Python platform: Linux-5.10.112-108.499.amzn2.x86_64-x86_64-with-glibc2.2.5
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 470.103.01
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.22.4
[pip3] torch==1.13.0.dev20220724+cu116
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.12.0+cu113
[pip3] torchvision==0.14.0.dev20220724+cu116

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @SciPioneer @H-Huang @kwen2501 @SsnL @VitalyFedyunin @ejguan @NivekT