Why Rank 0 does a lot of work for loading the data? Why NaN in evaluation?

princethewinner: <p>I am trying to understand how <code>DataLoader</code> works with multiple GPUs and multiple workers. This question has four parts.</p>
<ol>
<li>The first part is asked here: <a href="https://discuss.huggingface.co/t/same-seed-across-different-gpus-in-multiple-workers/76535" class="inline-onebox" rel="noopener nofollow ugc">Same seed across different gpus in multiple workers - Intermediate - Hugging Face Forums</a> (While writing this post, there was no answer on this as well, I would appreciate it if someone can answer it as well)</li>
<li>To investigate how the data is loaded during training I dumped the samples generated by each <code>rank</code> and <code>worker</code> combination. It seems that most of the work to load the data is done by the workers of the <code>rank_0</code> . (In the figure below, <code>rank_0</code> workers have the largest dump – 18M – while others has only 13K dump size. ). What could be the reason for this?</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f083ad25ad28e794ad52a7d8bb1653baa28a91b4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f083ad25ad28e794ad52a7d8bb1653baa28a91b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f083ad25ad28e794ad52a7d8bb1653baa28a91b4_2_690x44.jpeg" alt="image" data-base62-sha1="yjGHUjOVZy53HYyjMWbNHn5vCyE" width="690" height="44" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f083ad25ad28e794ad52a7d8bb1653baa28a91b4_2_690x44.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f083ad25ad28e794ad52a7d8bb1653baa28a91b4_2_1035x66.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/0/f083ad25ad28e794ad52a7d8bb1653baa28a91b4_2_1380x88.jpeg 2x" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2170×140 111 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="3">
<li>
<p>Evaluation steps result in <code>nan</code>, the investigation of <code>nan</code> suggests that the <code>custom_loss</code> function in <code>transformer</code>’s <code>Trainer</code> API receiving a blank input, although there are more samples left for evaluation.</p>
</li>
<li>
<p>Multi-GPU evaluation and single-GPU evaluation utilize a different number of samples. Further, in multi-GPU evaluation, all the GPUs report independent metrics. I am using <code>transformer</code>’s <code>Trainer.evaluate</code> API with <code>torchrun --nproc_per_node 2 eval.py</code> .</p>
</li>
</ol>
<p>Training setup (for part 2 of the question):</p>
<pre data-code-wrap="python"><code class="lang-python">train_data_loader: IterableDataset = ...
mlm_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm_probability=mlm_probability
)

deepspeed_config = {
    "train_micro_batch_size_per_gpu": "auto",
    "gradient_accumulation_steps": "auto",
    "fp16": {
        "enabled": False
    },
    "zero_optimization": {
        "stage": 0
    }
}

training_arguments: TrainingArguments = TrainingArguments(.  # from HF-transformer
    output_dir=checkpoint_dir,
    logging_dir=tensorboard_dir,
    evaluation_strategy=evaluation_strategy,
    learning_rate=2e-5,
    weight_decay=0.01,
    push_to_hub=False,
    max_steps=max_steps,
    dataloader_pin_memory=True,
    dataloader_num_workers=32,
    per_device_train_batch_size=64,
    accelerator_config={"split_batches": True},
    gradient_accumulation_steps=32,
    dataloader_prefetch_factor=5,
    deepspeed=deepspeed_config,
    fp16=False,
    disable_tqdm=True
)

trainer : Trainer = Trainer(...)  # from HF-transformers

</code></pre>
<p>Run command:</p>
<pre data-code-wrap="bash"><code class="lang-bash">deepspeed --master_port=29600 --num_gpus=4 train.py
</code></pre>