Repository: huggingface/transformers

Issue:
Distributed DataSampler has fixed data order despite random seeds.
When using a distributed data loader with `shuffle = True` in the Hugging Face trainer, it calls the underlying torch data loader. If `shuffle` is set to True, the data loader seeds the generator with `seed + epoch` ([here](https://github.com/pytorch/pytorch/blob/f84a50109f794d4feab922056b77d7c358076776/torch/utils/data/distributed.py#L100)).

When calling the data loader in HF trainer ([here](https://github.com/huggingface/transformers/blob/3ed5e97ba04ce9b24b4a7161ea74572598a4c480/src/transformers/trainer.py#L553)), the seed is _not_ passed to the torch data loader and thereby gets set to the default seed of 0. This means the data loader generator will always gets initialized to the epoch, despite a different seed to HF.

I would think we'd want the data order to be random, too.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.0.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes (with DeepSpeed)

### Who can help
@sgugger (trainer)

## Information

Model I am using (Bert, XLNet ...): GPT2

The problem arises when using:
* The hugging face trainer with a distributed data sampler

The tasks I am working on is:
* Training GPT2 from scratch using DDP with DeepSpeed

## To reproduce

Steps to reproduce the behavior:

Using a different seed with distributed data loader does not change the data order.

## Expected behavior

The random seed should be passed to the data loader so the data order to randomized with the seed changing.



Discussion:
Follow-up note (part of @lorr1's team that encountered this). This is particularly insidious for any sort of code that tries training with multiple random seeds; there's an assumption that across seeds, weight initialization (for pre-training, fine-tuning weights), dropout, AND data order are all different (and all do have significant bearing on results). 

Consistent data order (as in the existing code) runs counter to that expectation.
I'm guessing we want a new argument to control that seed though, not the current `args.seed` that is set at the beginning of training, what do you think? 
I think the seed set at the beginning of training would be fine -- that would be the expected behavior (weights get randomly initialized, then data order is random _conditioned on a single seed_.

Adding a separate seed just for data order means it's just one more thing you need to keep track of.

There's a backwards compatibility issue here possibly (if folks doing multiple random seeds worth of runs have been relying on/reporting those results), but this feels like the simplest solution?
It's definitely the easiest solution. For the backward compatibility issue, I hope users save their version of Transformers and PyTorch along the seeds for reproducibility. PyTorch does not guarantee the same results across versions (we had an issue with multinomial that changed behavior for instance). So I think it's fine to change the behavior, especially as it's a bug fix.

Will make a PR with the change.

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -547,6 +547,7 @@ def _get_train_sampler(self) -> Optional[torch.utils.data.sampler.Sampler]:
                     rank=self.args.process_index,
                     lengths=lengths,
                     model_input_name=model_input_name,
+                    seed=self.args.seed,
                 )
 
         else:
@@ -562,10 +563,14 @@ def _get_train_sampler(self) -> Optional[torch.utils.data.sampler.Sampler]:
                     batch_size=self.args.per_device_train_batch_size,
                     num_replicas=self.args.world_size,
                     rank=self.args.process_index,
+                    seed=self.args.seed,
                 )
             else:
                 return DistributedSampler(
-                    self.train_dataset, num_replicas=self.args.world_size, rank=self.args.process_index
+                    self.train_dataset,
+                    num_replicas=self.args.world_size,
+                    rank=self.args.process_index,
+                    seed=self.args.seed,
                 )
 
     def get_train_dataloader(self) -> DataLoader:


Test patch: