Repository: huggingface/accelerate

Issue:
[Feature Request] Split `sampler.generator` and `loader.generator` to leave loader's generator unsynchronized by default.
`DataLoaderShard` will synchronize `self.generator`(i.e. `loader.generator`) of all processes at each new iteration.
The [documentation](https://huggingface.co/docs/accelerate/quicktour#internal-mechanism) says "the generator attribute of a given sampler" will be synchronized.
However current implementation also synchronize loader's generator, because `prepare_data_loader` will pass sampler's generator to `DataLoaderShard.__init__`, which executes `self.generator = generator`
Theoretically, only `sampler.generator` is required to be synchronized so that each process sees the same random permutation of samples.
Is this intended, or the expected behavior should be `self.sampler_generator = generator`?

PyTorch `DataLoader` will use `loader.generator` to generate `base_seed` for workers.
Without `accelerate`, one can simply call `torch.manual_seed` with different seed, or pass different generator to `DataLoader` in different process, to get different `base_seed`.

`base_seed` will be used to seed main random number generator of worker process:
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/_utils/worker.py#L208-L229

However, loader prepared by `accelerate` will synchronize `loader.generator`, which is used to generate `base_seed`.

It took me 1 hour to figure out why `torch.manual_seed(device.index)` doesn't work -- all GPU's DataLoader get the same `base_seed` for workers. (Initially I expect `loader.generator is None`, however it doesn't. It turns out that `accelerate` will create a generator for prepared loader.)

The warning in the [documentation](https://huggingface.co/docs/accelerate/quicktour#internal-mechanism) clearly show that "Synchronization of the main torch (or CUDA or XLA) random number generator" can cause duplicated random data augmentation.
> Synchronization of the main torch (or CUDA or XLA) random number generator will affect any other potential random artifacts you could have in your dataset (like random data augmentation) in the sense that all processes will get the same random numbers from the torch random modules (so will apply the same random data augmentation if it’s controlled by torch).

However, with the mechanism described above, loader prepared by `accelerate` can cause duplicated random data augmentation without "Synchronization of the main torch (or CUDA or XLA) random number generator".

Even worse, this problem only happens if `num_workers > 1`! I think this is undesired.


Discussion:
Please use the [forums](https://discuss.huggingface.co/) to ask such questions as we keep the issues for bugs and feature requests only. Not sure what the actual question is, but the sync is needed so that each process sees the same random permutation of samples. You can read more about it in the [documentation](https://huggingface.co/docs/accelerate/quicktour#internal-mechanism).
Sorry for my unclear presentation. The actual question is, can we split sampler's generator and dataloader's generator?
Detailed motivation: Pytorch DataLoader will use `loader.generator` to generate `base_seed` for workers. Without `accelerate`, users can simply manual_seed for different GPU process, to get different `base_seed`.
Ah, thanks for clarifying! Right now it's not possible, but it's certainly something we can look into. Can you tell us a little bit more about the use case?
I completely rewrite this issue and describe the use case.
To a certain extent, this is a bug report instead of a feature request.
Since the synchronization of `base_seed` is unexpected.
Related PyTorch code:
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/dataloader.py#L424-L437
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/dataloader.py#L378-L383
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/dataloader.py#L601
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/dataloader.py#L1022-L1028
https://github.com/pytorch/pytorch/blob/eafc910d16a99200af089099e24468c7f8926a05/torch/utils/data/_utils/worker.py#L208-L229
As you can see, the execution of `DataLoaderShard.__iter__` (`num_workers > 1`)will be:
1. `DataLoaderShard` synchronize `self.generator`(i.e. `loader.generator`)
2. `DataLoaderShard` call `DataLoader.__iter__`
3. `_MultiProcessingDataLoaderIter` is created with `self._base_seed = torch.empty((), dtype=torch.int64).random_(generator=loader.generator).item()` (in `_BaseDataLoaderIter.__init__`)
4. Synchronized `self._base_seed` is passed to `_worker_loop` to initialize the worker.
5. `random.seed(base_seed + worker_id)`, `torch.manual_seed(base_seed + worker_id)`, `np.random.seed(_generate_state(base_seed, worker_id))`

Then the same random data augmentation will be applied for workers with the same `worker_id` belonging to different GPUs.

Resolving patch:
diff --git a/src/accelerate/data_loader.py b/src/accelerate/data_loader.py
--- a/src/accelerate/data_loader.py
+++ b/src/accelerate/data_loader.py
@@ -352,16 +352,16 @@ class DataLoaderShard(DataLoader):
         - **total_dataset_length** (`int`) -- Total length of the inner dataset across all processes.
     """
 
-    def __init__(self, dataset, device=None, rng_types=None, generator=None, **kwargs):
+    def __init__(self, dataset, device=None, rng_types=None, synchronized_generator=None, **kwargs):
         super().__init__(dataset, **kwargs)
         self.device = device
         self.rng_types = rng_types
-        self.generator = generator
+        self.synchronized_generator = synchronized_generator
         self.gradient_state = GradientState()
 
     def __iter__(self):
         if self.rng_types is not None:
-            synchronize_rng_states(self.rng_types, self.generator)
+            synchronize_rng_states(self.rng_types, self.synchronized_generator)
         self.gradient_state._set_end_of_dataloader(False)
         try:
             length = getattr(self.dataset, "total_dataset_length", len(self.dataset))
@@ -650,12 +650,12 @@ def prepare_data_loader(
     # Iterable dataset doesn't like batch_sampler, but data_loader creates a default one for it
     new_batch_sampler = dataloader.batch_sampler if not isinstance(new_dataset, IterableDataset) else None
     sampler_is_batch_sampler = False
-    generator = getattr(dataloader, "generator", None)
+    synchronized_generator = None
     # No change if no multiprocess
     if (num_processes != 1 or state.distributed_type == DistributedType.MEGATRON_LM) and not dispatch_batches:
         if isinstance(new_dataset, IterableDataset):
             if getattr(dataloader.dataset, "generator", None) is not None:
-                generator = dataloader.dataset.generator
+                synchronized_generator = dataloader.dataset.generator
             new_dataset = IterableDatasetShard(
                 new_dataset,
                 batch_size=dataloader.batch_size,
@@ -674,8 +674,7 @@ def prepare_data_loader(
             if hasattr(sampler, "generator"):
                 if sampler.generator is None:
                     sampler.generator = torch.Generator()
-                generator = sampler.generator
-                generator.manual_seed(int(torch.empty((), dtype=torch.int64).random_().item()))
+                synchronized_generator = sampler.generator
 
             batch_sampler = dataloader.sampler if sampler_is_batch_sampler else dataloader.batch_sampler
             new_batch_sampler = BatchSamplerShard(
@@ -692,10 +691,9 @@ def prepare_data_loader(
         "sampler",
         "batch_sampler",
         "drop_last",
-        "generator",
     ]
 
-    if rng_types is not None and generator is None and "generator" in rng_types:
+    if rng_types is not None and synchronized_generator is None and "generator" in rng_types:
         rng_types.remove("generator")
 
     kwargs = {
@@ -710,6 +708,7 @@ def prepare_data_loader(
         kwargs["batch_size"] = dataloader.batch_size // num_processes if split_batches else dataloader.batch_size
 
     if dispatch_batches:
+        kwargs.pop("generator")
         dataloader = DataLoaderDispatcher(
             new_dataset,
             split_batches=split_batches,
@@ -722,9 +721,9 @@ def prepare_data_loader(
             new_dataset,
             device=device if put_on_device and state.distributed_type != DistributedType.TPU else None,
             sampler=new_batch_sampler,
-            batch_size=getattr(dataloader, "batch_size", _PYTORCH_DATALOADER_KWARGS["batch_size"]),
+            batch_size=dataloader.batch_size,
             rng_types=rng_types,
-            generator=generator,
+            synchronized_generator=synchronized_generator,
             **kwargs,
         )
     else:
@@ -733,7 +732,7 @@ def prepare_data_loader(
             device=device if put_on_device and state.distributed_type != DistributedType.TPU else None,
             batch_sampler=new_batch_sampler,
             rng_types=rng_types,
-            generator=generator,
+            synchronized_generator=synchronized_generator,
             **kwargs,
         )
 


Test patch: