Repository: Lightning-AI/pytorch-lightning

Issue:
Trainer flag overfit_batches does not overwrite train dataloaders shuffle flag
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
Setting the trainer flag overfit_batches (e.g. =10) does not overwrite the shuffle flag set in the training dataloader, even though the warning reads:
`UserWarning: You requested to overfit but enabled training dataloader shuffling. We are turning it off for you.`
### To Reproduce

Steps to reproduce the behavior:
1. Create lightning module with method train_dataloader with flag shuffle=True: 
```
   def train_dataloader(self) -> loading.DataLoader:
        dataset = ProstateX(train=True)
        batch_transforms, gpu_transforms, sample_transforms = self.get_transformations()
        dataloader = loading.DataLoader(dataset,
                                        batch_size=self.hparams.tr_batch_size,
                                        batch_transforms=batch_transforms,
                                        shuffle=True,
                                        sample_transforms= sample_transforms,
                                        gpu_transforms=gpu_transforms,
                                        pseudo_batch_dim=True,
                                        num_workers=self.hparams.num_workers)
        return dataloader 
```
( I use a rising dataloader, bug should also occur with pytorch dataloaders though)

1. Create main.py with:
```
mymodel = model.Model3D(cfg)
trainer = pl.Trainer(gpus=1, precision=16, overfit_batches=10)
trainer.fit(mymodel)`
```
2. Run main.py
3. Find out that your model does not converge.
4. set shuffle=False when creating Dataloader in train_dataloader
5. See that your model converges after some epochs.

(Or log the samples loaded by the dataloader and check if they are the same each epoch.)
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->


#### Code sample
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

### Expected behavior
Either model also converges with shuffle=True, since warning says that it got overwritten (assuming model converges with shuffle=False) or at least warning should read that user has to change shuffle to False.

### Environment

* CUDA:
        - GPU:
                - GeForce GTX 1080 Ti
        - available:         True
        - version:           10.1
* Packages:
        - numpy:             1.19.0
        - pyTorch_debug:     False
        - pyTorch_version:   1.7.0.dev20200705+cu101
        - pytorch-lightning: 0.8.5
        - tensorboard:       2.2.2
        - tqdm:              4.47.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - 
        - processor:         x86_64
        - python:            3.7.7
        - version:           #109-Ubuntu SMP Fri Jun 19 11:33:10 UTC 2020


### Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Hi! thanks for your contribution!, great first issue!
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -167,6 +167,12 @@ def reset_train_dataloader(self, model: LightningModule) -> None:
             model: The current `LightningModule`
         """
         self.train_dataloader = self.request_dataloader(model.train_dataloader)
+        if (self.overfit_batches > 0):
+            if hasattr(self.train_dataloader, 'sampler') and isinstance(self.train_dataloader.sampler, RandomSampler):
+                rank_zero_warn('You requested to overfit but enabled training dataloader shuffling.'
+                               ' We are turning it off for you.')
+                self.train_dataloader = self.replace_sampler(
+                    self.train_dataloader, SequentialSampler(self.train_dataloader.dataset))
 
         # debugging
         self.dev_debugger.track_load_dataloader_call('train_dataloader', dataloaders=[self.train_dataloader])
@@ -247,7 +253,7 @@ def _reset_eval_dataloader(
 
                 # when overfitting, the dataloader should not have sampler
                 if self.overfit_batches > 0:
-                    rank_zero_warn('You requested to overfit but enabled training dataloader shuffling.'
+                    rank_zero_warn('You requested to overfit but enabled test/val dataloader shuffling.'
                                    ' We are turning it off for you.')
                     dataloaders[loader_i] = self.replace_sampler(loader, SequentialSampler(loader.dataset))
 


Test patch:
diff --git a/tests/trainer/test_trainer_tricks.py b/tests/trainer/test_trainer_tricks.py
--- a/tests/trainer/test_trainer_tricks.py
+++ b/tests/trainer/test_trainer_tricks.py
@@ -56,11 +56,13 @@ def test_overfit_batch_limits(tmpdir):
     # ------------------------------------------------------
     # get the training loader and batch
     # ------------------------------------------------------
+    # Create a reference train dataloader without shuffling.
     train_loader = DataLoader(model.train_dataloader().dataset, shuffle=False)
+    (xa, ya) = next(iter(train_loader))
+    train_loader = DataLoader(model.train_dataloader().dataset, shuffle=True)
     full_train_samples = len(train_loader)
     num_train_samples = int(0.11 * full_train_samples)
 
-    (xa, ya) = next(iter(train_loader))
 
     # ------------------------------------------------------
     # set VAL and Test loaders
@@ -87,7 +89,8 @@ def test_overfit_batch_limits(tmpdir):
 
     trainer = Trainer(overfit_batches=0.11)
     trainer.reset_train_dataloader(model)
-    assert trainer.train_dataloader is train_loader
+    # The dataloader should have been overwritten with a Sequential sampler.
+    assert trainer.train_dataloader is not train_loader
     assert trainer.num_training_batches == num_train_samples
 
     # make sure the loaders are the same