Repository: Lightning-AI/pytorch-lightning

Issue:
Validation every epoch with non-finite dataloader
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->

Providing a way to do validation every epoch with non-finite (`__len__` not implemented) dataloaders.

### Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

Doing validation every epoch is a natural choice, and with finite dataloader you can do it easily by setting `val_check_interval=1.0`. However, with non-finite dataloader you cannot set `val_check_interval` to be float. There's no simple way to work around it yet.

### Pitch

<!-- A clear and concise description of what you want to happen. -->

There can be several way to make it happen.

One solution on top of my head is to let `val_check_interval` to be `None`. if it is none, just do the validation according to the number of `check_val_every_n_epoch` at the end of every epoch.

### Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

Alternatively, you can let the user set `val_check_interval` to be 1.0 even with non-finite dataloaders. Anything below 1.0 would be invalid but only 1.0 can be valid. If it is 1.0 then do the validation according to the number of `check_val_every_n_epoch` at the end of every epoch.

### Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Not all dataloaders without `__len__` implemented are infinite dataloaders. Some just cannot decide length in advance. With these dataloaders the concept of 'epoch' is still valid. Pytorch-lightning needs to serve this kind of dataloaders better.



Discussion:
Thanks for the issue! Totally agree that we should support this. It might be a bit painful with how validation works at the moment. Currently we validate when the batch index reaches a certain value. But with Iterable stuff we'd need to do it once the itetation is over (and we can't know ahead of time when that will be). I guess that the way to do it is to add a seperate clause which deals specifically with the case when `val_check_interval=1.0`, although it's not the most elegant solution :/
cool, @ethanwharris @dreamgonfly mind drafting a PR?
cc: @PyTorchLightning/core-contributors 

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading.py b/pytorch_lightning/trainer/data_loading.py
--- a/pytorch_lightning/trainer/data_loading.py
+++ b/pytorch_lightning/trainer/data_loading.py
@@ -136,16 +136,19 @@ def reset_train_dataloader(self, model: LightningModule) -> None:
                     'If you want to disable validation set `val_percent_check` to 0.0 instead.')
         else:
             if not _has_len(self.train_dataloader):
-                raise MisconfigurationException(
-                    'When using an infinite DataLoader (e.g. with an IterableDataset or when '
-                    'DataLoader does not implement `__len__`) for `train_dataloader`, '
-                    '`Trainer(val_check_interval)` must be an int. An int k specifies checking '
-                    'validation every k training batches.')
-
-            self._percent_range_check('val_check_interval')
+                if self.val_check_interval == 1.0:
+                    self.val_check_batch = float('inf')
+                else:
+                    raise MisconfigurationException(
+                        'When using an infinite DataLoader (e.g. with an IterableDataset or when '
+                        'DataLoader does not implement `__len__`) for `train_dataloader`, '
+                        '`Trainer(val_check_interval)` must be `1.0` or an int. An int k specifies '
+                        'checking validation every k training batches.')
+            else:
+                self._percent_range_check('val_check_interval')
 
-            self.val_check_batch = int(self.num_training_batches * self.val_check_interval)
-            self.val_check_batch = max(1, self.val_check_batch)
+                self.val_check_batch = int(self.num_training_batches * self.val_check_interval)
+                self.val_check_batch = max(1, self.val_check_batch)
 
     def _reset_eval_dataloader(self, model: LightningModule,
                                mode: str) -> Tuple[int, List[DataLoader]]:
diff --git a/pytorch_lightning/trainer/training_loop.py b/pytorch_lightning/trainer/training_loop.py
--- a/pytorch_lightning/trainer/training_loop.py
+++ b/pytorch_lightning/trainer/training_loop.py
@@ -400,8 +400,8 @@ def run_training_epoch(self):
             train_dataloader = train_dataloader.per_device_loader(device)
 
         # run epoch
-        for batch_idx, batch in self.profiler.profile_iterable(
-            enumerate(train_dataloader), "get_train_batch"
+        for batch_idx, (batch, is_last_batch) in self.profiler.profile_iterable(
+            enumerate(_with_is_last(train_dataloader)), "get_train_batch"
         ):
             # stop epoch if we limited the number of training batches
             if batch_idx >= self.num_training_batches:
@@ -429,8 +429,10 @@ def run_training_epoch(self):
             # ---------------
             is_val_check_batch = (batch_idx + 1) % self.val_check_batch == 0
             can_check_epoch = (self.current_epoch + 1) % self.check_val_every_n_epoch == 0
-            should_check_val = not self.disable_validation and can_check_epoch
-            should_check_val = should_check_val and (is_val_check_batch or early_stop_epoch)
+            can_check_val = not self.disable_validation and can_check_epoch
+            should_check_val = is_val_check_batch or early_stop_epoch
+            should_check_val = should_check_val or (is_last_batch and self.val_check_batch == float('inf'))
+            should_check_val = can_check_val and should_check_val
 
             # fast_dev_run always forces val checking after train batch
             if self.fast_dev_run or should_check_val:
@@ -740,3 +742,16 @@ def call_checkpoint_callback(self):
         if self.checkpoint_callback is not None:
             self.checkpoint_callback.on_validation_end(self, self.get_model())
         self.on_validation_end()
+
+
+def _with_is_last(iterable):
+    """Pass through values from the given iterable with an added boolean indicating if this is the last item.
+    See `https://stackoverflow.com/a/1630350 <https://stackoverflow.com/a/1630350>`_"""
+    it = iter(iterable)
+    last = next(it)
+    for val in it:
+        # yield last and has next
+        yield last, False
+        last = val
+    # yield last, no longer has next
+    yield last, True


Test patch:
diff --git a/tests/trainer/test_dataloaders.py b/tests/trainer/test_dataloaders.py
--- a/tests/trainer/test_dataloaders.py
+++ b/tests/trainer/test_dataloaders.py
@@ -372,7 +372,6 @@ class CurrentTestModel(
         )
         trainer.fit(model)
 
-    # logger file to get meta
     trainer = Trainer(
         default_save_path=tmpdir,
         max_epochs=1,
@@ -383,6 +382,15 @@ class CurrentTestModel(
     # verify training completed
     assert result == 1
 
+    trainer = Trainer(
+        default_save_path=tmpdir,
+        max_epochs=1
+    )
+    result = trainer.fit(model)
+
+    # verify training completed
+    assert result == 1
+
 
 def test_inf_val_dataloader(tmpdir):
     """Test inf val data loader (e.g. IterableDataset)"""