Repository: Lightning-AI/pytorch-lightning

Issue:
Support running validation every fixed number of examples
**Is your feature request related to a problem? Please describe.**
For IterableDataset, we may not know the length of the dataset in advance. Running validation every X examples would be helpful.



Discussion:
I really would love this! Any updates on that?
:+1: on this, I guess it should be easy to implement with a [Hook](https://williamfalcon.github.io/pytorch-lightning/Trainer/hooks/#on_batch_end)
I'm able to do something similar that by setting up the trainer like this:
```
trainer = Trainer(
    ...,
    val_percent_check=kwargs["val_check_interval"],
    val_check_interval=kwargs["val_check_interval"],
)
```
If you set `val_check_interval` to `(batch_size / len(train_dataset)) * nb_batches` validating every `nb_batches` works. Just be sure to make the `DataLoaders` you use shuffle the data, otherwise the same data will be used over and over again for validation.

Also the `ModelCheckpoint`-call back has to be adjusted:

```
checkpoint_callback = ModelCheckpoint(
    ...,
    period=1 / kwargs["val_check_interval"],
)
```

Early-stopping seems not be influenced by the validation frequency.

Only caveat is that the validation batches are sampled now randomly from the validation dataset, and it's not guaranteed that all data is used after one epoch. Not too sure if that's an issue tho.

Resolving patch:
diff --git a/pytorch_lightning/trainer/data_loading_mixin.py b/pytorch_lightning/trainer/data_loading_mixin.py
--- a/pytorch_lightning/trainer/data_loading_mixin.py
+++ b/pytorch_lightning/trainer/data_loading_mixin.py
@@ -2,6 +2,9 @@
 
 from torch.utils.data.distributed import DistributedSampler
 import torch.distributed as dist
+from torch.utils.data import IterableDataset
+
+from pytorch_lightning.utilities.debugging import MisconfigurationException
 
 try:
     from apex import amp
@@ -15,8 +18,11 @@ class TrainerDataLoadingMixin(object):
     def layout_bookeeping(self):
 
         # determine number of training batches
-        self.nb_training_batches = len(self.get_train_dataloader())
-        self.nb_training_batches = int(self.nb_training_batches * self.train_percent_check)
+        if isinstance(self.get_train_dataloader(), IterableDataset):
+            self.nb_training_batches = float('inf')
+        else:
+            self.nb_training_batches = len(self.get_train_dataloader())
+            self.nb_training_batches = int(self.nb_training_batches * self.train_percent_check)
 
         # determine number of validation batches
         # val datasets could be none, 1 or 2+
@@ -34,8 +40,13 @@ def layout_bookeeping(self):
             self.nb_test_batches = max(1, self.nb_test_batches)
 
         # determine when to check validation
-        self.val_check_batch = int(self.nb_training_batches * self.val_check_interval)
-        self.val_check_batch = max(1, self.val_check_batch)
+        # if int passed in, val checks that often
+        # otherwise, it checks in [0, 1.0] % range of a training epoch
+        if isinstance(self.val_check_interval, int):
+            self.val_check_batch = self.val_check_interval
+        else:
+            self.val_check_batch = int(self.nb_training_batches * self.val_check_interval)
+            self.val_check_batch = max(1, self.val_check_batch)
 
     def get_dataloaders(self, model):
         """
@@ -127,6 +138,16 @@ def get_dataloaders(self, model):
             self.get_test_dataloaders()
             self.get_val_dataloaders()
 
+        # support IterableDataset for train data
+        self.is_iterable_train_dataloader = isinstance(self.get_train_dataloader(), IterableDataset)
+        if self.is_iterable_train_dataloader and not isinstance(self.val_check_interval, int):
+            m = '''
+            When using an iterableDataset for train_dataloader,
+            Trainer(val_check_interval) must be an int.
+            An int k specifies checking validation every k training batches
+            '''
+            raise MisconfigurationException('when using ')
+
     def determine_data_use_amount(self, train_percent_check, val_percent_check,
                                   test_percent_check, overfit_pct):
         """
diff --git a/pytorch_lightning/trainer/train_loop_mixin.py b/pytorch_lightning/trainer/train_loop_mixin.py
--- a/pytorch_lightning/trainer/train_loop_mixin.py
+++ b/pytorch_lightning/trainer/train_loop_mixin.py
@@ -31,7 +31,12 @@ def train(self):
 
             # init progress_bar when requested
             if self.show_progress_bar:
-                self.progress_bar.reset(self.total_batches)
+                nb_iterations = self.total_batches
+
+                #  for iterable train loader, the progress bar never ends
+                if self.is_iterable_train_dataloader:
+                    nb_iterations = float('inf')
+                self.progress_bar.reset(nb_iterations)
 
             # changing gradient according accumulation_scheduler
             self.accumulation_scheduler.on_epoch_begin(epoch_nb, self)
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -101,7 +101,7 @@ def __init__(self,
         :param train_percent_check: int. How much of train set to check
         :param val_percent_check: int. How much of val set to check
         :param test_percent_check: int. How much of test set to check
-        :param val_check_interval: int. Check val this frequently within a train epoch
+        :param val_check_interval: float/int. If float, % of tng epoch. If int, check every n batch
         :param log_save_interval: int. Writes logs to disk this often
         :param row_log_interval: int. How often to add logging rows
         :param add_row_log_interval: int. How often to add logging rows. Deprecated.
@@ -160,6 +160,7 @@ def __init__(self,
         self.get_train_dataloader = None
         self.get_test_dataloaders = None
         self.get_val_dataloaders = None
+        self.is_iterable_train_dataloader = False
 
         # training state
         self.model = None


Test patch: