Repository: Lightning-AI/pytorch-lightning

Issue:
The total number of batches shows by the progress bar of the sanity check is wrong
## 🐛 Bug
The `total` of the sanity check progress bar is set by
https://github.com/PyTorchLightning/pytorch-lightning/blob/4d0406ec8bf1c9147b34eb607411b78a9cd28243/pytorch_lightning/callbacks/progress.py#L296

The progress bar will always show `trainer.num_sanity_val_steps` even if  the length of the validation `DataLoader` is less than `trainer.num_sanity_val_steps`.

Maybe the `total` could be computed by
```python
from pytorch_lightning.trainer import data_loading

num_full_val_dataloader_batches = [
    len(dataloader) if data_loading._has_len(dataloader) else float('inf')
    for dataloader in trainer.val_dataloaders
]
self.val_progress_bar.total = convert_inf(
    sum(min(num_batches, trainer.num_sanity_val_steps)
            for num_batches in num_full_val_dataloader_batches))
```

We use the private function `data_loading._has_len` to check if `dataloader` has `__len__`, maybe we could make `data_loading._has_len` public. 

Or we could make `num_full_val_dataloader_batches` (and `num_full_train_dataloader_batches`) a member variable of `Trainer` and update the value in `pytorch_lightning.trainer.data_loading.TrainerDataLoadingMixin`. 

### To Reproduce

The progress bar of the sanity check in the following code (`num_sanity_val_steps == 999` and `len(val_data_loader) == 10`) shows
```
Validation sanity check:   1%|          | 9/999 [00:09<16:31,  1.00s/it]`
```

#### Code sample
```python
import time

import pytorch_lightning as pl
from torch.utils import data


class Dataset(data.Dataset):

  def __init__(self, length):
    self._elements = list(range(length))

  def __getitem__(self, item):
    return self._elements[item]

  def __len__(self):
    return len(self._elements)


class Model(pl.LightningModule):

  def forward(self, *args, **kwargs):
    pass

  def training_step(self, *args, **kwargs):
    pass

  def train_dataloader(self):
    pass

  def configure_optimizers(self):
    pass

  def validation_step(self, *args, **kwargs):
    time.sleep(1)
    return pl.EvalResult()


if __name__ == '__main__':
  model = Model()

  val_dataset_length = 10
  val_dataset = Dataset(val_dataset_length)
  val_data_loader = data.DataLoader(val_dataset)

  trainer = pl.Trainer(num_sanity_val_steps=999, limit_val_batches=999,
                       max_epochs=0)
  trainer.fit(model, val_dataloaders=val_data_loader)

```
### Expected behavior
The program above should be
```
Validation sanity check: 100%|██████████| 10/10 [00:10<00:00,  1.00s/it]
```
### Environment
* CUDA:
	- GPU:
	- available:
	- version:
* Packages:
	- numpy:             1.18.5
	- pyTorch_debug:     False
	- pyTorch_version:   1.6.0+cpu
	- pytorch-lightning: 0.9.0rc11
	- tensorboard:       1.15.0
	- tqdm:              4.48.2
* System:
	- OS:                Windows
	- architecture:
		- 64bit
		- WindowsPE
	- processor:
	- python:            3.7.3
	- version:           10.0.18362

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:
@manipopopo I guess the first solution gets things done without changing anything else. Mind submitting a PR?
I think once #2882 gets resolved this issue will be resolved automatically.
It seems that resolve #2882 will make the sanity check of `Trainer(num_sanity_val_steps=5)` run exactly `5` steps (or `5 * len(self.val_dataloaders)` ) if the validation `DataLoader` has at least 5 batches.

Resolve this issue will make the `total` of the progress bar for the sanity check be the size of the validation `DataLoader` if it is less than `num_sanity_val_steps` (except `num_sanity_val_steps == -1`).
then I would suggest fixing #2882 first in which we can make self.num_sanity_val_steps to be a list of correct num_steps for each val_dataloader and then we can just do `sum(self.num_sanity_val_steps)` here to assign it to progress_bar.
It seems that we have several ways to tackle the problem.

- Make self.num_sanity_val_steps a list of correct numbers of steps as @rohitgr7  [suggests](https://github.com/PyTorchLightning/pytorch-lightning/issues/2891#issuecomment-671043532). `ProgressBar` can update `self.val_progress_bar.total` using `self.num_sanity_val_steps`. Accessing the member `num_sanity_val_steps` may get values different from the one passed into `Trainer.__init__`.

- Add a new member to save a list of correct numbers of steps or save a list of the sizes of validation `DataLoader`s. `ProgressBar` can update `self.val_progress_bar.total` using the new member.  Users of `Trainer` can still get `num_sanity_val_steps` passed into `Trainer.__init__` by accessing the member `num_sanity_val_steps`. (Supposing #2882 is resolved and the member `num_sanity_val_steps` is independent of `limit_val_batches `)

- Compute the correct number of steps in the `ProgressBar` with the help of `pytorch_lightning.trainer.data_loading._has_len`



It seems that the values of the public member `Trainer.num_sanity_val_steps` in stable `0.8.5` and `master` are different.

- In `0.8.5`, `Trainer.num_sanity_val_steps` is `num_sanity_val_steps` passed in `__init__`.

- https://github.com/PyTorchLightning/pytorch-lightning/blob/1e68968ed7fb9b8f73df148dd48194d469655ea3/pytorch_lightning/trainer/trainer.py#L491
sets `Trainer.num_sanity_val_steps` to `float('inf')` when the one passed in `__init__` is `-1`.

- https://github.com/PyTorchLightning/pytorch-lightning/blob/a59e140ee814d8818d121405582133cf6b767e1a/pytorch_lightning/trainer/trainer.py#L461
decides `Trainer.num_sanity_val_steps` according to `limit_val_batches ` and `num_sanity_val_steps` passed in `__init__`.
Fixed by #2917.
It's broken again due to refactors, opening this so that we don't forget to fix this. If required, let's fix this after a week once refactors are done.
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/trainer.py#L1256
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/evaluation_loop.py#L246
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/trainer/evaluate_loop.py#L53-L57
https://github.com/PyTorchLightning/pytorch-lightning/blob/85cd558a3f71a4e7559bda9a31c8e416d4297a16/pytorch_lightning/callbacks/progress.py#L341-L344
this issue is annoying. will fix this.

Resolving patch:
diff --git a/pytorch_lightning/callbacks/progress.py b/pytorch_lightning/callbacks/progress.py
--- a/pytorch_lightning/callbacks/progress.py
+++ b/pytorch_lightning/callbacks/progress.py
@@ -340,8 +340,9 @@ def on_train_batch_end(self, trainer, pl_module, batch, batch_idx, dataloader_id
 
     def on_validation_start(self, trainer, pl_module):
         super().on_validation_start(trainer, pl_module)
-        self.val_progress_bar = self.init_validation_tqdm()
-        self.val_progress_bar.total = convert_inf(self.total_val_batches)
+        if not trainer.running_sanity_check:
+            self.val_progress_bar = self.init_validation_tqdm()
+            self.val_progress_bar.total = convert_inf(self.total_val_batches)
 
     def on_validation_batch_end(self, trainer, pl_module, batch, batch_idx, dataloader_idx):
         super().on_validation_batch_end(trainer, pl_module, batch, batch_idx, dataloader_idx)


Test patch:
diff --git a/tests/callbacks/test_progress_bar.py b/tests/callbacks/test_progress_bar.py
--- a/tests/callbacks/test_progress_bar.py
+++ b/tests/callbacks/test_progress_bar.py
@@ -193,3 +193,37 @@ def on_test_batch_end(self, trainer, pl_module, batch, batch_idx, dataloader_idx
 
     trainer.test(model)
     assert progress_bar.test_batches_seen == progress_bar.total_test_batches
+
+
+@pytest.mark.parametrize(['limit_val_batches', 'expected'], [
+    pytest.param(0, 0),
+    pytest.param(5, 7),
+])
+def test_num_sanity_val_steps_progress_bar(tmpdir, limit_val_batches, expected):
+    """
+    Test val_progress_bar total with 'num_sanity_val_steps' Trainer argument.
+    """
+    class CurrentProgressBar(ProgressBar):
+        def __init__(self):
+            super().__init__()
+            self.val_progress_bar_total = 0
+
+        def on_validation_epoch_end(self, trainer, pl_module):
+            self.val_progress_bar_total += trainer.progress_bar_callback.val_progress_bar.total
+
+    model = EvalModelTemplate()
+    progress_bar = CurrentProgressBar()
+
+    trainer = Trainer(
+        default_root_dir=tmpdir,
+        max_epochs=1,
+        num_sanity_val_steps=2,
+        limit_train_batches=0,
+        limit_val_batches=limit_val_batches,
+        callbacks=[progress_bar],
+        logger=False,
+        checkpoint_callback=False,
+        early_stop_callback=False,
+    )
+    trainer.fit(model)
+    assert trainer.progress_bar_callback.val_progress_bar_total == expected
diff --git a/tests/trainer/test_trainer.py b/tests/trainer/test_trainer.py
--- a/tests/trainer/test_trainer.py
+++ b/tests/trainer/test_trainer.py
@@ -956,7 +956,6 @@ def test_num_sanity_val_steps(tmpdir, limit_val_batches):
         max_steps=1,
     )
     assert trainer.num_sanity_val_steps == num_sanity_val_steps
-    val_dataloaders = model.val_dataloader__multiple_mixed_length()
 
 
 @pytest.mark.parametrize(['limit_val_batches'], [
@@ -980,7 +979,6 @@ def test_num_sanity_val_steps_neg_one(tmpdir, limit_val_batches):
         max_steps=1,
     )
     assert trainer.num_sanity_val_steps == float('inf')
-    val_dataloaders = model.val_dataloader__multiple()
 
 
 @pytest.mark.parametrize("trainer_kwargs,expected", [