Repository: Lightning-AI/pytorch-lightning
Pull request: support num_sanity_val_steps=-1
## What does this PR do?

<!--
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
-->

Fixes #1715 
`Trainer(num_sanity_val_steps=-1)` is now possible and will run all val dataloaders in full.

TODO: 
- [x] is `limit_val_batches` supposed to influence sanity val steps? ok, clarified by william

# Before submitting

- [x] Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), Pull Request section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together? Otherwise, we ask you create a separate PR for every change.
- [x] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests? 
- [x] Did you verify new and existing tests pass locally with your changes?
- [x] If you made a notable change (that affects users), did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)?

<!-- For CHANGELOG separate each item in unreleased section by a blank line to reduce collisions -->

## PR review    
Anyone in the community is free to review the PR once the tests have passed.     
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
Run full validation epoch before training
## ❓ Questions and Help

#### What is your question?
How can I manually trigger a full validation step before training?

I want to compute and log the validation metrics before I start the training (ideally also updating the progress bar dictionary).

The reason why I want to do this is that I am fine-tuning a pre-trained model, and I want to check the performances before training.

Hints from issue discussion:
To evaluate the performance of an existing model in your case, it is best practice to implement the test methods in the lightning module and then invoke the Trainer.test(). So I imagine your workflow will roughly be

```
model = YourLightningModule.load_from_checkpoint(...)

trainer = Trainer(options for testing)
Trainer.test()  # in the future, this will return results directly
# for now:
metrics = trainer.progress_bar_metrics  # for example to print them

# now training
model = YourLightningModule.load_from_checkpoint(...) 
trainer = Trainer(args for training)
trainer.fit()
```

There is a reason why the validation is tied to the training and cannot be run so easily from the outside. The validation is conceptually not the same as the test and does not reflect the true performance of a model, because we do things like early stopping based on validation loss etc.
Hi @awaelchli 
I understand that I can run the test before training**, but that's a bit different from what I am trying to achieve.

Currently, the `Trainer` class accepts the `num_sanity_val_steps` which allows users to define how many validation steps to execute before running.
It would be great to be able to set this to a special value, say `-1`, to tell the `Trainer` to run a full validation epoch before training instead of just some fixed number of steps.
Then inside `on_sanity_check_end_called` people would be able to use `metrics_callback` to do whatever they want with the validation results***.

**:  Currently you can't directly call `test()` on a pretrained model unless you manually call `prepare_data()` before the `test()` call, see https://github.com/PyTorchLightning/pytorch-lightning/issues/1562.

***: Currently the `trainer.metrics_callback` inside the `on_sanity_check_end_called` is not populated with the validation metrics.
I'm agree with @simonepri. it's good to run full `validation_sanity_check` before training.
If we can know full `num_sanity_val_steps` before declaring `trainer`, it gonna be simple, however i don't think it's possible.
There is not option for such feature for now, isn't it?
yes maybe we could think about the option `num_sanity_val_steps=-1`
@PyTorchLightning/core-contributors 
I agree. It's good practice to run the validation before the train, and I'll surely use it!
It's pretty straight forward, I guess I'm not the first to write this:)
```
class run_validation_on_start(Callback):
    def __init__(self):
        pass

    def on_train_start(self, trainer: Trainer, pl_module):
        return trainer.run_evaluation(test_mode=False)
```
@awaelchli yes, perfect. let’s do it that way. 

num_sanity_val_steps=-1

PR for 0.8.0?
happy to make it but it would be easier to do after #1920.

Commit message: Merge branch 'master' into feature/sanity-val-full-data

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 26dcb32ad8519..3a3417fda7722 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -8,6 +8,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- Added support for `Trainer(num_sanity_val_steps=-1)` to check all validation data before training ([#2246](https://github.com/PyTorchLightning/pytorch-lightning/pull/2246))
+
 ### Changed
 
 - Changed epoch indexing from 0 instead of 1 ([#2289](https://github.com/PyTorchLightning/pytorch-lightning/pull/2289))
diff --git a/docs/source/debugging.rst b/docs/source/debugging.rst
index 18536082399f4..3384d3c9f25cc 100644
--- a/docs/source/debugging.rst
+++ b/docs/source/debugging.rst
@@ -129,4 +129,4 @@ argument of :class:`~pytorch_lightning.trainer.trainer.Trainer`)
 .. testcode::
 
     # DEFAULT
-    trainer = Trainer(num_sanity_val_steps=5)
\ No newline at end of file
+    trainer = Trainer(num_sanity_val_steps=2)
\ No newline at end of file
diff --git a/pytorch_lightning/callbacks/progress.py b/pytorch_lightning/callbacks/progress.py
index 358575273f945..4ea9ea1c91b4f 100644
--- a/pytorch_lightning/callbacks/progress.py
+++ b/pytorch_lightning/callbacks/progress.py
@@ -295,7 +295,7 @@ def init_test_tqdm(self) -> tqdm:
     def on_sanity_check_start(self, trainer, pl_module):
         super().on_sanity_check_start(trainer, pl_module)
         self.val_progress_bar = self.init_sanity_tqdm()
-        self.val_progress_bar.total = trainer.num_sanity_val_steps * len(trainer.val_dataloaders)
+        self.val_progress_bar.total = convert_inf(trainer.num_sanity_val_steps * len(trainer.val_dataloaders))
         self.main_progress_bar = tqdm(disable=True)  # dummy progress bar
 
     def on_sanity_check_end(self, trainer, pl_module):
diff --git a/pytorch_lightning/trainer/__init__.py b/pytorch_lightning/trainer/__init__.py
index 6e276360470cf..01e137a90407c 100644
--- a/pytorch_lightning/trainer/__init__.py
+++ b/pytorch_lightning/trainer/__init__.py
@@ -603,16 +603,19 @@ def on_train_end(self, trainer, pl_module):
 
 Sanity check runs n batches of val before starting the training routine.
 This catches any bugs in your validation without having to wait for the first validation check.
-The Trainer uses 5 steps by default. Turn it off or modify it here.
+The Trainer uses 2 steps by default. Turn it off or modify it here.
 
 .. testcode::
 
     # default used by the Trainer
-    trainer = Trainer(num_sanity_val_steps=5)
+    trainer = Trainer(num_sanity_val_steps=2)
 
     # turn it off
     trainer = Trainer(num_sanity_val_steps=0)
 
+    # check all validation data
+    trainer = Trainer(num_sanity_val_steps=-1)
+
 num_tpu_cores
 ^^^^^^^^^^^^^
 .. warning:: .. deprecated:: 0.7.6
diff --git a/pytorch_lightning/trainer/evaluation_loop.py b/pytorch_lightning/trainer/evaluation_loop.py
index 59614c2d5fb39..a1353103b4cef 100644
--- a/pytorch_lightning/trainer/evaluation_loop.py
+++ b/pytorch_lightning/trainer/evaluation_loop.py
@@ -72,15 +72,16 @@
 -----------------------------------------
 
 Lightning runs a few steps of validation in the beginning of training.
- This avoids crashing in the validation loop sometime deep into a lengthy training loop.
+This avoids crashing in the validation loop sometime deep into a lengthy training loop.
 
 .. code-block:: python
 
     # DEFAULT
-    trainer = Trainer(num_sanity_val_steps=5)
+    trainer = Trainer(num_sanity_val_steps=2)
 
 
-You can use `Trainer(num_sanity_val_steps=0)` to skip the sanity check.
+You can use `Trainer(num_sanity_val_steps=0)` to skip the sanity check or `Trainer(num_sanity_val_steps=-1)`
+to check all the validation data.
 
 # Testing loop
 
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
index c207d8255d0c2..6f092f7b552a2 100644
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -278,7 +278,8 @@ def __init__(
 
             amp_level: The optimization level to use (O1, O2, etc...).
 
-            num_sanity_val_steps: Sanity check runs n batches of val before starting the training routine.
+            num_sanity_val_steps: Sanity check runs n validation batches before starting the training routine.
+                Set it to `-1` to run all batches in all validation dataloaders. Default: 2
 
             truncated_bptt_steps: Truncated back prop breaks performs backprop every k steps of
 
@@ -376,7 +377,7 @@ def __init__(
         self.max_steps = max_steps
         self.min_steps = min_steps
 
-        self.num_sanity_val_steps = num_sanity_val_steps
+        self.num_sanity_val_steps = float("inf") if num_sanity_val_steps == -1 else num_sanity_val_steps
         # Backward compatibility, TODO: remove in v0.9.0
         if print_nan_grads:
             rank_zero_warn("Argument `print_nan_grads` has no effect and will be removed in v0.9.0."
diff --git a/tests/trainer/test_trainer.py b/tests/trainer/test_trainer.py
index 84b0525c8b2e7..30455537ac5ed 100644
--- a/tests/trainer/test_trainer.py
+++ b/tests/trainer/test_trainer.py
@@ -6,6 +6,7 @@
 import types
 from argparse import Namespace
 from pathlib import Path
+from unittest.mock import patch
 
 import cloudpickle
 import pytest
@@ -804,6 +805,24 @@ def test_tpu_choice(tmpdir, tpu_cores, expected_tpu_id, error_expected):
         assert trainer.tpu_id == expected_tpu_id
 
 
+def test_num_sanity_val_steps(tmpdir):
+    """ Test that num_sanity_val_steps=-1 runs through all validation data once. """
+    model = EvalModelTemplate()
+    trainer = Trainer(
+        num_sanity_val_steps=-1,
+        # TODO: limit_val_batches influences num_sanity_val_step. Fix it.
+        # limit_val_batches=0,
+        max_steps=1,
+        default_root_dir=tmpdir
+    )
+    assert trainer.num_sanity_val_steps == float('inf')
+    val_dataloaders = model.val_dataloader__multiple()
+
+    with patch.object(trainer, 'evaluation_forward', wraps=trainer.evaluation_forward) as mocked:
+        trainer.fit(model, val_dataloaders=val_dataloaders)
+        assert mocked.call_count == sum(len(dl) for dl in val_dataloaders)
+
+
 @pytest.mark.parametrize("trainer_kwargs,expected", [
     pytest.param(
         dict(distributed_backend=None, gpus=None),


Code review comments:
diff_hunk: @@ -804,6 +805,24 @@ def test_tpu_choice(tmpdir, tpu_cores, expected_tpu_id, error_expected):
         assert trainer.tpu_id == expected_tpu_id
 
 
+def test_num_sanity_val_steps(tmpdir):
+    """ Test that num_sanity_val_steps=-1 runs through all validation data once. """
+    model = EvalModelTemplate()
+    trainer = Trainer(
+        num_sanity_val_steps=-1,
+        # TODO: limit_val_batches influences num_sanity_val_step. Fix it.
+        # limit_val_batches=0,
+        max_steps=1,
+        default_root_dir=tmpdir
path: tests/trainer/test_trainer.py
text: @user1:
```suggestion
        default_root_dir=tmpdir,
```