Repository: Lightning-AI/pytorch-lightning
Pull request: More explicit exception message when testing with fast_dev_run=True
## What does this PR do?

Fixes https://github.com/PyTorchLightning/pytorch-lightning/issues/6615.

## Before submitting
- [x] Was this discussed/approved via a GitHub issue? (not for typos and docs)
- [x] Did you read the [contributor guideline](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [x] Did you make sure your PR does only one thing, instead of bundling different changes together?
- [x] Did you make sure to update the documentation with your changes? (if necessary)
- [x] Did you write any new necessary tests? (not for typos and docs)
- [x] Did you verify new and existing tests pass locally with your changes?
- [x] Did you update the [CHANGELOG](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/CHANGELOG.md)? (not for typos, docs, test updates, or internal minor changes/refactorings)

<!-- For CHANGELOG separate each item in the unreleased section by a blank line to reduce collisions -->

## PR review
Anyone in the community is free to review the PR once the tests have passed.
Before you start reviewing make sure you have read [Review guidelines](https://github.com/PyTorchLightning/pytorch-lightning/wiki/Review-guidelines). In short, see the following bullet-list:

 - [x] Is this pull request ready for review? (if not, please submit in draft mode)
 - [x] Check that all items from **Before submitting** are resolved
 - [x] Make sure the title is self-explanatory and the description concisely explains the PR
 - [x] Add labels and milestones (and optionally projects) to the PR so it can be classified

## Did you have fun?
Make sure you had fun coding 🙃


Problem statement:
Calling trainer.test() when using fast_dev_run throws confusing error
## 🐛 Bug

Calling trainer.test() when using fast_dev_run throws confusing error:
```bash
Traceback (most recent call last):                                                                                                                                                                 
  File "main.py", line 89, in <module>
    trainer.test(test_dataloaders=test)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 916, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)
  File "/home/ash/miniconda3/envs/tmp/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 927, in __test_using_best_weights
    raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: ckpt_path is "best", but ModelCheckpoint is not configured to save the best model.
```

## Please reproduce using the BoringModel

```python

from pytorch_lightning import LightningModule
import torch
from torch.utils.data import DataLoader, Dataset
import pytorch_lightning as pl


class RandomDataset(Dataset):
    def __init__(self, size, num_samples):
        self.len = num_samples
        self.data = torch.randn(num_samples, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class BoringModel(LightningModule):

    def __init__(self):
        super().__init__()
        self.layer = torch.nn.Linear(32, 2)

    def forward(self, x):
        return self.layer(x)

    def loss(self, batch, prediction):
        return torch.nn.functional.mse_loss(prediction, torch.ones_like(prediction))

    def training_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"loss": loss}

    def training_step_end(self, training_step_outputs):
        return training_step_outputs

    def training_epoch_end(self, outputs) -> None:
        torch.stack([x["loss"] for x in outputs]).mean()

    def validation_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        return {"x": loss}

    def validation_epoch_end(self, outputs) -> None:
        torch.stack([x['x'] for x in outputs]).mean()

    def test_step(self, batch, batch_idx):
        output = self.layer(batch)
        loss = self.loss(batch, output)
        self.log('fake_test_acc', loss)
        return {"y": loss}

    def test_epoch_end(self, outputs) -> None:
        torch.stack([x["y"] for x in outputs]).mean()

    def configure_optimizers(self):
        optimizer = torch.optim.SGD(self.layer.parameters(), lr=0.1)
        lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1)
        return [optimizer], [lr_scheduler]


num_samples = 10000

train = RandomDataset(32, num_samples)
train = DataLoader(train, batch_size=32)

val = RandomDataset(32, num_samples)
val = DataLoader(val, batch_size=32)

test = RandomDataset(32, num_samples)
test = DataLoader(test, batch_size=32)

model = BoringModel()

trainer = pl.Trainer(
    fast_dev_run=True
)

trainer.fit(model, train, val)

trainer.test(test_dataloaders=test)
```

Hints from issue discussion:
Dear @hobogalaxy,

This happens because trainer.fit didn't generate a checkpoint and you don't provide model to trainer.test.
Therefore, it doesn't find a model to load.
This is expected behaviour but I agree the MisconfigurationException could be more explicit on what's wrong.

Best,
T.C
@tchaton Thank you for explanation. It is confusing though and it probably happens to many users. I think lightning should detect the case when fast_dev_run is set to True and either:
1. Throw python warning instead of exception and simply not execute testing
2. Raise exception that informs explicitly you shouln't call trainer.test() when using fast_dev_run
@hobogalaxy mind send a PR with one of your suggestions?
I think I prefer (2) as it is more explicit

Commit message: update changelog

Patch to review:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 780a8790b9fdd..0dc0ef5cc3ab8 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ### Added
 
+- Added more explicit exception message when trying to execute `trainer.test()` or `trainer.validate()` with `fast_dev_run=True` ([#6667](https://github.com/PyTorchLightning/pytorch-lightning/pull/6667))
+
 
 - Added a way to print to terminal without breaking up the progress bar ([#5470](https://github.com/PyTorchLightning/pytorch-lightning/pull/5470))
 
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
index f0f1d3e6b11e1..6fcb563749154 100644
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -956,10 +956,17 @@ def __load_ckpt_weights(
         ckpt_path: Optional[str] = None,
     ) -> Optional[str]:
         # if user requests the best checkpoint but we don't have it, error
-        if ckpt_path == 'best' and not self.checkpoint_callback.best_model_path:
-            raise MisconfigurationException(
-                'ckpt_path is "best", but `ModelCheckpoint` is not configured to save the best model.'
-            )
+        if ckpt_path == 'best':
+            if not self.checkpoint_callback.best_model_path and self.fast_dev_run:
+                raise MisconfigurationException(
+                    'You cannot execute `trainer.test()` or trainer.validate()`'
+                    ' when `fast_dev_run=True`.'
+                )
+
+            if not self.checkpoint_callback.best_model_path:
+                raise MisconfigurationException(
+                    'ckpt_path is "best", but `ModelCheckpoint` is not configured to save the best model.'
+                )
 
         # load best weights
         if ckpt_path is not None:
diff --git a/tests/trainer/test_trainer.py b/tests/trainer/test_trainer.py
index 4ca2f737f5106..e1c26e80094ef 100644
--- a/tests/trainer/test_trainer.py
+++ b/tests/trainer/test_trainer.py
@@ -1777,3 +1777,19 @@ def on_fit_start(self, trainer, pl_module: LightningModule) -> None:
 
     trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, callbacks=[TestCallback()])
     trainer.fit(model, datamodule=dm)
+
+
+@pytest.mark.parametrize("fast_dev_run", [True, False])
+def test_exception_when_testing_or_validating_with_fast_dev_run(tmpdir, fast_dev_run):
+    model = BoringModel()
+
+    trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, fast_dev_run=fast_dev_run)
+
+    trainer.fit(model)
+
+    if fast_dev_run:
+        with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
+            trainer.validate()
+
+        with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
+            trainer.test()


Code review comments:
diff_hunk: @@ -1777,3 +1777,19 @@ def on_fit_start(self, trainer, pl_module: LightningModule) -> None:
 
     trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, callbacks=[TestCallback()])
     trainer.fit(model, datamodule=dm)
+
+
+@pytest.mark.parametrize("fast_dev_run", [True, False])
+def test_exception_when_testing_or_validating_with_fast_dev_run(tmpdir, fast_dev_run):
+    model = BoringModel()
+
+    trainer = Trainer(default_root_dir=tmpdir, max_epochs=1, fast_dev_run=fast_dev_run)
+
+    trainer.fit(model)
+
+    if fast_dev_run:
+        with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
+            trainer.validate()
+
+        with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
+            trainer.test()
path: tests/trainer/test_trainer.py
text: @user1:
No need to parametrize this test

```suggestion
def test_exception_when_testing_or_validating_with_fast_dev_run(tmpdir):
    model = BoringModel()
    trainer = Trainer(default_root_dir=tmpdir, fast_dev_run=True)
    trainer.fit(model)

    with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
        trainer.validate()

    with pytest.raises(MisconfigurationException, match=".*when `fast_dev_run=True`*"):
        trainer.test()
```

diff_hunk: @@ -956,10 +956,17 @@ def __load_ckpt_weights(
         ckpt_path: Optional[str] = None,
     ) -> Optional[str]:
         # if user requests the best checkpoint but we don't have it, error
-        if ckpt_path == 'best' and not self.checkpoint_callback.best_model_path:
-            raise MisconfigurationException(
-                'ckpt_path is "best", but `ModelCheckpoint` is not configured to save the best model.'
-            )
+        if ckpt_path == 'best':
+            if not self.checkpoint_callback.best_model_path and self.fast_dev_run:
+                raise MisconfigurationException(
+                    'You cannot execute `trainer.test()` or trainer.validate()`'
+                    ' when `fast_dev_run=True`.'
path: pytorch_lightning/trainer/trainer.py
text: @user1:
```suggestion
                    ' with `fast_dev_run=True`.'
```

diff_hunk: @@ -956,10 +956,17 @@ def __load_ckpt_weights(
         ckpt_path: Optional[str] = None,
     ) -> Optional[str]:
         # if user requests the best checkpoint but we don't have it, error
-        if ckpt_path == 'best' and not self.checkpoint_callback.best_model_path:
-            raise MisconfigurationException(
-                'ckpt_path is "best", but `ModelCheckpoint` is not configured to save the best model.'
-            )
+        if ckpt_path == 'best':
+            if not self.checkpoint_callback.best_model_path and self.fast_dev_run:
+                raise MisconfigurationException(
+                    'You cannot execute `trainer.test()` or trainer.validate()`'
path: pytorch_lightning/trainer/trainer.py
text: @user1:
```suggestion
                    'You cannot execute `trainer.test()` or `trainer.validate()`'
```