Repository: Lightning-AI/pytorch-lightning

Issue:
Support to turn on cudnn benchmark mode
Cudnn benchmark mode is [recommended for improved performance](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/11).

It would be great to have a one-liner option to turn this on. This could be a `Trainer` constructor arg. I'm happy to do a quick PR/documentation update for this, just want to get feedback/confirmation before I do it.

Only downside/complexity is [when determinism is also enabled](https://github.com/pytorch/pytorch/issues/6351). AFAICT, there's no option for determinism in Lightning, so seems like this shouldn't be a problem.
Support to turn on cudnn benchmark mode
Cudnn benchmark mode is [recommended for improved performance](https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936/11).

It would be great to have a one-liner option to turn this on. This could be a `Trainer` constructor arg. I'm happy to do a quick PR/documentation update for this, just want to get feedback/confirmation before I do it.

Only downside/complexity is [when determinism is also enabled](https://github.com/pytorch/pytorch/issues/6351). AFAICT, there's no option for determinism in Lightning, so seems like this shouldn't be a problem.


Discussion:
Why would you want this inside lightning instead of you doing it yourself? I'm hesitant to change the global environment inside lightning
I was thinking about it as a performance best practice along the lines of
ddp that should be on in most cases. Happy to keep it outside as well (what
I am doing locally already anyways) if you think that is cleaner.

On Wed, Oct 16, 2019 at 2:09 PM William Falcon <notifications@github.com>
wrote:

> Why would you want this inside lightning instead of you doing it yourself?
> I'm hesitant to change the global environment inside lightning
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/williamFalcon/pytorch-lightning/issues/370?email_source=notifications&email_token=AADYHNUQSPXVQ5OZ5LSFZVDQO57I5A5CNFSM4JAWOR4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBN6GTA#issuecomment-542892876>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYHNRQO6IMIM4NKVVZU7DQO57I5ANCNFSM4JAWOR4A>
> .
>


-- 
----------------------------------------------------------------------------------------------------
David Kossnick | Latent Space | We're hiring!
<https://angel.co/company/latentspace/jobs> | david@latentspace.co

I think it's a good idea to add to our pl-example repo or some pl-best practice repo
@kossnick why don't you submit a PR with a new flag:

```python
Trainer(enable_benchmarking=False)
```

And do what needs to be done to enable this. Also, make sure to write a test for this.
In the future this flag can be used to group all of these optimization tricks under.


This feature is quite useful in my usage. How is the status of this issue for adding a new flag ? Is there any plan to release this feature? 
Hi all, sorry for the delay. I wrote the very simple / limited code, but haven't had time to get a full setup with multi-GPU to run all the required tests. I'm unassigning myself for now, anyone feel free to pick up!
@jwwandy @kossnick anyone want to look at this?
Why would you want this inside lightning instead of you doing it yourself? I'm hesitant to change the global environment inside lightning
I was thinking about it as a performance best practice along the lines of
ddp that should be on in most cases. Happy to keep it outside as well (what
I am doing locally already anyways) if you think that is cleaner.

On Wed, Oct 16, 2019 at 2:09 PM William Falcon <notifications@github.com>
wrote:

> Why would you want this inside lightning instead of you doing it yourself?
> I'm hesitant to change the global environment inside lightning
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/williamFalcon/pytorch-lightning/issues/370?email_source=notifications&email_token=AADYHNUQSPXVQ5OZ5LSFZVDQO57I5A5CNFSM4JAWOR4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBN6GTA#issuecomment-542892876>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYHNRQO6IMIM4NKVVZU7DQO57I5ANCNFSM4JAWOR4A>
> .
>


-- 
----------------------------------------------------------------------------------------------------
David Kossnick | Latent Space | We're hiring!
<https://angel.co/company/latentspace/jobs> | david@latentspace.co

I think it's a good idea to add to our pl-example repo or some pl-best practice repo
@kossnick why don't you submit a PR with a new flag:

```python
Trainer(enable_benchmarking=False)
```

And do what needs to be done to enable this. Also, make sure to write a test for this.
In the future this flag can be used to group all of these optimization tricks under.


This feature is quite useful in my usage. How is the status of this issue for adding a new flag ? Is there any plan to release this feature? 
Hi all, sorry for the delay. I wrote the very simple / limited code, but haven't had time to get a full setup with multi-GPU to run all the required tests. I'm unassigning myself for now, anyone feel free to pick up!
@jwwandy @kossnick anyone want to look at this?

Resolving patch:
diff --git a/pytorch_lightning/trainer/trainer.py b/pytorch_lightning/trainer/trainer.py
--- a/pytorch_lightning/trainer/trainer.py
+++ b/pytorch_lightning/trainer/trainer.py
@@ -110,6 +110,7 @@ def __init__(
             truncated_bptt_steps: Optional[int] = None,
             resume_from_checkpoint: Optional[str] = None,
             profiler: Optional[BaseProfiler] = None,
+            benchmark: bool = False,
             reload_dataloaders_every_epoch: bool = False,
     ):
         r"""
@@ -583,12 +584,26 @@ def __init__(
                     trainer = Trainer(profiler=profiler)
             reload_dataloaders_every_epoch: Set to True to reload dataloaders every epoch
 
+            benchmark (bool): If true enables cudnn.benchmark.
+                This flag is likely to increase the speed of your system if your
+                input sizes don't change. However, if it does, then it will likely
+                make your system slower.
+
+                The speedup comes from allowing the cudnn auto-tuner to find the best
+                algorithm for the hardware `[see discussion here]
+                <https://discuss.pytorch.org/t/what-does-torch-backends-cudnn-benchmark-do/5936>`_.
+
         .. warning:: Following arguments become deprecated and they will be removed in v0.8.0:
 
             - `nb_sanity_val_steps`
 
         """
 
+        # benchmarking
+        self.benchmark = benchmark
+        if benchmark:
+            torch.backends.cudnn.benchmark = True
+
         # Transfer params
         # Backward compatibility
         if nb_gpu_nodes is not None:


Test patch:
diff --git a/tests/test_trainer.py b/tests/test_trainer.py
--- a/tests/test_trainer.py
+++ b/tests/test_trainer.py
@@ -782,6 +782,40 @@ def test_trainer_min_steps_and_epochs(tmpdir):
         trainer.current_epoch > 0, "Model did not train for at least min_steps"
 
 
+def test_benchmark_option(tmpdir):
+    """Verify benchmark option."""
+    tutils.reset_seed()
+
+    class CurrentTestModel(
+        LightningValidationMultipleDataloadersMixin,
+        LightningTestModelBase
+    ):
+        pass
+
+    hparams = tutils.get_hparams()
+    model = CurrentTestModel(hparams)
+
+    # verify torch.backends.cudnn.benchmark is not turned on
+    assert not torch.backends.cudnn.benchmark
+
+    # logger file to get meta
+    trainer_options = dict(
+        default_save_path=tmpdir,
+        max_epochs=1,
+        benchmark=True,
+    )
+
+    # fit model
+    trainer = Trainer(**trainer_options)
+    result = trainer.fit(model)
+
+    # verify training completed
+    assert result == 1
+
+    # verify torch.backends.cudnn.benchmark is not turned off
+    assert torch.backends.cudnn.benchmark
+
+
 def test_testpass_overrides(tmpdir):
     hparams = tutils.get_hparams()
     from pytorch_lightning.utilities.debugging import MisconfigurationException