Repository: Lightning-AI/pytorch-lightning

Issue:
lr_find doesn't return the correct suggestion if some losses are nan
<!-- 
### Common bugs:
1. Tensorboard not showing in Jupyter-notebook see [issue 79](https://github.com/PyTorchLightning/pytorch-lightning/issues/79).    
2. PyTorch 1.1.0 vs 1.2.0 support [see FAQ](https://github.com/PyTorchLightning/pytorch-lightning#faq)    
-->

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

lr_finder doesn't return the correct suggestion if some losses are `nan`. The returned loss is the one corresponding to the `nan` value, which is very big in my case.

![image](https://user-images.githubusercontent.com/441707/82111685-3513ea00-978a-11ea-9de2-f436d2fb1750.png)

### To Reproduce

This depends on the dataset. Please see the code sample.

#### Code sample
<!-- Ideally attach a minimal code sample to reproduce the decried issue. 
Minimal means having the shortest code but still preserving the bug. -->

I believe this is caused by numpy. The related code should be https://github.com/PyTorchLightning/pytorch-lightning/blob/b84b02400a312240a6429c186cc63514eeb45a82/pytorch_lightning/trainer/lr_finder.py#L325

```python
example_losses = [0.90, 0.89, 0.87, 0.86, 0.85, 0.84]
print(np.gradient(example_losses).argmin())
example_losses = [0.90, 0.89, 0.87, 0.86, 0.85, 0.84, float('nan')]
print(np.gradient(example_losses).argmin())
```
Output:
```
1
5
```
### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

Return the correct suggested loss.

### Environment

* CUDA:
	- GPU:
	- available:         False
	- version:           10.2
* Packages:
	- numpy:             1.18.4
	- pyTorch_debug:     False
	- pyTorch_version:   1.5.0
	- pytorch-lightning: 0.7.6
	- tensorboard:       2.2.0
	- tqdm:              4.45.0
* System:
	- OS:                Linux
	- architecture:
		- 64bit
		- 
	- processor:         
	- python:            3.7.6
	- version:           #1 SMP Debian 4.19.118-2 (2020-04-29)

### Additional context

<!-- Add any other context about the problem here. -->
NA


Discussion:
Hi! thanks for your contribution!, great first issue!
Good catch. I guess that we can not do much about how `np.gradient` behave but we can filter `Nan` values before doing the calculation. @binshengliu are you up for doing a PR? 
Filtering out `nan` would be a reasonable approach. Locally I just reset `nan` to `inf` also to avoid dealing with indexes.

Sorry I'm quite inundated with my own projects recently and may not have enough time to shape a proper PR.

Resolving patch:
diff --git a/pytorch_lightning/trainer/lr_finder.py b/pytorch_lightning/trainer/lr_finder.py
--- a/pytorch_lightning/trainer/lr_finder.py
+++ b/pytorch_lightning/trainer/lr_finder.py
@@ -321,8 +321,9 @@ def suggestion(self, skip_begin: int = 10, skip_end: int = 1):
 
         """
         try:
-            loss = self.results["loss"][skip_begin:-skip_end]
-            min_grad = (np.gradient(np.array(loss))).argmin()
+            loss = np.array(self.results["loss"][skip_begin:-skip_end])
+            loss = loss[np.isfinite(loss)]
+            min_grad = np.gradient(loss).argmin()
             self._optimal_idx = min_grad + skip_begin
             return self.results["lr"][self._optimal_idx]
         except Exception:


Test patch:
diff --git a/tests/trainer/test_lr_finder.py b/tests/trainer/test_lr_finder.py
--- a/tests/trainer/test_lr_finder.py
+++ b/tests/trainer/test_lr_finder.py
@@ -124,7 +124,7 @@ def test_call_to_trainer_method(tmpdir):
     # logger file to get meta
     trainer = Trainer(
         default_save_path=tmpdir,
-        max_epochs=5,
+        max_epochs=5
     )
 
     lrfinder = trainer.lr_find(model, mode='linear')
@@ -170,7 +170,7 @@ def test_suggestion_parameters_work(tmpdir):
     # logger file to get meta
     trainer = Trainer(
         default_save_path=tmpdir,
-        max_epochs=10,
+        max_epochs=10
     )
 
     lrfinder = trainer.lr_find(model)
@@ -179,3 +179,24 @@ def test_suggestion_parameters_work(tmpdir):
 
     assert lr1 != lr2, \
         'Skipping parameter did not influence learning rate'
+
+
+def test_suggestion_with_non_finite_values(tmpdir):
+    """ Test that non-finite values does not alter results """
+
+    hparams = EvalModelTemplate.get_default_hparams()
+    model = EvalModelTemplate(hparams)
+
+    # logger file to get meta
+    trainer = Trainer(
+        default_save_path=tmpdir,
+        max_epochs=10
+    )
+
+    lrfinder = trainer.lr_find(model)
+    before_lr = lrfinder.suggestion()
+    lrfinder.results['loss'][-1] = float('nan')
+    after_lr = lrfinder.suggestion()
+
+    assert before_lr == after_lr, \
+        'Learning rate was altered because of non-finite loss values'