Repository: Lightning-AI/pytorch-lightning
Pull request: Add a warning to the `detect_anomaly` flag
## What does this PR do?

<!--
Please include a summary of the change and which issue is fixed.
Please also include relevant motivation and context.
List any dependencies that are required for this change.

If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

The following links the related issue to the PR (https://docs.github.com/en/free-pro-team@latest/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
-->

Fixes #17359

<!-- Does your PR introduce any breaking changes? If yes, please list them. -->

<details>
  <summary><b>Before submitting</b></summary>

- Was this **discussed/agreed** via a GitHub issue? (not for typos and docs)
- [ ] Did you read the [contributor guideline](https://github.com/Lightning-AI/lightning/blob/master/.github/CONTRIBUTING.md), **Pull Request** section?
- [ ] Did you make sure your **PR does only one thing**, instead of bundling different changes together?
- Did you make sure to **update the documentation** with your changes? (if necessary)
- Did you write any **new necessary tests**? (not for typos and docs)
- [ ] Did you verify new and **existing tests pass** locally with your changes?
- Did you list all the **breaking changes** introduced by this pull request?
- Did you **update the CHANGELOG**? (not for typos, docs, test updates, or minor internal changes/refactors)

<!-- In the CHANGELOG, separate each item in the unreleased section by a blank line to reduce collisions -->

</details>

## PR review

Anyone in the community is welcome to review the PR.
Before you start reviewing, make sure you have read the [review guidelines](https://github.com/Lightning-AI/lightning/wiki/Review-guidelines). In short, see the following bullet-list:

<details>
  <summary>Reviewer checklist</summary>

- [ ] Is this pull request ready for review? (if not, please submit in draft mode)
- [ ] Check that all items from **Before submitting** are resolved
- [ ] Make sure the title is self-explanatory and the description concisely explains the PR
- [ ] Add labels and milestones (and optionally projects) to the PR so it can be classified

</details>

<!--

Did you have fun?

Make sure you had fun coding 🙃

-->


Problem statement:
Add a performance warning to pl.Trainer parameter detect_anomaly.
### Description & Motivation

This detect_anomaly parameter greatly affects performance and is usually only used in debugging.  
Sometimes after debugging, user forget setting detect_anomaly to False, resulting in a performance loss during training.   
Hope add a warning to notice user that detect_anomaly flag will cause performance lose.  

In my experiment, the train speed when detect_anomaly is turned off is 2.5x when it is turned on.

### Pitch

Add a performance warning when detect_anomaly enabled.

### Alternatives

_No response_

### Additional context

_No response_

cc @borda @justusschock @awaelchli

Hints from issue discussion:
I am not opposed to this. Since the flag is only meant for debugging stability. 
@carmocca @justusschock any objection to this? 
I agree. @One-sixth Would you like to contribute it?
@carmocca Yes, I opened a PR.

Commit message: [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Patch to review:
diff --git a/src/lightning/pytorch/trainer/trainer.py b/src/lightning/pytorch/trainer/trainer.py
index c19598471c18b..823fea0562143 100644
--- a/src/lightning/pytorch/trainer/trainer.py
+++ b/src/lightning/pytorch/trainer/trainer.py
@@ -454,6 +454,10 @@ def __init__(
             GradClipAlgorithmType(gradient_clip_algorithm.lower()) if gradient_clip_algorithm is not None else None
         )
 
+        if detect_anomaly:
+            rank_zero_info(
+                "You have turned on detect_anomaly flag, which can significantly slow down compute speed and is recommended only for model debugging."
+            )
         self._detect_anomaly: bool = detect_anomaly
 
         setup._log_device_info(self)


Code review comments:
diff_hunk: @@ -454,6 +454,10 @@ def __init__(
             GradClipAlgorithmType(gradient_clip_algorithm.lower()) if gradient_clip_algorithm is not None else None
         )
 
+        if detect_anomaly:
+            rank_zero_info(
+                "You have turned on detect_anomaly flag, which can significantly slow down compute speed and is recommended only for model debugging."
path: src/lightning/pytorch/trainer/trainer.py
text: @user1:
```suggestion
                "You have turned on `Trainer(detect_anomaly=True)`. This will significantly slow down compute speed and is recommended only for model debugging."
```