Repository: ourownstory/neural_prophet

Issue:
Make Lightning trainer .test() verbosity configurable
**Prerequisites**

* [X] Put an X between the brackets on this line if you have done all of the following:
    * Checked the Answered Questions on the Github Disscussion board: https://github.com/ourownstory/neural_prophet/discussions
      If you have the same question but the Answer does not solve your issue, please continue the conversation there.
    * Checked that your issue isn't already filed: https://github.com/ourownstory/neural_prophet/issues
      If you have the same issue but there is a twist to your situation, please add an explanation there.
    * Considered whether your issue might need further discussing before being defined as a feature request:
      Please [post an idea or feedback](https://github.com/ourownstory/neural_prophet/discussions/categories/ideas-feedback)

**Is your feature request related to a problem? Please describe.**

When doing cross-validation, my output gets flooded with the Lightning .test() output, which is frustrating. It drowns out the per-trial results, which is what I'm actually interested in, and lengthens the output by 20x.

**Describe the solution you'd like**

Have some sort of verbosity parameter to `.test()` which then gets passed into the `verbosity={value}` parameter of `self.trainer.test`.

**Describe alternatives you've considered**

There seems to be no good alternative -- Lightning doesn't use a logger, it just `print()`s the output based on the verbose flag. What I've ended up doing in the meantime is disabling stdout altogether when running `.test()`, then re-enabling it before printing out the trial results.

**Additional context**

N/A



Discussion:
Hi @w-biggs, thanks for raising this. Most of our core-developers are on summer break right now, so I'll try to help. Unfortunately, I don't get the problem. Could you elaborate on it?
When cross-validating, every time a model is tested, Lightning outputs something like:

```────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       Test metric             DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
        Loss_test          0.0026712114922702312
         MAE_val            18.709611892700195
        RMSE_val             22.74985122680664
      RegLoss_test                  0.0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
```

If I'm doing 100 tuning trials with 5 different cross-validation splits, that means this output gets printed 500 times. If I'm working in a notebook, this becomes super unwieldy, and it drowns out the info I'm actually interested in. Lightning's `.test()` function, which is what is outputting this, has a verbosity parameter that can be used to disable this output. It would be nice if NeuralProphet's `.test()` function could also take in a verbosity parameter, which would then be passed to Lightning's `.test()` function when it's called.
The easiest solution to me is to surface the `verbose` argument in the `.test()` API. 
Another verbosity issue I faced is on `fit` method. So if `learning_rate = 'auto'`, then there's no way one can disable the learning rate finder's progress bar, even when `minimal = True` or `progress=False` as those only control the main fitting progress bar.

Resolving patch:
diff --git a/neuralprophet/forecaster.py b/neuralprophet/forecaster.py
--- a/neuralprophet/forecaster.py
+++ b/neuralprophet/forecaster.py
@@ -1101,13 +1101,15 @@ def predict(self, df: pd.DataFrame, decompose: bool = True, raw: bool = False):
         self.predict_steps = self.n_forecasts
         return df
 
-    def test(self, df: pd.DataFrame):
+    def test(self, df: pd.DataFrame, verbose: bool = True):
         """Evaluate model on holdout data.
 
         Parameters
         ----------
             df : pd.DataFrame
                 dataframe containing column ``ds``, ``y``, and optionally``ID`` with with holdout data
+            verbose : bool
+                If True, prints the test results.
         Returns
         -------
             pd.DataFrame
@@ -1132,7 +1134,7 @@ def test(self, df: pd.DataFrame):
         )
         loader = self._init_val_loader(df)
         # Use Lightning to calculate metrics
-        val_metrics = self.trainer.test(self.model, dataloaders=loader)
+        val_metrics = self.trainer.test(self.model, dataloaders=loader, verbose=verbose)
         val_metrics_df = pd.DataFrame(val_metrics)
         # TODO Check whether supported by Lightning
         if not self.config_normalization.global_normalization:


Test patch: