Repository: ourownstory/neural_prophet

Issue:
Automatic Future Regressor Removal with Global Model
**Describe the bug**

When training a global model (multiple series identified by the ID column) with future regressors, NeuralProphet automatically removes the specified future regressor if it's constant on one of those series. However, while it might be constant / not useful for one particular series (ID group), it might carry relevant for other series. For example, when forecasting electrical consumption, we can include estimated PV generation - which will be 0 for houses that don't have a PV installation, and different from that for houses that do have it.

While browsing through the code, I have narrowed the issue to this line of code (that removes a regressor if it's constant only across one of the series):
https://github.com/ourownstory/neural_prophet/blob/main/neuralprophet/df_utils.py#L449

For some reason, this regressor removal is only activated if a `validation_df` is supplied during the model fitting (and not otherwise).

**To Reproduce**

Steps to reproduce the behavior:
```
from neuralprophet import NeuralProphet
import pandas as pd
import datetime

n_samples = 100
start = datetime.datetime(2023, 1, 1)
one_day = datetime.timedelta(days=1)
df = pd.DataFrame({
    'y': [float(x) for x in range(n_samples)]*2,
    'fut_reg_1':  [float(x)**2 for x in range(n_samples)] * 2,
    'fut_reg_2':  [float(x)**0.5 for x in range(n_samples)] * 2,
    'ID': ['a']*n_samples + ['b']*n_samples,
    'ds': [start + x*one_day for x in range(n_samples)] * 2
})

df.loc[df.ID == 'b', 'fut_reg_2'] = 0

model = NeuralProphet(
    growth="off",
    daily_seasonality=False,
    yearly_seasonality=False,
    weekly_seasonality=5,
    n_lags=4,
    ar_reg=1,
    learning_rate=0.01,
)

df_train, df_val = model.split_df(df, valid_p=0.2)
    
model.add_future_regressor(name='fut_reg_1')
model.add_future_regressor(name='fut_reg_2')

print(model.config_regressors)

metrics = model.fit(
    df=df_train, 
    validation_df=df_val,
    freq='1D',    
)
```

**Expected behavior**

When training a global model, NeuralProphet should only remove regressors if they are constant across all series, and not if they are only constant across some of the series.

**What actually happens**

The future regressor gets automatically removed, which, in return, raises an exception (`ValueError: Unexpected column fut_reg_2 in data`).

**Screenshots**

```
INFO - (NP.df_utils._infer_frequency) - Major frequency D corresponds to 99.0% of the data.
INFO - (NP.df_utils._infer_frequency) - Dataframe freq automatically defined as D
INFO - (NP.df_utils._infer_frequency) - Major frequency D corresponds to 99.0% of the data.
INFO - (NP.df_utils._infer_frequency) - Dataframe freq automatically defined as D
WARNING - (NP.forecaster.fit) - When Global modeling with local normalization, metrics are displayed in normalized scale.
WARNING - (NP.df_utils.check_single_dataframe) - Encountered future regressor with only unique values in training set. Automatically removed variable.
WARNING - (NP.forecaster._check_dataframe) - Removing regressor fut_reg_2 because it is not present in the data.
INFO - (NP.df_utils._infer_frequency) - Major frequency D corresponds to 98.765% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - 1D
INFO - (NP.df_utils._infer_frequency) - Major frequency D corresponds to 98.765% of the data.
INFO - (NP.df_utils._infer_frequency) - Defined frequency is equal to major frequency - 1D
INFO - (NP.config.set_auto_batch_epoch) - Auto-set batch_size to 16
INFO - (NP.config.set_auto_batch_epoch) - Auto-set epochs to 419
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [30], in <cell line: 38>()
     34 model.add_future_regressor(name='fut_reg_2')
     36 print(model.config_regressors)
---> 38 metrics = model.fit(
     39     df=df_train, 
     40     validation_df=df_val,
     41     freq='1D',    
     42 )

File C:\Anaconda3\envs\dev_3_10\lib\site-packages\neuralprophet\forecaster.py:821, in NeuralProphet.fit(self, df, freq, validation_df, epochs, batch_size, learning_rate, early_stopping, minimal, metrics, progress, checkpointing, continue_training, num_workers)
    819     df_val = self._check_dataframe(df_val, check_y=False, exogenous=False)
    820     df_val = self._handle_missing_data(df_val, freq=self.data_freq)
--> 821     metrics_df = self._train(
    822         df,
    823         df_val=df_val,
    824         progress_bar_enabled=bool(progress),
    825         metrics_enabled=bool(self.metrics),
    826         checkpointing_enabled=checkpointing,
    827         continue_training=continue_training,
    828         num_workers=num_workers,
    829     )
    831 # Show training plot
    832 if progress == "plot":

File C:\Anaconda3\envs\dev_3_10\lib\site-packages\neuralprophet\forecaster.py:2609, in NeuralProphet._train(self, df, df_val, progress_bar_enabled, metrics_enabled, checkpointing_enabled, continue_training, num_workers)
   2607 if df_val is not None:
   2608     df_val, _, _, _ = df_utils.prep_or_copy_df(df_val)
-> 2609     val_loader = self._init_val_loader(df_val)
   2611 # Init the model, if not continue from checkpoint
   2612 if continue_training:

File C:\Anaconda3\envs\dev_3_10\lib\site-packages\neuralprophet\forecaster.py:2562, in NeuralProphet._init_val_loader(self, df)
   2550 """Executes data preparation steps and initiates evaluation procedure.
   2551 
   2552 Parameters
   (...)
   2559     torch DataLoader
   2560 """
   2561 df, _, _, _ = df_utils.prep_or_copy_df(df)
-> 2562 df = self._normalize(df)
   2563 dataset = self._create_dataset(df, predict_mode=False)
   2564 loader = DataLoader(dataset, batch_size=min(1024, len(dataset)), shuffle=False, drop_last=False)

File C:\Anaconda3\envs\dev_3_10\lib\site-packages\neuralprophet\forecaster.py:2496, in NeuralProphet._normalize(self, df)
   2494 data_params = self.config_normalization.get_data_params(df_name)
   2495 df_i.drop("ID", axis=1, inplace=True)
-> 2496 df_aux = df_utils.normalize(df_i, data_params).copy(deep=True)
   2497 df_aux["ID"] = df_name
   2498 df_norm = pd.concat((df_norm, df_aux), ignore_index=True)

File C:\Anaconda3\envs\dev_3_10\lib\site-packages\neuralprophet\df_utils.py:393, in normalize(df, data_params)
    391 for name in df.columns:
    392     if name not in data_params.keys():
--> 393         raise ValueError(f"Unexpected column {name} in data")
    394     new_name = name
    395     if name == "ds":

ValueError: Unexpected column fut_reg_2 in data
```

**Environement (please complete the following information):**

 - Python environment [Python 3.10, in standalone anaconda env with no other packages]
 - NeuralProphet version and install method [0.5.1, installed from PYPI with `pip install neuralprophet==0.5.1`]


Discussion:
@milannnnn This is a valid point. Thank you for raising it!

@judussoari  We should indeed only remove a variable if it is single-valued across all series in a panel dataset (unless the comonent were to be set to be local). 
We may also want to de-escalate the Error to a Warning if unused columns are passed. 

Resolving patch:
diff --git a/neuralprophet/df_utils.py b/neuralprophet/df_utils.py
--- a/neuralprophet/df_utils.py
+++ b/neuralprophet/df_utils.py
@@ -142,6 +142,7 @@ def data_params_definition(
     config_regressors=None,
     config_events: Optional[ConfigEvents] = None,
     config_seasonality: Optional[ConfigSeasonality] = None,
+    local_run_despite_global: Optional[bool] = None,
 ):
     """
     Initialize data scaling values.
@@ -215,9 +216,13 @@ def data_params_definition(
         for reg in config_regressors.keys():
             if reg not in df.columns:
                 raise ValueError(f"Regressor {reg} not found in DataFrame.")
+            norm_type = config_regressors[reg].normalize
+            if local_run_despite_global:
+                if len(df[reg].unique()) < 2:
+                    norm_type = "soft"
             data_params[reg] = get_normalization_params(
                 array=df[reg].values,
-                norm_type=config_regressors[reg].normalize,
+                norm_type=norm_type,
             )
     if config_events is not None:
         for event in config_events.keys():
@@ -310,10 +315,17 @@ def init_data_params(
         )
     # Compute individual  data params
     local_data_params = OrderedDict()
+    local_run_despite_global = True if global_normalization else None
     for df_name, df_i in df.groupby("ID"):
         df_i.drop("ID", axis=1, inplace=True)
         local_data_params[df_name] = data_params_definition(
-            df_i, normalize, config_lagged_regressors, config_regressors, config_events, config_seasonality
+            df_i,
+            normalize,
+            config_lagged_regressors,
+            config_regressors,
+            config_events,
+            config_seasonality,
+            local_run_despite_global,
         )
         if global_time_normalization:
             # Overwrite local time normalization data_params with global values (pointer)
@@ -439,14 +451,13 @@ def check_single_dataframe(df, check_y, covariates, regressors, events, seasonal
         raise ValueError("Column ds has timezone specified, which is not supported. Remove timezone.")
     if len(df.ds.unique()) != len(df.ds):
         raise ValueError("Column ds has duplicate values. Please remove duplicates.")
-    regressors_to_remove = []
     if regressors is not None:
         for reg in regressors:
             if len(df[reg].unique()) < 2:
                 log.warning(
-                    "Encountered future regressor with only unique values in training set. Automatically removed variable."
+                    "Encountered future regressor with only unique values in training set. "
+                    "Variable will be removed for global modeling if this is true for all time series."
                 )
-                regressors_to_remove.append(reg)
 
     columns = []
     if check_y:
@@ -489,7 +500,7 @@ def check_single_dataframe(df, check_y, covariates, regressors, events, seasonal
         df.index.name = None
     df = df.sort_values("ds")
     df = df.reset_index(drop=True)
-    return df, regressors_to_remove
+    return df
 
 
 def check_dataframe(
@@ -521,14 +532,20 @@ def check_dataframe(
     """
     df, _, _, _ = prep_or_copy_df(df)
     checked_df = pd.DataFrame()
-    regressors_to_remove = []
     for df_name, df_i in df.groupby("ID"):
-        df_aux, reg = check_single_dataframe(df_i, check_y, covariates, regressors, events, seasonalities)
+        df_aux = check_single_dataframe(df_i, check_y, covariates, regressors, events, seasonalities)
         df_aux = df_aux.copy(deep=True)
-        if len(reg) > 0:
-            regressors_to_remove.append(*reg)
         df_aux["ID"] = df_name
         checked_df = pd.concat((checked_df, df_aux), ignore_index=True)
+    regressors_to_remove = []
+    if regressors is not None:
+        for reg in regressors:
+            if len(df[reg].unique()) < 2:
+                log.warning(
+                    "Encountered future regressor with only unique values in training set across all IDs."
+                    "Automatically removed variable."
+                )
+                regressors_to_remove.append(reg)
     if len(regressors_to_remove) > 0:
         regressors_to_remove = list(set(regressors_to_remove))
         checked_df = checked_df.drop(*regressors_to_remove, axis=1)


Test patch: