One Potential Cause of Overfitting That I Never Noticed Before | by Yufeng | Towards Data Science

“Why is my tuned model still overfitting?”
“Did you use cross-validation?”
“Of course yes.”
“What model did you use and what are the hyperparameters you tuned?”
“Random Forest regressor and I tuned tree number and maximum feature numbers per tree.”
“I need to check your code.”
This was the dialogue between my wife and me when she was doing a mini-project.
Honestly, based on her description, I couldn’t think of anywhere wrong in the procedure. However, after my checking her code line by line, I found the issue that caused the overfitting problem and I have never thought of before.
Let’s dissect her code together.
This is the block of code that tuned the model in the training dataset and it is also where the problem happened.
def train_pipeline_rf(X,y):    # X are factors    # y is output    # impute missing X by median    X_prepared = pre_pipeline.fit_transform(X)    # set cross-validation    tscv = TimeSeriesSplit(n_splits=10)    data_split = tscv.split(X_prepared)    # hyper-parameter space    param_grid_RF = {        'n_estimators' : [10,20,50,100,200,500,1000],        'max_features' : [0.6,0.8,"auto","sqrt"]    }    # build random forest model    rf_model = RandomForestRegressor(random_state=42,n_jobs=-1)    # gridsearch for the best hyper-parameter    gs_rf = GridSearchCV(rf_model, param_grid=param_grid_RF, cv=data_split, scoring='neg_mean_squared_error', n_jobs=-1)    # fit dataset    gs_rf.fit(X_prepared, y)    return gs_rf
The pre_pipeline in the code is a pipeline for missing value imputation and feature scaling, both of which are essential steps in data pre-processing. Here is what it looks like:
pre_pipeline = Pipeline([        ('imputer', SimpleImputer(strategy="median")),        ('std_scaler', StandardScaler()),    ])
The imputer she used here is to replace the NA values by the column median, and the scaler is the standard scaler which normalizes the columns as:
z = (x — u) / s
where u is the mean and s is the standard deviation of the column [1]. Some other imputers and scalers in the sklearn package can also be used. But this part is not related to the overfitting problem.
    # set cross-validation    tscv = TimeSeriesSplit(n_splits=10)    data_split = tscv.split(X_prepared)
Here she used a method specifically designed for time-series data, TimeSeriesSplit.
Usually, people use k-fold cross-validation to randomly split the training data or stratified k-fold cross-validation to preserve the percentage of samples for each class [2]. But these two methods are not suitable for time-series data because they don’t keep the raw order of the data points.
This step is pretty standard and cannot be related to the overfitting problem.
    # hyper-parameter space    param_grid_RF = {        'n_estimators' : [10,20,50,100,200,500,1000],        'max_features' : [0.6,0.8,"auto","sqrt"]    }    # build random forest model    rf_model = RandomForestRegressor(random_state=42,n_jobs=-1)
As for hyperparameter tuning, a set of candidates should be given, which are usually defined as a dictionary format in sklearn package. The names of the hyperparameters are from the model to build, for example, RandomForestRegressor in my wife’s code.
The so-called “tuning-hyperparameter” step is to select the best hyperparameter combination from your given candidates that outperforms other combinations of hyperparameters in the cross-validation procedure.
It is the part that resulted in the overfitting problem.
If we refer to the manual page of Random Forest Regressor in sklearn, we can see a long list of parameters in the function RandomForestRegressor (not listed here, please refer to the webpage if interested).
However, in the code above, only n_estimators and max_features were passed to the function during the tuning process, which resulted in that all the other parameters took the default values.
One of the parameters in the function is called max_depth, which is the maximum depth of the tree in your random forest model. The default value of it is “None”, which means that the decision tree will go as deep as that each leave is pure or all leaves have at most m samples, where m is defined by min_samples_split (default = 2).
This setting significantly increased the complexity of the trained model. And based on the tradeoff between bias and variance, the model trained in the code above had low bias and high variance. Of course, it finally led to the overfitting problem.
I solved the problem by adding max_depth to the hyperparameter space.
    # hyper-parameter space    param_grid_RF = {        'n_estimators' : [10,20,50,100,200,500,1000],        'max_features' : [0.6,0.8,"auto","sqrt"],        'max_depth' : [4,5,6]    }
Actually, there are some other parameters that can affect the complexity of the model in RandomForestRegressor, but it is not recommended to put all of them into the hyperparameter space.
The extra hyperparameter in the code above already increased the computation cost by three folds. So, it’s not worth increasing the running time exponentially to just tune more parameters.
Instead, you can set some of them in the model initiation like this.
# hyper-parameter space    param_grid_RF = {        'n_estimators' : [10,20,50,100,200,500,1000]    }# build random forest model    rf_model = RandomForestRegressor(random_state=42,n_jobs=-1,max_features=0.6,max_depth=5)
In the code above, the only tuned hyperparameter is n_estimators, and max_features, as well as max_depth, are fixed during the training process. The tuned or fixed parameters do not have to be like that.
The rationale of the setting is to reduce the computational cost when you already have some idea on the selection of some hyperparameters.
Yes, my wife’s problem has been solved. To be honest, however, I could not have found it until I went over all the default parameters in the package. That’s why I want to share this experience with you so that you can be more careful about the default parameters in your machine learning model training processes.
Overfitting is one of the most common problems in data science, which mostly comes from the high complexity of the model and the lack of data points.
To avoid it, it’s better to take full control of the packages you use.
Hope this piece of advice can help you.
https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScalerhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.htmlhttps://en.wikipedia.org/wiki/Random_forest
https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedKFold.html
https://en.wikipedia.org/wiki/Random_forest
I would like to thank Antonio Carlos for pointing out this issue in my original post. He also suggested a nice post “Pre-Process Data with Pipeline to Prevent Data Leakage during Cross-Validation.” to the topic. I really appreciate that.
It is related to the pre-processing part of the pipeline. It shouldn’t have been done on the entire training dataset, but on each iteration of the Cross-Validation step because the overall normalization of the training data will cause leakage between the train and validation dataset in the CV step.
So, I put the corrected code with a corresponding revision as below:
def train_pipeline_rf(X,y):    # X are factors    # y is output    # set cross-validation    tscv = TimeSeriesSplit(n_splits=10)    data_split = tscv.split(X)        # build a pipeline of pre-processing and random forest model    my_pipe = Pipeline([        ('imputer', SimpleImputer(strategy="median")),         ('std_scaler', StandardScaler()),        ('rf_model', RandomForestRegressor(random_state=42,n_jobs=-1))    ])    # hyper-parameter space    param_grid_RF = {        'rf_model__n_estimators' : [10,20,50,100,200,500,1000],        'rf_model__max_features' : [0.6,0.8,"auto","sqrt"],        'rf_model__max_depth' : [4,5,6]    }    # gridsearch for the best hyper-parameter within the pipeline.    gs_rf = GridSearchCV(my_pipe, param_grid=param_grid_RF, cv=data_split, scoring='neg_mean_squared_error', n_jobs=-1)    # fit dataset    gs_rf.fit(X, y)    return gs_rf