A Succinct Scikit-learn Solution for Kaggle House Prices Prediction Challenge
By: Jiahui Wang
Tags: Gridsearchcv, Crossvalidation, Scikit Learn, Regression, Pipeline

1. Overview of the Process Flow

The source code is split into two parts: Exploratory Data Analysis (EDA) and Scikit-learn modeling, and the code can be run on Google Colab. The EDA file loads the raw training and testing data from Kaggle. After the exploratory data analysis and data transformation, the transformed data is exported by using pickle. Then, also by using pickle, the second Scikit-learn modeling file loads the transformed data.

For the EDA part, the detailed code and data transformation process can be found in the source code.

Process flow of EDA and Scikit-learn hyperparameter tuning

2. Scikit-learn Hyperparameter Tuning

Cross-validation

When modeling this house price dataset, my focus was to implement Scikit-learn cross-validation procedure. Since Scikit-learn only provides high-level estimators to the users, we cannot modify the implementation of these estimators. Thus, the only tunable elements left for the users are the estimator hyperparameters. Tuning hyperparameters requires splitting the training data into training and validation data. In addition to the simple train_test_split method, Scikit-learn also provides cross-validation mechanism to avoid wasting training data.

Pipeline

However, if not used properly, cross-validation may suffer from data leakage, which means the validation data statistics may get leaked to the training data. To avoid data leakage, Scikit-learn introduces pipeline , which enables chaining multiple preprocessors together. Chaining the preprocessing estimators together with the final classifier/regressor estimator ensures that the training data remain independent from the validation data. Meanwhile, the pipeline can be easily applied to the testing data.

ColumnTransformer

To apply different preprocessing estimators to the numeric and categorical columns, ColumnTransformer is used. After separately defining two pipelines and specifying the numeric and categorical columns, ColumnTransformer is then chained to a regressor estimator for generating final output.

GridSearchCV

To jointly tune the hyperparameters of the pipeline, gridsearch parameters are defined. By calling GridSearchCV , the optimized hyperparameter combination can be found.

Here, the process of how to tune the ElasticNet estimator hyperparameters is shown. Similarly, other regressor estimators, such as Ridge, Lasso, gradient boosting, can be optimized.

# define numeric column preprocessing pipeline

numeric_features = numericColumns

numeric_transformer = Pipeline(steps=[

('imputer', SimpleImputer(strategy='mean')),

('scaler', StandardScaler())]) # define categorical column preprocessing pipeline

categorical_features = categoricalColumns

categorical_transformer = Pipeline(steps=[

('imputer', SimpleImputer(strategy='constant', fill_value='missing')),

('onehot', OneHotEncoder(handle_unknown='ignore'))]) # define ColumnTransformer preprocessing pipeline

preprocessor = ColumnTransformer(transformers=[

('num', numeric_transformer, numeric_features),

('cat', categorical_transformer, categorical_features)]) # add regressor estimator to the pipeline

ElasticNet_pipeline = Pipeline(steps=[

('preprocessor', preprocessor),

('regressor', ElasticNet())]) # define GridSearch parameters

ElasticNet_param_grid = {

'regressor__alpha': [0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5, 10,100],

'regressor__l1_ratio':[0.5,0.6,0.7,0.8,0.9]} # 'neg_mean_squared_error', metrics.mean_squared_error is used here

scoring_metrics = make_scorer(mean_squared_error, greater_is_better=False) ElasticNet_grid_search = GridSearchCV(ElasticNet_pipeline, ElasticNet_param_grid, scoring = scoring_metrics, cv = 5) ElasticNet_grid_search.fit(df_train[numericColumns+categoricalColumns], df_train['SalePrice']) print(f"best score is {ElasticNet_grid_search.best_score_}, best parameter from grid search is{ElasticNet_grid_search.best_params_}")

Conclusion

This post focuses on the implementation of Scikit-learn key concepts of cross-validation , pipeline , ColumnTransformer , and GridSearchCV . Don’t forget to check out the source code on Github, to find out the code, intermediate visualization output, and the detailed code annotation.

Meanwhile, I have also modeled the same Kaggle House Prices Prediction dataset using TensorFlow 2.0. If you are interested in the differences between Scikit-learn and TensorFlow 2.0, please stay tuned! I will be updated the post on how I model using TensorFlow 2.0 soon!