Useful scikit-learn tips-2
By: Bala Priya C
Tags: Machine Learning, Scikit Learn, Python, Data, Data Preprocessing

Photo by Bonnie Kittle on Unsplash

This blog post covers yet another set of tips from Kevin Markham’s useful scikit-learn tips series; Have you ever felt confused in understanding the fit and transform methods? Has there been an occasion of applying fit_transform on the test data? Are you familiar with the advantages of using sklearn’s pre-processing pipeline even though pandas is your favourite choice for data wrangling? Need a refresher on encoding categorical variables? Read ahead to find out more…

Understanding fit, transform, and fit_transform methods

The “fit” method is used when the transformer learns something about the data; the “transform” method is used to apply the learned transformation to the data. The following are some examples, explaining what the “fit” and “transform” methods do in each case:

Using fit_transform

Instead of calling the fit and transform methods separately, we might as well call the fit_transform method; but we should be sure to use fit_transform only on the training data and use only the transform method on the test data. If we use fit_transform on the test data, then we are essentially asking our model to learn from the test data as well which is not allowed leading to the subtle problem of ‘data leakage’.

Advantages of preferring sci-kit-learn over pandas for preprocessing

Can cross-validate the entire workflow: If we want to call cross_val_score on the entire workflow, then it’s a very good practice to use sklearn preprocessing pipeline. In this case, the preprocessing is applied separately on each fold of the data rather than on the whole of the data at once, which results in a more accurate model.

If we want to call on the entire workflow, then it’s a very good practice to use sklearn preprocessing pipeline. In this case, the preprocessing is applied separately on each fold of the data rather than on the whole of the data at once, which results in a more accurate model. Can grid search model and preprocessing hyperparameters: We generally use a grid search to find out optimal hyperparameters for our model, however, we might as well do a grid search on the preprocessing hyperparameters such as the best strategy for imputation, normalization, etc.

We generally use a grid search to find out optimal hyperparameters for our model, however, we might as well do a grid search on the preprocessing hyperparameters such as the best strategy for imputation, normalization, etc. Avoids adding new columns to the source DataFrame: We often need to add new columns and apply a transformation to certain columns; Using sklearn’s ColumnTransformer is a much better option than adding additional columns to the source DataFrame which would make the DataFrame difficult to explore.

We often need to add new columns and apply a transformation to certain columns; Using sklearn’s ColumnTransformer is a much better option than adding additional columns to the source DataFrame which would make the DataFrame difficult to explore. pandas lack separate fit/transform steps to prevent data leakage: Data Leakage if left unnoticed can impact model performance severely; it’s always one of the best practice to use sklearn as the fit/transform methods provide clear distinction as to which method to apply on the training and test data obtained from the train_test_split.

Encoding Categorical Features- OneHotEncoder and OrdinalEncoder