3 Steps to Improve your Efficiency when Hypertuning ML Models
By: Khuyen Tran
Tags: Machine Learning, Command Line, Data Science, Gridsearchcv, Python

3 Steps to Improve your Efficiency when Hypertuning ML Models

Switch between Different Models Effortlessly without Messing with the Code

Photo by Markus Petritz on Unsplash

Motivation

You may hear about “no free lunch” (NFL) theorem, which indicates that there is no best algorithm for every data. One algorithm may perform well in one data but perform poorly in other data. That is why there are so many machine learning algorithms available to train data.

How do we know which machine learning model is the best? We cannot know until we experiment and compare the performance of different models. But experimenting with different models can be a mess, especially when you when to find the best parameters for your model with GridSearchCV.

For example, when we finished experimenting with RandomForestClassifier and switched to SVC, we might wish to save the parameters of RandomForestClassifier in case we want to reproduce the results we have with RandomForestClassifier. But how do we save these parameters efficiently?

Wouldn’t it be nice if we have the information about each model saved in different configuration files like below?

experiments/

├── data_preprocess.yaml

├── hyperparameters.yaml

└── model

├── random_forest.yaml

└── svc.yaml

Each file under model will specify their parameters like this

When we want to use a specific model (let’s say RandomForestClassifier), all we need to do is to run the training file and specify the model we want to train with model=modelname

python train.py model=random_forest

Being able to do this has helped me experiment with different models much faster without being afraid of losing the hyperparameters of a particular model used for GridSearchCV. This article will show you how to switch between different models effortlessly like above with Hydra.

1: Add Hydra

Hydra is a framework for elegantly configuring complex applications. I wrote how you could use hydra for your data science projects here. Besides configuring one configuration file, hydra also makes it much easier to work different configuration files.

To follow this tutorial, simply clone this repo. This is the tree structure of our project

.

├── data

├── experiments

│ ├── data_preprocess.yaml

│ ├── hyperparameters.yaml

│ └── model

│ ├── random_forest.yaml

│ └── svc.yaml

├── predict.py

├── preprocessing.py

├── train_pipeline.py

└── train.py

To access the config files in experiment , we will add hydra.main as the decorator of our function in train.py like below. The config_path specifies the path of the config files and config_name specifies the name of the config file

This is how hyperparameters.yaml looks like

YAML is the easiest language to understand and work with config files. As you can see, to access the training data, we simply use config.processed_data.text.train

In hyperparameters.yaml , we put general information related to training such as data path, scoring of GridSearchCV, but not the hyperparameters for a specific model. We want to keep this file static while changing the configuration for the model based on the model we use.

If we want to start training our model with SVC, in hyperparameters.yaml , we set the default model with

defaults:

- model: svc

Step 2: Configure Machine Learning Models

Now when we run python train.py , Hydra will try to search for the file svc.yaml under the directory model

Thus, our next step is to create a file with the name svc.yaml (or any other name that you would like) under the model directory like this

experiments/

├── data_preprocess.yaml

├── hyperparameters.yaml

└── model

├── random_forest.yaml

└── svc.yaml

Our svc.yaml file will contain the name of the model and the hyperparameters for GridSearchCV to search over

Now when we run

python train.py

Hydra will automatically access the svc.yaml config within the model directory and use the parameters in svc.yaml !

If you want to instead use RandomForestClassifier, create the file named random_forest.yaml then insert the information about our RandomForestClassifier

Instead of changing the default model in the hyperparameter.yaml file, we can override the default model in the terminal!

python train.py model=random_forest

Now our function in train.py can get access to these parameters via config. For example, if I use svc model, this will be what I see

>>> print(config.model)

SVC >>> print(hyperparameters))

{ classifier__C: [.05, .12] classifier__kernel: ['linear', 'poly'] classifier__gamma: [0.1, 1] classifier__degree: [0, 1, 2]

}

Pretty cool!

Last Step: Config File into the Parameters for GridSearchCV

To use turn the string ‘SVC’ into a class, use eval

from sklearn.svm import SVC classifier = eval(config.model)()

Now classifier works like a normal SVC class!

To use the hyperparameters in our config file into a Python dictionary whose values are Python list, we use

Now you can freely switch between different models in the terminal like this

python train.py model=random_forest

From now on, if you want to use a new model with different hyperparameters, all you need to do is to add the config file for that model then run

python train.py model=<newmodel>

Your model will run without the need of changing the code in train.py!

But what if I want to use Python function in the Config file?

Sometimes, instead of writing down some specific values that are recognized by YAML file such as a list classifier__C: [.05, .12] , we might want to use the function that is not recognized by YAML such as classifier__C: np.logspace(-4, 4, 20)

model: LogisticRegression hyperparameters: classifier__penalty: ['l1', 'l2'] classifier__C: np.logspace(-4, 4, 20) classifier__solver: ['qn', 'lbfgs', 'owl']

Don’t worry! We can still find a way to work around that situation by wrapping eval() function around the string! This would turn your string np.logspace(-4, 4, 20) into a real python function!

for key, value in param_grid.items(): if isinstance(value, str): param_grid[key] = eval(param_grid[key])

Then turn all of your value in the dictionary into Python lists so they would be valid parameters for GridSearchCV

param_grid = {ele: (list(param_grid[ele])) for ele in param_grid}

Conclusion

Congratulation! You have just learned how to use Hydra to experiment with different machine learning models. With this tool, you can keep your code separately from the specific information of your data.

You don’t need to go back to your code if you want to change the hyperparameters of your machine learning model. You just need to add one more config file for an additional model and switch between them with

python train.py model=<modelname>

Here is the example project that uses hydra.cc and config file.

I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.

Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these