Introduction to Hydra.cc: A Powerful Framework to Configure your Data Science Projects | by Khuyen Tran | Towards Data Science

It is fun to play with different feature engineering methods and machine learning models, but you will most likely need to adjust your feature engineering methods and tuning your machine learning models before getting a good result.
For example, in the speed dating data below, you might want to drop iid, id, idg, wave, career considering that they are not important features. But after doing more research about the data, you realize that career would be an important feature to predict whether two people would have the next date. So you decide not to dropcareer column.
If you are hard coding, which means to embed data directly into the source code of a script, like below
and your file is long, it might take a while for you to find the code that specifies which columns to drop. Wouldn’t it be great if you fix the columns from a simple text that solely contains information about the data without other python code like this instead?
This is when you need a configuration file.
A configuration file contains plain text parameters that define settings running a program. It is a good practice to avoid hard-coding in your python scripts while keeping all information related to the data such as which columns to drop, categorical variables in your config file.
This practice not only saves you from wasting time searching for a specific variable in your scripts but also make your scripts more reproducible.
For example, I could reuse this code for an entirely different data because there are no columns’ names specified in the code. To make the code work for the new data, all I need to fix is to change the columns’ names in my config file!
A common language of a config file is YAML. YAML is a human-friendly data serialization standard for all programming languages. The syntax is easy to read and almost similar to Python. Find out more about YAML syntax here.
I hope the short explanation above helps you somewhat understand the importance of a config file. But how do we go about accessing the parameters inside a config file?
There are some tools out there to read a config file such as PyYaml but my favorite one is Hydra.cc. Why? Because it allows me to:
Seamlessly change my default parameters in the terminal
Switch between different config groups
Automatically log the results
Let’s find out how to get started with hydra.cc and explore the benefits of using this powerful tool.
Install Hydra.cc with
pip install hydra-core --upgrade
Let’s start with a concrete example:
For example, if you have the config file like below with all the specific information about the data path, encoding, the kind of pipeline, and target column
All you need to do is to add the decorator @hydra.main(config_path='path/to/config.yaml') to the function that will use the config file. Make sure to add config inside the function to get access to the config file.
Now you are all set to use any parameters in your config file! If you want to get the name of the target,
target: match
all you need to do is to call config.target to get the string ‘match’!
Notice that you don’t need to put the quotation mark around the word ‘match’. The YAML file will consider it as a string if it is a word.
Hydra.cc allows you to override your default parameters inside the config file in the terminal. For example, if you want to switch the machine learning model from decision tree to logistic regression
you don’t need to rewrite the config file. You could instead type in the terminal the alternative parameters of your variables when running a file
python file.py model=logisticregression
And the model will be switched to logistic regression!
Better yet, if your config file is complex, Hydra.cc also allows you to access parameters in the file easier with the tab completion! You could find the details about tab completion here.
To keep your config files short and structured, you might want to create different files for different models along with their parameters such as this
You can specify the config file of the model you want to train on the command line
python file.py model=logistic
Now you can switch between different models and get access to their hyperparameters effortlessly!
Logging is important if you want to keep track of the results of your run. But many people don’t use Python logging because of the setup cost. Hydra.cc make it easy by automatically creating and saving all of your results in the folder ‘outputs’ based on the day
Each day folder is organized based on hours and minutes. You will see all the logs associated with your runs as well as the config files you use for that run!
If you happened to change your config file and don’t remember how the config file you used to produce a certain output looks like, you can look at the folder that day to find out!
Find more about logging here.
Since you are in the ‘output’s directory when running the function that is wrapped around by the hydra decorator, make sure to use utils.to_absolute_path('path/to/file') if you want to get access to other files in the parent directory.
Current working directory  : /Users/khuyentran/dev/hydra/outputs/2019-10-23/10-53-03Original working directory : /Users/khuyentran/dev/hydrato_absolute_path('foo')    : /Users/khuyentran/dev/hydra/footo_absolute_path('/foo')   : /foo
Congratulations! You have learned about why the configuration file is important and how to seamlessly configure your data science projects. I find it much more organized when I have all information that is related to the data in a separate file. I also find it easier to experiment with different parameters when all I need to do is to call python file.py variable=new_value. I hope you also gain the same benefits by incorporating both the configuration files and Hydra.cc in your data science practice.
Here is the example project that uses hydra.cc and config file.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these