Configure XGBoost Models with YAML | by Cole Brendel | Towards Data Science

How to use YAML files to make your models rapidly sharable, wickedly flexible, and to prepare them for production
There is nothing more enjoyable than starting work on a new ML project. First, you get your hands dirty with some data cleaning, then you get to load that squeaky-clean data into a pandas df to start some exploratory data analysis (where you inevitably discover you need to go back and do more data cleaning...) Next, you are onto feature engineering and then finally, you are at the stage where you are ready to build some models and experiment!
Throughout this whole process, you’ve likely put all of your code together in your trusty jupyter notebook. Now you’d like to actually put your model to use, but how do you even begin? You certainly can’t put a notebook into production.
One could write a book on the productionization of machine learning models. It is not my intent to teach you all of the engineering steps required for production. Instead, think of this article as a productionization kaizen: one small change you can make to your code to prepare for production.
In the article, I will show you how to use YAML configuration files to make your code cleaner, more scalable, easier to maintain, and more efficient at running experiments.
YAML is a recursively named language for storing information in a human readable format [1].
A detailed discussion on the structure of yaml would take us too far afield, so being sufficient for our purposes, let’s consider the following yaml block for initializing an XGBoost Classifier. This block defines a dictionary “model_settings”, taking another dictionary as its value the parameters dictionary. The parameters dictionary holds the values for each of the parameters of the xgboost model that we would like to set.
# basic yaml configmodel_settings:  random_state: 90210  objective: 'binary:logistic'  eval_metric: ['error', 'map', 'auc', 'aucpr']  n_jobs: -1  learning_rate: 0.01  colsample_bytree: 0.8  subsample: 0.75  reg_alpha: 2.5  reg_lambda: 1.25  max_depth: 7  gamma: 0.2  n_estimators: 1251  max_delta_step: 8.5  scale_pos_weight: 2.0
By writing the parameters of your model in a yaml config, you derive 4 primary benefits:
The parameters are no longer hard-coded in the initialization of the model making code cleanerCollaborators can easily implement your model by passing a single file (preferably in a git repo)Making changes to your model no longer requires code refactoringFollowing (3), you can easily conduct “what-if” experiments on your model. E.g., What if I changed max-depth to go one level deeper?
The parameters are no longer hard-coded in the initialization of the model making code cleaner
Collaborators can easily implement your model by passing a single file (preferably in a git repo)
Making changes to your model no longer requires code refactoring
Following (3), you can easily conduct “what-if” experiments on your model. E.g., What if I changed max-depth to go one level deeper?
We want to be able to pass the information contained in the yaml file to other scripts in your project. To this end, we define a super-class for your project holding all sub-classes (think of them as modules) which will hold the values in your yaml. By doing this, we can enforce types so that a user can’t pass strings where ints are expected. Let’s define one such object.
class YamlDemo:    class ModelSettings:        def __init__(self, random_state: int, objective: str, eval_metric: list, n_jobs: int,                      learning_rate: float, colsample_bytree: float, subsample: float,                      reg_alpha: float, reg_lambda: float, max_depth: int, gamma: float,                      n_estimators: int, max_delta_step: float, scale_pos_weight: float):            self.random_state = random_state            self.objective = objective            self.eval_metric = eval_metric            self.n_jobs = n_jobs            self.learning_rate = learning_rate            self.colsample_bytree = colsample_bytree            self.subsample = subsample            self.reg_alpha = reg_alpha            self.reg_lambda = reg_lambda            self.max_depth = max_depth            self.gamma = gamma            self.n_estimators = n_estimators            self.max_delta_step = max_delta_step            self.scale_pos_weight = scale_pos_weight                def __init__(self, model_settings: ModelSettings, config_path: str):        self.model_settings = model_settings        self.config_path = config_path
With your yaml config and config object in place, the last component is the authoring of a parser to read your yaml and populate your config object with the values. A simple config parser function like the following can do this for you.
import yamldef parse_config(config_path: str) -> YamlDemo:    """    parses yaml config; returns populated YamlDemo object    """    with open(config_path, 'r') as stream:            try:                config_settings = yaml.safe_load(stream)            except yaml.YAMLError as e:                print(e)model_settings = YamlDemo.ModelSettings(            random_state = config_settings["model_settings"]["random_state"],            objective = config_settings["model_settings"]["objective"],            eval_metric = config_settings["model_settings"]["eval_metric"],            n_jobs = config_settings["model_settings"]["n_jobs"],            learning_rate = config_settings["model_settings"]["learning_rate"],            colsample_bytree = config_settings["model_settings"]["colsample_bytree"],            subsample = config_settings["model_settings"]["subsample"],            reg_alpha = config_settings["model_settings"]["reg_alpha"],            reg_lambda = config_settings["model_settings"]["reg_lambda"],            max_depth = config_settings["model_settings"]["max_depth"],            gamma = config_settings["model_settings"]["gamma"],            n_estimators = config_settings["model_settings"]["n_estimators"],            max_delta_step = config_settings["model_settings"]["max_delta_step"],            scale_pos_weight = config_settings["model_settings"]["scale_pos_weight"]        )    config = YamlDemo(model_settings=model_settings, config_path=config_path)    assert isinstance(config, YamlDemo)    return config
Now, the last step in the process is to pass the information in your config object to your model.
First, we must parse our config to obtain a populated object:
config_settings = parse_config('/Users/cole.brendel/Desktop/demo.yaml')
Next, let’s get an example data set to see the model “in action”:
from sklearn.datasets import load_irisX,y = load_iris(return_X_y=True)# notice how I set the value for `random_state` from the config!from sklearn.model_selection import train_test_splitX_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.33, random_state=config_settings.model_settings.random_state)
Lastly, we’ll pass this config to a function which trains the model.
# fits a model on the training set/evals on test setimport pandas as pdfrom pandas import DataFrameimport xgboostfrom xgboost import XGBClassifierfrom sklearn.metrics import classification_reportdef fit_xgb_model(config_settings: YamlDemo, X_train: DataFrame, y_train: DataFrame, X_test: DataFrame, y_test: DataFrame) -> xgboost.XGBClassifier:    """    fits an XGB model and returns model obejct     """        eval_set = [(X_test, y_test)]  # used for early stopping# populate the model with settings from config    xgb = XGBClassifier(        random_state=config_settings.model_settings.random_state,        objective=config_settings.model_settings.objective,        eval_metric=config_settings.model_settings.eval_metric,        eval_set=eval_set,        n_jobs=config_settings.model_settings.n_jobs,        learning_rate=config_settings.model_settings.learning_rate,        colsample_bytree=config_settings.model_settings.colsample_bytree,        subsample=config_settings.model_settings.subsample,        reg_alpha=config_settings.model_settings.reg_alpha,        reg_lambda=config_settings.model_settings.reg_lambda,        max_depth=config_settings.model_settings.max_depth,        gamma=config_settings.model_settings.gamma,        n_estimators=config_settings.model_settings.n_estimators,        max_delta_step=config_settings.model_settings.max_delta_step,        scale_pos_weight=config_settings.model_settings.scale_pos_weight    )    # fit the model    xgb_fit = xgb.fit(X_train, y_train)        # predict on test set    y_pred = xgb.predict(X_test)        # show results    print(classification_report(y_test, y_pred, digits=4, target_names=['setosa', 'versicolor', 'virginica']))        return xgb_fitxgb = fit_xgb_model(config_settings, X_train, y_train, X_test, y_test)
And violá! The model has been trained and evaluated!
You may be asking yourself:
Why would I write all this code just to pass some parameters to my model? It seems like a ton of work and way more code than is necessary!
It is true that you could simply populate a model with all of the parameters, but you are now locking yourself into that configuration! To make changes, you will need to go in and alter the code. What if you want to show your teammate your latest configuration which yielded the best results? Will you copy/paste your models code? Why not just slack them the config file?
This config is just the tip of the iceberg. By using a process such as the one I am outlining here, you can easily specify much more additional information that is relevant to your model pipeline. For example, where are the training files located? Put that path in the config! Which ETLs to run and where to store the temporary files after the application of each ETL? Config that! How many iterations of RandomSearch should be run? Config again! To give you an idea on the scope of these configs, consider the following highly simplified example:
ec2_settings:  ip_address: xxx.xxx.xxx  pem_file: '/Users/{}/Desktop/demo/ssh/my.pem'data_settings:  deploy_date: 'today' # or 'YYYY-MM-DD'  target_names: ['no', 'yes']  raw_bucket: '/Users/{}/Desktop/demo/datasets/raw'  etl_bucket_train: '/Users/{}/Desktop/demo/datasets/etl_train'  etl_bucket_score: '/Users/{}/Desktop/demo/datasets/etl_score'  tmp_bucket: '/Users/{}/Desktop/demo/datasets/tmp'  df_name: 'df.csv'logger_settings:  log_name: 'demo.log'  logs_path: '/Users/{}/Desktop/demo/logs'etl_settings:  pickles: '/Users/{}/Desktop/demo/src/etls/pickles'  target_variable: 'xxx_yyy'  idx_column: 'idx_col'model_settings:  pickles: '/Users/{}/Desktop/demo/src/modeling/pickles'  predictions: '/Users/{}/Desktop/demo/predictions'  eval_metric: ['error', 'map', 'aucpr']  n_iters: 500  sampling_points: 1000  optimization_metric: 'f1_weighted'  cv_folds: 5
With such a system in place, the sharing of results and experiments becomes as simple as slacking your fellow DS a yaml file.
You now have all the tools necessary to begin writing configs for you machine learning projects! This process takes some time to set up, but it pays off in the long run.
You enable rapid experimentation and you don’ t have to interact with the code to make changes to your model. You enable other scientists to simply change some settings and test away. By reading the YAML into a python class, you can easily enforce types and prevent erroneous values from being passed before spending money on expensive compute resources.
[1] https://yaml.org/
Data structures for statistical computing in python, McKinney, Proceedings of the 9th Python in Science Conference, Volume 445, 2010.
@software{reback2020pandas,    author       = {The pandas development team},    title        = {pandas-dev/pandas: Pandas},    month        = feb,    year         = 2020,    publisher    = {Zenodo},    version      = {latest},    doi          = {10.5281/zenodo.3509134},    url          = {https://doi.org/10.5281/zenodo.3509134}}
Harris, C.R., Millman, K.J., van der Walt, S.J. et al. Array programming with NumPy. Nature 585, 357–362 (2020). DOI: 10.1038/s41586–020–2649–2.
Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825–2830, 2011.
Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). New York, NY, USA: ACM. https://doi.org/10.1145/2939672.2939785