Doing XGBoost hyper-parameter tuning the smart way — Part 1 of 2 | by Mateo Restrepo | Towards Data Science

In this post and the next, we will look at one of the trickiest and most critical problems in Machine Learning (ML): Hyper-parameter tuning. After reviewing what hyper-parameters, or hyper-params for short, are and how they differ from plain vanilla learnable parameters, we introduce three general purpose discrete optimization algorithms aimed at search for the optimal hyper-param combination: grid-search, coordinate descent and genetic algorithms. We report on the results of an experiment in which we use each of these to come up with good hyperparameters on an example ML problem taken from Kaggle. Although we focus on optimizing XGBoost hyper-parameters in our experiment, pretty much all of what we will present applies to any other advanced ML algorithms.
The most powerful ML algorithms are famous for picking up patterns and regularities in the data by automatically tuning thousands (or even millions) of so-called “learnable” parameters. For instance, in tree-based models (decision trees, random forests, XGBoost), the learnable parameters are the choice of decision variables at each node and the numeric thresholds used to decide whether to take the left or right branch when generating predictions. In neural networks, the learnable parameters are the weights used on each connection to amplify or negate the activation from a previous layer into the next one.
Learnable parameters are, however, only part of the story. In fact, they are the easy part. The more flexible and powerful an algorithm is, the more design decisions and adjustable hyper-parameters it will have. These are parameters specified by “hand” to the algo and fixed throughout a training pass. Further, the algo typically does not include any logic to optimize them for us. In tree-based models, hyper-parameters include things like the maximum depth of the tree, the number of trees to grow, the number of variables to consider when building each tree, the minimum number of samples on a leaf, the fraction of observations used to build a tree, and a few others. For neural networks, the list includes the number of hidden layers, the size (and shape) of each layer, the choice of activation function, the drop-out rate and the L1/L2 regularization constants.
From a computational point of view, supervised ML boils down to minimizing a certain loss function (e.g. the MSE or the classification error) that depends on the data training input data (Xt, Yt), the learnable params which we denote by a, and the hyper-params. The crucial observation here is that this minimization is done by letting only the learnable parameters vary, while holding the data and the hyper-params constant. To choose the optimal set of hyper-params, the usual approach is to perform cross-validation. Given a hyper-param vector h, its quality is assessed by evaluating the loss function on a held-out set of validation data using the best set of learnable parameters, a*, for that value of h. Symbolically, we have
Introducing the Hyper-Parameter Grid
One important thing to note about hyper-parameters is that, often, they take on discrete values, with notable exceptions being things like drop-out rates or regularization constants. Thus, for practical reasons and to avoid the complexities involved in doing hybrid continuous-discrete optimization, most approaches to hyper-parameter tuning start off by discretizing the ranges of all hyper-parameters in question. For example, for our XGBoost experiments below we will fine-tune five hyperparameters. The ranges of possible values that we will consider for each are as follows:
{"learning_rate"    : [0.05, 0.10, 0.15, 0.20, 0.25, 0.30 ] , "max_depth"        : [ 3, 4, 5, 6, 8, 10, 12, 15], "min_child_weight" : [ 1, 3, 5, 7 ], "gamma"            : [ 0.0, 0.1, 0.2 , 0.3, 0.4 ], "colsample_bytree" : [ 0.3, 0.4, 0.5 , 0.7 ] }
For an explanation of what each of these means in the context of XGBost we refer the reader to this post.
Notice that despite having limited the range for the (continuous) learning_rate hyper-parameter to only six values, that of max_depth to 8, and so forth, there are 6 x 8 x 4 x 5 x 4 = 3840 possible combinations of hyper parameters. This discrete subspace of all possible hyper-parameters is called the hyper-parameter grid. In what follows, we will use the vector notation symbol h = [h0, h1, ..., hp] to denote any such combination, that is, any point in the grid.
Exhaustive grid search (GS) is nothing other than the brute force approach that scans the whole grid of hyper-param combinations h in some order, computes the cross-validation loss for each one and finds the optimal h* in this manner. Generally, the use of lexicographic ordering, that is, the dictionary order imposed on hyper-param vectors, is discouraged and a different order should be considered. The reason is that with lexicographic ordering there is a high chance that the search will focus on a rather uninteresting part of the search space for a rather long time. An interesting alternative is scanning the whole grid in a fully randomized way that is, according to a random permutation of the whole grid . With this type of search, it is likely that one encounters close-to-optimal regions of the hyper-param space early on. We show some evidence of this in the example below.
Coordinate descent (CD) is one of the simplest optimization algorithms. It’s an iterative algorithm, similar to gradient descent, but even simpler! The basic idea is that, at each iteration, only one of the coordinate directions of our search vector h is altered. To pick which one, we examine each coordinate direction turn and minimize the objective function by varying that coordinate and leaving all the other constant. Then we pick the direction that yields the most improvement. The algorithm stops when none of the directions yields any improvement. To learn all the details we refer the reader to the Wikipedia article.
The main advantage of this simple approach is that it is easily adaptable to the fully discrete case for which the arbitrary directions given by gradient descent cannot be easily used. We provide an implementation here.
Genetic algorithms (GAs) are a whole class of optimization algorithms of rather general applicability and are particularly well adapted for high-dimensional discrete search spaces. A genetic algorithm tries to mimic nature by simulating a population of feasible solutions to a(n optimization) problem as they evolve through several generations and survival of the fittest is enforced. There are two basic mechanisms for generating a new generation from the previous one. One is cross-breeding in which two individuals (feasible solutions) are combined to produce two offspring. The other one is mutation. With a given mutation probability, any individual can change any of their params to another valid value. Survival of the fittest is enforced by letting fitter individuals cross-breed with higher probability than less fit individuals. The fitness of an individual is of course the negative of the loss function.For the whole details we refer the reader to Wikipedia or to AIMA, Part II Chapter 4. Our own implementation is available here.
To get a feeling for each of the hyper-param optimization approaches mentioned above, we performed an experiment using actual data taken from the Home Credit Default Risk binary-classification competition, currently being held on Kaggle. We only considered the application_(train|test) data sets and performed a rather minimal data preparation step (see function prepare_data here). Further, to keep training and validation times short and allow for full exploration of the hyper-param space in a reasonable time, we sub-sampled the training set, keeping only 4% of the records (approx. 12,000). For validation, we used a separate 3.2% of the records (approx. 10,000). Even with this small amount of data, a single round of training and validation takes around 3 seconds, for a given set of hyper-params, which is not negligible considering that we intend to scan the whole hyper-param grid. For the reasons just mentioned, the cross-validation AUC values we get are not competitive with the ones at the top of the leader board, which surely are making use of all available records in all data sets provided by the competition.
As mentioned before, we chose XGBoost as our machine-learning architecture. Due to the outstanding accuracy obtained by XGBoost, as well as its computational performance, it is perhaps the most popular choice among Kagglers and many other ML practitioners for purely “tabular” problems such as this one.
Now, for each of the three hyper-param tuning methods mentioned above, we ran 10,000 independent trials. Each trial first resets the random seed to a new value, then initializes the hyper-param vector to a random value from our grid, and then proceeds to generate a sequence of hyper-param vectors following the optimization algorithm being tested. When testing GS, the trial just goes through hyper-param vectors according to a random permutation of the whole grid. With CD, the generated hyper-param vectors are all the ones tried out in intermediate evaluations of the CD algorithm. When CD gets stuck at a local optimum, it gets restarted to a new initial vector uniformly at random. With the genetic algo, the hyper-param vectors are the ones contained in the “DNA” of every individual generated in every successive generation.
By the way, we were able to run these many trials despite the cost of each CV-loss evaluation thanks to a technique called memoization (https://en.wikipedia.org/wiki/Memoization). Essentially, once the CV-loss of a hyper-param vector was evaluated for the first time, we kept it in a look-up table shared among the three methods and never had to re-evaluate it again.
While running each trial, we keep a log of all the hyper-param vectors tried and their corresponding cross-validation losses. The following plots show the behavior of a typical trial, for each of the three methods, The y-axis shows the negative of the CV loss, which in this case is just the CV_AUC metric which we are maximizing. The X-axis keeps track of “time”, that is, the total number CV-AUC evaluations up to a point.
The orange “envelope” line keeps track of the “running best,” which is the best AUC value seen among all function evaluations prior to a point. In a sense, this line is all we care about, as every method will always return the best of all hyper-param vectors evaluated and not simply the last one. Notice how different all three methods look at this level. GS succeeds in gradually maximizing the AUC by mere chance. The deliberate “effort” that coordinate descent and the genetic algorithm make in finding progressively better values is apparent from their plots as well.
However, one randomized trial does not tell the whole story. Let’s look at the “envelope” lines for each of 30 independent trials of each of the three methods.
Interesting... The envelope lines of the grid-search and genetic algos quickly cluster towards the top, unlike the CD one... This means that, with high probability, almost all trials of GS and genetic are likely to give near optimal solutions quicker than CD. On the other hand, the overall best solution is achieved around function evaluation #40 by one of the CD trials, and we don’t see the same for the other methods. Also coordinate descent beats the other two methods after function evaluation #100 or so, in that all of the 30 trials are nearly optimal and show a much smaller variance!
Let’s look at a final plot. What if we take the 10% percentile of the running-best AUC among all 10000 trials? This number is interesting because it gives us a sense of how the running-best of most trials evolves as a function of the number of evaluations. Another way to put it is that it gives us a 90% confidence bound on the worst we can expect from each method. Here it is:
Aha! This is more compelling evidence of what was mentioned above as it tells us that,with probability at least 90%, a randomized trial of coordinate descent will have seen, by CV-AUC evaluation #90, a hyper-param vector with an AUC of around 0.740 or better, whereas the corresponding value for GS is only around 0.738. This might seem like a minor difference, but it is quite significant as far as ML metrics go. Perhaps more telling than that is the fact that, up until the point it plateaus, the CD curve in the plot above has a distinctly higher slope than the GS curve.
The GA, on the other hand, seems to be dominated by GS throughout. This perhaps shouldn’t surprise us. On the one hand GA’s depend on several adjustable parameters (which would be hyper-hyper-parameters in our formulation) such as generation size, mutation rate, ordering of the instance variables to build a DNA sequence and the function linking AUC to the probability of cross-breading. For our experiment, we only tried a couple of settings for these. It is likely that with different settings GA would have beat GS. On the other hand, it’s known (see, for instance AIMA , at the end of page 148 ) that genetic algorithms work best when there are contiguous blocks of genes (hyper-params in our case) for which there are certain combinations of values that work better on average. It is not clear if this is the case in for the rather arbitrary ordering of XGBoost hyper-params that we have chosen.
In this post we clarified the distinction between params and hyper-params in the context of supervised ML and showed, through an experiment, that optimizing over hyper-params can be a tricky and costly business, in terms of computing time and electricity. We see, however, that there are approaches such as coordinate descent that go beyond plain vanilla grid search in terms of speed and quality of the discovered hyper-param vectors.
Stay tuned for a second post in this series in which we shall try out three other optimization approaches: Bayesian optimization, HyperOpt and auto-tune.