# Predicting Hospital Admissions to Reduce Crowding in the Emergency Departments
## 3 Method and Materials
### Gradient Boosting  
Gradient boosting is a machine learning algorithm that builds an ensemble of weak trees in a sequential fashion, with each tree being trained with respect to the previous and reducing the marginal error of the whole ensemble learned so far [31]. Subsequent trees, therefore, help classify observations that are not well classified. Later, weak trees are combined in a gradual and additive manner into a powerful and strong model with high predictability, which is hard to beat with other algorithms [32]. The theory behind this process is to train the new base learners to be maximally correlated with the negative gradient of the loss function associated with the whole ensemble until the gradient descent is zero. In other words, the gradient descent measures the local gradient of the loss (cost) function for a given set of parameters (\(\Theta\)) as illustrated in Figure 3.  
For datasets with a categorical response \(y\in[0,1]\), the two most used loss functions are the Binomial and the Adaboost loss function, which are more generally referred to as the Bernoulli loss functions. Given that gradient boosting machines (GBM) learn sequentially from previous weak models, it is of high relevance to defining an adequate shrinkage value. This value can help regularize and control model complexity by potentially reducing the impact of unstable regression coefficients and reducing the size of incremental steps in the model learning process. The main principle applied is that taking many small steps in improving the model is better than taking fewer large steps. The shrinkage parameter is defined as \(\lambda\in(0,1]\)[33]. Although the number of weak trees to train is defined by the researcher, for small values of \(\lambda\) the gradient boosting algorithm will require many trees (>1000) to arrive at a satisfactory result (see Figure 3), which can be computationally expensive and time-consuming. In spite of this, reducing shrinkage too much can result in a negative outcome, known as overfitting. This is defined as an excessive improvement of the model to the training dataset, learning and adapting too much on its particularities and resulting in a decrease in the performance on the test dataset. This can be solved with the early stopping technique, which allows the algorithm to stop the number of iterations before the initially pre-specified  
Figure 3: (**a**) Too small learning rate \(\lambda^{29}\); (**b**) gradient descent in GBM function[29]; (**c**) too large learning rate \(\lambda^{29}\). one if it detects overfitting, or what is the same, a decrease in the predicting power of observations in the test set. This optimal number of boosts is therefore dependent on the shrinking parameter \(\lambda\). To deal with the trade-off existing between the learning rate and number of boosts is usually approached using a cross-validation procedure, which allows for testing the model on withheld portions of data, while still using all of the data at the processing stages. To do so, a cross-validating parameter k is defined and used to partition the data into k disjoint non-overlapping subsets, each of which will be used as a validation set for the GBM model fitted with the rest of the subsets. Only after doing the process for all the subsets will the validation performance from each of the folds be aggregated to serve as estimate of model generalization on the validation set [31]. One of the disadvantages of gradient boosting machines, apart from overfitting, is the comparably low interpretability of the results obtained. There are two main tools to address this issue. The first one is the relative variable influence analysis [34], which is a common tool used in many cases to base the feature selection, although it does not provide any specific explanation of how each variable actually affects variation in the result. The second tool is partial dependence plots, which are commonly used to visualize the effect of one selected variable on the response while controlling for the effect of all other explanatory variables.