# High-Frequency Direction Forecasting of the Futures Market Using a Machine-Learning-Based Method
## 2 Related Methodology  
### eXtreme Gradient Boosting (XGBoost)  
As an effective ensemble learning algorithm, XGBoost was proposed by [PERSON] and [PERSON] in 2016 [15]. XGBoost is considered as an improved GBDT algorithm [39]. The main operating principle of XGBoost is that it is composed of several base classifiers that can be linearly superimposed into a more robust classifier to obtain the final optimization algorithm. Unlike the GBDT algorithm, first, a second-order Taylor expansion is utilized for the optimization process of the loss function in XGBoost instead of the first-order Taylor expansion in GBDT. Second, XGBoost adds a regularization term to the objective function to effectively reduce the model complexity and prevent the overfitting problem [40; 41]. The superiority of high accuracy, generalization ability, and fast running speed makes it an efficient and excellent computing tool in the fields of machine learning and data mining [42]. The prediction results \(\hat{y}_{i}\) summarize the leaf scores of the \(K\) weak learners produced by a boosting method. The formula is given by Equation (1):  
\[\hat{y}_{i}=\sum\limits_{k=1}^{K}f_{k}(x_{i}),f_{k}\in F \tag{1}\]  
where \(f_{k}(\cdot)\) represents the score for the \(k\)-th weak learner and \(F\) is the set of all CART decision trees. When establishing an XGBoost model, the main task is to search for the optimal parameters by minimizing the objective function \(Obj\). As shown in Equation (2), \(Obj\) consists of two components, which are the loss function term \(l\) and the model regularization term \(\Omega\):  
\[Obj=\sum\limits_{i=1}^{n}l(y_{i},\hat{y}_{i})+\sum\limits_{k=1}^{K}\Omega(f_ {k}) \tag{2}\]  
In Equation (2), the former part is an item used to measure the differences between the prediction \(\hat{y}_{i}\) and the target \(y_{i}\), and the latter part is employed to punish the complexity of the tree structure. The regularization term \(\Omega\) is defined as:  
\[\Omega(f)=\gamma T+\frac{1}{2}\lambda\sum\limits_{j=1}^{T}w_{j}^{2} \tag{3}\]  
where \(\gamma\) denotes the complexity parameter, \(T\) represents the total number of leaf nodes, \(\lambda\) is a regularization coefficient, and \(\omega\) is the score of the leaf nodes. Notably, the model is trained in an additive manner, that is, it adds a new \(f_{t}(x_{i})\) to develop the current model, and it establishes a new loss function. The objective function can be expressed as:  
\[Obj^{(t)}=\sum\limits_{i=1}^{n}l\Big{(}y_{i},\hat{y}_{i}^{(t-1)}+f_{t}(x_{i}) \Big{)}+\Omega(f_{t}) \tag{4}\]  
The loss function is extended by performing a second-order Taylor expansion during the process of gradient boosting, and the constant term is then removed to simplify the objective function. The final \(Obj\) is shown in Equation (5), while \(g_{i}\) and \(h_{i}\) are defined in Equation (6), representing the gradient statistics of the loss function:  
\[ \begin{array}{rcl}Obj^{(t)}&\cong&\sum\limits_{i=1}^{n}\Big{[}g_{i}f_{t}(x _{i})+\frac{1}{2}h_{i}f_{t}^{2}(x_{i})\Big{]}+\Omega(f_{t})\\ &=&\sum\limits_{j=1}^{T}\Big{[}\Big{(}\sum_{i\in I_{j}}g_{i}\Big{)}\omega_{j} +\frac{1}{2}\Big{(}\sum_{i\in I_{j}}h_{i}+\lambda\Big{)}w_{j}^{2}\Big{]}+ \gamma T\\ \\ &&g_{i}=\partial_{g_{i}^{(t-1)}}l\Big{(}y_{i},\hat{y}_{i}^{(t-1)}\Big{)},\ h_{i}= \partial_{\hat{y}_{i}^{(t-1)}}^{2}l\Big{(}y_{i},\hat{y}_{i}^{(t-1)}\Big{)}\\ \end{array}  \tag{5}\]  
Assume that an instance set \(I_{j}\) in the leaf \(j\) is \(I_{j}=\{i|q(x_{i})=j\}\). When the tree structure is given, the formula for calculating the optimal leaf weight \(w_{j}^{*}\) and the loss function at the leaf node \(j\) can be evaluated by Equations (7) and (8), respectively:  
\[w_{j}^{*}=-\frac{\sum_{i\in I_{j}}g_{i}}{\sum_{i\in I_{j}}h_{j}+\lambda} \tag{7}\]\[Obj^{(t)}(q)=-\frac{1}{2}{\sum_{j=1}^{T}}\frac{\left(\sum_{i\in I_{j}}g_{i}\right) ^{2}}{\sum_{i\in I_{j}}h_{i}+\lambda}+\gamma T \tag{8}\]  
In XGBoost, the greedy search algorithm is used to evaluate the split candidates and to find an optimal tree structure.