# Fault Diagnosis for Wind Turbines Based on ReliefF and eXtreme Gradient Boosting
## 2 Introduction to the Relieff and XGBoost Algorithms
### The Principle of the XGBoost Algorithm  
The XGBoost algorithm was originally proposed by [PERSON] in 2016. It is a large-scale parallel machine learning algorithm based on Integrated Learning [20]. The XGBoost algorithm is a combination of multiple Classification and Regression Tree (CART) based on boosting integrated learning ideas. The CART is a kind of decision tree algorithm that can be used for both classification and regression. The generation of a CART is the process of recursively constructing a binary decision tree. Assuming that the characteristics of the data set are a, b, and c, the binary tree constructed by a CART is shown in Figure 2.  
The CART uses a top-down design method. Each iteration loop selects a feature attribute to be forked until it cannot be forked again. In the process of a CART construction, there are two major issues. The first is how to choose the best bifurcation feature attribute, and the second is how to get the best prediction score. There are two main ideas of boosting learning. The first point is that the selection of training set in each iteration is related to the learning results of the previous rounds; the second point is that it changes the data distribution by updating the weight of each sample in each iteration. There are two steps to train the classifier by using boosting learning. The first step is to learn each weak classifier step by step. The final strong classifier is composed of all the classifiers generated step by step. The second step is to change the weight of each sample according to the result in the previous step. The main idea of the XGBoost algorithm is to build a classification regression tree by using all the features in the data set. Each time a classification feature node is added, it is to learn a new function, and use the learning results to fit the last prediction residual; after the training, it is a K tree containing K classification feature nodes. For the prediction score of each sample, we need to find all  
Figure 2: Classification and Regression Tree (CART). the corresponding leaf nodes according to the characteristics of the sample, and the sum of the scores of all the leaf nodes is the prediction value of the sample. The XGBoost is integrated by multiple CART trees. The model expression is shown as  
\[\hat{y}_{i}=\sum\limits_{k=1}^{K}f_{k}(x_{i})\ \ f_{k}\in F, \tag{3}\]  
where \(\hat{y}_{i}\) is the predicted value; \(x_{i}\) is the sample of i-th input; \(K\) is the total number of trees, \(F\) represents the function space of decision tree (all cart trees), and \(f_{k}\) represents a function in function space \(F\). In order to better learn the aforementioned model, it is necessary to minimize the objective function. The objective function of the XGBoost algorithm is shown as  
\[Obj(\Phi)=\sum\limits_{i=1}^{n}l(\hat{y}_{i},y_{i})+\sum\limits_{k=1}^{K} \Omega(f_{k}). \tag{4}\]  
The objective function consists of two parts. The first part is the loss function used to measure the difference between the predicted value and the true value. The second part is the penalty term of the model complexity, it is represented by \(\Omega\). The expansion of \(\Omega\) is shown in Equation (5)  
\[\Omega(f)=\gamma T+\frac{1}{2}\lambda\|\omega\|^{2}, \tag{5}\]  
where \(\gamma\) represents the regularization parameter of leaf number, which is used to restrain the node from further splitting; \(\lambda\) represents the regularization parameter of leaf weight, which is used to prevent the node score from being too large; \(T\) represents the number of leaf nodes; \(\omega\) represents the score of leaf nodes. In the process of constructing a CART, the XGBoost algorithm solves the problem of bifurcation feature selection through greedy thoughts on the one hand, and solves the problem of obtaining prediction scores by optimizing the objective function on the other.