# A Novel Intelligent Method for Fault Diagnosis of Steam Turbines Based on T-SNE and XGBoost
## 2 Methods
### Imbalanced Data Recognition Model Based on SMOTE and XGBoost  
Sampling methods are very popular for balancing the class distribution. Over- and under-sampling methodologies have received considerable attention to counteract the effect of imbalanced data sets. The SMOTE algorithm is simple and efficient, has good anti-noise ability, and can improve the generalization of the model [27; 28]. The formal procedure is as follows. The minority class is over-sampled by taking each minority class sample and inserting synthetic examples along the line segments connecting any/all of the k minority class nearest neighbors. Depending on the amount of over-sampling required, neighbors are randomly selected from the k nearest neighbors. Synthetic samples are generated as follows: take the difference between the feature vector of sample and its nearest neighbor; multiply this difference by a random number between 0 and 1, and add it to the feature vector under consideration. This results in the selection of a random point along the line segment between two specific features. This approach effectively forces the decision region of the minority class to become more general [29]. Boosting is a machine learning technique that can be used for regression and classification problems. It generates a weak learner at each step and accumulates them in the overall model. If the weak learner for each step is based on the gradient direction of the loss function, it can be called gradient boosting decision tree (GBDT) [30]. The difference with GBDT is that only the first derivative of the loss function is used to compute the objective function. The XGBoost approximates the loss function using the second order Taylor expansion. The main algorithm is as follows and the source code of the XGBoost algorithm is in Appendix A. Assume that a data set is \(D=\{(x_{i},y_{i})\}(|D|=m,x_{i}\in R^{n},y_{i}\in R)\), then we obtain n observations with m features each and with a corresponding variable \(y\). Let \(\hat{y}\) be defined as a result given by an ensemble represented by the generalized model as follows:  
\[\hat{y}_{i}=\phi(\textbf{x}_{i})=\sum_{k=1}^{K}f_{k}(\textbf{x}_{i}),f_{k}\in F \tag{8}\]  
where \(f_{k}\) is a regression tree, and \(f_{k}(\textbf{x}_{i})\) represents the score given by the \(k\)-th tree to the \(i\)-th observation in data. In order to functions \(f_{k}\), the following regularized objective function should be minimized:  
\[L(\phi)=\sum_{i}l(\hat{y}_{i},y_{i})+\sum_{k}\Omega(f_{k}) \tag{9}\]  
where \(l\) is the loss function. To prevent too large complexity of the model, the penalty term \(\Omega\) is included as follows:  
\[\Omega(f_{k})=\gamma T+\frac{1}{2}\lambda\|\omega\|^{2} \tag{10}\]  
where \(\gamma\) and \(\lambda\) are parameters controlling penalty for the number of leaves \(T\) and magnitude of leaf weights \(\omega\) respectively. The purpose of \(\Omega(f_{k})\) is to prevent over-fitting and to simplify models produced by this algorithm. An iterative method is used to minimize the objective function. The objective function that minimized in \(j\)-th iterative to add \(f_{\text{j}}\) is:  
\[L^{(j)}=\sum_{i=1}^{n}l\left(y_{i},\hat{y}_{i}{}^{(t-1)}+f_{\text{j}}(\textbf{ x}_{i})\right)+\Omega(f_{\text{j}}) \tag{11}\]Equation (11) can be simplified by using the Taylor expansion. Then, a formula can be derived for loss reduction after the tree split from a given node:  
\[L_{split}=\frac{1}{2}\Bigg{[}\frac{\left(\sum_{i\in I_{L}}g_{i}\right)^{2}}{ \sum_{i\in I_{L}}h_{i}+\lambda}+\frac{\left(\sum_{i\in I_{R}}g_{i}\right)^{2}}{ \sum_{i\in I_{R}}h_{i}+\lambda}-\frac{\left(\sum_{i\in I}g_{i}\right)^{2}}{ \sum_{i\in I}h_{i}+\lambda}\Bigg{]}-\gamma \tag{12}\]  
where \(I\) is a subset of the available observations in the current node and \(I_{L}\), \(I_{R}\) are subsets of the available observations in the left and right nodes after the split.