# Transformer Fault Diagnosis Method Based on Incomplete Data and TPE-XGBoost
## 2 Theory
### TPE-XGBoost Method  
The extreme gradient boosting (XGBoost) algorithm uses CART as the base classifier and integrates it with gradient boosting [22]. The gradient boosting framework of the XGBoost model makes it more efficient and flexible [23]. Compared with the Gradient Boosting Decision Tree algorithm, the generalization capability of XGBoost is improved by using advanced regularization. The basic principle is that each time a new CART is added as a base evaluator, the prediction residuals of the previous CART are fitted. The prediction results of all CARTs are accumulated to obtain the final results of the model. At the same time, the XGBoost algorithm adds a regularization term to the loss function, which greatly reduces the model complexity and can achieve a balance between model accuracy and complexity. Suppose the feature dimension of the sample data is \(m\), and the training dataset \(S=\{(x_{1},y_{1}),\,\ldots,\,(x_{n},y_{n})\}\) includes \(n\) samples, where \(x_{i}=(x_{i1},\,\ldots,\,x_{im})\). If the XGBoost model contains \(t\) weak evaluators, then the classification result of sample \(x_{i}\) is  
\[y_{i}^{t}=\sum_{k=1}^{t}f_{k}(x_{i}),f_{k}\in F\]  
where \(y_{i}\) denotes the diagnostic result of sample \(x_{i},f_{k}\) denotes the \(k\)-th weak evaluator, and \(F\) denotes the function space containing every potential regression tree. The objective function \(L\) of the XGBoost model is  
\[L^{(t)}=\sum_{i=1}^{n}l(y_{i},y_{i}^{(t-1)}+f_{t}(x_{i}))+\Omega(f_{t})\]  
where \(l\) is the loss function, which represents the difference between the classification result and the real value; \(\Omega\) is the regularization term, which is used to reduce the risk of overfitting in the classification process, and the expression is  
\[\Omega(f_{k})=\gamma T+\frac{1}{2}\lambda\sum_{j=1}^{T}\omega_{j}^{2}\]  
where \(\gamma\) and \(\lambda\) are the parameters used to prevent overfitting, \(T\) is the number of nodes, and \(\omega\) denotes the weight of each node. The objective function after the second-order Taylor series expansion is  
\[L^{(t)}\approx\sum_{i=1}^{n}\biggl{[}l(y_{i}^{t-1},y_{i})+g_{i}f_{t}(x_{i})+ \frac{1}{2}h_{i}f_{t}^{2}(x_{i})\biggr{]}+\Omega(f_{t})\]where \(g_{i}\) and \(h_{i}\) are the first- and second-order derivatives, respectively, and the expressions are  
\[g_{i}=\partial_{y_{i}^{(t-1)}}l(y_{i}^{(t-1)},y_{i})\]  
\[h_{i}=\partial_{y_{i}^{(t-1)}}^{2}l(y_{i}^{(t-1)},y_{i})\]  
Further, by removing the constant term of the objective function, then  
\[L^{(t)}\approx\sum\limits_{i=1}^{n}\left[g_{i}f_{t}(x_{i})+\frac{1}{2}h_{i}f_ {t}^{2}(x_{i})\right]+\Omega(f_{t}) \tag{2}\]  
If the dataset of sample numbers in leaf node \(j\) is defined as  
\[I_{j}=\{i|q(x_{i})=j\}\]  
where \(q(x_{i})\) is the value of the leaf label corresponding to \(x_{i}\). Then, the solution of Equation (2) is  
\[w_{j}^{*}=-\frac{\sum\limits_{i\in I_{j}}g_{j}}{\sum\limits_{i\in I_{j}}h_{j} +\lambda}\]  
\[f_{obj}=-\frac{1}{2}\sum\limits_{j=1}^{T}\left(\frac{\sum\limits_{i\in I_{j}}g _{j}}{\sum\limits_{i\in I_{j}}h_{j}+\lambda}\right)+\gamma T\]