# Distributed Kernel Extreme Learning Machines for Aircraft Engine Failure Diagnostics
## 2 Review of Basic and Kernel Elm
### Kernel ELM  
Kernel ELM was developed from ELM using the kernel transforming technology, which enables it to have better generalization performance than ELM due to the kernel transformation from input space to kernel space. Minimizing the training errors as well as the output weights at the same time, KELM can be derived as the following constrained optimization form  
\[ \begin{split}&\min_{\mathbf{\beta}}\frac{1}{2}\|\mathbf{\beta}\|_{2}^{2}+ \frac{C}{2}\sum_{i=1}^{n}\varepsilon_{i}^{2}\\ & s.t.\ \mathbf{\varphi}^{T}(\mathbf{x}_{i})\mathbf{\beta}=t_{i}-\varepsilon_{i},i=1,2, \cdots,n\end{split}  \tag{4}\]where \(\mathbf{\varphi}(\cdot)\) defines the kernel transformation from input space to kernel space, \(\varepsilon_{i}\) is the \(ith\) training error, the specified parameter \(C\) is used to represent the tradeoff between \(\mathbf{\beta}\) and \(\varepsilon\). Through proper kernel transformation, the linear unsolvable problem in the input space can be converted to a linear solvable problem in the kernel space. According to the Karush-Kuhn-Tucker (KKT) theorem and after introducing the Lagrange multiplier \(\alpha_{i}\), the following dual optimization problem can be utilized to solve the output weights \(\mathbf{\beta}\)  
\[\min_{\mathbf{\beta},\mathbf{\alpha},\varepsilon}\left\{L=\frac{1}{2}\|\mathbf{\beta}\|_{2 }^{2}+\frac{C}{2}\sum_{i=1}^{n}\varepsilon_{i}^{2}-\sum_{i=1}^{n}\alpha_{i} \left(\mathbf{\varphi}^{\mathsf{T}}(\mathbf{x}_{i})\mathbf{\beta}-t_{i}+\varepsilon_{i} \right)\right\} \tag{5}\]  
Take the partial derivatives and make them zero, the KKT conditions can be written as  
\[\frac{\partial L}{\partial\beta_{j}}=0,j=1,\cdots,L\rightarrow\mathbf{\beta}= \Phi^{\mathsf{T}}\mathbf{\alpha} \tag{6}\] \[\frac{\partial L}{\partial\varepsilon_{i}}=0,i=1,\cdots,n \rightarrow\mathbf{\alpha}=C\varepsilon\] \[\frac{\partial L}{\partial\alpha_{i}}=0,i=1,\cdots,n \rightarrow\Phi\mathbf{\beta}-\mathbf{t}+\varepsilon=\mathbf{0}\]  
where \(\Phi=\left[\mathbf{\varphi}(\mathbf{x}_{1})\cdots\mathbf{\varphi}(\mathbf{x}_{n})\right]^{T}\) denotes the kernel output function. With a little bit of simple substitution and derivation, the output function may be transformed into the following expression  
\[f(\mathbf{x})=\mathbf{\varphi}^{\mathsf{T}}(\mathbf{x})\Phi^{\mathsf{T}}\mathbf{\alpha}=\mathbf{ \varphi}^{\mathsf{T}}(\mathbf{x})\Phi^{\mathsf{T}}\left(I_{n}/C+\Phi\Phi^{\mathsf{ T}}\right)^{-1}\mathbf{t} \tag{7}\]  
where \(I_{n}\) denotes identity matrix with \(n\)-dimension, and based on the ridge regression theory, the enhancement of the regulation item \(I_{n}/C\) is able to improve the generalization performance [35]. For the convenience of calculation, kernel transformation is uniformly written as inner product, and the kernel matrix is defined as  
\[K=\Phi\Phi^{\mathsf{T}}:K_{i,j}=\mathbf{\varphi}(\mathbf{x}_{i})\cdot\mathbf{\varphi}(\bm {x}_{j})=k(\mathbf{x}_{i},\mathbf{x}_{j}) \tag{8}\]  
The network structure of the original KELM algorithm is illustrated by Figure 1, including the input feature layer, the kernel mapping layer and the output layer. In the kernel mapping layer, all the training samples are used as the hidden nodes. Thus, the output function could be represented in the compact formulation as  
\[f(\mathbf{x})=\left[k(\mathbf{x},\mathbf{x}_{1})\cdots k(\mathbf{x},\mathbf{x}_{n})\right]\mathbf{\alpha} \tag{9}\]  
where \(\mathbf{\alpha}=\left(I_{n}/C+K\right)^{-1}\mathbf{t}\) denotes the output weights in terms of kernel mapping. KELM is similar to the SVM in that it only needs to define the kernel matrix without needing knowledge about the specific kernel mapping and the dimension of kernel space. It can be proved theoretically that if the expression of SVM does not contain bias terms, the result of KELM is consistent with that of least squares SVM. KELM adopts structural risk minimization strategy to ensure good generalization performance by comprehensively considering empirical risks as well as confidence intervals.