# KAN-Transformer Model for UltraShort-Term Wind Power Prediction Based on EWMA Data Processing
## 2 Research Methodology and EWMA-KAN-Transformer Model
### Structure of KAN-Transformer Prediction Model Based on EWMA Data Processing
#### 2.2.6 KAN Layer  
In Transformer models and their variants, the final prediction results are typically obtained through a simple linear layer for dimensionality reduction. Using self-learned nonlinear unit functions instead of a simple linear layer, the model can better capture nonlinear relationships and improve prediction accuracy. To address this, the KAN network, which is based on the KART mathematical principle, is introduced. The principle is described using Formula (5). \[f(x)=f(x_{1},\ldots,x_{n})=\sum_{q=1}^{2n+1}\Phi_{q}\left(\sum_{p=1}^{n}\phi_{q, p}(x_{p})\right), \tag{5}\]  
where \(q\) represents the number of unit functions, \(p\) represents the number of input units, \(\phi_{q,p}(x_{p})\) represents the unary function of the first layer in the KART network, and \(\phi_{q}\) represents the unary function of the second layer in the KART network. Based on Formula (5), it can be concluded that a multivariate function can be obtained by summing a finite number of nonlinear unit functions. In this process, if \(n=2\), then the first layer of the KART network will have a fixed number of nodes, specifically \((2n+1)=5\) nodes. The KAN network is a variant of this formula, where the fixed number of \((2n+1)\) nodes can be manually set. For ease of analysis, the basic architecture of the KAN network, using the example of predicting a future wind power value, is illustrated in Figure 10.  
Figure 9: Workflow of Decoder. In Figure 10, the 512 outputs from the Decoder, denoted as \(x_{1}\) to \(x_{512}\), are mapped to 512 unit functions \(\phi_{1}\)-\(\phi_{512}\). These unit functions are then summed to obtain the multivariate function \(f(x_{1},\ldots x_{512})\), which is used to derive the final prediction result \(pr\). The core of the KAN network is to learn each of these nonlinear unit functions. The learning process is described using Formula (6):  
\[\left\{ \begin{array}{l}\phi(x)=w(b(x)+\text{spline}(x))\\ b(x)=\text{silu}(x)=x/(1+e^{-x})\\ \text{spline}(x)=\sum c_{i}B_{i}(x)\end{array} \right., \tag{6}\]  
where \(w\) represents the learnable parameters, spline(\(x\)) represents the B-spline function, and \(c_{i}\) represents the learnable parameters. In Formula (6), each nonlinear unit function \(\phi(x)\) is defined by adding a B-spline function spline(\(x\)) to the \(\text{silu}(x)\) function and then scaling it by a learnable parameter \(w\) to ensure smoothness of \(\phi(x)\). The most important part is spline(\(x\)), which is the B-spline function; the KAN network uses a cubic B-spline. To facilitate a clear explanation of the process of learning the nonlinear unit function, the code for the most important component, spline(\(x\)), in the KAN network is shown in Figure 11. Additionally, the learning process of spline(\(x\)) and other components within the KAN network is elaborated in detail, as illustrated in Figure 12.  
Figure 11: spline(\(x\)) code writing process. Figure 10: KAN network infrastructure. In Figure 12, \(b(x)\) is a fixed silu function and does not require learning, while \(w\) is a learnable parameter used for scaling. The most critical aspect of the figure is the learning process of spline(\(x\)). The process is as follows: First, \(G_{1}=5\) is set, which indicates that there are five grid points where the nonlinear unit function will be formed. Second, seven B-spline functions are initialized with \(K=3\). By learning the weights \(c_{i}\) of these seven B-spline functions, one can control the shape of each B-spline function. These seven learned B-spline functions are summed to obtain spline(\(x\)). Then, a smooth curve \(\text{silu}(x)\) is added to spline(\(x\)) to enhance the smoothness of the unit function. Finally, \(\{\text{spline}(x)+\text{silu}(x)\}\) is multiplied by \(w\) to obtain the final unit function \(\phi(x)\). It is important to note that the process described is for learning a single unit function. In practical applications, one needs to map the 512 outputs of the Decoder layer to 512 nonlinear unit functions, each learnable.