## 4 Gated Transformer Models
### Input Module
#### 4.1.1 Encoder Input  
* Input data design:  
The Input of Encoder mainly includes three data types. Historical wind power data are \(\mathrm{P}=\{P_{i},P_{i+1},\cdots,P_{i+T_{i}-1}\}\in\mathbb{R}^{\mathrm{B} \times\mathrm{T_{s}}\times 1}\), where \(P_{i}\) is the \(i\)-th random data point as the starting point of historical input, \(P_{i+T_{s}-1}\) is the endpoint of historical input, and the length of the historical power input sequence is \(T_{s}\). Historical NWP data are \(\mathrm{W}=\{W_{i},W_{i+1},\cdots,W_{i+T_{s}-1}\}\in\mathbb{R}^{\mathrm{B} \times\mathrm{T_{s}}\times\mathrm{F}}\). Similarly, this data sequence is a time series with a length of \(T_{s}\) and a dimension of F. Future NWP data are collected by a meteorological prediction system and are \(\mathrm{\widetilde{W}}=\{W_{i+T},W_{i+T+1},\cdots,W_{i+T+p-1}\}\in\mathbb{R}^{ \mathrm{B}\times\mathrm{T_{p}}\times\mathrm{F}}\). Unlike historical data, the length of the future sequence is \(T_{\mathrm{P}}\), which depends on the length of the power which needs to be predicted;  
* Data processing design:  
After data collection, data need to be preprocessed. Due to different types of data, the unit and fluctuation range of NWP data are inconsistent, so all the data are unified  
Figure 6: The pipeline of the gated transformer.  
under the same scale by standardized method for observation. In addition, data filling and data repair are used to avoid the bad influence on the prediction model due to data loss or abnormality, and improve the data quality. During this process, the format of the input data does not change;
* Dilated convolutional design:  
Then, the data are fused and extracted by the dilated convolution module. To ensure the high efficiency of convolution calculation during the input of long sequence data, the module is designed to be composed of three pyramid-shaped cavity convolution layers. This scheme can expand the data receptive field and extract features more efficiently on the premise of maintaining the computational complexity. Here, the Conv1d convolution model is adopted and the dilated rates are set to be {1, 2, 4}. We introduce the padding mechanism to ensure consistency in length. In addition, dilated convolution is generally suitable for power prediction of long input sequences. In short-term prediction, too large of a Receptive field will lose the variability in the data themselves. Therefore, an adaptive approach is adopted, which can automatically adjust the dilated ratio during short-term input, thus ensuring the efficient feature extraction of convolution. Note that the format of the data will change \(\mathbb{R}^{\mathbb{B}\times T_{\text{g}}\times\mathbb{F}}\rightarrow\mathbb{R }^{\mathbb{B}\times T_{\text{g}}\times\mathbb{D}}\), \(\mathbb{R}^{\mathbb{B}\times T_{\text{p}}\times\mathbb{F}}\rightarrow\mathbb{R }^{\mathbb{B}\times T_{\text{p}}\times\mathbb{D}}\) here, where \(\mathbb{D}=128\) is the set hidden layer dimension;
* Position embedding design:  
Finally, the data reconstruction is completed through the position coding module. It encodes the given input sequence according to odd and even positions. The odd positions are encoded using cosine functions \(\text{PE}_{(pos,2i+1)}=\cos(pos/10000^{2i/D})\). Even positions are encoded according to sinusoidal positions \(\text{PE}_{(pos,2i)}=\sin(pos/10000^{2i/D})\). After completing the encoding, the location information is added to the data sequence.