## 4 STGPT2 UGAN Design
### Generator (Gen) Design
The key \(K_{z}^{S}\in\mathbb{R}^{D\times(\frac{D}{N_{z}})}\), query \(Q_{z}^{S}\in\mathbb{R}^{D\times(\frac{D}{N_{z}})}\), and value \(V_{z}^{S}\in\mathbb{R}^{D\times(\frac{D}{N_{z}})}\) are obtained using linear projection \(HST_{t_{k}}^{b-1}W_{K}^{S}\), \(HST_{t_{k}}^{b-1}W_{Q^{\prime}}^{S}\), and \(HST_{t_{k}}^{b-1}W_{V}^{S}\), respectively. \(W_{K}^{S}\), \(W_{Q}^{S}\), and \(W_{V}^{S}\in\mathbb{R}^{D\times(\frac{D}{N_{z}})}\) stand for the learnable parameters for linear projection. \(N_{z}\) denotes the total number of heads. \(\alpha\) is the scaling factor. After obtaining \(HS_{z,t_{k}}^{b}\) and \(z\in[1,N_{z}]\), the spatial representations \(HS_{t_{k}}^{b}\in\mathbb{R}^{N\times D}\) at timestep \(t_{k}\) are computed as follows:  
\[HS_{t_{k}}^{b}=\Big{[}HS_{1,t_{k}}^{b},HS_{2,t_{k}}^{b},\ldots,HS_{N_{z},t_{k }}^{b}\Big{]}W_{\sigma}^{S}, \tag{3}\]  
where \(W_{\sigma}^{S}\in\mathbb{R}^{D\times D}\) represents the weight parameters. \([*,\ldots,*]\) stands for the concatenation operation. _Temporal attention_: To model temporal dependencies, the temporal attention with one multi-head self-attention layer is presented, as depicted in Figure 2c. Similar to spatial attention, the model is able to dynamically allocate weights to various timesteps based on their importance. In the \(b\)th STA module, let the spatial representations of historical \(P\) timesteps \(\mathcal{HS}^{b}\in\mathbb{R}^{P\times N\times D}\) be the input of temporal attention, in which \(HS_{i}^{b}\in\mathbb{R}^{P\times D}\) denotes the spatial representations of turbine \(i\). \(\mathcal{HST}^{b}\in\mathbb{R}^{P\times N\times D}\) is the spatio-temporal representation generated by the one multi-head self-attention layer, in which \(HST_{i}^{b}\in\mathbb{R}^{P\times D}\) is denoted as  
Figure 2: The framework of the generator. (**a**) **STGPT-2**. \(\mathcal{X}\) is the input of feature embedding layer. \(\mathcal{H}\) is the input of \(B\) STA modules. In the \(b\)th STA module, \(\mathcal{HS}^{b}\) is the output of spatial attention. \(\mathcal{HST}^{b}\) is the output of temporal attention. \(\mathcal{HST}^{B}\) denotes the output of \(B\) STA modules. \(\mathcal{HZ}^{GL}\) is the input of tokenization. \(\mathcal{\hat{Y}}_{WP}\) stands for the predicted wind speed sequences in the future \(Q\) timesteps. (**b**) **Spatial attention. (c) Temporal attention. **  
the spatio-temporal representations of turbine \(i\). For the turbine \(i\), the \(z\)th head operation of the one multi-head self-attention layer is represented as follows:  
\[HST_{z,i}^{b}=\text{Softmax}\bigg{(}\beta Q_{z}^{ST}\Big{(}K_{z}^{ST}\Big{)}^{ \top}\bigg{)}V_{z}^{ST}, \tag{4}\]  
where \(HST_{z,i}^{b}\in\mathbb{R}^{P\times\big{(}\frac{D}{R_{z}}\big{)}}\) indicates the spatio-temporal representations of the \(z\)th head. \(K_{z}^{ST}=HST_{i}^{b}W_{K}^{ST}\), \(Q_{z}^{ST}=HST_{i}^{b}W_{Q}^{ST}\), and \(V_{z}^{ST}=HST_{i}^{b}W_{V}^{ST}\) are the key, query, and value, respectively. \(W_{K}^{ST}\), \(W_{Q}^{ST}\), and \(W_{V}^{ST}\in\mathbb{R}^{D\times\big{(}\frac{D}{R_{z}}\big{)}}\) are the trainable parameters for linear projection. \(\beta\) stands for the scaling factor. After acquiring \(HST_{z,i}^{b}\) and \(z\in[1,N_{z}]\), the spatio-temporal representations \(HST_{i}^{b}\in\mathbb{R}^{P\times D}\) of turbine \(i\) are calculated as  
\[HST_{i}^{b}=\Big{[}HST_{1,i}^{b},HST_{2,i}^{b},\dots,HST_{N_{z,i}}^{b}\Big{]}W _{o}^{ST}, \tag{5}\]  
where \(W_{o}^{ST}\in\mathbb{R}^{D\times D}\) denotes the weight parameters. (3) _TGPT-2_: According to the spatio-temporal representations generated by the STB, we develop TGPT-2 to estimate future wind speed. TGPT-2 consists of four structures, called tokenization, token and position encoding, GPT-2 [25], and output layer. In the following, we describe how each structure coordinates with each other to estimate the future wind speed.