## 4 STGPT2 UGAN Design
### Generator (Gen) Design
_Tokenization_: To accommodate spatio-temporal representations for the subsequent GPT-2, channel independence and patching [26] are adopted to tokenize spatio-temporal representations to reach the input demands of GPT-2. Because GPT-2 has an inability to directly deal with the high-dimensional spatio-temporal representations \(\mathcal{HST}^{B}\in\mathbb{R}^{P\times N\times D}\), two FC layers are presented to decrease the dimensions from \(D\) to \(K\), identified as \(\mathcal{HCI}\in\mathbb{R}^{P\times N\times K}\). Channel independence then separates the multi-turbine spatio-temporal representations into the spatio-temporal representations of multiple single turbines, thus switching the representations' dimensions to \(\mathbb{R}^{P\times 1\times K}\). In the end, for the spatio-temporal representations of each turbine, patching is developed to integrate adjacent timesteps into a single patch, scaling down the dimensions from \(P\) to \(P_{v}\), where \(P_{v}\) is the total count of patches. The dimensions of each turbine's spatio-temporal representations are extended from \(1\times K\) to \(L_{d}\times K\), where \(L_{d}\) is each patch's length. In detail, according to the restored spatio-temporal representations \(\mathcal{HCI}\), channel independence (**CI**) and patching (**pa**) are leveraged to produce a sequence of patches:  
\[HI_{i}^{Pa}=\textbf{pa}\Big{(}\textbf{CI}\Big{(}\mathcal{HCI}^{GL}\Big{)}\Big{)},i\in[1,N], \tag{6}\]  
where \(HI_{i}^{Pa}\in\mathbb{R}^{P_{v}\times L_{d}\times K}\) denotes the patched spatio-temporal representations of turbine \(i\). _Token and position encoding_: In NLP practices, token encoding is devised to match patches' dimensions with the embedding dimensions of GPT-2 [27]. Analogous to the NLP practices, it is necessary to leverage token encoding to handle the patched spatio-temporal representations for the same purpose. However, in previous studies, token encoding was exclusively designed for facing text series in NLP. Such a model is extremely suited for scalar tokens while the patched spatio-temporal representations are vectors. Hence, a novel token encoding is created. Since 1D CNNs (**1D-Conv**) are able to deal with vectors and preserve local semantic information leveraging the convolution operation, we view **1D-Conv** as the token encoding. In detail, given the turbine \(i\)'s patched spatio-temporal representations \(HI_{i}^{Pa}\in\mathbb{R}^{P_{v}\times L_{d}\times K}\), we formalize the the token encoding as  
\[\textbf{e}_{i}^{\text{to}}\ =\textbf{1D-Conv}\Big{(}HI_{i}^{Pa}\Big{)}, \tag{7}\]where \(\mathbf{e}_{i}^{\text{to}}\in\mathbb{R}^{P_{v}\times T_{d}\times K}\) stands for the patching embedding produced by the token encoding. \(T_{d}\) is the dimensions of patching embedding, matching with the embedding dimensions of GPT-2. To learn the position correlations between various timesteps, the position encoding is used, deriving from the Transformer [15]. Given the index of the turbine \(i\)'s patch locations \(index_{i}\in\mathbb{R}^{P_{v}}\), a learnable lookup table \(\mathbf{P}^{\text{posi}}\) is adopted to map the patch locations:  
\[\mathbf{e}_{i}^{\text{posi}}=\mathbf{P}^{\text{posi}}\left(index_{i}\right), \tag{8}\]  
where \(\mathbf{e}_{i}^{\text{posi}}\in\mathbb{R}^{P_{v}\times T_{d}\times K}\) denotes the position embedding created by the position encoding. After obtaining the token encoding \(\mathbf{e}_{i}^{\text{to}}\) and the position encoding \(\mathbf{e}_{i}^{\text{posi}}\), \(\mathbf{e}_{i}^{\text{to}}\) and \(\mathbf{e}_{i}^{\text{posi}}\) are added to obtain \(\mathbf{e}_{i}\in\mathbb{R}^{P_{v}\times T_{d}\times K}\):  
\[\mathbf{e}_{i}=\mathbf{e}_{i}^{\text{to}}+\mathbf{e}_{i}^{\text{posi}}. \tag{9}\]  
Then, \(\mathbf{e}_{i}\) is fed into the GPT-2. _GPT-2_: Several studies linked to LLMs advise that preserving most parameters without training has a tendency to obtain an enhanced performance compared to LLMs with training from scratch [28; 29]. Therefore, pre-trained GPT-2 is leveraged to generate output embedding.