# Solar Irradiance Forecasting with Transformer Model
## 2 Dataset and Methods
### Transformer Model for Solar Irradiance  
The Transformer model is a neural network architecture used typically for natural language translation. Its basic element is the attention mechanism, which links related features in sequential data. Transformer is composed of basic parts, which are self-attention blocks and position-wise fully connected feed-forward network blocks. Self-attention blocks are formed by a normalization layer, a Multi-head Attention layer, a dropout layer, and finally a residual connection. Position-wise fully connected feed-forward network blocks are composed of a normalization layer, a pair of fully connected (dense) layers, a dropout layer, and finish with a residual connection. By stacking these basic blocks, two main parts of the model are created, the encoder and decoder. Self-attention block maps query and key-value inputs to one output from this block. The output is expressed as a weighted sum of the input value, where the weight is formed by the query input function with the corresponding key input, according to Equation (5). To improve the accuracy of the model, a combination of several attention matrices into one output was used according to Equation (6). The principle of the position-wise fully connected feed-forward network block is expressed by Equation (7) [4]. \[Attention(Q,\ K,V)=softmax\bigg{(}\frac{QK^{T}}{\sqrt{d_{k}}}\bigg{)}V \tag{5}\]  
\[MultiHead(Q,\ K,V)=Concat(head_{1},\ \dots,head_{h})W^{O} \tag{6}\]  
\[\text{where}\ head_{i}=Attention(QW_{i}^{O},KW_{i}^{K},VW_{i}^{V})\]  
Here, the matrix \(Q\) represents the query input, the matrix \(K\) represents the key input, and the matrix \(V\) represents the value input to the attention block. The value \(d_{k}\) expresses the size of the key input. Matrices \(W_{i}^{Q},\ W_{i}^{K},\ W_{i}^{V},\ \text{and}\ W^{O}\) represent trained model parameters for the projection of features [4]. \[FFN(x)=GeLU(xW_{1}+b_{1})W_{2}+b_{2} \tag{7}\]  
The GeLU (Gaussian Error Linear Unit) in Equation (7) represents nonlinearity in the model [25], and matrices \(W\) and \(b\) represent weights and bias. In the model used in this work (see Figure 5), unlike the original Transformer model [4], there are normalization layers before each block, and each block is terminated only by a residual connection, as is the case with the Vision Transformer, which, similarly to the problem of solar irradiance prediction, works on the input of the model with a real signal [25]. The truncated normal method with a standard deviation of 0.02 (as in the case of BEIT [26]) is used to initialize the training parameters of the model. The first input layer consists of a normalization layer adapted to training data, the task of which is to transform various measured physical variables with different ranges of values into variables with zero mean and unit standard deviation. The output from the model consists of a fully connected (dense) layer with the linear activation function that is followed by the denormalization layer, which helps to maintain a zero mean and one standard deviation of  
Figure 4: Visualization of the frequency of occurrences of wind characteristics by wind vector. targets. The denormalization layer works in the same manner as the normalization layer but in the inverse. The normalization layers and the denormalization layer are adapted to training examples only, and this is done before the training process. Their memorized values of means and standard deviations are also applied to both the validation and test datasets. The advantage of the last layer being linear is that the gradients are unaffected by the activation function of the neurons during the backpropagation since the first derivative of the linear function equals one. A learning rate cosine scheduler was also used for training, which optimizes the learning process [27] and allows the model only to warm up in the first steps of the learning process while the gradients are too high. Figure 5: Transformer model for Solar irradiance.