## 2 Time Series Forecasting for Energy Reduction
### Seq2 Seq with Attention  
Instead of compressing the entire input into a single vector, for each input the encoder reads, the attention-mechanism takes into account the rest of the inputs and decides which ones are important by attributing different weights to those inputs. The Attention layer selects the relevant elements of the input sequence giving more _attention_ to them. Then the decoder will take the encoded sentence and the weights provided by the Attention mechanism to produce the output. Figure 8 presents a representation of the algorithm when translating Spanish into English.  
First, the encoder creates a representation of the Spanish sentence as a list of vectors. Then the decoder starts translating the sentence one word at a time. To generate the translation, the attention mechanism defines what words are more important and provides more context to translate the current Spanish word (what words to pay more attention) by creating a weighted distribution of the input words. In the image, when translating the second word \"European\", the attention mechanism decides that the word  
Figure 8: Example of text translation with the Attention mechanism.  
Figure 7: Sequence to Sequence model.  
with more weight is going to be the immediate translation and assigns some weight to \"Economica\" because it adds some context to the translation, while the rest of the words are not important. This allows the network to process long sequences by paying attention only to the important parts.  
Attention Mechanism  
The attention layer receives all the hidden states \(h_{t}\) from the encoder LSTMs units instead of a single vector containing all the information and the previous hidden state from the decoder \(s_{t-1}\) and computes the context vector \(v_{t}\), which tells the decoder what parts of the input are more important. The decoder receives the output of the attention layer \(v_{t}\), the previous hidden states \(s_{t-1}\), and the previous output \(y_{t-1}\) to generate the current output, as shown in Figure 9.  
The context vector, defined in Equation (15), is a weighted sum of the hidden states \(h_{t}\) and the alignment scores. The alignment score \(a_{t,j}\), defined in Equation (14), is produced by comparing the previous hidden state from the decoder \(S_{t-1}\) with the current hidden states from the encoder \(h_{i}\). The alignment assigns the score \(\alpha_{t,j}\) to every input based on the match of \((y_{t},X_{i})\), and these are weights that define how important every input is to produce the current output. The scores are normalized using a _Softmax_ layer to keep the values between the [0,1] interval. In this study, the [PERSON]'s Dot-Product attention score was used, as presented in [26] and defined in Equation (16), which is a dot product between the hidden states from the encoder and decoder.  
\[\alpha_{t,j}=align(y_{t},x_{i})=\frac{exp(score(s_{t-1},h_{t}))}{\sum_{j=1}^{n }exp(score(s_{t-1},h_{j}))} \tag{14}\]  
\[v_{t}=\sum_{i=1}^{n}\alpha_{t,i}h_{i} \tag{15}\]  
Figure 9: Sequence to Sequence with the Attention mechanism adapted from [9].  
\[score(s_{t},h_{i})=s_{t}^{\top}h_{i} \tag{16}\]  
Seq2 Seq and Attention models for time series forecasting have been used in many research articles, showing promising results when dealing with long sequences [19; 27; 28; 29].