# Prediction of Building's Thermal Performance Using LSTM and MLP Neural Networks
## 2 Materials and Methods
### Artificial Neural Networks Developed
#### 2.1.2 Long Short-Term Memory Neural Networks  
LSTM is a recurrent neural network architecture specifically for modeling time sequences and their long-range dependencies more accurately than conventional RNNs. LSTM introduces specific units (memory blocks) into the recurrent hidden layer that store the temporal state of the network through self-connections. In addition, the memory block also contains units known as gates to control the flow of information [38; 40]. A forget gate was incorporated into the memory blocks to prevent the LSTM model from processing continuous input flows without being segmented in subsequences. Furthermore, more complex LSTM networks also incorporate peephole connections between internal cells and their gates to learn the timing of the outputs [32; 35].  
The structure of traditional RNNs [57] can be presented by deterministic transitions from previous to current hidden states in form of a function:  
\[h_{t}^{l-1},\;h_{t-1}^{l}\rightarrow\;h_{t}^{l}\]  
where \(h_{t}^{l}\) represents a hidden state in layer \(l\) in timestep \(t\).  
However, LSTM counts with a complex dynamic that allows memorizing information for many timesteps. Long-term information is stored in a vector of memory cells \(m_{t}^{l}\in\mathbb{R}^{n}\)[21; 37]. The LSTM networks are empowered to decide whether to overwrite, retrieve or maintain this information for the next timestep. Their architecture is the following:  
\[h_{t}^{l-1},\;h_{t-1}^{l},\;m_{t-1}^{l}\to h_{t}^{l},\;m_{t}^{l}\]  
Let \(x_{1},\;x_{2},\;\dots,\;x_{k}\) be a classic input sequence (with \(k\) lags), where \(x_{t}\;\in\;\mathbb{R}^{D}\) denotes the vector of \(D\) real values at the timestep \(t\). The LSTM architecture defines an input gate \(i_{t}\), a forget gate \(f_{t}\) and an output gate \(o_{t}\). At specific time \(t\), as is observable in Figure 2a where an LSTM cell is shown, the hidden layer output would be \(h_{t}\), the cell input state \(\widetilde{N}_{t}\) and the cell output state \(N_{t}\)[21; 38; 39; 41].  
Figure 2: (a) Long short-term memory (LSTM) cell structure and (b) forecasting framework of LSTM neural network architecture.  
The two values \(N_{t}\) (Equation (1)) and \(h_{t}\) (Equation (2)) that are transmitted to the next timestep, following the steps explained in [21; 39], can be calculated as follows:  
\[\text{N}_{t}=\text{ i}_{t}\times\widetilde{N}_{t}+\text{ f}_{t}\times\text{N}_{t-1}, \tag{1}\]  
\[h_{t}=\text{ }o_{t}\times tanh(N_{t}) \tag{2}\]  
A particular LSTM neural network was built for the prediction of indoor temperatures and thermal demand of the studied building (see Figure 2b). In this case, the input matrix of the network would be the observed data \(X_{t}\) (see Table 1) and the one-dimensional output the predicted future data \(\widetilde{Y}_{t+1}\) (Equation (3)). Once known \(h_{t}\) (Equation (2)) the network output will be:  
\[\widetilde{Y}_{t+1}=\text{ }W_{2}\times h_{t}+b, \tag{3}\]  
where \(W_{2}\) is the weight matrix connecting the last hidden layer to the output layer and \(b\) the bias term in the output layer (see Figure 2b).  
Specifically, the LSTM architecture used in this study is also variable due to the different complexity of the models based on the number of time lags introduced. The variable parameters were the number of hidden and LSTM layers and the number of hidden neurons. In this case, the layer structures (LSTM and hidden layers) in the tested grid were: 2-1, 2-2, 3-1 and 3-2. On the other hand, the number of hidden neurons was a grid with eight values between 5 and 500. In addition, as in the MLP models, the activation function of the models was the reLU function [54], the kernel initializer was normal, the optimizing algorithm was Adam [55] and the batch size equal to 64 [56]. In order to avoid overfitting problems, they were trained with the same early stop as MLP neural networks (100 epochs) [52; 53].