# Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data
## 2 Learning Algorithms and Methodologies
### Long Short-Term Memory (LSTM) Networks  
Neural networks try to model the behavior of the human brain. They consist of artificially created neurons and a set of edges that connect those neurons. Furthermore, each neuron has its associated activation function, which models neuron impulses. The RNN is a special type of neural network, where the input is a sequence. An RNN is very powerful because it not only uses the input to predict the output but it also utilizes the information from previously observed timestamps. All RNNs form a sequence of connected units that represent the state of the network at a timestamp, \(t\). A single module takes data from the previous unit, \(h_{t}-1\), and input for that timestamp, \(x_{t}\), then uses the \(tanh\) function to compute the output for a timestamp, \(t\). According to [32], a finite-sized RNN can compute any function that exists. However, RNNs suffer either from exploding or vanishing gradients when back-propagating through time. To update the weights, the neural network computes partial derivatives of the loss function of the current layer at each timestamp. Consequently, when the gradients are very small, either learning happens at a very slow rate or not at all. To overcome this issue with RNNs, the LSTM network is proposed, as suggested in [33]. The LSTM network is a special kind of RNN architecture. Instead of only using the tanh function in a unit, the LSTM network utilizes three gate units: a forget gate, an input gate, and an output gate. The forget gate is responsible for keeping only the relevant information, as given in (9a). It takes an input at timestamp \(x_{t}\) and the data from the previous hidden layer, \(h_{t}-1\). Then, the sigmoid function is applied to those inputs, and as a result, the output of the forget gate is somewhere between 0 and 1. The output closer to 0 will be forgotten, and the output with a numeric value of 1 will be kept for further calculations. Furthermore, the input gate decides how the memory cell will be updated, as shown in (9b). First, the candidate value is computed using (9c), then the result is scaled by the output of the input gate to decide by how much the cell state will be updated, as shown in (9d). Finally, the LSTM network employs an output gate, which is a filtered version of the cell state. First, the cell state is normalized using the tanh function, then the sigmoid layer that is presented in (9e) is utilized to decide which parts of the memory will be output, as presented in (9f). The outputs of the hidden state, \(h_{t}\), and the prediction, \(y_{t}\), is the same; however, the notation \(h_{t}\) are used as a hidden state input at timestamp \(t+1\). The structure of the LSTM network for a single unit is given in Figure 1.  
\[f_{t} =\sigma(W_{f}[h_{t-1},x_{t}]+b_{f}) \tag{9a}\] \[i_{t} =\sigma(W_{i}[h_{t-1},x_{t}]+b_{i})\] (9b) \[\tilde{c_{t}} =\tanh(W_{c}[h_{t-1},x_{t}]+b_{c})\] (9c) \[c_{t} =f_{t}\otimes c_{t-1}+i_{t}\otimes\tilde{c_{t}}\] (9d) \[o_{t} =\sigma(W_{o}[h_{t-1},x_{t}]+b_{o})\] (9e) \[h_{t} =\tanh(c_{t})\otimes o_{t} \tag{9f}\]  
The LSTM architecture is better suited for time series problems compared to the other mentioned algorithms. The LSTM model will learn the previously observed sequence before predicting the output, whereas the mentioned models treat each row in the dataset as an individual training sample.