## 2 Methodology: Data Pre-Processing and Models
### Long Short Term Memory Model  
Using the TensorFlow libraries ([[https://www.tensorflow.org/](https://www.tensorflow.org/)]([https://www.tensorflow.org/](https://www.tensorflow.org/))), we designed a stacked LSTM that allows for a configuration of multiple parameters including: the past history window, configuring how long of the past history should be used to predict current and future weather conditions, number of the gated recurrent units (GRU) in each hidden layer of the LSTM, number of stacked layers to determine the memory length, and how far into the future the network should forecast the grid behavior (number of output variables) [18; 19]. The cost function minimization is defined as the mean between each of the predicted time-bound variables, while the Adam optimizer orchestrates the learning (Adam: an extension of a gradient descent optimization that uses the adaptive moment estimator) [20]. Furthermore, the stacked LSTM consists of layers of GRUs, and as the network's depth increases (by increasing the number of hidden layers), so does the system's memory of the past used to make the future predictions. The learning rate is reduced over time, described as:  
```
while(step*self.batch_size<self.training_iters)and (testing_loss_value>target_loss): current_learning_rate=self.learning_rate current_learning_rate*=0.1** (step*self.batch_size)*self.training_iter_step_down_every)
```  
Figure 3 shows the illustration LSTM with three hidden layers of memory cells controlled at time \(t\) by the input \(x_{t}\), the sate \(c_{t}\) and the outputs parameters \(o_{t}\) and \(c_{t}\). The input gate transform \(i_{t}\) is defined as \(i_{t}=g(W_{xi}x_{t}+W_{hi}s_{t-1}+b_{i})\), the forget gate \(f_{t}\) where \(f_{t}=g(W_{xf}x_{t}+W_{hf}s_{t-1}+b_{f})\), the output \(o_{t}=g(W_{xo}x_{t}+W_{ho}s_{t-1}+b_{o})\), the input transform \(c^{\prime}_{int}=tanh(W_{xc}x_{t}+W_{hc}s_{t-1}+b_{cin})\), the cell's internal memory is updated by \(c_{t}=f_{c}*c_{t-1}*i_{t}*c_{int}\), and the state is updated using \(s_{t}=o_{t}*tanh(c_{t})\). The activation function \(g\) is implemented as a sigmoid function and the weights \(W\) are evolved by training the model on data [13]. Our LSTM instance used three hidden layers with 300 memory cells in each layer. The model's weights were updated using a batch size of 100 instances. The LSTM parameters were empirically tuned with the following parameter values: the learning rate = \(2.5\times 10^{-5}\), the past history windows length = 30, the number of output cells = 5, the number or GRU = 200, the number of hidden layers = 8, the criteria to terminate the model training is either the error loss = \(1.0\times 10^{-7}\) or a maximum of training epochs = \(1.0\times 10^{6}\), and the LSTM had 4 outputs where each output predicted the weather conditions for 10 min into the future (the total of 40 min). Each plot series in Figure 4 shows the error loss during the LSTM training, in other words, the plot lines show how well each the LSTM generalized the weather condition training data. The LSTMs trained on the data for wind direction, conditions, events and the wind gust speeds were the hardest to generalize and resulted in LSTMs with poor quality of predictions (top four plot series in Figure 4 respectively). A poor quality of LSTM models could be a result of an inconsistency or subjective judgment of a human observer in the categorization of the weather data and the subsequent labeling of the weather conditions. The labeled data series are the overall weather conditions and the weather events. The models failed to generalize the wind gust direction or the wind gust speed weather conditions because the weather data was collected from a limited geographical area and the microclimate data does not captivate information from a larger geographical region that would allow to predict the local \"anomaly\" events. The humidity, wind speed, temperature and dew point LSTMs had the most significant error loss and predicted the current and future weather conditions most accurately.