# Predicting the Gap in the Day-Ahead and Real-Time Market Prices Leveraging Exogenous Weather Data
## 4 Data Preparation  
### Data Collection  
Three distinct datasets are collected and merged to form an extensive dataset for studying the price gap across the DAM and RTM. The first dataset is the one with information from the electricity market. The California Independent System Operator (CAISO) provides an Open-Access Same-time Information System (OASIS) Application Programming Interface (API), which produces reports for the energy market and power grid information  
\begin{table}
\begin{tabular}{l c} \hline \hline MAE & \(\left[\frac{1}{s}\sum_{t=1}^{s}|y_{t}-\hat{y}_{t}|\right]\) \\ \hline RMSE & \(\sqrt{\frac{1}{s}\sum_{t=1}^{s}(y_{t}-\hat{y}_{t})^{2}}\) \\ \hline nRMSE & \(\sqrt{\frac{1}{s}\sum_{t=1}^{s}(y_{t}-\hat{y}_{t})^{2}}\Big{/}[y_{\max}-y_{\min}]\) \\ \hline Max Error & \(max(|y_{t}-\hat{y}_{t}|)\forall t\in\{0,s\}\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Prediction errors for Station A while using temporal data for various learning algorithms.  
Figure 1: The LSTM architecture of a single unit.  
in real-time. To demo the results of this paper, the _MUIRRAY6N015_ node, located in San Diego, CA, is chosen, and its reports for the energy market and power grid information in real-time are leveraged. The LMP, the LMP congestion component, the LMP energy component, and the LMP loss component are collected for the DAM and RTM. Furthermore, a seven-day-ahead load forecast as well as load forecasts for the next two days are acquired using the CAISO API. The period of collection for the dataset is two years, starting from the 1 January 2017.  
The second one has meteorological data. For the historical hourly weather dataset, the Meteostat API is used. Meteostat collects hourly weather measurements from more than 5000 weather stations around the world. In addition, it offers comprehensive historical datasets that combine their measurements with the NOAA's Global Historical Climatology Network's dataset. The weather data are obtained from the San Diego International Airport weather station, which is the closest weather station to the node of interest. The collected data include information about temperature, dew point, humidity, wind speed, wind direction, weather condition, sea level pressure, wind gust, cloud layers, and weather forecasts for the next 3 and 6 days. Weather conditions directly and indirectly influence both the demand and supply of electricity, which in turn affects the price. Solar power generation is directly influenced by the amount of sunlight, while the production of electricity from wind turbines depends on wind speeds. If wind speeds are predicted to be low, wind-generated electricity might be reduced, potentially leading to higher prices. On the other hand, weather conditions may influence the demand for electricity. Extreme temperatures, both hot and cold, increase the demand for electricity.  
The third dataset concerns renewable energy availability. The mesonet API is utilized to acquire a dataset for solar irradiation. This API offers quality-controlled, surface-based environmental data such as Global Horizontal Irradiance (GHI), Direct Normal Irradiance (DNI), Diffuse Horizontal Irradiance (DHI), solar zenith angle, cloud type, and precipitable water. The GHI is the total amount of terrestrial irradiance received from above by a surface horizontal to the ground. The DNI means the radiation that comes in a straight line directly from the sun and is absorbed by a unit perpendicular to the rays. Furthermore, the DHI is the radiation that does not arrive on a direct path from the sun, and it is equally absorbed by the particles in the atmosphere. It should be noted that the historical weather data (i.e., the second dataset), the forecast weather data (i.e., the third dataset), and the forecast demand data (i.e., the first dataset) are utilized.