SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Improving day-ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context
Oussama Boussif  $^{1,2}$  * Stefano Massaroli  $^{1,2}$
Ghai Tianle Yuan4
ukachab  $^{1,3}$  Loubna Benabbou
Dan Assouline  $^{1,2}$  *  
Yoshua Bengio  $^{1,2}$
<sup>1</sup>Mila - Québec AI Institute  
<sup>2</sup>Université de Montréal
<sup>3</sup>Université du Québec à Rimouski  
<sup>4</sup>NASA Goddard Space Flight Center
Solar power harbors immense potential in mitigating climate change by substantially reducing  $\mathrm{CO}_{2}$  emissions. Nonetheless, the inherent variability of solar irradiance poses a significant challenge for seamlessly integrating solar power into the electrical grid. While the majority of prior research has centered on employing purely time series-based methodologies for solar forecasting, only a limited number of studies have taken into account factors such as cloud cover or the surrounding physical context. In this paper, we put forth a deep learning architecture designed to harness spatio-temporal context using satellite data, to attain highly accurate day-ahead time-series forecasting for any given station, with a particular emphasis on forecasting Global Horizontal Irradiance (GHI). We also suggest a methodology to extract a distribution for each time step prediction, which can serve as a very valuable measure of uncertainty attached to the forecast. When evaluating models, we propose a testing scheme in which we separate particularly difficult examples from easy ones, in order to capture the model performances in crucial situations, which in the case of this study are the days suffering from varying cloudy conditions. Furthermore, we present a new multi-modal dataset gathering satellite imagery over a large zone and time series for solar irradiance and other related physical variables from multiple geographically diverse solar stations. Our approach exhibits robust performance in solar irradiance forecasting, including zero-shot generalization tests at unobserved solar stations, and holds great promise in promoting the effective integration of solar power into the grid.
Solar power has become an increasingly important source of renewable energy in recent years, with the potential to help mitigate the effects of climate change by reducing greenhouse gas emissions (Doblas-Reyes et al., 2021; IEA, 2021). However, the variability of solar irradiance - the amount of solar radiation that reaches the earth's surface - presents a challenge for integrating solar power into the grid. Accurate forecasting of solar irradiance can assist grid operators in dealing with the variability of solar power, leading to a more efficient and dependable integration of solar power into the grid. As a result, this can help to reduce the requirement for costly and environmentally damaging backup power sources.
\*Equal contribution. Corresponding authors: oussama.boussif@mila.quebec, ghait.boukachab@mila.quebec, dan.assouline@mila.quebec Code & dataset: https://github.com/gitbooo/CrossViVit
Figure 1: CrossViViT architecture, in its Multi-Quantile version. (1) The spatio-temporal context videos are tokenized, partially masked, and encoded with a vision transformer, using Rotary Positional Embedding, (2) The time series are tokenized and encoded in parallel with a transformer, (3) Both resulting latents are fed into  $L$  layers of a cross transformer to mix them into one latent, (4) the output is passed into another transformer decoder, (5) and passed to multiple MLP heads which predict multiple quantiles, forming a prediction interval for each day-ahead time-series prediction.
Solar irradiance is influenced by a range of factors, including the time of day, the season, weather patterns, and the position of the sun in the sky. However, one of the most significant factors affecting solar irradiance variability is cloud cover. Clouds can block or scatter solar radiation, leading to rapid changes in solar irradiance at the earth's surface. Forecasting solar irradiance accurately thus requires modeling cloud cover, as well as accounting for the inherent variability of the system.
While a lot of previous work has focused on using pure time-series approaches to forecast solar irradiance (Yang et al., 2022), few have incorporated cloud cover (Nielsen et al., 2021; Bone et al., 2018; Si et al., 2021) and even fewer tackled the challenging day-ahead forecasting, often focusing on the easier very short term predictions (2 to 4 hours). When forecasting solar irradiance at a specific station, relying solely on its local physical variables is insufficient due to the significant spatial variability of cloud cover. To accurately anticipate the impact of clouds on incoming solar radiation, it is necessary to consider their motion and trajectory within a larger spatial context encompassing the station. In this paper, we incorporate satellite imaging to forecast solar irradiance at any chosen station and propose a multi-modal architecture that can in principle be used to forecast any physical variable. This study reveals the inadequacy of conventional testing schemes using conventional metrics like MAE or RMSE on an entire dataset for evaluating solar irradiance forecasting models. They fail to capture the models' performance in crucial cloud-related scenarios. Hence, we emphasize the necessity of a separate evaluation approach to address the complexity of cloud impact on solar irradiance variability. To allow for the metrics to show such performances by limiting the smoothing effect of the averaging, we propose a testing scheme based on multiple splits of the test data, separating particularly difficult examples from easy ones, for the task at hand. This is particularly important for practical downstream tasks related to solar irradiance estimation, such as the monitoring of solar power plants, where the correct prediction of such difficult examples can have a very high impact, as easy examples (in the presence of light cloud cover) can be treated fairly well by very simple models (such as a clear-sky or persistence model, as presented in section 5.1).
Contributions: Our focus is on the application of machine learning to time-series forecasting, with a particular emphasis on utilizing multi-modal spatio-temporal data that includes physical, weather, and remotely sensed variables. The main contributions of this paper can be summarized as follows:
- We present a deep learning architecture called CrossViViT, designed to leverage spatiotemporal context (such as satellite data) in order to achieve highly accurate medium-term (1 day horizon) time-series forecasting at any given station. This paper focuses specifically on forecasting Global Horizontal Irradiance (GHI).
- We present a Multi-Quantile version of the model which allows to extract uncertainty estimation attached to each prediction. This methodology, although applied here in our context, should be applicable to any forecasting task and framework.
- We present a multi-modal dataset that combines satellite imagery with solar irradiance and other physical variables. The dataset covers a period of 14 years, from 2008 to 2022, and includes data from six geographically diverse solar stations. This dataset is unique in its combination of diverse variables and long time span, and is intended to facilitate the development and evaluation of new multi-modal forecasting models for solar irradiance.
- We propose a forecasting testing scheme based on multiple time splits of the test data, separating particularly difficult examples from easy ones, therefore allowing to capture the models' performances in problematic examples.
- We experimentally show that the proposed approach can generalize to a new station not seen during training in a zero-shot generalization forecasting setting.
Machine Learning for time-series forecasting Deep learning approaches have gained popularity for time-series forecasting in recent years due to their ability to model complex nonlinear relationships and capture temporal dependencies. These approaches have demonstrated superior performance compared to traditional statistical methods, motivating further research in this area. In a recent survey (Wen et al., 2022), it was found that transformers, renowned for their success in natural language processing and computer vision, were also effective for time-series analysis. The authors discussed the strengths and limitations of transformers and compared the structure and performance of recent transformer-based architectures on a benchmark weather dataset (Zhou et al., 2021). The particular case of solar irradiance forecasting represents an interesting application for time-series models (Wang et al., 2019a; Narvaez et al., 2021; Alzahrani et al., 2017). One recent study developed a multi-step attention-based model for solar irradiance forecasting that generates deterministic predictions and quantile predictions as well (Sharda et al., 2021). In a similar perspective, Jønler et al. (2023) developed a probabilistic solar irradiance transformer that incorporates gated recurrent units and temporal convolution networks, demonstrating strong performance for short-term horizons.
Context mixing / Multimodal learning for time-series forecasting Previous studies highlight the potential of time-series methods for solar irradiance forecasting, emphasizing the significance of short-term horizons in solar energy management. However, day-ahead forecasting remains challenging due to the influence of cloud cover on surface irradiance (Bone et al., 2018; Si et al., 2021), a problem which we aim to address in this paper. Thus, it is crucial to account for cloud effects in solar irradiance forecasting regardless of the chosen method. For instance, Zhang et al. (2023) investigated the impact of cloud movement on irradiance prediction and proposed an approach to automatically learn the relationship between sky image appearance and solar irradiance. A concurrent work (Liu et al., 2023) proposed a multimodal-learning framework for ultra-short-term (10min-ahead) solar irradiance forecasting. They used Informer (Zhou et al., 2021) to encode historical time-series data, then utilized Vision Transformer (Dosovitskiy et al., 2020) to handle sky images. Finally, they employed cross-attention to couple the two modalities. The studies discussed above highlight the potential of incorporating external data sources, such as sky images and satellite images, in combination with time-series approaches to improve the accuracy of solar forecasting.
Operator Learning Utilizing available satellite imagery to forecast GHI over a region presents limitations as it may not capture clouds that exist at a resolution beyond that of the satellite data. To ensure accurate forecasting of quantities of interest, the ability to query the model at any possible resolution and any point within the domain becomes crucial. Recent advancements have witnessed
the rise of algorithms focusing on learning operators capable of mapping across functional spaces, with a focus on solving partial differential equations (PDE) (Lu et al., 2019; Li et al., 2021; Kovachki et al., 2021; Li et al., 2020). These operators can effectively map initial conditions to PDE solutions, making it possible to query the learned solution theoretically anywhere within its domain. Fourier Layers, developed by Li et al. (2021), enable zero-shot prediction on both uniform and non-uniform grids with learnable deformations (Li et al., 2022). Pathak et al. (2022) replace attention in ViT (Dosovitskiy et al., 2020) with Fourier layer mixing for competitive weather forecasting results with faster inference. MeshFreeflowNet (Jiang et al., 2020) learns high-resolution frames from corresponding lower resolution ones by querying the model at any point of the domain for irregular grids. Similarly, Boussif et al. (2022) employ message passing with a low-resolution graph for zero-shot super-resolution PDE learning. Additionally, message passing neural PDE solvers (Brandstetter et al., 2022) exhibit spatio-temporal multi-scale capabilities benefiting from long-expressive memory (Equer et al., 2023; Rusch et al., 2022). We note that while these approaches were developed for PDEs in mind, they can still be used for weather-related applications.
Uncertainty estimation When performing solar irradiance forecasting, deterministic forecasts are not sufficient to characterize the inherent variance and uncertainty in solar irradiance data. Probabilistic forecasts, providing uncertainty information, are crucial for energy system management (Wang et al., 2019b). (Doubleday et al., 2020) and (Yagli et al., 2020) benchmark solar forecasting methods, emphasizing calibration and sharpness in prediction intervals. Specifically in short term solar irradiance forecasting, Zelikman et al. (2020) delves into post-hoc calibration for better predictions. Turkoglu et al. (2022) introduces FiLM-Ensemble, balancing predictive accuracy and calibration in uncertainty estimation. The deep ensembles approach by Lakshminarayanan et al. (2016) aggregates neural network predictions, capturing both data noise and model uncertainties. Few studies tackle uncertainty evaluation in a regression setting, rather than in classification one, which consists in the estimation of prediction intervals. Sønderby et al. (2020), performing precipitation forecasting, suggests an easy solution: the output is separated in 512 bins and the model predicts the precipitation rate (probability) in each of the bins, resulting ultimately in a distribution. Other methodologies include bootstrapping and ensembling methods, drawing a distribution out of multiple predictions from submodels, following the spirit of Quantile Regression Forests (Meinhausen and Ridgeway, 2006). The evolving landscape of solar irradiance forecasting underscores the importance of a calibrated, comprehensive, and robust probabilistic approach to address inherent uncertainties.
We develop a framework for solar irradiance time-series forecasting, incorporating spatio-temporal context alongside historical time-series data from multiple stations. This framework is inspired by recent advancements in video transformer models (Arnab et al., 2021; Feichtenhofer et al., 2022) and multi-modal models that leverage diverse data sources such as images and time series (Liu et al., 2023). To establish the foundation for our framework, we provide a brief overview of the Rotary Positional Embedding (Su et al., 2021). Subsequently, we present the proposed architecture, CrossViViT, in detail, outlining its key components and design principles. Details about the ViT (Dosovitskiy et al., 2020) and ViViT architectures can be found in the Appendix.
3.1 Rotary Positional Embedding
As we are dealing with permutation-variant data, assigning positions to patches before using attention is necessary. A common approach is to use additive positional embedding, which considers the absolute positions of the patches added into the input sequence. However, for our case, it is more appropriate to have dot-product attention depend on the relative "distance" between the patches. This is because the station should only be concerned with the distance and direction of nearby clouds, and therefore, a relative positional embedding is more sensible.
To this end we make use of RoPE (Su et al., 2021) to mix the station time series and the context. The station and each patch in the context are assigned their normalized latitude and longitude coordinates in  $[-1,1]$  which are used as positions for RoPE. Details about the formulation used are left to the Appendix.
3.2 Cross Video Vision Transformer for time-series forecasting (CrossViViT)
CrossViViT. Our approach aims to integrate the available historical time-series data with video clips of spatio-temporal physical context to enhance the prediction of future time series for solar irradiance. The overall methodology, depicted in Figure 1, can be summarized as follows:
1. Tokenizing: The video context  $\mathbf{V} \in \mathbb{R}^{T \times C_{ctx} \times H \times W}$ , with  $T$  frames for each of the  $C_{ctx}$  channels, and  $H$  and  $W$  respectively the height and width of the video images, is divided into  $N_p$  non-overlapping patches and linearly projected into a sequence of  $d$ -dimensional context tokens  $\mathbf{z}^{ctx} \in \mathbb{R}^{T \times N_p \times d}$ . We use the Uniform frame sampling ViViT scheme Arnab et al. (2021) to embed the videos we have at hand, the frames being concatenated along the batch dimension, and the sample frequency being defined at 30 minutes. The historical time series  $\mathbf{t} \in \mathbb{R}^{T \times C_{ts}}$  are linearly projected into a sequence of  $d$ -dimensional time-series tokens  $\mathbf{z}^{ts} \in \mathbb{R}^{T \times d}$ . We augment the context tokens with RoPE, as presented in section 3.1, and a learnt positional encoding for the time-series tokens.
2. Masking: As a regularizing mechanism, we allow the model to mask a portion of the past time series and the video context before adding the positional encodings. During the training phase, a masking ratio  $m_{ctx}$  is randomly sampled from a uniform distribution  $U(0, 0.99)$  for the context, and the corresponding patches are masked accordingly. We also explored masking the time series to encourage the model to rely more on the context but in practice, no masking gave the best performance. We note that during inference, no masking is applied.
3. Encoding: We encode the time series and the past video context separately with two transformer architectures: a spatio-temporal encoder similar to a ViT for the video context, and a multi layer transformer for the input time series. More specifically, the context tokens alone and time series alone are passed through  $L$  separate transformer layers (we keep the same number of layers  $L$  for both encoders), including Multi-Head Self-Attention (MSA), LayerNorm (LN) and Multi-Layer Perceptron (MLP) blocks, so that for each layer  $l$ , we perform the following operations:
$$
\mathbf {y} _ {l} ^ {c t x} = \operatorname {M S A} (\ln \left(\mathbf {z} _ {l} ^ {c t x}\right)) + \mathbf {z} _ {l} ^ {c t x} \quad \mathbf {y} _ {l} ^ {t s} = \operatorname {M S A} (\ln \left(\mathbf {z} _ {l} ^ {t s}\right)) + \mathbf {z} _ {l} ^ {t s} \tag {1}
$$
$$
\mathbf {z} _ {l + 1} ^ {c t x} = \operatorname {M L P} \left(\ln \left(\mathbf {y} _ {l} ^ {c t x}\right)\right) + \mathbf {y} _ {l} ^ {c t x} \quad \mathbf {z} _ {l + 1} ^ {t s} = \operatorname {M L P} \left(\ln \left(\mathbf {y} _ {l} ^ {t s}\right)\right) + \mathbf {y} _ {l} ^ {t s} \tag {2}
$$
4. Mixing: We combine the resulting context and time-series latents, respectively  $\mathbf{z}_L^{ctx}$  and  $\mathbf{z}_L^{ts}$ , within  $L$  layers of a Transformer with Cross Attention (CA) Vaswani et al. (2017) (we keep the same number of layers in the entire encoder-mixer architecture). After adding ROPE, the two  $L$ -th layers are mixed with CA and passed through an MLP block. The output of each layer becomes a mixed latent which is in turn mixed with the context latent  $\mathbf{z}_L^{ctx}$  and again passed through a block of MLP. Formally, the following operations are performed respectively at the first layer (left equations) and on the remaining layers (right equations) of the CA:
$$
\mathbf {y} _ {1} ^ {m i x} = \operatorname {C A} \left(\operatorname {L N} \left(\mathbf {z} _ {L} ^ {c t x}, \mathbf {z} _ {L} ^ {t s}\right)\right) + \mathbf {z} _ {L} ^ {t s} \quad \mathbf {y} _ {l} ^ {m i x} = \operatorname {C A} \left(\operatorname {L N} \left(\mathbf {z} _ {L} ^ {c t x}, \mathbf {z} _ {l} ^ {m i x}\right)\right) + \mathbf {z} _ {l} ^ {m i x} \tag {3}
$$
$$
\mathbf {z} _ {2} ^ {m i x} = \operatorname {M L P} \left(\operatorname {L N} \left(\mathbf {y} _ {1} ^ {m i x}\right)\right) + \mathbf {y} _ {1} ^ {m i x} \quad \mathbf {z} _ {l + 1} ^ {m i x} = \operatorname {M L P} \left(\operatorname {L N} \left(\mathbf {y} _ {l} ^ {m i x}\right)\right) + \mathbf {y} _ {l} ^ {m i x} \tag {4}
$$
5. Decoding: The sequence of mixed tokens  $\mathbf{z}_L^{mix}$  returned by the layers of Cross Transformer is then passed through  $N$  layers of another Transformer as a decoder, before adding a learnt positional embedding to the token sequence. Each layer  $n$  of the Transformer is again formed by MSA, LN and MLP blocks:
$$
\mathbf {y} _ {n} = \operatorname {M S A} \left(\ln \left(\mathbf {z} _ {n}\right)\right) + \mathbf {z} _ {n} \tag {5}
$$
$$
\mathbf {z} _ {n + 1} = \mathrm {L N} (\operatorname {M L P} (\mathrm {L N} (\mathbf {y} _ {n})) + \mathbf {y} _ {n}) \tag {6}
$$
The output decoded sequence  $\mathbf{z}_N$  is passed through a final MLP head to output the final predicted future time series  $\mathbf{t}_{pred} \in \mathbb{R}^{T \times C_{ts}}$ .
3.3 Multi-Quantiles: Extracting prediction intervals
To obtain prediction intervals for each forecasted value, we propose a easy methodology which can be used for any deep learning forecasting model, here resulting in an alternative version of the
WV073
Figure 2: Stations and satellite data. Left: Location of the six meteorological stations considered in the study with the red border indicating the spatial extent (TAM is out of the considered window). Additionally, three of the eleven spectral channels under investigation are highlighted: IR_108, VIS_008 and WV_073 channels are infrared  $(10.8\mu)$ , visible  $(0.8\mu m)$  and water vapor  $(7.3\mu m)$  channels respectively. Right: Table summarizing the geographic coordinates and elevation of each station used in the paper.
Station	Latitude	Longitude
Cabauw (CAB) (Knap, 2007)	51°58′N	4°55′E
Cener (CNR) (Olano, 2022)	42°48′N	1°36′W
Izana (IZA) (Cuevas-Aguilló, 2014)	28°18′N	16°29′W
Palaiseau (PAL) (Haeffelin, 2014)	48°42′N	2°12′E
Payerne (PAY) (Vuilleumier, 2018)	46°48′N	6°56′E
Tamanrasset (TAM) (Baika, 2023)	22°47′N	5°31′E
CrossViViT architecture. In this modified version, the original MLP head is replaced with  $N_{\text{heads}}$  parallel MLPs, each dedicated to predicting a specific quantile of the distribution for each time step. To achieve this, we employ distinct quantile loss functions for each MLP head. By summing these quantile losses, we obtain a comprehensive Multi-Quantile loss, which serves as the training objective for the model. The quantile loss (Koenker and Hallock, 2001)  $L_{\alpha}(y,\hat{y})$  for the  $\alpha$  quantile is defined as:
$$
L _ {\alpha} (y, \hat {y}) = \max  \left\{\alpha (\hat {y} - y), (1 - \alpha) (y - \hat {y}) \right\}. \tag {7}
$$
The Multi-Quantile loss, aiming to learn multiple quantile predictions  $\hat{y}_{\alpha}$  for a chosen set of quantiles  $v_{A}$ , is then defined as:  $MQL(y,\hat{y}_{\alpha \in v_A}) = \sum_{\alpha \in v_A}L_\alpha (y,\hat{y}_\alpha)$ . The selection of quantile heads  $v_{\alpha}$  is a crucial hyperparameter that determines the density of the output distribution generated by the model. To achieve a  $96\%$  prediction interval while maintaining a sufficiently dense distribution, we set the list of quantiles as  $v_{A} = [0.02, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 0.98]$ . It is worth noting that, it is possible to assign different weights to the quantiles to guide the learning process. This approach can be beneficial in scenarios where the task requires a preference for overestimation or underestimation. However, in this study, our primary objective was to provide a prediction interval, leading to the conservative choice of quantile distribution and uniformly weighing the  $L_{\alpha}$ 's.
4 Dataset
This section provides a comprehensive description of the dataset designed for this study and shared publicly, including all the applied pre-processing steps.
4.1 Time series