Title: Improving *day-ahead* Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context
Venue: NeurIPS 2023
Accepted: yes
Keywords: Time series forecasting

multi-modal learning

solar irradiance

context-enriched learning
Abstract: Solar power harbors immense potential in mitigating climate change by substantially reducing CO$_{2}$ emissions. Nonetheless, the inherent variability of solar irradiance poses a significant challenge for seamlessly integrating solar power into the electrical grid. While the majority of prior research has centered on employing purely time series-based methodologies for solar forecasting, only a limited number of studies have taken into account factors such as cloud cover or the surrounding physical context.
In this paper, we put forth a deep learning architecture designed to harness spatio-temporal context using satellite data, to attain highly accurate day-ahead time-series forecasting for any given station, with a particular emphasis on forecasting Global Horizontal Irradiance (GHI). We also suggest a methodology to extract a distribution for each time step prediction, which can serve as a very valuable measure of uncertainty attached to the forecast. When evaluating models, we propose a testing scheme in which we separate particularly difficult examples from easy ones, in order to capture the model performances in crucial situations, which in the case of this study are the days suffering from varying cloudy conditions. Furthermore, we present a new multi-modal dataset gathering satellite imagery over a large zone and time series for solar irradiance and other related physical variables from multiple geographically diverse solar stations. Our approach exhibits robust performance in solar irradiance forecasting, including zero-shot generalization tests at unobserved solar stations, and holds great promise in promoting the effective integration of solar power into the grid.

Improving day-ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context

Abstract

Solar power harbors immense potential in mitigating climate change by substantially reducing CO 2 emissions. Nonetheless, the inherent variability of solar irradiance poses a significant challenge for seamlessly integrating solar power into the electrical grid. While the majority of prior research has centered on employing purely time series-based methodologies for solar forecasting, only a limited number of studies have taken into account factors such as cloud cover or the surrounding physical context. In this paper, we put forth a deep learning architecture designed to harness spatio-temporal context using satellite data, to attain highly accurate day-ahead time-series forecasting for any given station, with a particular emphasis on forecasting Global Horizontal Irradiance (GHI). We also suggest a methodology to extract a distribution for each time step prediction, which can serve as a very valuable measure of uncertainty attached to the forecast. When evaluating models, we propose a testing scheme in which we separate particularly difficult examples from easy ones, in order to capture the model performances in crucial situations, which in the case of this study are the days suffering from varying cloudy conditions. Furthermore, we present a new multi-modal dataset gathering satellite imagery over a large zone and time series for solar irradiance and other related physical variables from multiple geographically diverse solar stations. Our approach exhibits robust performance in solar irradiance forecasting, including zero-shot generalization tests at unobserved solar stations, and holds great promise in promoting the effective integration of solar power into the grid.

1 Introduction

Solar power has become an increasingly important source of renewable energy in recent years, with the potential to help mitigate the effects of climate change by reducing greenhouse gas emissions (Doblas-Reyes et al., 2021; IEA, 2021). However, the variability of solar irradiance - the amount of solar radiation that reaches the earth's surface - presents a challenge for integrating solar power into the grid. Accurate forecasting of solar irradiance can assist grid operators in dealing with the variability of solar power, leading to a more efficient and dependable integration of solar power into the grid. As a result, this can help to reduce the requirement for costly and environmentally damaging backup power sources.

∗

Equal contribution.

Corresponding

authors:

Contributions: Our focus is on the application of machine learning to time-series forecasting, with a particular emphasis on utilizing multi-modal spatio-temporal data that includes physical, weather, and remotely sensed variables. The main contributions of this paper can be summarized as follows:

We present a deep learning architecture called CrossViViT, designed to leverage spatiotemporal context (such as satellite data) in order to achieve highly accurate medium-term (1 day horizon) time-series forecasting at any given station. This paper focuses specifically on forecasting Global Horizontal Irradiance (GHI).

We present a Multi-Quantile version of the model which allows to extract uncertainty estimation attached to each prediction. This methodology, although applied here in our context, should be applicable to any forecasting task and framework.

We present a multi-modal dataset that combines satellite imagery with solar irradiance and other physical variables. The dataset covers a period of 14 years, from 2008 to 2022, and includes data from six geographically diverse solar stations. This dataset is unique in its combination of diverse variables and long time span, and is intended to facilitate the development and evaluation of new multi-modal forecasting models for solar irradiance.

We propose a forecasting testing scheme based on multiple time splits of the test data, separating particularly difficult examples from easy ones, therefore allowing to capture the models' performances in problematic examples.

We experimentally show that the proposed approach can generalize to a new station not seen during training in a zero-shot generalization forecasting setting.

2 Related works

Machine Learning for time-series forecasting Deep learning approaches have gained popularity for time-series forecasting in recent years due to their ability to model complex nonlinear relationships and capture temporal dependencies. These approaches have demonstrated superior performance compared to traditional statistical methods, motivating further research in this area. In a recent survey (Wen et al., 2022), it was found that transformers, renowned for their success in natural language processing and computer vision, were also effective for time-series analysis. The authors discussed the strengths and limitations of transformers and compared the structure and performance of recent transformer-based architectures on a benchmark weather dataset (Zhou et al., 2021). The particular case of solar irradiance forecasting represents an interesting application for time-series models (Wang et al., 2019a; Narvaez et al., 2021; Alzahrani et al., 2017). One recent study developed a multi-step attention-based model for solar irradiance forecasting that generates deterministic predictions and quantile predictions as well (Sharda et al., 2021). In a similar perspective, Jønler et al. (2023) developed a probabilistic solar irradiance transformer that incorporates gated recurrent units and temporal convolution networks, demonstrating strong performance for short-term horizons.

Context mixing / Multimodal learning for time-series forecasting Previous studies highlight the potential of time-series methods for solar irradiance forecasting, emphasizing the significance of shortterm horizons in solar energy management. However, day-ahead forecasting remains challenging due to the influence of cloud cover on surface irradiance (Bone et al., 2018; Si et al., 2021), a problem which we aim to address in this paper. Thus, it is crucial to account for cloud effects in solar irradiance forecasting regardless of the chosen method. For instance, Zhang et al. (2023) investigated the impact of cloud movement on irradiance prediction and proposed an approach to automatically learn the relationship between sky image appearance and solar irradiance. A concurrent work (Liu et al., 2023) proposed a multimodal-learning framework for ultra-short-term (10min-ahead) solar irradiance forecasting. They used Informer (Zhou et al., 2021) to encode historical time-series data, then utilized Vision Transformer (Dosovitskiy et al., 2020) to handle sky images. Finally, they employed cross-attention to couple the two modalities. The studies discussed above highlight the potential of incorporating external data sources, such as sky images and satellite images, in combination with time-series approaches to improve the accuracy of solar forecasting.

Operator Learning Utilizing available satellite imagery to forecast GHI over a region presents limitations as it may not capture clouds that exist at a resolution beyond that of the satellite data. To ensure accurate forecasting of quantities of interest, the ability to query the model at any possible resolution and any point within the domain becomes crucial. Recent advancements have witnessed

the rise of algorithms focusing on learning operators capable of mapping across functional spaces, with a focus on solving partial differential equations (PDE) (Lu et al., 2019; Li et al., 2021; Kovachki et al., 2021; Li et al., 2020). These operators can effectively map initial conditions to PDE solutions, making it possible to query the learned solution theoretically anywhere within its domain. Fourier Layers, developed by Li et al. (2021), enable zero-shot prediction on both uniform and non-uniform grids with learnable deformations (Li et al., 2022). Pathak et al. (2022) replace attention in ViT (Dosovitskiy et al., 2020) with Fourier layer mixing for competitive weather forecasting results with faster inference. MeshFreeflowNet (Jiang et al., 2020) learns high-resolution frames from corresponding lower resolution ones by querying the model at any point of the domain for irregular grids. Similarly, Boussif et al. (2022) employ message passing with a low-resolution graph for zeroshot super-resolution PDE learning. Additionally, message passing neural PDE solvers (Brandstetter et al., 2022) exhibit spatio-temporal multi-scale capabilities benefiting from long-expressive memory (Equer et al., 2023; Rusch et al., 2022). We note that while these approaches were developed for PDEs in mind, they can still be used for weather-related applications.

Uncertainty estimation When performing solar irradiance forecasting, deterministic forecasts are not sufficient to characterize the inherent variance and uncertainty in solar irradiance data. Probabilistic forecasts, providing uncertainty information, are crucial for energy system management (Wang et al., 2019b). (Doubleday et al., 2020) and (Yagli et al., 2020) benchmark solar forecasting methods, emphasizing calibration and sharpness in prediction intervals. Specifically in short term solar irradiance forecasting, Zelikman et al. (2020) delves into post-hoc calibration for better predictions. Turkoglu et al. (2022) introduces FiLM-Ensemble, balancing predictive accuracy and calibration in uncertainty estimation. The deep ensembles approach by Lakshminarayanan et al. (2016) aggregates neural network predictions, capturing both data noise and model uncertainties. Few studies tackle uncertainty evaluation in a regression setting, rather than in classification one, which consists in the estimation of prediction intervals. Sønderby et al. (2020), performing precipitation forecasting, suggests an easy solution: the output is separated in 512 bins and the model predicts the precipitation rate (probability) in each of the bins, resulting ultimately in a distribution. Other methodologies include bootstrapping and ensembling methods, drawing a distribution out of multiple predictions from submodels, following the spirit of Quantile Regression Forests (Meinshausen and Ridgeway, 2006). The evolving landscape of solar irradiance forecasting underscores the importance of a calibrated, comprehensive, and robust probabilistic approach to address inherent uncertainties.

3 Methodology

We develop a framework for solar irradiance time-series forecasting, incorporating spatio-temporal context alongside historical time-series data from multiple stations. This framework is inspired by recent advancements in video transformer models (Arnab et al., 2021; Feichtenhofer et al., 2022) and multi-modal models that leverage diverse data sources such as images and time series (Liu et al., 2023). To establish the foundation for our framework, we provide a brief overview of the Rotary Positional Embedding (Su et al., 2021). Subsequently, we present the proposed architecture, CrossViViT, in detail, outlining its key components and design principles. Details about the ViT (Dosovitskiy et al., 2020) and ViViT architectures can be found in the Appendix.

3.1 Rotary Positional Embedding

As we are dealing with permutation-variant data, assigning positions to patches before using attention is necessary. A common approach is to use additive positional embedding, which considers the absolute positions of the patches added into the input sequence. However, for our case, it is more appropriate to have dot-product attention depend on the relative "distance" between the patches. This is because the station should only be concerned with the distance and direction of nearby clouds, and therefore, a relative positional embedding is more sensible.

To this end we make use of RoPE (Su et al., 2021) to mix the station time series and the context. The station and each patch in the context are assigned their normalized latitude and longitude coordinates in [ -1 , 1] which are used as positions for RoPE. Details about the formulation used are left to the Appendix.

3.2 Cross Video Vision Transformer for time-series forecasting (CrossViViT)

CrossViViT. Our approach aims to integrate the available historical time-series data with video clips of spatio-temporal physical context to enhance the prediction of future time series for solar irradiance. The overall methodology, depicted in Figure 1, can be summarized as follows:

Tokenizing : The video context V ∈ R T × C ctx × H × W , with T frames for each of the C ctx channels, and H and W respectively the height and width of the video images, is divided into N p non-overlapping patches and linearly projected into a sequence of d -dimensional context tokens z ctx ∈ R T × N p × d . We use the Uniform frame sampling ViViT scheme Arnab et al. (2021) to embed the videos we have at hand, the frames being concatenated along the batch dimension, and the sample frequency being defined at 30 minutes. The historical time series t ∈ R T × C ts are linearly projected into a sequence of d -dimensional time-series tokens z ts ∈ R T × d . We augment the context tokens with RoPE, as presented in section 3.1, and a learnt positional encoding for the time-series tokens.

Masking : As a regularizing mechanism, we allow the model to mask a portion of the past time series and the video context before adding the positional encodings. During the training phase, a masking ratio m ctx is randomly sampled from a uniform distribution U (0 , 0 . 99) for the context, and the corresponding patches are masked accordingly. We also explored masking the time series to encourage the model to rely more on the context but in practice, no masking gave the best performance. We note that during inference, no masking is applied.

Encoding: We encode the time series and the past video context separately with two transformer architectures: a spatio-temporal encoder similar to a ViT for the video context, and a multi layer transformer for the input time series. More specifically, the context tokens alone and time series alone are passed through L separate transformer layers (we keep the same number of layers L for both encoders), including Multi-Head Self-Attention (MSA), LayerNorm (LN) and Multi-Layer Perceptron (MLP) blocks, so that for each layer l , we perform the following operations:

Mixing: We combine the resulting context and time-series latents, respectively z ctx L and z ts L , within L layers of a Transformer with Cross Attention (CA) Vaswani et al. (2017) (we keep the same number of layers in the entire encoder-mixer architecture). After adding ROPE, the two L -th layers are mixed with CA and passed through an MLP block. The output of each layer becomes a mixed latent which is in turn mixed with the context latent z ctx L and again passed through a block of MLP. Formally, the following operations are performed respectively at the first layer (left equations) and on the remaining layers (right equations) of the CA:

Decoding: The sequence of mixed tokens z mix L returned by the layers of Cross Transformer is then passed through N layers of another Transformer as a decoder, before adding a learnt positional embedding to the token sequence. Each layer n of the Transformer is again formed by MSA, LN and MLP blocks:

The output decoded sequence z N is passed through a final MLP head to output the final predicted future time series t pred ∈ R T × C ts .

3.3 Multi-Quantiles: Extracting prediction intervals

To obtain prediction intervals for each forecasted value, we propose a easy methodology which can be used for any deep learning forecasting model, here resulting in an alternative version of the

Context

IR_108

XZA

VIS008

WV_073

CrossViViT architecture. In this modified version, the original MLP head is replaced with N heads parallel MLPs, each dedicated to predicting a specific quantile of the distribution for each time step. To achieve this, we employ distinct quantile loss functions for each MLP head. By summing these quantile losses, we obtain a comprehensive Multi-Quantile loss, which serves as the training objective for the model. The quantile loss (Koenker and Hallock, 2001) L α ( y, ˆ y ) for the α quantile is defined as:

The Multi-Quantile loss, aiming to learn multiple quantile predictions ˆ y α for a chosen set of quantiles v A , is then defined as: MQL ( y, ˆ y α ∈ v A ) = ∑ α ∈ v A L α ( y, ˆ y α ) . The selection of quantile heads v α is a crucial hyperparameter that determines the density of the output distribution generated by the model. To achieve a 96% prediction interval while maintaining a sufficiently dense distribution, we set the list of quantiles as v A = [0 . 02 , 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 9 , 0 . 98] . It is worth noting that, it is possible to assign different weights to the quantiles to guide the learning process. This approach can be beneficial in scenarios where the task requires a preference for overestimation or underestimation. However, in this study, our primary objective was to provide a prediction interval, leading to the conservative choice of quantile distribution and uniformly weighing the L α 's.

4 Dataset

This section provides a comprehensive description of the dataset designed for this study and shared publicly, including all the applied pre-processing steps.

4.1 Time series

The time-series measurements were obtained from Baseline Surface Radiation Network datasets (Driemel et al., 2018). The experiments in this paper use data from six locations (Figure 2), collected at a 30-minute resolution over a 15-year period (2008-2022). The data captures diverse patterns, ranging from consistent irradiance levels under clear skies to fluctuations caused by intermittent clouds affecting surface solar radiation. The data contains measurements of the pressure in the station, clear sky components, Direct Normal Irradiance (DNI), and Diffuse Horizontal Irradiance (DHI).

The GHI component was computed using the formula: GHI = DNI × cos z + DHI where z is the zenith angle of the sun obtained from the pvlib python library (Holmgren et al., 2020). The Ineichen model (Ineichen, 2016) available through the same library is utilized to obtain the clear sky components. The 2008-2016 data is used for training while subsets of the 2017-2022 data and stations are used for validation and test performance evaluations. In this study, the models are trained and evaluated to forecast the GHI component for a 24-hour period ahead, based on a history of 24-hour measurements.

4.2 Satellite images

In this study, we utilize the EUMETSAT Rapid Scan Service (RSS) dataset (Holmlund, 2003), which spans a period of 15 years from 2008 to 2022, with an original resolution of 5 minutes, later aligned with the time series data. Our analysis focuses on the non-High Resolution Visible (non-HRV) channels, which encompass 11 spectral channels with a spatial resolution of 6-9km and provide comprehensive coverage of the upper third of the Earth, with a particular emphasis on Europe. These channels, including Infrared and Water vapor, offer valuable information for our investigation. To facilitate our analysis, we reprojected the original geostationary projection data onto the World Geodetic System 1984 (WGS 84) coordinate system (Jacob et al., 2022). The region of interest is depicted in Figure 2. Note that one of the 6 stations, TAM, lies slightly outside the region we consider, and therefore represents an out-of-distribution station in term of the context we use.

To augment the contextual information, we computed the optical flow for each channel using the TVL1 algorithm (Sánchez Pérez et al., 2013) from the OpenCV package (Bradski, 2000). The optical flow represents the "velocity" of pixels between consecutive frames, which in our case corresponds to the motion of clouds. Furthermore, we included the elevation map as an additional channel in our dataset. The pre-processed satellite data originally had a resolution of 512 2 , but for computational efficiency, we downscale it to 64 2 .

5 Experiments and Results

In this section, we outline the baselines utilized for comparison alongside the suggested architecture. We describe the experimental setup and present the results, benchmarking our framework against state-of-the-art forecasting models across various test configurations. Additionally, we employ a split methodology to assess model performance in challenging prediction scenarios, which hold significant implications for downstream tasks associated with solar irradiance estimation.

The models are trained using a dataset spanning a period of 9 years, from 2008 to 2016, encompassing the stations IZA, CNR, and PAL. Validation is performed on a separate dataset covering 3 years, from 2017 to 2019, for the PAY station. The TAM and CAB stations serve as the test dataset, consisting of 3 years from 2020 to 2022 for CAB, and from 2017 to 2019 for TAM. Each model takes a historical

input of 24 hours and predicts the GHI for the subsequent 24 hours, employing a sliding window approach. Details about the implementation and hyperparameter tuning can be found in the Appendix.

5.1 Baselines

We conduct a comprehensive comparison between our approach and several state-of-the-art deep learning architectures specifically designed for forecasting tasks. These architectures are explicitly mentioned in the results tables, such as Table 1. Additionally, we propose dummy baselines that are tailored to solar irradiance forecasting, namely:

Persistence : This baseline relies solely on the past day's time-series data, considering it as the prediction for future values.

Clear Sky baseline : This baseline uses the computable clear sky components of solar irradiance (Ineichen model (Ineichen, 2016)), which represent the total amount of irradiance that would reach the station in the absence of clouds.

Fourier approximations : We compute Fourier approximations over the previous day's time series, and apply a low-pass filter to keep a limited number of modes. We consider 3 baselines: Fourier 3 , Fourier 4 , and Fourier 5 , corresponding to approximations with 3, 4, and 5 modes, respectively.

5.2 'Hard' vs. 'Easy' forecasting scenarios

To assess the capability of the suggested model in capturing cloud-induced variability in GHI forecasts, we perform evaluations on test stations using various time splits, aiming to identify its strengths and limitations in comparison to previous approaches. This analysis allows us to pinpoint the specific scenarios where CrossViViT excels, as well as the areas where it falls short. Furthermore, it provides insights into the comparative strengths and weaknesses of previous approaches.

Given the favorable performance of the Persistence baseline when the GHI values exhibit similarity between consecutive days, we propose a time split approach that categorizes examples as either "Easy" or "Hard" based on the extent of GHI variation. The "Easy" examples entail minimal changes in GHI across consecutive days (the Persistence baseline works well), while the "Hard" examples involve significant variations (the Persistence baseline fails). To quantify the similarity, we employ a measure based on the ratio of the area under the GHI curve for the two days. In order to assign equal importance to ratios such as 0.5 (indicating GHI half that of the previous day) and 2 (indicating

GHI double that of the previous day), we utilize the measure r = ∣ ∣ ∣ log y y prev ∣ ∣ ∣ . Here, y represents the GHI over a 24-hour period, and y prev represents the GHI over the previous 24 hours. Accordingly, we categorize cases as "Easy" when r < ∣ ∣ log ( 2 3 )∣ ∣ , and "Hard" otherwise.

5.3 Performance on stations and years outside the training distribution

Table 1 presents a comprehensive comparison of the proposed CrossViViT approach with state-ofthe-art timeseries models and dummy baselines. The evaluation is conducted on the test stations TAM and CAB, covering the periods 2017-2019 and 2020-2022, respectively. It is important to note that due to the unavailability of data for TAM during the 2020-2022 period, we perform the evaluation using the 2017-2019 data instead.

During the 2017-2019 period, CrossViViT achieves the lowest MAE compared to the time-series models on the TAM station. However, it is important to note that the persistence baseline still outperforms our approach. This performance disparity can be attributed to the characteristics of the TAMstation, which is situated in a desert region characterized by predominantly clear and sunny days. As we incorporate cloud information, it may occasionally underestimate the GHI in such clear-sky conditions. Furthermore, the training dataset consists of data from only one "sunny" station located in the Canary Islands (IZA), limiting the availability of examples to effectively learn clear-sky patterns. Based on these results, one can see that for stations characterized by low irradiance intermittency, a combination of persistence and clear-sky models might be sufficient. For the 2020-2022 period on the CAB station, CrossViViT outperforms all baselines across different time splits. This notable improvement can be attributed to the specific meteorological conditions of the CAB station, which experiences a higher frequency of cloudy days. This aligns with the primary focus of our research, which aims to accurately forecast GHI under cloudier conditions. Predictions visualisations can be

(a)

(b)

(c)

(d)

MAE: 23.52

MAE: 5.74

median MAE:43.35

median MAE:5.44

500

100

Prediction

Median Pred.

Ground truth

200

Ground Truth

0

0

0

0亡

-24h

-12h

oh

12h

24h

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

MAE: 3.34

MAE: 36.94

median MAE: 7.18

median MAE: 83.5

200

500

500

0

0

0-

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

MAE: 38.2

MAE: 9.89

Amplitude

103

500

100

Prediction

103

102

Ground truth

101

0

10

-24h

-12h

oh

12h

24h

-24h

-12h

oh

12h

24h

CrossFormer

CrossViviT——Ground truth

104

MAE:15.85

MAE:48.44

Amplitude

200

103

500

IH9

102

0-

0

5

10

15

20

25

0

5

10

15

20

25

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

Modes

Modes

seen in Figure 3, along with the comparison of the fourier spectra of our prediction, the ground truth and a strong baseline, CrossFormer.

5.4 Zero-shot forecasting on unseen stations and in-distribution years

To evaluate the zero-shot capabilities of CrossViViT in terms of performance on unseen stations, we maintain the same time period as the training data , thereby emphasizing the spatial dimension of the analysis. The radar plots presented in Figure 4 demonstrate that our approach consistently outperforms the baselines across all splits for the CAB and PAY stations, which are characterized by higher cloud cover. However, it is noteworthy that Persistence remains competitive for the TAM station, particularly in the "easy" splits where irradiance variations are minimal throughout the day. This observation further underscores the efficacy of CrossViViT in accurately accounting for cloud conditions, particularly when examining the performance metrics of the "Hard" split.

5.5 Multi-Quantile results

Table 1 also shows results of the Multi-Quantile CrossViViT, including the MAE of the median prediction (the 0.5 quantile), along with the test confidence p t obtained for the prediction interval: the probability for the ground-truth to be included within the interval, for each time step, averaged across the entire dataset considered. As for the other models, the evaluation is conducted on the test stations TAM and CAB, covering the periods 2017-2019 and 2020-2022, respectively. It is important to note that for this Multi-Quantile version, the goal is not to provide the best prediction from the median but rather to provide confident prediction intervals, with a high p t , ideally close to 96%, which we theoretically should reach using 0.02 and 0.98 extreme quantiles. We include a small version of the Multi-Quantile model, and a large one, matching the number of parameters used for our Cross ViViT model. The small model provides the best performance.

The prediction intervals achieved a high level of confidence, surpassing 0.9, for the unseen CAB station. However, for the TAM station, the harsh environmental conditions of the desert posed a

(a)

(b)

(c)

MAE

MAE

MAE

49.0

50.0

49.0

RMSE hard

68.0

RMSE

RMSE hard

55.0

RMSE

RMSE hard

53.0

RMSE

178.0

87.0

106.6

113.0

60.0

100.6

137.0

57.0

109.0

196.0

142

126.0

110.

149.0

1180

2140

232.0214.0

106.0

178.0

139.0

157

.0

65.0

130.0

120.0

161.0

173.0

61.0

136.0

127.0

97.0

104.0

111.0

128.0106.0

214.087.0

178.0

68.0

49.0

56.0

62.0

68.0

74.0

106.0

113.0

51.0

48.0

67.0

74.0

81.0

86.0

95.0

100L0

56.0

52.0

48.0

440

MAE hard

142.0

MAE easy

MAE hard

99.0

MAE easy

MAE hard

90.0

MAE easy

106.0

85.0

Persistence

CrossViViT

RMSE easy

Persistence

CrossViViT

RMSE easy

Persistence

CrossViViT

RMSE easy

CrossFormer

CrossFormer

CrossFormer

challenge for reliable estimation of prediction intervals, due to the abundance of clear sky components which is not seen during training. Although the median prediction results were comparatively inferior to those of a baseline method, it demonstrates consistent patterns in its variation across easy and difficult cases. Furthermore, the test confidence of the proposed method remains relatively constant across different splits, for both stations. Prediction visualisations can be seen on Figure 3.

5.6 Ablating the Rotary Positional Embedding

In order to assess the influence of RoPE, we ablate this design choice against a learned positional encoding which was the standard encoding used in ViViT. As Table 1 shows, RoPE contributes to the good performance in OOD stations and OOD years. Moreover, in the hard cases of the CAB station, RoPE outperforms the learnable positional encoding showcasing its importance in capturing cloud-induced variations.

6 Conclusion, Limitations, and Future Work

We present CrossViViT, an architecture for accurate day-ahead time-series forecasting of solar irradiance using the spatio-temporal context derived from satellite data. We suggest a testing scheme that captures model performance in crucial situations, such as days with varying cloudy conditions, and we enable the extraction of a distribution for each time step prediction. We also introduce a new multi-modal dataset that provides a diverse collection of solar irradiance and related physical variables from multiple geographically diverse solar stations. CrossViViT exhibits robust performance in solar irradiance forecasting, including zero-shot tests at unobserved solar stations during unobserved years, and holds great promise in promoting the effective integration of solar power into the grid.

However, there are some limitations to our study. Firstly, we would benefit to include more test years and stations to further validate the effectiveness of our approach. It would also be interesting to explore different past and future horizons to further evaluate the robustness of our model across different prediction and context horizons. The training time remains our key limitation since it takes close to 5 hours per epoch on a single GPU (see Appendix), but we plan on optimizing the architecture further in future work. Lastly, we plan to investigate the use of a cropping methodology on the context as a regularization method. Despite these limitations, our study clearly highlights the importance of incorporating spatio-temporal context in solar irradiance forecasting and demonstrates the potential of deep learning in addressing the challenges associated with variability in solar irradiance. Given the promising results observed in our current study, we intend to investigate the applicability of CrossViViT to other physical variables that are significantly influenced by their surrounding context.

Acknowledgments and Disclosure of Funding

This research has been funded by the Government of Quebec, with additional computational resources supplied by Mila (mila.quebec). It is conducted as part of an international partnership project between Mila, Moroccan agency for sustainable energy (Masen), l'UQAR, Polytechnique Montreal, l'Ecole Mohammadia d'Ingénieurs, and l'Institut de Valorisation des Données (IVADO). Our special thanks to MASEN R&D team for their precious collaboration. We extend our sincere appreciation to Jacob Bieker from Open Climate Fix, whose facilitation in accessing EUMETSAT data, coupled with his invaluable assistance regarding data availability, markedly enriched our study.

References

(2021). Net Zero by 2050: A Roadmap for the Global Energy Sector . OECD.

Alzahrani, A., Shamsi, P., Dagli, C., and Ferdowsi, M. (2017). Solar Irradiance Forecasting Using Deep Neural Networks. Procedia Computer Science , 114:304-313.

Arnab, A., Dehghani, M., Heigold, G., Sun, C., Luˇ ci´ c, M., and Schmid, C. (2021). Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 6836-6846.

Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450 .

Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings .

Baika, S. (2023). Meteorological synoptical observations from station Tamanrasset (2022-12). Backup Publisher: National Meteorological Office of Algeria Type: data set.

Bone, V., Pidgeon, J., Kearney, M., and Veeraragavan, A. (2018). Intra-hour direct normal irradiance forecasting through adaptive clear-sky modelling and cloud tracking. Solar Energy , 159:852-867.

Boussif, O., Bengio, Y., Benabbou, L., and Assouline, D. (2022). Magnet: Mesh agnostic neural pde solver. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems , volume 35, pages 31972-31985. Curran Associates, Inc.

Bouthillier, X., Tsirigotis, C., Corneau-Tremblay, F., Schweizer, T., Dong, L., Delaunay, P., Normandin, F., Bronzi, M., Suhubdy, D., Askari, R., Noukhovitch, M., Xue, C., Ortiz-Gagné, S., Breuleux, O., Bergeron, A., Bilaniuk, O., Bocco, S., Bertrand, H., Alain, G., Serdyuk, D., Henderson, P., Lamblin, P., and Beckham, C. (2022). Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization.

Bradski, G. (2000). The OpenCV Library. Dr. Dobb's Journal of Software Tools .

Brandstetter, J., Worrall, D. E., and Welling, M. (2022). Message passing neural pde solvers. International Conference On Learning Representations .

Cuevas-Agulló, E. (2014). Radiosonde measurements from station Izana (2014-08). Backup Publisher: Izaña Atmospheric Research Center, Meteorological State Agency of Spain Type: data set.

Doblas-Reyes, F., Sörensson, A., Almazroui, M., Dosio, A., Gutowski, W., Haarsma, R., Hamdi, R., Hewitson, B., Kwon, W.-T., Lamptey, B., Maraun, D., Stephenson, T., Takayabu, I., Terray, L., Turner, A., and Zuo, Z. (2021). Linking Global to Regional Climate Change Supplementary Material .

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .

Doubleday, K., Van Scyoc Hernandez, V., and Hodge, B.-M. (2020). Benchmark probabilistic solar forecasts: Characteristics and recommendations. Solar Energy , 206:52-67.

Driemel, A., Augustine, J., Behrens, K., Colle, S., Cox, C., Cuevas-Agulló, E., Denn, F. M., Duprat, T., Fukuda, M., Grobe, H., Haeffelin, M., Hodges, G., Hyett, N., Ijima, O., Kallis, A., Knap, W., Kustov, V., Long, C. N., Longenecker, D., Lupi, A., Maturilli, M., Mimouni, M., Ntsangwane, L., Ogihara, H., Olano, X., Olefs, M., Omori, M., Passamani, L., Pereira, E. B., Schmithüsen, H., Schumacher, S., Sieger, R., Tamlyn, J., Vogt, R., Vuilleumier, L., Xia, X., Ohmura, A., and König-Langlo, G. (2018). Baseline surface radiation network (bsrn): structure and data description (1992-2017). Earth System Science Data , 10(3):1491-1501.

Equer, L., Rusch, T. K., and Mishra, S. (2023). Multi-scale message passing neural pde solvers. ARXIV.ORG .

Feichtenhofer, C., Li, Y., He, K., et al. (2022). Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems , 35:35946-35958.

Haeffelin, M. (2014). Basic measurements of radiation at station Palaiseau (2009-02). Backup Publisher: Laboratoire de Météorologie Dynamique du C.N.R.S., Ecole Polytechnique Type: data set.

Hendrycks, D. and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 .

Holmgren, W., Calama-Consulting, Hansen, C., Mikofski, M., Lorenzo, T., Krien, U., bmu, Anderson, K., Stark, C., DaCoEx, Driesse, A., konstant_t, mayudong, de León Peque, M. S., Heliolytics, Miller, E., Anoma, M. A., Boeman, L., jforbess, tylunel, Guo, V., Morgan, A., Stein, J., Leroy, C., R, A. M., JPalakapillyKWH, Dollinger, J., Anderson, K., MLEEFS, and Dowson, O. (2020). pvlib/pvlib-python: v0.8.0.

Holmlund, K. (2003). Eumetsat current and future plans on product generation and dissemination.

Ineichen, P. (2016). Validation of models that estimate the clear sky global and beam solar irradiance. Solar Energy , 132:332-344.

Jacob, B., Peter, D., and Jack, K. (2022). Ocf datapipes. https://github.com/openclimatefix/ ocf_datapipes .

Jiang, C. M., Esmaeilzadeh, S., Azizzadenesheli, K., Kashinath, K., Mustafa, M., Tchelepi, H. A., Marcus, P., Prabhat, and Anandkumar, A. (2020). Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis , SC '20. IEEE Press.

Jønler, J. F., Brunø Lottrup, F., Berg, B., Zhang, D., and Chen, K. (2023). Probabilistic forecasts of global horizontal irradiance for solar systems. IEEE Sensors Letters , 7(1):1-4.

Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer. In International Conference on Learning Representations .

Knap, W. (2007). Horizon at station Cabauw. Backup Publisher: Koninklijk Nederlands Meteorologisch Instituut, De Bilt Type: data set.

Koenker, R. and Hallock, K. F. (2001). Quantile regression. Journal of economic perspectives , 15(4):143-156.

Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2021). Neural operator: Learning maps between function spaces.

Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. NIPS .

Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. (2020). Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 6755-6766. Curran Associates, Inc.

Li, Z.-Y., Huang, D., Liu, B., and Anandkumar, A. (2022). Fourier neural operator with learned deformations for pdes on general geometries. ARXIV.ORG .

Li, Z.-Y., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2021). Fourier neural operator for parametric partial differential equations. ArXiv , abs/2010.08895.

Liu, J., Zang, H., Cheng, L., Ding, T., Wei, Z., and Sun, G. (2023). A Transformer-based multimodallearning framework using sky images for ultra-short-term solar irradiance forecasting. Applied Energy , 342:121160.

Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.

Lu, L., Jin, P., and Karniadakis, G. (2019). Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. ARXIV.ORG .

Meinshausen, N. and Ridgeway, G. (2006). Quantile regression forests. Journal of machine learning research , 7(6).

Narvaez, G., Giraldo, L. F., Bressan, M., and Pantoja, A. (2021). Machine learning for site-adaptation and solar radiation forecasting. Renewable Energy , 167:333-342.

Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with transformers.

Nielsen, A. H., Iosifidis, A., and Karstoft, H. (2021). Irradiancenet: Spatiotemporal deep learning model for satellite-derived solar irradiance short-term forecasting. Solar Energy , 228:659-669.

Olano, X. (2022). Basic measurements of radiation at station Cener (2022-04). Backup Publisher: National Renewable Energy Centre Type: data set.

Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z.-Y., Azizzadenesheli, K., Hassanzadeh, P., Kashinath, K., and Anandkumar, A. (2022). Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. ARXIV.ORG .

Rusch, T. K., Mishra, S., Erichson, N. B., and Mahoney, M. W. (2022). Long expressive memory for sequence modeling. International Conference On Learning Representations .

Sharda, S., Singh, M., and Sharma, K. (2021). Rsam: Robust self-attention based multi-horizon model for solar irradiance forecasting. IEEE Transactions on Sustainable Energy , 12(2):1394-1405.

Si, Z., Yu, Y ., Yang, M., and Li, P. (2021). Hybrid Solar Forecasting Method Using Satellite Visible Images and Modified Convolutional Neural Networks. IEEE Transactions on Industry Applications , 57(1):5-16.

Sønderby, C. K., Espeholt, L., Heek, J., Dehghani, M., Oliver, A., Salimans, T., Agrawal, S., Hickey, J., and Kalchbrenner, N. (2020). Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003.12140 .

Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. ARXIV.ORG .

Sánchez Pérez, J., Meinhardt-Llopis, E., and Facciolo, G. (2013). TV-L1 Optical Flow Estimation. Image Processing On Line , 3:137-150. https://doi.org/10.5201/ipol.2013.26 .

Turkoglu, M. O., Becker, A., Gündüz, H., Rezaei, M., Bischl, B., Daudt, R. C., D'aronco, S., Wegner, J. D., and Schindler, K. (2022). Film-ensemble: Probabilistic deep learning via feature-wise linear modulation. Neural Information Processing Systems .

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems , 30.

Vuilleumier, L. (2018). Meteorological synoptical observations from station Payerne (2015-01). Backup Publisher: Swiss Meteorological Agency, Payerne Type: data set.

Wang, H., Lei, Z., Zhang, X., Zhou, B., and Peng, J. (2019a). A review of deep learning for renewable energy forecasting. Energy Conversion and Management , 198:111799.

Wang, H., Lei, Z., Zhang, X., Zhou, B., and Peng, J. (2019b). A review of deep learning for renewable energy forecasting. Energy Conversion and Management , 198:111799.

Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, L. (2022). Transformers in time series: A survey. arXiv preprint arXiv:2202.07125 .

Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M. (2023). Timesnet: Temporal 2dvariation modeling for general time series analysis. In International Conference on Learning Representations .

Wu, H., Xu, J., Wang, J., and Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems , 34:22419-22430.

Yagli, G. M., Yang, D., and Srinivasan, D. (2020). Ensemble solar forecasting using data-driven models with probabilistic post-processing through GAMLSS. Solar Energy , 208:612-622.

Yang, D., Wang, W., Gueymard, C. A., Hong, T., Kleissl, J., Huang, J., Perez, M. J., Perez, R., Bright, J. M., Xia, X., van der Meer, D., and Peters, I. M. (2022). A review of solar forecasting, its dependence on atmospheric sciences and implications for grid integration: Towards carbon neutrality. Renewable and Sustainable Energy Reviews , 161:112348.

Zelikman, E., Zhou, S., Irvin, J., Raterink, C., Sheng, H., Avati, A., Kelly, J., Rajagopal, R., Ng, A. Y., and Gagne, D. (2020). Short-term solar irradiance forecasting using calibrated probabilistic models. arXiv preprint arXiv: 2010.04715 .

Zeng, A., Chen, M.-H., Zhang, L., and Xu, Q. (2022). Are transformers effective for time series forecasting? ARXIV.ORG .

Zhang, T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., and Li, J. (2022). Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures.

Zhang, X., Zhen, Z., Sun, Y., Wang, F., Zhang, Y., Ren, H., Ma, H., and Zhang, W. (2023). Prediction interval estimation and deterministic forecasting model using ground-based sky image. IEEE Transactions on Industry Applications , 59(2):2210-2224.

Zhang, Y. and Yan, J. (2023). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations .

Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference , volume 35, pages 1110611115. AAAI Press.

Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. (2022a). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.

Zhou, T., Ma, Z., xue wang, Wen, Q., Sun, L., Yao, T., Yin, W., and Jin, R. (2022b). FiLM: Frequency improved legendre memory model for long-term time series forecasting. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems .

A Experimental details

In this section, we present details about training CrossViViT and its Multi-Quantile variant, as well as the time-series baselines.

A.1 Training setup

A.1.1 CrossViViT

CrossViViT integrates two modalities: satellite video data and time-series station data. For both modalities, essential information such as geographic coordinates, elevation, and precise time-stamps is available. In this section, we provide a comprehensive explanation of the encoding process for each feature and conclude by presenting the hyperparameters of the model.

We first start by encoding the timestamps. For each time point, we have access to the following time features : The year, the month, the day, the hour and the minute at which the measurement was made. We use a cyclical embedding to encode these time features discarding the year. For a time feature x , its corresponding embedding can be expressed as:

Where ω ( x ) is the frequency for time feature x (see Table 2) for the frequency of each time feature). We concatenate these features to form our final time embedding that we simply concatenate to the context and the time-series channels respectively. Furthermore, we incorporate the elevation data for each coordinate in both the context and the time-series. Specifically, the elevation values are concatenated to their corresponding channels in the context and time-series representations.

Regarding the geographic coordinates, we possess information regarding the latitude and longitude for both the context and the station. These coordinates are normalized so as to lie in [ -1 , 1] :

Rotary Positional Emebedding Next, we embed these coordinates using Rotary Positional Embedding (RoPE) that we provide a PyTorch implementation for:

Training configuration CrossViViT was trained on two RTX8000 GPUs, over 17 epochs with early stopping. Its Multi-Quantile variant was also trained on two RTX8000 GPUs, over 12 epochs with early stopping. The remaining settings are identical for both variants: An effective batch size of 20 was utilized for both models; The training process employed the AdamW optimizer (Loshchilov and Hutter, 2019) with a weight decay of 0.05; A cosine warmup strategy was implemented, gradually increasing the learning rate from 0 to 0.0016 over five epochs before starting the decay phase.

Below, we highlight the relevant model hyperparameters for CrossViViT and Multi-Quantile CrossViViT:

A.1.2 Time-series baselines

We conducted training on a total of nine baseline models, and we emphasize the importance of the hyperparameters used for each of these models. Below, we provide a comprehensive overview of the hyperparameters employed in our study:

seq_len : Input sequence length.

label_len : Start token length.

pred_len : Prediction sequence length.

enc_in : Encoder input size.

dec_in : Decoder input size.

e_layers : Number of encoder layers.

d_layers : Number of decoder layers.

c_out : Output size.

d_model : Dimension of model.

n_heads : Number of attention heads.

d_ff : Dimension of Fully Connected Network.

factor : Attention factor.

embed : Time features encoding, options:[timeF, fixed, learned].

distil : Whether to use distilling in encoder.

moving average : Window size of moving average kernel.

We adapted the majority of the baselines using the Time Series Library (TSlib (Wu et al., 2023)), which served as a valuable resource in our experimentation. We refer the reader to the original papers, which served as a base for the hyperparameters utilized in our study, in order to have a comprehensive understanding of the models and the training settings.

The training of the baselines took place on a single RTX8000 GPU over the course of 100 epochs. During training, a batch size of 64 was consistently employed. For model optimization, we utilized the AdamW optimizer (Loshchilov and Hutter, 2019), incorporating a weight decay value set to 0.05. Moreover, we implemented a learning rate reduction strategy known as Reduce Learning Rate on Plateau, which gradually decreased the learning rate by a factor of 0.5 after a patience of 10 epochs.

For the hyperparameter tuning of the baselines, we employed the Orion package (Bouthillier et al., 2022), which is an asynchronous framework designed for black-box function optimization. Its primary objective is to serve as a meta-optimizer specifically tailored for machine learning models. As an example, we present the details of the Crossformer hyperparameter tuning scheme, showcasing the approach we followed to optimize its performance:

B Additional results and visualisations

B.1 Performance on (unobserved) validation PAY station, on validation and test years

This section presents a comprehensive comparative analysis that assesses the performance of CrossViViT in relation to various timeseries approaches and solar irradiance baselines specifically for the PAY station. The evaluation encompasses both the validation period (2017-2019) and the test period (2020-2022), with the validation period serving as the basis for model selection. The results, as presented in Table 3, offer compelling evidence of the superior performance of CrossViViT compared to the alternative approaches across all evaluation splits. These findings underscore the crucial role of accurately capturing cloud dynamics in solar irradiance forecasting, which is particularly pronounced in the "Hard" splits.

B.2 Performance in the operational setting: 00:00 to 23:00 window only

In this section, we present an assessment of CrossViViT's performance in an operational context, specifically focusing on evaluations conducted within the 00:00 to 23:00 time window. This approach provides insights into the model's effectiveness in its intended real-world application, where it is typically executed once daily, precisely at midnight. This evaluation setup offers a more accurate representation of the model's practical utility compared to the sliding window approach. Table 4 shows that the relative performance of CrossViViT over baselines is still maintained which demonstrates the usefulness of our approach in real-world scenarios.

B.3 Performance evaluation with MAPE

In this section, we report the Mean Absolute Percentage Error (MAPE) 3 on test station CAB during test years (2020-2022) and test station TAM during val years (2017-2019) (see Table 5). The MAPE highlights even more the strength of our approach compared to the baselines. We also present results

of the ablated model for which we replace RoPE with a traditional learned Positional Encoding. Despite showing a better performance than most baselines, it does showcase a lower performance than the original CrossViViT, which underlines the importance of the choice of RoPE.

B.4 Inference times

Table 6 shows inference time metrics of CrossViViT and the time-series baselines and we do so by reporting two main metrics:

Latency : The time it takes for the model to process one instance (batch size=1).

MAC : Number of multiply-accumulate operations (MAC). A multiply-accumulate operation corresponds to a +( b × c ) which counts as one operation.

The above metrics have all been computed on a single RTX8000 GPU. From Table 6, we can see that CrossViViT and its multi-quantile variant have the longest inference and training times which is normal given that we incorporate two different modalities, one of which is usually very expensive to process (i.e. satellite data). But we argue that in an operational setting, CrossViViT only needs to be executed once a day which means that while it is desirable to have a relatively short inference time, it may not be of paramount importance, especially if such speed gains come at the expense of predictive accuracy. It is worth acknowledging that a noteworthy limitation lies in the training time, which could potentially be improved through optimization efforts in future work.

B.5 Visualisations for day ahead time series predictions

This section presents visualizations of predictions generated by CrossViViT and CrossFormer on the PAY station for both the validation period (2017-2019) (see Figure 6) and the test period (2020-2022) (see Figure 5). We also present visualizations of predictions generated by the Multi-Quantile version, for the two periods, on the PAY station. The predictions depicted in red are the median (50% quantile) estimation from the model, and the generated prediction interval is defined as the interval between the two predefined quantiles: [ q 0 . 02 , q 0 . 98 ]

CrossViViT:35.34|CrossFormer:63.47

CrossViViT:20.69|CrossFormer:84.67

IH9

500

500

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT: 125.17|CrossFormer: 132.93

CrossViViT: 75.48| CrossFormer: 61.95

500

W

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:71.39|CrossFormer:72.97

CrossViViT:86.61|CrossFormer:32.72

500-

10

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT: 140.89| CrossFormer: 180.98

CrossViViT: 119.64| CrossFormer: 103.14

500

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:58.17|CrossFormer:34.66

CrossViViT:51.14|CrossFormer:45.79

400

IH9

500

200

W

10

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:51.06|CrossFormer:55.18

IH9

500

500

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT

CrossFormer

Ground Truth

CrossViViT:40.94|CrossFormer:55.1

CrossViViT:45.39|CrossFormer:33.05

200

200

10

一

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT: 16.85| CrossFormer: 22.0

CrossViViT: 35.88|CrossFormer: 25.25

200

200

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:35.52|CrossFormer:36.67

CrossViViT:49.4|CrossFormer:52.58

400

400

200-

200

10

1

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT: 41.59 | CrossFormer: 40.64

CrossViViT: 23.25|CrossFormer: 51.19

400

500

1

250

10

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:90.78|CrossFormer:70.99

CrossViViT:65.7|CrossFormer:90.07

500-

500

250-

250

10

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT:15.58|CrossFormer:32.26

CrossViViT:47.97|CrossFormer:42.3

500

IH9

200

250

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT

CrossFormer

Ground Truth

CrossViViT MultiQ:83.18|CrossFormer:63.47

CrossViViTMultiQ:112.33|CrossFormer:84.67

1000

500

500

10

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ:133.19|CrossFormer:132.93

CrossViViT MultiQ: 65.39|CrossFormer: 61.95

1000

500

500

W

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 92.76| CrossFormer: 72.97

CrossViViT MultiQ: 95.3| CrossFormer: 32.72

1000

500

500

10

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 64.88|CrossFormer:180.98

CrossViViT MultiQ: 113.68 |CrossFormer: 103.14

1000

1000

GHI

500

500

-0

10

-24h

-12h

Oh

12h

24h

-24h

-12h

oh

12h

24h

CrossViViT MultiQ: 49.17| CrossFormer: 34.66

CrossViViT MultiQ: 67.56| CrossFormer: 128.6

500

500

10

0

-24h

-12h

Oh

12h

24h

-24h

-12h

oh

12h

24h

CrossViViT MultiQ:42.11|CrossFormer:45.79

CrossViViT MultiQ: 34.43|CrossFormer:55.18

1000

IH9

500

500

W

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

oh

12h

24h

CrossViviT_MultiQuantile

CrossFormer

GroundTruth

CrossViViTMultiQ:43.68|CrossFormer:55.1

CrossViViTMultiQ:33.26|CrossFormer:33.05

400

400

200

200-

10

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 27.55|CrossFormer: 22.0

CrossViViT MultiQ: 11.05|CrossFormer: 25.25

400

400

200-

200

0

0

-24h

-12h

oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 37.63| CrossFormer: 36.67

CrossViViT MultiQ: 70.46| CrossFormer: 52.58

500-

500-

250-

10

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 31.6|CrossFormer: 40.64

CrossViViT MultiQ: 17.18|CrossFormer: 51.19

500

500

250-

250

10

-0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 78.21| CrossFormer: 70.99

CrossViViT MultiQ: 51.18| CrossFormer: 90.07

IH9

500

500

-0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT MultiQ: 64.16| CrossFormer: 32.26

CrossViViT MultiQ: 29.94| CrossFormer: 42.3

IH9

500

500-

0

0

-24h

-12h

Oh

12h

24h

-24h

-12h

Oh

12h

24h

CrossViViT_MultiQuantile

一

CrossFormer

GroundTruth

±

C Vision Transformers (ViT) and Video Vision Transformers (ViViT)

ViT The Vision Transformer (ViT) model (Dosovitskiy et al., 2020) leverages self-attention mechanisms to efficiently process images, inspired by the popular transformer architecture (Bahdanau et al., 2015; Vaswani et al., 2017) notably based on the scaled dot product attention: given the query matrix Q , key matrix K , and value matrix V : Attention ( Q,K,V ) = softmax ( QK T √ d k ) V where d k is the dimensionality of the keys. In the ViT model, the input image of dimensions H × W is divided into non-overlapping patches x i ∈ R h × w , which are linearly projected and transformed into d -dimensional vector tokens z i ∈ R d using a learned weight matrix E that applies 2D convolution. The sequence of patches is defined as z = [ z class , E x 1 , ..., E x P ] + E pos, where z class is an optional learned classification token representing the class label, and E pos ∈ R P × d is a 1D learned positional embedding that encodes position information.

To extract global features from the image, the embedded patches undergo K transformer layers. Each transformer layer k consists of Multi-Headed Self-Attention (MSA) (Vaswani et al., 2017), layer normalization (LN) (Ba et al., 2016), and MLP blocks with residual connections. The MLPs consist of two linear layers separated by the Gaussian Error Linear Unit (GELU) activation function (Hendrycks and Gimpel, 2016). The output of the final layer can be used for image classification, either by directly utilizing the classification token or by applying global average pooling to all tokens z L if the classification token was not used initially.

ViViT The Video Vision Transformer (Arnab et al., 2021) extends the ViT model to handle video classification tasks by incorporating both spatial and temporal dimensions within a transformer-like architecture. The authors propose multiple versions of the model, with a key consideration being how to embed video clips for attention computation.

Two approaches are presented: (1) Uniform frame sampling and (2) Tubelet embedding . The first method involves uniformly sampling n t frames from the video clip, applying the same ViT embedding method Dosovitskiy et al. (2020) to each 2D frame independently, and concatenating the resulting tokens into a single sequence. The second method extracts non-overlapping spatio-temporal "tubes" from the input volume and linearly projects them into R d , effectively extending ViT's embedding technique to 3D data, akin to performing a 3D convolution. Intuitively, the second method allows for the fusion of spatio-temporal information during tokenization, whereas the first method requires independent temporal fusion post-tokenization. Experimental results, however, show only slightly superior performance for the second method in specific scenarios, despite its significantly higher computational complexity (Arnab et al., 2021).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents a multi-modal model, called CrossViViT, to perform day-ahead solar global horizontal irradiance predictions. In that, the model combines spatial information from satellites, i.e. RGB, IR and vapor channels, across Europe with time series information from six point-like stations, i.e. clear sky, pressure, direct normal irradiance, diffuse horizontal irradiance as well as a derived proxy global horizontal irradiance based on the Ineichen model. CrossViViT's performance has been compared again various other statistical and numerical models (Persistence and FFT) as well as other state-of-the-art deep learning approaches based on the transformer building blocks. The performance, measured in RMSE or MAE,\n\nSTRENGTHS:\n- Application case from natural sciences incl. challenging real-world problems of multi-modality and missing data\n- Distinction between 'easy' and 'hard' prediction cases; this is commonly overlooked\n- Open discussion of strength and limitations of CrossViViT in contrast to other method, in particular:\n    * Showcasing that it does not win across the board\n    * Improving the interesting 'hard' cases for domain applications\n\nWEAKNESSES:\n- The reviewer thinks that the evaluation in the manuscript could generally be improved with findings from other papers as follows:\n    * Normalize that forecasting values into a stated range (e.g. [0-1]) or state the value ranges otherwise an RMSE/MAE improvement by certain value cannot be put into a frame of reference\n    * Alternatively, consider reporting MAPE, mean average percentage, improvements instead that already includes that maximum range\n    * By extension the plots in Fig. 4 can be considered somewhat misleading as they do not show the minimum value 0 or give a magnitude of improvement\n- The authors dismiss the use of fourier-layers, e.g. AFNO, without clear substantiation\n- Since cross-attention is such an integral part for mixing tokens from the spatial and temporal domains, it would be meaningful to show the equation in your manuscript beyond the reference\n- It would be meaningful to extend the discussion of the findings towards the domain and/or real-world. What does it mean that you can achieve\n- Non-adherence to the conference paper template, fonts in tables are too small, figures are outside of the text margins\n- Color palette is difficult to read for color-blind people (red-green)\n- The study is not reproducible as there is no code given; there is no indication that it will be released in case of acceptance\n- Might be good to show the rough location, possibly with an arrow, for the TAM station on the map in Fig. 2\n- To the reviewer's personal taste: change the title from a question to a typical title like \"Solar Irradiance Time Series Forecasting with Spatio-Temporal Context\"\n\nQUESTIONS:\nNo direct question but an invitation to provide additional details or comments to the points raised in 'Weaknesses'\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presents a method to integrate information about cloud (using satellite images) with timeseries data related to Solar Irradiance to improve the solar irradiance forecasting.\n\nSTRENGTHS:\nHere some interesting aspects of the paper:\n- the release of a new dataset containing both timeseries and satellite images for many years for several sites. \n- an attempt to build a multimodal architecture based on transformer.\n- usually the subdivision between sunny days and cloudy days is done in the forecasting works related to solar irradiance (e.g. pv production). Authors have proposed to subdivide the days in \"hard\" and \"easy\" based on the similarity between two consecutives days. I think this approach is interesting and help the fair assessment of the model.\n\nWEAKNESSES:\nThe main weakness I see is that day-ahead use case is not explicitely evaluated.\nI know that the sliding window is more general but an important real word use case is to have a real day-ahead prediction. \nAuthors could test their algorithm on day-ahead use case (extracting properly the sliding windows of interest, i.e. from 0:00 to 23:00).\n\nMoreover, some details on the real input of the model seem missing.\n\nQUESTIONS:\nFig.2 --> The TAM is outside the area of interest. This is described in the text but, since the table in fig. reports TAM but it is not present in the first satellite image, the authors should insert a comment in the caption.\n\nRow 230: It is not true for TAM as wrote in row 266\n\nFig.3 and Fig.4 contain some discussions of the results but it would be better if these comments were moved in a proper section in the text.\n\nIt could be interesting to consider a day-ahead situation, in other words considering the sliding window starting at 0:00 and ending at 23.00 of day-1 and compute the next day. In this way the most common day-ahead use case is tested.\n\nI think the satellite image are related to the same time of the time series. I didn't find this information in the text (or I missed it).\n\nSome details are not clear to me.\n1. the context image is used as a separated channel?\n2. all data from all stations are used for the training?\n3. What are effectively the inputs of the model, a list and a detailed encoding would be preferrable. I gave a look at the appendix, some details are present other not.\n4. what are the computational time required for the training of the proposed approach?\n\nAn interesting future perspective could be to incorporate the forecasting of atmosphere's state (images obtained by a Weather Prediction service provider). \n\nRow 51\nSound prediction ?\n\nLIMITATIONS:\nSome limitations have been identified and discussed by authors.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission presents a multimodal model for next-day solar irradiance prediction. They use time series of past irradiance and satellite image to predict irradiance 24h in advance. The model consists of one transformer branch for each modality and a shared (cross-modal) transformer. Their method can be used to predict uncertainty as well. They display improvements over the SOTA. Finally, they released  in open access a dataset acquired with 6 stations across 15 years.\n\nSTRENGTHS:\n- The problem is interesting, difficult, useful, and not a lot of work has been done with machine learning on the subject\n\n- The model architecture is reasonable\n\n- the authors compare their method to many baselines and competing methods\n\n- The authors provide a large-scale (at least temporally) dataset in open access\n\nWEAKNESSES:\n- Some details are missing, making it hard to understand how the method works precisely.\n\n- The uncertainty prediction with quantile lacks a proper evaluation, related work, and comparison baselines.\n\nQUESTIONS:\nQ1) the cross-attention module needs to better explained. There are TxNp video tokens and T temporal tokens; how are they mixed? The authors use \"learned positional encoding\" for these tokens, but it is not explained how. Is the absolute spatiotemporal position encoded? The different sampling rates of eumetsat (5min) and bsrn (1h), yet they have the same number of observations T? \n\nQ2) What is the influence of ROPE compared to a more standard positional encoding? Since the authors make it an integral part of their method, its effect should be quantified in an ablation study\n\nQ3) we allow the model to mask a portion of the past time-series -> Why would the model want to do that? That can only decrease the train performance\n\nS1) The title shouldn't be a question, especially such a hyper-precise and niche one! It should be: Improved Day-Ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context or something in that vein\n\nLIMITATIONS:\nnot provided", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a transformer-based day-ahead forecasting model for solar irradiance at a ground station.\nThe model ingests previous irradiances and contextual (image-sequence) information with a temporal and vision transformer.\nA cross-former merges the tokens, and a temporal transformer decoder estimates irradiances in a 24-hour window.\n\nAn optional multi-quantile output head also allows the model to estimate uncertainty by forecasting quantiles. \nThis multi-quantile loss produces predictions with slightly lower accuracy, which may be justifiable at the benefit of uncertainty quantification.\n\nFurther considerations involve rotary positional encodings for the context image information, which is motivated by the images centered on the measurement station.\n\nOverall, the paper presents a combination of state-of-the-art methods (ViVIT, transformers) with problem-specific ideas (ROPE, Quantile regression) crafted towards a suitable application (irradiance forecasting).\n\nSTRENGTHS:\n* important application (irradiance forecast) addressed with state-of-the-art machine learning (temporal + vision) and architecture crafted towards the forecasting application.\n* uncertainty quantification with a loss function inspired by quantile regression (Koenker & Hallock, 2001)\n* separate evaluation in easy and hard cases. Comparison to reasonable comparison methods and baselines\n* evaluations between different stations to test out-of-domain generalization\n\nWEAKNESSES:\n* some design decisions are not justified or unclear (see questions: learned positional encoding). \n* fast/sloppy preparation of some parts manuscript:\n  * errors/typos in domain-specific equations: GHI = DNI + DHI x cos(z) <- I believe the x cos(z) should be with the DNI (direct normal irradiance) and not the DHI to account for the sun angle.\n  * equation 7: shouldnt it be \\hat{y}_\\alpha, as there is a prediction \\hat{y} for each quantile \\alpha?\n  * style and references are not consistent and some are not retrievable (ArXiv.org vs ArXiv, abs/2010.08895. Or “Rothfuss, H. (2015); Data access at eumetsat.” has no meaning and can not be retrieved); is “348 [BSRN] BSRN. Baseline surface radiation network.” from A Driemel · 2018 <- Cited by 249,  Baseline Surface Radiation Network (BSRN) ?\n\nQUESTIONS:\n* Why did the authors use a learned positional encoding for the temporal transformer rather than a regular periodic function?\n* Why does Multi-Quantile CrossVivit have fewer parameters (almost half) than CrossVivit? Following my intuition, it should have more, as it has more MLP heads. Were the hyperparameters (i.e., number of layers, etc) different?\n* According to the DNI/DHI Equation: Can the authors verify this is just a typo not implemented in the data generation?\n* How applicable is the method compared to related work? In particular with respect to the runtime and inference time. Can the authors provide some numbers on the runtime compared to other approaches\n\nLIMITATIONS:\nA limitations section is included in the conclusion. However, these limitations are rather phrased toward future work and, for instance, describe a lack of data that is hard to address.\nOther potential limitations like computational runtime, which is an important factor towards the applicability of this method, are not discussed. I feel this should be somewhat considered given that the proposed method is crafted for a particular application field.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}