Improving *day-ahead* Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context
Authors: Oussama Boussif
Ghait Boukachab
Dan Assouline
Stefano Massaroli
Tianle Yuan
Loubna Benabbou
Yoshua Bengio
Venue: NeurIPS 2023 poster (2023)
Keywords: Time series forecasting
multi-modal learning
solar irradiance
context-enriched learning
TL;DR: We present a novel deep learning architecture which leverages the spatio-temporal context (e.g. satellite data) around a meteorological station to improve the forecasting of solar irradiance at this station.

Abstract: Solar power harbors immense potential in mitigating climate change by substantially reducing CO$_{2}$ emissions. Nonetheless, the inherent variability of solar irradiance poses a significant challenge for seamlessly integrating solar power into the electrical grid. While the majority of prior research has centered on employing purely time series-based methodologies for solar forecasting, only a limited number of studies have taken into account factors such as cloud cover or the surrounding physical context.
In this paper, we put forth a deep learning architecture designed to harness spatio-temporal context using satellite data, to attain highly accurate day-ahead time-series forecasting for any given station, with a particular emphasis on forecasting Global Horizontal Irradiance (GHI). We also suggest a methodology to extract a distribution for each time step prediction, which can serve as a very valuable measure of uncertainty attached to the forecast. When evaluating models, we propose a testing scheme in which we separate particularly difficult examples from easy ones, in order to capture the model performances in crucial situations, which in the case of this study are the days suffering from varying cloudy conditions. Furthermore, we present a new multi-modal dataset gathering satellite imagery over a large zone and time series for solar irradiance and other related physical variables from multiple geographically diverse solar stations. Our approach exhibits robust performance in solar irradiance forecasting, including zero-shot generalization tests at unobserved solar stations, and holds great promise in promoting the effective integration of solar power into the grid.

Improving day-ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Tempora Context
Oussama Boussif 1,2
Ghait Boukachab 1,3
Dan Assouline 1,2
Stefano Massaroli
Tianle Yuan
Loubna Benabbou'
Yoshua Bengio 1.2
Mila Québec AT Institute Université de Montréal
Universitéé du Quéhec Rimouski "NASA Goddard Space Flight Center
Abstract
Solar power harbors immense potential in mitigating climate change by substan- tially reducing CO2 emissions. Nonetheless, the inherent variability of solar irradiance poses significant challenge for seamlessly integrating solar power into the electrical grid. While the majority of prior research has centered on employing purely time besies-based methodologies for solar forecasting, only limited number of studies have taken into account factors such as cloud cover or the surrounding physical context. In this paper, we put forth deep learning architecture designed to harness spatio-temporal context using satellite data, to attain highly accurate day-ahead time-series forecasting for any given station, with particular emphasis on forecasting Global Horizontal Irradiance (GHI). We also suggest methodol- ogy to extract a distribution for each time step prediction, which can serve as a very valuable measure of uncertainty attached to the forecast. When evaluating models, we propose a testing scheme in which we separate particularly difficult examples from easy ones, in order to capture the model performances in crucial situations, which in the case of this study are the days suffering from varying cloudy conditions. Furthermore, satellite imagery over a large zone and time series for solar irradiance and other related physical variables from multiple geographically diverse solar stations. Our approach exhibits robust performance in solar irradiance forecasting, including zero-shot generalization tests at unobserved solar stations, and holds great promise in promoting the effective integration of solar power into the grid.
present
multi-modal dataset gathering
Introduction
Solar power has become an increasingly important source of renewable energy recent years, with the potential to help mitigate the effects of climate change by reducing greenhouse gas emissions (Doblas-Reyes et al. 2021 IEA 202T). However, the variability of solar irradiance the amount of solar radiation that reaches the earth's 's surface presents challenge for integrating solar power into the grid. Accurate forecasting of solar irradiance can assist grid operators in dealing with the variability of solar power, leading to more efficient and dependable integration of solar power into the grid. a result, this can help to reduce the requirement costly and environmentally damaging backup power sources.
"quual contribution.
Corresponding authors:
ousammabonsii@@mlla.uu
ghait.boukachab @ mila mila.quebec
dan. assouline @ www...alaubeeee
Code dataset: teettp::/ggithee com/ gitbboooo/Croaaaiiitt
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Leaer
Past Spatiel Content
Vision Transfameer
Berrry Entrlkin
Postiosa
Laye
Masking
En brddiingg
Nalshead Mesccsatiin
Crow Trars ammer
Tru Teaaforee
Mail boot
Laye Herm
Erots
MIP
Nose tice
Post Time Series
Terppraa Tranafomer ilnemer
Toiania
Oational Masking
Layer
harrt Posiiional
Nulshead
FPN
Muliple
Laye Norn
MiPy
MIP
Layer Norr
Seare
Future time scrics prediction
Figure 1: ViVis architecture, in its Hulti-Quantil version. (1) The spatio-temporal context videos are tokenized, partially masked, and encoded with a vision transformer, using Rotary Positional Embedding, (2) The time series are tokenized and encoded in parallel with a transformer, (3) Both resulting latents are fed into layers of cross transformer to mix them into one latent, (4) the output is passed into another transformer decoder, (5) and passed to multiple MLP heads which predict multiple quantiles, forming a prediction interval for each day-ahead time-series prediction.
Solar irradiance is influenced by a range of factors, including the time day, the season, weather patterns, and the position of the in the sky. However, of the most significant factors affecting solar irradiance variability is cloud cover. Clouds can block or scatter solar radiation, leading to rapid changes in solar irradiance at the earth's surface. Forecasting solar irradiance accurately thus requires modeling cloud cover, as well as accounting for the inherent variability of the system.
While a lot of previous work has focused using pure time-series approaches forecast solar irradiance (Yang et al.) 2022). few have incorporated cloud cover (Nielsen et al.) [202]] Bone et al.) 2018 Si ct al.] (221)) and even fewer tackled the challenging day-ahead forecasting. often locusing on the easier very short term predictions (2 to 4 hours). When forecasting solar irradiance at a specific station, relying solely on its local physical variables is insufficient due the significant spatial variability of cloud cover. To accurately anticipate the impact of clouds on incoming solar radiation, it is necessary consider their motion and trajectory within larger spatial context encompassing the station. In this paper, we incorporate satellite imaging to forecast solar irradiance at any chosen station and propose multi-modal architecture that can principle be used to forecast any physical variable. This study reveals the inadequacy of conventional testing schemes using conventional metrics like MAE or RMSE on an entire dataset for evaluating solar irradiance forecasting models. They fail to capture the models' performance in crucial cloud-related scenarios. Hence, we emphasize the necessity of separate evaluation approach address the complexity of cloud impact on solar irradiance variability. To allow for the metrics to show such performances by limiting the smoothing effect of the averaging. we propose testing scheme based on multiple splits of the test data, separating particularly difficult examples from easy ones, for the task at hand. This is particularly important for practical downstream tasks related to solar irradiance estimation, such as the monitoring of solar power plants, where the correct prediction of such difficult examples can have very high impact, as easy examples (in the presence of light cloud cover) can be treated fairly well by very simple models (such clear-sky persistence model, as presented in section 5.1

---

Contributions: Our focus is on the application of machine learning to time-series forecasting, with a particular emphasis on utilizing multi modal spatio-tempon data that includes physical, weather, and remotely sensed variables. The main contributions of this paper can be summarized as follows:
We present a deep learning architecture called CrossViViTy designed to leverage spatio- temporal context (such satellite data) in order to achieve highly accurate medium-term (1 day horizon) time-series forecasting at any given station. This paper focuses specifically on forecasting Global Horizontal Irradiance (GHI).
We present a Multi-Quantile version of the model which allows to extract uncertainty estimation attached to each prediction. This methodology, although applied here in our context, should be applicable any forecasting task and framework.
We present multi-modal dataset that combines satellite imagery with solar irradiance and other physical variables. The dataset covers period of 14 years, from 2008 to 2022, and includes data from six geographically diverse solar stations. This dataset is unique in its combination of diverse variables and long time span, and is intended to facilitate the development and evaluation of new multi-modal forecasting models for solar irradiance.
We propose forecasting testing scheme based on multiple time splits of the test data, separating particularly difficult examples from easy ones, therefore allowing to capture the models' performances in problematic examples.
We experimentally show that the proposed approach can generalize to a new station not seen during training in zero-shot generalization forecasting setting.
2
Related works
Machine Learning for time-series forecasting Deep learning approaches have gained popularity for time-series forecasting in recent years due to their ability to model complex nonlinear relationships and capture temporal dependencies. These approaches have demonstrated superior performance compared to traditional statistical methods, motivating further research in this area. In recent survey (Wen al. 2022). it was found that transformers, renowned for their success in natural language processing and computer vision, were also effective for time-series analysis. The authors discussed the strengths and limitations of transformers and compared the structure and performance of recent transformer-ban based architectures on a benchmark weather dataset (Zhou et al. (02T). The particular case of solar irradiance forecasting represents an interesting application for time-series models (Wang) et al. 2019a] Narvaez et al.] [021]] Allzahrani et al., (017). One recent study developed multi-step attention-based model for solar irradiance forecasting that generates deterministic predictions and quantile predictions as well (Sharda et al. 2021). In a similar perspective, Jønler et al. (2023) developed probabilistic solar irradiance transformer that incorporates gated recurrent units and temporal convolution networks, demonstrating strong performance for short-term horizons.
Context mixing/ / Multimodal learning for time-series forecasting Previous studies highlight the potential of time-series methods for solar irradiance forecasting, emphasizing the significance of short- term horizons in solar energy management. However, day-ahead forecasting remains challenging due to the influence of cloud cover on surface irradiance (Bone et al.) 2018 Si et al.) (021), a problem which we aim to address in this paper. Thus, is crucial to account for cloud effects in solar irradiance forecasting regardless of the chosen method. For instance, /hang et al. (2023) investigated the impact of cloud movement on irradiance prediction and proposed an approach to automatically learn the relationship between sky image appearance and solar irradiance. A concurrent work (Liu et al.) 2023) proposed a multimodal-learingg framework for ultra-short-term (10min-ahcad) solar irradiance forecasting. They used Informer (Zhou et al. 102T to encode historical time-series data, then utilized Vision Transformer (Dosovitskiy et al.) 2020) to handle sky images. Finally, they employed cross-attention to couple the two modalities. The studies discussed above highlight the potential of incorporating external data sources, such as sky images and satellite images, in combination with time-series approaches to improve the accuracy of solar forecasting.
Operator Learning Utilizing available satellite imagery to forecast GHI over a region presents limitations as it may not capture clouds that exist at a resolution beyond that of the satellite data. To ensure accurate forecasting of quantities of interest, the ability to query the model at any possible resolution and any point within the domain becomes crucial. Recent ad advancements have witnessed
3

---

the rise of algorithms focusing on learning operators capable of mapping across functional spaces,
with focus on solving partial differential equations (PDE) (Lu al. 2019 Li et al. 2021 Kovachki
et al. 202T Li et al. 2020). These operators can effectively map initial conditions to PDE solutions, making it possible to query the learned solution theoretically anywhere within its domain. Fourier Layers, developed by [i et al. (202T), enable zero-shot prediction on both uniform and non-uniform grids with learnable deformations (Li et al., (022). Pathak et al. (2022) replace attention in ViT (Dosovitskiy al. 2020) with Fourier layer mixing for competitive weather forecasting results with faster inference. MeshFreeflowNer (Jiang et al. 2020) learns high-resolution frames from corresponding lower resolution ones by querying the model at any point of the domain for irregular grids. Similarly, Boussif et al. (2022) employ message passing with a fow-resolution graph for zero- shot super-resolution PDE learning. Additionally, message passing neural PDE solvers (Brandstetter iet al. 2022) exhibit spatio-temporal multi-scale capabilities benefiting from long-expressive memory (Equer al. 2023 Rusch et al. 2022). We note that while these approaches were developed for PDEs in mind, they can still be used for weather-related applications.
Uncertainty estimation When performing solar irradiance forecasting, deterministic forecasts are not sufficient to characterize the inherent variance and uncertainty in solar irradiance data. Probabilistic forecasts, providing uncertainty information, are crucial for energy system management (Wang et al., 2019b). (Doubleday et al.) 2020) and (Yagli et al., 2020) benchmark solar forecasting methods, emphasizing calibration and sharpness in prediction intervals. Specifically in short term solar irradiance forecasting, Zelikman et al. (2020) delves into post-hoc calibration for better predictions. Turkoglu et al. (2022) introduces FiLM Enseembll,,, balancing predictive accuracy and calibration in uncertainty estimation. The deep ensembles approach by Lakshminarayanan et al. (2016) aggregates neural network predictions, capturing both data noise and model uncertainties. Few studies tackle uncertainty evaluation in a regression setting, rather than in classification one, which consists in the estimation of prediction intervals. Sønderby al. (2220), performing precipitation forecasting, suggests an easy solution: the output is separated in 512 bins and the model predicts the precipitation rate (probability) in each of the bins, resulting ultimately in distribution. Other methodologies include bootstrapping and ensembling methods, drawing a distribution out of multiple predictions from submodels, following the spirit of Quantile Regression Forests (Meinshausen and Ridgeway 2006). The evolving landscape of solar irradiance forecasting underscores the importance of a calibrated, comprehensive, and robust probabilistic approach to address inherent uncertainties.
3 Methodology
We develop a framework for solar irradiance time-series forecasting, incorporating context alongside historical time-series data from multiple stations. This framework is inspired by recent advancements in video transformer models (Arnab al., 2021 Feichtenhofer et al.) 2022) and multi-modal models that leverage diverse data sources such as images and time series (Liu et al.) 2023). To establish the foundation for our framework, we provide brief overview of the Rotary Positional Embedding (Su et al. 2021). Subsequently, we present the proposed architecture, Cross ViVIT, in detail, outlining its key components and design principles. Details about the ViT (Dosovitskiy et al. 2020) and ViViT architectures can be found in the Appendix.
3.1 Rotary Positional Embedding
As we are dealing with permutation-vant variant data, assigning positions to patches before using attention is necessary. A common approach is to use additive positional embedding, which considers the absolute positions of the patches added into the input sequence. However, for our case, it is more appropriate to have dot-product attention depend on the relative "distance" between the patches. This is because the station should only be concerned with the distance and direction of nearby clouds, and therefore, relative positional embedding is more sensible.
To this end we make use of RoPE (Su et al. 2021 to mix the station time series and the context. The station and each patch in the context are assigned their normalized latitude and longitude coordinates in [-1, 1] which are used as positions for RoPE Details about the formulation used are left to the Appendix.

---

3.2 Cross Video Vision Transformer for time-series forecasting (CrossVivil)
CrossViVill Our approach aims to integrate the available historical time-series data with video clips of spatio-temporal physical context to enhance the prediction of future time series for solar irradiance. The overall methodology, depicted in Figure T can be summarized as follows:
1. Tokenizing: The video context V XTxxCxtxX*> XHxW with T frames for each of the Cete channels. and H and W respectively the height and width of the video images, is divided into N, non-overlapping patches and linearly projected into a sequence of d-dimensional context tokens = R'''N,,xd, We use the Uniform frame sampling ViVIT scheme Arnab et al. (2021) to embed the videos we have at hand, the frames being concatenated along the batch dimension, and the sample frequency being defined at 30 minutes. The historical time series t E Tx are linearly projected into sequence of d-dimensional time-series tokens zd E Txd" We augment the context tokens with RoPE, as presented in section 3.1 and a learnt positional encoding for the time-series tokens.
2. Masking: As a regularizing mechanism, we allow the model to mask a portion of the past time series and the video context before adding the positional encodings. During the training phase, a masking ratio Maz is randomly sampled from a uniform distribution (O, 0.99) for the context, and the corresponding patches are masked accordingly. We also explored masking the time series to encourage the model to rely more on the context but in practice, no masking gave the best performance. We note that during inference, no masking is applied.
3. Encoding: We encode the time series and the past video context separately with two transformer architectures: a spatio-temporal encoder similar to a ViT for the video context, and a multi layer transformer for the input time series. More specifically, the context tokens alone and time series alone are passed through L separate transformer layers (we keep the same number of layers L for both encoders), including Multi-Head Self-Attention (MSA), LayerNorm (INN) and Multi-Layer Perceptron (MLP) blocks, so that for each layer I, we perform the following operations:
yf2² MSA(LN(Z) (2) + zi cta
y/ (SA(LN(z()) + 2
(1)
MLP(LN(Y? + y
2 MLP(LN(y + y (2)
4. Mixing: We combine the resulting context and time-series latents, respectively n and z within L layers of Transformer with Cross Attention (CA) Vaswani et al. (2017) (we keep the same number of layers in the entire encoder-mixer architecture). Affer adding ROPE, the two I.-th layers are mixed with CA and passed through an MLP block. The output of each layer becomes a mixed latent which is in turn mixed with the context latent 2 and again passed through block of MLP. Formally, the following operations are performed respectively at the first layer (left equations) and on the remaining layers (right equations) of the CA:
mir CA(LN ctm .z12) Z mán MLP(LN(y MiR) + y min
y mir CA(LN(ZI," ata 22(" + zi
(+1 = MIP(LN((771)) + y mii
miz
5. Decoding:
The sequence of mixed tokens 2pvi returned by the layers of Cross Transformer is then passed through N layers of another Transformer as decoder, before adding learnt positional embedding to the token sequence. Each layer n of the Transformer is again formed by MSA, LN and MLP blocks:
yr MSA(LN(Z..) + 2n
(5)
Zn+1 LN(MLP(LN(y.))) + yy)
(6)
The output decoded sequence nn is passed through a final MLP head to output the final predicted future time series tpred RTxC,,
3.3 Multi- Multi-Quantiles: Extracting prediction intervals
To obtain prediction intervals for each forecasted value, we propose a easy methodology which can be used for any deep learning forecasting model, here resulting in alternative version of the
5

---

Contect
18 20B
Station
Latitude
Longitude
Cabanw (CAB)
Knap
51958 N
4"53'
Cener (CNR) O Oano 202g
42"48'N
13
Izana (IZA) Aucyas-Agullo
2014
28"18'N
16"29'W
Palaiscau (PAL) Hacllei 2014
48"42'N
22'
Puyeme (PAY) Vullleumer
2018
46
6"56'E
Tamanraseet (TAM) Baaka
2123
N
5'3T'
V5038
WV 073
Figure 2:
Stations and satellite data. Left: Lo- cation of the six meteorological stations consid- ered in the study with the red border indicating the spatial extent (TAM is out of the considered window), Additionally, three of the eleven spec- tral channels under investigation
highlighted: IR 108, 1S_008 and WV WV073 channels are in- frared (10.8p). visible (0.8pm) and water vapor (7 (7.3pm) channels respectively. Right: Table sum- marizing the geographic coordinates and eleva- tion of each station used in the paper.
ViViT architecture. this modified version, the original MLP head replaced with Neends parallel MIPs, each dedicated to predicting a specific quantile of the distribution for each time step. To achieve this, we employ distinct quantile loss functions for each MLP head. By summing these quantile losses, obtain comprehensive for the model. The quantile loss (Koenker and Hallock 2001) La (y. 9) for the a quantile is defined as:
loss, which serves the training objective
Ly 9) = max - v), (1 - av 0)}-
(7)
The Multi-Quantin loss, aiming to learn multiple quantile predictions da for chosen set of quantiles is then defined as: (Q MoEE.) = En Eng (y, Ba). The selection of quantile heads crucial hyperparameter that determines the density the output distribution generated by the model. To achieve a 96% prediction interval while maintaining sufficiently dense distribution, we set the list of quantiles as U 10.02, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.9, 0.98]. It is worth noting that, it is possible to assign different weights to the quantiles to guide the learning process. This approach can be beneficial scenarios where the task requires preference for overestimation or underestimation. However, in this study, our primary objective was to provide prediction interval, leading to the conservative choice of quantile distribution and uniformly weighing the L.g 's.
Dataset
This section provides comprehensive description of the dataset designed for this study and shared publicly, including all the applied pre-pre pre-processing steps.
4.1 Time series
The time-series measurements were obtained from Baseline Surface Radiation Network datasets (Driemel et al. 2018). The experiments this paper use data from six locations (Figure collected at 30-minute resolution over a 15-year period (2008-2022) The data captures diverse patterns, ranging from consistent irradiance levels under clear skies to fluctuations caused by intermittent clouds affecting surface solar radiation. The data contains measurements of the pressure in the station, clear sky components, Direct Normal Irradiance (DNI), and Diffuse Horizontal Irradiance (DHI).
The GHI component was computed using the formula: GHI = DNI x cos 2 + DHI where is the zenith angle of the sun obtained from the pvlib python library (Holmgren et al. 2020). The Ineichen model (Ineichen] 2016) available through the same library is utilized to obtain the clear sky components. The 2008-20 201 6 data used for training while subsets the 2017-2022 data and stations are used for validation and test performance evaluations. In this study, the models are trained and evaluated to forecast the GHI component for 24-hour period ahead, based on history of 24-hour measurements.

---

Table 1: Comparison of model performances across test stations TAM and CAB, during test years (2020-2022) for CAB, and val years (2017-2019) for TAM. We report the MAE and RMSE for the easy and difficult splits presented in section along with the number of data points for each split. We add the MAE resulting from the Multi- QuatiiQuaatile Cross ViT median prediction, along with pt. the probability for the ground-truth to be included within the interval, averaged across time steps. Additionally, we ablate RoPE against a learned positional embedding.
CAB (2028-2013)
TAM (2017-2019)
Medel
Parameters
All (9203)
Easy (5814)
Hard (3889)
(2209)
Fasy (2064)
Hard (235)
MAB
RMSE
MAE
RMSE
RMISE
RMISE
MAB
RMSE
MAE
RMSE
Persiatance
N/A
63.57
131.44
Fourier;
N/A
68.91
121.22
03.854
87.46
29
56.0
85
62.63
148.42
Fourier,
N/A
65:74
123.15
96
83.56
44.02
33.15
57.56
56
Pourler;
NAA
64.67
124.22
32.67
97.94
$2.51
155.43
40.25
91.36
28.94
55.57
139.68
233.50
Clear Sky Ineichen 2016
N/A
67.19
140.11
60.55
125.06
77.12
159 26
40.61
98.02
31.07
63.4
124.42
242 26
ReFormer KNlgey
1020
5.6M
57.42
102.
53.75
92.97
62.92
115.81
$1.6
137.04
78.57
129.72
108.22
189.55
Informer
56.7M
72.26
122.89
70..5
118.85
74.35
12% 69
63.43
140.38
$2.6
135.46
90.66
156.22
HILM 2hi i Min
9.4M
68:37
116.86
$9..40
U5. 35
$1.4
143.11
62.72
99.71
$4,99
77.63
130.66
210.58
Panch 1
9.6
60.76
119.41
$4.77
107. 71
69.7
135.0
66.94
132.44 44
60.4
124.25
106.77
188 72
LighTS
2012
32K
54.91
102.88
39.5 55
89.28
62.90
120 38
68.51
114.50
64.61
104.98
102.77
177. 98
CrossPurmar (ching 100
212
227M
55.98
101.84
$1.59
90.2
62.55
117.11
68.85
116.45
65.4
107.88
99.16
175.08
PEDFormer 21
23.6M
56.38
99.27
53.08
90.13
61.31
111.54
92.12
146.52
91.13
142,85
100.82
175.08
DLiinar (Zag & 20124
4.7K
75.01
121.013
05.21
99.72
89.65
147.21
73.54
115.40
09:01
98.74
132.67
211.28
AutoPordar Wirers 2011
50.4M
64.34
104.53
60.81
95.14
69.63
117.17
115.88
170.91
117.36
171.07
102.87
169.47
CroSVVVI
145M
50.35
99.18
47.04
89.6
55.30
112.00
49,46
94.96
44,01
79.91
97.40
179.30
Cross (Learned PE:
145M
51.11
103.66
47.31
95.13
56,84
115:31
109.28
196.44
111.33
197.63
91.29
185.61
MAB
Ph
MAB
Ph
MAB
PL
MAE
PL
MAB
Ph
MAE
Pi
Multi-Quantile VIVIE (small)
78.8
61,30
0.91
57.00
0.93
68.94
0.90
$1.20
0.71
78.93
0.70
101.18
0.75
Mulli-Quantlle CrossViVi (lurge)
145 5M4
74:26
0.89
68.83
0.91
$2.39
0.57
79.73
0.76
76.08
0.76
111.74
0.75
4.2 Satellite images
>, which spans a period of 15 years from 2008 to 2022, with an original resolution of minutes, later aligned with the time series data. Our analysis focuses on the non-High Resolution Visible (non-HRV) channels, which encompass 11 spectral channels with a spatial resolution of 6-9km and provide comprehensive coverage of the upper third of the Earth, with a particular emphasis on Europe. These channels, including Infrared and Water vapor, offer valuable information for our investigation. To facilitate our analysis, we reprojected the original geostationary projection data onto the World Geodetic System 1984 (WGS 84) coordinate system (Jacob al., 2022). The region of interest is depicted in Figure Note that one of the 6 stations, TAM, lies slightly outside the region we consider, and therefore represents an nat-of-distribution station in term of the context we use.
In this study, we utilize the EUMETSAT Rapid Scan Service (RSS) dataset (Holmlund) 2003
To augment the contextual information, we computed the optical flow for each channel using the TVL1 algorithm (Sánchez Pérez et al., (013) from the OpenCV package (Bradski. 2000). The optical flow represents the "velocity" of pixels between consecutive frames, which in our case corresponds to the motion of clouds. Furthermore, dataset. The pre-processed satellite data originally had a resolution of 5122, but for computational efficiency, we downscale it to 642.
included the elevation map an additional channel in our
5 Experiments and Results
In this section, we outline the baselines utilized for comparison alongside the suggested architecture. We describe the experimental setup and present the results, benchmarking our framework against state-of-the-an art forecasting models across various test configurations. Additionally, we employ split methodology to assess model performance in challenging prediction scenarios, which hold significant implications for downstream tasks associated with solar irradiance estimation.
The models are trained using a dataset spanning a period of 9 years, from 2008 to 2016, encompassing the stations IZA, CNR, and PAL. Validation is performed on a separate dataset covering years, from 2017 to 2019, for the PAY station. The TAM and CAB stations serve as the test dataset, consisting of 3 years from 2020 to 2022 for CAB, and from 2017 to 2019 for TAM. Each model takes a historical
We replace RoPE with learnable spatial positional encoding. This was done by adding a learnable parameter E g Nxa to the encoded input, where N is the number of tokens in each satellite frame.
7

---

input of 24 hours and predicts the GHI for the subsequent 24 hours, employing sliding window approach. Details about the implementation and hyperparameter tuning can be found in the Appendix.
5.1 Baselines
We conduct a comprehensive comparison between our approach and several state-of-the-ar deep learning architectures specifically designed for forecasting tasks. These architectures are explicitly mentioned in the results tables, such as Table Additionally, we propose dummy baselines that are tailored solar irradiance forecasting, namely:
Persistence: This baseline relies solely on the past day's time-series data, considering as the prediction for future values.
This baseline uses the computable clear sky components of solar irradiance (Ineichen model (Ineichen] 2016)), which represent the total amount of irradiance that would reach the station in the absence of clouds.
Clear Sky baseline:
Fourier approximations: We compute Fourier approximations over the previous day's time series, and apply low-pass filter to keep a limited number of modes. We consider 3 baselines: Fouriers, Fourier, and Fouriers, corresponding to approximations with 3, 4, and modes, respectively.
5.2 "Hard" vs. "Easy" forecasting scenarios
To assess the capability of the suggested model in capturing cloud-induced variability in GHI forecasts, we perform evaluations on test stations using various time splits, aiming to identify its strengths and limitations in comparison to previous approaches. This analysis allows us pinpoint the specific scenarios where ViViT excels, as well as the areas where it falls short. Furthermore, it provides insights into the comparative strengths and weaknesses of previous approaches.
Given the favorable performance of the Persistence baseline when the GHI values exhibit similarity between consecutive days, propose time split approach that categorizes examples either "Easy" or "Hard" based on the extent of GHI variation. The "Easy" examples entail minimal changes in GHI across consecutive days (the Persistence baseline works well), while the "Hard" examples involve significant variations (the Persistence baseline fails). To quantify the similarity, we employ measure based the ratio of the area under the GHI curve for the two days. In order to assign equal importance to ratios such as 0.5 (indicating GHI half that of the previous day) and 2 (indicating log "
GHI double that of the previous day), we utilize the measure GHI over a 24-hour period, and Uprev represents the GHI over the previous 24 hours. Accordingly, we categorize cases as "Easy" when < log ( and "Hard" otherwise.
yarev Here, y represents the
5.3 Performance on stations and years outside the training distribution
Table presents a comprehensive comparison of the proposed ViVIT approach with state-of- the-art timeseries models and dummy baselines. The evaluation is conducted on the test stations TAM and CAB, covering the periods 2017-2019 and 2020-2022, respectively. is important to note that due to the unavailability of data for TAM during the 2020-2022 period, we perform the evaluation using the 2017-2019 data instead.
During the 201 2017-2019 period, ViViT achieves the lowest MAE compared to the time-series models the TAM station. However, it is important note that the persistence baseline still outperforms our approach. This performance disparity can be attributed to the characteristics of the TAM station, which is situated in desert region characterized by predominantly clear and sunny days. As we incorporate cloud information, it may occasionally underestimate the GHI in such clear-sky conditions. Furthermore, the training dataset consists of data from only one "sunny" station located in the Canary Islands (IZA), limiting the availability of examples to effectively learn clear-sky patterns. Based on these results, one can see that for stations characterized by low irradiance intermittency. a combination of persistence and clear-sky models might be sufficient. For the 2020-2022 period on the CAB station, CrossViViT outperforms all baselines across different time splits. This notable improvement can be attributed to the specific meteorological conditions of the CAB station, which experiences higher frequency of cloudy days. This aligns with the primary focus of our research, which aims to accurately forecast GHI under cloudier conditions. Predictions visualisations can be
8

---

MAE: 23.52
MAE: 5.74
median MAE: 43.35
median MAE: 5.44
500
100
Prad
Trd
GHI
GHI
200
0
24hh
-12h
0h
12h
24h
-24h
-12h
0h
12h
24h
-24h
-12h
Oh
12h
24h
-24h
-12h
0h
12h
24h
MAE: 3.34
MAE: 36.94
median MAE:
7.18
median MAE: 83.5
GHI
200
500
GHI
250
500
-12h
-12h
(a)
(b)
38.2
Amplitude
10a
500
100
cton
107
GHI
10a
10
10
24hh
-12h
0h
12h
24h
-4hh
12h
0h
12h
24h
-DrssFomme some Cressivll Ground touth
MAE: 15.85
MAE: 48.44
Amplttude
10a
10
GHI
200
500
10a
10
102
10a
10
20
10
20
-12h
12h
-12h
(c)
Figure 3: Prediction visualisations from Cross for four examples in CAB station, the 2020-2022 test period. (a) ViViT predictions. (b) Multi-Quantill Cross median (40.30 quantile) predictions with [q0.02 (60.02-40.90 98] prediction interval. (c) Predictions from strong baseline, CrossFormer, (d) Fourier spectrum of the target, our prediction, and CrossFormer prediction. Figure (a) illustrates that Cross closely aligns with the ground truth by effectively capturing cloud variations, whereas CrossFormer assumes a clear-sky pattern. This is confirmed by the Fourier spectra depicted in (d), where CrossFormer's spectrum exhibits rapid decay in contrast ViVI..
seen in Figure 3 along with the comparison of the fourier spectra of our prediction, the ground truth and strong baseline, Cromerrrmer.
5.4 Zero-shot forecasting unseen stations and in-distribution years
To evaluate the zero-shot capabilities of Cross ViVIT in terms performance on unseen stations, we maintain the same time period as the training data, thereby emphasizing the spatial dimension of the analysis. The radar plots presented Figure demonstrate that our approach consistently outperforms the baselines across all splits for the CAB and PAY stations, which characterized by higher cloud cover. However, it is noteworthy that Persistence remains competitive for the TAM station, particularly in the "casy" splits where irradiance variations This observation further underscores the efficacy of Cross ViViT accurately accounting for cloud conditions, particularly when examining the performance metries of the "Hard" split.
minimal throughout the day.
5.5
Multi-Quantile results
Table also shows results of the Multi- Sulti-Quantile CrossViViT, including the MAE the median prediction (the 0.5 quantile), along with the test confidence pt obtained for the prediction interval: the probability for the ground-truth to be included within the interval, for cach time step, averaged across the entire dataset considered. As for the other models, the evaluation is conducted the test stations TAM and CAB, covering the periods 2017-2019 and 2020-200 2022, respectively. that for this Multi-Quantile version, the goal is not to provide the best prediction from the median but rather to provide confident prediction intervals, with a high Pt. ideally close to 96%, which we theoretically should reach using 0.02 and 0.98 extreme quantiles. We include small version of the Multi- Multi-Quantile model, and a large one, matching the number of parameters used for our Cross ViVIT model. The small model provides the best performance.
important to note
The prediction intervals achieved a high level of confidence, surpassing 0.9, for the unseen CAB station. However, for the TAM station, the harsh environmental conditions of the desert posed

---

MAE
MAE
MAEE
IIASE Twird
SASE
RARSE
RMISE
RAIST hard
RIASS
2143
MAE hard
WAE 200y
NAE hard
NAE ensy
NAE hard
NAE eisy
NOSSE
RMS: may
Residend
RNSE
(a)
(b)
(c)
Figure 4: Radar plots illustrating the comparative analysis of Cross performance with Persis- tence and CrossFormer models, during training years (the in-distribution period from 2008 2016), on the unobserved stations of TAM, CAB, and PAY, respectively on (a), (b) and (c). CrossViViT demonstrates greater versatility compared CrossFormer forecasting under diverse conditions. Persistence struggles to accurately predict under challenging conditions, particularly on "Hard" days.
challenge for reliable estimation of prediction intervals, due the abundance of clear sky components which is not seen during training. Although the median prediction results were comparatively inferior those of baseline method, it demonstrates consistent patterns in its variation across easy and difficult cases. Furthermore, the test confidence of the proposed method remains relatively constant across different splits, for both stations. Prediction visualisations can be seen on Figure 3
5.6 Ablating the Rotary Positional Embedding
In order to assess the influence of RoPE, we ablate this design choice against learned positional encoding which was the standard encoding used in ViViT. As Table shows, RoPE contributes to the good performance OOD stations and OOD years. Moreover, in the hard cases of the CAB station, RoPE outperforms the learnable positional encoding showcasing its importance in capturing cloud-induced variations.
Conclusion, Limitations, and Future Work
We present CrossViViTy an architecture for accurate day-ahead time-series forecasting of solar irradiance using the spatio-temporal context derived from satellite data. We suggest a testing scheme that captures model performance in crucial situations, such as days with varying cloudy conditions, and we enable the extraction of a distribution for each time step prediction. We also introduce a new multi-modal dataset that provides a diverse collection of solar irradiance and related physical variables from multiple geographically diverse solar stations. Cross ViVIT exhibits robust performance in solar irradiance forecasting, including zero-shot tests at unobserved solar stations during unobserved years, and holds great promise promoting the effective integration of solar power into the grid.
However, there are some limitations to our study. Firstly, we would benefit to include more test years and stations further validate the effectiveness of our approach. It would also be interesting to explore different past and future horizons to further evaluate the robustness of our model across different prediction and context horizons. The training time remains our key limitation since it takes close to hours per epoch on single GPU (see Appendix), but we plan on optimizing the architecture further in future work. Lastly, we plan investigate the use of a cropping methodology on the context as regularization method. Despite these limitations, our study clearly highlights the importance of incorporating spatio-temporal context in solar irradiance forecasting and demonstrates the potential of deep learning in addressing the challenges associated with variability in solar irradiance. Given the promising results observed our current study, we intend to investigate the applicability of Cross VIVIT to other physical variables that are significantly influenced by their surrounding context.
10

---

Acknowledgments and Disclosure of Funding
This research has been funded by the Government of Quebec, with additional computational resources supplied by Mila (mila.quebec). It is conducted as part of international partnership project between Mila, Moroccan agency for sustainable energy (Masen), IUQAR, Polytechnique Montreal, l'Ecole Mohammadia d' Ingénieurs. and I'Institut de Valorisation des Données (IVADO). Our special thanks to MASEN R&D team for their precious collaboration. We extend our sincere appreciation to Jacob Bieker from Open Climate Fix, whose facilitation in accessing EUMETSAT data, coupled with his invaluable assistance regarding data availability, markedly enriched our study.
References
(2021). Net Zero by 2050: A Roadmap for the Global Energy Sector. OECD,
Alzahrami, A., Shamsi, Dagli, C., and Ferdowsi, M. (2017). Solar Irradiance Forecasting Using Deep Neural Networks. Procedia Computer Science, 1 11::04-313.
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučié, M., and Schmid, (2021). Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836-6846.
Ba, J. L.,, Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint 1607
Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Baika, S. (2023). Metcorological synoptical observations from station Tamanrasset (2022-12).
Backup Publisher: National Meteorological Office of Algeria Type: data set.
Bone, V., Pidgeon, J., Kearney, M., and Veeraragavan, A. (2018), Intra-hour direct normal irradiance forecasting through adaptive clear-sky modelling and cloud tracking. Solar Energy, 159:852-888 867.
Boussif, O., Bengio, Y., Benabbou, L.., and Assouline, D. (2022). Magnet: Mesh agnostic neural pde solver. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems, volume 35, pages 31972-31985. Curran Associates, Inc.
Bouthillier, X., Tsirigotis, C., Corneau-Trenmnn
F., Schweizer, T., Dong, L., Delaunay, P., Nor- mandin, F,, Bronzi, M., Suhubdy, D., Askari, R., Noukhovitch, M., Xue, C., Ortiz-Gagné, S., Breuleux, O., Bergeron, A., Bilaniuk, O., Bocco, S., Bertrand, H., Alain, G., Serdyuk, D., Hen- derson, P., Lamblin, P., and Beckham, C. (2022). Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization.
Bradski, (2000). The OpenCV Library. Dr. Dobb's Journal of Software Tools.
Brandstetter, J., Worrall, D. E., and Welling, M. (2022). Message passing neural pde solvers. International Conference On Learning Representations.
Cuevas-Agullé, E. (2014). Radiosonde measurements from station Izana (2014-08). Backup Publisher: Izaña Atmospheric Research Center, Meteorological State Agency Spain Type: data set.
Doblas-Rews Reyes, F., Sörensson, A., Almazroui, M., Dosio, A., Gutowski, W., Haarsma. R., Hamdi, R., Hewitson, B., Kwon, W.-T., Lamptey, B., Maraun, D.. Stephenson, T., Takayabu, I., Terray, L., Turner, A., and Zuo, Z. (2021). Linking Global to Regional Climate Change Supplementary Material.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. (2020). An image is worth 16x words: Transformers for image recognition scale. arXiv preprint a7X.992010.11929.
11

---

Doubleday, K., Van Scyoc Hernandez, V., and Hodge, B.-M. (2020), Benchmark probabilistic solar forecasts: Characteristics and recommendations. Solar Energy, 206:52-67.
Driemel, A., Augustine, J., Behrens, K., Colle, S., Cox, C.,, Cuevas-Agulóó, E., Denn, F. M., Duprat,
T., Fukuda, M., Grobe, H., Haeffelin, M., Hodges, G., Hyett, N., Ijima, O., Kallis, A., Knap, W., Kustov, V., Long. C. N., Longenecker, D., Lupi, A., Maturilli, M., Mimouni, M.. Ntsangwane, L.. Ogihara, H., Olano. X., Olefs, M., Omori, M.. Passamani, L., Pereira, E. B.. Schmithüsen, H., Schumacher, S.. Sieger, R., Tamlyn, J., Vogt, R., Vuilleumier, L., Xia, X., Ohmura, A., and König-Langlo, G. (2018). Baseline surface radiation network (bsrn): structure and data description (1992-2017). Earth System Science Data, 1033):141--1011.
Equer, L., Rusch, T. K., and Mishra, (2023). Multi-scale message passing neural pde solvers. ARXIVORG.
Feichtenhofer, C., Li, Y., He, K., et al. (2022). Masked autoencoders as spatiotemporal learners. Advances in neural information processing systems, 35:35946-35958.
Haeffelin, M. (2014). Basic measurements of radiation at station Palaiseau (2009-02). Backup
Publisher: Laboratoire de Météorologie Dynamique du C.N.R.S., Fcole Polytechnique Type: data set.
Hendrycks, D. and Gimpel, K. (2016). Gaussian error linear units (gelus). arXiv preprint 1606.0885.
Holmgren, W.,
Hansen, C., Mikofski, M., Lorenzo, T., Krien, U., bmu, Anderson, K., Stark, C., DaCoEx, Driesse, A., konstant_c, It, mayudong, de León Peque, M. S., Heliolytics, Miller, E., Anoma, M. A., Boeman, L.,, jforbess, tylunel, Guo, V., Morgan, A., Stein, J., Leroy,
C., R. A. M., JPalakppilyyKHH. Dollinger, J., Anderson, K., MLEEFS, and Dowson, O. (2020). pvlib/pylib-pythoo:: v0.8.0.
Holmlund, K. (2003). Eumetsat current and future plans on product generation and dissemination.
Ineichen, P. (2016). Validation models that estimate the clear sky global and beam solar irradiance. Solar Energy, 132:332-344 344.
Jacob, B., Peter, D., and Jack, K. (2022). Ocf datapipes. http::///gtchubb.om/pmppnp ocf loof_cattaaaaaa sdd_ddatpppess
Jiang, C. M., Esmaeilzadeh, S., Azizzadenesheli, K., Kashinath, K.,, Mustafa, M., Tchelepi, H. A..
Marcus, P., Prabhat, and Anandkumar, A. (2020). Meshfreeflownet: A physics-constrainet deep continuous space-time super-resolution framework. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC '20. IEEE Press.
Jønler, J. F., Brunø Lottrup. F., Berg. B., Zhang, D., and Chen, K. (2023). Probabilistic forecasts of global horizontal irradiance for solar systems. IEEE Sensors Letters, 7(1):1-4.
Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer. In International Conference on Learning Representations.
Knap, W. (2007). Horizon at station Cabauw. Backup Publisher: Koninklijk Nederlands Meteorolo- gisch Instituut, De Bilt Type: data set.
Koenker, R. and Hallock, K. F. (2001). Quantile regression. Journal of economic perspectives, 15(4):143-156.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2021). Neural operator: Learning maps between function spaces.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. (2016). Simple and scalable predictive uncertainty estimation using deep ensembles. NIPS.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. (2020). Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and I. Lin, H., editors, Advances Neural Information Processing Systems, volume 33, pages 6755-6766, Curran Associates, Inc
12

---

Li, Z.-Y., Huang. D.. Liu, B., and Anandkumar, A. (2022). Fourier neural operator with learned deformations for pdes on general geometries. ARX/VOOG..
Li, Z.-Y., Kovachki, N. B., Azizzadenesheli, K., Liu, B., Bhattacharya, K. Stuart, A., and Anand- kumar, A. (2021). Fourier neural operator for parametric partial differential equations. ArXiv, abs/2010.08895.
Liu, J., Zang. H., Cheng, L.,, Ding, T., Wei, Z., and Sun, G. (2023). Transformer- -ased multimodal- learning framework using sky images for ultra-short-term solar irradiance forecasting. Applied Energy, 342:121 160.
Loshchilov, and Hutter, F. (2019). Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Lu, L.,, Jin, P., and Kariiadakis, G. (2019). Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. ARXIV.OR..
Meinshausen, N. and Ridgeway, G. (2006). Quantile regression forests. Journal of machine learning research, 7(6).
Narvaez, G., Giraldo, F., Bressan, M., and Pantoja, A. (2021). Machine learning for site-adaptation and solar radiation forecasting. Renewable Energy, 167:333-342.
Nie, Y., Nguyen, N. H., Sinthong. P., and Kalagnanam, J. (2023). A time series is worth 64 words: Long-term forecasting with transformers.
Nielsen, H., Iosifidis, A., and Karstoft, (2021). Irradiancenet: Spatiotemporal deep learning model for satellite-derived solar irradiance short-term forecasting, Solar Energy, 228:659-661
Olano, X. (2022). Basic measurements of radiation at station Cener (2022-04). Backup Publisher: National Renewable Energy Centre Type: data set.
Pathak. J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall,
D., Li, Z., Azizzadenesheli, K., Hassanzadeh, P., Kashinath, K., and Anandkumar, A. (2022). Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. ARXIV.OR..
Rusch, T. K., Mishra, S., Erichson, N. B., and Mahoney, M. W. (2022). Long expressive memory for sequence modeling. International Conference On Learning Representations.
Sharda, S., Singh, M., and Sharma, (2021). Rsam: Robust self-attention based multi-horizon model for solar irradiance forecasting. IEEE Transactions on Sustainable Energy, 12(2):1394-1405.
Si, Z., Yu, Y., Yang, M., and Li, P. (2021). Hybrid Solar Forecasting Method Using Satellite Visible Images and Modified Convolutional Neural Networks. IEEE Transactions on Industry Applications, 57(1):5-16.
Sønderby, C. K., Espeholt, I., Heek, J., Dehghani, M., Oliver, A., Salimans, T., Agrawal, S., Hickey, J., and Kalchbrenner, N. (2020). Metnet: A neural weather model for precipitation forecasting. arXiv preprint arXiv:2003. /2/40.
Su, J., Lu, Y., Pan, S., Wen, B., and Liu, Y. (2021). Roformer: Enhanced transformer with rotary position embedding. ARXIV.RR..
Sánchez Pérez, J., Meinhardt-LLopie E., and Facciolo, (2013). TV-L1 Optical Flow Estimation.
Image Processing On Line, 3:137-150, htttps://doi.or/// ot.po//00/....0/.p..l....... 2013. 2616.26
Turkoglu, M. O., Becker, A., Gündüzz H., Rezaei, M., Bischl, B., Daudt, R. C., D' D'aronco, S., Wegner, D., and Schindler, K. (2022). Film-ensemble: Probabilistic deep learning via feature-wise linear modulation. Neural Information Processing Systems.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, E.., and Polosukhin, (2017). Attention all you need. Advances in neural information processing systems, 30.
13

---

Vuilleumier, L. (2018). Meteorological synoptical observations from station Payerne (2015-01). Backup Publisher: Swiss Meteorological Agency, Payerne Type: data set.
Wang, H., Lei, Z., Zhang, X., Zhou, B., and Peng. (2019a). A review deep learning for renewable energy forecasting. Energy Conversion and Management, 198:111799,
Wang. Lei, Z.., Zhang. X., Zhou, B., and Peng, (2019b). A review of deep learning for renewable energy forecasting. Energy Conversion and Management, 198:111799.
Wen, Q. Zhou, T., Zhang, C.,, Chen, W., Ma, Z., Yan, J., and Sun, L. (2022). Transformers time series: A survey, arXiv preprint arX.v:220..07125.
Wu, H., Hu, T., Liu, Y., Zhou, H., Wang. J., and Long, M. (2023). Timesnet: Temporal 2d- variation modeling for general time series analysis. In International Conference on Learning Representations.
Wu, H., Xu, J., Wang, J., and Long, M. (2021). Autoformer:
Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-2244 22430..
Yagli, G. M., Yang, D., and Srinivasan, D. (2020). Ensemble solar forecasting using data-driven models with probabilistic post-processing through GAMLSS. Solar Energy, 208:612-622.
Yang, D., Wang, W., Gueymard, A., Hong, T., Kleissl, J., Huang, J., Perez, M. J., Perez, R., Bright, J. M., Xia, X., van der Meer, D., and Peters, I. M. (2022). A review of solar forecasting, its dependence on atmospheric sciences and implications for grid integration: Towards carbon neutrality. Renewable and Sustainable Energy Reviews, 161:1 12344.
Zelikman, E., Zhou, S., Irvin, J., Raterink, C., Sheng, H., Avati, A., Kelly, J., Rajagopal, R., Ng. A. Y., and Gagne, D. (2020). Short-term solar irradiance forecasting using calibrated probabilistic models. arXiv preprint arXiv: 2010.04715.
Zeng, A., Chen, M.-H., Zhang, L., and Xu, Q. (2022). Are transformers effective for time series forecasting? ARXIV.ORG.
Zhang. T., Zhang, Y., Cao, W., Bian, J., Yi, X., Zheng, S., and Li, J. (2022). Less is more: Fast multivariate time series forecasting with light sampling-oriented mip structures.
Zhang. X., Zhen, Z., Sun. Y., Wang. F., Zhang, Y., Ren, H.. Ma, H., and Zhang, W. (2023). Prediction interval estimation and deterministic forecasting model using ground-basee basedd sky image. IEEE Transactions on Industry Applications, 59(2):2210-2224,
Zhang, Y. and Yan, (2023). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations.
Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. (2021). Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106- 11115. AAAI Press.
Zhou, T., Ma, Z., Wen, Q.. Wang, X., Sun, L., and Jin, R. (2022a). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.
Zhou, T., Ma, 7.., xue wang, Wen, Q., Sun, I.,, Yao, T., Yin, W., and Jin, R. (2022b). FiLM: Frequency improved legendre memory model for long-term time series forecasting. In Oh, H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.
14

---

A
Experimental details
In this section, we present details about training Vivis and its Multi-Quantile variant, as well the time-series baselines.
A.1
Training setup
A.1.1 Cross
VivST integrates two modalities: satellite video data and time-series station data. For both modalities, essential information such as geographic coordinates, elevation, and precise time-stamps is available. In this section, provide a comprehensive explanation of the encoding process for each feature and conclude by presenting the hyperparameters of the model.
We first start by encoding the timestamps. For each time point, we have access to the following time features: The year, the month, the day, the hour and the minute which the measurement was made. We use a cyclical embedding to encode these time features discarding the year. For time feature æ, its corresponding embedding can be expressed as:
2TTæ w(x)
2Ta (xx)
sin
.oos
Where w(r) is the frequency for time feature æ (see Table 2) for the frequency of each time feature). We concatenate these features form final time embedding that simply concatenate the context and the time-series channels respectively. Furthermore, we incorporate the elevation data for each coordinate in both the context and the time-series. Specifically, the elevation values concatenated to their corresponding channels in the context and time-series representations.
Regarding the geographic coordinates, we possess information regarding the latitude and longitude for both the context and the station. These coordinates are normalized so as to lie in [-1,
lat + 2
Iat+-90
1
180
(9)
lon 2 180
Table 2: Frequency of each time feature.
Time feature
Frequency
Month
12
Day
31
Hour
24
Minute
60
Rotary Positional Emebedding Next, we embed these coordinates using Rotary Positional Em- bedding (RoPE) that provide Py Torch implementation for:
15

---

Rotary Positional Embedding
class AriallotaryBabeddinnggi Module) :
def __init__ dim, max_freq).
inittrO)..iit__.0)
self dim dim scales torch linspace .0, max free 2, dim // 4)
self register_buffeer watteeerrbufferresceeeeeee ("scales", scales)
def forward(self, coords: torch. :
Args:
coords (torch. Tensor); Coordinates of shape [B, height, wedth) "**
seq_x coorda[:, 0, 0, :]
seq_x seq_x. rqq.x.usqueeze (-1)
seq_y coords 1, 0]
seq_y seq-y wns.uuuzqueeeeeeee (-1)
18
scales self scales ((* ((None, None)), Ellipsis)]
scales scales. (c
22
seq_x x seq_x scales pi
23
seq-y aeq-y * scales * pi
x sinu (seq,,
bij d" j=aeq_y shape [1])
y y_sinu repeat "b bij d", i=seq_x. shape [1]pe[1])
27
sin torch caalcctt(x_smxxxx
y_sinu.sin dim=-1) torch. cat ((x_((x_saa simu ()))) y_sinu.cmm c)) dim=-1)
30
31
sin, cos map(lambda t: rearrange(t, "b i d -> b (i j) d"), (sin, cos))
sin, map(lambda repeat(t, "b bn (d j)", j=2), (sin,
cos))
33
return sin, cos
Training configuration CrossViViT trained two RTX8000 GPUs, 17 epochs with early stopping. Its Multi- Quantile variant was also trained on two RTX8000 GPUs, over 12 epochs with early stopping. The remaining settings are identical for both variants: An effective batch size of 20 was utilized for both models; The training process employed the AdamW optimizer (Loshchiloy and Hutter 2019) with weight decay 0.05; cosine warmup strategy was implemented, gradually increasing the learning rate from 0 to 0.0016 over five epochs before starting the decay phase.
Below, we highlight the relevant model hyperparameters for Cross ViVIT and Multi- cuati--uuattle Cross- ViViT:
16

---

CrossViViT hyperparametest ameters
patch_ size: [8, 8]
use_glu: True max_freq: 128
num_mlp_heads:
5
ctx ttx_masking.ratio: 0.99 ts_masking.ratio:
These hyperparameters apply to the encoding transformers and cross-attention dim: 384
depth: 16
heads: 12
mip_ratio:
14 head: 64
dropout: 0.4
These only apply the decoding transformer decoder_dim: 128
becoder_depth. 4
decoder_heads
@ooder_dim__eeee
Multi- Quantile CrossViViT hyperparameters
patch_ size: [8,
use_glu: True max_freq: 128
num_mlp.heads: 11
5 quantiles: [0.02, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.98]
ctx etx_easking_ratio: 0.99
ts_ msmaskinggratio: 0
These hyperparameters apply to the encoding transformers and cross cross-attention dim: 256
depth: 16
heads: 12
mlp_ratio: dim dim_head: 64
dropout: 0.4
# These only apply the decoding transformer
pecoder_dim. 128
decoder denderr_depth: 4
decoder_heads
decoder_dim_hett
A.1.2 Time-series baselines
We conducted training on total of nine haseline models, and we emphasize the importance of the hyperparameters used for each of these models. Below, we provide comprehensive overview of the hyperparameters employed our study:
seq len: Input sequence length.
label len: Start token length.
pred len: Prediction sequence length.
enc in: Encoder input size.
17

---

dec in: Decoder input size.
e layers: Number of encoder layers.
d layers: Number of decoder layers.
c_out: Output size.
d model: Dimension of model.
n heads: Number of attention heads.
d_ d_ff Dimension of Fully Connected Network.
factor: Attention factor.
embed: Time features encoding, options:[timeF, fixed, learned].
distil: Whether use distilling in encoder.
moving average: Window size of moving average kernel.
We adapted the majority of the baselines using the Time Series Library (TSlib (Wu et al. 2023) which served as valuable resource our experimentation. We refer the reader to original papers, which served as base for the hyperparameters utilized in our study, in order to have a comprehensive understanding of the models and the training settings.
LightTS
FILM
DLinear
model:
model:
model:
enc_in:
2
enc_in: 10
enc_in: 10
3 seq_len: 48
seq_len: 48
seq_len: 48
pred_len: 48
label_ 24
pred_len: 48
d_model: 256
pred_lent 48
moving_avg:
dropout: 0.05
e_layers:
individual: False
chunk_size size:
ratio: 0.4
Crossf ormer
Reformer
PatchTST
model:
model:
model:
enc_in:
enc_in: 10
enc_in:
seq_len: 48
3
c_out: 1
c_out: 1
pred_len: 48
seq_len: 48
seq_len: 48
d_model: 1024
pred_len: 48
pred_len: 48
n_heads:
d_model: 512
d_model: 1024
e_layers:
n_heads: 8
n_heads:
d_ff: 2048
e_layers: 3
e_layers: 3
factor: 10
d_ff: 2048
d_ff: 2048
dropout: 0.01
factor:
factor:
11
dropout: 0.05
11
dropout: 0.05
12
embed: timeF
12 embed: timeF
13
activation: gelu
activation: gelu
18

---

Autof ormer
Informer
FEDf ormer
model:
model:
model:
enc_in: 10
enc_in: 10
enc_in:
5
dec_in:
dec_in: 10
dec_in:
c_out:
c_out: 1
c_out: 1
seq_len: 48
label_len 24
seq_len: 48
label_len: 24
pred_len: 48
label_len: 24
pred_len:
d_model: 2048
pred_len:
8
moving_avg:
n_heads: 4
moving_avg:
d_model: 1024
e_layera: 2
d_model: 512
n_heads:
d_layers: 2
n_heads: 8
e_layers: 3
d_ff: 2048
e_layers: 3
d_layers:
12
factor:
12
d_layers:
d_ff: 2048
13
dropout: 0.
13
d_ff: 2048
factor: 10
14
embed: timeF
14
dropout: 0.05
dropout: 0.01
activation: gelu 16 distil: True
version: fourier mode_select: random
embed: timeF
17
activation: gelu
17 modes: 32
The training of the baselines took place on single RTX8000 GPU over course of 100 epochs. During training. batch size of 64 was consistently employed. For model optimization, utilized the Adam W optimizer (Loshchilov and Hutter 2019), incorporating a weight decay value set to 0.05. Moreover, implemented learning rate reduction strategy known Reduce Learning Rate Platcau, which gradually decreased the learning rate by a factor of 0.5 after patience of 10 epochs.
For the hyperparameter tuning of the baselines, we employed the Orion package (Bouthillier et al.) 2022), which an asynchronous framework designed for black-box function optimization, Its primary objective is to serve meta-optimizer specifically tailored for machine learning models. As an example, we present the details of the Crossformer hyperparameter tuning scheme, showcasing the approach we followed to optimize its performance:
Crossformer hyperparameters tuning
parama:
2
optimizer. lr: loguniform (1e-8, 0.1)
optimizer. weight_ _ loguniform (1e-10, 1)
d_model: choices [1228 256, 512, 1024 2048])
d_ff: Choices([1024, 2048, 4096]) 2044,,4996])
6 n_heads: choices ([1,,, (1,,,4,,,,66)
e_layers: choices [1, (1,2,3,4,51)
dropout: choices 01, [00.01,0.05,00
factor: choices([2,5,100))
max_epochs: Sidelity(lows) high=100, base=3)
B Additional results and visualisations
B.1 Performance on (unobserved) validation PAY station, on validation and test years
This section presents comprehensive comparative analysis that assesses the performance of Cross- ViVIT in relation to various timeseries approaches and solar irradiance baselines specifically for the PAY station. The evaluation encompasses both the validation period (2017-2019) and the test period (2020-2022). with the validation period serving as the basis for model selection. The results, as presented 3 offer compelling evidence the superior performance of Cross ViVIT compared to the alternative approaches across all evaluation splits. These findings underscore the crucial role of accurately capturing cloud dynamics in solar irradiance forecasting, which particularly pronounced in the "Hard" splits.
19

---

Table 3: Comparison of model performances in the val station PAY, during test years (2020-2022) and val years (2017-2019). We report the MAE and RMSE for the easy and difficult splits (presented in the main paper) along with the number of data points for each split. We add the MAE resulting from the Multi-Quanticn Cross CrossViViT ViVIT median prediction, along with pt. the probability for the ground-truth to be included within the interval, averaged across time steps.
PAY (Ted) years- 2010-3022)
PAY (Val years 2017-2019)
Models
Parameters
All (12171)
Easy (8053)
Harrd (4118)
ATl (27166)
Easy (16683)
Hard (10483)
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
Persistence
N/A
61.9
140.05
44.96
105.47
94.99
190.31
63.61
141.16
48.57
106.9]
87.54
180,99
Foursers
09.21
129.29
91.61
102.53
130.58
94.81
Fourfer:
131.13
93.68
183.48
132.41
96.71
174.79
Fourier:
98.61
175.95
Sky Innichen
138.70
116.72
64.72
174,03
62.12
77.25
ReFormer
Kitse et
8.6M
50.15
75
51.67
91.67
73.79
138
59.3
109.3
54.06
94.31
Informer Zimircep PU1
56.7M
79.18
133.47
75.8
126.07
K5.78
146.85
77.51
131.22
75.65
126.58
$0.65
52
FILM C Puzdis
9.4M
69.87
124.86
$5.93
91.15
97.11
172.72
71.1
125.33
58.35
94.17
91.39
163
1 Mic Ki1
9.0M
63.21
131.64
53.06
113.78
83.06
100.93
63.88
131.14
55.47
115.59
77.25
152.64
LighTS
Phinggrr Pilaa
32K
55.57
111.22
48.93
83.69
77.4
145.53
58.22
110.84
50.72
91.24
70.16
136. 53
CrossFormer MArin Jn 1023
227M
59.64
111.03
$1.04
90.34
76.45
143.08
59.39
111.49
52.55
93.15
70.29
135.66
VEDFonggr (2inurr F
23.6
63.44
110.62
58.15
97.33
73.79
132.85
62.45
109 999.92
59,88
100.61
66.57
123.3
DiLinear Zedy at 20r2s
4.7K
75.09
12%.6
59.42
93.82
105.74
178.04
76.76
129.45
62.64
97.93
99.29
167.81
AntoForner (Wieer 2021
50.4M
73.36
117.22
68.89
105. 52
K2. 12
137.25
71.39
114,96
60,42
106.79
74.54
126.88
CoosViV
145M
51.47
107.73
41.59
K5.14
70,81
141.87
52.02
108.77
44.71
90.47
63,65
132.79
MAR
P4
MAE
Pt
MAE
Pt
MAE
Pt
MAB
Pt
MAE
Pr
Multi Multi-Quantile Crois (small)
78.8M
29.06
0.83
54.96
0.33
57.06
0.84
57.05
0.86
51,75
0.87
67.05
0.86
Multi-Quantile CrossVIViT (large)
145.
74.18
0.89
68.69
0.91
82. 38
0.87
59.97
0.87
$2.67
0.88
71,99
0.85
B.2 Performance in the operational setting: 00:00 to 23:00 window only
In this section, we present assessment of CrossViViT's performance in an operational context, specifically focusing on evaluations conducted within the 00:00 to 23:00 time window. This approach provides insights into the model's effectiveness in its intended real-world application, where it is typically executed once daily, precisely at midnight. This evaluation setup offers a more accurate representation of the model's practical utility compared to the sliding window approach. Table that the relative performance of Cross CrossViViT ViVIT over baselines is still maintained which demonstrates the usefulness of our approach in real-world scenarios.
Table 4: Comparison of model performances in the test station CAB, during test years (2020-2022) and test station TAM on val years (2017-2015 for 00:00 to 23:00 windows only. We report the MAE and RMSE for the easy and hard splits along with the number of data points for each split.
2022)
2019)
Model
Parameters
(207)
Easy (120)
All (48)
Easy
Iard
MAB
RMSE
MAB
RMSE
MAB
RMSE
RMSE
MAB
RMSE
MAE
RMSE
Persistence
N/A
63.19
49.91
$1.09
160.7
238.15
Fourler,
NIA
69.49
122.65
54.9
57.7
44.57
253.2
Fourier;
N/A
65.78
123.80
51.71
92.03
85.19
05.08
31.95
140.79
Fourier,
N/A
64:55
124.51
50..4
03.08
84.5
157.9
41.91
94.95
27.79
49.58
140.77
33
Clear Sky Ineichm 2016
N/A
67.31
140.42
58.63
121.49
79.21
162.90
40.93
99.03
29.42
55.82
121.5
90
ReFormer
Kinev
1020
3.6
60.64
104.52
92.
67.07
119:35
$1.5
134.89
77.95
126.06
106.36
Informer P2PaEEL
2011
56.7M
72:82
125.44
69.23
118.1
77.78
134.91
$4.51
137.51
$1.91
131.83
102.72
172.11
FILM Zhork 707211
9.4M
69.36
120.87
38.65
95.01
$4.14
149 36
63.42
105.46
53.4
73.02
133.5%
217.82
Patch
Lo 2304
9.6M
64.37
124.09
36.75
108.37
74.84
142.50 96
73.16
142.01
67.18
130.89
115.03
200 49
LighTT Miag
32K
61.54
105. 14
$4.56
X7.32
71.15
125.73
61.75
101.48
$2.95
80.20
125. 19
193.04
CrnssFormer Pening mile Tan 2023
227M
58.19
104.66
52.96
30003
65.4
121.90
66.25
113.67
58.81
95.04
118.33
200.37
PEDFurmer Phon
20720
23.6M
59.11
104
52.68
88.
67.71
123.25
114.47
176.32 24
116.55 .9
176.21
99:62
176.42
DLinear (20g 20 2014
4,7K
27:36
125.1
65.61
100.19
93.57
153.08
79.35
121.69
70.71
99.87
139.9
220.55
AutoPordeer (1
2011
50.4M
72.16
110.54
67.57
97.6
78.07
126.23
141.97
204.4
146.02
208.11
113.62
176.2
CrossViVII
145M
$1.91
101.58
46.34
87
59.73
118.8
52.72
99.11
46.47
82.53
96,42
173,79
B.3 Performance evaluation with MAPE
In this section, we report the Mean Absolute Percentage Error (MAPE) on test station CAB during test years (2020-2022) and test station TAM during val years (2017-2019) (see Table The MAPE highlights even more the strength of our approach compared to the baselines. We also present results
'Sinee during the night, the GHI values are zero, we clamp the denominator by = 11 since the precision of the GHI values in the dataset is 1.
20

---

Table Comparison of model performances across test stations TAM and CAB, during test years (2020-202)) for CAB, and val years (2017-2019) for TAM. using the Mean Absolute Percentage Error (MAPE) metric. We report the MAPE for the easy and difficult splits presented in section 5.2 along with the number of data points for each split. We add the MAPE resulting from the Multi-Quee Qaati-Quanttil CrossViVil median prediction, along with pt. the probability for the ground-truth to
be included within the interval, averaged across time steps. Additionally, ablate RoPE against
learned positional embedding.
(2020 2022)
(2017-2019)
Models
Parameters
All (9703)
Easy (5814)
Hard (3889)
All (2299)
Easy (2064)
Hard (235)
N/A
Fourier;
NIA
8.43
8.22
8.76
19.98
20.37
16.6
Fourier;
N/
5.42
5.18
7.54
8.83
Fouriers
NIA
4.21
4.17
4.26
7.55
7.44
8.47
Clear Sky (neichen 2016
N/A
0.72
0.53
1,02
0,28
0,21
0,9
ReForrer Kiaev
al 20207
8.6M
4.05
3.7
4.59
5.88
5.89
5.78
Informer 21mv of 2021
56.7M
6.12
4.92
7.93
5,44
5.42
5.57
Zhvu e1
Patch1SI
[N66 2. K176
9.6M
2.53
2.4
2.74
3.32
3.29
3.59
LighTS (Zhang
32K
7.08
16.29
CrossForimer Zhant and Yan 2023
227M
3.07
2.76
3.54
3.57
3.54
3.86
FEDFormer Zigu
23.6M
5,35
4.95
5,96
12.83
13.15
10.01
DL.nnar Zerprein Kize)
4.7K
13.98
12.39
16.37
13.11
12.98
14.21
AutoFormer au 2021
50.4M
10:47
9.77
11.51
23.33
24.18
15.91
Cross
0.58
VIVIs VIVIT (Learned PE
145M
0.83
0.64
1,13
0.95
0.99
0.6
MAPE
Pr
MAPE
MAPE
MAPE
MAPE
MAPE
25
Multi-Quantile VIVIL (small)
78.8M
0.61
0.92
0.43
0.93
0.87
0.90
0.61
0.92
0.41
0.93
0.87
0.90
Multi-Quantile ViVIE (large)
145.5M
0.69
0.89
0.49
0.69
0.89
0.49
0.91
0.99
0.87
of the ablated model for which we replace RoPE with traditional learned Positional Encoding. Despite showing better performance than most baselines, it does showcase lower performance than the original Cross ViViT, which underlines the importance of the choice of RoPE.
B.4 Inference times
Table 6 shows inference time metrics of CrossViVis and the time-series baselines and we do so by reporting two main metrics:
Lateney: The time it takes for the model to process instance (batch size=1).
MAC: Number of multiply-accumulaat operations (MAC). A multiply-accmmulate operation
corresponds + (b x c) which counts one operation.
The above metrics have all been computed on single RTX8000 GPU. From Table 6 that CrossViViT and its multi-quantile variant have the longest inference and training times which is normal given that we incorporate two different modalities, one of which is usually very expensive to process (i.c. satellite data). But we argue that in an operational setting, Cross ViViT only needs to be executed once day which means that while it is desirable to have relatively short inference time, it may not be of paramount importance, especially such speed gains come at the expense of predictive accuracy. It is worth acknowledging that noteworthy limitation lies in the training time, which could potentially be improved through optimization efforts in future work.
we can see
B.5 Visualisations for day ahead time series predictions
This section presents visualizations of predictions generated by Vivss and CrossFormer on the PAY station for both the validation period (2017-2019) (see Figure and the test period (2020-2022) (see Figure 5) We also present visualizations of predictions generated by the Multi- luatt--Qaatilee version, for the two periods, on the PAY station. The predictions depicted in red are the median (50% quantile) estimation from the model, and the generated prediction interval is defined as the interval between the two predefined quantiles: [go.02, 40.98]
We replace RoPE with learnable spatial positional encoding. This was done by adding a learnable parameter g Nxa to the encoded input, where N the number of tokens in each satellite frame.
21

---

CrossViViT: 35.34 CrossFormer: 63.47
CrossViViT: 20.69 CrossFormer: 84.67
GHI
500
-24h
-12h
12h
24h
-24h
-12h
12h
24h
CrossViViT: 125.17 CrossFormer: 132.93
CrossViViT: 75.48 CrossFormer: 61.95
500
-24h
-12h
12h
-24h
-12h
12h
CrossViViT: 71.39 CrossFormer: 72.97
CrossViViT: 86.61
CrossFormer: 32.72
GH
500
500
12h
12h
CrossViViT: 140.89 CrossFormer: 180.98
CrossViViT: 119.64 CrossFormer: 103.14
500
500
12h
-12h
12h
CrossViViT: 58.17 CrossFormer: 34.66
CrossViViT: 51.14 CrossFormer: 45.79
400
500
200
W
-24h
-12h
12h
24h
-24h
-12h
12h
CrossVivIT: 51.06 CrossFormer: 55.18
CrossViViT: 112.08 CrossFormer: 122.89
GHI
500
-24h
-12h
12h
24h
-24h
-12h
12h
24h
CrossVivIT
CrossFormer
Ground Truth
Figure 5: Visualizations of ViossViViT and CrossFormer are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the test period spanning from 2020 to 2022.
22

---

CrossViViT: 40.94 CrossFormer: 55.1
CrossViViT: 45.39 CrossFormer: 33.05
GH
200
-24h
-12h
12h
24h
-24h
-12h
12h
CrossViVIT: 16.85 CrossFormer: 22.0
CrossViViT: 35.88
CrossFormer: 25.25
200
200
100
-24h
-12h
12h
-24h
-12h
12h
CrossViviT: 35.52 CrossFormer: 36.67
CrossViviT: 49.4
CrossFormer: 52.58
400
400
200
200
12h
12h
CrossViViT: 41.59 CrossFormer: 40.64
CrossViViT: 23.25 CrossFormer: 51.19
400
500
200
250
-12h
12h
CrossViViT: 90.78 CrossFormer: 70.99
CrossViViT: 65.7 CrossFormer: 90.07
500
500
250
250
-24h
-12h
12h
24h
-24h
-12h
12h
CrossViViT: 15.58
CrossFormer: 32.26
CrossViViT: 47.97 CrossFormer: 42.3
500
GHI
200
250
-24h
-12h
12h
24h
-24h
-12h
12h
24h
CrossVivIT
CrossFormer
Ground Truth
Figure 6: Visualizations of CrossViViT and CrossFormer are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2017 to 2019.
23

---

CrossVivIT MultiQ: 83.18 CrossFormer: 63.47
CrossViviT MultiQ: 112.33 CrossFormer: 84.67
1000
GHI
500
500
-24h
-12h
-24h
-12h
CrossViviT MultiQ: 133.19 CrossFormer: 132.93
CrossViviT MultiQ: 65.39 CrossFormer: 61.95
500
GH
-24h
24h
-24h
CrossViviT MultiQ: 92.76 CrossFormer: 72.97
CrossVivIT MultiQ: 95.3 CrossFormer: 32.72
1000
GHI
500
500
-24h
-12h
24h
-24h
-12h
CrossViviT MultiQ: 64.88 CrossFormer: 180.98
CrossViVIT MultiQ: 113.68
CrossFormer: 103.14
1000
1000
GHI
500
500
-74h
-12h
12h
74h
-24h
-17h
12h
24h
CrossViviT MultiQ: 49.17 CrossFormer: 34.66
CrossViviT MultiQ: 67.56 | CrossFormer: 128.6
GHI
500
500
0
-24h
-12h
0h
12h
24h
-24h
-12h
0h
12h
24h
CrossViVIT MultiQ: 42.11 CrossFormer: 45.79
CrossVivIT MultiQ: 34.43 CrossFormer: 55.18
500
GHI
500
W
0
-24h
-12h
0h
12h
24h
-24h
-12h
0h
12h
24h
CrossVivIT MultiQuantile
CrossFormer
Ground Truth
Figure 7: Visualizations Multi-Quantile CrossViViT (median prediction and [yo.02, Q0.98 prediction interval) and CrossFormer predictions presented for subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2020 to 2022.
24

---

CrossViViT MultiQ: 43.68 CrossFormer: 55.1
CrossViVIT MultiQ: 33.26 CrossFormer: 33.05
400
400
GHI
200
200
-2hh
-12h
12h
-24h
-12h
12h
CrossViViT MultiQ: 27.55
CrossFormer: 22.0
CrossViViT MultiQ: 11.05
CrossFormer: 25.25
400
400
200
200
-44h
12h
12h
CrossViVIT MultiQ: 37.63 CrossFormer: 36.67
CrossVivIT MultiQ: 70.46 CrossFormer: 52.58
500
500
250
-24h
-12h
12h
24h
-24h
-12h
12h
CrossViviT MultiQ: 31.6 CrossFormer: 40.64
CrossViViT MultiQ: 17.18
CrossFormer: 51.19
500
500
250
250
-74h
-12h
12h
74h
-24h
-12h
12h
24h
CrossViViT MultiQ: 78.21
CrossFormer: 70.99
CrossViviT MultiQ: 51.18
CrossFormer: 90.07
GHI
500
500
-24h
-12h
0h
12h
24h
-24h
-12h
0h
12h
24h
CrossViViT MultiQ: 64.16 CrossFormer: 32.26
CrossViVIT MultiQ: 29.94
CrossFormer: 42.3
500
GHI
500
-24h
-12h
0h
12h
24h
-24h
-12h
0h
12h
24h
CrossVivIT MultiQuantile
CrossFormer
Ground Truth
Figure 8: Visualizations Multi-Quantile Cross (median prediction and [q0.02, Q0.98 prediction interval) and CrossFormer predictions presented for subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2017 to 2019.
25

---

Table 6: Comparison of models' inference metrics and training times. We report the mean latency and standard deviation in milliseconds as well as Giga MACs and the training time per epoch. All these metrics were computed for a single NVIDIA RTX8000 GPU.
Models
Latency (ms)
Giga MACs
Training time per epoch (s)
ReFormer (Kitacy ct al. 2020
5,43 + 0.32
1.66
387
Informer (Zhou et 2021
8,63 0.30
2.54
623
FiLM (Zhou et al. 2022b
9.5 0.25
445
PatchTST (Nic ci al. 2023
2.69 0.13
0.57
106
LighTS (Zhang
0.82 - 0.087
<.001
370
CrossFormer (Zhang and Yan.
2023
17.89 + 0.43
7.05
1300
FEDFormer (Zhou et al. 2022a
66,41 0.63
1.03
1134
DLinear (Zeg et al. 2022
0.29 =
0.01
365
AutoFormer (Wu et (2021)
32.03 = 0.54
2.92
1500
Vross (145M)
65.03 + 0.43
180.47
18000
Multi-Quantice Cross VIVIT (78.8M)
50.48 = 0.24
100.45
18000
C Vision Transformers (ViT) and Video Vision Transformers (ViViT)
The Vision Transformer (ViT) model (Dosovitskiy et al. 2020) leverages self-attention mech- anisms to efficiently process images, inspired by the popular transformer architecture (Bahdanau et al. 2015 Vaswani et al.) 2017) notably based on the scaled dot product attention: given the query
VIT
Attention(Q, K, V) = softmax OKK V where vdz
matrix Q, key matrix K, and value matrix
dj is the dimensionality of the keys. In the VIT model, the input image of dimensions H x W is divided into non-overlapping patches 2; E R which are linearly projected and transformed into d d-dimensional vector tokens Id using a learned weight matrix E that applies 2D convolution.
The sequence of patches is defined as z [Zclass, E .... Exp] + Epos. where Zelass is an optional learned classification token representing the class label, and Epos E Pxd is 1D learned positional embedding that encodes position information.
To extract global features from the image, the embedded patches undergo transformer layers. Each transformer layer & consists of Multi-Headed Self-Attention (MSA) (Vaswani et al.) 2017). layer normalization (LN) (Ba et al. 2016). and MLP blocks with residual connections. The MLPs consist of two linear layers separated by the Gaussian Error Linear Unit (GELU) activation function (Hendrycks and Gimpel| 2016). The output of the final layer can be used for image classification, either by directly utilizing the classification token by applying global average pooling to all tokens 2 if the classification token was not used initially.
VIVIT The Video Vision Transformer (Amab al. 202 extends the ViT model to handle video classification tasks by incorporating both spatial and temporal dimensions within a transformer like architecture. The authors propose multiple versions of the model, with key consideration being how to embed video clips for attention computation.
Two approaches presented: (1) Uniform frame sampling and (2) Tubelet embedding. The first method involves uniformly sampling n, frames from the video clip, applying the same VIT embedding method Dosovitskiy et al. (2020) to each 2D frame independently. and concatenating the resulting tokens into a single sequence. The second method extracts non-overlapping spatio-tempon from the input volume and linearly projects them into Rd, effectively extending VIT's embedding technique to 3D data, akin to performing a 3D convolution. Intuitively, the second method allows for the fusion of spatio-temporal information during tokenization, whereas the first method requires independent temporal fusion post-tokenization. Experimental results, however, show only slightly superior performance for the second method in specific scenarios, despite its significantly higher computational complexity (Arnab et al. 2021).
"tubes"
26


Reviews:
[{"review_id": "Bt91Z53Xn8", "reviewer": "Reviewer_1GwN", "summary": "The paper proposes a transformer-based day-ahead forecasting model for solar irradiance at a ground station.\nThe model ingests previous irradiances and contextual (image-sequence) information with a temporal and vision transformer.\nA cross-former merges the tokens, and a temporal transformer decoder estimates irradiances in a 24-hour window.\n\nAn optional multi-quantile output head also allows the model to estimate uncertainty by forecasting quantiles. \nThis multi-quantile loss produces predictions with slightly lower accuracy, which may be justifiable at the benefit of uncertainty quantification.\n\nFurther considerations involve rotary positional encodings for the context image information, which is motivated by the images centered on the measurement station.\n\nOverall, the paper presents a combination of state-of-the-art methods (ViVIT, transformers) with problem-specific ideas (ROPE, Quantile regression) crafted towards a suitable application (irradiance forecasting).", "questions": "* Why did the authors use a learned positional encoding for the temporal transformer rather than a regular periodic function?\n* Why does Multi-Quantile CrossVivit have fewer parameters (almost half) than CrossVivit? Following my intuition, it should have more, as it has more MLP heads. Were the hyperparameters (i.e., number of layers, etc) different?\n* According to the DNI/DHI Equation: Can the authors verify this is just a typo not implemented in the data generation?\n* How applicable is the method compared to related work? In particular with respect to the runtime and inference time. Can the authors provide some numbers on the runtime compared to other approaches", "limitations": "A limitations section is included in the conclusion. However, these limitations are rather phrased toward future work and, for instance, describe a lack of data that is hard to address.\nOther potential limitations like computational runtime, which is an important factor towards the applicability of this method, are not discussed. I feel this should be somewhat considered given that the proposed method is crafted for a particular application field.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* important application (irradiance forecast) addressed with state-of-the-art machine learning (temporal + vision) and architecture crafted towards the forecasting application.\n* uncertainty quantification with a loss function inspired by quantile regression (Koenker & Hallock, 2001)\n* separate evaluation in easy and hard cases. Comparison to reasonable comparison methods and baselines\n* evaluations between different stations to test out-of-domain generalization", "weaknesses": "* some design decisions are not justified or unclear (see questions: learned positional encoding). \n* fast/sloppy preparation of some parts manuscript:\n  * errors/typos in domain-specific equations: GHI = DNI + DHI x cos(z) <- I believe the x cos(z) should be with the DNI (direct normal irradiance) and not the DHI to account for the sun angle.\n  * equation 7: shouldnt it be \\hat{y}_\\alpha, as there is a prediction \\hat{y} for each quantile \\alpha?\n  * style and references are not consistent and some are not retrievable (ArXiv.org vs ArXiv, abs/2010.08895. Or “Rothfuss, H. (2015); Data access at eumetsat.” has no meaning and can not be retrieved); is “348 [BSRN] BSRN. Baseline surface radiation network.” from A Driemel · 2018 <- Cited by 249,  Baseline Surface Radiation Network (BSRN) ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and suggestions to improve the paper. We respond to the reviewer's comments below:\n\n> errors/typos in domain-specific equations: GHI = DNI + DHI x cos(z) <- I believe the x cos(z) should be with the DNI (direct normal irradiance) and not the DHI to account for the sun angle.\n\nIt was a typo indeed, we double-checked the data-generation process and it contained the right formula.\n\n> equation 7: shouldnt it be \\hat{y}_\\alpha, as there is a prediction \\hat{y} for each quantile \\alpha?\n\nThere is indeed a different prediction for each quantile, so we changed the notation regarding the MultiQuantile loss, just under the equation 7 (yet not for equation 7 itself which simply defines a single quantile loss).\n\n> style and references are not consistent and some are not retrievable (ArXiv.org vs ArXiv, abs/2010.08895. Or “Rothfuss, H. (2015); Data access at eumetsat.” has no meaning and can not be retrieved); is “348 [BSRN] BSRN. Baseline surface radiation network.” from A Driemel · 2018 <- Cited by 249, Baseline Surface Radiation Network (BSRN) ?\n\nThe three above points were corrected. \n\n> Why did the authors use a learned positional encoding for the temporal transformer rather than a regular periodic function?\n\nThis is a design choice, motivated by the current literature on transformers. In particular, it was used in the architecture that partly gave us inspiration, the Video VIT (https://arxiv.org/pdf/2103.15691.pdf), as mentioned in the paper. It is therefore natural that we employ the same positional encoding here, as it appears to offer more expressivity and function the best.\n\n> Why does Multi-Quantile CrossVivit have fewer parameters (almost half) than CrossVivit? Following my intuition, it should have more, as it has more MLP heads. Were the hyperparameters (i.e., number of layers, etc) different?\n\nWe tried to train the Multi quantile version with the same number of parameters, and the performance was simply a bit lower. The version shown in the paper, with different hyperparameters indeed (using smaller number of layers and smaller dimensionality as shown in the appendix), seemed to work better. Please note that in the case of this version, the goal is not to offer the best median prediction, but rather the best confidence interval, meaning with a highest probability of including the observed values, which was offered by the smaller version. For completeness however, we added the results of the larger model, with the same hyperparameters as the normal CrossViViT, in the tables. This version has 145.5M parameters (a little bit more than the normal CrossViViT, since, as pointed out, it has more MLP heads).\n\n> How applicable is the method compared to related work? In particular with respect to the runtime and inference time. Can the authors provide some numbers on the runtime compared to other approaches\n\nBelow, we present a table with different inference and training metrics:\n\n| Model                    | Mean Latency (ms) | STDev. Latency (ms) | Giga MACs | Training time per epoch (s) |\n|--------------------------|-------------------|---------------------|-----------|-----------------------------|\n| Reformer                 | 5.43              | 0.32                | 1.66      | 387                         |\n| Informer                 | 8.63              | 0.30                | 2.54      | 623                         |\n| FiLM                     | 9.5               | 0.25                |           | 445                         |\n| PatchTST                 | 2.69              | 0.13                | 0.57      | 106                         |\n| LightTS                  | 0.82              | 0.087               | 0.0004    | 370                         |\n| CrossFormer              | 17.89             | 0.43                | 7.05      | 1300                        |\n| FEDFormer                | 66.41             | 0.63                | 1.03      | 1134                        |\n| DLinear                  | 0.29              | 0.007               | 0.00004   | 365                         |\n| AutoFormer               | 32.03             | 0.54                | 2.92      | 1500                        |\n| CrossViViT               | 65.03             | 0.43                | 180.47    | 18000                       |\n| CrossViViT MultiQuantile | 50.48             | 0.24                | 100.45    | 18000                       |\n\n*Latency: The time it takes for the model to process one instance (batch size=1)*\n*MAC: Number of multiply-accumulate operations. A multiply-accumulate operation corresponds to the operation a+(b*c) which counts as one operation.\nAs it can be observed on the table, the lightest models are indeed very fast and theoretically much faster than Ours, CrossViViT. However, all models have an inference time that lies within 1 second, and in practice it therefore does not make such a difference. \nWe would like to note within the context of a solar installation, day-ahead forecasting would be done once a day for each station, so as long as the inference time is not unreasonably high, which is the case here, it seems acceptable to sacrifice inference time for better performance, given that time-series models are less compute-hungry in comparison. \nGiven the previous comments, we suggest not including it in the main text, and instead in the appendix, to avoid making the tables too heavy. \n\n> Limitations\n\nA small note on this was added to the conclusion."}, {"review_id": "mlsNIPeNDn", "reviewer": "Reviewer_zf5t", "summary": "This submission presents a multimodal model for next-day solar irradiance prediction. They use time series of past irradiance and satellite image to predict irradiance 24h in advance. The model consists of one transformer branch for each modality and a shared (cross-modal) transformer. Their method can be used to predict uncertainty as well. They display improvements over the SOTA. Finally, they released  in open access a dataset acquired with 6 stations across 15 years.", "questions": "Q1) the cross-attention module needs to better explained. There are TxNp video tokens and T temporal tokens; how are they mixed? The authors use \"learned positional encoding\" for these tokens, but it is not explained how. Is the absolute spatiotemporal position encoded? The different sampling rates of eumetsat (5min) and bsrn (1h), yet they have the same number of observations T? \n\nQ2) What is the influence of ROPE compared to a more standard positional encoding? Since the authors make it an integral part of their method, its effect should be quantified in an ablation study\n\nQ3) we allow the model to mask a portion of the past time-series -> Why would the model want to do that? That can only decrease the train performance\n\nS1) The title shouldn't be a question, especially such a hyper-precise and niche one! It should be: Improved Day-Ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context or something in that vein", "limitations": "not provided", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is interesting, difficult, useful, and not a lot of work has been done with machine learning on the subject\n\n- The model architecture is reasonable\n\n- the authors compare their method to many baselines and competing methods\n\n- The authors provide a large-scale (at least temporally) dataset in open access", "weaknesses": "- Some details are missing, making it hard to understand how the method works precisely.\n\n- The uncertainty prediction with quantile lacks a proper evaluation, related work, and comparison baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful feedback and suggestions to improve the paper. We respond to the reviewer's comments below:\n\n> Some details are missing, making it hard to understand how the method works precisely.\n\nWe are not sure we understand what the reviewer means. What type of details are missing?\n\n> The uncertainty prediction with quantile lacks a proper evaluation, related work, and comparison baselines.\n\nThe Multi Quantile version serves as a supplementary version to offer a way to extract the uncertainty attached to the prediction, which can be implemented in most deep learning architectures, but is not the main contribution of the work, so we had to keep the results and comparison light in this regard. However, we do evaluate its median prediction (even though it is not meant as the definitive prediction) and its ability to include the observed true values. Do you have suggestions for more things we can include?\n\n> Q1) the cross-attention module needs to better explained. There are TxNp video tokens and T temporal tokens; how are they mixed?\n\nWe first mix tokens spatially, that is we have no regard for time initially which is implemented by simply reshaping the time dimension into the batch size. This leaves for each time-step mixing $N$ context tokens with one time-series token which is done using cross-attention and results in one token. We believe this was clearly stated in the methods section and illustrated in Figure 1, but if not, let us know how we can make it clearer.\n\n> The authors use \"learned positional encoding\" for these tokens, but it is not explained how. Is the absolute spatiotemporal position encoded?\n \nThe mixing is done first spatially by considering each time point separately.  We first mix the tokens spatially using cross-attention through Rotary Positional Encoding (RoPE) and then the resulting tokens are concatenated in a temporal sequence which is processed by a transformer that adds a learned positional encoding. This is similar to how Video ViT works, as described in the appendix.\n\n> The different sampling rates of eumetsat (5min) and bsrn (1h), yet they have the same number of observations T?\n\nThe original sampling rates of EUMETSAT is 5 min and BSRN is 1 minute, we down-sample them to 30min each, so that both context and time-series have a sampling rate of 30 minutes.\n\n> Q2) What is the influence of ROPE compared to a more standard positional encoding? Since the authors make it an integral part of their method, its effect should be quantified in an ablation study\n\nWe thank the reviewer for this observation. We did the ablation and will include the results in the paper. The findings summarized in the table below (which extends Table 1) suggest that RoPE does help improve the performance.\n\nOn CAB (2020-2022):\n\n| Model                   | MAE       | RMSE      | MAE Easy  | RMSE Easy | MAE Hard  | RMSE Hard  |\n|-------------------------|-----------|-----------|-----------|-----------|-----------|------------|\n| CrossViViT              | **50.35** | **99.18** | **47.04** | **89.60** | **55.30** | **112.00** |\n| CrossViViT without RoPE | 51.11     | 103.66    | **47.31** | 95.13     | 56.84     | 115.31     |\n\nOn TAM (2017-2019):\n\n| Model                   | MAE       | RMSE      | MAE Easy  | RMSE Easy | MAE Hard  | RMSE Hard  |\n|-------------------------|-----------|-----------|-----------|-----------|-----------|------------|\n| CrossViViT              | **49.46** | **94.96** | **44.01** | **79.91** | 97.40     | **179.30** |\n| CrossViViT without RoPE | 109.28    | 196.44    | 111.33    | 197.63    | **91.29** | 185.61     |\n\n> Q3) we allow the model to mask a portion of the past time-series -> Why would the model want to do that? That can only decrease the train performance\n\nThe idea behind the possibility of masking the past time series was to encourage the model to use the spatial context, rather than relying too much on the past time series, which is the first natural thing to do for the model. In particular, we wanted to prevent the model from simply repeating the past, making persistence-like predictions. \nInterestingly, we realized that even when masking entirely the past time series, the predictions were quite good, therefore showing that the model was indeed using the spatial context. In practice, however, the performance was still slightly better when keeping the time series masking to 0, so we kept it this way (while mentioning the possibility in the text and figures). \n\n> S1) The title shouldn't be a question, especially such a hyper-precise and niche one! It should be: Improved Day-Ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context or something in that vein.\n\nWe will change the title accordingly.\n\n> Limitations: not provided\n\nWe did mention a few limitations! Yet, it was rather limited, so we increased this part a little since the review. "}, {"review_id": "LjqXTEqTdn", "reviewer": "Reviewer_eJfQ", "summary": "The work presents a method to integrate information about cloud (using satellite images) with timeseries data related to Solar Irradiance to improve the solar irradiance forecasting.", "questions": "Fig.2 --> The TAM is outside the area of interest. This is described in the text but, since the table in fig. reports TAM but it is not present in the first satellite image, the authors should insert a comment in the caption.\n\nRow 230: It is not true for TAM as wrote in row 266\n\nFig.3 and Fig.4 contain some discussions of the results but it would be better if these comments were moved in a proper section in the text.\n\nIt could be interesting to consider a day-ahead situation, in other words considering the sliding window starting at 0:00 and ending at 23.00 of day-1 and compute the next day. In this way the most common day-ahead use case is tested.\n\nI think the satellite image are related to the same time of the time series. I didn't find this information in the text (or I missed it).\n\nSome details are not clear to me.\n1. the context image is used as a separated channel?\n2. all data from all stations are used for the training?\n3. What are effectively the inputs of the model, a list and a detailed encoding would be preferrable. I gave a look at the appendix, some details are present other not.\n4. what are the computational time required for the training of the proposed approach?\n\nAn interesting future perspective could be to incorporate the forecasting of atmosphere's state (images obtained by a Weather Prediction service provider). \n\nRow 51\nSound prediction ?", "limitations": "Some limitations have been identified and discussed by authors.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Here some interesting aspects of the paper:\n- the release of a new dataset containing both timeseries and satellite images for many years for several sites. \n- an attempt to build a multimodal architecture based on transformer.\n- usually the subdivision between sunny days and cloudy days is done in the forecasting works related to solar irradiance (e.g. pv production). Authors have proposed to subdivide the days in \"hard\" and \"easy\" based on the similarity between two consecutives days. I think this approach is interesting and help the fair assessment of the model.", "weaknesses": "The main weakness I see is that day-ahead use case is not explicitely evaluated.\nI know that the sliding window is more general but an important real word use case is to have a real day-ahead prediction. \nAuthors could test their algorithm on day-ahead use case (extracting properly the sliding windows of interest, i.e. from 0:00 to 23:00).\n\nMoreover, some details on the real input of the model seem missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The main weakness I see is that day-ahead use case is not explicitely evaluated. I know that the sliding window is more general but an important real word use case is to have a real day-ahead prediction. Authors could test their algorithm on day-ahead use case (extracting properly the sliding windows of interest, i.e. from 0:00 to 23:00).\n\nThank you for the suggestion, we did the test for day-ahead use case, practically selecting only the 00:00 to 23:00 windows when computing the metrics. The resulting table can be found in the rebuttal PDF.\n\nAs the reviewer can see, it only slightly decreases the absolute performance of all models but comparatively, CrossViViT still outperforms the baselines. While we agree that this demonstrates the performance in an operational setting, we believe it would fit better in the appendix rather than the main text.  \n\n> Moreover, some details on the real input of the model seem missing.\n\nWe are not sure we understand what the reviewer means. What type of details are missing?\n\n>**Fig.2 --> The TAM is outside the area of interest. This is described in the text but, since the table in fig. reports TAM but it is not present in the first satellite image, the authors should insert a comment in the caption.\n\nWe thank the reviewer for this suggestion and agree with the premise. To clarify the position of TAM (slightly) outside of the area of interest, we increased the area covered by Figure 2 top left figure, including TAM, but highlighting the area we are considering in red. We also inserted a comment in the caption. \n\n> Row 230: It is not true for TAM as wrote in row 266\n\nThis was corrected. \n\n> Fig.3 and Fig.4 contain some discussions of the results but it would be better if these comments were moved in a proper section in the text.\n\nMost comments are in the discussion part of the paper, yet we thought it would make for a better reading experience to also put some significant comments directly in the caption, with the figure of interest close by, rather than looking for the figure corresponding to the comment. It is in our opinion not uncommon to discuss results in captions. \n\n> I think the satellite image are related to the same time of the time series. I didn't find this information in the text (or I missed it).\n\nYes, the time series and the satellite images are aligned temporally. We clarified that in the text. \n\n> the context image is used as a separated channel?\n\nWhat we call the spatial context is the ensemble of satellite data, which include multiple video channels, as described in the satellite data section. There are 11 channels in total. \n\n> all data from all stations are used for the training?\n   \nAs mentioned in the paper: 3 stations (IZA, CNR, PAL) are used for training, 1 for validation (PAY) and the two remaining for testing (TAM and CAB). It allows us to evaluate the spatial generalization capabilities of the model.  \n  \n> What are effectively the inputs of the model, a list and a detailed encoding would be preferrable. I gave a look at the appendix, some details are present other not.\n\nWe presented all inputs in the method section and the type of data in the satellite data and time series section. Other details are indeed in the appendix. If you think more details are missing, we will be happy to add them in the appendix.\n\n> what are the computational time required for the training of the proposed approach?  \n\nThe table below highlights all the inference and training metrics for all models (Note that all the times presented there are for **one GPU**):\n\n|Model|Mean Latency (ms)|STDev. Latency (ms)|Giga MACs|Training time per epoch (s)|\n|---|---|---|---|---|\n|Reformer|5.43|0.32|1.66|387|\n|Informer|8.63|0.30|2.54|623|\n|FiLM|9.5|0.25||445|\n|PatchTST|2.69|0.13|0.57|106|\n|LightTS|0.82|0.087|0.0004|370|\n|CrossFormer|17.89|0.43|7.05|1300|\n|FEDFormer|66.41|0.63|1.03|1134|\n|DLinear|0.29|0.007|0.00004|365|\n|AutoFormer|32.03|0.54|2.92|1500|\n|CrossViViT|65.03|0.43|180.47|18000|\n|CrossViViT MultiQuantile|50.48|0.24|100.45|18000|\n\n*Latency: The time it takes for the model to process one instance (batch size=1)*\n*MAC: Number of multiply-accumulate operations. A multiply-accumulate operation corresponds to the operation a+(b*c) which counts as one operation.\n\nWe don't report MACs for FiLM since it's based on S4 model and the library we used doesn't support it. \n\n> An interesting future perspective could be to incorporate the forecasting of atmosphere's state (images obtained by a Weather Prediction service provider).\n\nIt is indeed in our plan for future work! Thanks for the suggestion. \n\n> Row 51 Sound prediction ?\n\nBy sound prediction we meant correct prediction; We replaced it in the text to avoid any confusion."}, {"review_id": "k3nAelKbB4", "reviewer": "Reviewer_WJ1J", "summary": "The work presents a multi-modal model, called CrossViViT, to perform day-ahead solar global horizontal irradiance predictions. In that, the model combines spatial information from satellites, i.e. RGB, IR and vapor channels, across Europe with time series information from six point-like stations, i.e. clear sky, pressure, direct normal irradiance, diffuse horizontal irradiance as well as a derived proxy global horizontal irradiance based on the Ineichen model. CrossViViT's performance has been compared again various other statistical and numerical models (Persistence and FFT) as well as other state-of-the-art deep learning approaches based on the transformer building blocks. The performance, measured in RMSE or MAE, ", "questions": "No direct question but an invitation to provide additional details or comments to the points raised in 'Weaknesses'", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Application case from natural sciences incl. challenging real-world problems of multi-modality and missing data\n- Distinction between 'easy' and 'hard' prediction cases; this is commonly overlooked\n- Open discussion of strength and limitations of CrossViViT in contrast to other method, in particular:\n    * Showcasing that it does not win across the board\n    * Improving the interesting 'hard' cases for domain applications", "weaknesses": "- The reviewer thinks that the evaluation in the manuscript could generally be improved with findings from other papers as follows:\n    * Normalize that forecasting values into a stated range (e.g. [0-1]) or state the value ranges otherwise an RMSE/MAE improvement by certain value cannot be put into a frame of reference\n    * Alternatively, consider reporting MAPE, mean average percentage, improvements instead that already includes that maximum range\n    * By extension the plots in Fig. 4 can be considered somewhat misleading as they do not show the minimum value 0 or give a magnitude of improvement\n- The authors dismiss the use of fourier-layers, e.g. AFNO, without clear substantiation\n- Since cross-attention is such an integral part for mixing tokens from the spatial and temporal domains, it would be meaningful to show the equation in your manuscript beyond the reference\n- It would be meaningful to extend the discussion of the findings towards the domain and/or real-world. What does it mean that you can achieve\n- Non-adherence to the conference paper template, fonts in tables are too small, figures are outside of the text margins\n- Color palette is difficult to read for color-blind people (red-green)\n- The study is not reproducible as there is no code given; there is no indication that it will be released in case of acceptance\n- Might be good to show the rough location, possibly with an arrow, for the TAM station on the map in Fig. 2\n- To the reviewer's personal taste: change the title from a question to a typical title like \"Solar Irradiance Time Series Forecasting with Spatio-Temporal Context\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Normalize that forecasting values into a stated range (e.g. [0-1]) or state the value ranges otherwise an RMSE/MAE improvement by certain value cannot be put into a frame of reference\n\nWe thank the reviewer for the suggestion. We think, however, that given that all the models are compared on the same setups and the same unobserved years for each station, the improvement of one model should be interpretable with respect to the others. However, if the reviewer means that normalizing would be helpful in order to compare the results across stations as well, we agree that it is a valid point. Yet, since the different stations have different patterns, we believe it is better to compare models for each station and not compare between stations, so that we keep in mind the order of magnitude of the values which can differ greatly from one station to another.\n\n> Alternatively, consider reporting MAPE, mean average percentage, improvements instead that already includes that maximum range\n\nIndeed, MAPE could be a good choice, but unfortunately it is problematic in our cases given that we are also considering nights through our sliding window training process, which involves ground truth values of 0, which cannot be handled by the MAPE by definition.\n\n> By extension the plots in Fig. 4 can be considered somewhat misleading as they do not show the minimum value 0 or give a magnitude of improvement\n\nThe goal of the radar plots on Fig 4 is to compare the three models appearing on them, regarding each of the metrics we present. This, in our opinion, is easily doable using the plots. What would be the advantage of showing the minimum value 0?\n\n> The authors dismiss the use of fourier-layers, e.g. AFNO, without clear substantiation\n\nDoes the reviewer mean mentioning fourier layers in related works, or using them within the architecture of CrossViViT? If it is the former, we did mention FNO, AFNO and GeoFNO in related works, as they are indeed very important works regarding weather prediction, PDE solving and video prediction in general. However, we feel that we wanted to build an architecture that only uses “basic” building blocks (including transformers, given the revolution they brought in our field) rather than already advanced architectures. There is of course a possibility to integrate many different existing methods to possibly improve the results, but we believe it is better to start without them at first. Furthermore, since we are not doing directly video prediction (although it would have been possible to predict the context as an auxiliary target, but we chose not to at first), we do not think fourier layers were necessarily first in line among all recent methods to improve our forecasting performance.\n\n> Since cross-attention is such an integral part for mixing tokens from the spatial and temporal domains, it would be meaningful to show the equation in your manuscript beyond the reference\n\nWe thank the reviewer for this comment, we added the equation to the manuscript.\n\n> It would be meaningful to extend the discussion of the findings towards the domain and/or real-world. What does it mean that you can achieve\n\nWe apply CrossViViT to a real-world problem of forecasting solar-irradiance which can be useful for mitigating climate change by encouraging the use of solar energy. We also mentioned in the introduction that CrossViViT can in principle be used to forecast any other physical variables.\n\n> Non-adherence to the conference paper template, fonts in tables are too small, figures are outside of the text margins\n\nWe modified the paper to fit everything within the text margins, except for Figure 3 showing prediction visualizations, which would unfortunately be unreadable if keep it within the margins; if it is a problem, we can put some of the visualizations in the supplementary material.\n\n> Color palette is difficult to read for color-blind people (red-green)\n\nWhen making the plots, we tried to make sure that they were color-blind friendly as one of our co-authors is color-blind as well (and they made the plots). Can you give suggestions of colors we can use instead?\n\n> The study is not reproducible as there is no code given; there is no indication that it will be released in case of acceptance\n\nYou can find the repository here: https://anonymous.4open.science/r/CrossViVit-57C2\n\n> Might be good to show the rough location, possibly with an arrow, for the TAM station on the map in Fig. 2\n\nFigure 2 was changed accordingly.\n\n> To the reviewer's personal taste: change the title from a question to a typical title like \"Solar Irradiance Time Series Forecasting with Spatio-Temporal Context\"\n\nThe title was changed accordingly."}]

Author rebuttal:
We thank the reviewers for their efforts and reviews of high quality. We are happy that they appreciated our contributions to the forecasting literature, including our architecture and the “easy and hard” cases evaluation.

We carefully considered your suggestions, and believed that it ultimately made our paper better; we therefore thank you again for it. We answered each reviewer separately, point by point, using the discussion tool. 

In a nutshell, these are the main points and additions made to the original manuscript:

- Ablated the RoPE Positional Encoding (replacing it by a learned PE) to measure its impact on the architecture:
On CAB (2020-2022):

| Model                   | MAE       | RMSE      | MAE Easy  | RMSE Easy | MAE Hard  | RMSE Hard  |
|-------------------------|-----------|-----------|-----------|-----------|-----------|------------|
| CrossViViT              | **50.35** | **99.18** | **47.04** | **89.60** | **55.30** | **112.00** |
| CrossViViT without RoPE | 51.11     | 103.66    | **47.31** | 95.13     | 56.84     | 115.31     |

On TAM (2017-2019):

| Model                   | MAE       | RMSE      | MAE Easy  | RMSE Easy | MAE Hard  | RMSE Hard  |
|-------------------------|-----------|-----------|-----------|-----------|-----------|------------|
| CrossViViT              | **49.46** | **94.96** | **44.01** | **79.91** | 97.40     | **179.30** |
| CrossViViT without RoPE | 109.28    | 196.44    | 111.33    | 197.63    | **91.29** | 185.61     |    
- Added results for a larger Multi-Quantile version of the model, matching the size of CrossViViT
On CAB (2020-2022):

| Model                             | Parameters | MAE   | $p_t$  | MAE Easy | $p_t$  | MAE Hard | $p_t$  |
|-----------------------------------|------------|-------|------|----------|------|----------|------|
| Multi-Quantile CrossViViT (small) | 78.8M      | 61.8  | 0.91 | 57.03    | 0.93 | 68.94    | 0.9  |
| Multi-Quantile CrossViViT (large) | 145.5M     | 74.26 | 0.89 | 68.83    | 0.91 | 82.39    | 0.87 |

On TAM (2017-2019):

| Model                             | Parameters | MAE   | $p_t$  | MAE Easy | $p_t$  | MAE Hard | $p_t$  |
|-----------------------------------|------------|-------|------|----------|------|----------|------|
| Multi-Quantile CrossViViT (small) | 78.8M      | 81.2  | 0.71 | 78.93    | 0.70 | 101.18    | 0.75  |
| Multi-Quantile CrossViViT (large) | 145.5M     | 79.73 | 0.76 | 76.08    | 0.76 | 111.74    | 0.75 |
- Added a table showing inference times of all models considered including our own, as well as training times 

|Model|Mean Latency (ms)|STDev. Latency (ms)|Giga MACs|Training time per epoch (s)|
|---|---|---|---|---|
|Reformer|5.43|0.32|1.66|387|
|Informer|8.63|0.30|2.54|623|
|FiLM|9.5|0.25||445|
|PatchTST|2.69|0.13|0.57|106|
|LightTS|0.82|0.087|0.0004|370|
|CrossFormer|17.89|0.43|7.05|1300|
|FEDFormer|66.41|0.63|1.03|1134|
|DLinear|0.29|0.007|0.00004|365|
|AutoFormer|32.03|0.54|2.92|1500|
|CrossViViT|65.03|0.43|180.47|18000|
|CrossViViT MultiQuantile|50.48|0.24|100.45|18000|
- Added CrossViViT results for day-ahead only cases. Due to the character limit, we refer the reviewers to the attached rebuttal PDF that contains results for CAB (2020-2022) and TAM (2017-2019).
- Corrected the text, added some details and made a few things clearer about the method in general. 
- Changed the name of the paper: *Improved Day-Ahead Solar Irradiance Time Series Forecasting by Leveraging Spatio-Temporal Context*.
- We added the anonymous repository to our code: https://anonymous.4open.science/r/CrossViVit-57C2/README.md 

Meta-review: This paper proposes a transformer-based model which leverages satellite data to provide the spatio-temporal context for solar irradiance time series forecasting. It also makes available a new multi-modal dataset that can benefit subsequent research by others.

We all agree that the application studied is important and challenging. The proposed transformer-based architecture is a reasonable design for the application. The distinction between easy and hard prediction cases, which is usually overlooked, seems to be a novel idea that is effective for the problem. One strength of this work is to release an important dataset that will likely arouse more interest in the application. Also, thorough experimental evaluation involving many baselines and out-of-domain generalization study is another strength of this work. Nevertheless, quite a few questions were raised by the reviewers. We appreciate the effort of the authors in answering the questions, helping to clarify many points. Further experiments were also conducted during the rebuttal phase to help answer some of the questions.

The authors are highly recommended to take the comments and suggestions of the reviewers into consideration when revising their paper.
Decision: Accept (poster)