SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The time-series measurements were obtained from Baseline Surface Radiation Network datasets (Driemel et al., 2018). The experiments in this paper use data from six locations (Figure 2), collected at a 30-minute resolution over a 15-year period (2008-2022). The data captures diverse patterns, ranging from consistent irradiance levels under clear skies to fluctuations caused by intermittent clouds affecting surface solar radiation. The data contains measurements of the pressure in the station, clear sky components, Direct Normal Irradiance (DNI), and Diffuse Horizontal Irradiance (DHI).
The GHI component was computed using the formula:  $GHI = DNI \times \cos z + DHI$  where  $z$  is the zenith angle of the sun obtained from the pvlib python library (Holmgren et al., 2020). The Ineichen model (Ineichen, 2016) available through the same library is utilized to obtain the clear sky components. The 2008-2016 data is used for training while subsets of the 2017-2022 data and stations are used for validation and test performance evaluations. In this study, the models are trained and evaluated to forecast the GHI component for a 24-hour period ahead, based on a history of 24-hour measurements.
Table 1: Comparison of model performances across test stations TAM and CAB, during test years (2020-2022) for CAB, and val years (2017-2019) for TAM. We report the MAE and RMSE for the easy and difficult splits presented in section 5.2 along with the number of data points for each split. We add the MAE resulting from the Multi-Quantile CrossViViT median prediction, along with  $p_t$ , the probability for the ground-truth to be included within the interval, averaged across time steps. Additionally, we ablate RoPE against a learned positional embedding.
Models	Parameters	CAB (2020-2022)	TAM (2017-2019)
All (9703)	Easy (5814)	Hard (3889)	All (2299)	Easy (2064)	Hard (235)
MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
Persistence	N/A	63.57	131.44	52.56	109.05	80.04	159.14	32.26	94.71	20.8	59.47	132.92	238.12
Fourier3	N/A	68.91	121.23	56.51	93.854	87.46	153.29	56.0	94.85	45.48	62.62	148.42	231.47
Fourier4	N/A	65.74	123.15	53.82	96.38	83.56	154.76	44.02	92.22	33.15	57.56	139.56	232.61
Fourier5	N/A	64.67	124.22	52.67	97.94	82.61	155.44	40.26	91.36	28.94	55.57	139.68	233.52
Clear Sky (Ineichen, 2016)	N/A	67.19	140.11	60.55	125.66	77.12	159.28	40.61	98.02	31.07	63.4	124.42	242.26
ReFormer (Kitaev et al., 2020)	8.6M	57.42	102.73	53.75	92.97	62.92	115.81	81.6	137.04	78.57	129.72	108.22	189.55
Informer (Zhou et al., 2021)	56.7M	72.26	122.89	70.85	118.85	74.35	128.69	83.43	140.38	82.6	138.46	90.66	156.22
FiLM (Zhou et al., 2022b)	9.4M	68.37	116.86	59.66	95.35	81.4	143.11	62.72	99.71	54.99	77.63	130.66	210.58
PatchTST (Nie et al., 2023)	9.6M	60.76	119.41	54.77	107.71	69.7	135.01	66.94	132.44	62.4	124.25	106.77	189.72
LighTS (Zhang et al., 2022)	32K	54.91	102.88	49.55	89.28	62.92	120.38	68.51	114.59	64.61	104.98	102.77	177.98
CrossFormer (Zhang and Yan, 2023)	227M	55.98	101.84	51.59	90.2	62.55	117.11	68.85	116.45	65.4	107.88	99.16	175.08
FEDFormer (Zhou et al., 2022a)	23.6M	56.38	99.27	53.08	90.13	61.31	111.54	92.12	146.52	91.13	142.83	100.82	175.64
DLinear (Zeng et al., 2022)	4.7K	75.01	121.01	65.21	99.72	89.65	147.21	75.54	115.40	69.04	98.74	132.67	211.28
AutoFormer (Wu et al., 2021)	50.4M	64.34	104.53	60.81	95.14	69.63	117.17	115.88	170.91	117.36	171.07	102.87	169.47
CrossViViT	145M	50.35	99.18	47.04	89.6	55.30	112.00	49.46	94.96	44.01	79.91	97.40	179.30
CrossViViT (Learned PE)2	145M	51.11	103.66	47.31	95.13	56.84	115.31	109.28	196.44	111.33	197.63	91.29	185.61
MAE	pt	MAE	pt	MAE	pt	MAE	pt	MAE	pt	MAE	pt
Multi-Quantile CrossViViT (small)	78.8M	61.80	0.91	57.03	0.93	68.94	0.90	81.20	0.71	78.93	0.70	101.18	0.75
Multi-Quantile CrossViViT (large)	145.5M	74.26	0.89	68.83	0.91	82.39	0.87	79.73	0.76	76.08	0.76	111.74	0.75
4.2 Satellite images
In this study, we utilize the EUMETSAT Rapid Scan Service (RSS) dataset (Holmlund, 2003), which spans a period of 15 years from 2008 to 2022, with an original resolution of 5 minutes, later aligned with the time series data. Our analysis focuses on the non-High Resolution Visible (non-HRV) channels, which encompass 11 spectral channels with a spatial resolution of  $6 - 9\mathrm{km}$  and provide comprehensive coverage of the upper third of the Earth, with a particular emphasis on Europe. These channels, including Infrared and Water vapor, offer valuable information for our investigation. To facilitate our analysis, we reprojected the original geostationary projection data onto the World Geodetic System 1984 (WGS 84) coordinate system (Jacob et al., 2022). The region of interest is depicted in Figure 2. Note that one of the 6 stations, TAM, lies slightly outside the region we consider, and therefore represents an out-of-distribution station in term of the context we use.
To augment the contextual information, we computed the optical flow for each channel using the TVL1 algorithm (Sánchez Pérez et al., 2013) from the OpenCV package (Bradski, 2000). The optical flow represents the "velocity" of pixels between consecutive frames, which in our case corresponds to the motion of clouds. Furthermore, we included the elevation map as an additional channel in our dataset. The pre-processed satellite data originally had a resolution of  $512^2$ , but for computational efficiency, we downscale it to  $64^2$ .
In this section, we outline the baselines utilized for comparison alongside the suggested architecture. We describe the experimental setup and present the results, benchmarking our framework against state-of-the-art forecasting models across various test configurations. Additionally, we employ a split methodology to assess model performance in challenging prediction scenarios, which hold significant implications for downstream tasks associated with solar irradiance estimation.
The models are trained using a dataset spanning a period of 9 years, from 2008 to 2016, encompassing the stations IZA, CNR, and PAL. Validation is performed on a separate dataset covering 3 years, from 2017 to 2019, for the PAY station. The TAM and CAB stations serve as the test dataset, consisting of 3 years from 2020 to 2022 for CAB, and from 2017 to 2019 for TAM. Each model takes a historical
2We replace RoPE with learnable spatial positional encoding. This was done by adding a learnable parameter  $\mathbf{p} \in \mathbb{R}^{N \times d}$  to the encoded input, where  $N$  is the number of tokens in each satellite frame.
input of 24 hours and predicts the GHI for the subsequent 24 hours, employing a sliding window approach. Details about the implementation and hyperparameter tuning can be found in the Appendix.
5.1 Baselines
We conduct a comprehensive comparison between our approach and several state-of-the-art deep learning architectures specifically designed for forecasting tasks. These architectures are explicitly mentioned in the results tables, such as Table 1. Additionally, we propose dummy baselines that are tailored to solar irradiance forecasting, namely:
**Persistence:** This baseline relies solely on the past day's time-series data, considering it as the prediction for future values.
Clear Sky baseline: This baseline uses the computable clear sky components of solar irradiance (Ineichen model (Ineichen, 2016)), which represent the total amount of irradiance that would reach the station in the absence of clouds.
Fourier approximations: We compute Fourier approximations over the previous day's time series, and apply a low-pass filter to keep a limited number of modes. We consider 3 baselines:  $\mathrm{Fourier}_3$ ,  $\mathrm{Fourier}_4$ , and  $\mathrm{Fourier}_5$ , corresponding to approximations with 3, 4, and 5 modes, respectively.
5.2 "Hard" vs. "Easy" forecasting scenarios
To assess the capability of the suggested model in capturing cloud-induced variability in GHI forecasts, we perform evaluations on test stations using various time splits, aiming to identify its strengths and limitations in comparison to previous approaches. This analysis allows us to pinpoint the specific scenarios where CrossViViT excels, as well as the areas where it falls short. Furthermore, it provides insights into the comparative strengths and weaknesses of previous approaches.
Given the favorable performance of the Persistence baseline when the GHI values exhibit similarity between consecutive days, we propose a time split approach that categorizes examples as either "Easy" or "Hard" based on the extent of GHI variation. The "Easy" examples entail minimal changes in GHI across consecutive days (the Persistence baseline works well), while the "Hard" examples involve significant variations (the Persistence baseline fails). To quantify the similarity, we employ a measure based on the ratio of the area under the GHI curve for the two days. In order to assign equal importance to ratios such as 0.5 (indicating GHI half that of the previous day) and 2 (indicating GHI double that of the previous day), we utilize the measure  $r = \left| \log \frac{y}{y_{\text{prev}}} \right|$ . Here,  $y$  represents the GHI over a 24-hour period, and  $y_{\text{prev}}$  represents the GHI over the previous 24 hours. Accordingly, we categorize cases as "Easy" when  $r < \left| \log \left( \frac{2}{3} \right) \right|$ , and "Hard" otherwise.
5.3 Performance on stations and years outside the training distribution
Table 1 presents a comprehensive comparison of the proposed CrossViViT approach with state-of-the-art timeseries models and dummy baselines. The evaluation is conducted on the test stations TAM and CAB, covering the periods 2017-2019 and 2020-2022, respectively. It is important to note that due to the unavailability of data for TAM during the 2020-2022 period, we perform the evaluation using the 2017-2019 data instead.
During the 2017-2019 period, CrossViViT achieves the lowest MAE compared to the time-series models on the TAM station. However, it is important to note that the persistence baseline still outperforms our approach. This performance disparity can be attributed to the characteristics of the TAM station, which is situated in a desert region characterized by predominantly clear and sunny days. As we incorporate cloud information, it may occasionally underestimate the GHI in such clear-sky conditions. Furthermore, the training dataset consists of data from only one "sunny" station located in the Canary Islands (IZA), limiting the availability of examples to effectively learn clear-sky patterns. Based on these results, one can see that for stations characterized by low irradiance intermittency, a combination of persistence and clear-sky models might be sufficient. For the 2020-2022 period on the CAB station, CrossViViT outperforms all baselines across different time splits. This notable improvement can be attributed to the specific meteorological conditions of the CAB station, which experiences a higher frequency of cloudy days. This aligns with the primary focus of our research, which aims to accurately forecast GHI under cloudbier conditions. Predictions visualisations can be
Figure 3: Prediction visualisations from CrossViViT for four examples in CAB station, on the 2020-2022 test period. (a) CrossViViT predictions. (b) Multi-Quantile CrossViViT median ( $q_{0.50}$  quantile) predictions with  $[q_{0.02}, q_{0.98}]$  prediction interval. (c) Predictions from a strong baseline, CrossFormer, (d) Fourier spectrum of the target, our prediction, and CrossFormer prediction. Figure (a) illustrates that CrossViViT closely aligns with the ground truth by effectively capturing cloud variations, whereas CrossFormer assumes a clear-sky pattern. This is confirmed by the Fourier spectra depicted in (d), where CrossFormer's spectrum exhibits a rapid decay in contrast to CrossViViT.
seen in Figure 3, along with the comparison of the fourier spectra of our prediction, the ground truth and a strong baseline, CrossFormer.
5.4 Zero-shot forecasting on unseen stations and in-distribution years
To evaluate the zero-shot capabilities of CrossViViT in terms of performance on unseen stations, we maintain the same time period as the training data, thereby emphasizing the spatial dimension of the analysis. The radar plots presented in Figure 4 demonstrate that our approach consistently outperforms the baselines across all splits for the CAB and PAY stations, which are characterized by higher cloud cover. However, it is noteworthy that Persistence remains competitive for the TAM station, particularly in the "easy" splits where irradiance variations are minimal throughout the day. This observation further underscores the efficacy of CrossViViT in accurately accounting for cloud conditions, particularly when examining the performance metrics of the "Hard" split.
5.5 Multi-Quantile results
Table 1 also shows results of the Multi-Quantile CrossViViT, including the MAE of the median prediction (the 0.5 quantile), along with the test confidence  $p_t$  obtained for the prediction interval: the probability for the ground-truth to be included within the interval, for each time step, averaged across the entire dataset considered. As for the other models, the evaluation is conducted on the test stations TAM and CAB, covering the periods 2017-2019 and 2020-2022, respectively. It is important to note that for this Multi-Quantile version, the goal is not to provide the best prediction from the median but rather to provide confident prediction intervals, with a high  $p_t$ , ideally close to  $96\%$ , which we theoretically should reach using 0.02 and 0.98 extreme quantiles. We include a small version of the Multi-Quantile model, and a large one, matching the number of parameters used for our Cross ViViT model. The small model provides the best performance.
The prediction intervals achieved a high level of confidence, surpassing 0.9, for the unseen CAB station. However, for the TAM station, the harsh environmental conditions of the desert posed a
(b)
Figure 4: Radar plots illustrating the comparative analysis of CrossViViT performance with Persistence and CrossFormer models, during training years (the in-distribution period from 2008 to 2016), on the unobserved stations of TAM, CAB, and PAY, respectively on (a), (b) and (c). CrossViViT demonstrates greater versatility compared to CrossFormer in forecasting under diverse conditions. Persistence struggles to accurately predict under challenging conditions, particularly on "Hard" days.
challenge for reliable estimation of prediction intervals, due to the abundance of clear sky components which is not seen during training. Although the median prediction results were comparatively inferior to those of a baseline method, it demonstrates consistent patterns in its variation across easy and difficult cases. Furthermore, the test confidence of the proposed method remains relatively constant across different splits, for both stations. Prediction visualisations can be seen on Figure 3.
5.6 Ablating the Rotary Positional Embedding
In order to assess the influence of RoPE, we ablate this design choice against a learned positional encoding which was the standard encoding used in ViViT. As Table 1 shows, RoPE contributes to the good performance in OOD stations and OOD years. Moreover, in the hard cases of the CAB station, RoPE outperforms the learnable positional encoding showcasing its importance in capturing cloud-induced variations.
6 Conclusion, Limitations, and Future Work
We present CrossViViT, an architecture for accurate day-ahead time-series forecasting of solar irradiance using the spatio-temporal context derived from satellite data. We suggest a testing scheme that captures model performance in crucial situations, such as days with varying cloudy conditions, and we enable the extraction of a distribution for each time step prediction. We also introduce a new multi-modal dataset that provides a diverse collection of solar irradiance and related physical variables from multiple geographically diverse solar stations. CrossViViT exhibits robust performance in solar irradiance forecasting, including zero-shot tests at unobserved solar stations during unobserved years, and holds great promise in promoting the effective integration of solar power into the grid.
However, there are some limitations to our study. Firstly, we would benefit to include more test years and stations to further validate the effectiveness of our approach. It would also be interesting to explore different past and future horizons to further evaluate the robustness of our model across different prediction and context horizons. The training time remains our key limitation since it takes close to 5 hours per epoch on a single GPU (see Appendix), but we plan on optimizing the architecture further in future work. Lastly, we plan to investigate the use of a cropping methodology on the context as a regularization method. Despite these limitations, our study clearly highlights the importance of incorporating spatio-temporal context in solar irradiance forecasting and demonstrates the potential of deep learning in addressing the challenges associated with variability in solar irradiance. Given the promising results observed in our current study, we intend to investigate the applicability of CrossViViT to other physical variables that are significantly influenced by their surrounding context.
This research has been funded by the Government of Quebec, with additional computational resources supplied by Mila (mila.quebec). It is conducted as part of an international partnership project between Mila, Moroccan agency for sustainable energy (Masen), l'UQAR, Polytechnique Montreal, l'Ecole Mohammadia d'Ingénieurs, and l'Institut de Valorisation des Données (IVADO). Our special thanks to MASEN R&D team for their precious collaboration. We extend our sincere appreciation to Jacob Bieker from Open Climate Fix, whose facilitation in accessing EUMETSAT data, coupled with his invaluable assistance regarding data availability, markedly enriched our study.
(2021). Net Zero by 2050: A Roadmap for the Global Energy Sector. OECD.
Alzahrani, A., Shamsi, P., Dagli, C., and Ferdowsi, M. (2017). Solar Irradiance Forecasting Using Deep Neural Networks. Procedia Computer Science, 114:304-313.
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. (2021). Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836-6846.
Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. arXiv preprint arXiv:1607.06450.
Bahdanau, D., Cho, K., and Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In Bengio, Y. and LeCun, Y., editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Baika, S. (2023). Meteorological synoptical observations from station Tamanrasset (2022-12). Backup Publisher: National Meteorological Office of Algeria Type: data set.
Bone, V., Pidgeon, J., Kearney, M., and Veeraragavan, A. (2018). Intra-hour direct normal irradiance forecasting through adaptive clear-sky modelling and cloud tracking. Solar Energy, 159:852-867.
Boussif, O., Bengio, Y., Benabbou, L., and Assouline, D. (2022). Magnet: Mesh agnostic neural pde solver. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Neural Information Processing Systems, volume 35, pages 31972-31985. Curran Associates, Inc.
Bouthillier, X., Tsirigotis, C., Corneau-Tremblay, F., Schweizer, T., Dong, L., Delaunay, P., Normandin, F., Bronzi, M., Suhubdy, D., Askari, R., Noukhovitch, M., Xue, C., Ortiz-Gagné, S., Breuleux, O., Bergeron, A., Bilaniuk, O., Bocco, S., Bertrand, H., Alain, G., Serdyuk, D., Henderson, P., Lamblin, P., and Beckham, C. (2022). Epistimio/orion: Asynchronous Distributed Hyperparameter Optimization.
Bradski, G. (2000). The OpenCV Library. Dr. Dobb's Journal of Software Tools.
Brandstetter, J., Worrall, D. E., and Welling, M. (2022). Message passing neural pde solvers. International Conference On Learning Representations.
Cuevas-Agulló, E. (2014). Radiosonde measurements from station Izana (2014-08). Backup Publisher: Izaña Atmospheric Research Center, Meteorological State Agency of Spain Type: data set.
Doblas-Reyes, F., Sorensson, A., Almazroui, M., Dosio, A., Gutowski, W., Haarsma, R., Hamdi, R., Hewitson, B., Kwon, W.-T., Lamptey, B., Maraun, D., Stephenson, T., Takayabu, I., Terray, L., Turner, A., and Zuo, Z. (2021). Linking Global to Regional Climate Change Supplementary Material.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
In this section, we present details about training CrossViViT and its Multi-Quantile variant, as well as the time-series baselines.
A.1 Training setup
A.1.1 CrossViViT
CrossViViT integrates two modalities: satellite video data and time-series station data. For both modalities, essential information such as geographic coordinates, elevation, and precise time-stamps is available. In this section, we provide a comprehensive explanation of the encoding process for each feature and conclude by presenting the hyperparameters of the model.
We first start by encoding the timestamps. For each time point, we have access to the following time features: The year, the month, the day, the hour and the minute at which the measurement was made. We use a cyclical embedding to encode these time features discarding the year. For a time feature  $x$ , its corresponding embedding can be expressed as:
$$
\left[ \sin \left(\frac {2 \pi x}{\omega (x)}\right), \cos \left(\frac {2 \pi x}{\omega (x)}\right) \right] \tag {8}
$$
Where  $\omega(x)$  is the frequency for time feature  $x$  (see Table 2) for the frequency of each time feature). We concatenate these features to form our final time embedding that we simply concatenate to the context and the time-series channels respectively. Furthermore, we incorporate the elevation data for each coordinate in both the context and the time-series. Specifically, the elevation values are concatenated to their corresponding channels in the context and time-series representations.
Regarding the geographic coordinates, we possess information regarding the latitude and longitude for both the context and the station. These coordinates are normalized so as to lie in  $[-1, 1]$ :
$$
\left\{ \begin{array}{l} \operatorname {l a t} \leftarrow 2 \left(\frac {\operatorname {l a t} + 9 0}{1 8 0}\right) - 1 \\ \operatorname {l o n} \leftarrow 2 \left(\frac {\operatorname {l o n} + 1 8 0}{3 6 0}\right) - 1 \end{array} \right. \tag {9}
$$
Table 2: Frequency of each time feature.
Time feature	Frequency
Month	12
Day	31
Hour	24
Minute	60
Rotary Positional Embedding Next, we embed these coordinates using Rotary Positional Embedding (RoPE) that we provide a PyTorch implementation for:
Rotary Positional Embedding
class AxialRotaryEmbedding(nnModule): def __init__(self, dim, max_freq): super().__init_.() self.dim = dim scales = torch.linspace(1.0, max_freq / 2, dim // 4) self.register_buffer("scales", scales) def forward(self, coords: torch.Tensor): "" Args: coords (torch.Tensor): Coordinates of shape [B, 2, height, width] "" seq_x = coords[:, 0, 0, :] seq_x = seq_x unsqueeze(-1) seq_y = coords[:, 1, :, 0] seq_y = seq_y unsqueeze(-1) scales = self~-1) scales = self~-1) seq_x = seq_x * scales * pi seq_y = seq_y * scales * pi x_sinu = repeat(seq_x, "b i d -> b i j d", j=seq_y.shape[1]) y_sinu = repeat(seq_y, "b j d -> b i j d", i=seq_x.shape[1]) sin = torch.cat((x_sinu.sin(), y_sinu.sin(), dim=-1) cos = torch.cat((x_sinu.cos(), y_sinu.cos(), dim=-1) sin, cos = map(Lambda t: rearrange(t, "b i j d -> b (i j) d"), (sin,  $\leftrightarrow$  cos)) sin, cos = map(Lambda t: repeat(t, "b n d -> b n (d j)", j=2), (sin,  $\leftrightarrow$  cos)) return sin, cos