SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Training configuration CrossViViT was trained on two RTX8000 GPUs, over 17 epochs with early stopping. Its Multi-Quantile variant was also trained on two RTX8000 GPUs, over 12 epochs with early stopping. The remaining settings are identical for both variants: An effective batch size of 20 was utilized for both models; The training process employed the AdamW optimizer (Loshchilov and Hutter, 2019) with a weight decay of 0.05; A cosine warmup strategy was implemented, gradually increasing the learning rate from 0 to 0.0016 over five epochs before starting the decay phase.
Below, we highlight the relevant model hyperparameters for CrossViViT and Multi-Quantile CrossViViT:
CrossViViT hyperparameters
1 patch_size: [8, 8]  
2 use glu: True  
3 max_freq: 128  
4 num_mlp_heads: 1  
5  
6 ctx_masking_ratio: 0.99  
7 ts_masking_ratio: 0  
8  
9 # These hyperparameters apply to the encoding transformers and cross-attention  
10 dim: 384  
11 depth: 16  
12 heads: 12  
13 mlp_ratio: 4  
14 dim_head: 64  
15 dropout: 0.4  
16  
17 # These only apply to the decoding transformer  
18 decoder_dim: 128  
19 decoder_depth: 4  
20 decoder_heads: 6  
21 decoder_dim_head: 128
Multi-Quantile CrossViViT hyperparameters
patch_size: [8, 8]  
use glu: True  
max_freq: 128  
num_mlp_heads: 11  
quantiles: [0.02, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.98]  
ctx_masking_ratio: 0.99  
ts_masking_ratio: 0  
# These hyperparameters apply to the encoding transformers and cross-attention  
dim: 256  
depth: 16  
heads: 12  
mlp_ratio: 4  
dim_head: 64  
dropout: 0.4  
# These only apply to the decoding transformer  
decoder_dim: 128  
decoder_depth: 4  
decoder_heads: 6  
decoder_dim_head: 128
A.1.2 Time-series baselines
We conducted training on a total of nine baseline models, and we emphasize the importance of the hyperparameters used for each of these models. Below, we provide a comprehensive overview of the hyperparameters employed in our study:
- seq_len: Input sequence length.
- label_len: Start token length.
- pred_len: Prediction sequence length.
- enc_in: Encoder input size.
dec_in: Decoder input size.
- e_layers: Number of encoder layers.
- d_layers: Number of decoder layers.
- c_out: Output size.
- d_model: Dimension of model.
- n_heads: Number of attention heads.
- d_ff: Dimension of Fully Connected Network.
- factor: Attention factor.
- embed: Time features encoding, options:[timeF, fixed, learned].
- distil: Whether to use distilling in encoder.
- moving average: Window size of moving average kernel.
We adapted the majority of the baselines using the Time Series Library (TSlib (Wu et al., 2023)), which served as a valuable resource in our experimentation. We refer the reader to the original papers, which served as a base for the hyperparameters utilized in our study, in order to have a comprehensive understanding of the models and the training settings.
LightTS
1 model:  
2 enc_in: 10  
3 seq_len: 48  
4 pred_len: 48  
5 d_model: 256  
6 dropout: 0.05  
7 chunk_size: 24
FiLM
1 model:  
2 enc_in: 10  
3 seq_len: 48  
4 label_len: 24  
5 pred_len: 48  
6 e_layers: 2  
7 ratio: 0.4
DLinear
1 model:  
2 enc_in: 10  
3 seq_len: 48  
4 pred_len: 48  
5 moving_avg: 25  
6 individual: False
Crossformer
1 model:  
2 enc_in: 10  
3 seq_len: 48  
4 pred_len: 48  
5 d_model: 1024  
6 n_heads: 2  
7 e_layers: 4  
8 d_ff: 2048  
9 factor: 10  
10 dropout: 0.01
Reformer
1 model:  
2 enc_in: 10  
3 c_out: 1  
4 seq_len: 48  
5 pred_len: 48  
6 d_model: 512  
7 n_heads: 8  
8 e_layers: 3  
9 d_ff: 2048  
10 factor: 5  
11 dropout: 0.05  
12 embed: timeF  
13 activation: gelu
PatchTST
1 model:  
2 enc_in: 10  
3 c_out: 1  
4 seq_len: 48  
5 pred_len: 48  
6 d_model: 1024  
7 n_heads: 6  
8 e_layers: 3  
9 d_ff: 2048  
10 factor: 10  
11 dropout: 0.05  
12 embed: timeF  
13 activation: gelu
The training of the baselines took place on a single RTX8000 GPU over the course of 100 epochs. During training, a batch size of 64 was consistently employed. For model optimization, we utilized the AdamW optimizer (Loshchilov and Hutter, 2019), incorporating a weight decay value set to 0.05. Moreover, we implemented a learning rate reduction strategy known as Reduce Learning Rate on Plateau, which gradually decreased the learning rate by a factor of 0.5 after a patience of 10 epochs.
For the hyperparameter tuning of the baselines, we employed the Orion package (Bouthillier et al., 2022), which is an asynchronous framework designed for black-box function optimization. Its primary objective is to serve as a meta-optimizer specifically tailored for machine learning models. As an example, we present the details of the Crossformer hyperparameter tuning scheme, showcasing the approach we followed to optimize its performance:
Crossformer hyperparameters tuning   
1 params:   
2 optimizer.lr: loguniform(1e-8, 0.1)   
3 optimizer.weight Decay: loguniform(1e-10, 1)   
4 d_model: choices([128,256,512,1024,2048])   
5 d_ff: choices([1024,2048,4096])   
6 n_heads: choices([1,2,4,8,16])   
7 e_layers: choices([1,2,3,4,5])   
8 dropout: choices([0.01,0.05,0.1,0.2,0.25])   
9 factor: choices([2,5,10])   
10 max_epochs: fidelity(low=5, high=100, base=3)
B Additional results and visualisations
B.1 Performance on (unobserved) validation PAY station, on validation and test years
This section presents a comprehensive comparative analysis that assesses the performance of CrossViViT in relation to various timeseries approaches and solar irradiance baselines specifically for the PAY station. The evaluation encompasses both the validation period (2017-2019) and the test period (2020-2022), with the validation period serving as the basis for model selection. The results, as presented in Table 3, offer compelling evidence of the superior performance of CrossViViT compared to the alternative approaches across all evaluation splits. These findings underscore the crucial role of accurately capturing cloud dynamics in solar irradiance forecasting, which is particularly pronounced in the "Hard" splits.
Table 3: Comparison of model performances in the val station PAY, during test years (2020-2022) and val years (2017-2019). We report the MAE and RMSE for the easy and difficult splits (presented in the main paper) along with the number of data points for each split. We add the MAE resulting from the Multi-Quantile CrossViViT median prediction, along with  $p_t$ , the probability for the ground-truth to be included within the interval, averaged across time steps.
Models	Parameters	PAY (Test years - 2020-2022)	PAY (Val years - 2017-2019)
All (12171)	Easy (8053)	Hard (4118)	All (27166)	Easy (16683)	Hard (10483)
MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
Persistence	N/A	61.9	140.05	44.98	105.47	94.99	190.31	63.61	141.16	48.57	108.91	87.54	180.99
Fourier3	N/A	69.21	129.29	52.17	91.61	102.53	181.63	71.71	130.58	56.03	94.81	96.68	172.86
Fourier4	N/A	65.67	131.13	48.77	93.68	98.72	183.48	67.35	132.41	51.8	96.71	92.08	174.79
Fourier5	N/A	63.39	132.38	46.14	95.08	97.14	184.71	65.71	133.87	50.03	98.61	90.66	175.98
Clear Sky (Ineichen, 2016)	N/A	65.99	138.78	56.41	116.72	84.72	174.03	62.12	131.69	52.59	109.91	77.28	160.36
ReFormer (Kitaev et al., 2020)	8.6M	59.15	109.75	51.67	91.67	73.79	138.44	59.3	109.3	54.06	94.31	67.64	129.62
Informer (Zhou et al., 2021)	56.7M	79.18	133.47	75.8	126.07	85.78	146.86	77.51	131.72	75.65	126.58	80.46	139.52
FiLM (Zhou et al., 2022b)	9.4M	69.87	124.86	55.93	91.15	97.11	172.72	71.1	125.32	58.35	94.17	91.39	163.06
PatchTST (Nie et al., 2023)	9.6M	63.21	131.64	53.06	113.78	83.06	160.93	63.88	131.14	55.47	115.59	77.25	152.64
LighTS (Zhang et al., 2022)	32K	58.57	111.22	48.93	88.69	77.4	145.53	58.22	110.84	50.72	91.24	70.16	136.33
CrossFormer (Zhang and Yan, 2023)	227M	59.64	111.03	51.04	90.34	76.45	143.08	59.39	111.49	52.55	93.15	70.29	135.66
FEDFormer (Zhou et al., 2022a)	23.6M	63.44	110.62	58.15	97.33	73.79	132.83	62.46	109.92	59.88	100.61	66.57	123.3
DLinear (Zeng et al., 2022)	4.7K	75.09	128.64	59.42	93.82	105.74	178.04	76.78	129.45	62.64	97.93	99.29	167.81
AutoFormer (Wu et al., 2021)	50.4M	73.36	117.22	68.89	105.52	82.12	137.25	71.39	114.96	69.42	106.79	74.54	126.88
CrossViViT	145M	51.47	107.73	41.59	85.14	70.81	141.87	52.02	108.77	44.71	90.47	63.65	132.79
MAE	pt	MAE	pt	MAE	pt	MAE	pt	MAE	pt	MAE	pt
Multi-Quantile CrossViViT (small)	78.8M	59.06	0.83	54.96	0.83	67.06	0.84	57.65	0.86	51.75	0.87	67.05	0.86
Multi-Quantile CrossViViT (large)	145.5M	74.18	0.89	68.69	0.91	82.38	0.87	59.97	0.87	52.67	0.88	71.59	0.85
B.2 Performance in the operational setting: 00:00 to 23:00 window only
In this section, we present an assessment of CrossViViT's performance in an operational context, specifically focusing on evaluations conducted within the 00:00 to 23:00 time window. This approach provides insights into the model's effectiveness in its intended real-world application, where it is typically executed once daily, precisely at midnight. This evaluation setup offers a more accurate representation of the model's practical utility compared to the sliding window approach. Table 4 shows that the relative performance of CrossViViT over baselines is still maintained which demonstrates the usefulness of our approach in real-world scenarios.
Table 4: Comparison of model performances in the test station CAB, during test years (2020-2022) and test station TAM on val years (2017-2019) for 00:00 to 23:00 windows only. We report the MAE and RMSE for the easy and hard splits along with the number of data points for each split.
Models	Parameters	CAB (2020-2022)	TAM (2017-2019)
All (207)	Easy (120)	Hard (87)	All (48)	Easy (42)	Hard (6)
MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE	MAE	RMSE
Persistence	N/A	63.19	130.5	49.91	103.23	81.49	160.7	33.94	98.15	19.75	53.91	133.32	238.15
Fourier3	N/A	69.49	122.68	54.9	90.38	89.63	156.67	57.7	99.09	44.57	58.76	149.62	233.2
Fourier4	N/A	65.78	123.84	51.71	92.03	85.19	157.5	45.55	95.98	31.95	52.41	140.79	233.4
Fourier5	N/A	64.55	124.51	50.24	93.08	84.3	157.9	41.91	94.95	27.79	49.58	140.77	234.33
Clear Sky (Ineichen, 2016)	N/A	67.31	140.42	58.68	121.49	79.21	162.96	40.93	99.03	29.42	55.82	121.5	237.99
ReFormer (Kitaev et al., 2020)	8.6M	60.64	104.52	55.98	92.28	67.07	119.35	81.5	134.89	77.95	126.06	106.36	185.29
Informer (Zhou et al., 2021)	56.7M	72.82	125.44	69.23	118.1	77.78	134.91	84.51	137.51	81.91	131.83	102.74	172.11
FILM (Zhou et al., 2022b)	9.4M	69.36	120.87	58.65	95.01	84.14	149.36	63.42	105.46	53.4	77.02	133.58	217.82
PatchTST (Nie et al., 2023)	9.6M	64.37	124.09	56.78	108.37	74.84	142.96	73.16	142.01	67.18	130.89	115.03	203.49
LighTS (Zhang et al., 2022)	32K	61.54	105.14	54.56	87.22	71.18	125.73	61.73	101.48	52.95	80.29	123.19	193.04
CrossFormer (Zhang and Yan, 2023)	227M	58.19	104.66	52.96	90.03	65.4	121.99	66.25	113.67	58.81	95.04	118.33	200.37
FEDFormer (Zhou et al., 2022a)	23.6M	59.11	104.56	52.88	88.58	67.71	123.25	114.47	176.24	116.59	176.21	99.62	176.42
DLinear (Zeng et al., 2022)	4.7K	77.36	125.17	65.61	100.19	93.57	153.08	79.36	121.69	70.71	99.87	139.9	220.55
AutoFormer (Wu et al., 2021)	50.4M	72.16	110.54	67.87	97.6	78.07	126.23	141.97	204.4	146.02	208.11	113.62	176.2
CrossViViT	145M	51.91	101.58	46.24	87	59.73	118.8	52.72	99.11	46.47	82.53	96.42	175.79
B.3 Performance evaluation with MAPE
In this section, we report the Mean Absolute Percentage Error (MAPE) $^3$  on test station CAB during test years (2020-2022) and test station TAM during val years (2017-2019) (see Table 5). The MAPE highlights even more the strength of our approach compared to the baselines. We also present results
<sup>3</sup>Since during the night, the GHI values are zero, we clamp the denominator by  $\epsilon = 1$  since the precision of the GHI values in the dataset is 1.
Table 5: Comparison of model performances across test stations TAM and CAB, during test years (2020-2022) for CAB, and val years (2017-2019) for TAM, using the Mean Absolute Percentage Error (MAPE) metric. We report the MAPE for the easy and difficult splits presented in section 5.2 along with the number of data points for each split. We add the MAPE resulting from the Multi-Quantile CrossViViT median prediction, along with  $p_t$ , the probability for the ground-truth to be included within the interval, averaged across time steps. Additionally, we ablate RoPE against a learned positional embedding.
Models	Parameters	CAB (2020-2022)	TAM (2017-2019)
All (9703)	Easy (5814)	Hard (3889)	All (2299)	Easy (2064)	Hard (235)
Persistence	N/A	0.54	0.37	0.8	0.14	0.08	0.67
Fourier3	N/A	8.43	8.22	8.76	19.98	20.37	16.6
Fourier4	N/A	5.42	5.58	5.18	7.54	7.4	8.83
Fourier5	N/A	4.21	4.17	4.26	7.55	7.44	8.47
Clear Sky (Ineichen, 2016)	N/A	0.72	0.53	1.02	0.28	0.21	0.9
ReFormer (Kitaev et al., 2020)	8.6M	4.05	3.7	4.59	5.88	5.89	5.78
Informer (Zhou et al., 2021)	56.7M	6.12	4.92	7.93	5.44	5.42	5.57
FiLM (Zhou et al., 2022b)	9.4M	8.88	9.18	8.45	11.62	11.64	11.43
PatchTST (Nie et al., 2023)	9.6M	2.53	2.4	2.74	3.32	3.29	3.59
LighTS (Zhang et al., 2022)	32K	7.08	7.08	7.08	12.28	11.82	16.29
CrossFormer (Zhang and Yan, 2023)	227M	3.07	2.76	3.54	3.57	3.54	3.86
FEDFormer (Zhou et al., 2022a)	23.6M	5.35	4.95	5.96	12.83	13.15	10.01
DLinear (Zeng et al., 2022)	4.7K	13.98	12.39	16.37	13.11	12.98	14.21
AutoFormer (Wu et al., 2021)	50.4M	10.47	9.77	11.51	23.33	24.18	15.91
CrossViViT	145M	0.43	0.33	0.58	0.17	0.14	0.47
CrossViViT (Learned PE)4	145M	0.83	0.64	1.13	0.95	0.99	0.6
MAPE	pt	MAPE	pt	MAPE	pt	MAPE	pt
Multi-Quantile CrossViViT (small)	78.8M	0.61	0.92	0.43	0.93	0.61	0.92	0.44	0.93
Multi-Quantile CrossViViT (large)	145.5M	0.69	0.89	0.49	0.91	0.99	0.89	0.49	0.91
of the ablated model for which we replace RoPE with a traditional learned Positional Encoding. Despite showing a better performance than most baselines, it does showcase a lower performance than the original CrossViViT, which underlines the importance of the choice of RoPE.
B.4 Inference times
Table 6 shows inference time metrics of CrossViViT and the time-series baselines and we do so by reporting two main metrics:
- Latency: The time it takes for the model to process one instance (batch size=1).
- MAC: Number of multiply-accumulate operations (MAC). A multiply-accumulate operation corresponds to  $a + (b \times c)$  which counts as one operation.
The above metrics have all been computed on a single RTX8000 GPU. From Table 6, we can see that CrossViViT and its multi-quantile variant have the longest inference and training times which is normal given that we incorporate two different modalities, one of which is usually very expensive to process (i.e. satellite data). But we argue that in an operational setting, CrossViViT only needs to be executed once a day which means that while it is desirable to have a relatively short inference time, it may not be of paramount importance, especially if such speed gains come at the expense of predictive accuracy. It is worth acknowledging that a noteworthy limitation lies in the training time, which could potentially be improved through optimization efforts in future work.
B.5 Visualisations for day ahead time series predictions
This section presents visualizations of predictions generated by CrossViViT and CrossFormer on the PAY station for both the validation period (2017-2019) (see Figure 6) and the test period (2020-2022) (see Figure 5). We also present visualizations of predictions generated by the Multi-Quantile version, for the two periods, on the PAY station. The predictions depicted in red are the median (50% quantile) estimation from the model, and the generated prediction interval is defined as the interval between the two predefined quantiles:  $[q_{0.02}, q_{0.98}]$
4We replace RoPE with learnable spatial positional encoding. This was done by adding a learnable parameter  $\mathbf{p}\in \mathbb{R}^{N\times d}$  to the encoded input, where  $N$  is the number of tokens in each satellite frame.
Figure 5: Visualizations of CrossViViT and CrossFormer are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the test period spanning from 2020 to 2022.
Figure 6: Visualizations of CrossViViT and CrossFormer are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2017 to 2019.
Figure 7: Visualizations of Multi-Quantile CrossViViT (median prediction and  $[q_{0.02}, q_{0.98}]$  prediction interval) and CrossFormer predictions are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2020 to 2022.
Figure 8: Visualizations of Multi-Quantile CrossViViT (median prediction and  $[q_{0.02}, q_{0.98}]$  prediction interval) and CrossFormer predictions are presented for a subset of 12 randomly selected days from the "Hard" split on the PAY station during the validation period spanning from 2017 to 2019.
Table 6: Comparison of models' inference metrics and training times. We report the mean latency and standard deviation in milliseconds as well as Giga MACs and the training time per epoch. All these metrics were computed for a single NVIDIA RTX8000 GPU.
Models	Latency (ms)	Giga MACs	Training time per epoch (s)
ReFormer (Kitaev et al., 2020)	5.43 ± 0.32	1.66	387
Informer (Zhou et al., 2021)	8.63 ± 0.30	2.54	623
FiLM (Zhou et al., 2022b)	9.5 ± 0.25	-	445
PatchTST (Nie et al., 2023)	2.69 ± 0.13	0.57	106
LighTS (Zhang et al., 2022)	0.82 ± 0.087	< 0.01	370
CrossFormer (Zhang and Yan, 2023)	17.89 ± 0.43	7.05	1300
FEDFormer (Zhou et al., 2022a)	66.41 ± 0.63	1.03	1134
DLinear (Zeng et al., 2022)	0.29 ± 0.007	< 0.01	365
AutoFormer (Wu et al., 2021)	32.03 ± 0.54	2.92	1500
CrossViViT (145M)	65.03 ± 0.43	180.47	18000
Multi-Quantile CrossViViT (78.8M)	50.48 ± 0.24	100.45	18000
C Vision Transformers (ViT) and Video Vision Transformers (ViViT)
ViT The Vision Transformer (ViT) model (Dosovitskiy et al., 2020) leverages self-attention mechanisms to efficiently process images, inspired by the popular transformer architecture (Bahdanau et al., 2015; Vaswani et al., 2017) notably based on the scaled dot product attention: given the query matrix  $Q$ , key matrix  $K$ , and value matrix  $V$ : Attention  $(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$  where  $d_k$  is the dimensionality of the keys. In the ViT model, the input image of dimensions  $H \times W$  is divided into non-overlapping patches  $x_i \in \mathbb{R}^{h \times w}$ , which are linearly projected and transformed into  $d$ -dimensional vector tokens  $z_i \in \mathbb{R}^d$  using a learned weight matrix  $\mathbf{E}$  that applies 2D convolution. The sequence of patches is defined as  $\mathbf{z} = [z_{\mathrm{class}}, \mathbf{E}x_1, \dots, \mathbf{E}x_P] + \mathbf{E}_{\mathrm{pos}}$ , where  $z_{\mathrm{class}}$  is an optional learned classification token representing the class label, and  $\mathbf{E}_{\mathrm{pos}} \in \mathbb{R}^{P \times d}$  is a 1D learned positional embedding that encodes position information.
To extract global features from the image, the embedded patches undergo  $K$  transformer layers. Each transformer layer  $k$  consists of Multi-Headed Self-Attention (MSA) (Vaswani et al., 2017), layer normalization (LN) (Ba et al., 2016), and MLP blocks with residual connections. The MLPs consist of two linear layers separated by the Gaussian Error Linear Unit (GELU) activation function (Hendrycks and Gimpel, 2016). The output of the final layer can be used for image classification, either by directly utilizing the classification token or by applying global average pooling to all tokens  $\mathbf{z}^L$  if the classification token was not used initially.
ViViT The Video Vision Transformer (Arnab et al., 2021) extends the ViT model to handle video classification tasks by incorporating both spatial and temporal dimensions within a transformer-like architecture. The authors propose multiple versions of the model, with a key consideration being how to embed video clips for attention computation.
Two approaches are presented: (1) Uniform frame sampling and (2) Tubelet embedding. The first method involves uniformly sampling  $n_t$  frames from the video clip, applying the same ViT embedding method Dosovitskiy et al. (2020) to each 2D frame independently, and concatenating the resulting tokens into a single sequence. The second method extracts non-overlapping spatio-temporal "tubes" from the input volume and linearly projects them into  $\mathbb{R}^d$ , effectively extending ViT's embedding technique to 3D data, akin to performing a 3D convolution. Intuitively, the second method allows for the fusion of spatio-temporal information during tokenization, whereas the first method requires independent temporal fusion post-tokenization. Experimental results, however, show only slightly superior performance for the second method in specific scenarios, despite its significantly higher computational complexity (Arnab et al., 2021).