# A simple method for quickly estimating solar irradiance forecast errors

Philippe Lauret, Mathieu David, Josselin Le Gal La Salle, Elke Lorenz, Richard Perez, Cyril Voyant

► To cite this version:

Philippe Lauret, Mathieu David, Josselin Le Gal La Salle, Elke Lorenz, Richard Perez, et al.. A simple method for quickly estimating solar irradiance forecast errors. Solar Energy, 2025, 301, pp.113821. 10.1016/j.solener.2025.113821. hal-05220664

HAL Id: hal-05220664

https://univ-reunion.hal.science/hal-05220664v1

Submitted on 23 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# A simple method for quickly estimating solar irradiance forecast errors

Philippe Lauret<sup>a ID,*</sup>, Mathieu David<sup>a ID</sup>, Josselin Le Gal La Salle<sup>a ID</sup>, Elke Lorenz<sup>b</sup>, Richard Perez<sup>c ID</sup>, Cyril Voyant<sup>d,e ID</sup>

## ARTICLE INFO

Dataset link: https://github.com/Laboratoire-Piment/solar-predict-rmse.git

**Keywords:**

Solar forecast error

Solar variability

Root Mean Square Error (RMSE)

Bounds of RMSE

Forecast verification

ECMWF

SURFRAD

## ABSTRACT

In this work, we develop simple linear models that allow users to predict solar irradiance forecast errors based solely on solar variability at a specific location on Earth. These straightforward yet actionable models enable solar forecasters to quickly estimate forecast errors for a given site, providing a clear indication of how well their forecasting models are likely to perform. The error in deterministic solar irradiance forecasts is measured by the Root Mean Square Error (RMSE), while solar variability is quantified by the standard deviation of an hourly time series of changes in the dimensionless clear sky index.

Sixty sites distributed around the globe are used to build two types of RMSE prediction models. The first type is for intra-day forecasts (1-hour to 6-hour forecast horizons), while the second is for day-ahead forecasts (24-hour horizon). The derivation of the intra-day forecast error prediction model leverages on a non-linear time series approach whereas the one for day-ahead forecast error relies on forecasts issued by the European Centre for Medium-Range Weather Forecasts (ECMWF). For each type of model, we calculate also the 2.5% and 97.5% percentiles of the distribution in order to estimate the 95% uncertainty interval associated with the prediction. This uncertainty interval defines the bounds of the RMSE within which 95% of future RMSE values are expected to fall. These error bounds can provide solar forecasters with valuable insights into the performance of their solar forecasting methods in relation to the forecast challenges posed by site-specific variability.

Verification against published results in the literature, specifically for seven sites of the SURFRAD network, demonstrates that these models can satisfactorily predict intra-day and day-ahead forecast RMSEs using only site-specific solar variability data.

## 1. Introduction

In the solar forecasting community, it is implicitly accepted that forecast difficulty is directly linked to the solar variability experienced at a particular site [1–3]. However, to the best of our knowledge, no empirical evidence explicitly demonstrates a statistical dependence, let alone a correlation, between the solar variability at a specific site and the forecast error. Such a correlation, expressed as a parameterized model that predicts forecast error in relation to solar variability, could be valuable for solar forecasters during a verification procedure. More specifically, such an approach may allow solar forecasters to

quickly verify the consistency of the predictions generated by their forecasting models. Surprisingly, to the best of our knowledge, only one study [4] has proposed an approach to determine the correlation between forecasting error and a set of 18 statistical parameters. This study was conducted at 8 locations and concluded that there is a statistical dependence between forecast error (defined as either normalized RMSE or normalized MAE) and a statistical parameter from the field of econometrics: the mean absolute log return. In their conclusion, the authors suggested that the study should be extended to more locations. A recent work by Karimi-Arpanahi et al. (2023) proposed a systematic framework for quantifying the forecast difficulty of renewable generation time series. By evaluating several entropy-based metrics, they identified Weighted Permutation Entropy (WPE) as the most effective

<footnote><sup>a</sup> University of La Réunion - PIMENT laboratory, 15, avenue René Cassin, 97715 Saint-Denis, France</footnote>

<footnote><sup>b</sup> Fraunhofer Institute for Solar Energy Systems ISE, Heidenhofstr. 2, 79110 Freiburg, Germany</footnote>

<footnote><sup>c</sup> State University of New York, Albany, United States</footnote>

<footnote><sup>d</sup> Sciences for Environment, UMR CNRS 6134, University of Corsica Pasquale Paoli, Ajaccio, France</footnote>

<footnote><sup>e</sup> Center for Observation, Impacts, Energy, MINES ParisTech, Sophia Antipolis Cedex, France</footnote>

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: philippe.lauret@univ-reunion.fr (P. Lauret), mathieu.david@univ-reunion.fr (M. David), josselin.le-gal-la-salle@univ-reunion.fr (J.L.G.L. Salle), elke.lorenz@ise.fraunhofer.de (E. Lorenz), solarperez@gmail.com (R. Perez), cyril.voyant@minesparis.psl.eu (C. Voyant).</footnote>
indicator across different forecast models and horizons. Their analysis — based on more than 300 PV systems in Australia — showed that WPE correlates well with forecast error and can guide system planning and market operations. However, calculating WPE requires tuning a hyperparameter and it has no clear physical interpretation, making it less accessible to practitioners.

In addition, an uncertainty interval that bounds the prediction of the forecast error would be also highly valuable. As noted by Yang [5], knowledge of forecast error bounds (e.g., the bounds of RMSE in this case) is crucial for forecast verification. Indeed, by evaluating the forecasts of interest to the RMSE bounds estimated using data from the same location and ideally over the same time period, one gains insight into how the forecasts perform relative to the inherent difficulty of the forecasting situation. In his work [5], the upper bound of the RMSE is given by the performance of a reference model based on a optimal convex combination of climatology and persistence while the lower bound is approximated by the predictability error growth, which is the difference between the controlled and perturbed forecasts from a perfect NWP model.

It should be noted that Yang [5] discussed the notion of predictability. While related, the concepts of forecast error estimation and predictability serve different purposes. This study does not aim to define or quantify predictability in the theoretical sense — typically involving reference models or error bounds — but instead focuses on a practical, data-driven estimation of expected RMSE based on observed solar variability. This allows us to assess the intrinsic difficulty of forecasting at a site without relying on a formal predictability framework.

Therefore, in this paper, we will not address this concept of predictability. For more details regarding predictability, the interested reader is referred to Karimi-Arpanahi et al. [6], Liu et al. [7], Liu et al. [8], Yang [5], Voyant et al. [9], Yang et al. [10], Pedro and Coimbra [11].

In this study, we first hypothesize a correlation between site hourly variability measured by the standard deviation of an hourly time series of the change in clear sky indices and forecast error given by the Root Mean Square Error (RMSE). Secondly, we build two linear models that enable users to predict intra-day or day-ahead forecast errors given the site variability at any location worldwide. Indeed, with the sole knowledge of solar variability, it might be useful to estimate the most likely forecast error before implementing any (possibly complex) solar forecasting methodology. In other words, the proposed method aims to estimate the expected RMSE of solar forecasts at a given site using only a measure of solar variability. This makes it particularly suitable for pre-deployment applications, as it allows practitioners to assess the intrinsic forecast difficulty of a site before any forecasting system is implemented.

The first model is for intra-day forecasts (1 h–6 h forecast horizon with a 1 h time step) while the second one is for day-ahead forecasts (24 h forecast horizon). A 95% uncertainty interval is associated with the models’ predictions, defining in this work the RMSE bounds within which 95% of future RMSE values are expected to lie. These error bounds can offer solar forecasters valuable insights into the performance of their forecasting methods, particularly in addressing the forecast challenges posed by site-specific variability.

The building of the linear prediction models requires intra-day forecasts provided by a nonlinear Neural Network time series model [1] and day-ahead forecasts issued by ECMWF. Sixty locations with quality-controlled Global Horizontal Irradiance (GHI) provided by subtask 1 of IEA PVPS T16 [12,13] are used to establish the linear correlation, in the Pearson sense, between the RMSE forecast error and solar variability. Verification of the models is done by comparing their predictions with published results from the literature.

The rest of the paper is organized as follows. First, Section 2 highlights the correlation between forecast error and solar variability. Section 3 details the derivation of the forecast error prediction models while Section 4 proposes a verification of the different models. Section 5 discusses the usefulness and limitations of the model. Finally, Section 6 concludes this paper.

## 2. Variability, forecast error and correlation between the two metrics

This work focuses on solar irradiance forecasts as Global Horizontal Irradiance (GHI) is the main driver of PV power generation. Let us recall that, in the field of solar forecasting, it is a standard procedure to use a clear sky model to remove daily and annual seasonality of the GHI. Specifically, a new des seasonalized series, known as the clear sky index $k_c$ time series, is derived by employing the following data transformation

$$k_c = \frac{I}{I_c} \qquad (1)$$

where I is the measured global horizontal irradiance and $I_c$ is the output of a clear sky model. Here, we will use the McClear clear sky model [14].

Solar irradiance variability refers to fluctuations in the amount of solar energy received at a specific location on the Earth’s surface over time. This variability is due to factors such as cloud movement, atmospheric conditions, diurnal cycles and seasonal changes, all of which have an impact on the efficiency of solar energy production.

Some researchers proposed metrics to characterize the variability e.g., Perez et al. [15], Coimbra et al. [16]. However, in this paper, we will use the nominal variability proposed by Perez et al. [17] which is defined as

$$\sigma(\Delta k_c) = \sqrt{\mathrm{VAR}(\Delta k_c)} \qquad (2)$$

Here $\Delta k_c = k_c(t) - k_c(t - \Delta T)$ with $\Delta T = 1h$, i.e., $\Delta k_c$ represents the hourly changes of the dimensionless clear sky index $k_c$. In this work, a time span of 1 year is used to calculate the variance (VAR) or equivalently the standard deviation $\sigma$ of the time series of $\Delta k_c$. This definition is appealing as $\Delta k_c$ is directly linked to the ramp rate’s magnitude [17]. More precisely, this choice is motivated by the physical behavior of cloud movements, which often cause rapid, localized fluctuations in irradiance. The first-order difference captures these short-term dynamics, while the standard deviation provides a summary measure of their intensity. The work of Perez et al. (2016) showed that this metric effectively distinguishes between stable and highly variable sites, even when average clearness conditions are similar, and that forecast performance is highly sensitive to such short-term variability—particularly under partly cloudy conditions. Our method builds directly upon this foundation: we use $\sigma(\Delta k_c)$ as a proxy for expected forecast error, enabling practitioners to assess site-level forecast difficulty using historical data alone.

Moreover, the advantage of using this definition of variability is that it allows for the use of GHI satellite gridded-data instead of ground truth data, to compute the nominal variability, as the proposed metric relies on the difference between two consecutive clear sky indices.

Regarding the verification of forecasting methods that generate deterministic or point forecasts $\hat{I}$, we retain the proposition of Yang et al. [10] to measure the forecast accuracy by using the Root Mean Square Error (RMSE). For N pairs of GHI forecasts/observations ($\hat{I}/I$), RMSE reads as

$$\mathrm{RMSE} = \sqrt{\frac{1}{N} \sum_{n=1}^{N} (\hat{I}_n - I_n)^2} \qquad (3)$$

As mentioned in the introduction, we hypothesize a correlation between nominal variability and forecast error measured by the RMSE. In order to highlight this correlation, we propose to evaluate the forecast errors of (i) intra-day forecasts (1 h–6 h forecast horizon) provided by a time series Neural Network model and (ii) day-ahead forecasts (24 h forecast horizon) issued by the European Centre for Medium-Range Weather Forecasts (ECMWF). For 60 ground stations representing diverse climatic conditions (see Appendix A and Fig. 1), we compute their nominal variability using Eq. (2) and the RMSE (Eq. (3)) of the deterministic intra-day and day-ahead forecasts generated for these locations. Note that, to calculate the clear sky index ($k_c$), we extracted time series of McClear clear sky estimates from the CAMS website [18].
<figure><img src="image_4.png" /><figcaption>Fig. 1. Location of the sites used to build the models. The reader is referred to Table 5 for the complete name of the stations.</figcaption></figure>

<table><caption>**Table 1**
Equations of the forecast error prediction models: *x* represents the nominal variability σ(Δk_c) and *y* the forecast error (RMSE).</caption><thead><tr><th>Model Name</th><th>Equation</th></tr></thead><tbody><tr><td>ID-1h (Intra-day 1 h)</td><td>2.5%: y = 416.46x - 5.68<br>97.5%: y = 747.18x - 19.56<br>regression: y = 558.43x - 10.28</td></tr><tr><td>ID-2h (Intra-day 2 h)</td><td>2.5%: y = 534.16x - 6.71<br>97.5%: y = 840.77x - 7.40<br>regression: y = 683.69x - 7.80</td></tr><tr><td>ID-3h (Intra-day 3 h)</td><td>2.5%: y = 597.18x - 7.04<br>97.5%: y = 841.74x + 4.26<br>regression: y = 748.57x - 6.21</td></tr><tr><td>ID-4h (Intra-day 4 h)</td><td>2.5%: y = 604.18x - 5.77<br>97.5%: y = 719.43x + 30.09<br>regression: y = 785.03x - 4.30</td></tr><tr><td>ID-5h (Intra-day 5 h)</td><td>2.5%: y = 633.91x - 9.50<br>97.5%: y = 808.20x + 23.34<br>regression: y = 817.80x - 4.84</td></tr><tr><td>ID-6h (Intra-day 6 h)</td><td>2.5%: y = 647.41x - 11.29<br>97.5%: y = 880.59x + 17.58<br>regression: y = 838.37x - 5.37</td></tr><tr><td>D24 - day-ahead model</td><td>2.5%: y = 406.53x + 10.21<br>97.5%: y = 767.26x + 18.02<br>regression: y = 661.48x - 2.12</td></tr></tbody></table>

## 2.1. Intra-day forecasts

To highlight the correlation between the RMSE of intra-day forecasts and nominal variability, we employ the following procedure for each site listed in Appendix A:

1. For each forecast horizon from 1 h to 6 h, we train six nonlinear Neural Networks (NN) time series models as depicted in Lauret et al. [1]. Year 1 of Table 5 in Appendix A is used for training the NN model.

2. We compute the RMSE of the NN models and nominal variability on the testing year (Year 2 of Table 5).

3. For each forecast horizon, scatter plots of the RMSE of the NN model and nominal variability computed on the testing year (Year 2) are displayed.

Note that for computing RMSE and variability over 1 year, GHI data for which solar elevation < 10° are filtered out.

Fig. 2a–f plot the forecast error (RMSE) vs nominal variability σ(Δk_c) for the 60 sites under study. As shown, a clear positive correlation (Pearson correlation between 0.82 and 0.84) between RMSE and nominal variability is exhibited.

## 2.2. Day-ahead forecasts

For day-ahead forecasts, HRES operational GHI forecasts at the 00Z run from ECMWF were extracted from the Meteorological Archival and Retrieval System (MARS). Since forecasts may not be available for the specific locations listed in Table 5, we used the forecast corresponding to the closest grid point to those locations. The RMSE of these day-ahead forecasts and the nominal variability were computed on the testing year (Year 2) of Table 5. Fig. 2g shows the scatter plot of the RMSE of day-ahead forecasts vs nominal variability. Again a statistically significant correlation of 0.72 is found for this type of forecasts.

Based on the above experimental evidence, we propose in the following section to derive two types of models: one for intra-day forecasts and the other for day-ahead forecasts. These models will predict the forecast error (RMSE) in relation to the nominal variability (σΔk_c).

## 3. Model derivations

For both types of forecasts, we hypothesize a linear correlation<fnref n="1" /> between RMSE and variability and therefore build a linear regression model. Also, we compute the conditional quantiles of the distribution (RMSE - σ(Δk_c)) at nominal probabilities of 2.5% and 97.5%. This permits estimating the 95% uncertainty interval associated with each prediction and therefore provides the RMSE bounds of the prediction models. Table 1 lists the equations of the regression models as well as the equations for the conditional quantiles at nominal probabilities of 2.5% and 97.5%. The prediction models denoted ID-1h, ..., ID-6h in Table 1 are for intra-day forecasts while model denoted D24 in

<footnote n="1">¹ An exponential fit was also tested but yielded similar results in terms of R².</footnote>
<figure><img src="image_5.png" /><figcaption>(a) Timeseries NN Model : 1h forecast horizon</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Timeseries NN Model : 2h forecast horizon</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) Timeseries NN Model : 3h forecast horizon</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(d) Timeseries NN Model : 4h forecast horizon</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(e) Timeseries NN Model : 5h forecast horizon</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(f) Timeseries NN Model : 6h forecast horizon</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(g) ECMWF day-ahead forecasts</figcaption></figure>

**Fig. 2.** Scatter plots of forecast error vs. nominal variability. The high correlation coefficients indicate a strong positive linear relationship between nominal variability and RMSE. The p-value being extremely low suggests that the correlations are statistically significant.

Table 1 is for day-ahead forecasts. By applying these simple models, the user can quickly obtain an initial estimate of the deterministic RMSE forecast error based on the site's variability.

Fig. 3a–f show the predictions of models ID-1h to ID-6h and Fig. 3g the day-ahead forecast error estimated by model D24. For each case, the conditional quantiles at nominal probabilities of 2.5% and 97.5% are

also plotted. In these figures, in addition to the regression model that estimates the conditional expectation (or mean) of RMSE given $\sigma(\Delta k_c)$, we also plot the conditional median. As observed, for all prediction models, both the regression line and the median are practically equal meaning that the distribution (RMSE - $\sigma(\Delta k_c)$) is almost symmetric and not subject to outliers.
<figure><img src="image_12.png" /><figcaption>(a) Intra-day RMSE prediction model ID-1H</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) Intra-day RMSE prediction model ID-2H</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>(c) Intra-day RMSE prediction model ID-3H</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(d) Intra-day RMSE prediction model ID-4H</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(e) Intra-day RMSE prediction model ID-5H</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(f) Intra-day RMSE prediction model ID-1-6H</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>(g) Day-ahead RMSE prediction model D24</figcaption></figure>

**Fig. 3.** Regression models and associated quantiles. For the intra-day regression models, the *R*² values between 0.67 and 0.71 indicates a relatively strong relationship between the independent and dependent variables while for day-ahead the value of 0.52 suggests a moderate fit.

By sorting the 60 sites by increasing variability, Fig. 4 provides another perspective on the predictions made by model D24. For each site, the reference RMSE from the day-ahead forecast produced by ECMWF is plotted as a black cross.

For most sites, the D24 model predicts the RMSE reasonably well, and, as expected, the uncertainty interval (which here defines the RMSE bounds) increases with the site's variability. Some deviations between D24 model predictions and reference RMSEs are observed at certain sites, but overall, the RMSEs issued by ECMWF remain within the

uncertainty bounds. Furthermore, this figure illustrates the substantial range of RMSEs estimated by the proposed model.

## 4. Verification of the models

To assert the relevance of the proposed models, we conduct three levels of verification. First, we compare the intra-day RMSE predictions (models ID-1h to ID-6h) with published results that pertain only to insular sites (see Table 2). Second, day-ahead predictions (model
<figure><img src="image_19.png" /><figcaption>Fig. 4. Application of the day-ahead prediction model D24 to the 60 sites. The reference RMSEs are those generated by the IFS ECMWF forecast model.</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Fig. 5. Test Sites locations. The reader is referred to Tables 2–4 for the complete name of the stations.</figcaption></figure>

D24) are tested against the operational day-ahead forecast issued by ECMWF on 10 sites listed in Table 3. Finally, the RMSE predictions of models ID-1h, ..., ID-6h and D24 are evaluated on seven sites of the Surface Radiation Budget monitoring network (SURFRAD). Many solar forecasters have used this network to implement or verify their models. In particular, we can cite the work of Perez et al. [19], Perez et al. [20], Yang [5], Zhang et al. [21] that used this database to generate intra-day and day-ahead deterministic forecasts. Table 4 lists the main characteristics of the SURFRAD stations.

Fig. 5 shows the location of the twenty-two sites used in the verification procedure.

## 4.1. Verification of the intra-day prediction models ID-1h/ID-6h on insular sites

We proceed first on the verification of the intra-day models (ID-1h, ..., ID-6h) by assessing their predictions against reported RMSE of intra-day forecasts of the literature. The forecasts are related to insular sites (see Table 2). To predict the RMSE, we rely on the regression models of Table 1 that take as input the nominal variability listed in Table 2.

For the insular stations AJA, RAI and SPI, the reported forecasts are generated by a nonlinear Gaussian Process technique described in Lauret et al. [1]. The models used only 5 past ground GHI measurements to generate the forecasts. For stations LAS and POZ, NN models are

<caption>**Table 2**</caption>

Verification of the RMSE predictions of models ID-1h, ..., ID-6h on insular sites. Column $\sigma(\Delta K_c)$ is the nominal variability used to calculate the RMSE predictions.

<table><thead><tr><th>Site</th><th>Code</th><th>Latitude</th><th>Longitude</th><th>Altitude</th><th>σ(ΔK<sub>c</sub>)</th></tr></thead><tbody><tr><td>Ajaccio</td><td>AJA</td><td>41.917</td><td>8.733</td><td>4</td><td>0.194</td></tr><tr><td>Las Palmas</td><td>LAS</td><td>28.1108</td><td>5.4269</td><td>17</td><td>0.175</td></tr><tr><td>Pozo Izquierdo</td><td>POZ</td><td>27.8175</td><td>15.4244</td><td>47</td><td>0.141</td></tr><tr><td>Le Raizet</td><td>RAI</td><td>16.264</td><td>-61.516</td><td>11</td><td>0.237</td></tr><tr><td>Saint-Pierre</td><td>SPI</td><td>-21.34</td><td>55.491</td><td>75</td><td>0.188</td></tr></tbody></table>

designed to produce intra-day solar forecasts using ground and exogenous data. Ground data were obtained from two measurement stations in Gran Canaria Island. Exogenous data are satellite GHI data (from Helioclim-3) as well as solar radiation and Total Cloud Cover forecasts provided by ECMWF. The interested reader is referred to Lauret et al. [1], Aguiar et al. [22] for details regarding the implementation and the performance in terms of RMSE of the forecasting models selected in this section.

As depicted in Fig. 6, irrespective of location, all reported RMSE values are close to those predicted by models ID-1H, ..., ID-6H. Overall, a ±25 W/m² difference is observed between reported RMSEs and predictions made by the regression models. Moreover, these reported RMSEs fall within the 95% uncertainty interval.
<figure><img src="image_21.png" /><figcaption>Fig. 6. Verification of the proposed models for the intra-day case.</figcaption></figure>

## 4.2. Verification of the day-ahead prediction model D24

In this subsection, we focus on the day-ahead RMSE predictions made by model D24. For 10 sites provided by sub task 1 of IEA PVPS T16 [13] and listed in Table 3, we plot the predicted RMSE and the uncertainty interval in relation with the nominal variability of the site (last column of Table 3). We insert also in Fig. 7 the reference RMSE i.e., the RMSE obtained by the ECMWF operational run on the year listed in Table 3 (see black crosses in Fig. 7). In the following, the forecasts used to compute the reference RMSEs will serve as test forecasts in our study.

As shown by Fig. 7, except for stations DWN, MUT and MLE, the day-ahead forecast error predictions are very close to the reference day-ahead RMSEs. Further, all reference RMSEs fall within the RMSE bounds. Overall, the magnitude of these bounds varies between 50 and 75 W/m<fnref n="2" /> and may seem significant. However, the proposed RMSE bounds have similar magnitude than those estimated by Yang [5].<fnref n="2" />

## 4.3. Verification of the models on the SURFRAD stations

Finally, in this section, we verify both the intra-day and day-ahead RMSE predictions on the SURFRAD stations. Fig. 8 plots for each SURFRAD station the predicted RMSE in relation with the nominal variability listed in Table 4. The predicted uncertainty interval is also provided.

For each station, we also compute the reference RMSE, i.e., the RMSE obtained by the intra-day Neural Network (NN) model or the ECMWF operational run on the testing set.

<caption>Table 3</caption>

Day-ahead verification sites. Column $\sigma(\Delta k_c)$ is the nominal variability of the testing year which is used as input of the RMSE day-ahead D24 prediction model.

<table><thead><tr><th>Site</th><th>Code</th><th>Source</th><th>Year</th><th>Latitude</th><th>Longitude</th><th>Altitude</th><th>σ(Δk<sub>c</sub>)</th></tr></thead><tbody><tr><td>Bac Ninh</td><td>BAC</td><td>ESMAP</td><td>2018</td><td>21.201</td><td>106.063</td><td>42</td><td>0.187</td></tr><tr><td>Darwin</td><td>DWN</td><td>BSRN</td><td>2017</td><td>-12.424</td><td>130.893</td><td>32</td><td>0.179</td></tr><tr><td>Ghardaia</td><td>GHA</td><td>BSRN</td><td>2016</td><td>32.386</td><td>3.780</td><td>463</td><td>0.129</td></tr><tr><td>Gurgaon</td><td>GUR</td><td>BSRN</td><td>2018</td><td>28.425</td><td>77.156</td><td>259</td><td>0.160</td></tr><tr><td>Howrah</td><td>HOW</td><td>BSRN</td><td>2018</td><td>22.553</td><td>88.306</td><td>51</td><td>0.147</td></tr><tr><td>Lauder</td><td>LAU</td><td>BSRN</td><td>2017</td><td>-45.045</td><td>169.689</td><td>350</td><td>0.216</td></tr><tr><td>Male</td><td>MLE</td><td>ESMAP</td><td>2016</td><td>4.193</td><td>73.528</td><td>10</td><td>0.198</td></tr><tr><td>Mutanda</td><td>MUT</td><td>ESMAP</td><td>2016</td><td>-12.423</td><td>26.215</td><td>1318</td><td>0.184</td></tr><tr><td>Ny-Aesund</td><td>NYA</td><td>BSRN</td><td>2016</td><td>78.925</td><td>11.930</td><td>11</td><td>0.140</td></tr><tr><td>Tiruvallur</td><td>TIR</td><td>BSRN</td><td>2018</td><td>13.092</td><td>79.974</td><td>36</td><td>0.173</td></tr></tbody></table>

Notice that, prior to the verification of the models, the NN intra-day models for the SURFRAD sites are built using the training year Year 1 (see Table 4).

Fig. 8 reveals that mostly all the predicted RMSEs (blue points) are close to the reference ones (black crosses) i.e. those effectively obtained by the forecasting methods. It must be noted also that all reference RMSEs related to the SURFRAD network fall within the RMSE bounds.

Selecting another point of view, Fig. 9³ plots in the same graph, for each SURFRAD station, the intra-day (1 h–6 h forecast horizons) and day-ahead (24 h forecast horizon) predicted RMSE in relation with the

<footnote n="2">² see Figure 6 of Yang [5].</footnote>

<footnote n="3">³ This figure is similar to the one proposed by Yang [5] (see Figure 6 in his paper).</footnote>
<figure><img src="image_22.png" /><figcaption>Fig. 7. Predicted day-ahead RMSE vs. nominal variability with the associated 95% uncertainty interval. The black cross represents the reference RMSE.</figcaption></figure>

<caption>**Table 4**</caption>

SURFRAD stations. Year 1 is used for building the NN time series model for the SURFRAD stations. Year 2 is used to calculate the reference RMSE (i.e. the RMSE obtained by the NN model or by IFS ECMWF model) of the forecasts on the testing set. Column $\sigma(\Delta k_c)$ is the nominal variability of the testing year which is used as input of the RMSE prediction models.

<table><thead><tr><th>Site</th><th>Code</th><th>Year 1</th><th>Year 2</th><th>Latitude</th><th>Longitude</th><th>Altitude</th><th>σ(Δk<sub>c</sub>)</th></tr></thead><tbody><tr><td>Bondville</td><td>BON</td><td>2015</td><td>2016</td><td>40.067</td><td>-88.367</td><td>213</td><td>0.190</td></tr><tr><td>Desert Rock</td><td>DRA</td><td>2015</td><td>2016</td><td>36.626</td><td>-116.018</td><td>1007</td><td>0.137</td></tr><tr><td>Fort Peck</td><td>FPK</td><td>2015</td><td>2016</td><td>48.317</td><td>-105.100</td><td>634</td><td>0.178</td></tr><tr><td>Goodwin creek</td><td>GWN</td><td>2015</td><td>2016</td><td>34.255</td><td>-89.873</td><td>98</td><td>0.175</td></tr><tr><td>Penn State Uni.</td><td>PSU</td><td>2015</td><td>2016</td><td>40.720</td><td>-77.930</td><td>376</td><td>0.189</td></tr><tr><td>Sioux Falls</td><td>SXF</td><td>2015</td><td>2016</td><td>43.730</td><td>-96.620</td><td>473</td><td>0.183</td></tr><tr><td>Table Mountain</td><td>TBL</td><td>2015</td><td>2016</td><td>40.125</td><td>-105.237</td><td>1689</td><td>0.210</td></tr></tbody></table>

forecast horizon. The 95% uncertainty interval (filled in gray color in Fig. 9) is associated with each model predictions.

For the intra-day case (left part of the graph), the RMSE predictions (blue dot line in Fig. 9) obtained by applying the intra-day models ID-1H to ID-6H (see Regression equations in Table 1) are again compared to the reference RMSEs i.e. those generated by the time series NN model on the testing year (see black crosses in Fig. 9).

In addition, following the work of Yang [5] who estimated the RMSE bounds on the SURFRAD stations and compared this estimation with published results of the literature, we insert in the graph the RMSE results of (1) SolarAnywhere v2 forecasts [19], (2) SolarAnywhere v4 forecasts [20] and post-processed High-Resolution Rapid Refresh forecasts (HRRR) [21]. As shown for all stations, except for intra-day forecasts generated by SolarAnywhere V4, all RMSE fall inside the RMSE bounds. Overall, for the intra-day case, the magnitude of the RMSE bounds ranges from 25 W/m² (see station DRA at 1 h Horizon) to 75 W/m² (see station TBL at 6 h Horizon).

For the day-ahead case (right part of the graph), the RMSE predictions (orange dot in Fig. 9) obtained by applying the day-ahead model D24 (see Regression equation in Table 1) are again compared to the reference ECMWF RMSE i.e. those provided the ECMWF operational forecasts on the testing year (see black cross in Fig. 9). Again, similarly to Yang [5], we add the published results of (1) forecasts from the North American Mesoscale (NAM) model [23], (2) the post-processed NAM forecasts [24], (3) ECMWF controlled forecasts of Ref. [25] and (4) SolarAnywhere v4 forecasts [20].

As shown by Fig. 9 (forecast horizon 24 h), practically all the RMSE fall within the proposed uncertainty interval. Moreover, for this day-ahead case, it must be noted that the RMSE bounds are of similar magnitude (≈ 75 W/m²) as those estimated by Yang [5].

## 5. Usefulness and limitations of the proposed models

The three levels of verification conducted above suggest that the proposed models may help solar forecasters quickly gain a good idea of how well their forecast models will perform for a particular site if the site's variability is known. This information should also be of particular interest to solar power generation entities, as these forecast error prediction models give them the ability to roughly assess the potential economic benefit of implementing a solar farm at a specific location.

For the estimation of the RMSE bounds, unlike the method proposed by Yang [5], which used a physics-based approach, our approach is purely statistical. Recall that in this work, we define the RMSE bounds by the 95% uncertainty interval; therefore, it is expected that 5% of the forecast error lies outside the proposed bounds. One might then question the usefulness of our proposition.

Nevertheless, based on the performance of the reported forecasts relative to the proposed RMSE bounds, we do think that our approach is a viable and efficient means for verifying forecasts of interest for a specific site. As shown in the previous section, our proposed RMSE bounds are of similar magnitude to those proposed by Yang [5], particularly in the day-ahead case. We believe that solar forecasters could initially use our prediction models to quickly and roughly evaluate their models' performance relative to the proposed error bounds. If necessary, they could then, as a second step, apply the method of Yang [5] to refine the verification of their forecasts.

As mentioned in the introduction, other works such as Karimi-Arpanahi et al. (2023) have proposed alternative metrics to relate forecast error to underlying time series characteristics. For the sake of completeness, we include a complementary analysis in Appendix B, comparing WPE-based RMSE predictions to those of our proposed model on the SURFRAD sites.

Our predictions models require the knowledge of the solar variability for a particular location. One may then question the availability of ground GHI data to calculate the variability metric. Nonetheless, the availability of GHI satellite-gridded databases with global coverage, along with easy access to clear sky times series provided, for instance, by CAMS McClear [18], allows for the calculation of nominal variability
<figure><img src="image_23.png" /><figcaption>(a) RMSE vs. nominal variability. Model ID-1H</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>(c) RMSE vs. nominal variability. Model ID-3H</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>(e) RMSE vs. nominal variability. Model ID-5H</figcaption></figure>

<figure><img src="image_26.png" /><figcaption>(g) RMSE vs. nominal variability. Model D24</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>(b) RMSE vs. nominal variability. Model ID-2H</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>(d) RMSE vs. nominal variability. Model ID-4H</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>(f) RMSE vs. nominal variability. Model ID-6H</figcaption></figure>

<caption>**Fig. 8.** SURFRAD stations - Predicted RMSE vs. nominal variability with the associated 95% uncertainty interval. The black cross represents the reference RMSE.</caption>

time series for any location in the world. Consequently, users can quickly estimate RMSE forecast error along with its bounds.

Regarding the limitations of the models, it is important to note that the intra-day forecasting model used to infer the relationship between nominal variability and forecast error make use of only of 5 past ground measurements as inputs. Therefore, it is expected that intra-day forecasting models that combine different sources of input data may generate better RMSE which could be badly predicted by the proposed prediction models. See for instance the case of SolarAnywhere

V4 RMSEs in Fig. 9 which lie outside the predicted RMSE bounds. Additionally, RMSEs obtained from naive models, such as persistence or from linear models, are likely to fall outside the uncertainty bounds.

The long-term stability of the proposed model is an important consideration. Its robustness is partly due to the construction of the input metric: the standard deviation of the first-order difference $\Delta k_c$ over a 1-hour lag helps filter out slow trends and seasonal drifts, making the model relatively insensitive to long-term changes in irradiance levels. However, if significant shifts occur in local site characteristics
<figure><img src="image_30.png" /><figcaption>(a) station: BON</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>(b) station: DRA</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>(c) station: FPK</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>(d) station: GWN</figcaption></figure>

<figure><img src="image_34.png" /><figcaption>(e) station: PSU</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>(f) station: SXF</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>(g) station: TBL</figcaption></figure>

Fig. 9. Verification of the proposed models on the SURFRAD stations. In addition to the predicted RMSE, RMSE from the literature are also reported. For the intra-day case, the following RMSEs are plotted: (1) SolarAnywhere v2 [19] (2) SolarAnywhere v4 [20] and (3) post-processed High-Resolution Rapid Refresh forecasts [21]. For the day-ahead case, the following RMSE are inserted in the graph: (1) forecasts from the North American Mesoscale (NAM) model [23], (2) the post-processed NAM forecasts [24], (3) ECMWF controlled forecasts of Ref. [25] and (4) SolarAnywhere SolarAnywhere v4 forecasts [20]. The predicted 95% uncertainty interval is given by the shaded area.

or the underlying forecasting system, we recommend periodic validation and, if necessary, recalibration to ensure sustained operational accuracy.

Finally, the value of R² = 0.52 obtained for the forecast error day-ahead prediction D24 model might be considered modest. Consequently, predictions of this model may have a considerable amount of
unexplained variability. There may be other relevant variables or inter-
actions that are not included in the model, which could explain more
of the variability. Adding these variables could potentially improve the
predictions of the model.

6. Summary and conclusion

In this work, we proposed simple linear regression models to predict
the intra-day and day-ahead forecast errors based on the variability
experienced at a specific site. The forecast error was measured using
the RMSE, while variability was quantified by the standard deviation
of a time series of 1-hour changes in the dimensionless clear sky index.
The models also predicted RMSE bounds. They were developed using
GHI data from sixty sites distributed across the globe.

Verification of the prediction models was conducted on twenty-
two sites with diverse climates. This evaluation demonstrates that the
proposed models can reasonably predict the RMSE forecast error based
on the nominal variability of a site. These straightforward yet effective
prediction models enable users to quickly estimate the forecast error
for a specific site before resorting to more complex solar forecasting
models. Additionally, the predicted RMSE bounds allow solar forecast-
ers to quickly evaluate the performance of their forecasts relative to
these bounds. Thus, we believe that the proposed models can serve as
valuable tools for solar forecasters.

CRediT authorship contribution statement

**Philippe Lauret:** Writing – original draft, Software, Methodology, Investigation, Formal analysis. **Mathieu David:** Writing – review & editing, Validation, Resources, Methodology, Investigation. **Josselin Le Gal La Salle:** Writing – review & editing, Software, Investigation. **Elke Lorenz:** Writing – review & editing, Validation. **Richard Perez:** Writing – review & editing, Validation, Methodology, Investigation, Formal analysis, Conceptualization. **Cyril Voyant:** Writing – review & editing, Validation, Investigation, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing finan-
cial interests or personal relationships that could have appeared to
influence the work reported in this paper.

Acknowledgments

The authors express their gratitude to Anna Forstinger for provid-
ing the ground datasets. We would also like to thank the following
institutions and colleagues for the provision of data for this work:

• NamPower in Namibia

* Mr. Yuldash Sobirov from the Institute of Material Science in Uzbekistan

• Frank Vignola from the University of Oregon

• David Pozo from the University Jaen

• Dietmar Baumgartner from the University of Graz

• Julian Gröbner at PMOD

• Nicolas Fernay from the University Lille

• Peter Armstrong at the Masdar Institute

• Laurent Vuilleumier at MeteoSwiss

• Irena Balog at ENEA

• Sophie Pelland at CanmetÉNERGIE Varennes

• Emmanuel Guillot at CNRS-PROMES, Odeillo

* Department of Civil Engineering at the Technical University of Denmark

• Swedish Meteorological and Hydrological Institute

• Australian Government Bureau of Meteorology

• INPE National Institute of Space Research CCST Center for Earth System Sciences with FINEP Financier of Studies and Projects Ministry of Science and Technology and PETROBRAS Petróleo Brasileiro

• SKYNET, in particular Hitoshi Irie and Tamio Takamura (CERES/Chiba-U.), Chiba University, Tadahiro Hayasaka (Tohoku University), Chulalongkorn University

• ESMAP program of the World Bank Group in particular Joana Zerbin, Clara Ivanescu, Branislav Schnierer, Roman Affolter, GeoSUN Africa, Rachel Fox, Margot King

• Cairo University in Egypt

• University of Oujda

• Institute Research Solar Energy Et Energies Nouvelles (IRESEN) in Morocco

• Research and Technology Centre of Energy (CRTE) in Tunisia

• University of Jordan in Jordan and the Centre de Developpement des Energies Renouvelables (CDER) in Algeria.

• NOAA Global Monitoring Laboratory

• Baseline Surface Radiation Network (https://bsrn.awi.de/).

The study received support from the TwInSolar project funded by the European Union's Horizon Europe research and innovation program grant number 101076447.

Appendix A. List of sites used to build the prediction models ID-1, ..., ID-6H and D24

See Table 5.

Appendix B. Comparison with entropy-based metric

To further assess the value of our proposed method, we compare its
performance against an entropy-based predictability index — Weighted
Permutation Entropy (WPE) — as proposed by Karimi-Arpanahi et al.
(2023). WPE is a widely used metric that quantifies the complexity
or disorder of a time series and has been shown to correlate with
forecast errors in renewable energy applications. Appendix C gives
details regarding the computation of WPE of a timeseries.

Using the training dataset of 60 sites, we first built a linear regres-
sion model to relate the WPE of the clear sky index time series to the
observed RMSE of day-ahead forecasts. This provided the prediction
model that could then be applied to new test sites for evaluation. Notice
that the hyperparameter embedding dimension *d* was set to 6 in the
calculation of the WPE metric.

We then apply this prediction model with WPE values of the time
series of the clear sky index for the SURFRAD test sites and compared
them with both the reference RMSE (from ECMWF day-ahead fore-
casts). Fig. 10b plots the prediction of the WPE-based model. For ease
of comparison, Fig. 10a (from our D24 model) is shown alongside the
WPE-based predictions.

The RMSE predictions based on WPE are of comparable order
of magnitude to those obtained from our variability-based model.
However, it appears that the WPE metric does not produce a strong
variation in predicted RMSE across sites. Indeed, from the data depicted
in Fig. 10, the predicted RMSE values from the σ(Δk_c)-based model
span a range of 48 W/m² while the WPE-based predictions span only
18.8 W/m². This limited dispersion illustrates the reduced sensitivity
of WPE-based model to site-specific forecast difficulty. A normalization
of the WPE metric using the min-max method leads to a similar
conclusion.

In addition, entropy-based metrics such as WPE require careful
tuning of hyperparameters and may be less intuitive to interpret than
physically grounded metrics like σ(Δk<sub>c</sub>).

In contrast, our variability-based predictor produces a wider dy-
namic range across sites and more clearly captures the relationship
with forecast error. This supports the notion that σ(Δk<sub>c</sub>) is a more
interpretable and actionable metric for estimating site-level forecast
difficulty, particularly in early deployment contexts where entropy
metrics may require tuning and lack physical intuition.
<caption>**Table 5**</caption>

List of sites used to build the RMSE prediction models. Year 1 is used to train the NN time series. Year 2, the testing year, is used to compute the RMSE of the NN time series and the RMSE of IFS ECMWF forecast. $\sigma(\Delta k_c)$ is the nominal variability of the testing year and mean GHI the mean of observations of the testing Year2.

<table><thead><tr><th>Site</th><th>Code</th><th>Source</th><th>Year 1</th><th>Year 2</th><th>Latitude</th><th>Longitude</th><th>Altitude</th><th>σ(Δk<sub>c</sub>)</th></tr></thead><tbody><tr><td>Adrar</td><td>ADR</td><td>Enermena</td><td>2015</td><td>2016</td><td>27.880</td><td>-0.274</td><td>262.0</td><td>0.118</td></tr><tr><td>Albuquerque</td><td>ALB</td><td>NOAA</td><td>2018</td><td>2019</td><td>35.038</td><td>-106.622</td><td>1617.0</td><td>0.179</td></tr><tr><td>Alice Springs</td><td>ALI</td><td>BSRN</td><td>2017</td><td>2018</td><td>-23.798</td><td>133.888</td><td>547.0</td><td>0.132</td></tr><tr><td>Barrow</td><td>BAR</td><td>NOAA</td><td>2016</td><td>2017</td><td>71.323</td><td>-156.611</td><td>11.0</td><td>0.128</td></tr><tr><td>Burns</td><td>BUR</td><td>SRML</td><td>2017</td><td>2018</td><td>43.519</td><td>-119.022</td><td>1265.0</td><td>0.168</td></tr><tr><td>Cabauw</td><td>CAB</td><td>BSRN</td><td>2018</td><td>2019</td><td>51.971</td><td>4.927</td><td>0.0</td><td>0.205</td></tr><tr><td>Cachoeira Paulista</td><td>CAC</td><td>Sonda</td><td>2017</td><td>2018</td><td>-22.690</td><td>-45.006</td><td>574.0</td><td>0.200</td></tr><tr><td>Cape Hedo</td><td>CAH</td><td>SKYNET</td><td>2017</td><td>2018</td><td>26.867</td><td>128.248</td><td>65.0</td><td>0.199</td></tr><tr><td>Cairo University</td><td>CAI</td><td>Enermena</td><td>2017</td><td>2018</td><td>30.036</td><td>31.009</td><td>104.0</td><td>0.139</td></tr><tr><td>Carpentras</td><td>CAR</td><td>BSRN</td><td>2017</td><td>2018</td><td>44.083</td><td>5.059</td><td>100.0</td><td>0.188</td></tr><tr><td>Central Highlands</td><td>CEH</td><td>ESMAP</td><td>2018</td><td>2019</td><td>12.753</td><td>107.876</td><td>277.0</td><td>0.207</td></tr><tr><td>Cener</td><td>CEN</td><td>BSRN</td><td>2017</td><td>2018</td><td>42.816</td><td>-1.601</td><td>471.0</td><td>0.188</td></tr><tr><td>Chiba</td><td>CHB</td><td>SKYNET</td><td>2017</td><td>2018</td><td>35.625</td><td>140.104</td><td>21.0</td><td>0.190</td></tr><tr><td>Chileka</td><td>CHI</td><td>ESMAP</td><td>2016</td><td>2017</td><td>-15.680</td><td>34.972</td><td>769.0</td><td>0.215</td></tr><tr><td>Chilanga</td><td>CHL</td><td>ESMAP</td><td>2016</td><td>2017</td><td>-15.548</td><td>28.248</td><td>1226.0</td><td>0.174</td></tr><tr><td>Concordia</td><td>CON</td><td>BSRN</td><td>2017</td><td>2018</td><td>-75.100</td><td>123.383</td><td>3233.0</td><td>0.062</td></tr><tr><td>Da Nang</td><td>DAN</td><td>ESMAP</td><td>2018</td><td>2019</td><td>16.013</td><td>108.186</td><td>11.0</td><td>0.209</td></tr><tr><td>Dar Es Salaam</td><td>DAR</td><td>ESMAP</td><td>2016</td><td>2017</td><td>6.781</td><td>39.204</td><td>122.0</td><td>0.200</td></tr><tr><td>De Aar</td><td>DEA</td><td>BSRN</td><td>2017</td><td>2018</td><td>-30.667</td><td>23.993</td><td>1287.0</td><td>0.144</td></tr><tr><td>Feni</td><td>FEN</td><td>ESMAP</td><td>2018</td><td>2019</td><td>22.800</td><td>91.358</td><td>7.0</td><td>0.206</td></tr><tr><td>Florianopolis</td><td>FLO</td><td>BSRN</td><td>2017</td><td>2018</td><td>-27.605</td><td>-48.523</td><td>11.0</td><td>0.200</td></tr><tr><td>Fukuoka</td><td>FUK</td><td>BSRN</td><td>2017</td><td>2018</td><td>33.582</td><td>130.376</td><td>3.0</td><td>0.186</td></tr><tr><td>Fukue</td><td>FUU</td><td>SKYNET</td><td>2017</td><td>2018</td><td>32.752</td><td>128.682</td><td>80.0</td><td>0.188</td></tr><tr><td>Gan</td><td>GAN</td><td>ESMAP</td><td>2016</td><td>2017</td><td>-0.691</td><td>73.150</td><td>5.0</td><td>0.194</td></tr><tr><td>Gobabeb</td><td>GOB</td><td>BSRN</td><td>2017</td><td>2018</td><td>-23.561</td><td>15.042</td><td>407.0</td><td>0.143</td></tr><tr><td>Hanford</td><td>HAF</td><td>NOAA</td><td>2017</td><td>2018</td><td>36.314</td><td>-119.632</td><td>73.0</td><td>0.124</td></tr><tr><td>Hanimaadhoo</td><td>HAN</td><td>ESMAP</td><td>2016</td><td>2017</td><td>6.748</td><td>73.170</td><td>4.0</td><td>0.180</td></tr><tr><td>Hermiston</td><td>HER</td><td>SRML</td><td>2017</td><td>2018</td><td>45.818</td><td>-119.285</td><td>188.0</td><td>0.153</td></tr><tr><td>Ishigakijima</td><td>ISH</td><td>BSRN</td><td>2017</td><td>2018</td><td>24.337</td><td>124.164</td><td>5.7</td><td>0.191</td></tr><tr><td>Izana</td><td>IZA</td><td>BSRN</td><td>2015</td><td>2016</td><td>28.309</td><td>-16.499</td><td>2372.9</td><td>0.113</td></tr><tr><td>Univ. of Jordan</td><td>JOR</td><td>Enermena</td><td>2016</td><td>2017</td><td>30.172</td><td>35.818</td><td>1012.0</td><td>0.099</td></tr><tr><td>Kasungu</td><td>KAS</td><td>ESMAP</td><td>2016</td><td>2017</td><td>-13.015</td><td>33.468</td><td>1067.0</td><td>0.196</td></tr><tr><td>Kadhdhoo</td><td>KDO</td><td>ESMAP</td><td>2016</td><td>2017</td><td>1.860</td><td>73.520</td><td>2.0</td><td>0.186</td></tr><tr><td>Langley Res. Center</td><td>LAN</td><td>BSRN</td><td>2017</td><td>2018</td><td>37.104</td><td>-76.387</td><td>3.0</td><td>0.194</td></tr><tr><td>Madison</td><td>MAD</td><td>NOAA</td><td>2017</td><td>2018</td><td>43.072</td><td>-89.411</td><td>271.0</td><td>0.185</td></tr><tr><td>Milan RSE</td><td>MIL</td><td>RSE</td><td>2017</td><td>2018</td><td>45.476</td><td>9.255</td><td>150.0</td><td>0.182</td></tr><tr><td>Minamitorishima</td><td>MIN</td><td>BSRN</td><td>2017</td><td>2018</td><td>24.288</td><td>153.983</td><td>7.1</td><td>0.172</td></tr><tr><td>Missour</td><td>MIS</td><td>Enermena</td><td>2016</td><td>2017</td><td>32.860</td><td>-4.107</td><td>1107.0</td><td>0.204</td></tr><tr><td>Miyako</td><td>MIY</td><td>SKYNET</td><td>2016</td><td>2017</td><td>24.737</td><td>125.327</td><td>50.0</td><td>0.197</td></tr><tr><td>Mzuzu</td><td>MZU</td><td>ESMAP</td><td>2016</td><td>2017</td><td>-11.420</td><td>33.995</td><td>1287.0</td><td>0.213</td></tr><tr><td>Natal</td><td>NAT</td><td>Sonda</td><td>2017</td><td>2018</td><td>-5.837</td><td>-35.206</td><td>58.0</td><td>0.134</td></tr><tr><td>Univ. of Oujda</td><td>OUJ</td><td>Enermena</td><td>2018</td><td>2019</td><td>34.650</td><td>-1.900</td><td>617.0</td><td>0.159</td></tr><tr><td>Palaiseau</td><td>PAL</td><td>BSRN</td><td>2017</td><td>2018</td><td>48.713</td><td>2.208</td><td>156.0</td><td>0.201</td></tr><tr><td>Payerne</td><td>PAY</td><td>BSRN</td><td>2017</td><td>2018</td><td>46.815</td><td>6.944</td><td>491.0</td><td>0.190</td></tr><tr><td>Petrolina</td><td>PET</td><td>BSRN</td><td>2017</td><td>2018</td><td>-9.068</td><td>-40.319</td><td>387.0</td><td>0.198</td></tr><tr><td>Salt Lake City</td><td>SAL</td><td>NOAA</td><td>2015</td><td>2016</td><td>40.772</td><td>-111.955</td><td>1288.0</td><td>0.177</td></tr><tr><td>Sapporo</td><td>SAP</td><td>BSRN</td><td>2016</td><td>2018</td><td>43.060</td><td>141.329</td><td>17.2</td><td>0.208</td></tr><tr><td>Seattle</td><td>SEA</td><td>NOAA</td><td>2017</td><td>2017</td><td>47.687</td><td>-122.257</td><td>20.0</td><td>0.181</td></tr><tr><td>Seattle</td><td>SET</td><td>SRML</td><td>2016</td><td>2017</td><td>47.654</td><td>-122.309</td><td>70.0</td><td>0.181</td></tr><tr><td>Silver Lake</td><td>SIL</td><td>SRML</td><td>2016</td><td>2018</td><td>43.119</td><td>-121.059</td><td>1324.0</td><td>0.189</td></tr><tr><td>Song Binh</td><td>SON</td><td>ESMAP</td><td>2018</td><td>2017</td><td>11.264</td><td>108.345</td><td>61.0</td><td>0.197</td></tr><tr><td>Sterling</td><td>STR</td><td>NOAA</td><td>2017</td><td>2018</td><td>38.972</td><td>-77.487</td><td>85.0</td><td>0.195</td></tr><tr><td>Plataforma Solar</td><td>TAB</td><td>CIEMAT</td><td>2017</td><td>2018</td><td>37.091</td><td>-2.358</td><td>500.0</td><td>0.181</td></tr><tr><td>Tamanrasset</td><td>TAM</td><td>BSRN</td><td>2016</td><td>2017</td><td>22.790</td><td>5.529</td><td>1385.0</td><td>0.147</td></tr><tr><td>Tan-Tan</td><td>TAN</td><td>Enermena</td><td>2015</td><td>2016</td><td>28.498</td><td>-11.322</td><td>75.0</td><td>0.168</td></tr><tr><td>Tataouin</td><td>TAT</td><td>Enermena</td><td>2015</td><td>2016</td><td>32.974</td><td>10.485</td><td>210.0</td><td>0.153</td></tr><tr><td>Toravere</td><td>TOR</td><td>BSRN</td><td>2017</td><td>2018</td><td>58.254</td><td>26.462</td><td>70.0</td><td>0.192</td></tr><tr><td>Tri An</td><td>TRI</td><td>ESMAP</td><td>2017</td><td>2018</td><td>11.102</td><td>107.038</td><td>59.0</td><td>0.219</td></tr><tr><td>Kiruna</td><td>KIR</td><td>SMHI</td><td>2017</td><td>2018</td><td>67.842</td><td>20.410</td><td>424.0</td><td>0.196</td></tr><tr><td>Norrköping</td><td>NOR</td><td>SMHI</td><td>2017</td><td>2018</td><td>58.582</td><td>16.148</td><td>53.0</td><td>0.190</td></tr><tr><td>Parkent</td><td>PAR</td><td>p.C.</td><td>2015</td><td>2016</td><td>41.320</td><td>69.740</td><td>1081.0</td><td>0.168</td></tr><tr><td>Visby</td><td>VIS</td><td>SMHI</td><td>2017</td><td>2018</td><td>57.673</td><td>18.345</td><td>49.0</td><td>0.195</td></tr></tbody></table>

## Appendix C. Permutation entropy and weighted permutation entropy

The Permutation entropy (PE) metric can be calculated by determining the ordinal patterns of vectors in a time series and extracting their probability distribution.

The core idea of PE is that certain ordinal patterns may occur more frequently than others in a predictable time series. For example, if a

few patterns dominate in a time series, the PE will be close to zero, indicating high predictability. Conversely, if all patterns have nearly equal occurrence, the PE approaches 1, reflecting randomness or low predictability.

To calculate the PE of a time series {$x_t$}$_{t=1}^N$ for an embedding dimension $d$, the series is first divided into $N - (d - 1)$ overlapping vectors of length $d$:

$$X_t^d = (x_t, x_{t+1}, \dots, x_{t+d-1})$$
<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /><figcaption>Fig. 10. Comparison of predicted and reference RMSE using (a) a variability-based model and (b) the entropy-based WPE indicator, across SURFRAD test sites. Blue circles indicate RMSE predicted by the D24 and WPE models, and black crosses show reference RMSE from ECMWF forecasts. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

The dimension *d* is a hyperparameter which must be tuned. For practical purposes, the embedding dimension should be a number between 3 and 7.

Each vector is assigned to one of the *d!* possible permutations $\pi_i \in P$ based on its ordinal ranking. For instance, if $X_t^d = (4, 3, 7)$, the ordinal pattern is 2-1-3.

The relative frequency of a permutation $\pi_i$ occurring in the time series is:

$$P(\pi_i) = \frac{|\{t : f(X_t^d) = \pi_i\}|}{N - (d - 1)}$$

The permutation entropy is then defined as:

$$PE(d) = - \sum_{\pi_i \in P} P(\pi_i) \log_2 P(\pi_i)$$

To make the values comparable across different *d*, PE is normalized by $log_2(d!)$.

A major limitation of PE is that it considers only the ordinal structure, ignoring amplitude variations and being sensitive to small fluctuations. To address this, weighted permutation entropy (WPE) incorporates both order and amplitude information.

The weight $w_t$ of vector $X_t^d$ is calculated as:

$$w_t = \frac{1}{d} \sum_{s=1}^{d} (x_{t+(s-1)} - \bar{X}_t^d)^2$$

where $\bar{X}_t^d$ is the mean of the elements of $X_t^d$.

The weighted probability of permutation $\pi_i$ is then:

$$P_w(\pi_i) = \frac{\sum_{t=1}^{N-(d-1)} w_t \cdot \mathbb{I}(f(X_t^d) = \pi_i)}{\sum_{t=1}^{N-(d-1)} w_t}$$

where $\mathbb{I}(a=b)$ is 1 if $a=b$, and 0 otherwise.

The WPE is computed as:

$$WPE(d) = - \sum_{\pi_i \in P} P_w(\pi_i) \log_2 P_w(\pi_i)$$

Like PE, WPE is also normalized by $log_2(d!)$ to facilitate comparison across different *d*. In this study, the normalized WPE is referred to simply as “WPE”.

## Data availability

The code and data are provided as supplementary materials and are available in the following GitHub repository: https://github.com/Laboratoire-Piment/solar-predict-rmse.git.

In the provided data files, readers will have access to the RMSE values predicted by the models and the RMSE values reported in the literature. Additionally, readers will be able to compute the relative RMSE, as the mean GHI for each site is provided.

## References

[1] P. Lauret, C. Voyant, T. Soubdhan, M. David, P. Poggi, A benchmarking of machine learning techniques for solar radiation forecasting in an insular context, Sol. Energy 112 (2015) 446-457.

[2] M. David, F. Ramahatana, P. Trombe, P. Lauret, Probabilistic forecasting of the solar irradiance with recursive ARMA and GARCH models, Sol. Energy 133 (2016) 55-72.

[3] R. Alonso-Suárez, M. David, V. Branco, P. Lauret, Intra-day solar probabilistic forecasts including local short-term variability and satellite information, Renew. Energy 158 (2020) 554-573, URL https://www.sciencedirect.com/science/article/pii/S0960148120307448.

[4] C. Voyant, T. Soubdhan, P. Lauret, M. David, M. Muselli, Statistical parameters as a means to a priori assess the accuracy of solar forecasting models, Energy 90 (2015) 671-679.

[5] D. Yang, Correlogram, predictability error growth, and bounds of mean square error of solar irradiance forecasts, Renew. Sustain. Energy Rev. 167 (2022) 112736, http://dx.doi.org/10.1016/j.rser.2022.112736.

[6] S. Karimi-Arpanahi, S.A. Pourmousavi, N. Mahdavi, Quantifying the predictability of renewable energy data for improving power systems decision-making, Patterns 4 (4) (2023) Publisher: Elsevier.

[7] B. Liu, D. Yang, M.J. nos Mayer, C.F. Coimbra, J. Kleissl, M. Kay, W. Wang, J.M. Bright, A.X. Xiang, X. Lv, D. Srinivasan, Y. Wu, H.G. Beyer, G.M. Yagli, Y. Shen, Predictability and forecast skill of solar irradiance over the contiguous United States, Renew. Sustain. Energy Rev. 182 (2023) 113359.

[8] B. Liu, J. Wang, J. Chen, B. Li, D. Sun, G. Zhang, A probabilistic perspective on predictability of solar irradiance using bootstrapped correlograms and ensemble predictability error growth, Sol. Energy 260 (2023) 17-24.

[9] C. Voyant, P. Lauret, G. Notton, J.-L. Duchaud, A. Fouilloy, M. David, Z.M. Yaseen, T. Soubdhan, A Monte Carlo based solar radiation forecastability estimation, J. Renew. Sustain. Energy 13 (2) (2021) 026501, eprint: https://pubs.aip.org/aip/jrse/article-pdf/doi/10.1063/5.0042710/19752243/026501_1_online.pdf.

[10] D. Yang, S. Alessandrini, J. Antonanzas, F. Antonanzas-Torres, V. Badescu, H.G. Beyer, R. Blaga, J. Boland, J.M. Bright, C.F. Coimbra, M. David, A. Frimane, C.A. Gueymard, T. Hong, M.J. Kay, S. Killinger, J. Kleissl, P. Lauret, E. Lorenz, D. van der Meer, M. Paulescu, R. Perez, O. Perpinan-Lamigueiro, I.M. Peters, G. Reikard, D. Renne, Y.-M. Saint-Drenan, Y. Shuai, R. Urraca, H. Verbois, F. Vignola, C. Voyant, J. Zhang, Verification of deterministic solar forecasts, Sol. Energy 210 (2020) 20-37, Special Issue on Grid Integration.

[11] H.T. Pedro, C.F. Coimbra, Short-term irradiance forecastability for various solar micro-climates, Sol. Energy 122 (2015) 587-602.

[12] A. Forstinger, Y.-M. Saint-Drenan, S. Wilbert, A. Jensen, B. Kraas, C.F. Peruchena, C. Gueymard, D. Ronzio, D. Yang, E. Collino, J.P. Martinez, J. Ruiz-Arias, N. Hanrieder, P. Blanc, IEA-PVPS Task-16 Reference Solar Measurements, MINES ParisTech, 2021.

[13] A. Forstinger, S. Wilbert, A. Jensen, B. Kraas, C. Peruchena, C. Gueymard, D. Ronzio, D. Yang, E. Collino, J. Martinez, J. Ruiz-Arias, N. Hanrieder, P. Blanc, Y. Saint-Drenan, Expert quality control of solar radiation ground data sets, in: Proceedings of SWC 2021: ISES Solar World Congress, International Solar Energy Society, 2021, pp. 1037-1048, SWC 2021: ISES Solar World Congress; Conference date: 25-10-2021 Through 29-10-2021.

[14] M. Lefèvre, A. Oumbe, P. Blanc, B. Espinar, B. Gschwind, Z. Qu, L. Wald, M. Schroedter-Homscheidt, C. Hoyer-Klick, A. Arola, A. Benedetti, J.W. Kaiser, J.-J. Morcrette, McClear: a new model estimating downwelling solar radiation at ground level in clear-sky conditions, Atmos. Meas. Tech. 6 (9) (2013) 2403-2418.
[15] R. Perez, S. Kivalov, J. Schlemmer, K. Hemker, T. Hoff, Parameterization of site-specific short-term irradiance variability, Sol. Energy 85 (7) (2011) 1343-1353.

[16] C.F. Coimbra, J. Kleissl, R. Marquez, Overview of solar-forecasting methods and a metric for accuracy evaluation, in: Solar Energy Forecasting and Resource Assessment, Elsevier, 2013, pp. 171–194, http://dx.doi.org/10.1016/B978-0-12-397177-7.00008-5.

[17] R. Perez, M. David, T.E. Hoff, M. Jamaly, S. Kivalov, J. Kleissl, P. Lauret, M. Perez, Spatial and temporal variability of solar energy, Found. Trends- ÆRenew. Energy 1 (1) (2016) 1–44.

[18] CAMS McClear, https://www.tsv.soda-pro.com/web-services/radiation/cams-mcclear, (Accessed 22 October 2024).

[19] R. Perez, S. Kivalov, J. Schlemmer, K. Hemker, D. Renne, T.E. Hoff, Validation of short and medium term operational solar radiation forecasts in the US, Sol. Energy 84 (12) (2010) 2161–2172.

[20] R. Perez, J. Schlemmer, S. Kivalov, J. Dise, P. Keelin, M. Grammatico, et al., A new version of the SUNY solar forecast model: A scalable approach to site-specific model training, in: IEEE 45th Photovoltaic Specialists Conference, 2018, pp. 1–6.

[21] G. Zhang, D. Yang, G. Galanis, E. Androulakis, Solar forecasting with hourly updated numerical weather prediction, Renew. Sustain. Energy Rev. 154 (2022) 111768.

[22] L.M. Aguiar, B. Pereira, P. Lauret, F. Díaz, M. David, Combining solar irradiance measurements, satellite-derived data and a numerical weather prediction model to improve intra-day solar forecasting, Renew. Energy 97 (2016) 599–610.

[23] D. Yang, R. Perez, Can we gauge forecasts using satellite-derived solar irradiance? J. Renew. Sustain. Energy 11 (2) (2019) 023704, eprint: https://pubs.aip.org/aip/jrse/article-pdf/doi/10.1063/1.5087588/15680943/023704_1_online.pdf.

[24] D. Yang, Post-processing of NWP forecasts using ground or satellite-derived data through kernel conditional density estimation, J. Renew. Sustain. Energy 11 (2) (2019) 026101, eprint: https://pubs.aip.org/aip/jrse/article-pdf/doi/10.1063/1.5088721/15682732/026101_1_online.pdf.

[25] R. Perez, E. Lorenz, S. Pelland, M. Beauharnois, G. Van Knowe, K. Hemker, D. Heinemann, J. Remund, S.C. Muller, W. Traunmuller, G. Steinmauer, D. Pozo, J.A. Ruiz-Arias, V. Lara-Fanego, L. Ramirez-Santigosa, M. Gaston-Romero, L.M. Pomares, Comparison of numerical weather prediction solar irradiance forecasts in the US, Canada and Europe, Sol. Energy 94 (2013) 305–326.