## 3 Evaluation Metrics for Probabilistic Forecasting Models
### Continuous Ranked Probability Score (CRPS)  
CRPS is a univariate strictly proper scoring rule which measures the compatibility of a cumulative distribution function \(F\) with an observation \(x\in\mathbb{R}\) as  
\[\text{CRPS}(F,x)=\int_{\mathbb{R}}(F(y)-\mathds{1}\{x\leq y\})^{2}\,\mathrm{d} y\,, \tag{4}\]  
where \(\mathds{1}\{x\leq y\}\) is the indicator function, which is one if \(x\leq y\) and zero otherwise.  
The predictive distributions are often expressed in terms of samples, possibly through Monte-Carlo sampling [28]. Fortunately, there are several methods to estimate CRPS given only samples from a predictive distribution. The precision of these approximation methods depends on the number of samples we use for estimation. Below you can find a list of the most used techniques.  
Empirical CDF:  
In this technique, we approximate the CDF of a predictive model using its samples.  
\[\hat{F}(y)=\frac{1}{n}\sum_{i=1}^{n}\mathds{1}\{x_{i}\leq y\}\,. \tag{5}\]  
Then, we can use \(\hat{F}(y)\) in conjunction with Equation (4) to approximate CRPS.  
Quantile based:  
The pinball loss or quantile loss at a quantile level \(\alpha\in[0,1]\) and with a predicted \(\alpha\)th quantile \(q\) is defined as  
\[\Lambda_{\alpha}(q,x)=(\alpha-\mathds{1}\{x<q\})(x-q). \tag{6}\]  
The CRPS has an intuitive definition as the pinball loss integrates over all quantile levels \(\alpha\in[0,1]\),  
\[\text{CRPS}\Big{(}F^{-1},x\Big{)}=\int_{0}^{1}2\Lambda_{\alpha}\Big{(}F^{-1}( \alpha),x\Big{)}d\alpha\,, \tag{7}\]  
where \(F^{-1}\) represents the quantile function. In practice, we approximate quantiles based on the samples we have. Therefore, Equation (7) can be approximated as a summation over N quantiles. The precision of our approximation depends on the number of quantiles as well as the number of samples we have.  
Sample Estimation:  
Using lemma 2.2 of [33] or identity 17 of [34], we can approximate CRPS by  
\[\text{CRPS}(F,x)=\mathds{E}_{F}|X-x|-\frac{1}{2}\mathds{E}_{F}\big{|}X-X^{ \prime}|\,, \tag{8}\]  
where \(X\) and \(X^{\prime}\) are independent copies of a random variable with distribution function F and a finite first moment [28].  
To investigate the significance of sample size on the accuracy of different approximation methods, we ran a simple experiment. In this experiment, we assumed that the probabilistic model follows a Gaussian distribution with \(\mu=0\) and \(\sigma=1\). Then, we approximated \(\text{CRPS}(F,x)\) where \(x=0\) with various sample sizes in range \([200,5000]\). Since we know the probabilistic model distribution, we can calculate the value of CRPS analytically, i.e., \(\text{CRPS}(F,x)\approx 0.2337\).  
From Figure 1a,b, we can perceive that the empirical CDF method and sample estimation method can converge to the close vicinity of the true value efficiently. However, the empirical CDF method has less variance in comparison to sample estimation. The method based on pinball loss depends on sample size and the number of quantiles. Figure 1c portrays how these two factors affect the approximation. We can see that with a number of quantiles greater than 20, the pinball loss method can produce a very good approximation using only a few samples (circa 500 samples).