## 5 Closer Look into CRPS-Sum in Practice  
In the previous section, we discussed the properties of CRPS-Sum and indicated its shortcomings in hypothetical scenarios using toy data settings. In this section, we aim to investigate CRPS-Sum capabilities with real datasets. To do so, we conducted experiments by constructing simple models that are based on random noise and investigate their  
Figure 2: The relative change in CRPS-Sum (**a**) and ES (**b**) with respect to \(\rho\) and \(\varrho\). The correlation of forecast (\(\varrho\)) is presented on the \(x\) axis, and the correlation of data (\(\rho\)) is depicted with different lines. Unlike ES, the CRPS-Sum figure is not symmetric, which indicates that it is biased with regard to the \(\rho\) value. performance using CRPS-Sum. In our first experiment, we employed the exchange-rate dataset [35]. The exchange-rate dataset is a multivariate time series dataset which contains the daily exchange rate of eight countries, namely, Australia, British, Canada, Switzerland, China, Japan, New Zealand, and Singapore, which was collected between 1990 and 2016. This dataset has few dimensions, which lets us use ES alongside CRPS and CRPS-Sum. Additionally, it is easier to perform qualitative assessment on lower dimensionalities. We used the dataset in the same setting that is proposed in [18]. We also utilized the low-rank Gaussian copula processes method (GP-copula) from [18]. GP-copula combines an RNN-based time-series model with a Gaussian copula process output model for probabilistic forecasting. Furthermore, the model employs a low-rank covariance structure to reduce the computational complexity and handle non-Gaussian marginal distributions. We selected this model since the model performance has been reported in CRPS-Sum. Our first model is a dummy univariate model which follows a Gaussian distribution. The mean of the Gaussian distribution is \(\mu=\mu_{last}\) where \(\mu_{last}\) is the mean of the last values in the input vector over the dimensions, i.e.,  
\[\mu_{last}=\frac{1}{D}\sum_{i=1}^{D}x_{T}^{i}. \tag{15}\]  
We used \(\sigma=10^{-4}\) as the standard deviation of the dummy univariate model in our experiments; however, the results are not dependent on the \(\sigma\) value (more discussion on \(\sigma\) values can be found in Appendix B). We used this model to generate the forecast for every dimension. For the second model, we employed a multivariate Gaussian distribution to build a dummy multivariate forecaster. The mean of the \(i\)-th dimension of the multivariate Gaussian distribution is the value of the last time step in the input window, i.e., \(\mu_{i}=x_{T}^{i}\). The covariance matrix is zero everywhere except on its main diagonal, which is filled with \(10^{-4}\). In other words, we extended the last observation of the input window as the prediction and apply a small perturbation from a Gaussian distribution. Table 1 presents the CRPS-Sum, CRPS, and ES of the two dummy models and the result of the GP-copula model from [18] on the exchange-rate dataset. Note that all values in this paper were calculated using the sampling method. We calculated these metrics based on the quantile method as well, which yielded almost similar results. While the CRPS-Sum suggests that the dummy univariate model is much better than GP-copula, the CRPS and ES indicate that the performance of the dummy univariate model is worse than GP-copula. The results reported by CRPS and ES are aligned with our expectations; however, the CRPS-Sum reports a misleading assessment. On the other hand, the quantitative results for the dummy multivariate model are quite surprising. All three assessment methods denote that the dummy multivariate has a superior performance in comparison to GP-copula. To provide further explanation for this unexpected result, we analyzed the performance of these models qualitatively.