## 5 Closer Look into CRPS-Sum in Practice
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **GP-copula** & **Dummy Univariate** & **Dummy Multivariate** \\ \hline CRPS-Sum & 0.0070 & 0.0049 & 0.0048 \\ CRPS & 0.0092 & 0.4425 & 0.0077 \\ ES & 0.0043 & 0.2037 & 0.0032 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: This table illustrates the results from dummy models on the exchange-rate dataset and compares their performance with GP-copula based on CRPS-Sum, CRPS, and ES. It shows that CRPS-Sum dummy models have better performance in comparison to GP-copula. Figure 3a depicts the forecasts from GP-copula for the first dimension of the exchange-rate dataset (the rest of the dimensions are visualized in Appendix A) and Figure 3b presents samples from the dummy multivariate model. This experiment shows us that the border between a dummy model and a genuine model can become very blurry if we rely solely on CRPS-Sum. Furthermore, we learn that CRPS and visualization can help us to acquire a better understanding of model performance. In the second experiment, we performed a similar experiment on the taxi dataset [36]. The taxi dataset contains the spatio-temporal traffic time-series of New York taxi rides taken at 1214 locations every 30 min in the months of January 2015 (training set) and January 2016 (test set). This dataset consists of 1214 dimensions. Table 2 presents results for the experiment on the taxi dataset. In contrast to the previous experiment on the exchange-rate dataset, we cannot examine the discrimination ability of CRPS-Sum by comparing it to other metrics in higher dimensionalities. As mentioned in Section 3.2, the ES is not a reliable indicator of model performance in higher dimensionalities. CRPS cannot reflect the dependency structure learned by the model. Furthermore, we cannot crosscheck the CRPS-Sum discrimination ability with the qualitative performance of the model, since it is not possible to investigate the model's performance intuitively due to the size of data and the unintuitive nature of time-series data. For instance, Figure 4a,b illustrates the performance of GP-copula on the dimensions where the model has the best and the worst performance based on CRPS. By comparing these two figures, we can perceive clearly that the qualitative analysis of model performance is not feasible and straightforward. This experiment emphasizes again the importance of a strictly proper scoring rule for probabilistic multivariate time-series forecasting, which is sound in its definition and analyzed carefully with real-world datasets with low dimensionalities. Figure 3: These figures illustrate the 400 forecast samples from GP-copula and dummy multivariate model for the first dimension of the exchange-rate dataset, alongside the expected ground truth. While dummy multivariate model forecasts look like random noise as expected, it is hard to spot any meaningful pattern in GP-copula forecasts in comparison to the expected value. **(a)** Visualization of GP-copula samples. **(b)** Visualization of forecast samples from dummy multivariate model. Figure 4: These figures illustrate the 400 forecast samples from the GP-copula model on the taxi dataset. (**a**) Visualization of the GP-copula model on four dimensions of the taxi dataset with the best performance based on CRPS. (**b**) Visualization of the GP-copula model on four dimensions of the taxi dataset with the worst performance based on CRPS.