Title: MVG-CRPS: A Robust Loss Function for Multivariate Probabilistic Forecasting
Venue: NeurIPS 2025
Accepted: no
Keywords: Probabilistic forecasting

scoring rules

time series
Abstract: Multivariate probabilistic forecasting typically leverages neural network-based distributional regression, often employing Gaussian assumptions to simplify computation. While the standard negative log-likelihood provides analytical convenience, its sensitivity to outliers can severely degrade forecasting accuracy. Conversely, robust alternatives like the Energy Score, although less sensitive to extreme values, rely heavily on computationally expensive sampling approximations, limiting scalability in neural network training. To bridge this gap, we introduce the MVG-CRPS, a novel, strictly proper scoring rule for multivariate Gaussian distributions that maintains robustness to outliers while providing a closed-form expression, enabling efficient training and evaluation. Our approach leverages a whitening transformation, decorrelating multivariate outputs and reducing the multivariate scoring task to tractable univariate CRPS computations. Experiments on real-world datasets for both multivariate autoregressive and univariate sequence-to-sequence (Seq2Seq) forecasting tasks demonstrate that MVG-CRPS enhances robustness and predictive performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the probabilistic multivariate time series forecasting task by using a novel loss called MVG-CRPS. It is developed based on the original CRPS score, and adapted to the multivariate time series scenario. The loss is evaluated on both synthetic and real-world data sets, and shows better performance than the log-score and the energy score, when evaluated with the CRPS_sum score.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  The proposed loss function and the decomposition is to the best of my knowledge novel. \n\n  The task and the motivation are well described, and the illustration of the two ways of decomposing the multivariate time series -- autoregressive and seq2seq -- with a figure is also helpful to understand the method. \n\n- Weaknesses\n\n  The biggest question to me is the evaluation of the method. That is, the paper proposes a CRPS-based loss and it compares this loss with other losses using the CRPS_sum as metric in table 1, and CRPS_mean and ES in appendix. It is not surprising to me when the proposed loss performs better on the CRPS based metrics. But I don't understand why the proposed loss also outperforms ES loss when evaluated with ES as metric. Imagine we use mean squared error (MSE) as metric and solve a regression task with MSE and MAE as losses, I would expect the MSE loss offers better results. Could you elaborate why the proposed loss works better than ES loss when evaluated with ES as metric in Table A4?\n\n  Typo in line 87\n\n  I could not find a definition of the function $\\theta(\\cdot)$.\n\nQUESTIONS:\n- It is not clear to me how seq2seq is factored, are the TS assumed to be independent or is there autoregressive applied to each TS?\n\n- What information does the column VAR in table 1 deliver?\n\nLIMITATIONS:\nThe paper discussed that the scalability of Eigen-decomposition is one limitation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new loss function for multivariate probabilistic forecasting. This loss function is based on the CRPS (a commonly used univariate score) applied to the components of a multivariate Gaussian distribution after a whitening transformation is applied. Thus, this score allows for exact computation for multivariate Gaussian distribution models (instead of requiring sampling-based estimations). At the same time, the score is robust to outliers, which is not the case for the log-score, another common score that can be computed exactly; this ensures the introduced score works better in the presence of model misspecification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The derivation of the MVG-CRPS is well explained and that scoring rule seems original.\n\nWeaknesses: \n\n- *quality/significance*: The main weakness of the paper is the following: the MVG-CRPS is motivated as solving the need of obtaining a scoring rule that can be analytically computed (for multivariate Gaussian models) while being robust to outliers (which the log score fails to be). They argue that, while the multivariate energy score is not sensitive to outliers, it requires sampling-based approximation. However, while that is true in general, I think it’s possible to obtain a closed-form formulation of the energy score for the case of multivariate Gaussian models (which is the only case in which MVG-CRPS is strictly proper and hence meaningful to apply). If this is the case, the problem that MVG-CRPS aims to solve did not exist in the first place. However, the paper fails to mention closed form expression for the energy score or why this is impossible to obtain.\n- *significance:* Even if the above point did not apply, MVG-CRPS is only meaningful for multivariate Gaussian models, which is quite restrictive, as I guess many forecasting models go beyond this assumption.\n- *clarity/quality*: In Sec 4, I don’t understand why the authors are then considering a univariate forecasting task; in those conditions, doesn’t the MVG-CRPS just fall back to the normal CRPS?\n- *clarity/quality*: Table 2 shows per-epoch and total training time. I don’t think the former matters; what matters is the overall training time, so I think highlighting the lower per-epoch time of MVG-CRPS is meaningless (plus, the difference between that and the log score is extremely small and can be due to implementation differences).\n\nMinor/notation weaknesses\n\n- *clarity* :  Sec 2.2 starts by defining scoring rules as metrics to compare predicted distributions with observed outcomes, but never introduces a notation with probabilities and outcomes, instead directly introducing $s(p,q)$, where both p and q are probabilities. This is confusing as later on ES(F,z) is introduced, where F is a cumulative density function and z is an observation. Then, lines 110-112 (characterising the ES) are quite handwavy. Finally, 115-120 fails to mention that the variogram score is proper but not strictly so.\n- *clarity:* line 159: “Eq. (4) and Eq. (6), when used as scoring rules to optimize the model, are generally referred to as the log-score” this is not strictly true, those equations are the log score for the multivariate normal distribution, but not in general.\n- *clarity*: line 180 does not mention what the allowed values of beta for ES to be strictly proper are.\n- *clarity*: Eq 11: that is not an equality: the right-hand side is the sample-based approximation of the energy score\n- *clarity*: line 193-271: z_t denotes the observation, so it is not strictly true that it is distributed according to N(µt, Σt). There should be a clearer distinction between the observation and predictive distribution.\n- *quality*: lines 209-211 are all intuition, it is not a concrete argument. It should be presented as such.\n- *clarity*: the structure of Sec 4 is confusing: 4.1 introduces some models and (but no real datasets, even though the title suggests so); then, sec 4.2 introduces a toy example, which is however unrelated to the models in 4.1. Those models appear again in 4.3 and 4.4, where however I don’t understand what is the ground truth. Then, 4.4 does not discuss the energy score anymore.\n\nQUESTIONS:\n- Does the energy score have a closed-form expression for multivariate Gaussian models? If this is not the case, can the authors explain why the corresponding integrals cannot be solved?\n- In 1D, doesn’t the MVG-CRPS just fall back to the normal CRPS?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new loss function to train multivariate probabilistic forecasting models that make a multivariate Gaussian assumption on the output.\nAuthors argue that the widely used Negative Log-likelihood suffers from quadratic growth, making it very sensitive to outliers.\nOn the other hand, Energy score, which is a multivariate generalization of CRPS score, not only lacks a closed-form expression but is also insensitive to the errors in variance.\n\nTo address these limitations, authors proposed a Multivariate Gaussian CRPS, which extends the univariate CRPS of the Gaussian distribution to a multivariate setup.\nIn MVG-CRPS, the covariance matrix of a multivariate Gaussian is transformed into a diagonal covariance matrix using the whitening transformation.\nThe final loss is the sum of CRPS on univariate marginals of the transformed distribution.\n\nUnlike Energy Score, MVG-CRPS has closed form expression (avoiding the necessity of sampling) and is also robust to outliers.\nAuthors proved that MVG-CRPS is a strictly proper scoring rule for multivariate Gaussian distributions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n[S1] The paper is generally well written and easy to read\n\n[S2] MVG-CRPS is novel and useful for multivariate probabilistic forecasting, as it is robust to outliers\n\n[S3] Experiments show that training a model using MVG-CRPS provides similar results compared to Energy Score while being faster, as MVG-CRPS does not require sampling\n\n**Weakness:**\n\n[W1] Model trained with energy score as loss function performs worse than the model trained with MVG-CRPS when evaluated for energy score (Tables A4 and A7). This is counterintuitive as we expect a model to perform better on the loss function it was trained for.\n\n[W2] CRPS-sum is the main evaluation metric (I call this main evaluation metric because its results are in the main paper, Fig. 1 and Table 1, and the results of CRPS-mean and ES are in the appendix). However, CRPS-sum is not a useful metric for evaluating multivariate distributions. [1] demonstrated that a model which is similar to a random distribution can have better CRPS-sum than a good probabilistic model. Hence, using CRPS-sum is questionable for comparison.\n\n[W3] Evaluation metrics are not helpful for the demonstration of the usefulness of the loss function:\n\n1. CRPS-sum is not useful, see W2\n\n2. CRPS-mean measures the average error on univariate marginal distributions and cannot evaluate a multivariate distribution correctly\n\n3. Energy score: authors themself mentioned that it is not a good metric to evaluate multivariate distributions ([2] has a detailed study)\n\n[W4] Continuing on W3, models were not evaluated on NLL and MVG-CRPS\n\n[W5] MVG-CRPS is similar to weighted CRPS on the reparameterized covariance matrix, with weights being the root of eigenvalues. Intuitively, it captures the multivariate nature of the distribution. However, it is not clear if it is capturing a true multivariate Gaussian distribution. A detailed analysis is required.\n\nQUESTIONS:\n[Q1] MVG-CRPS is a loss function for predicting multivariate Gaussian distributions, which is not limited to time series data. What is the reason for choosing time series data?\n\n[Q2] How do the loss functions fare when they are used for the density estimation task?\nE.g., the toy example in Sec. 4.2 can be used to generate data with some additional noise. Train three models with three different loss functions: NLL, ES, and MVG-CRPS; and demonstrate which model closely predicts the ground truth parameters.\n\n[Q3] In related work, authors mentioned, \"Probabilistic forecasting methods typically fall into two main categories: parametric methods, which assume explicit probability density functions (PDFs), and non-parametric methods, which rely on quantile estimation\". Could authors clarify in which category normalizing flows, diffusion models, etc, fall?\n\nReferences\n[1] Koochali, Alireza, et al. \"Random noise vs. state-of-the-art probabilistic forecasting methods: A case study on CRPS-Sum discrimination ability.\" Applied Sciences 12.10 (2022): 5104.\n\n[2] Marcotte, Étienne, et al. \"Regions of reliability in the evaluation of multivariate probabilistic forecasts.\" International Conference on Machine Learning. PMLR, 2023.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new learning objective for probabilistic forecasting extending the continuous ranked probability score with the aim of addressing issues related to both computational problems (sample complexity of energy score evaluation) and outlier sensitivity (poor learning under multivariate Gaussian likelihoods, or \"log-scores\", in the face of outliers).\n\nThe proposed MVG-CRPS makes use of a singular value decomposition to whiten the data before analytically evaluating the CRPS of the resultant normalized values.\nAs a theoretical contribution, the paper provides a proof that the MVG-CRPS is a strictly proper scoring rule.\n\nThe study provides a series of probabilistic forecasting experiments, comparing the MVG-CRPS to two learning objective baselines (Gaussian log-likelihood, energy score), different architectures (vector autoregression, GPVar, Transformer, N-HiTS) across a suite of datasets - including, e.g., experiments on robustness to induced outliers in the supplementary. MVG-CRPS generally outperforms log-score on CRPS-based metrics and is often, but not consistently, an improvement over using the energy score. The authors argue that MVG-CRPS is more computationally efficient based on training time measurements in their early stopping-based experiments, and also show a case study of parametrized variances and forecasts comparing MVG-CRPS to log-scores.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper's main experimental findings argue that MVG-CRPS is an on par (if not slightly better) learning objective than energy scores with (much) lower convergence times. I find that the experimental results suitably support this claim, generally, with a reasonable approach, overall, with some minor weaknesses related to, e.g., the analysis related to Table 2 (see next paragraph). The authors do openly discuss how the proposed method isn't necessarily a strong improvement to energy score in performance on, e.g., CRPS-sum. The authors provide a proof for the claim that MVG-CRPS is a strictly proper scoring rule.\n\nThe analysis related to Table 2 would be stronger if it included comparisons of, for instance, the complexity on a single training step (for-/backward pass) in terms of, e.g., FLOP and/or memory requirements (potentially with consideration of the ameliorating effect that parallel computation provides for the energy score). Consideration of the data complexity would be a strong addition (i.e., does MVG-CRPS require more data to achieve a certain level of performance compared to energy scores?) or an evaluation of gradient variance for the different methods. I appreciate the difficulty in designing an experiment that can compare times to convergence, but the main reason I find the results in Table 2 compelling is the magnitude of the difference between energy score and MVG-CRPS.\n\n\nOverall I find the clarity of the presentation decent. The paper provides suitably (if borderline overly) detailed introduction and background, detailing the motivations nicely. Since the experimental results focus on the multivariate autoregressive results, and not the seq2seq setting, the authors could consider moving details related to the seq2seq setting to the supplementary, possibly in favor of, e.g., a sketch of the theoretical results you provide in Section A and/or the outlier robustness experiments Section D. The supplementary seems to provide adequate details on their experimental setup to attempt reproduction. I find that there are corrections needed that currently reduce clarity (see \"Possible corrections\" below).\n\n\nThe work does, to the best of my knowledge, provide a novel albeit somewhat incremental contribution in a learning objective with the main insight that properties of SVD allow analytical evaluation of CRPS. It's common practice to preprocess data before whatever (neural) forecasting method sees the data. While standardization (\"to z-score\") is ubiquitous in (probabilistic) forecasting and more common than (PCA) whitening/sphering, a whitening procedure is also not uncommon as a preprocessing procedure in, e.g., neuroscience data processing pipelines. Given the compelling performance and general simplicity of the approach, I find the presented method has some minor-to-moderate significance - I find it likely that the community would find it worthwhile applying in other high-dimensional probabilistic forecasting problems and in scenarios where energy score evaluation is, for some reason, prohibitive/too high variance with low samples but the standard multivariate Gaussian log-likelihood provides too poor performance.\n\nPossible corrections:\n* l. 87: \"A scoring rule is deemed proper if it incentivizes honest forecasting, achieving its minimal expected score when the predicted probability distribution p matches the true distribution q.\" Surplus \"when the predicted distribution\"?\n* l. 138: forecast, not forecasted?\n* l. 154: DL not introduced - presumably deep learning\n* Fig 1: not starting the CRPS axis/y axis at 0 is, in my opinion, a little misleading, even if I know you're just interested in showing the most detail.\n* Figure 2 and Eq. 2: Compared to the individual time-series introduced in l. 133, why the z bolded if you refer to the individual univariate time-series? I'm not sure it's super important to make the distinction between predicting a single step ahead or predicting a range, nor between \"univariate and multivariate\", in this level of detail.\n* l. 155: it's a little confusing to me to refer to a case of $N=1$ here (\"the number of time-series is 1\" as per l. 133) and saying this is a case where \"the same model is being shared across all time series\" (l. 156). Since, I believe, the intention is to convey that the same functions $\\mu$ and $\\sigma$ are mapping from $h_{i,t}$ to univariate Normal distributions (i.e., assuming independence between the N time series, but still modeling, possibly, a $N>1$ problem - as indicated in l. 232).\n* Eq. 11: consider using $\\approx$?\n* I'm assuming it's simply a matter of having changed the naming during preparation, but: is there a particular reason you switch between MVG-CRPS and MCRPS - you're using them interchangeably, right?\n* Table 1 includes a \"VAR\" column with no introduction in the main paper; I'm assuming it's the naive baseline described in C.2, but it should probably be included in the main paper.\n\nQUESTIONS:\nSince you're specifically considering robustness to outliers as an argument in favor of the proposed method, I find your experimental evidence would be more compelling if it included a Student's t distribution likelihood baseline model. The negative log-likelihood under a Student's t distribution is much more robust to outliers and is commonly used as a robust alternative when basic Gaussian assumptions fail. Using a Student T observation model is considered in, e.g., Section 5.3 of Sprangers et al. (2023) [5]. I would suggest you add a comparison to a heavy-tailed observation model. Similarly, it might be interesting to see how MVG-CRPS handles a basic non-Gaussian toy problem; assuming that a subset of important forecasting applications have non-Gaussian distributions, it would be worthwhile knowing if the MVG-CRPS objective, while otherwise compelling, still handles, e.g., heavier-tailed distributions (like precipitation in weather forecasting).\n\nRe. Figure A1: Did you consider if the whitening causes the model to have a hard time learning accurate estimates of $\\rho$, both generally, but notably even in noise-free settings? How do you reconcile the fact that the energy score seems to estimate this parameter with less bias with the results in Figure 3?\n\nSince you're main argument for the proposed method relies on computational efficiency, I would strongly consider improving my score if the related analysis (i.e., Table 2 and beyond) could provide more robust justification for the claim (as outlined in my response under Strengths and Weakness).\n\nRelated works suggestions:\n* You could discuss, e.g., Zamo & Naveau (2018) [3] or Lang et al. (2024) [4] in related works; these studies, and related ones, discuss the error in CRPS estimation and how it affects comparison.\n* While it's very recent work, it might be useful to mention Alet et al. (2025) [6] as a recent state-of-the-art probabilistic forecasting model training towards CRPS.\n\n[1] Lam, Remi, et al. \"Learning skillful medium-range global weather forecasting.\" Science 382.6677 (2023): 1416-1421.\n\n[2] Price, Ilan, et al. \"Probabilistic weather forecasting with machine learning.\" Nature 637.8044 (2025): 84-90.\n\n[3] Zamo, Michaël, and Philippe Naveau. \"Estimation of the continuous ranked probability score with limited information and applications to ensemble weather forecasts.\" Mathematical Geosciences 50.2 (2018): 209-234.\n\n[4] Lang, Simon, et al. \"AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score.\" arXiv preprint arXiv:2412.15832 (2024).\n\n[5] Sprangers, Olivier, Sebastian Schelter, and Maarten de Rijke. \"Parameter-efficient deep probabilistic forecasting.\" International Journal of Forecasting 39.1 (2023): 332-345.\n\n[6] Alet, Ferran, et al. \"Skillful joint probabilistic weather forecasting from marginals.\" arXiv preprint arXiv:2506.10772 (2025).\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-11"}