Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary:
The authors propose an approach to calibrate conditional distribution estimation models. The approach uses normalizing flows to transform an existing model's predictions into a prediction that better matches the empirical quantiles to the theoretical quantiles. After this remapping procedure, the authors introduce a new plot to visualize calibration. Empirical benchmarks are run on a suite of UCI datasets.

Review:

I'm not sure what is really that interesting here. My high-level problems:

- The remapping that the authors propose is just using a normalizing flow with a simple quantile calibration. Why do we need normalizing flows for this at all? Any model can be recalibrated using any other model here. Is there some special reason for normalizing flows here?

- The new plot introduced is more confusing than illuminating. I really didn't follow it at all. It is very crowded and takes a lot of inspection to understand what is going on. I suspect all of this could have been visualized much clearer by using a handful of simpler plots that are straightforward.

- The benchmarks do not really compare against very competitive models. The authors choose to use a model that was only pushed to the arxiv a month ago as the baseline. Why? Then the alternative choices are strawmen: a Bayesian linear regression model, a variational dropout model, etc. There is a wealth of conditional distribution estimation literature with companion code publicly available on github (NADE, MAF, MDNs, etc). Why not use those?

- Does this really help us do anything new? Is this just "my model is 0.1% better on 8 UCI datasets than 4 other models"? Seems like a pretty uninspiring result if that's the idea.

Overall, I just don't know what to see as the big contribution here. The paper feels a little rushed and could use a slower, more methodical pace where the authors carefully think through the contribution(s) and why they're necessary. A more thorough comparison to related work is also called for.
review_title: Not sure what is novel here.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a normalizing flow approach for calibrated predictions in regression tasks. Experimental results on toy and UCI datasets demonstrate the proposed method improves model calibration.

It is not clear what the technical contributions are. Also, most of the important details are missing.

**Strengths**
- The reviewer appreciates the effort towards improved calibration models; important for reliable predictions

**Weaknesses**

*Lack of clarity*:
- The paper was difficult to follow, it omits several crucial details necessary to understand the proposed method. 

Below are some general questions or suggestions:
1) While the paper's principal focus is on calibration and recalibration, it is unclear why there are claims to address aleatoric uncertainty
2) The concept of recalibration is introduced in Section 2 as a classification problem. However, in Section 4, the focus is on regression with normalizing flows
3)  In Figures 1 and 2, what is $X, c, W_1, W_2$? 
4) Figure 2, add labels to x-axis and y-axis
5) Improve caption quality across all Figures and tables

*CDF performance plot*:
- The paper claims the CDF performance plot is one of the main contributions, yet it is difficult to interpret or follow. Why are calibrated CDFs uniformly distributed?
1) What is $\sigma$?
2) What is **pull**?
3) What is $\psi$

*Recalibration Normalizing flows*:
- What are the complete formulations of the likelihoods optimized in 1) and 2)?
-  How is normalizing flow extended to multivariate calibration?

*Weak experiments*:
- How is the *Calib*  metric computed?
- The experiments are on toy data and small UCI datasets. Additional large scale image datasets or regression tasks would strengthen the submission.

 
review_title: Difficult to follow due to missing details

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a method for calibrating uncertainty estimates for regression models. It builds off a method proposed by Kuleshov et al (2018). The newly proposed method has the following steps:
  1) Train a conditional density model for a regression outcome y given an input x on training data. The authors use conditional normalizing flows for this task.
  2) For each set of points (x_t, y_t) in a validation set, pass an input x_t to the model in step 1 to learn an induced CDF for the output F_t(). Calculate F_t(y_t), i.e. the induced CDF evaluated at the actual output, and use another flow-based density model to learn the distribution of the CDF values. If the model in step 1 is perfectly calibrated, this density should be uniform, but in practice it seldom is.
  3) For future data points, the composition of densities in steps 1) and 2) provide a new, uncertainty-calibrated density.

The overall problem area is interesting, important, and underexplored. There has been a lot of work on calibration estimates for classification tasks, but there are less methods for regression. Using flows here is a cool idea. Any exact density model can be used for the method above, and normalizing flows are a good solution because they are monotonic by construction. 

That being said, after reading the paper, I'm not convinced that the proposed method is a significant improvement over the method by Kuleshov et al. that it's building off of. Kuleshov et al. proposes a similar 2-step process, but instead of explicitly learning a distribution over the induced CDF values, it uses an isotonic regression to calibrate the CDF. Looking at the experimental results, it seems that the isotonic recalibration performs on par with the flow recalibration in terms of test set calibration error. 

Put another way, why should someone use flow-based recalibration instead of isotonic recalibration? A possible answer mentioned in the paper is that flow-based recalibration can be used to compute distribution statistics, such as the mean, while isotonic recalibration cannot compute these statistics. (Side question: why can't the isotonic recalibration be used to compute the mean? It seems like isotonic recalibration explicitly transforms an inverse-CDF to another inverse-CDF. Can't distribution statistics be imputed from this transformed inverse-CDF?). 

Even though flow recalibration can compute distribution statistics, the MSE never improves after flow recalibration; in some instances, it gets worse. What is the benefit of having distribution statistics? On the one hand, the worsened MSE might be expected behavior. Is uncertainty calibration expected to behave like a regularizer? If so, that should be stated and discussed in the paper. If not, then of course we shouldn't expect improvements in MSE after recalibration, because we're changing the model that had the best training-set performance. This could explain the results we see. In any case, there should be some justification in the paper for a) why computing distribution statistics is important, b) whether we should expect recalibration to behave like a regularizer, c) a discussion about the tradeoff between calibration performance and model error, and d) an illustration of scenarios where distribution statistics are crucial [and e), why the Kuleshov et al method can't be used to calculate test error].

Additionally, the paper proposes a way to visualize recalibration results. To be honest, I found the CDF performance plot confusing and hard to interpret. How should we interpret the x-axis (are predictions standardized, and if not, what units are they in)? I found the standard qq-plot-like calibration graph a lot more interpretable. What does the new visualization answer? I think the new visualization should be better explained (it also didn't help that the legend in Figure 5 blocked the middle of the graph).

Overall, I think the paper proposes an interesting model, but it doesn't adequately justify when/why the model should be used over the existing method. I think there could definitely be scenarios where it is useful -- I just don't think the paper has adequately and convincingly illustrated them.

Pros:
- Interesting and underexplored problem area
- Thorough experiments
- Normalizing flows are an interesting and new model for this problem

Cons:
- Doesn't justify meaningful ways the method is different from existing methods
- Visualizations are confusing
review_title: Interesting model but with unconvincing improvements over recalibration baseline

final_score: rating: 7: Good paper, accept
review_content: The paper proposes to use normalizing flows to improve estimates of aleatoric uncertainty in regression tasks. First, the paper suggests that since normalizing flows can improve the flexibility of output distribution, they can be used to mitigate issues of underfitting. Second, the paper proposes an approach that uses normalizing flows for recalibration, which allows people to still query the model’s statistical properties. The authors also introduce a plot that is useful for diagnosing calibration issues. 

The paper’s approach of applying normalizing flows in recalibration seems that it could be useful to the community. The supporting experimental results look reasonable. In addition, the presentation of the paper looks nice, the experiments are well documented, and the diagnosing plots seem like a helpful tool. Given these plus points, I would recommend acceptance. 

One suggestion is that it feels quite obvious that since flows can represent flexible distributions, using them to model aleatoric uncertainty can reduce underfitting issues. I am not sure if it is worth stating in the paper. It seems that the interesting part of the paper is recalibration, so perhaps it might be better to focus more on that. 

Questions for the authors:
- I wonder whether using flows to recalibrate is susceptible to overfitting. 
- The paper focuses on aleatoric uncertainty. How can the approach proposed in the paper be combined with approaches for addressing epistemic uncertainty?
review_title: seems like a good paper, but I am not an expert

Meta-review: The paper proposes to recalibrate predictive models by fitting a
normalizing flow on top of the predictive model on a held out validation
set using side information. At a high level this idea has some potential,
especially in the multivariate setting, but there are several directions for
improvement:

- Comparison with a broader set of baselines as suggested by the reviewers


- Clarity on why recalibrate with a normalizing flow especially in the 1-d case


- Why not any other model with explicit density? Are there other important desiderata?


- A motivating experiment that makes the potential value clear
Decision: Reject