CONTRA: Conformal Prediction Region via Normalizing Flow Transformation

Authors: Zhenhan Fang, Aixin Tan, Jian Huang
Primary area: generative models
Keywords: Uncertainty Quantification, Conditional Density Estimation, Conformalized Quantile Regression, Coverage Probability, Multi-dimensional Prediction

Abstract:
Density estimation and reliable prediction regions for outputs are crucial in supervised and unsupervised learning. While conformal prediction effectively generates coverage-guaranteed regions, it struggles with multi-dimensional outputs due to reliance on one-dimensional nonconformity scores. To address this, we introduce CONTRA: CONformal prediction region via normalizing flow TRAnsformation. CONTRA utilizes the latent spaces of normalizing flows to define nonconformity scores based on distances from the center. This allows for the mapping of high-density regions in latent space to sharp prediction regions in the output space, surpassing traditional hyperrectangular or elliptical conformal regions. Further, for scenarios where other predictive models are favored over flow-based models, we extend CONTRA to enhance any such model with a reliable prediction region by training a simple normalizing flow on the residuals. We demonstrate that both CONTRA and its extension maintain guaranteed coverage probability and outperform existing methods in generating accurate prediction regions across various datasets. We conclude that CONTRA is an effective tool for (conditional) density estimation, addressing the under-explored challenge of delivering multi-dimensional prediction regions.

Decision: accept (poster)
Reviewer scores: 5, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730126783373

thread_type:
official_review

<thread_object>
creator: reviewer_bedm

<content>
summary:
The authors propose a technique to compute valid prediction regions in multiple-output regression tasks. The approach is based on learning a Normalizing Flow that transforms the output conditional probability into a 'simpler' base distribution. The invertibility of the flow allows mapping valid convex intervals of the feature space, e.g. rectangles, to non-convex smooth regions in the original space.

strengths:
- Using Normalizing Flow to improve Conformal Prediction is intriguing and flexible. 
- Normalizing Flows and their likes are increasingly popular for training generative models. The proposed method could be further improved by leveraging advances in that domain.
- Multiple-output Conformal Prediction is an important but often overlooked problem.

weaknesses:
- The use of Normalizing Flow is not theoretically justified. The introduction should explain the differences compared to learning more standard feature maps.
- The authors should describe in detail the overlaps between their work and Colombo 2024, which is not mentioned with other related work but seems closely related to the proposed approach.
- It is unclear why the proposed approach is superior to existing multiple-output Conformal Prediction approaches.
- The authors should give more details about the invertible flow, $t_\theta$, and discuss possible overfitting or underfitting problems. Especially in comparison with other non-invertible probability maps (e.g.diffusion models).

questions:
- What is the difference between the proposed approach and methods that learn a feature space map before applying CP, e.g. STDQR?
- What is the difference between using a normalizing flow as a prediction model and a model for the residuals? From Figure 1, the latter seems to produce smoother regions. Is this always the case?
- Can the method be extended to tasks with many more outputs, e.g. images?
- In practice, requiring the map to be invertible may make Normalizing Flow hard to train or less flexible than unconstrained models. Have you tried Normalizing Flow with different structures?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620974508

thread_type:
official_review

<thread_object>
creator: reviewer_rpvj

<content>
summary:
CONTRA is a novel method for multi-dimensional output prediction that utilizes normalizing flows to define conformal prediction regions. By mapping high-density regions in latent space to output space, it generates more precise and connected prediction regions, avoiding the oversized or irregular regions produced by traditional methods. The paper also introduces ResCONTRA, an extension that can enhance any predictive model with reliable prediction regions. Experiments demonstrate that CONTRA not only guarantees the desired coverage but also produces prediction regions with smaller volumes and smoother boundaries, outperforming existing methods on both synthetic and real-world datasets.

strengths:
The manuscript is generally easy to follow and includes a sufficiently comprehensive discussion of relevant prior works. Experimental results are presented well.  The proposed solution has solid theoretical foundations.

weaknesses:
- In line 186, how to construct the prediction region? \hat{E} is a q-dim ball, which contains infinitely many vectors, making its implementation unclear.
- More discussion about CNF will be valuable.  
        1. Providing details about choosing q_1 in Appendix D will be better. Equal division? Moreover, for Bio dataset, the dim-y is only 1, that can not be split.   
         2. A possible limitation is that normalizing flows is notoriously difficult to train in high-dimensional spaces. Extending CONTRA to handle high-dimensional outputs (e.g., image generation) would significantly enhance its practicability.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714456325

thread_type:
official_review

<thread_object>
creator: reviewer_e6sy

<content>
summary:
This work addresses the limitation of most of the current conformal prediction algorithms which struggle with multi-dimensional outputs. The proposed method CONTRA, uses conditonal normalizing flows (CNFs) to learn a bijection, mapping a latent variable form a simple base distribution to the output variable, and then mapping the high density regions of the base distribution to the output. Using Conformal Prediction ( particularly the split conformal prediction algorithm) , CONTRA then calibrates the results to provide marignal coverage gaurantees of CP on including the true prediction region.

strengths:
1. The writing is clear, and to the point. The contributions are presented honestly and placed well with respect to the prior work. 
2. The experiments are elaborate, and show the main metrics in CP: efficiency in terms of the size ( volume ) of the prediction sets and coverage. Both synthetic and real world datasets are considered, and the results are consistent across the two settings ( synthetic vs real-world). 
3. the output regions of CONTRA do seem to visually support the authors' claims in that they are interpretable and have smoother boundaries compared to the related work. In most settings/datasets CONTRA seems to be outperforming most other baselines ( except PCP ). 
4. They address an important and practical problem (multi-modal outputs ) which extends the applicability of Conformal Prediction. 
5. Their methods seems simple to follow and implement.
6. They introduce variation of the method, to adopt to different choices of the fitted model for the conditional density estimation.

weaknesses:
1. PCP ~ ( a competing method) is able to outperform both variations of CONTRA and ResCONTRA in some cases in terms of providing smaller prediction regions while maintaining coverage guarantees.
2. the authors mention instances in which CONTRA could deteriorate in performance. Particularly when its hard to learn bijection well. but no results are provided to illicit the deterioration trends or to show the remedy solutions discussed by the authors can help.

questions:
1. with respect to point 2 in the weaknesses, how do competing methods, particularly PCP, are expected to perform in cases that CONTRA is not able to perform well. A discussion ~ potentially a brief experiment on synthetic data ~ where we understand the trend of performance deterioration in CONTRA, and the effectiveness of the solutions proposed on page 10,  could illicit a more clear choice to follow in whether one should try to apply the proposed solutions or go with another method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716314054

thread_type:
official_review

<thread_object>
creator: reviewer_vtsj

<content>
summary:
The authors propose a new conformal prediction method for multi-dimensional regression. It is based on fitting a (conditional) normalizing flow in the space of outputs. Authors perform numerical experiments on public datasets that illustrate certain benefits of the proposed method.

strengths:
- The proposed method makes a lot of sense as flexible estimator of conditional label distribution is obviously required to achieve optimal set size, especially in multidimensional case. 
- The results are adequately explained.
- Experimental results show certain benefits of the proposed approach in terms of prediction set size.

weaknesses:
- The proposed approach boils down to application of the normalizing flows to the split conformal prediction framework. There are no technical difficulties on this way. Any other conditional density estimator can be used here instead.

- All the theorems in the paper are standard (not sure, why even proofs are provided). There seems to be no theoretical contribution for the present paper.

questions:
- Why don't you study other density estimation methods? 

- Performance of normalizing flows heavily depends on the size of the dataset. How will it influence the results of the proposed method? I think that this should be studied.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732262043793

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you very much for your valuable suggestions and insightful questions. Based on your feedback, we have revised our draft, highlighted the updated sections in blue, and carefully replied to all the questions. These revisions and responses address concerns regarding the choice of normalizing flows for conformal prediction, additional comparisons with competing methods (e.g., diffusion models with PCP and ST-DQR), more details about normalizing flows (NFs), the novelty of ResCONTRA, and discussions on issues of underfitting/overfitting, strategies to avoid them, and performance with small training sizes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The main claim of the paper is that conformal prediction with multidimensional outputs can be effectively handled using a new proposed method called CONTRA. 

The method is simple to implement and outperforms most baselines, and addresses an important and practical problem in CP.  Therefore, it is of interest to the wider CP community attending ICLR. 

The main weakness of the work is that the theoretical results (Theorems 1-3) are existing standard results, and the proposed method is a rather straightforward applications of normalizing flows to conformal prediction. 

I encourage the authors to take the reviewer's feedback into account for the final version of the paper.