Peacock: Multi-Objective Optimization for Deep Neural Network Calibration

Authors: Dexter Neo, Tsuhan Chen
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: deep neural network calibration, uncertainty calibration, robustness, safety, out-of-distribution

Abstract:
The rapid adoption of deep neural networks underscores an urgent need for models to be safe, trustworthy and well-calibrated. Despite recent advancements in network calibration, the optimal combination of techniques remains relatively unexplored. By framing the task as a multi-objective optimization problem, we demonstrate that combining state-of-the-art methods can further boost calibration performance. We feature a total of seven state-of-the-art calibration algorithms and provide both theoretical and empirical motivation for their equal and weighted importance unification. We conduct experiments on both in and out-of-distribution computer vision and natural language benchmarks, investigating the speeds and contributions of different components.

Decision: reject
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730300667303

thread_type:
official_review

<thread_object>
creator: reviewer_jftv

<content>
summary:
The paper proposes a multi-objective calibration strategy that relies on Pareto optimal solutions that combines several losses that have been shown to provide good calibrated networks. The approach is evaluated on several small size problems (image and text).

strengths:
- The principle of combining multiple losses designed for good calibration makes sense.

- Comprehensive evaluation and ablation study on multiple architectures and datasets.

- The results are constantly good on the evaluated problems (but only small size problems).

I rate the work as “marginally below the acceptance threshold”: I may change my score after reading the rebuttal and other reviewers' comments. .

weaknesses:
- The proposed approach is basically a multi-objective optimization using the latest calibration losses from the literature: the algorithm novelty is limited.

- I had a problem with the mathematical notations which are often not clearly defined. This makes the paper difficult to check. See the questions section.

- Evaluation is only for small size problems (according to current standards).

questions:
- I had problems with the mathematical notations. For example (but not only): 
   - l.139 and 141: is $P_i(y_k | x)=P(\{y=y_k\}|x_i)$ the likelihood or score of hypothesis $k$ when $x_i$ is the input ? What is $h_i$?
  - Eq. 1: ECE is a scalar measure that characterizes globally the predictor (Naeini et al., 2015): How can it depend on a sample $x$?
  - l. 281: what is $\bar{y}$?
  - Eq.11:  I don’t understand this equation (expectation of a non random scalar).
  - Please make your mathematical notations sound.

- How does your approach scale with larger models and datasets (ImageNet) ?

- How does the approach compare with post-hoc calibration, which is a much lighter calibration strategy, for instance [1] cited in the paper or [2] not cited? 

[1] Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods
for uncertainty calibration in deep learning. In ICML, 2020.

[2] Kanil Patel, William H Beluch, Bin Yang, Michael Pfeiffer, and Dan Zhang. Multi-class uncertainty
calibration via mutual information maximization-based binning. In ICLR, 2021
</content>
</thread_object>

</thread>
<thread>
creation date:
1730477355014

thread_type:
official_review

<thread_object>
creator: reviewer_qmtl

<content>
summary:
This paper introduces a framework (Peacock) designed to improve neural network calibration through multi-objective optimization. Emphasizing the importance of accurate model confidence in safety-critical applications, the framework proposes combining multiple calibration methods into a single framework. Peacock optimizes these methods to reduce calibration errors for in-distribution (ID) and OOD data. They operate in two modes: Equal Importance, where all methods are weighted equally, and Weighted Importance, where weights are dynamically adjusted using a self-attention mechanism. The results show that Peacock achieves better calibration accuracy across various datasets than any individual method.

strengths:
1. Peacock’s approach to combining calibration methods in a multi-objective framework is novel and reflects a strong and thoughtful attempt to address the limitations of single-method approaches.
2. Provide a theoretical foundation, arguing that Peacock’s calibration error is bounded by the average error of its components. This mathematical rigor adds to the credibility and applicability of their approach however I need to see other reviewers’ opinion on it.
3. Peacock’s use of a self-attention-based weighting mechanism in its Weighted Importance formulation is a valuable addition, allowing the framework to adapt to different calibration needs.
4. The extensive evaluation on synthetic and real-world datasets, including CIFAR variants and iWildCam, demonstrates Peacock’s potential for generalizable performance in ID and OOD contexts.

weaknesses:
1. The work could improve clarity around why certain calibration methods were selected and how they interact synergistically within Peacock. It would be helpful if the authors compared Peacock with other excluded methods to understand its compatibility and potential flexibility.
2. While the self-attention mechanism for adaptive weighting is innovative, the authors do not provide a clear explanation of how specific weights are adjusted in various calibration scenarios. Detailed case studies or visualizations could strengthen the interpretability of the model’s weighting adjustments.
3. Peacock’s performance is documented on many typical benchmarks, but it remains unclear if it scales efficiently in more resource-constrained settings (less human supervision specially) or real-time applications. Details on Peacock’s resource usage across varying computational capacities could improve the overall proposal of framework.
4. The dependence on Expected Calibration Error (ECE) and related metrics is standard, yet calibration results could benefit from additional metrics or more especially contextual tests. It would be great to see the limitation of Peacock in unusual or extreme OOD scenarios.

questions:
1. Please clarify the selection of these specific calibration techniques. How does Peacock handle any dependencies or conflicting effects among these methods, and what might its performance look like if additional calibration techniques (e.g., ensemble-based or regularization-based) were included? It will be helpful to justify the proposed framework and provide comprehensive details for future research.
2. Interpretability of Adaptive Weighting: For the self-attention mechanism, can the authors provide insights into how weight adjustments occur across different datasets? Examples of how specific weights change or stay consistent across OOD shifts would clarify the mechanism’s adaptability.
3. Handling of Adversarial and Non-Standard OOD Scenarios: Peacock performs well in ID and standard OOD settings, but could the authors elaborate on its behavior in more adversarial or highly unstructured OOD contexts? Even very limited empirical evidence will be suffice for it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688710221

thread_type:
official_review

<thread_object>
creator: reviewer_crxd

<content>
summary:
This paper presents a comprehensive survey on “Multi-objective optimization for deep neural network optimization”. Motivated from the previous literature, this paper advocate a method of unifying the previous baselines into one, dubbed the method “Peacock”. 

This method, according to figure 1(a), encompasses a few recent works: AdaFocal, Dual Focal, and MaxEnt for entropy based one, as well as margin-based CPC and regularizers based RankMixup and Post-hoc processing AdaTS. 

Experiments are conducted on many domain shift datasets, like CIFAR10/100-C, FmoW, CivilComments and so on.

strengths:
+ This paper presents a complete survey of previous works. 
+ It seems to be a good reading material for the newcomers to this field. 
+ Experiments shown in this paper indicates averaging some of the good methods can improve the performance further. This is good to be known for the community

weaknesses:
There is no solid scientific contribution in terms of novel methods. Instead, it basically composes several state-of-the-art methods and constitutes as an engineering effort to unify previous methods. I think in general this paper does not have technical algorithmic novelty, and seems to be a technical report rather than a scientific paper. 

There exists some presentation mistakes:
- In the figure of page 2, It should be Figure 1, but this caption is missing. 
- I am not sure why this method is called “Peacock”, it lacks of any interpretation but seems to be a meaningless shortcut.

questions:
As above. 

My biggest concern is this paper is taking a short cut, summarizing many previous works and put them into one in a naive manner. This might take some of the works, but not the actual contribution that excites the community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730772092721

thread_type:
official_review

<thread_object>
creator: reviewer_pwaz

<content>
summary:
The paper presents Peacock, a calibration algorithm that performs multi-objective optimization of existing calibration techniques. The method is shown to improve in and out-of-distribution calibration while retaining performance and speed on computer vision and natural language benchmarks.

strengths:
– The paper is very well written and easy to follow

– The paper does a good job of thoroughly reviewing prior work

– The paper includes a comprehensive appendix with extensive details as well as a detailed codebase which will aid in reproducibility

– The theoretical justification for the multi-objective formulation is intuitive and well-explained

– The proposed method seems effective on a range of datasets without adding a computational overhead

weaknesses:
– The paper framing refers to calibrating “deep neural networks” in general rather than classification models. From what I can tell the proposed approach is designed only with classification in mind – it would be helpful to either update the paper’s framing to reflect this, or to include additional discussion (or if possible, even a proof of concept experiment) demonstrating how Peacock may be generalized to other tasks (or even to specialized classification tasks such as dense prediction)

– The technical contribution of the paper is somewhat limited – 1) it shows that multi-objective optimization of existing calibration strategies is both theoretically and empirically better than or equivalent to any one in isolation (which is consistent with related findings in loss and model ensembling, as the paper acknowledges), and 2) it proposes an algorithm to compute “pareto optimal importance weights” efficiently. That said, it does a very thorough job of exploring various design choices, motivating its method, and reviewing the literature, due to which I do not consider this a significant weakness.

– The “weighted importance formulation” would benefit from additional motivation and . L306 claims that “loss terms can often be conflicting” as the motivation for importance weighting – it would be good to experimentally demonstrate this. Further, do any of the theoretical guarantees for the equal weighting formulation apply to the importance weighted version? Finally, why does importance weighting not consistently outperform equal weighting in Table 1 and 2? 

– Some aspects of the method design need elaboration. For eg. why is “direction weighted self-attention” required to learn loss weights, rather than say a simple linear layer?

questions:
Please see weaknesses above for a detailed list of questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733209845286

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response and Summary of Discussion Phase

comment:
We sincerely thank all reviewers for their valuable feedback and constructive criticism. We also express our gratitude to the ACs and SACs for their time and guidance.

In response to the insightful comments from Reviewers pwAz and QmtL, we have conducted additional experiments to address their concerns regarding the adaptive weighting mechanism of Peacock and its performance under adversarial attacks. These experiments have been included in our rebuttal, and we are pleased to note that Reviewer QmtL has increased their score from 5 to 6.

Additionally, we have addressed the final comments from Reviewer jFTV by conducting further experiments on scalability and post-hoc processing. We believe these experiments adequately address their concerns.

We thank the reviewers again for their insights, which has significantly improved our work. We wish you all a happy holiday season.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Peacock, a method to improve calibration through a multi-objective optimization viewpoint. The method specifically combines multiple calibration methods by determining a weighting importance for each method via a direction-weighted self-attention block. Results show improved ID and OOD calibration performance across a wide set of benchmarks. The reviewers appreciated the method's comprehensive description of prior works and attempt to unify them, but raised a number of concerns. While the idea of combining methods is intuitive (and in-line with ensembling/other variants for other domains), a number of considerations such as why particular methods were selected, how they interact (and as mentioned by authors, can conflict), and how this ties to both theory and empirical performance (e.g. closeness of results in many cases with equal weighting). Indeed, Additional concerns included issues in the writing and theoretical notations, interpretation/justification of the self-attention based weight adjustment, handling of adversarial/other OOD scenarios, etc. The authors provided a rebuttal, with a good bit of back-and-forth with the reviewers, including some additional experiments and explanations. Reviewer QmtL was overall satisfied with the response, but reviewers pwAz and jFTV expressed significant concerns were remaining. 

After considering all of the materials, I recommend not accepting this paper at this time. Overall, there were a large number of concerns expressed by reviewers, and while we appreciate the author's effort in addressing some of them, many remain outstanding. While the idea itself, especially with theoretical motivation, can be interesting, there is not sufficient scientific rigor in the formulation, demonstrating the need for adaptive weighting, its relationship to when it would/would not work given comments about considerations such as conflicts across methods, and ties to implementation (self-attention). On the flip side, the paper could use at least stronger empirical demonstration of superiority especially of the dynamic weighting component and at larger scale settings that are now common. We recommend that the authors address these to strengthen the contributions for a future resubmission.