The Disparate Benefits of Deep Ensembles

Authors: Kajetan Schweighofer, Adrian Arnaiz-Rodriguez, Sepp Hochreiter, Nuria M Oliver
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: deep ensembles, algorithmic fairness, disparate benefits, post-processing

Abstract:
Ensembles of Deep Neural Networks, Deep Ensembles, are widely used as a simple way to boost predictive performance. However, their impact on algorithmic fairness is not well understood yet. Algorithmic fairness investigates how a model’s performance varies across different groups, typically defined by protected attributes such as age, gender, or race. In this work, we investigate the interplay between the performance gains from Deep Ensembles and fairness. Our analysis reveals that they unevenly favor different groups in what we refer to as a disparate benefits effect. We empirically investigate this effect with Deep Ensembles applied to popular facial analysis and medical imaging datasets, where protected group attributes are given and find that it occurs for multiple established group fairness metrics, including statistical parity and equal opportunity. Furthermore, we identify the per-group difference in predictive diversity of ensemble members as the potential cause of the disparate benefits effect. Finally, we evaluate different approaches to reduce unfairness due to the disparate benefits effect. Our findings show that post-processing is an effective method to mitigate this unfairness while preserving the improved performance of Deep Ensembles.

Decision: reject
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1729856418499

thread_type:
official_review

<thread_object>
creator: reviewer_hhja

<content>
summary:
The paper examines how deep neural network ensembles affect group fairness, demonstrating that these ensembles provide disproportionate benefits to protected groups (a phenomenon termed the "disparate benefits effect"). The authors establish that this effect is linked to the predictive diversity among ensemble members, noting that deep ensembles show greater sensitivity to prediction thresholds compared to individual models due to their sensitivity to calibration. Additionally, they investigate post-processing techniques to mitigate the unfairness introduced by the disparate benefits effect while maintaining ensemble performance. Their findings indicate that Hardt post-processing is particularly effective, as it successfully preserves the ensemble's performance while reducing unfairness.

strengths:
The main strengths are:
1. The focus on how deep ensemble models affect fairness violations in group fairness scenarios, a relatively understudied area in current research.
2. The authors provide reasonable arguments on the why (group) unfairness emerges in deep ensembles. Their claims are thoroughly tested by conducting comprehensive analyses through controlled experiments and large-scale benchmark evaluations, examining multiple group fairness metrics under various distribution shifts.
3. The argument of using post-hoc methods to edit calibration of the predictive distribution, rather than focusing on non-homogeneous members weighting is both practical and convicing.

weaknesses:
The paper presents valuable insights into the relationship between deep ensembles and group fairness. The authors' thorough analysis could be further enriched by expanding their investigation to include other fairness definitions discussed in the literature [1,2]. For instance, while the authors focus on group fairness, their findings could be contextualized alongside the results from [3], which examines min-max fairness [2] and shows that deep ensembles improve worst-group accuracy (their findings align with the "FalseFalseTrue" phenomenon described in [4]).
To strengthen the paper's contribution to the broader fairness literature (which would help create a more comprehensive understanding of deep ensembles' impact on algorithmic fairness), I suggest either:
1. Clarifying the strategic choice to focus on group fairness and its specific implications, or
2. Preferably, extending the analysis to examine how the authors' hypotheses validity holds across different fairness definitions, particularly in relation to the findings in [3].


[1] Verma, Sahil, and Julia Rubin. "Fairness definitions explained." Proceedings of the international workshop on software fairness. 2018. 

[2] Zietlow, Dominik, et al. "Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Ko, Wei-Yin, et al. "Fair-ensemble: When fairness naturally emerges from deep ensembling." arXiv preprint arXiv:2303.00586 (2023).

[4] Lin, Yong, et al. "Spurious feature diversification improves out-of-distribution generalization." In ICLR (2024).

questions:
Continuing from the Weaknesses section of the review, given that the min-max fairness setting [1] addresses inefficiencies of statistical parity/equality measures, when evaluating fairness, given your findings on predictive diversity, it would be interesting also to understand:
1. How does your analysis of predictive diversity relate to the worst-group accuracy (WGA) findings in [2]?
2. In the context of min-max fairness, how would you assess the applicability of the Hardt post-processing method from a calibration standpoint? Are there any benefits in using it?


[1] Zietlow, Dominik, et al. "Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[2] Ko, Wei-Yin, et al. "Fair-ensemble: When fairness naturally emerges from deep ensembling." arXiv preprint arXiv:2303.00586 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730581560955

thread_type:
official_review

<thread_object>
creator: reviewer_njbd

<content>
summary:
The paper explore the impact of Deep Ensembles on group fairness through empirical studies across three datasets—FairFace, UTKFace, and CheXpert—using statistical parity difference (SPD), equal opportunity difference (EOD), and average odds difference (AOD). Their results show that while ensembles improve performance globally, they generally tend to favor advantaged demographic groups according to fairness measures, which they call "the disparate benefits effect". They link this effect to how ensemble members' predictions vary across demographic groups. Finally, the study shows that because Deep Ensembles are better calibrated than individual models, post-processing techniques, such as the algorithm by Hardt et al. (2016), are effective in mitigating these fairness disparities while preserving the performance benefits of Deep Ensembles.

strengths:
The study addresses an important aspect of Deep Ensembles by examining the fairness implications of their performance improvements.  This kind of evaluation is crucial to ensure that gains in predictive accuracy are achieved without unintended consequences for different demographic groups.

weaknesses:
- The reported differences in fairness metrics between models are often very small, making it difficult to assess the significance of the disparate benefits effect. Including the original results for individual models in Table 1 would help clarify these differences.
- The baseline selection process is unclear - while I assume the baseline represents the aggregated average across all individual models, the paper doesn't explicitly specify it (is it average results on all individual models / architectures and different seeds?). It should be clarified for proper comparison.
- The paper lacks intuitive explanations for the relationship between prediction diversity and fairness. While the average predictive diversity (DIV) shows strong differences between demographic groups, it's unclear why this specifically impacts fairness criteria. 
- The study relies solely on Hardt's post-processing method for mitigation. Comparing with other approaches, like in-processing techniques, would strengthen the claims.

questions:
1) See Point 3 above - Could you provide an intuition on how diversity among ensemble members contributes to fairness violation? Why does high predictive diversity (DIV) across demographic groups could lead to increased disparate impact or affect equalized odds? I do not see the causation;  
2) Are the calibration benefits of Deep Ensembles evenly distributed across the two different demographic groups? 
3) a) The controlled experiment lacks testing of different levels of ensemble diversity. How would increasing the number of diverse training images for A=1 (two/three/four/five different images) affect the fairness metrics (DP and EO)? Does it decreases gradually? Such experimental variations might provide additional insights into the observed relationship.

b) What are the corresponding average metrics of individual members (at least on 10 members) in this controlled experiment? Do individual members also show better fairness metrics compared to the deep ensemble?

4)The baseline selection process requires more clarification. Is it the average metrics over the 10 individual models on the 5 architectures and the different seeds?  On page 6, at the end of the caption, it states “...and the average ensemble member.” Should this instead read “...and the average of individual members”?

5) Why was the Deep Ensemble not tested with fairness constraints through in-processing methods? Testing fairness constraints during training of the members could help strengthen the claims about the suited post-processing mitigation approaches.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730647421055

thread_type:
official_review

<thread_object>
creator: reviewer_eq4t

<content>
summary:
This paper empirically investigates fairness violations in Deep Ensemble models. It finds that, in most cases, Deep Ensemble models exhibit poorer fairness performance compared to single models, despite achieving significantly better overall performance, which echoes existing research. The paper aims to understand the reasons behind the amplification of unfairness and demonstrates that post-processing can mitigate unfairness issues in Deep Ensemble models.

strengths:
This work offers a perspective on fairness issues in Deep Ensembles. 

The presentation style is clear and easy to follow.

 Additionally, I find the experiments conducted to be convincing.

weaknesses:
(1) A key limitation of this work lies in the interpretation of results presented in Section 6. The author hypothesizes that disparities in the average predictive diversity among groups contribute to the observed disparate benefits effect. However, the benefit remains ambiguous. For instance, in Figure 2, while the subgroup with A=1 demonstrates a higher TPR, the subgroup with A=0 shows a lower False Positive Rate (FPR), making the ensemble effect difficult to assess.

(2) In Figure 13(a), the disadvantaged group shows a higher TPR and nearly the same FPR as the advantaged group, which is inconsistent with the analysis in Section 6.

(3) The fairness concern addressed is the disparity across sensitive groups. However, the author's approach to explaining unfairness through performance differences does not fully explore the underlying reasons for fairness issues in deep ensembles.

(4) The proposed solution is also relatively unremarkable. The use of model-agnostic post-processing methods, which are not specifically tailored for deep ensembles, is unsurprising. The key question is how the deep ensemble model differs in effectively addressing fairness concerns compared to other high-performing models.

questions:
(1) One reason for unfairness arises from skewed data distribution. Figure 3(c) shows a much more skewed distribution compared to Figure 3(a). However, I found that the fairness issues in CX (age) are much milder than those in FF (age/gender), and I am curious why this is the case. 

(2) The trade-off between utility and fairness shows similar behavior in relation to fairness issues across classification tasks. This raises the question: what distinguishes the behavior of a deep ensemble model with strong utility performance in terms of fairness?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732033826019

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all reviewers again for their time and effort to provide their valuable feedback to our work. In response to individual reviewers’ requests, we conducted additional experiments and added results and discussions to our manuscript as follows:

* Additional controlled experiment (**NjdB**): We conducted an additional controlled experiment to examine the relationship between the average predictive diversity and changes in accuracy, SPD, EOD and AOD due to ensembling in more detail. A detailed description of the setup and results are presented in Sec.F.2 in the appendix.
* Investigation of min-max fairness (**HhJa**): Our main experiments are based on well-established group fairness metrics that capture the difference of e.g. error rates between groups. We provide an additional investigation on the recently proposed notion of min-max fairness that is complementary to the established difference based group fairness metrics. In a nutshell, min-max fairness is concerned with the worst group e.g. accuracy or error rates. Our additional experiments show that Deep Ensembles generally improve accuracy or error rates for both groups, also the worst group. Thus there is no min-max fairness impairment due to Deep Ensembles, both groups improve. However, as our main results show, the groups do not benefit equally, which we denote as disparate benefits effect. This effect often amplifies discrepancies between groups, further exacerbating unfairness according to the well-established group fairness metrics we consider in our work.
* Calibration per protected group attribute (**HhJa**): We provide a more detailed analysis of the calibration of Deep Ensembles in Fig.6a, subdividing per group attribute. A discussion is added in Sec.F.7, with the results shown in Fig.30. We observe that while individual members and the Deep Ensemble sometimes have different levels of calibration for the two groups, Deep Ensembles are generally more calibrated than individual members for any given group.

For easier reference, we highlighted new or strongly altered sections in brown in the revised version of our manuscript.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The submission studies the disparate impact of ensemble models on different groups, focusing on statistical parity and equality of opportunity. The main conclusion is that ensemble methods disproportionately benefit the majority group, and thus exacerbate the disparity between the majority and the minority groups. The authors also show that post-processing techniques are effective in mitigating the unfairness of ensemble models. 

The paper has received mixed reviews. The main concern is the lack of theoretical analysis, i.e., why do ensemble methods tend to favor the majority groups? Despite the positive-leaning ratings, during the AC-reviewer discussion, the majority of the reviewers expressed concerns about the theoretical grounding of the work. The finding of using post-processing to improve fairness is not new either, given the recent line of work in the literature on algorithmic fairness, both theoretically and empirically [1-2]. The authors should discuss and ideally adopt these methods into the empirical studies as well.

[1].    Unprocessing Seven Years of Algorithmic Fairness
[2].    Fair and Optimal Classification via Post-Processing

I strongly urge the authors to take the reviewers' suggestion and the above comments when preparing for the next submission.