Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds
Authors: Michael-Andrei Panaitescu-Liess
Yigitcan Kaya
Sicheng Zhu
Furong Huang
Tudor Dumitras
Venue: ICLR 2024 poster (2024)
Keywords: poisoning
adversarial machine learning
group robustness
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We identify that inadvertently (i) approaches to group robustness without annotations amplify poison samples and (ii) poisoning defenses eliminate legitimate minority samples.

Abstract: Group robustness has become a major concern in machine learning (ML) as conventional training paradigms were found to produce high error on minority groups. Without explicit group annotations, proposed solutions rely on heuristics that aim to identify and then amplify the minority samples during training. In our work, we first uncover a critical shortcoming of these methods: an inability to distinguish legitimate minority samples from poison samples in the training set. By amplifying poison samples as well, group robustness methods inadvertently boost the success rate of an adversary---e.g., from 0\% without amplification to over 97\% with it. Notably, we supplement our empirical evidence with an impossibility result proving this inability of a standard heuristic under some assumptions. Moreover, scrutinizing recent poisoning defenses both in centralized and federated learning, we observe that they rely on similar heuristics to identify which samples should be eliminated as poisons. In consequence, minority samples are eliminated along with poisons, which damages group robustness---e.g., from 55\% without the removal of the minority samples to 41\% with it. Finally, as they pursue opposing goals using similar heuristics, our attempt to alleviate the trade-off by combining group robustness methods and poisoning defenses falls short. By exposing this tension, we also hope to highlight how benchmark-driven ML scholarship can obscure the trade-offs among different metrics with potentially detrimental consequences.

Published as a conference paper at ICLR 2024
LIKE OIL AND WATER: GROUP ROBUSTNESS METHODS AND POISONING DEFENSES MAY BE AT ODDS
Michael-Andrei Panaitescu-Liess University of Maryland College Park, MD, USA mpanaite umd.edu
Yigitcan Kaya University of California Santa Barbara, CA, USA yigitcan ucssbeddu
Sicheng Zhu University of Maryland College Park, MD, USA sczhu @umd.edu
Furong Huang University of Maryland College Park, MD, USA furongh @ udd.edu
Tudor Dumitras
University of Maryland College Park, MD, USA tudor@ aedor@umd.com umddedd
ABSTRACT
Group robustness has become major concern in machine learning (ML) as con- ventional training paradigms were found to produce high error on minority groups. Without explicit group annotations, proposed solutions rely on heuristics that aim to identify and then amplify the minority samples during training. In our work, we first uncover a critical shortcoming of these methods: an inability to distinguish legitimate minority samples from poison samples in the training set. By ampli- fying poison samples as well, group robustness methods inadvertently boost the success rate of an adversary e.g., from 0% without amplification to over 97% with it. Notably, we supplement proving this inability scrutinizing recent poisoning defenses both in centralized and federated learning, we observe that they rely on similar heuristics to identify which samples should be eliminated as poisons. In consequence, minority samples are eliminated along with poisons, which damages group robustness--e of the minority samples to 41% with it. Finally, as they pursue opposing goals using similar heuristics, our attempt to alleviate the trade-off by combining group robustness methods and poisoning defenses falls short. By exposing this tension, we also hope to highlight how benchmark-din drrivvn ML scholarship can obscure the trade-offs among different metrics with potentially detrimental consequences.
empirical evidence with impossibility result standard heuristic under some assumptions. Moreover,
from 55% without the removal
INTRODUCTION
As ML finds adaptations in many fields with diverse priorities, new metrics of success, aside from prediction performance (e.g., accuracy), have come into play. For example, in security-critical applications, robustness to adversarial examples (Chen et al. 2021 or poisoning attacks (Steinhardi al. 2017); or, demographically--ennn sensiiive applications, faurness (Hashimoto et al., (2018) or group robustness (Liu et al. 22021) are popular metrics the MI community aims to improve. The sheer number of such metrics has led a paradigm where researchers demonstrate progress on benchmarks often designed with single metric in mind.
Recent work has exposed previously unknown trade-offss between some of these metrics, e.g., between privacy and fairness (Bagdasaryan et al. (019) between robustness and privacy (Song et al. 2019). As applications in practice require balancing various, often mission-critical, metrics, such unknown trade-offs might have catastrophic consequences. This makes the research into studying the intersection of multiple metrics to identify tensions and interactions particularly crucial. With this motivation, in systematical quantitative study, our work uncovers an inherent tension between approaches designed for two critical metrics: group robustness methods and poisoning defenses.
$ Equal contribution.

---

Published as a conference paper at ICLR 2024
Group robustness has become concern as standard training via empirical risk minimization (ERM) has been shown to perform well on an average sample but poorly on samples belonging to under- represented, minority groups (Tatman 2017). Effective solutions, such minority upsampling (Byrd & Liptonn] 2019), are not always feasible as the explicit group annotations they rely on are often not available due privacy (e.g., demographic annotations) or financial concerns (e.g., large-scale data sets). To this end, research has proposed heuristics for identifying the minority training samples as proxy for annotations (Liu et al. 202I). A common observation behind these heuristics is that minority samples are often difficult to learn and the model cannot achieve low training eπror on them. The candidates identified as minority samples are then amplified during training, e.g., through upsampling, which is shown to improve group robustness significantly.
when the training set contains poison samples, group robustness heuristics cannot distinguish legitimate minority samples from them. Poison samples are injected by an attacker to teach the model an undesirable behavior, e.g., a backdoor (Saha et al. 2020). As result, two recent group robustness methods (Sohoni et al. 2020 Liu et al. 2021) end up assisting the attacker by encouraging low error on poison samples along with minority samples achieves 15- 97% higher success rate due amplification. Aiming to understand this vulnerability, we observe that poison samples can be as difficult to learn as minority samples, especially in a realistic attack that can inject only a few samples. This suggests that any group robustness method that relies on difficulty-basee
First, we expose a vulnerability (Figure D:
heuristics might carry a similar vulnerability. To supplement our empirical results, we prove that under specific assumptions, loss-based difficulty heuristics cannot distinguish between legitimate minority samples and poisons.
Second, on the other side of the coin, we show that poisoning defenses hurt group robustness when the training set contains legitimate minority samples. In particular, we focus on recent sanitization-bast defenses in centralized (Yang et al., 2022) and federated learning (Panda al. 2022) settings. As is challenging to detect poisons reliably (Shan et al. (2022), these methods pursue a simpler goal by relying on heuristics to identify outliers. Such heuristics are empirically shown to eliminate poisons without hurting the overall accuracy. However, due to providing distinct learning signals during train- ing, we observe that minority samples are often inadvertently identified (and eliminated) as outliers. This poses trade-off for the defender: the more poisons are eliminated (lower attack success), the more minority samples will be eliminated as well (lower group robustness). In consequence, the accuracy on the minority group drops by up to 15% after applying an effective defense.
Finally, we make an (unsuccessful) attempt to mitigate these tensions by applying poisoning defenses and group robustness methods in tandem. When aiming for low attack success, the defense removes enough minority samples to render group robustness methods ineffective. When aiming for high group robustness, the defense ignores enough poison samples that are still amplified by group robustness, which leads to high attack success. Ultimately, this implies an unintended alignment between defensive and group robustness heuristics. We hope to encourage future work to develop heuristics that conciliate these two critical success metrics.
In summary, we make the following contributions: (i) We show experimentally that group robustness methods fail to distinguish minority groups from poisons. which leads to the risk of amplifying the attacks, and complement our findings with a theoretical result (Section 4 (ūi We find that poisoning defenses also fail to distinguish poisons from under-represented groups and they introduce a risk of lower group robustness in both centralized and federated learning (Section 5 and Appendix A.4;; (iii) We demonstrate that a straightforward combination of group robustness methods and poisoning defenses cannot fully mitigate these tensions (Appendix A.5
2 RELATED WORK
Group Robustness Methods. Group robustness focuses on training models that obtain good performance on each pre-defined group in the data set. Approaches to group robustness fall into two broad categories. The approaches proposed by [Sagawa et al. (019)): Byrd & Lipton| (019 Cao et al. (219)) rely explicit group annotations during training. For example, group distributionally robust optimization (group DRO) (Sagawa et al. 2019) directly minimizes the worst-group error on more realistic scenario where annotations
the training set. The second set of approaches focuses
are not available during training (Liu et al. 2021 Nam et al. 2020) [Shhoni et al., 2020; Namkoong
& Duchi 2017). Our work focuses on promising type of approach within this set that essentially
2

---

Published as a conference paper at ICLR 2024
Spurious features
Class
Class 1
Poisoned sample
Misclassified sample
Core features
lif
Amplify
Illustration of group robustness methods without (left) and with poisons (right) in the training set. For the former, the methods operate regularly as they identify and amplify the minority groups, but for the latter, they also amplify poisons and, therefore, the attacker's influence over the decision boundary. The left and the right panels of each figure correspond respectively to the identification phase of the group robustness methods and the resulting model trained after this phase. Note that the lighter-colored circles and triangles represent amplified points.
Figure 1:
aims to obtain pseudo group labels by deploying various heuristics (Liu et al. 2021 We include
discussion on the heuristics used by several other group robustness methods (Zhang al. 2022
LaBonte et al., 2022 Kim et al. 2022 LaBonte et al. (024) in Appendix[A.2 (A.2
Poisoning Attacks. In poisoning attack, the adversary injects set of malicious samples into the victim's 's training set. The poisons are designed to induce certain vulnerabilities in the victim model. In dirty-label attacks, the adversary fully controls how the injected sample crafted and labeled (Gu et al. 2017). In clean-label attacks, however, they can only make minor changes to existing training samples without changing their ground truth label (Suciu al., 2018). terms of the attacker's goal, indiscriminate attacks hurt the model's overall accuracy (Koh et al. (022), targeted attacks cause misclassifications on specific samples (Shafahi et al., 2018), and backdoor attacks teach the model to misclassify any sample that contains a trigger pattern (Gu al. 2017). In our work, we use range of poisoning attacks to demonstrate a limitation of group robustness methods in distinguishing legitimate samples from poisons.
Poisoning Defenses. Based on their core assumptions, poisoning defenses can be split into three. (i)
First category corresponds to the assumption that poisons are difficult to be learned and this includes data sanitization defenses that detect anomalies and outliers the training set (Steinhardt et al., 2017] Chen et al. 2019; Yang al., 2022p. Note that these are popular strategies against poisoning attacks, because of their low computational overhead and minimal impact on accuracy. In this work, we focus on defenses from this category and show that they lead to the problem we identify as they end up eliminating difficult-to- to-eur minority samples as poisons. (ii) Defenses from the second category assume that poisons are easy to learned, making them suitable against strong adversaries (Li et al.) 202Tb. (ii) Third category corresponds to the assumption that poisons follow different distribution from clean samples. State-of-the-ae clean samples separate the clean training points from the poisons. We discuss the limitations and challenges of using defenses from the second and third categories in Section
defenses (Pan et al. 2023 Qi et al., 2023) use a small base set of
3 PROBLEM FORMULATION
3.1 DEFINITIONS
In this part, we draw inspiration from Sagawa et al. (2019) to define the notion of groups.
Groups. Given the set of class labels y and spurious attribute (feature) labels A, we consider a data generation process such that X is generated using class Y and a spurious attribute A. We denote data point by a triplet (x. y, a) and its group (label) by the tuple (y, a) E y x A. The data distribution of group (v. a) over X' x ) A is given by (D(y.a) (1.a))3 3, a). Note that this definition only refers to legitimate and clean-label groups. Later, in Section 4.3 we will also consider a (restricted) definition for poisons as darty-label samples generated from legitimate group.
3

---

Published as a conference paper at ICLR 2024
Minority and Majority Groups. Since prior work is missing formal definition for minority and majority groups, we propose one here. Given a dataset D containing samples from the groups. consider that a group = (y, a) E y x A is minority group if and only if the number of samples from is significantly lower than the average amount of samples from each group, i.e. lgl < [D[-IA] Z [hl, where [R] denotes the number of samples of group k y x A from D and [29] and |A] denote the cardinals of the sets y and A. respectively. Also, we consider that group is a majority group if and only if it is not minority group.
3.2 SETUP
Datasets. We consider two benchmark data sets: Waterbirds (Sagawa et al. 2019) and CelebA (Liu
al. 2013). Waterbirds contains 4, 795 training images of "land-bird" and "water-bird" classes, on either land or water backgrounds. CelebA contains of 162, 770 training images of faces, either male or female, and the task proposed by [Sagawa et al. (2019) to classify them as "blond" or "not blond". In our experiments in Section we randomly sample 10% of CelebA. We include experiments with different subset sizes for CelebA in Appendix A.3 and all the other experiments from the paper are performed on the full dataset. We refer to the lowest-represented group (water-birds on land and blond males) as LRG-1; and the highest-represented groups (land-bird) on land and water-birch birds on water blond females, non-blond females and non-blond males) as HRG. Additionally, Waterbirds contains a second under-represented group (land-birds on water), which we refer to as LRG-2
Models. Following prior work (Liu et al. 2021 we consider standard ResNet architectures (He et al.] 2016), starting from ImageNet-pretraiie ResNet-18, then we include additional results on ResNet-50 as well as for the scenario when models are trained from scratch in Appendix A.3
weights. In our main manuscript, we show results on
Group Robustness Methods. We consider two popular techniques from recent work: Just Train Twice (JTT) (Liu et al. 202I) and GEORGE (Sohoni et al.] 2020). Both methods have two main phases: (i) identification uses heuristics to identify pseudo group annotations for training samples; and (ii) amplification uses these annotations to amplify under-represented groups. In (i), JTT trains a heavily regularized model via ERM and identifies the training samples this model misclassifies as belonging to an under-represented group. In (ii), it simply upsamples these samples to train a second model that achieves higher group robustness. On the other hand, in (i), GEORGE trains a standard model via ERM, clusters its latent representations on the training samples, and treats the cluster labels as pseudo group annotations. In (ii), it applies group DRO (Sagawa et al. 2019) on these groups, which ends up amplifying the samples in smaller clusters under-represented groups.
Poisoning Attacks. We consider (i) a dirty-label backdoor (DLBD) attack that inserts samples containing& trigger with wrong label; (ii) a sub-population attack (SA) that targets a specific group indiscriminately (Jagielski et al., 202T); and (iii) Gradient Matching (GM) (Geiping et al.) 2020), state-of-the-art clean-label targeted poisoning attack.
Similar to prior work, we consider that 1% of the training set is poisoned for DLBD and GM, and 2% for SA. We also consider other poison percentages in Appendix A.3] For GM, we select the base samples (i.c.. the clean training samples the attack modifies into poisons) from LRG-1. For DLBD and SA, we select them from HRG and label them into the same class as LRG- For GM, we select 5 target samples from the class that does not contain LRG-1 and launch the attack to force the model to classify them as the class that contains LRG-1. We also consider setting with 100 target samples and show the results in Appendix A.3 Note that we consider that the attacker is aware of which groups are the easiest to learn so they can craft their poisons accordingly.
Poisoning Defenses. In our centralized training experiments, we consider EPIc (Yang et al. 2022). a state-of-the-an technique that iteratively (i) identifis the training samples isolated in gradient space outliers, and (ii) eliminates them potential poisons during training. Additionally, we also consider STRIP (Gao et 2019). a run-time backdoor detection defense, and include the results in Appendix A.4 In our federated learning experiments, we consider several robust aggregation mechanisms and poisoning defenses, including coordinate median update (Yin et al.) 2018). Trimmed Mean (Yin et al. 2018), and SparseFed (Panda et al. (022). These techniques aim to sanitize the updates sent by clients and prevent the model from learning outliers.
In Appendix A.1 we provide additional details about our setup and hyper-parameters.
4

---

Published as a conference paper at ICLR 2024
3.3 RELEVANT METRICS
We perform each experiment 3 times and report the average and the standard deviation of its results.
Accuracy Metrics. We report two metrics (as percentages) for the prediction performance of a model. First, we report the standard test accuracy (denoted as ACC) measured over the entire test set. ACC is dominated by HRG as it does not consider the group labels. To report the group robustness of model, we measure the Worst-Group Accuracy (denoted WGA). For WGA, truth annotations to measure the accuracy on each group (i.e., the percentage of the correctly classified samples from each group). We then report the lowest accuracy among all groups as WGA.
use the ground
Identification Success Metrics. Group robustness methods idcally identify and amplify only the samples in legitimate minority groups, whereas the other groups (including poisons) remain untouched. To report how close we are to the ideal, we measure the Identification-
(denoted as IDNF) of each ground truth group individually. IDNF measures what percentage of samples in the group end up being amplified. A small gap between IDNF on poisons and IDNF on LRG indicates failure scenario.
Attack Success Metries. In all attacks, we report Attack Success Rate (denoted as ASR)
percentage. The actual measurement ASR depends on the attack. For DLBD, we measure the percentage of test samples that correctly classified in the absence of the trigger, but misclassified in its presence; for GM, the percentage of the misclassified target samples, and for SA, the relative accuracy drop the target group of the attack defense not only prevents the accuracy drop by removing the poisons but also slightly increases the accuracy on the targeted group, then the ASR for the SA would be negative.
a non-poisoned model. Note that in case
Defense Success Metrics. An ideal defense only eliminates poisons and reduces the ASR, leaving ACC and WGA intact. For EPIc, measure the Elimination-Factor (denoted as ELMF) as the percentage of training samples removed from each individual ground truth group. A large gap between ELMF on LRG and ELMF on HRG indicates disparate impact. For the federated learning defenses (that operate on client updates, not on training samples), we report the drop in WGA and ACC over an undefended model.
LIMITATIONS OF GROUP ROBUSTNESS METHODS
In this section, show that heuristics deployed by group robustness methods identify the poisons as under-represented and amplify them. We illustrate this vulnerability in Figure We also study the implications of this limitation regarding the ASR of poisoning attacks.
4.1 GROUP ROBUSTNESS HEURISTICS IDENTIFY POISONS
We start by examining which samples are identified by group robustness methods. For JTT, these are the samples misclassified in the first phase, and, for GEORGE, the samples in the smallest cluster. In [ we present the IDNFs on each group (LRG-1, LRG-2, and HRG) for each method. We do not consider LRG-2 for GEORGE because creates clusters for each class individually and our poisons are only in the same class as LRG-1. In all experiments, the smallest cluster ends up in the same class as LRG-1, which indicates that GEORGE working as intended.
Across the board, we see that poisons and LRG-1 samples have the highest IDNF between 93.4% and 100%) Most notably, in many cases, the IDNF on poisons is up 5% higher than the IDNF on LRG-1. This suggests that group robustness methods are more likely to amplify poisons than legitimate under-represenned samples. For Waterbirds, there is an expected gap between the IDNFs on LRG-1 and LRG-2& LRG-2 contains over 3x more samples than LRG-1, which makes it less difficult to learn. Finally, in all settings, the IDNF on HRG is much lower than the rest-at most 12.5% in the case of GEORGE on Waterbirds.
We also experiment with letting GEORGE automatically adjust the number of clusters for each class by maximizing the silhouette score, done by Sohoni et al. (2020). This still ends up creating small cluster that mostly contains the poisons and LRG-1, meaning that it has failed to distinguish between represeepeee groups and poisons.
5

---

Published as a conference paper at ICLR 2024
Table 1: The CrentificationnFactooo (IDNFs) of group robustness methods different groups of samples in the training set. We highlight the alarming cases where the poison samples are more amplified than the lowest-represented group.
METHOD
DATASET
ATTACK
POISONS
LRG-1
LRG-2
HRG
JTT
WATERBIRDS
DLBD
98.5 + 1.2%
94.6 +: 1.4%
36.9 +: 1.8%
5.0 +: 0.3%
JTT
WATERBIRDS
SA
98.2 1 0.6%
94.6 1.7%
38.7 1 3.1%
4.8 - 0.3%
JTT
WATERBIRDS
GM
100.0 0.0%
96.2 6.4%
35.3 1.9%
5.4 0.3%
JTT CELEBA
DLBD
0.0%
96.7 + 0.2%
N/A
9.8 + 0.5%
GEORGE
WATERBIRDS
DLBD
98.5 + 1.2%
93.4 + 1.0%
12.5 + 1.2%
Takeaways. The heuristics used by group robustness methods achieve a high recall in identifying LRG. However, they often identify most of the poisons minority group, betwen 98. 28. and 100% which even exceeds the recall on the legitimate LRG. Overall, this supports our claim that current group robustness methods are limited in distinguishing between minority groups and poisons.
4.2 GROUP ROBUSTNESS METHODS AMPLIFY POISONS
After finding that group robustness methods identify poisons as an under-represented group, in this section, we study how this impacts poisoning attacks and their success. To this end, we consider three evaluation settings. In the standard case, we apply the group robustness method as-is. In the ideal and worst cases, we intervene in the method to prevent it from amplifying any poison or force it to amplify all poisons, respectively. These interventions aim to isolate the impact of amplifying poisons on the attack success rate (ASR). We implement these interventions by manually removing all poisons from (ideal) or placing all poisons into (worst) the set of samples identified by group robustness methods.
We present the results in Table 2 across different attacks, methods, and datasets. We first note that the ASR gap between the worst case and the standard case is often minimal. This highlights that the boost the attacker gains due to the shortcomings of group robustness heuristics is significant it can get. The large gap between (6.7% 97.4 47.4%) the standard and ideal cases shows an opportunity for better heuristics. We believe the larger amplification in the case of CelebA stems from the fact that this is a more complex data set with more variability and samples, which makes it easier for poisoning. Note that most of the models maintain relatively high WGA (at least 74.1%), which shows that group robustness methods are working as intended, but still worse than the case without any poisoning (86.7% in Liu al. (022I). The only exception this is the case of SA, where the WGA drops to 60.9%. However, this is expected as the goal of this attack is to hurt the accuracy on specific group. Finally, the models tend maintain a fairly high standard ACC, except against SA as it is an indiscriminate attack, which provides sanity check to our results.
Additionally, in Appendix A.3 we study the impact of the apper-parameters (early stopping for the identification model and upsampling factor) and consider more settings (different amount of poisoned samples, more targets for GM attack and using a larger model, as well as training the models from scratch). We observe that the results are consistent with our previous findings.
Takeaways. Due to identifying poisons as an under-represented group, group robustness methods end up directly or indirectly amplifying them. We show that this leads to boost in the success rate of poisoning attacks and, generally, this boost is almost as high as in the worst-case scenario.
4.3 IMPOSSIBILITY RESULT
In this section, show that under some assumptions, loss-based identification methods are inherently ineffective in separating legitimate minority groups from poisons.
We consider binary classification problem with one binary spurious attribute ()' ()'= A (0, 1)). We denote the identification model (i.e., the model used in the identification phase of the group robustness methods) by X - 1] and its output probability corresponding to the 7th class by Txx:x (i.e., the softmax output on class i). For group (y,a) e y A. denote

---

Published as a conference paper at ICLR 2024
Table 2: Evaluating the impact of amplification in group robustness methods worst group accuracy (WGA), attack success rate (ASR), and test accuracy (ACC). We highlight when there is small gap in ASR between the worst and standard cases as this indicates that method has given an advantage to the adversary.
METHOD
DATASET
ATTACK
CASE
WGA
ASR
ACC
WORST
76.9 + 3.5%
20.9 7.9%
86.4 + 1.2%
JTT
WATERBIRDS
DLBD
STANDARD
78.0 + 4.1%
20.4 4: 5.9%
86.7 + 1.2%
IDEAL
81.4 0.9%
0.5 1 0.3%
91.0 1 0.4%
WORST
40.9+4.5%%
24.0 + 3.3%
70.9 2.3%
JTT
WATERBIRDS SA
STANDARD
61.6 6.8%
31.4 4 6.0%
66.0 - 3.8%
IDEAL
0,8 1 1.3%
90.8 + 0.3%
WORST
76.1
20.0 0.0%
89.9 1.0%
WATERBIRDS
GM
STANDARD
76.1 3.2%
20.0 0.0%
89.9 1.0%
IDEAL
75.9 0.8%
13.3 1 11.5%
91.3 + 0.1%
WORST
19.7
97.7 0.1%
82.9 0.8%
JTT
CELEBA
DLBD
STANDARD
79.3 0.2%
97.7 0.1%
82.6 + 0.2%
IDEAL
79.2 1.1%
0.3 1 0.0%
83.6 1.4%
WORST
74.1 1.7%
16.5 + 2.9%
76.4 I 3.4%
GEORGE
WATERBIRDS
DLBD
STANDARD
77.3 2.5%
15.8 + 3.9%
79.4 1.6%
IDEAL
79.3 + 2.1%
0.4 0.0%
93.1 1.4%
Py.a := ((aaaaa)~(D (y. [ [ y (i.e., the expected class y probability output of the identification model on a sample from the group (y, a)). In the rest of the paper we will refer to this as expected class probability. We consider all the groups (minority and majority) to be clean-label and poisons to be dirty-label. To generate poison samples starting from legitimate samples from group (v. a), the FeatureMatch and Label Flipping
[1()),
the attacker needs to flip their label (this corresponds setting of the Subpopulation Attack from Jagielski et al. (202I)), so the distribution for the poisons (v, a)* (D(y,a); y. a) and we define Pz.a E -...1-v.a)~ (D)) (1x) 1-y]- We denote the maximum expected class probability number of darty-label samples is generally much lower than the number of clean samples (a realistic attacker cannot insert lot of poisons), we can assume that Pym.am. maa y. (.0.)).. (0.1)Pp..-
group Pym.aa. maF (0, L)yy.a Note that the
Lemma 4.1. For the setting described above, if we assume that there are no ties in maximum expected class probability among groups, then the identification model has less expected class probability on the poisons (ym, am)" in comparison to any legitimate group.
Proof. We include the proof Appendix A.6
Theorem 4.2. We consider the same setting as in Lemma 4.1 We denote the poisons (ym, am))" )* by Ip and let (y. a) = 9e be any group of samples (e.g., legitimate minority group). Also, we denote 1x)) for (x,, ym, um) (D 9D ymu a.m) by Gp and I(x), for (x. y, a) (Doc: y, a) by G. and the cross-entropy loss on E (Ge: Gp) by L(G). We assume Gp and G. are independent and ar (Gr( ())== = Var(G.) o² (i.e., the variances of the class probability for the identification model are equal for the legitimate group and for the poisons) and denote B(Gp) := Hp and B(G.) = Pe-
Then, for any 1), if vi-e
Me-Hr 2
we have (L) < L(G,)) >1 1
Proof. We include the proof Appendix A.6
Takeaways. An identification model amplifies hard-to-learn samples they are hypothesized to approximate legitimate minority groups. Our result Theorem 4.2 undermines this common hypothesis by proving that, under some conditions, the poisons are harder learn than minority group samples and, therefore, based thresholding will fail to separate them with a high probability. The strong correlations among popular sample difficulty metrics (Wu et al.) 2020), such as the loss value or learning epoch, give our result broader applicability despite our theorem's 's focus on a prototypical loss-based thresholding scheme.
7

---

Published as a conference paper at ICLR 2024
LRG2 LAG1
LAG2 LAGI Possons HRG
1120
samples eliminato
O1 cample eliminates
Pelsore
Poisons
HiG
HRG
samples
LRGI
Epoch
Epoch 10
Epech
HRG
Epoch
Figure 2: The elimination disparity between the under-representedd (LRG) and over-represented (HRG) groups against EPIc. The x-axes show the iterations of EPIc, and y-axes show the Elimination- Factor (ELMF) for each group. From left to right, the first three plots are for DLBD, SA, GM attacks on Waterbirds and the last one is for DLBD on CelebA
Table The impact EPIc on group robustmess, measured by the Worst-Group Accuracy (WGA).
DATASET
ATTACK
CASE
WGA
ASR
ACC
IDEAL
59.0 + 2.5%
0.1 1 0.1%
95.5 0.1%
WATERBIRDS
DLBD
STANDARD
55.8 6.0%%
0.1 + 0.0%
95.0 0.1%
WORST
50.7 - 6.9%
0.1 0.0%
94.3 0.8%
IDEAL
59.1 + 3.0%
-0.3 1.8%
94.6 0.4%
WATERBIRDS
SA
STANDARD
55.3 3.2%
0.2
94.2 0.4%
WORST
48.1
-
93.9 0.3%
IDEAL
50.2 2.0%
13.3 11.5%
95.8 0.3%
WATERBIRDS
GM
STANDARD
43.5 7.8%
13.3 11.5%
94.6 0.5%
WORST
44.3 + 8.3%
13.3 + 11.5%
94.6 0.4%
IDEAL
54.8 + 3.2%
0.3
93.9 + 0.0%
CELEBA
DLBD
STANDARD
40.5 + 4.5%
0.1 + 0.0%
94.0 + 0.1%
WORST
34.8 + 1.1%
0.0 + 0.0%
93.7 + 0.3%
POISONING DEFENSES HAVE DISPARATE IMPACT
After establishing how group robustness methods amplify poisons, in this section, we investigate whether poisoning defenses have any undesirable impact on under-represented samples and group robustness.
5.1 POISONING DEFENSES ELIMINATE MINORITY SAMPLES
We start our investigation by studying the impact of EPIc on under-represented samples. In Figure 2 show the Elimination-Facter (ELMF) of EPIc in four different settings. We observe that generally the poisons and LRG-1 samples are eliminated at similar rate (the two curves are close to each other, across the board). On the other hand, the ELMF on HRG samples is always significantly less. Interestingly, ELMF on LRG-2 samples from Waterbirds is less than the poisons and LRG-1 samples and more than HRG samples. large (reaching almost 100%) ELMF on poisons shows that EPle indeed an effective poisoning defense. However, by eliminating most poisons, EPIc also eliminates most LRG-1 samples as well. This suggests that legitimate under-represented samples are strong outliers from the perspective of EPIc, which demonstrates disparate impact.
5.2 POISONING DEFENSES REDUCE GROUP ROBUSTNESS
Here, study the effect of EPIc on group robustness by considering three scenarios. In ideal and worst-case scenarios, we make interventions on EPIc to never eliminate any under-represented sample eliminate under-tepresented samples early possible, respectively. In standard EPIc, make no intervention and apply the defense as-is. Through interventions, we hope to isolate the impact of EPIc minority samples. For each scenario, report WGA measure group robustness.
8

---

Published as a conference paper at ICLR 2024
We present the results in Table First, we observe that in different datasets and attacks, EPIc reduces the WGA by 3.2% - 14.3%. The most damage happens on CelebA data set as, we believe, it is more complex than Waterbirds. Overall, the ASR is low, indicating that EPIc works properly, with one exception for the GM attack, where the ASR is 13.3% for all three scenarios. In all cases, ACC is high, relatively close to ACC reported in prior work (Liu et al. 2021). We
significant gap in WGA after applying EPIc and applying group robustness methods (in Table 2). almost up to 40%. This is expected as EPIc does not make any attempts preserve group robustness, In Appendix A.5 we make an effort towards applying poisoning defenses while preserving group robustness.
Additionally, we study the impact that robust aggregation mechanisms and poisoning defenses have on the under-represented groups in Federated Learning, as well as more settings including different amounts of poisoned samples. We show the results in Appendix A..4 and observe that they are consistent with our previous findings.
Takeaways. Poisoning defenses either aim to identify and remove the outliers or make it more difficult for the model to learn poisons (e.g., in Federated Learning). They, however, also end up making minority groups more difficult to learn as well, which hurts the group robustness of the trained model. This shows that, despite the common practice, ACC can be over-optimistic in gauging the impact of a poisoning defense in the presence of under-represented groups.
DISCUSSION AND FUTURE WORK
In this work, we have focused only on defenses from the first category in Section 2 (that consider poisons as difficult to learn). We have exposed their vulnerability and the potentially harmful consequences of these defenses. We believe that defenses from the second category (that consider poisons as easy to learn), would not lead to these problems. However, it is important to note that, poisons will not be easy learn in realistic attacks where adversaries can only inject limited number of poisons, violating the assumption. As result, defenses from this category would be ineffective against such attacks and, therefore, group robustness methods would still inadvertently offer a needed boost to the weaker adversary. Finally, for the third category of defenses (poisons are different from clean samples), the state-of-the-ee defenses (Pan et al.) 2023 Qi al. 2023) use small base set (10-1000 samples) to model the distribution of clean samples and identity the training points most distinct from this distribution as poisons. These base sets are often assumed to follow the same distribution as the clean training set, which makes them unlikely to contain sufficiently many minority samples. We hypothesize that this will cause such defenses to still climinate clean minority samples poisons and hurt the WGA. However, it might be possible to avoid this problem by providing such defenses with carefully curated base sets that are balanced (in terms of groups) and free from poisons. In this case, instead of mistakenly penalizing difficult clean samples as poisons, they can isolate the real poisons from a more inclusive clean distribution captured by the base set. However, research suggests that making base poison-free (Zeng et al. (2022) or collecting enough labeled minority samples that capture their distribution properly (Lokhande et al. (2022) might be difficult in practice. We believe addressing these challenges is a promising avenue for future work to reconcile between group robustness and poisoning resilience.
7 CONCLUSIONS
In this work, we demonstrate a significant tension involving two critical metrics studied in the ML community: group robustness and poisoning resilience. The objective of group robustness methods is to amplify minority groups in the training set and create more equitable models. Our findings reveal that the samples injected by poisoning attacks consistently mislead these methods into amplifying them, resulting in an undesirable boost to the adversary. On the other hand, poisoning defenses aim to prevent attacks by removing problematic samples from the training set. However, these defenses remove legitimate under-tepeesented samples as well, hence compromising the model's equity. Finally, we wish to emphasize the pressing need for the ML community to focus the development of new methods that tackle the inherent challenges posed by data poisoning attacks and group robustness methods.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
Panaitescu-Liess Zhu and Huang are supported by National Science Foundation ASF-IIS-214722 FAI, DOD-DOO-OOHIE Office of Naval Research under award number N00014-22-1-2335. DOD-AFOSR- Air Force Office of Scientific Research under award number FA9550-23-1-0044 DOD-DARPA-
Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR001 12020007, Adobe, Capital One and JP Morgan faculty fellowships.
Kaya is supported by the Intelligence Community Postdoctoral Fellowship.
This effort was partially supported by the Intelligence Advanced Research Projects Agency (IARPA) under the contract W91 INF20C0035. The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
REFERENCES
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances neural information processing systems, 32, 2019.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. Advances in Neural Information Processing Systems, 34:10876-10889, 2021.
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9368-9377. 2018.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International Conference Machine Learning, pp. 872-881. PMLR, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-awaa margin loss. Advances in neural information processing systems, 32, 2019.
Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv preprint arXiv: 19.011331, 2019.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safety. CEUR-WS, 2019.
Yizheng Chen, Shiqi Wang, Yue Qin, Xiaojing Liao, Suman Jana, and David Wagner. Learning security classifiers with verified global robustness properties. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 477 494, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. Proceedings of the 35th Annual Computer Security Applications Conference, pp. 1 133-125, 2019.
Jonas Geiping, Liam Fowl, W Ronny Huang, Wojeiech Czaja, Gavin Taylor, Michael Moeller, and Tom Goldstein. Witches' brew: Industrial scale data poisoning via gradient matching. arXiv preprint arXiv:2009.0227 2020.
Tianyu Gu, Brendan Dolan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 2017.
Tatsunori Hashimoto, Megha Srivastava. Hongseok Namkoong, and Perey Liang. Fairesss without demographics in repeated loss minimization. In International Conference on Machine Learning. pp. 1929-1938. PMLR, 2018
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 3104-3122, 2021.
Nayeong Kim, Sehyun Hwang, Sungsoo Ahn, Jaesik Park, and Suha Kwak. Learning debiased classifier with biased committee. Advances in Neural Information Processing Systems, 35:18403- 18415, 2022.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, pp. 1-47, 2022.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Dropout disagreement: A recipe for group robustness with fewer annotations. In NeurlPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications. 2022.
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 36, 2024.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. Advances in Neural Information Processing Systems, 34: 14900-14912, 2021.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Vishnu Suresh Lokhande, Kihyuk Sohn, Jinsung Yoon, Madelcine Udell, Chen- Yu Lee, and Tomas Pfister. Towards group robustness in the presence of partial group labels. arXiv preprint arXiv:2201.0366 20..6666,,, 2022.
Aengus Lynch, Gbëtondji IS Dovonon, Jean Kaddour, and Ricardo Silva. Spawrious: A benchmark for fine control of spurious correlation biases. arXiv preprint 7xXi:22303.05700, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-efficieeel learning of deep networks from decentralized data. In Artificial intelli- gence and statistics, pp. 1282. PMLR, 2017.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-20684, 2020.
Hongseok Namkoong and John C Duchi. Variance- regularization with convex objectives. Advances in neural information processing systems, 30, 2017.
Minzhou Pan, Yi Zeng, Lingjuan Lyu, Xue Lin, and Ruoxi Jia. Asset: Robust backdoor data detection
across multiplicity of deep learning paradigms. arXiv preprint a0Xiv:2302. 1140, 2023.
Ashwinee Panda, Saeed Mahloujifar, Arjun Nitin Bhagoji. Supriyo Chakraborty, and Prateek Mittal. Sparsefed: Mitigating model poisoning attacks in federated learning with sparsification. In International Conference on Artificial Intelligence and Statisties, pp. 7587-7624. PMLR, 2022.
Xiangyu Qi, Tinghao Xie, Jiachen T. Wang. Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards proactive ml approach for detecting backdoor poison samples, 2023.
Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. Simple and fast group robustness by automatic feature reweighting. In International Conference on Machine Learning. pp. 28448- 28467. PMLR, 2023.
11

---

Published as a conference paper at ICLR 2024
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXhv:1911.08731, 2019.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11957- 1196, 2020.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. Advances in neural information processing systems, 31, 2018.
Shawn Shan, Arjun Nitin Bhagoji, Haitao Zheng. and Ben Y Zhao. Poison forensics: Traceback of data poisoning attacks in neural networks. In s1 USENIX Security Symposium (USENIX Security 22), pp. 3575-3592, 2022.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-graing grarse- classification problems. Advances in Neural Information Processing Systems, 33:19939-19352 2020.
Liwei Song, Reza Shokri, and Prateek Mittal. Membership inference attacks against adversarially robust deep learning models. In 2019 IEEE Security and Privacy Workshops (SPW), pp. 50-56. IEEE, 2019.
Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certified defenses for data poisoning attacks. Advances in neural information processing systems, 30, 2017.
Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning FAIL? Generalized transferability for evasion and poisoning attacks. In 27th USENIX Security Symposium (USENIX Security 18), pp. 1299-13111 1316, Baltimore, MD, August 2018. USENIX Association. ISBN 978-1 1-931133-----
Rachael Tatman. Gender and dialect bias in youtube's automatic captions. In Proceedings of the first ACL workshop on ethics in natural language processing, pp. 53-59, 2017.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers). pp. 1112-1122. Association for Computational Linguisties, 2018. URL httt:://ccwwbbb org/ anthology/NiL 1101
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? arXiv preprint arXiv:2012.03107, 2020.
Yu Yang, Tian Yu Liu, and Baharan Mirzasoleiman, Not all poisons are created equal: Robust training against data poisoning. In International Conference on Machine Learning, pp. 25154-25165. PMLR, 2022.
Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantinerrobust distributed learning: Towards optimal statistical rates. International Conference on Machine Learning, pp- 5650-5659. PMLR, 2018.
Yi Zeng, Minzhou Pan, Himanshu Jahagirdar, Ming Jin, Lingjuan Lyu, and Ruoxi Jia. How to sift out a clean data subset in the presence of data poisoning? arXiv preprint arXiv:2210.065PP 2022.
Michael Zhang, Nimit S Sohoni, Hongyang R Zhang, Chelsea Finn, and Christopher Ré. Correct-n- contrast: A contrastive approach for improving robustness to spurious correlations. arXiv preprint arXiv:2003.01517, 2022.
12

---

Published as a conference paper at ICLR 2024
Table 4: Proportions of each group in Waterbirds dataset. Labels O and 1 correspond the classes landbirds and waterbirds, respectively. Attributes and correspond to the spurious features land and water, respectively.
GROUP
GR. 0
GR.
GR.
GR. 3
PoISONS
LABEL/ ATTRIBUTE
0/0
0/1
1/0
1/1
1/0
PROPORTION
72.0%
3.8%
1.2%
22.0%
1.0%
A SUPPLEMENTARY MATERIAL
A.I ADDITIONAL DETAILS ON THE EXPERIMENTAL SETUP
For the models trained with JTT on Waterbirds, unless differently specified, we use SGD with momentum with a factor of 0.9 as the optimizer, batch size of 128, learning rate of Le 5 and weight decay of 1.0. We stop the identification model after 60 epochs, use an upsampling factor of 100, train the final model up to 100 epochs and choose the model that has the best WGA on the validation dataset, then report the results on the test dataset. In case of CelebA, we use the same as for Waterbirds, with a few exceptions: weight decay of 0.1, early stopping for the identification model after 1 epoch and we use an upsampling factor of 50. Also, we train the final model up to epochs and consider the same procedure to choose the best model as above. We build experiments top of the official code from
For the models trained with GEORGE, we use a similar optimizer, batch size, learning rate, weight decay as in the Waterbirds case from above. For clustering we let the model find the optimal number of clusters (up to 10) for each class based on Silhouette criterion as in (Sohoni et al. (2020). We train the final model up to 300 epochs and use the official code from P
For DLBD, the trigger is a 25x25 white square placed pixel away from the bottom-right corner of the poisoned samples.
For SA, we consider FeatureMatch and Label Flipping to create poisoned samples as in prior work (Jagielski et al. 2021).
For GM, we use the multi-target version of the attack, an epsilon of 16 and 8 restarts. To build the poisoned samples, we rely on the official code from 3]
For EPIc, in case of Waterbirds, we consider SGD with momentum with a factor of 0.9 as the optimizer, batch size of 128, learning rate of 1e 2 and weight decay of 1e 4. We train the models up to 40 epochs and consider the same selection criterion for the model based on WGA on the validation set as above. In the case of CelebA, we change the learning rate to le 3 and we train the models up to 10 epochs. We build our experiments on top of the official code from
For the federated learning experiments, we built the implementation on top of the code from here We consider total of 100 users and 10% of them are chosen at each round. We use SGD with momentum with a factor of 0.9 as the optimizer, a local batch size of 128 and learning rate le Also, we train each local model for 10 epochs at each round. In case of the non-IlI setting, we consider that only 10% of the users have samples from the under-represented groups
For the federated learning defenses, in the case of Trimmed Mean, remove the lowest and highest two values for each coordinate in the update and in the case of SparseFed we use a value of 400, 000 for the number of parameters that keep at each step which is equivalent to keeping less than 5%
of the parameters. We consider a momentum factor of p = 0.9 as in prior work (Panda et al. 2022) or not using momentum (p- 0) in our experiments.
httpss//jttlbbboooomoooscccnnnno wraprisnnniinngaadnnnnnnnaaaaaia mendiiintt-acching
"hatps://gttuub.com/ http://iiiuuucoooom/umuuaaaee wwmg.///.//ommmo//o/////C/C
PoToch
13

---

Published as a conference paper at ICLR 2024
Table 5: Proportions of each group in 10% of CelehA and Full CelebA. Labels 0 and correspond to the classes non-blond and blond, respectively, Attributes O and 1 correspond to the spurious features female and male, respectively.
GROUP
GR. 0
GR. 1
GR. 2
GR. 3
POISONS
LABEL/ / ATTRIBUTE
0/0
0/1
1/0
1/1
1/1
PROPORTION FOR 10% CELERA
44.7%
39.6%
13.8%
0.9%
1.0%
PROPORTION FOR FULL CELEBA
44.0%
40.1 %
14.1% %
0.9%
1.0%
In Tables 4 andd 5 we show the proportions of each group in Waterbirds and CelebA datasets,
respectively.
A.2 DISCUSSION ON HEURISTICS USED BY GROUP ROBUSTNESS METHODS
Although we experimented with methods that specifically loss-based heuristic, we believe that many other group robustness methods rely on similar hcuristics, perhaps less directly. this section, discuss four methods from recent literature that implemented seemingly different heuristics that will align with loss-based heuristics.
Zhang et al. (2022) applies contrastive learning push together the representations of training samples that are labeled into the same class but predicted differently. Since, the low-budget poisoning attacks we consider often generate hard-to-learn, misclassified samples, we believe this method will also suffer from the tension we identify, c.g., misclassified poison samples of class will be represented similarly to correctly classified samples of class, which will amplify their effectiveness the model becomes less likely to misclassify them.
LaBonte et (022) uses the disagreements between different forward passes of the model with dropout to find which training samples to amplify. Different dropout forward passes are known to disagree on samples where the model has high uncertainty (Gal & Ghahraman] (0166), which are often hard-to-learn, higher-loss samples. As result, we believe this method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples we generated will create dropout disagreements and will be amplified.
Kim et al. (0022) relies on disagreements among the members of an ensemble of models to identify the "biased" samples (i.e., the samples that contain spurious correlations). If sample has low ensemble agreement, it will be amplified by this method. Ensemble disagreement is a known uncertainty metric (Beluch al. 2018) in the literature. As a result, we believe that this method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples will be misclassified by more members of the ensemble, and will be amplified.
LaBonte et al. (2024) constructs a reweighting set hased on either (1) ERM model's misclassifications, or (2) the misclassifications of an early-stopping model, or (3) dropout disagreement (similar the Dropout Disagreement paper) or (4) the disagreements between the ERM and early-stopping models. All these heuristics identify minority group samples will end up identifying the hard-to-learn poisoning samples as well. The authors identified (4) as the most promising heuristic. It is known that hard-to-classify samples are learned at later iterations of model training (Baldock et al. (02T). As result, we believe that method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples will be classified differently by the early-stopping and regular models, and will be amplified.
These common threads show that, despite not using the loss heuristic directly, many methods in this line of work rely on related ideas that will amplify uncertain, misclassified, high-loss, or hard-to-learn samples. Moreover, is known that many example difficulty metrics are highly correlated with one another (Baldock et al. 2021 Carlini et al. 2019). Ultimately, believe that these related heuristics cannot avoid the tension we identified because Tow- bowgetdget poisoning attacks generate hard-to-learn samples.
14

---

Published as a conference paper at ICLR 2024
A.3 MORE RESULTS ON LIMITATIONS OF THE GROUP ROBUSTNESS METHODS
Impact of the
Here, we study how the hyper-parameters of JTT impact our findings from Section 2 We consider two
early stopping for the model in the first phase used to identify LRG; and the upsampling factor for the samples identified samples. Our previous experiments use the original hyper-parameters, i.e., early stopping is set to 60 epochs, and upsampling factor is set to 100x. However, due to the introduction of poisoning, these hyper- parameters might not be optimal anymore. To this end, we search for new hyper-parameeers in a grid (early stopping E (40, 80, 100, 200) and upsampling factors E {20, 50, 150) against DLBD attack on Waterbirds.
We present the results in Tables and 7 We observe that the results are consistent with our previous findings: ASRs are very close between the standard and the worst cases (i.e., high amplification), while both WGA and ACC are still relatively high with the exceptions when early stopping is set to 40 and when the upsampling factor is set to 150. In these cases, the WGAs are only 65.3% and 64.6%, respectively. For the former, we believe that it could be due how earlier stopping makes identification less selective (lower precision) and ends up upsampling HRG as well. Also, the ASR is higher by more than 15% compared the other early stopping values. For the latter, we believe that it is because the amplified samples appear too often during training compared to the non-amplified ones. Also, surprisingly, using smaller upsampling factors of 50x or 20x instead of 100x makes the attack even more powerful by 8.4% and 46.2% respectively, in terms of ASR.
Additional Settings. For completeness. in this section, we analyze several other scenarios on Waterbirds data set, including using different poisons percentages, different numbers of targets for the GM attack, different model architectures and different base samples. In Table g we consider JTT and GEORGE with the DLBD attack using several poison percentages in the range 0.4% 2%. The results are consistent with our previous findings. Additionally, we observe that an attacker who could introduce 2% poisoned samples, generally obtains higher ASR (of 52.5% 59.8%). In Table 9 we compare the results in two settings for the GM attack: (i) when the attack has 5 targeted samples (same as our previous experiments) and (ii) when the attack has 100 targeted samples. The results show that, in the case of 100 targets, the amplification is as high as it could be. The only distinction from the case with 5 targets is overall lower ASR, but this is expected the attack becomes more difficult as it targets more samples. We also consider a larger architecture, ResNet-50, instead of ResNet- 18, for the DLBD attack and several scenarios, including training the models from scratch. Our results in Table 10 are generally consistent, however, the ASR is significantly higher against ResNet-50. We believe the additional learning capacity of ResNet- over ResNet-18 facilitates the attack. Also, we observe that training the models from scratch and tuning them for high group robustness (WGA) damages the overall accuracy (ACC) significantly. This challenge explains the prior practice of using pre-trained models as they can alleviate this trade-off to some extent. In Table IT] we consider base samples from different groups of CelebA for DLBD. We observe that the casier a group is to learn (i.e., less average loss), the higher the ASR will be if the attacker crafts a DLBD attack with base samples from that group. In Table 12 we consider the scenario when the attacker is not aware of the presence of groups, observe high amplification (i.e., the small gap between the worst and standard cases) no matter which class is targeted by the attacker. As result, even a less informed attacker, who is unaware of the groups, would benefit from the amplification. In Table 13 we consider the Subpopulation Attack (SA) on CelebA when applying JTT and without applying any group robustness method (ERM) as a baseline. We observe that the amplification is generally high (large ASR gap between JTT and the baseline) while JTT still works as intended as it improves the WGA over the baseline. Additionally, in Table [4] we consider AFR (Qiu et al. 2023) on MultiNL dataset (Williams et al.) 2018) and observe similar results. We also consider several datasets from Spawrious benchmark (Lynch et al., 2023). We show the percentage of samples amplified by JTT from each group in Table 15 Again, the results are consistent with our previous findings. To conclude, our extensive experiments show that the limitation of group robustness methods is consistent across different settings, suggesting that this might be an inherent vulnerability.
they just attack one of the two classes of CelebA, with the DLBD. We
A.4 MORE RESULTS ON THE LIMITATIONS OF THE POISONING DEFENSES
Federated Learning Scenarios. To demonstrate that the limitation of EPIc we identified (in Section 5 is consistent in other defenses, we also run experiments on federated learning, where
15

---

Published as a conference paper at ICLR 2024
Table 6: Evaluating the impact of amplification in JTT on worst group accuracy (WGA), attack success rate (ASR) and test accuracy (ACC) when considering several early stopping epochs for the identification model.
EARLY STOPPING
CASE
WGA
ASR
ACC
WORST
65.4 + 9.4%
46.3 + 30.6%
83.3 + 4.4%
40
STANDARD
65.3 + 9.3%
45.7 31.5%
83.2 + 4.3%
IDEAL
76.8 1 3.2%
0.4 0.1%
91.5 0.5%
WORST
76.9 + 3.5%
20.9 7.9%
+6.4+1.2%%
60
STANDARD
78.0 + 4.1%
20.4 + 5.9%
86.7 + 1.2%
IDEAL
$1.400.90 0.9%
0.5 0.3%
91.0 1: 0.4%
WORST
80.3 2.3%
18.7
87.3 0.8%
STANDARD
80.9 2.8%
17.7 5.5%
87.6 + 1.0%
IDEAL
82.2 1: 2.4%
0.5 1 0.1%
91.3 + 0.5%
WORST
2.0%
19.7 8.9%
87.6 1.2%
100
STANDARD
82. 2.1%
23.4 8.7%%
87.2 0.9%
IDEAL
83.3 1.3+0.00 0.9%
0.9 0.1%
91.0 + 0.4%
WORST
83.5 f 2.4%
2
87.9 + 1.7%
200
STANDARD
29.5 8.5%
87.5 1.3%
IDEAL
+ 1.5%
0.7 + 0.1%
90.0 1.7%
Table 7: Evaluating the impact of amplification in JTT on worst group accuracy (WGA), attack success rate (ASR) and accuracy (ACC) when considering several upsampling factors.
UPSAMPLING FACTOR
CASE
WGA
ASR
ACC
WORST
71.9 4.0%
69.5 + 9.1%
91.4 0.6%
20
STANDARD
72.3 4.5%
66.6 + 10.0%
91.5 0.6%
IDEAL
74.9 2.3%
1.3 0.3%
93.0 0.3%
WORST
79.7 1.8%
+ +.5%
0.5%
50
STANDARD
79.6 1.9%
28.8 6.6%
9.9
IDEAL
1.0%%
0.9%0..4%
02.610.499
WORST
3.5%
+ +.9%
8..4
100
STANDARD
78,0 4.1%
2
86.7
IDEAL
81.4 0.9%
0.5 0.3%
91. 0.4%
WORST
65.7+10.3 10.9%
29.2 + 29.5%
74.1 +.7%%
150
STANDARD
64.6 + 9.9%
29.4 + 34.4%
73.2 + 3.8%
IDEAL
73.0 1 13.2%
0.7 1 0.1%
84.3 1 7.1%
16

---

Published as a conference paper at ICLR 2024
Table 8: Evaluating the impact of amplification in group robustness methods on worst group accu- racy (WGA), attack success rate (ASR) and test accuracy (ACC) when considering several poison percentages for the DLBD attack.
POISONS
METHOD
CASE
WGA
ASR
ACC
WORST
78.7 + 2.0%
4.8 + 0.7%
89.3 + 1.2%
0.4%
JTT
STANDARD
78,6 + 2.0%
3.5 1 1.7%
89.1 = 1.0%
IDEAL
81.1 1 1.9%
0.3 1 0.1%
91.4 0.3%
WORST
80.6 2.7%
8.7
89.4
0.6%
JTT
STANDARD
79.3 1.3++..7% 1.7%
9.6 + 1.8%
89.0 + 0.8%
IDEAL
81.2
0.5 + 0.1%
91.6 + 0.5%
WORST
78.5+3.449
15.8 4.4%
87.8 1.2%
0.8%
JTT
STANDARD
78.0 3.0%
16.8 5.4%
87.7 1.2%
IDEAL
8122 1.5%
0.5 1 0.1%
91.0 + 0.6%
WORST
76.9 + 3.5%
20.9 7.9%
86.4 1.2%
1%
JTT
STANDARD
+
20.4 5.9%
86.7 1.2%
IDEAL
81.4 0.9%
0.5 + 0.3%
91.0 - 0.4%
WORST
64.7 : 7.4%
56.6 + 41.9%
69.8 5.6%
2%
JTT
STANDARD
62.0 22.0+9.800 9.8%
59.8 + 45.9%
66.0 9.6%
IDEAL
82.5 1.6%
0.6 0.3%
91.1 0.2%
WORST
74.1 1.7%
16.5 2.9%
76.4 3.4%
1%
GEORGE
STANDARD
77.3 + 2.5%
15.8 + 3.9%
79.4 1 1.6%
IDEAL
79.3 + 2.11%
0.4 + 0.0%
93.1 I 1.4%
WORST
74.3 5.4%
13.6%
78.0 9.0%
2%
GEORGE
STANDARD
74.5 + 5.2%
52.5 + 7.1%
77.6 8.5%
IDEAL
80.2 1 0.4%
0.7 + 0.1%
92.4 0.9%
Table 9; Evaluating the impact of amplification in JTT worst group accuracy (WGA), attack success rate (ASR) and test accuracy (ACC) when considering different targets for the GM attack.
TARGETS
CASE
WGA
ASR
ACC
WORST
76.1
3.2%
20.0 0.0%
89.9 1.0%
STANDARD
76.1 + 3.2%
20.0 + 0.0%
89.9 + 1.0%
IDEAL
75.9 + 0.8%
13.3 + 11.5%
91.3 + 0.1%
I00
WORST
75.4 + 2.3%
8.6 + 0.5%
88.6 + 0.9%
STANDARD
75.8 2.2%
8.6 0.5%
88.8 1.0%
100
IDEAL
75.6 1 0.5%
5.6 + 0.5%
91.1 + 0.6%
Table 10: Evaluating the impact of amplification in JTT on worst group accuracy (WGA), attack success rate (ASR) and test accuracy (ACC) when considering different architecture (ResNet-50). Note that ES denotes early stopping for the identification model.
SETTING
ES
CASE
WGA
ASR
ACC
WORST
72.4 2.1%
70.6 13.8%
76.2 1.3%
PRE-TRAINED
60
STANDARD
72.4 2.1%
70.6 + 13.8%
76.2 1.3% 1.3%
IDEAL
85.1+ 1.4%
0.9 = 0.3%
00.2+1.0%%
WORST
8.14
58.0 23.7%
86.5 2.5%
PRE-TRAINED
200
STANDARD
83.5 1.9%
86.7 2.1%
IDEAL
84.7 1.5%
1.5%
0.5 0.5+0.55 0.5%
90.8 2.6%
WORST
39.5 7.8%
5.77
59.0
FROM SCRATCH
200
STANDARD
41.0 8.5%
34.5 26.4%
64.0 8.3%
IDEAL
37.8 1 6.9%
5.8 4.6%
61.0 = 12.9%
17

---

Published as a conference paper at ICLR 2024
Table 11: Evaluating the impact of the poison base samples' choice for JTT and CelebA
BASE SAMPLES' GROUP
GR. (HRG)
GR. (HRG)
GR. 2 (HRG)
GR. (LRG-1)
LABEL/ ATTRIBUTE
0/0
0/1
1/0
1/1
ASR
97.4 0.1
97.7 1 0.1
91.2 + 2.3
0.1 1 0.1
Loss AvG, (NO POISONS)
0.16
0.07
0.79
1.90
Table 12: Evaluating the scenario when the attacker does not have any knowledge about the groups. consider JTT and CelebA
BASE SAMPLES IN CLASS 0
BASE SAMPLES IN CLASS
ASR (WORST) ASR (STANDARD)
98.3 0.1
97.6 + 0.3
90.7 + 1.7
97.3 + 0.5
ASR (IDEAL)
0.1 = 0.0
0.5 0.1
Table 13: Evaluating the effect of JTT on CelobA with a different baseline (ERM) when varying the poison percentage. We consider the Subpopulation Attack (SA).
POISON %
0.05%
0.1%
0.2%
0.3%
0.5%
1%
2%
JTT (ASR)
0.1%
1.7%
2.4%
3.5%
6.3%
14.1%
46.8%
ERM (ASR)
0.1%
0.1%
0.2%
0.0%
0.1%
0.1%
0.3%
JTT (WGA)
81.7%
82.2%
83.3%
83.3%
83.3%
77.9%
48.2%
ERM (WGA)
43.3%
43.3%
47.2%
42.8%
41.1%
45.0%
45.6%
Table 14: Evaluating the effect of AFR CelebA with ERM baseline. We consider the Subpopulation Attack (SA).
POISON %
0.5%
1%
(ASR)
2.1%
14.4%
ERM (ASR)
1.0%
5.3%
AFR (WGA)
75.3%
78.4%
ERM (WGA)
68.0%
70.0%
Table 15: The percentage of samples amplified by JTT from each group in Spawrious benchmark.
DATASET
GR.0
GR.1
GR.2
GR.3
GR.4
GR.5
GR.6
GR.7
POISONS
M2M-EASY
4.9%
4.1%
2.5%
3.1%
4.8%
6.6%
2.1%
3.5%
99.6%
M2M-MEDIUM
4.0%
4,4%
2.6%
4.1%
10.0%
9.9%
2.7%
2.5%
97.4%
12M-HARD
4.1%
2.7%
4.0%
4.3%
3.4%
4.3%
4.6%
3.9%
100%
18

---

Published as a conference paper at ICLR 2024
Table 16: The impact of poisoning defenses in federated learning on group robustness.
METHOD
IID?
WGA DROP
ACC DROP
MEDIAN
YES
23.7 + 7.3%
-0.2 0.3%
TRIMMED MEAN
YES
45.7 1 8.8%
10.6 + 3.5%
SPARSEFED (p
YES
45.4 27.7%
11.0 17.7%
SPARSEFED (p 0.9)
YES
42.7 24.3%
19.0 14.6%
SPARSEFED (p = 0.9)
No
55.1 + 21.6%
12.3 + 16.3%
Table 17: The impact of EPIc on group robustness, when considering the DLBD attack and several poison percentages.
POISONS
CASE
WGA
ASR
ACC
IDEAL
61.2 + 2.1%
0.1 + 0.0%
95.4 + 0.5%
0.5%
STANDARD
57.6 + 4.9%
0.1 + 0.0%
95.0 0.6%
WORST
52.0 + 8.1%
0.1 + 0.0%
94.4+0.5%
IDEAL
19.0
0.1 0.1%
9.15
1%
STANDARD
55.8 1 6.0%
0.1 0.0%
95.0 0.1%
WORST
50.7 + 6.9%
0.1+0.0%
94.3 + 0.8%
IDEAL
57.3 1 2.5%
0.3 0.3%
94.5 14.50.000 0.6%
2%
STANDARD
57.0 1 0.4%
0.5 0.4%
94.5 1 0.4%
WORST
48.6 1 7.6%
0.2 1: 0.0%
93.8 1 0.1%
robust aggregation mechanisms are widely studied. In this scenario, the defense does not sanitize the training set but sanitizes the updates sent by each client to prevent poisoning. We consider FedAvg (McMahan et 2017) an un-defended baseline and study the drop WGA and ACC relative to it. We included more details about the experimental setup in Appendix A. In Table I6] we observe that the defenses we consider cause significantly more drop WGA than ACC, over the baseline. This exposes that all these methods, while attempting to fight against poisoning, end up having a disparate impact on the model's accuracy on under-represented groups.
More Settings. In Table 17 we study the impact of EPIc when there are 0.5% or 2% poisons for the DLBD attack, instead of 1%. We observe that EPIc drops the WGA by 0.3% 3.6%, while maintaining low ASR and high ACC. Also, aligning with our previous results, we observe that the overall WGA is low compared to the values obtained when considering a group robustness method. In Table 18 we run additional experiments for EPIc on CelebA with 20%, 50% and 100% of the dataset considered. The persistent gap between the Ideal and Standard cases demonstrates the negative impact of EPIc on group robustness. Overall, these results are consistent with our main claims.
Run-time defenses. Additionally, we run experiments using STRIP (ao et al. 2019). a run-time backdoor detection mechanism. The main assumption of this method is that backdoored model's outputs have lower entropy on perturbed triggered samples compared to perturbed clean samples. In Figure we show the percentage of samples from each group which detected by the defense when varying the entropy threshold (we use the first model from Table A in the standard case, run the experiment 3 times and provide the mean of the results). Basically, the result from Figure shows that there is no entropy threshold for which STRIP can detect the triggered samples without inadvertently detecting the clean samples, too. We believe that the strong regularization needed for the models to achieve group robustness (Liuu et al. [202]: Saaawa et al. (019)) contributes to the limitation of such defenses the model does not have very confident outputs, leading to generally high entropy values for all types of samples.
A.5 COMBINING GROUP ROBUSTNESS METHODS AND POISONING DEFENSES
In this section, we study the feasibility of achieving both high group robustness and poisoning resilience by combining the current state-of-the-art methods.
19

---

Published as a conference paper at ICLR 2024
Table 18: The impact (in terms of WGA) of EPlc group robustness, when considering the DLBD attack and several subset sizes of CelebA.
20% OF CELEBA
50% OF CELEBA
FULL CELEBA
IDEAL
52.3 1.6
50.3 + 1.9
45.9 5.8
STANDARD
43.1 + 5.5
46.4 + 2.8
43.6 + 3.0
WORST
32.3 + 2.7
32.0 : 0.8
37.7 + 3.8
1.0
LRG-2
LRG-1
Triggered Samples
0.8
of samples detected
0.2
0.0
0.900
0.950
Entropy Threshold
Figure 3: The percentage samples from each group detected by STRIP when varying the threshold.
We first apply EPIc to identify potential poisons the training set. Then, when we apply JTT, we intervene so that it does not amplify the potential poisons found in its first phase (i.c., remove them from JTT's upsampling set). We have considered the following two baselines in our pipeline: ideal EPIc that identifies only the poisons and not using EPIc that would want improve upon. Because EPIc removes samples iteratively, we considered three stopping epochs for the removal process, that have control over how many samples EPIc identifies poisons. As shown in Figure 2 stopping EPIc sooner leads to a lower percentage of samples from LRG that are removed, but also lower amount of poisons. Whereas, stopping EPIc later increases both of these rates.
For the rest of this section, we consider Waterbirds dataset and DLBD attack. More details on the experimental setup. including the hyper-parameters used are in Appendix A.1
First of all, as shown in Table 19 not using EPIc at all results in a WGA relatively close to the WGA that could be obtained if none of the poisons were amplified (Ideal EPIc). However, the ASR still high if we do not use EPIc to identify potential poisons. We evaluate three possible stopping epochs for EPlc and measure the effects WGA and ASR as function EPIC's stopping epoch. With higher stopping epoch (i.e., more samples are identified as poisons), the ASR decreases, however, the WGA also decreases. For example, to mitigate the attack and obtain below 1% ASR, we need to sacrifice over 35% WGA-signiffe
damage to group robustness. Also, in ideal EPIc (only poisons are climinated), we could obtain both high WGA (over 80%) and low ASR (lower than 1%). Moreover, note that for all the models, the ACC stays relatively high (over 80%), though, lower than the settings without any poisoning (e.g., 93.3% in Liu et al. (2021)). This further shows how ACC can be misleading to judge the side effects of a poisoning defense.
In conclusion, we have attempted to combine EPIc and JTT, in hopes achieving both high poisoning resilience and group robustness, but this task is not trivial. Both legitimate under-represented
20

---

Published as a conference paper at ICLR 2024
Table 19: Applying EPlc and JTT together combine poison resilience with group robustness.
EPIC
WGA
ASR 86.7 + 1.2%
ACC
No
78.0 + 4.1%
20.4 = 5.9%
STOP =3
76.5 3.8%
14.8 = 4.0%
91.9 1 0.3% 0.3%
STOP
73.4 + 7.4%
6.7 1.7%
82.8 + 7.3%
STOP =7
42.3 6.6%
0.9 + 0.6%
93.4 0.2%
IDEAL 81.4 0.9%
0.5 + 0.3%
91.0 + 0.4%
samples and poison samples in realistic attacks can be difficult-to-learn and without making specific assumptions, (e.g., poisons contain detectable artifacts), it might be difficult to distinguish them. Using EPIc (which makes no such assumptions) to identify potential poisons and use that information as intervention into JTT is not enough to mitigate the trade-off between WGA and ASR.
A.6 PROOFS
Lemma (Restated) For the setting described above, if we assume that there are no ties in maximum expected class probability among groups, then the identification model has less expected class probability on the poisons (Ym: am) in comparison to any legitimate group.
Proof. For any y, a E {0, 1}, we know Py > Pyg.a SO
um am < 1 - py.a
(1)
Also, for any y, a E (0, 1}, from the definition of , we have:
Py.a pg.
(2)
(B(saa)) [[xx) E (0,1- ly ay v.)) (xx)1_y]= ((.x,)) (o + [xx Ey- [{x)) [xx) Ex ((...).y.)))
[1] [1]= = =).
Hence, by substituting in we obtain:
Pi Amm
Py.a-
(3)
Therefore, the identification model has the least expected class probability on the poison samples
(Um, am) )=)*.
Theorem 4.2 (Restated) We consider the same setting as in Lemma 4.1 We denote the poisons
(ym, am)) by 9p and let (u,a) a) 9r. be any group of samples (e.g., legitimate minority group). Also, Ym am) by Gp and I(x) for (x, y, a) (Dac y, a) by Ge and the cross-entropy loss on G E [Go, by L(G). We assume Gp and Ge Var(G.) == o² (i.e., the variances of the class probability for the identification model are equal for the legitimate group and for the poisons) and denote
we denote T(x) 1-m for (ar, Ym: 0m)) (D
are independent and Var(Gp)
(BGp)) Up and BB(G)) pe- Then, for any (0, of & V + 1 P(L(G.) L(G,)) 1-
He By have
Proof. Let 6 E (0, 1). We observe that ( I(Gp)) =P(- (loooGG < -log
P(Gp < Go) > PPG < BetBe < Go) = P(G < bet ^ G. > "=="e)) = P(Gp <
2
HotHH) F(G. MotHe) P(G Up Ho PC.G. Me Po (e- (1 PPG 14p 2
He "e-Hee He (1 P(G. - E He [e-He)) ))
We know from Lemma
4.1 that Mp < Pc. (i.e., Mo > 0), so we can apply Cantelli's inequality to a²
obtain:
(1 PPG Mp (e="z)) (1 P(G. Me & -- "===12))
(1
x² pp)
(Me
n (
P (L(Gc) < I(Gp))
02 P
024(
02
- r² p0 Po-pp
21

---

Published as a conference paper at ICLR 2024
Since we know a &
V रí-
He we can conclude that P([((G) < I(G,))
2
2
(" ( y2 (
=1-6. c. (
22


Reviews:
[{"review_id": "4ziEdBbsZo", "reviewer": "Reviewer_wCf5", "summary": "This paper studies the existing tension between the group robustness and the resilience to poisoning attacks. The authors argued that, group robustness methods will unavoidably amplify the poisoning samples and boost poisoning performance. On the other hand, poisoning defenses will remove poisoning outliers will also unavoidably remove the minority samples. The authors advocate for tacking this inherent tension in future works.", "questions": "1. In page 4, what is the intuition behind configuring the poisoning attacks as described? The configuration seems to be different from the one in the original paper and the authors should clearly explain the reason. \n2. Will the gradient shaping method [1] enable a mitigation to the currently observed tension? \n\n[1] Hong et al., \"On the Effectiveness of Mitigating Data Poisoning Attacks with Gradient Shaping\", arXiv 2020.\n\n==========\n\nThanks for your detailed response. The rebuttal addresses most of my concerns, I have increased my rating.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The problem of the inherent tension between group robustness and poisoning resilience is an interesting problem/\n2. The presented empirical results are well-presented to support the main claim in the paper.", "weaknesses": "The main weakness of this paper is to make an impossibility type of claim without a more principled understanding on the problem.\nIn particular, the authors aim to find a tension between any poisoning attack and the distributionally robust optimization methods, which probably undermines the much-needed principled understanding. In particular, I am expecting some type of result that points out the tension between a poisoning with a clearly defined attacker capability and objective, and the group robustness optimization techniques (e.g., using loss-based thresholding). This result should show that, in order to achieve the attacker objective maximally, the poisoning points will also unavoidably become the amplified minority samples. This way, we can make claims on the impossibility of having group robustness and poisoning resilience at the same time. My major concern here is, the tensions between poisoning and group robustness are drawn some limited empirical results and the poisoning attacks also do not reflect what the attacker in practice may really care to do. For example, many ASR are quite low as presented in Table 2 and I am not sure if it is because the poisoning attacks are configured in some undesirable ways (with respect to the attack objective), who may choose a different approach to achieve higher ASR. The alternative, in turn, may no exhibit a strong tension. Related to this, the impossibility result in the paper only shows that there exist some poisoning points that are harder to learn but did not reason whether these poisoned points are useful for achieving the attacker objectives. I would suggest the authors to first clearly define the threat model and then conduct more rigorous analysis on how this can interfere with group robustness methods. Then, the insights are supported through extensive empirical evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tc2796JzhN", "reviewer": "Reviewer_a2i2", "summary": "This work identifies a critical conundrum between group robustness and poisoning resilience metrics in machine learning. The contributions are 3-fold:\n\n1) It presents empirical evidence illustrating that methods designed for group robustness are unable to differentiate between minority groups and poisoning data. This inability exacerbates the impact of poisoning attacks, a claim further substantiated by theoretical support under various assumptions.\n\n2) Further, it demonstrates through empirical studies that standard defenses against data poisoning struggle to distinguish poisoned data from that of minority groups. This leads to a compromise in group robustness, an issue observed in both centralized and federated learning setups.\n\n3) Finally, this paper concludes that merely combining group robustness strategies with poisoning defense mechanisms fails to address these challenges, indicating a need for more nuanced solutions.", "questions": "Please see Weaknesses section above for a list of all questions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) This paper is well written and it is easy to follow. The takeaways from each section are explicit, novel and clear.\n\n2) Figure 1 clearly conveys the key ideas.\n\n3) The findings are interesting and useful to the community.", "weaknesses": "1) Experiments. The results on CelebA (Section 4) only uses a randomly sampled 10% of the dataset which is concerning. Can the authors also consider the full CelebA setup, so that the results are more reliable?\n\n2) Why are the ASR values negative in Table 3 (Waterbirds / SA setup).\n\n3) Could the authors elaborate on whether any experiments were conducted involving the other two types of poisoning defenses mentioned in Section 2.1?\n\n\n\nMinor concerns\n\n1) Title Appropriateness: The broad focus implied by the current title, \"Poisoning Defenses,\" does not accurately reflect the specific emphasis of the work on poisons that are challenging to learn. A more precise title would set clearer expectations for readers.\n\n\n\nOverall I enjoyed reading this paper. In my opinion, the strengths of this paper outweigh the weaknesses. Authors please consider addressing the weaknesses above during rebuttal.\n\n\n\n\n======================\n\n**Post-rebuttal**\n\nThank you authors for your detailed response. The rebuttal addresses most of my concerns, and I maintain my initial rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CQdmDAxAaC", "reviewer": "Reviewer_Shsu", "summary": "The authors consider the interplay between a certain type of group robustness methods and poisoning attacks. They show that methods to achieve high accuracy on minority groups in the absence of (validation) group annotations also often flag poisoned examples, and thus amplify poisoning attacks. Additionally, they show that poisoning defenses often flag minority examples as poisons, thus removing them from the dataset. Finally, the authors complement their experimental findings with an formal impossibility result in a toy setting.", "questions": "### Is ASR actually amplified by group robustness methods\n\nI am confused the statement \"The large gap between (6.7% ´ 97.4%) the standard and ideal cases shows an opportunity for better heuristics\" (referring to Table 2). It is \"not the job\" of the group robustness method to remove the poisoning attack. Thus, a more fair comparison in my opinion would be to compare \"standard\" ASR against the baseline of not applying the group robustness method. What is the gap then?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The authors present an interesting connection/trade-off between two fields within ML. The problem of exploring trade-offs between competing objective is important.", "weaknesses": "### Overclaiming / limited scope of conclusions\nI believe that the statement\n\"We show experimentally that group robustness methods fail to distinguish minority groups from poisons.\"\nis misleading. In particular, the authors only consider group robustness methods which use a loss-based heuristic in lieu of *validation* group annotations. This significantly reduces the scope of authors' contributions (to the best of my knowledge, at the time of writing, only ~3 group robustness methods use the above heuristic), and in my opinion is not properly reflected in the writing. \n\n### No evidence the phenomenon persists for attacks with high efficacy (success rate)\n\nMy main concern with the paper is the following: if I have an attack (e.g., backdoor attack) with very high ASR, I would expect to have low loss on poisoned (e.g., backdoored) examples in the validation set. Hence, there's no reason for me to expect that JTT/GEORGE/etc would flag them as minority examples. In most of the reported setups, e.g. Table 2, the attack success rate is very low (e.g., 20%), even in the \"worst\" case when the group robustness method amplifies the attack. \n\n\nMinor point: It would have been nice to report results for the state of the art method (to the best of my knowledge) that uses the loss-based heuristic [1].\n\n[1] Qiu, Shikai, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. \"Simple and Fast Group Robustness by Automatic Feature Reweighting.\" arXiv preprint arXiv:2306.11074 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RfWe5n0i1", "reviewer": "Reviewer_Fmsc", "summary": "This paper studies the tension between group robustness methods and poisoning defenses; in particular, the authors show that group robustness methods which pseudo-label the minority group are often unable to distinguish minority samples from poison samples. On the other hand, poisoning defenses can make the minority group more difficult to learn, which hurts group robustness. The conclusions are supported by a variety of experiments and a simple theoretical result.", "questions": "1. I would encourage the authors to perform a more detailed literature review, as the related work subsection for group robustness is out of date and does not include any references more recent than 2021. There have been significant contributions since 2021, particularly in methods which pseudo-label the minority group (with or without an explicit identification model), e.g., [2,5,6,7,8,9].\n2. The definitions in Section 2.2 are unclear and should be improved.\n    * I believe the reason that the definitions of groups differs from [4] is to account for the dirty-label samples later in Section 3.3. This should be explicitly stated prior to the definitions, as otherwise the definitions seem inconsistent with the literature. Is the worst-group accuracy in the remainder of the paper computed with respect to these new groups (i.e., 8 groups) or the original groups (i.e., 4 groups)? This should also be made explicit.\n    * The definition of a minority group seems somewhat arbitrary and inconsistent with the literature. As far as I understand the definition given is that for a dataset of size $n$, a group $g\\in G_Y$ is a minority group iff the number of data belonging to $g$ is less than $n/|G_Y|$. If we use 4 groups for CelebA as is common in the literature, this would mean that blond-female is a minority group in CelebA (it has 22880 datapoints while the total data is 162770 over 4 groups), whereas the text states that the only minority group is blond-male. On the other hand, if we use 8 groups as I believe this work does, then blond-female is barely a majority group, and hence it is not clear that it remains a majority group when a random 10% subset of CelebA is taken.\n3. The bibtex could use an update: there are some extra braces and capitalization errors.\n4. It would be helpful for reference to include a table with the proportions of each group and class in each benchmark dataset, as well as the proportions of each group and class in the 10% sample of CelebA used in the experiments. See [9, 10] for examples.\n5. The definitions in Section 3.3 are unclear and should be improved. First, it should be made clear that $I(x)\\in [0,1]$, i.e., that it is a probability output rather than a logit output. Second, $I(x)_y$ is referred to as the “confidence” of model $I$ on input $x$ for class $y$, which is misleading: if $I(x)_y=0$ the model is in fact very confident (that $x$ does not belong to class $y$). I would expect the definition of “confidence” in this case to look more like $2|I(x)_y - 1/2|$, since $1/2$ can be considered as the “least confident” prediction in a binary classification problem.\n6. In Appendix A.2, should “DLDB” read “DLBD”?\n7. In Figure 1, it should be made clear that the lighter-colored circles and triangles are amplified points (it is otherwise an excellent figure). Also, the two shades of red are indistinguishable.\n8. The phrase “this boost is almost as high as it could have been” in Section 3.2 is confusing and could use a rephrase, perhaps connecting it to the worst-case approach tested earlier.\n9. In Section 5, “conciliate” should perhaps be replaced by “reconcile”.\n10. At the bottom of page 7, should the citation to Wu et al. be in parentheses, i.e., \\citep?\n\n***Recommendation***\n\nOverall, the novelty and importance of the contribution cause me to lean slightly more towards acceptance than rejection, but I believe the paper would be greatly improved with additional results on more rigorous benchmark datasets as detailed in the Weaknesses section, as well as improvements to the clarity of the writing.\n\n***After author discussion***\n\nIn light of the authors' response, which addressed all my concerns, I have concluded that this paper is valuable for the community and should be accepted to ICLR. Therefore, I have raised my score from a 6 to an 8.\n\n***References***\n\n[1] Idrissi et al. Simple data balancing achieves competitive worst-group-accuracy. CLeaR, 2022.\n\n[2] Taghanaki et al. MaskTune: Mitigating Spurious Correlations by Forcing to Explore. NeurIPS, 2022.\n\n[3] Lynch et al. Spawrious: A Benchmark for Fine Control of Spurious Correlation Biases. ArXiv, 2023.\n\n[4] Sagawa et al. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. ICLR, 2020.\n\n[5] Kim et al. Learning Debiased Classifier with Biased Committee. NeurIPS, 2022.\n\n[6] Sohoni et al. BARACK: Partially Supervised Group Robustness With Guarantees. ICML SCIS Workshop, 2022.\n\n[7] Zhang et al. Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations. ICML, 2022.\n\n[8] Qiu et al. Simple and Fast Group Robustness by Automatic Feature Reweighting. ICML, 2023.\n\n[9] LaBonte et al. Towards Last-layer Retraining for Group Robustness with Fewer Annotations. NeurIPS, 2023.\n\n[10] Kirichenko et al. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. ICLR, 2023.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "1. This paper makes an important contribution by exposing the incompatibility of current group robustness and poisoning defenses, especially showing that group robustness methods can make models more susceptible to poisoning adversaries. This observation is novel as far as I know, and should be interesting for the community.\n2. The experiments use a variety of robustness and poisoning techniques and metrics (if not datasets) and the extension to a federated learning application is welcome. The authors also provide a theoretical result in a simple yet well-motivated setting.\n3. The authors show that a naive combination of both techniques fails to resolve their tension, an interesting result which could represent a new direction of research for the community.", "weaknesses": "The main weakness of the paper is its limited dataset evaluation, which consists only of Waterbirds and a 10% subset of CelebA. While the experiments are indeed comprehensive with different techniques and metrics, the size and composition of the datasets questions whether the results hold more generally.\n\nIn particular, Waterbirds is known to the community to be a limited benchmark and should mostly be used for sanity checking or simple comparisons. First, Waterbirds is an easy benchmark in the sense that simple class balancing methodologies using ERM achieve similar worst-group accuracy performance to methods like Group DRO which utilize group information (as close as 1% WGA in [1]). Second, the validation and test distributions of Waterbirds are more amicable compared to the training dataset, as they are group-balanced conditioned on the classes; this can skew model selection in some cases. Third, Waterbirds is known to contain incorrect labels as well as images with both landbirds and waterbirds present [2]. Finally, while CelebA is a more complex dataset than Waterbirds, and the 10% subset was likely necessary due to computational restrictions, the fact remains that both datasets are small, belong to the vision domain, and only utilize binary-valued spurious features.\n\nSome suggestions may include Spawrious [3] for the vision domain (in particular because it is not too large; it is a difficult dataset which is smaller than CelebA) or MultiNLI [4] for the language domain (which would also introduce a non-binary-valued spurious feature).\n\nI also found the writing unclear in some parts, detailed in the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)