Combining Machine Learning Defenses without Conflicts
Authors: Vasisht Duddu
Rui Zhang
N. Asokan
Venue: Accepted by TMLR ()

Abstract: Machine learning (ML) models require protection against various risks to security, privacy, and fairness. Real-life ML models need simultaneous protection against multiple risks, necessitating combining multiple defenses effectively, without incurring significant drop in the effectiveness of the constituent defenses. We present a systematization of existing work based on how defenses are combined, and how they interact. We then identify unexplored combinations, and evaluate combination techniques to identify their limitations. Using these insights, we present, Def\Con, a combination technique which is (a) accurate (correctly identifies whether a combination is effective or not), (b) scalable (allows combining multiple defenses), (c) non-invasive (allows combining existing defenses without modification), and (d) general (is applicable to different types of defenses). We show that Def\Con achieves 90% accuracy on eight combinations from prior work, and 86% in 30 unexplored combinations evaluated empirically.

Published in Transactions on Machine Learning Research (08/2025)
Combining Machine Learning Defenses without Conflicts
Vasisht Duddu University of Waterioo
wusishl.cuuddccuuuuaeeee.....
Rui Zhang Zhejiang University
zhangrui28@:ju.eduu.
N. Asokan University of Waterloo
asokan Dacm. ovg
Reviewed on OpenReview: https: // openreview. net/ forun? id- CfF
Abstract
Machine learning (ML) models require protection against various risks to security, privacy, and fairness. Real-life ML models need simultaneous protection against multiple risks, ne- cossitating combining multiple defenses effectively, without incurring significant drop in the effectiveness of the constituent defenses. We present systematization of existing work based on how defenses are combined, and how they interact. We then identify unexplored combinations, and evaluate combination techniques to identify their limitations. Using these insights, we present, DEF DEF a combination technique which is (a) accurate (correctly identifies whether a combination is effective or not), (b) scalable (allows combining multiple defenses), (c) non-invasive (allows combining existing defenses without modification), and (d) general (is applicable to different types of defenses). We show that DEF CoN achieves 90% accuracy eight combinations from prior work, and 86% in 30 unexplored combina- tions which we empirically evaluated.
CBF
1 Introduction
Machine learning (ML) models are susceptible to a wide range of risks to security (Papernol el. al., 2018; Tian et al., 2022), privacy (De Cristofaro, 2020; Hu et al., 2022), and fairness (Mehrabi et al., 2021; Pes- sach & Shmucli, 2022). Several defenses have been proposed to mitigate them. Real-life models require simultaneous protection against multiple risks. But defenses designed to protect against one risk (Li et al., 2023a; Machado et al., 2021; De Cristofaro, 2020; Mehrabi et al., 2021) may impact susceptibility to other unrelated risks (Duddu et al., 2024a). This raises the question of how combine defenses effectively, with- out incurring significant drop in the protection provided by each constituent defense. Practitioners need effective combination lechniques, either by modifying eristing defenses (invasive), or by ident ifying whether existing defenses can be combined without modification (non-invasive). Prior work is either limited to specific defenses (e.g., Szyller & Asokan (2023); Chen et al. (2023); Fioretto et al. (2022); Noppel & Wressnegger (2024)), or study interactions among defenses with risks (Duddu et al., 2024a; Gittens et al., 2022). No prior work provides a way to quickly determine if defenses can be combined effectively.
We first systematically survey existing work on combining defenses based on: (a) how defenses are combined (i.e., what combination technique was used), and (b) how they interact (i.e,, whether they conflict align). We then identify previously unexplored combinations, and evaluate prior combination techniques to identify their limitations.
Non-invasive combination techniques are easier to deploy as they do not require expert knowledge from practitioners. Therefore, we identify "mutually exclusive placement" (Yaghini et al., 2023) as a promising technique. presumes that defenses can be effectively combined iff they operate on different stages in the ML pipeline: pre- in-, and post-training. However, it can result in ineffective combinations: (a) later-stage

---

Published in Transactions on Machine Learning Research (08/2025)
defenses can conflict with carlier ones (e.g., model or dataset watermarking with adversarial training differential privacy (Szyller & Asokan, 2023)), and (b) same-stage defenses may still be compatible (see $7).
Based on these insights, we present DEF CoN, a technique which is (i) accurate (correctly identifies whether a combination is effective or not), (ii) scalable (allows two or more defenses to be combined), (fi)) non-invasive (does not require changes to the defenses being combined), and (iv) general (applicable to different types of defenses). DEF CON is inspired by mutually exclusive placement (aka naïve technique), but overcomes its limitations by explicitly addressing the reasons that underlie conflicts among defenses: later-stage defense either (a) mitigates risk re-purposed by an carly-stage defense (Szyller & Asokan, 2023). We claim the following contributions:
defense by an early-stage defense, (b) overrides changes made
1. a systematization of prior work based on combination lechniques, and lypes of resulting interactions; ($4)
2. identifying unexplored combinations, and evaluating prior techniques for limitations.; ($5.1 and $5.2)
3. DEF CON', a scalable, non-invasive, and general combination technique ($5.3), which is more accurate than the naive technique, with a balanced accuracy of 90% (DEF), CON) vs. 40% (naive) using eight combinations from prior work as ground truth, ($7.2), 86% (DEF) CON) vs. 36% (naïve) via empirical evaluation of 30 unexplored combinations ($7.3 and 7.4). DEF CoN constitutes an inexpensive and fast technique for practitioners to determine if a combination of existing defenses without modification can be effective, without using expensive empirical evaluation.
2 Background: ML Notations
Consider X as the space of all possible input data records (e.g., images, text prompts) and as the space of corresponding outputs (c.g., classification labels for classifiers, prodicted next tokens for generative models). An ML model is a function 1 which maps x to y; ie., f X + where 0 indicates the model's paramelers. Hereafter, we denote fo by simply writing f. We consider two datasets of the form (z, y), where æ is the input data record and y is the output, for training an ML model using training dataset (D.r) and evaluate
the model on test dataset (Dre). We focus our evaluation ($7) on classifier models, and describe the training and inference for classifiers.
Training. We itcratively update using (x.8) E Drr over multiple epochs to minimize some objective function C: min [(f(2), y; + x R(0) where 1(f(z),)) y) is the prediction error on x for the ground truth y. 8
R(0) is the regularization function which restricts 0 from taking large values and À is a hyperparameter to control regularization. The parameters are updated as: 0 = a 80
ac where a the learning rate.
Inference. We measure the utility of f using its accuracy on Die computed as
Σ 1 [f(()=)) -
Óulf, Drest) D Drestt
(x.0)
where f(x) is the most likely class. If du is acceptable, f is deployed to provide predictions for input represented by f() for the probability vector across different classes.
3 Framework for Systematization
Given a list of defenses whose combinations are explored ($3.1), we present a framework to systematize prior work based on how the defenses are combined ($3.2), and how they interact (83.3). We then discuss the completeness of our framework and how to extend it ($3.4).
3.1 Defenses being Combined
We describe various defenses proposed to mitigate risks to ML models in the presence of an adversary (Adr). As part of our systematization (64), we will later enumerate all possible pairwise combination of these defenses.
Tink Code: http:///tttt...cco/////e//er///t
2

---

Published in Transactions on Machine Learning Research (08/2025)
Evasion robustness (EvsnRob) protects against evasion, which forces to misclassify an input by adding perturbation 8rab (aka adversarial examples) (Machado et al., 2021; Madry et al., 2018). Here, 8rob = urgmars,..!!fe + 8rob. y) and ||0roв|| < Erob, where Erob is a perturbation budget.
Poison robustness (PoisnRob) protects against poisoning which involves training f on poisons which are obtained by either tampering existing data records in Dtr or adding manipulated data records to Dir to degrade Φu (Tian et al., 2022). Alternatively, poisoning for backdoors forces f to incorrectly learn a mapping of some pattern in the poisons, to target class chosen by Adv. During inference, any data record with that pattern is then misclassified to the target class (Li et al., 2022).
Model Watermarking (MdIWM) checks for unauthorized model ownership, including model extraction allacks where Adv trains a local surrogale model to mimic the functionality of f (Orekondy et al., 2019).
MDLWM embeds watermarks in f that transfer to the surrogate model curing extraction. If a suspect model's watermark accuracy is above some pre-defined threshold, it identified as a surrogate.
Fingerprinting (Engrprnt) also checks for unauthorized model ownership by generating unique identifiers fingerprints (e.g., adversarial examples, embeddings), for f. These fingerprints transfer from to any surrogate model that are derived from it but are distinct from the fingerprints of independently trained models (Cao et al., 2021; Peng et al., 2022; Lukas et al., 2021; Zheng et al., 2022c; Maini et al., 2021).
Data watermarking (DtWM) checks for unauthorized data use where f is trained on datasets collected without consent (e.g., face images for facial recognition) (Sablayrolles et al., 2020; Huang et al., 2021;
Wenger et al., 2023). DTWM either augments Der with watermarks (e.g., backdoors) (Tekgul & Asokan, 2022; Sablayrolles et al., 2020), or selects high-influence samples from Dir as watermarks (Liu et al., 2022a). For verification, we check whether watermarks were in Der using statistical tests (Sablayrolles et al., 2020) or membership inference (Liu et al., 2022a). The difference between MDL WM and DTWM is how model trained from scratch on Drr is classified: DTWM flags it for unauthorized data use while MDLWM. M classifies it as independently trained.
Differential privacy (DiffPriv) protects against membership inference (whether a data record was in Dir) (Hu et al., 2022) and data reconstruction (reconstructing data records in Ptr) (Fredrikson al., 2015) by hiding whether individual's data record was used to train f (Abadi et al., 2016). Given two models trained on neighboring datasets differing by one record, DIFFPRIV bounds the privacy loss (distinguishability in predictions between the two models) by Ep H.. Here, dgp is the privacy budget and ddp is probability where the privacy loss is áp.
Group fairness (GpFair) minimizes discriminatory behavior to ensure equitable behavior across demo- graphic groups identified by a sensitive attribute in x (c.g., race or sex) (Mchrabi et al., 2021; Pessach & Shmueli, 2022). GPFAIR is measured using various metrics like accuracy parity, demographic parity (Zafar et al., 2019), equalized odds and equality of opportunity (Hardt et al., 2016).
Explanations (Expl) give insights into f's incomprehensibll behavior (Guidotti et al., 2018) which can be used to delect discriminalory behavior (Selvaraju et al., 2017; Kim et al., 2018). Explanations 7(x) indicate the influence of different input attributes in x on f(z). There are three main categories: Attribution- based (Ismail et al., 2021; Smilkoy et al., 2017; Sundararajan et al., 2017); influence-basse based (Koh & Liang, 2017); and recourse-based (Wachter al., 2017). We focus on attribution-based explanations which are popular in prior work on combining defenses, and applicable to various domains (e.g., labular, image). These explanations require training linear model in a region around point of interest (Ismail et al., 2021; Smilkov et al., 2017; Sundararajan et al., 2017). The cocfficients of the lincar model for an input x = .x) with n attributes, constitutes 7(z).
We summarize the defenses and their impact on du in Table 8.
3.2 Combination Techniques
Based on our survey described later in $4, we identify two combination techniques which either modify ezisting defenses, or identify whether eristing defenses can be combined without modification. We mark them T1 and T2 respectively, and describe them follows:
3

---

Published in Transactions on Machine Learning Research (08/2025)
T1 (Optimization) includes game-theoretic formalization, regularization, or constrained equation solving. TI incorporates defenses into the objective function (e.g,, regularization terms) so that the corresponding defense constraints can be satisfied during training for an effective combination (Xin et al., 2023; Hu et al., 2023a; Bu et al., 2022; Wu ct al., 2023; Zhang & Bu, 2022; He et al., 2020; Tran et al., 2022; Ali Mousavi et al., 2023; Benz et al., 2021; Ma et al., 2022; Nanda et al., 2021; Xu et al., 2021b; Sun et al., 2022; Li & Liu, 2023; Lee et al., 2024; Wei al., 2023; P & Abraham, 2021; Liu et al., 2021; Shekhar al., 2021; Zhang & Davidson, 2021; Tran et al., 2021b; Liu et al., 2022b; Lowy et al., 2023; Jagielski et al., 2019; Tran et al., 2021a; Yaghini et al., 2024; Ding et al., 2020; Xu et al., 2019a; Zhang et al., 2021; Esipova et al,, 2023; Xu et al., 2020; Tran et al., 2023; Lakkaraju et al., 2020; Chen et al., 2019; Li et al., 2023b). This also includes using variants of standard model architectures and algorithms, specifically catered for a particular combination to give better trade-ofis among the defenses (Ding et al., 2020; Xu et al., 2019a; Yang el al., 2022; Phan et al., 2019; 2020).
T2 (Mutually Exclusive Placement) consists of applying defenses at different stages of the ML pipeline i) pre-training (modifies Drr); ii) in-training (modifies training configuration such objective fune- tion), iii) post-training (modifies imputs or outputs of trained f during inference)- to avoid conflicts (Yaghini et al., 2023; Patel et al., 2022). We later refer to T2 the naive technique and use it as a baseline to compare with our proposed technique DEF CoN ($5.3 and $7).
3.3 Type of Interactions
Consider two defenses Dj and D2 which protect against risks Rk₁ and Rk2 respectively, with D2 is applied after Dj. The can interact in one of two ways:
Alignment. D1 and D2 are aligned if any of these hold: (i) D1 and D2 do not impact Rk₃ and Rky respectively; (ii) D1 reduces Rk2, increasing Dy's 's effectiveness; (iii) D1 generalizes D2, its effectiveness implies that of D2. Alignment leads to effective defense combination. When one defense implies the other (case (iii),, applying the first may be sufficient since we get the second for free (e.g., attribute privacy and group fairness (Aalmoes et al., 2022)).
Conflict. D1 and D2 conflict if any of the following hold: (i) D1 uses risk Rk (protected by D2), making D ineffective; (ii) D2 overrides Dj 's changes, making Dj ineefective. Conflict leads to an ineffective combination of the defenses, To avoid conflicts, we need accurate combination techniques.
3.4 Completeness of Framework
In $3.1, we identify some ML defenses for analysis. We do not claim that this list is complete. For instance, there are other defenses (e.g., individual fairness (Dwork et al., 2012), and interpretability (Kleinberg & hal,,,nathan, 2019)), or defenses specific to models other than classifiers (e.g., language and diffusion mod- els), and settings (e.g., federated learning). We can update the framework ($3.1) to add new defenses and enumerate all its combinations with other defenses (as shown later Table 1). Similarly, in $3.2, we do not claim that the list of combination techniques complete, but it covers all the techniques seen in our systematization (54). New combination techniques can be casily added into framework, and later used for systematization as shown in $4.1.
4 Systematizing Interactions among Defenses
We now use our framework to categorize existing literature. We present our methodology (64.1), and show the systematization of prior work ($4.2).
4.1 Methodology
We enumerate all defense combinations from $3.1, in Table 1. Each combination represented as a cell in Table 1, for which we indicate related work, combination technique used, and the type of resulting interaction.
4

---

Published in Transactions on Machine Learning Research (08/2025)
For each prior work, we identify the following: Combination Technique: We mark the technique to combine defenses as T1 or T2.
Type of Interaction: We mark the interactions as a conflict (E), alignment (E), or unexplored (E).
Justification. Prior surveys limited to specific defenses (e.g., Chen et al. (2023); Fioretto al. (2022); Noppel & Wressnegger (2024)), do not cover sufficient details to about combination techniques (e.g.. Gittens et al. (2022); Ferry et al. (2023)). This makes it challenging to design better combination techniques, and systematically compare with prior works. Our systematization addresses these limitations by (a) covering multiple defenses and their combinations, and (b) explicitly mapping them to the combination techniques and the lype of resulting interactions. As shown later $5, our systematization helps to identify gaps in existing literature (e.g., unexplored combinations, and limitations of prior techniques). Using the insights from systematization, we can design and evaluate a new combination technique ($5.3 and $7).
Selecting Papers for Analysis. We started surveying papers in Google Scholar using keywords (e.g, "combining <defense 1> and <defense 2>"). We selected all papers including those published in top-tier ML and security security/privacy privacy venues (e.g,, NeurIPS, ICML, ICLR, AAAI, CCS, S&P), related workshop papers, and unpublished papers on ArXiv. We examined their citations and related work to find other papers, Finally, we used papers from related surveys (e.g., Gittens et al. (2022); Chen et al. (2023); Fioretto et al. (2022); Noppel & Wressnegger (2024); Ferry al. (2023)) to ensure a comprehensive coverage.
4.2 Survey of Prior Work
We describe the defense combinations in the order of appearance along the columns in Table 1.
EvsnRob + PoisnRob. EVsNRoB suppresses adversarial examples (as outliers) while POISNROB sup- presses poisons (as outliers) in D,r- Hence, their objectives aligned. Viewing POISN as out-of- distribution (OOD) generalization, can modify adversarial training by incorporating noise from the new domain to improve domain generalization (Xin et al., 2023). This allows the model to learning robust fea- tures for OOD generalization, thereby aligning with POISNROB (E: T1). Hu et al. (2023a) defend against both poisons and evasion using a bi-level optimization (E: T1).
EvsnRob + MdIWM. Adversarial training, as EvSNROB.IN, suppresses the influence of backdoors which are used for MDLWM. (E: T2) (Szyller & Asokan, 2023). However, generating adversarial- example based watermarks with higher Grob than EvSNROB.IN, can result in an effective combination (E: T2) (Thakkar et al., 2023).
EvsnRob + DtWM. Radioactive data (Sablayrolles et al., 2020), as DTWM.PRE. adds backdoors as watermarks to Der by perturbing the inputs (similar to adversarial examples). Hence, adversarial training will suppress the influence of watermarks used for DTWM (E: T2) (Szyller & Asokan, 2023).
EvsnRob + Fngrprut. Dataset inference (Maini al., 2021) (as ENGRPRNT) is effective with EvsNRoB and incurs an acceptable performance drop (E: T2) (Szyller & Asokan, 2023). We attribute this to the defenses being applied at different stages (in-training vs. post-training), which reduces conflict between them. On the other hand, a variant of FNGRPRNT based on adversarial examples (i.e., "conferrable examples"), are ineffective when EVSNROB is applied for the target or the surrogate model (E: T2) (Lukas et al., 2021). We mark them both separately in Table 1.
EvsnRob + DiffPriv. Hayes et al. (2022) show that the generalization is worse on combining the objectives of adversarial training (as EVSNROB.IN) and DPSGD (as DIFFPRIV.IN), suggesting a conflict (E: T1).
Bu al. (2022) modify the minimax objective function of adversarial training to include DPSGD without violating its guarantees (E: T1). Wu et al. (2023) combine randomized smoothing with DPSGD by averaging the gradients of multiple training sample augmentations before clipping to account for the privacy budget of adversarial examples. Both techniques modify the objective function (E: T1).
Training f on some public data along with the choice of DIFFPRIV hyperparameters followed by task specific fine-tuning can result in better trade-off (E: T1) (Zhang & Bu, 2022; He et al., 2020). Other works have considered different optimizations: add DIFF PRIV noise to both input and hidden layers, ensemble adversarial
5

---

Published in Transactions on Machine Learning Research (08/2025)
Table 1: Overview of Pairwise Combinations among Defenses: For each combination cell, we citep related work and indicate the "interaction type" (= + alignment, = conflict, E unexplored), and "combination technique" used (T1-T2).
EvanRob
PoisnRob
MdIWM
Fngrprut
DLWM
DitPPriv
GpFair
E-+ T1: (Xia nl., 2023; al., 2023a)
T2: (Sayller Asokan, 2023)
MdIWM
(Thakkar al., 2023)
E-> T2: (Sayller Asokan, 2023)
Fngrpent
al., 2021)
DEWM
Σ- T2: (Sayller Asokan, 2023)
E
DitPriv
E++ T1: (Bu al., 2022; Wu
E- T2: (Xu
E- T2:
S- T2:
E- T2:
et al., 2021a; Vos sl., 2023; Ma et al., 2019)
(Szyller & Asokan, 2023)
(Sayller Asokan, 2023)
(Sayller
al., 2023; Phan
Asokan, 2023)
al., 9; 2020;
Zhang & Bu, 2022: He et ad., 2020)
T1: (Tran
E-TII (P &
E- T1: (Tran
GpPair
al., 2022:
Abraham, 2021:
et al., 2021b; Liu
Ali Mousavi sl.,
Liu al., 2021;
2022b;
2023; Benx et al.,
Shekhar et al.,
Lowy al., 2023;
2021; al.,
2021; Zhang Davidson, 2021)
Jagielski
2022; Nanda
2019; Tran et al.,
sl.,
2021s: Yaghini
et al., 2021b; Sun
es al., 2024; Ding
al., Li & Liu, 2023: Loe
2020; Xu
es al., 2019a:
sl., Wei
Zhang al.
et al., 2023)
2021; Esipova
2023; Xu es al., 2020; Tran 2023) =- T2: (Yaghini ct al., 2023)
(Sun al., 2022)
Expl
E-+ T1:
E- T1: (Yang 2022): (Patel et al., 2022)
(Lakkaraju al. 2020; Chen et al., 2019: Li et al., 2023b)
training to add adversarial examples to private Dtr, or modify objective function for DIFFPRIV guarantees on adversarial examples (E: T1) (Phan et al., 2020; 2019).
EvsnRob + GpFair. Adversarial training (as EvsNROB. IN) and group fairness (as GPFAIR.IN) have con- flicting objectives: EVSNROB.IN pushes the decision boundary away from Drr while GPFAIR.IN brings it closer (Tran et al., 2022). Also, EvsNRoB.IN IN increases the disparity among demographie subgroups due to class imbalance in D.r (Hu et al,, 2023c) and long-tailed distribution (Lee et al., 2024; Benz et al., 2021; Nanda et al., 2021; Hu et al., 2023c). Several works modily EvSNROB.IN' objective function to ensure equitable model behavior across demographic subgroups by assigning higher weight to minority subgroup (E: T1) (Ali Mousavi et al., 2023; Benz et al., 2021; Ma al., 2022; Nanda et al., 2021; Xu et al., 2021b; Sun al., 2022; Lee al., 2024: Li & Liu, 2023). Wei al. (2023) use different training configurations and assigning different weights to different classes to improve class-wise robustness T1).

---

Published in Transactions on Machine Learning Research (08/2025)
EvsnRob + Expl. Adversarial training (as EVSNROB.IN) improves the interpretability of the gradi- ents (Tsipras et al., 2019). This suggesting an alignment with explanations (Chalasani et al., 2020). Also, both defenses can be combined using minimax objective to construct high fidelity explanations while resisting adversarial examples (E: T1) (Lakkaraju et al., 2020; Chen et al., 2019; Li et al., 2023b).
PoisnRob + DiffPriv. DIFFPRIV reduces the influence of outliers thereby improving robust against poisons as shown in several works (Xu et al,, 2021a; Vos et al., 2023; Ma et al., 2019; Jagielski & Oprea, 2021). Hence, DPSGD mitigates poisons and the defenses have aligned objective, resulting in an effective combination (E: T2).
PoisnRob + GpFair. POISNROB may overly flag data records from the minority groups as outliers for removal, which increases the bias (Shekhar et al., 2021). This can be corrected by reweighing the scores assigned to outliers to account for sensitive attributes (E: T1) (P & Abraham, 2021; Liu et al., 2021). Also, an outlier detector can be trained to minimize the correlation between outlier scores and sensitive attributes TT) (Shekhar et al., 2021; Zhang & Davidson, 2021).
MdIWM + DiftPriv. DPSGD (as DIFFPRIV.IN reduces memorization of data records in Dir and reduces the impact of backdoors for watermarking (MDLWM). Hence, MDLWM conflicts with DIFFPRIV (E: T2) (Szyller & Asokan, 2023).
DtWM + DiffPriv. Ideally, DPSGD (as DIFFPRIV.IN IN) suppresses watermarks (in DTWM), suggesting a conflict. However, empirically, DrWM was effective when combined with DPSGD (Szyller & Asokan, 2023). The adversarial example-based watermarks radioactive watermarking (Sablayrolles al., 2020), were relatively inliers and not suppressed by DPSGD (2: T2) (Sayller & Asokan, 2023).
Fngrprnt + DiffPriv. Szyller and Asokan (Szyller & Asokan, 2023) found that DPSGD (as DIFFPRIV.IH IN) and dataset inference (ENGRPRNT) do not conflict, though no reason was provided. We attribute this to defenses being applied in different stages, reducing conflict (E: T2).
DiffPriv + GpFair. DPSGD (as DIFFPRIV.IN) shows disparate behavior over demographic sub- groups (Bagdasaryan et al., 2019). Theoretically, it is impossible to design high utility binary classifier that is private and fair (Cummings et al., 2019; Agarwal, 2021). Several works modify the objective function by using fairness constraints, regularization, and game theoretic optimization (3: T1) (Tran et al., 2021b; Liu et al., 2022b; Lowy et al., 2023; Tran et al., 2021a; Jagielski et al., 2019; Yaghini et al., 2024; Mozannar et al., 2020). Yaghini et al. (2023) combine demographic parity regularization with DPSGD, and estimate fairness on a public dataset to avoid consuming extra privacy budget (E: T1). Also, functional mechanism adds Laplace noise to the objective function, along with varied noise levels for different subgroups, which reduces discrimination (E: T1) (Ding et al., 2020; Xu et al., 2019a). However, this is limited the convex objective functions (e.g., logistic regression). Esipova et al. (2023) attribute unfairness in DPSGD to the differences in unclipped and clipped gradient directions. Subsequently, several works have used proposed variable gradient clipping to minimize discriminatory behavior while maintaining utility (E: T1) (Xu et al., 2020; Tran et al., 2023; Zhang et al., 2021). Yaghini et al. (2023) use PATE framework to apply fairness constraints and DIFFPRIV noise in the aggregated votes from the teacher's ensemble. Both fairness and privacy are applied in pre-training (E: T2).
DiffPriv + Expl. The objectives of these defenses are inherently conflicting: DIFFPRIV hides information to minimize leakage while EXPL releases additional information to improve comprehensibility (Banisar, 2011). Yang et al. (2022) train an autoencoder with DIFFPRIV (functional mechanism). This autoencoder is used to generate data records and compute counterfactuals that satisfy DIFFPRIV via the post-procossing property (E: T1). Patel et al. (2022) propose an adaptive DPSGD algorithm to generate high-quality explanations without consuming <dp, by reusing past explanations for similar data records (E: T2).
5 Insights from Systematization
We identify unexplored combinations ($5.1), requirements for an ideal technique, limitations of prior tech- niques ($5.2), and design new technique (55.3).
7

---

Published in Transactions on Machine Learning Research (08/2025)
5.1 Unexplored Combinations
We identify 14 unexplored combinations (E in Table 1): (i) POISNROB with (MDLWM, DrWM, FNGRPRNI, EXPL]; (ii) MDLWW with {DTWM, FNGRPPNT, GPFAIR, EXPL}; (ii) DTWM with JFNGRPRNT,GPEAIREEEEE (iv) FNNEPRNN with (GPFAIR, EXPL]; (v) GPFAIR with EXPL.
Takcaway: Unexplored combinations reveal research gaps and opportunitics for effective technique design.
We revisit these combinations in our evaluation ($6 and $7).
5.2 Evaluating Combination Techniques
Our systematization reveals that some techniques may lead to ineffective combinations. We outline require- ments for an ideal technique, and identify limitations in prior work.
Requirements. A combination technique should alow practitioner to quickly determine whether combina- tion can be effective combined. Empirical evaluation to determine the effectiveness of combination, while definitive, can be expensive, especially when multiple defenses are involved. An ideal combination technique should be: R1 (Accurate) correctly identifies whether a combination is effective or not; R2 (Scalable) allows Iwo or more to be combined simultaneously; R3 (Non-invasive) does not require modifying defenses, easing adoption and removing the need for expert knowledge; R4 (General) applicable to various defenses.
Takeaway: Combination techniques, including newly proposed ones, should be evaluated for R1-R4.
Limitations of Prior Techniques. We summarize the limitations of existing techniques (TI-T2) as per the requirements (R.1R44) in Table 2. We use O for requirement not satisfied, for partially satisfied, and for fully satisfied.
T1 (Optimization) where modifying the objective function for training, followed by hyperparameter tuning, can result in an effective combination. How- ever, this often results in trade-off between effec- tiveness of constituent defenses and model utility. Hence, we mark T1 as partially accurate (R1
Table 2: Requirements satisfied by various techniques:
Not satisfied; Partially satisfied; fully satisfied.
Technique
R3 Invasivv
R4 (General)
(Accurate)
(Scalable)
T1
C). This trade-off also explains why prior works have struggled to scale beyond two defenses (R2 +
T2
O). Furthermore, some defenses are not applicable for T1 (e.g., EXPL and ENGRPRNT), and require modi- fications or non-standard variants (R3 > O). Finally, these optimizations are tailored to specific defenses being combined, and do not apply to other defenses. They are also specific to some models (e.g., logistic regression with DIFFPRIV), and do not translate other models (e.g., neural networks). Hence, T1 has limited applicability (R4- O).
T2 (Mutually Exclusive Placement) can apply up to one defense in each of the three stages, thus, making it scalable (R2 + Defenses do not need any modification (R3 + and the combination technique is applicable to all types of defenses (R4- + However, the combinations may not be effective: (i) a defense in a later stage of the pipeline can conflict with carlier ones (false negatives) (Szyller & Asokan, 2023), and (ii) it rules out defenses in the same stage that do not conflict (false positives see $7). Hence, this may incorrectly identify effective combinations (partially accurate R1
Takeaway, Neither techniique satisfies all the requirements. T2 promising as it satisfies R2, R3,, and R4, but not R1 (incurs false positives and false negatives).
5.3
\ef{ Con: Design
From our systematization, we identify that T2 overlooks underlying causes for conflicts, leading to false Can we address the limitations of T2 and improve R1? Recall from $3 that conflicts arise when (i) an early-stage defense uses risk which is mitigated by later-stage defense, or (ii) changes
8

---

Published in Transactions on Machine Learning Research (08/2025)
by an early-stage defense is overridden by later-stage defense. We conjecture that by accounting for these underlying causes, can meet R.I. We present DEF CON, principled technique to identify effective defense combinations, by accounting for the reasons underlying conflicts
Methodology to Derive Def Con. We start with the naive technique and modily to include the underlying causes for conflicts among defenses. We iterate over the design of DEF CoN using prior work from $4, and evaluate the final design unexplored combinations (scc $7).
S1: Same Stage?
yes
$2:
D2 local/na change?
S3: uses
no
ves
yes
Def|Con Con Description. We describe DEF\ CoN us- ing the example of combining two defenses, D1 and D2 which protect against Rk₃ and Rk₂ respectively, and later discuss how to extend to more than two de- fenses. Following prior work (Duddu et al., 2024a), refer to unintended interactions between de- fense and risk if the defense either increases or de- creases the susceptibility to an unrelated risk (e.g., D2 and Rk2).
S4: D2 protects
n1
against
yes
Flowchart depicting various steps in DEF CON identify conflict between and D2 (D2 is applied after D1).
Figure 1:
We start by identifying variants of each of the defenses across pre-, in-, and post-training stages (see Table 8). We compare each variant of D1 with that of D2, and use A for alignment, and A for confliet. Assuming D is applied first and then D2, follow the steps below in sequence:
S1 Are D1 and D2 applied in the same stage?
If yes, go to $2 Elsc, go to S3 $2 The type of changes made by the defenses determines whether there is conflict. We classify the changes as global, local, and none. Global changes modily f (e.g., training with regularization term, pruning) or transform all records in Der (e.g,, synthetic data generation for DP or fairness during pre-training). Local changes affect specific data records (e.g., adding watermarks in pre-training or modifying certain predictions in post-training). FXGRPRNT: POST and ExPI.PPSST make no changes to I and Dtr.
D; makes global/local no changes while D2 makes local/no nno changes, we mark this as A.
Rationale: Changes by D1 will not interfere with local/no changes by D2, as D1 is applied first. Hence, no conflict.
Df makes global/local/mn changes while D2 makes global changes, mark as A.
Rationale: Global changes by D2 will override changes by Di, thereby reducing ils effecliveness. This is called catastrophic forgetting when the defenses applied sequentially during training (Kemker et al., 2018; Szyller & Asokan, 2023). This is conflict.
S3 D1 and D2 in different stages. Does Dj use risk Rk part of the defense (e.g, watermarking uses backdoors)?
If yes, go to S4. If no, mark as A.
Rationale: If Dj does not use Rk, the susceptibility to Rk will not be impacted after applying D2- Hence, Di and D2 are undikely to interfere with each olher.
$4 Does D2 protect against either explicitly or via unintended interaction?
If yes, mark as A. Rationale:
Since Dj uses Rk (either explicitly or via unintended interaction), D2 will reduce susceptibility to Rk making D1 less effective. Hence, there conflict.
If no, mark as A.
Rationale: Dj and D2 are unlikely to interfere with each other. Hence, there is no conflict.
We summarize the steps in DEF Con in Figure 1. DEF CoN evaluates combination effectiveness based solely on the effectiveness of the constituent defenses, without considering the model utility. We revisit model utility in $8. We also present formal analysis of DEF CoN covering consistency, soundness, and completeness in Appendix B.
9

---

Published in Transactions on Machine Learning Research (08/2025)
Note on DiffPriv, Combining DIFFPRIV with other defenses does not consume additional privacy budget. Any modification to DDr (e.g., adding watermarks) is done within the privacy boundary, and does not consume privacy budget. Any defense applied after DIFFPRIV is "free" (DIFFPRIV's post-processing property).
Extending Beyond Two Defenses (Multi-way Combinations). To extend DEF Con beyond two de- Tenses, our algorithm decomposes the multi-way combination into pairwise comparisons, invoking the original DEF CoN flowehart (Figure 1) to check for conflicts. Determining conflict for multi-way combinations from pairwise combinations is principled and not limiting. We have to consider two possible cases when decom- posing multi-way combinations into pairwise combinations:
Case 1: Defenses in different stages. When we combine three defenses in different stages, checking conflicts among all pairwise combinations is fine since the order in which we apply defenses is fixed. Any conflict detected among any pair, will be marked as an overall confict (marked as A).
Case 2: More than one defense in one stage. We have to check all possible permutations of the defenses in a given stage, and determine whether there a conflict. If there is no conflict, we combine with other stage defenses, and check for the conflicts.
We now present our algorithm to evaluate conflict for multi-way combinations, and assumn that we have a set of defenses- reffeses-partitioneee partitioned into three ordered stages. We iterate through each stage in order, and skip stages with no defenses:
MI If stage has single defense: For the stage 8, we consider the defense as D2 and check its compatibility with defenses in other stages. D, is D1, D2 D3 depending
M2 If stage has multiple defenses: For cach stage with multiple defenses:
M2.1 Permutation Checking: Consider all permutations of the defenses. For each permutation, check every consocutive pair of defenses using DEF flowchart to detect conflicts for two defenses. M2.2 Pruming and Selection: If a pair in a permutation causes a conflict, discard that permutation and prune others containing that conflicting pair (e.g., for defenses: A, B, and C, if "AB" conflict, then no need to check for "ABC" or "CAB"). If a conflict-free permutation is found, treat the entire sequence as a single composite defense, D, (e.g., let us say "CAB" does not conflict).
M3 Sequential Composition: Treat the resulting composite defense as an atomic unit (e.g., D1 = "CAB") and check for conflicts with the next stage's resolved defenses (e.g., D2 "DE"), using DEFACON flowchart between Dj and D2. In other words, we check whether any of the constituent defenses in D use risk which is mitigated by constituent defenses in D2 (S3 in Figure 1).
M4 Termination: If any invocation of DEF CoN for pairwisc check indicates a conflict, terminate and report as a conflict (marked as A). Else, indicate as aligument (marked as A).
We present a formal analysis of the algorithm to extend DEF CoN for more than two defenses in Appendix C. Having discussed the DEF CON's design, we comprehensively evaluate DEF CoN across R1--R4
6 Experimental Setup
6.1 Datasets and Models
We use two image datascts: FMNIST and UTKFACE. FMNIST consists of 28x28 grayscale images of ten clothing types, with 60,000 training and 10,000 testing images. We classify these using a two layer CNN with 16 and 32 filters, ReLU activation, and fully connected layer for ten-class classification. UTKFACE includes 48x48 RGB images, classifying individuals as young (under 30), with 11,852 training and 10,667 testing images. It also includes the sex of the individuals as a sensitive attribute. We use a VGG16 model with fully connected layer for binary classification.
We choose FMNIST since all of the defenses we consider for evaluation ($6.3) have used it for evaluation. We selected UTKFACE because many defenses effective for FMNIST are likely be applicable to it as well, given that both are image datasets. Also, UTKFACE includes sensitive attributes, making it suitable for GPFAIR.
10

---

Published in Transactions on Machine Learning Research (08/2025)
6.2 Revisiting Defenses
Since the naive technique and DEF CoN apply defenses at different ML pipeline stages, revisit and categorize defenses in $3.1 by the stage they are applied in. For each defense from $3.1, we specify the variants and post-training ("<defense>> seateenss....s")). For additional context, we indicate the impact of applying a defense on Φu compared to a "no defense" bascline, where "v" is decrease, "~ " is no effect, and "A" is an increase.
in pre-training
(" (*<defense>> "reffenss>.Pre)), in-training ("<defense>.In)
Evasion robustness (EvsnRob)
EvsnRob.Pre (Data Augmentation) where adding transformations of training data records improves robustness (Yun et al., 2019; Zhang et al., 2018b; DeVries & Taylor, 2017; Rebulfi et al., 2021) (but see $6.3). This improves Qu (^) by acting as regularization (Yun et al., 2019; Zhang et al., 2018b; DeVries & Taylor, 2017; Rebufli et al., 2021).
EvsnRob.In (Adversarial Training) modifies the objective function to minimize the max- imum loss from adversarial examples (Madry et al., 2018; Zhang et al., 2019): Laddur = ming Pi. Ex.yED.. max]já] ()) Alternatively, randomized smoothing modifies the training and inference to obtain certified robustness of f (Cohen et al., 2019; Lecuyer et al., 2019). These defenses decrease du (V) (Zhang et al., 2019; Tsipras et al., 2019).
EvsnRob.Post (Input Processing) removes adversarial perturbations before passing them to ʃ (e.g.- using generative models (Nie et al., 2022; Song et al., 2018) or input encoding (Buckman et al., 2018; Guo et al., 2018; Das et al., 2017)) checks for adversarial examples using statistical tests (Grosse et al., 2017). Defenses which modily input images using generative models decrease On (V) (Nie et al., 2022; Song et al., 2018; Guo et al., 2018; Das et al., 2017). If the input transformation is small, the decrease in @u is negligible (~) (Buckman et al., 2018; Grosse et al., 2017).
Poison robustness (PoisnRob)
PoisnRob. (Data Sanitization) includes detecting and removing outliers in Der (e.g., using Shapley values (Jia et al., 2021b; 2019; Doan et al., 2020) or anomaly detection (Cretu et al., 2008; Paudice et al., 2018; Tran al., 2018; Barreno et al., 2010; Chen et al., 2018)), followed by retraining. As the outliers are memorized and contribute to 6u, their removal degrades Φu (V) (Jia et al., 2021b; 2019).
PoisnRob.In (Fine-tuning) updates f to minimize outlier influence. This includes distillation to reduce the influence of poisons (Li et al., 2017) or fine-tuning on poison-free dataset (Diakonikolas et al., 2019; Zhu et al., 2023; Xu et al., 2019b; Liu & Guo, 2020; Patrini et al., 2017). These do not impact du (~). PoisnRob.Post (Pruning) roduces the effectiveness of backdoors by removing some model paramoters based on the observation that poisoned and clean samples have different activations (Liu el al., 2018; Wu & Wang, 2021; Zheng et al., 2022b;a; Li et al., 2023c). This degrades du (V).
Model Watermarking (MdIWM)
MdIWM.Pre (Backdoors) uses backdoor watermarks in Drr (Adi et al., 2018; Zhang et al., 2018c; Jia et al., 2021a; Uchida et al., 2017). These are designed to maintain du (~). MdIWM.In (Optimization) updates the original objective function to include watermark behav- ior (Bansal et al., 2022; Bagdasaryan & Shmatíkov, 2021). For instance, certified watermarking adds Gaussian noise to watermarks (added to Der) to get certification on watermark accuracy (Bansal et al., 2022). Also, backdoors can be introduced (hrough regularization, which can be repurposed for water- marking (Bagdasaryan & Shmatikov, 2021). This degrades Φu (V).
M.lWMM (API) modifies predictions to embed watermarks (Szyller et al., 2021) which are used by Adv to train the surrogale model. These are designed to maintain &u (~).
Fingerprinting (Fngrprnt). All fingerprints are post-training schemes (denoted as Engrprnt.Post) No retraining or modification of f is required and hence, FNGRPRNT has no effect on du (~).
Data watermarking (DtWM). All the current schemes are during pre-training (DtWM.Pre). and are designed to maintain @u (~). The difference between MDLWM and DrWM is how a model trained from scratch on Dir is classified: DTWM flags for unauthorized data use while MDLWM classifies it as independently trained.
11

---

Published in Transactions on Machine Learning Research (08/2025)
Differential privacy (DiffPriv)
DiffPriv.Pre (Private Data) from generative models with DIFFPRIV constraints, that can be used for downstream tasks instead of Der (Hu et al., 2023b; Xie et al., 2018; Torkzadehmahani et al., 2019; Zheng & Li, 2023). This decreases du (V).
DiffPriv.In (DPSGD) trains f by adding carefully computed noise to the gradients to minimize the influence of individual data records on f (Abadi et al., 2016). Private aggregation of teacher's ensembles (PATE) (Papernot el al., 2017) another framework for DP where multiple teacher models are trained disjoint private datasets, while a student model is trained via noisy voting from the teacher models. These defenses decrcase @u (V) (Jayaraman & Evans, 2019). DinfPriv.Post (Output Perturbation) includes adding calibrated noise to the output of empirical risk minimization objective (Chaudhuri et al., 2011). This decreases ou (V). The theoretical guarantees are poorer than other DP defenses and DIFFPRIV.PON POST requires the objective function to be convex. We omit this since does not cover neural networks.
public dataset with labels annotated
Group fairness (GpFair)
GpFair.Pre (Fair Data) modifies Der to reduce bias in the downstream model (Kamiran & Calders, 2011; Calmon et al., 2017; Zemel et al., 2013; Feldman et al., 2015). This degrades Φu (V). GpFair.In (Regularization) penalizes violating fairness constraints (Agarwal et al., 2018; 2019; Celis ct al., 2019; Kamishima et al., 2012). This dogrades du (V) (Zhang et al., 2018a; Louppe et al., 2017; Pinzón et al., 2023).
GpFair.Post (Calibration) adjusts the threshold over the predictions ensure that the prediction probabilities accurately reflect the true likelihood across each demographic group (Pleiss et al., 2017: Hardt et al., 2016; Kamiran et al., 2012: Geyik et al., 2019; Salvador et al., 2022; Kull et al., 2017: Hebert-Johnson et al., 2018) This degrades du (V) (Pleiss et al., 2017).
Explanations (Expl). EXPL are post-training defenses (Expl.Post) which does not require retraining, and hence does not degrade ou (~).
We summarize the defenses in Appendix A: Table 8.
6.3 Choosing Defenses for Evaluation
To select defenses for our evaluation, we began with those in $6.2 (summarized in Appendix A: Table 8). We remove defenses which are not robust: EVSNRoB.POST (Input Processing) and POISNROB. PRE (Data Sanitization) (Kang al., 2024; Koh al., 2022). We then evaluate the remaining defenses and exclude those which were ineffective et al., 2018b; DeVries & Taylor, 2017), DIFFPRIV. PRE (Private Data) (Zheng & Li, 2023), GPFAIR. PRE (Fair Data) (Zemel et al., 2013), and GPFAIR. (Calibration) (Pleiss et al., 2017). DIFFPRIV. PRe, GPFAIR. PRE, and GPFAIR. POST, were designed for labular datasets but ineffective on our image datasets. We speculate about these defenses in $8.
datasets: EVSNROB. (Data Angmentation) (Yun et al., 2019; Zhang
We are left with cleven defenses: (i) EVSNROB. IN (adversarial training), (ii) POISNROB. IN (fine-tuning). (iii) POIsNRoB.PoSe (model pruning), (iv) MoL.W. (backdoor watermarks), (v) MOLWM.IN (wa- termarks via objective function), (vi) door watermarks), (vili) ENGRPRNT.POS: (datasct inference), (ix) DIFFPRIV.IK IN (DPSGD), (x) GPFAIR.IN (regularization), (xi) EXPL.POST (altribution). We get 55 pairwise combinations from them but we remove combinations among defenses with the same objective: three combinations among watermark- ing (MDLWMM PRE, MDLWM.IN, MDLWM. three for EXCRPRNI.POST with MDLW MRP,, MDLWM.. MDL POST,, and one for POISNROB.IN and PoIsNRoB. This leaves us with 48 combinations.
(APl-based watermarks). (vii) DTWM.PRE (back-
6.4 Metrics and Implementations
We describe the metrics for evaluating the effectiveness of each defense, and the implementations taken from publicly available code from prior work. We measure th on Dze for all defenses. Our implementations for
12

---

Published in Transactions on Machine Learning Research (08/2025)
defenses are based on the state-of-the-art (ENGRPRNT, POISNROB), standard libraries (DIFFPRIV, GPFAIR, ExPL.), or seminal work (EvsNRoB, DrWM, MD..MM.. We use the standard hyperparameters which are either from the literature or the library documentation, such that the resulting individual defenses are effective (Table 3). When combining defenses, we use the same hyperparamcters, but revisit hyperparameter luning for defenses in combination (see $7.4). We report the mean and standard deviation across five runs.
Evasion Robustness (EvsnRob.In). We use the accuracy on Drob which obtained by replacing data records in Dr with the adversarial variants:
Probace (FVUSNRON Droh) - |Drabl (zo))
Σ
[(SEemRRO(()) =
Ideally, dEvaxttoa should be close to Φu- We use the original implementation of TRADES (Zhang et al., 2019)
Prbaac and an implementation of AutoAttack (Croce & Hein, 2020) in SecML library (Pintor et al., 2022b) from Pintor et al. (2022a). As we evaluate effectiveness of EVSNROB.IN IN using attacks, poorly optimized attacks can falscly suggest defense effectiveness (Carlini et al., 2019; Carlini & Wagner, 2017: Tramer et al., 2020). We individually optimize these attacks for evaluation with defenses and their combinations, following Pintor et al. (2022a) to address failures identified by various indicators (e.g., poor optimization). We evaluate across various AutoAttack variants by (a) modifying the loss function: cross entropy (CE) and difference of logits ratio (DLR), (b) applying expectation over transformations (EoT), and (c) using random starts. For FMNIST, we use DLR, CE, DOR+ and CE+FOT. For UTKFACE, we use CE and CE+FoT, as DLR is not applicable for binary classification. We report the best attack (least @robace).
Outlier Removal (PoisnRob). We compute accuracy on Dod (from adding backdoors to records in Die):
Σ T(fronNtOO(()) =
@fSs(foosntoo: Dra) = |Dbal
(z.g)EDod
where Yt is the target label chosen by Adv. Ideally, ØASR should be zero. We use BadNets (Gu et al., 2017) to generate poisons by adding a white patch of size 5x5 to the images, applied to 10% of Drr- For POISNROB.IN (Fino-tuning), we fine-tune the last layers of f using random sample of 10% of Der without poisons (Sha et al., 2022). For PoisNRoB.POS (Pruning), we use the inplementation from Zheng et al. (2022b). We sweep pruning thresholds from 0.6 to 1.5, in increments of 0.05, to get a model with highest d and lowest ASSR
Model- Wodde-WWatrmmarkng (MdIWM). We use the accuracy on Dwmm which is obtained by adding water- marks to data records in Die. We compute this watermark accuracy as:
Σ [[fanuma(=)= = 44
@wmace (fMMLMM PwmM) DDumM]
(w.y)EDwmM
where ym represents the target labels for watermarked records. Ideally, Owmace should be 100% if the model is successfully watermarked. For MoLWW. PRE (Backdoor), we use BadNets (Gu et al., 2017), similar to Szyller and Asokan (Szyller & Asokan, 2023), by adding a white patch of size 5x5 to 10% of the images in Der. For MDL WM.I (Modifying Loss), we use the certified neural network watermarking implementation by Bansal et al. (2022). For MoL.M. fraction of the predictions from target model as watcrmarks, which is later used to train the surrogate model. Following the original work (Szyller al., 2021), we apply the watermark to 0.2% of the predictions. Unlike other watermarking schemes, we compute ówmace on the surrogate model and not the target model.
(APT), we use DAWN (Szyller et al., 2021), which flips a
Fingerprinting (Fngrprnt.Poste We use dataset inference (Maini et al., 2021) as our fingerprinting scheme which extracts feature embeddings from J, and trains a classifier to distinguish bet Dr, and Die. A model is considered stolen if the distance of its embeddings is similar to f with high confidence, and veri- fication is successful if the p-value < 0.05. We use épval as the metric following Szyller and Asokan (Szyller & Asokan, 2023). We use the step size of 1.0 for L1 attack, 0.01 for Lg attack, and 0.001 for Linf, and 50 samples for computing p-value from the confidence regressor model.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Data- Watermarking (DtWM.Pre). To determine if dataset was used to train model, we compare the posterior probability of 100 watermarked testing samples against 100 benign ones using a pairwise t- test (Li et al., 2020). We then calculate the rate of successful detection (Ørea), which reflects the percentage of correctly identified watermarked samples from DWMD (Die with watermarks). Watermarks are generated using BadNets (Gu et al., 2017) where 10% of Der watermarked, and we use verification code from et al. (2020) compute Prdd Ideally, And should be 100% for watermarked models.
Differential Privacy (DiffPriv.In). We use Oap cdp: following Szyller and Asokan (Szyller & Asokan, 2023), where ideally, want low Edp- We use the implementation from Opacus library (Yousefpour et al., 2021) with noise multiplier of 1.0 and gradient norm clipping of 1.0 as used in their tutorial for MNIST.
Group Fairness (GpFair.In). We measure fairness using the equalized odds gap on Dte. for sensitive al- tributes S and model predictions Y, given by: @epudd = p( = = = 0, Y = y) - p(Ý = = =, = v) E {0, 1 where an ideal value of zero indicates perfect fairness. For GFFAIR.IN (Regularization), we use code from the fair fairness benchmark that adds a regularization term to penalize equalizod odds violations (Han el al., 2023). We set the regularization hyperparameter = which was sufficient. to reduce Oegodds with 2% drop in Qu.
Explanations (Expl.Post). We assess the quality fo explanations using converyence delta, measures the error between the explanation for data records and a baseline (Kokhlikyan et al., 2020). We report the average convergence delta across all Die records herr. We use DeepLift (Shrikumar et al., 2017) from Captum library which recommends using a zero vector as a baseline.
Table 3: Effectiveness of defenses. For metries, we use f (resp. 1) where higher (lower) value is better, and 'x' is shorthand for (.)
7 Evaluation
(m) (Øu is for context).
We evaluate individual defenses ($7.1), compare DEF' to naïve technique ($7.2 and $7.3), study the impact of hyperparameter tuning ($7.4), and confirm that DEF).CON meets all requirements (87.5).
Defense
Metric
FMITST
UTKPACE
u (t)
90.97 0.18
80.28 1.26
robace (1)
7.96 1.24
0.00 0.00
ASR (4)
99.95 0.04
99.98 0.05
No Defense
winace.Pre Pre (t)
9.98 + 0.28
0.00 = 0.00
(f)
6.28 1.20
62.21 6.03
wmace. Post (t)
0.00 + 0.00
13.33 = 6.32
RSD (t)
0.00
0.00
7.1 Evaluating Individual Defenses
egodds (4)
28.10 di 6.34
dp (4)
20
We evaluate the effectiveness of each defense by comparing the metrics o a "no defense" base- line. We report the results in Table 3. We also report & to provide context but do not use it to evaluate accuracy of the technique.
EveuRob.In
u (t)
86.93 0.23
73.38
robace (t)
76.09 =: 0.28
37.38 =: 1.30
PoienRob.Iu
u (t)
89.38 0.28
79.02 0.30
ASR (4)
9.94 0.24
56.62 37.83
PolsnRob. Post Post
u (t)
86.48 :: 2.35
66.42 3.27
ASR (4)
66.44 21.30
8.59 16.41
MdIWM.Pre
u (t)
0.27
We find that all the defense effectiveness metrics are better than the "no defense" baseline. Once the de- fenses are applied, we use their respective &., as the "single defense" baseline to compare the effective- ness of the defense combinations later in $7.3. For dere Post we do not have a "no defense" bascline to compare with. Assuming &f Øer we use it as the "single defense" baseline.
wmace (1)
99.91 0.05
100.00 J: 0.00
MdIWM.In
u (t)
80.87 = 0.88
66.71 # 10.19
wmace (t)
85.61 2.30
11.00
MosWM.Post
u (t)
99..5 0.34
80.82 0.45
wmace (t)
100.00 :: 0.00
78.10
9.33
DtWM.Pre
u (t)
90.31 0.27
79.93 0.37
RSD (1)
100.00 0.00
100.00 0.00
PosT is effective,
Post
u (t)
No change
No change
pval (4)
< 0.05
0.05
DitPrrv.Ia
u (f)
86.82 0.11
74.07 0.25
dp (4)
1.36
<dp 2,89
7.2 Accuracy: using Prior Work
GpPair.In
u (f)
76.85 1.99
equdds (4)
2.81
Before empirically evaluating 48 defense combina- tions, we first identify the combinations which have been empirically evaluated in prior work (54 and Table 5). We identify eight combinations (C1-C8) whose results can be used as ground truth to compare the predictions of DEF CoN and the naive technique (marked or E in Table 5). We use green and red to indicate alignment. and conflict among defenses, respectively.
Expl.Post
u (t)
No change
No change 0.50 + 0.05
ore (4)
0.12 + 0.03
14

---

Published in Transactions on Machine Learning Research (08/2025)
The prediction from a technique is accurate when A (or A) for DEF CON, vp (or v for naive technique, match = (or E) taken from prior work ($4) as ground truth. We present additional details to make predictions in S2, S3, and S4 using DEF CON in Table 4.
C1 (GpPair.Pre + DiffPrrv.PPe .rre) can be combined effectively in the pre-training stage (=) (Yaghini et al., 2023). Naive technique predicts v (same stage) while DEF CoN predicts A (defenses make local changes in S2).
C2 (EvsnRob.In + (Fngrprnt.Post can be effectively combined (=) (Szyller & Asokan, 2023). Naïve
technique predicts V (different stages) while DEF CON predicts A (S3=no).
C3 (DiffPriv.In + Fngrprnt.Post) can be effectively combined (=) (Szyller & Asokan, 2023). Naïve
technique predicts V (different stages) while DEF CoN predicts A (S3-no).
C4 (MdIWM.. Pre +.EvsnRob.In) are not effectively combined (=) (Szyller & Asokan, 2023). Naïve
technique predicts V/ (different stages) while DEF (CoN predicts á (EvsNROB.IN makes poisons ineffective via unintended interaction in S4).
C5 (DtWM.Pre +EvsnRob.In) cannot be effectively combined (=) (Szyller & Asokan, 2023). Naive technique predicts V (different stages) while DEF (CON predicts A (EvsnRoB.I) IN makes poisons ineffective via unintended interaction in S4).
C6 (MdIWM.Pre + DiffPriv.In cannot be effectively combined (=) (Szyller & Asokan, 2023). Similar to C5, DEF CON predictions this as A while the naive technique predicts V,
C7 (DtWM.Pre + DiffPriv.In) can be effectively combined (=) (Szyller & Asokan, 2023). Naïve DEFACON
lechnique predicts (different stages) while DEF CoN predicts A (DiFFPRIV.IN reduces effectiveness of poisons via unintended interaction in S4). Unlike backdoor-based watermarks used in our work, adversarial example-based watermarks used by Szyller and Asokan (Szyller & Asokan, 2023), are inliers which are not suppressed by DIFFPRIV.IN, IN. Hence, DEF) DEF)CON's prediction differs.
C8 (DiffPriv.In + Expl.Post) can be effectively combined (E) (Patel et al., 2022). Naïve technique predicts v (different stages) and DEF CoN predicts A (S3=no).
Of the eight combinations, Def Con predicts seven correctly, while naive technique pre- dicts four. This gives balanced accuracy of 90% (TP=4, TN=3, FP=0, FN=1) for Def. Con, and 40% (TP=4, TN=0, FP=3, FN=1) for the naëve technique.
Table 4: For each defense, we identify key parame- ters used in evaluating combination effectiveness (Fig- ure 1): type of change in S2 (Global, Local, or None); if defense uses risk in S3 ('Yes" for backdoors or adversarial examples); and if defense protects against risk in S4.
Defense
$2
S3
S4
7.3 Accuracy: via Empirical Evaluation
EvsnRob.In
Global
No Yes
PoisnRob.Tn
Global
No Yes
PoisnRob. Post
Global
No Yes
We now empirically evaluate the remaining, pre- viously unexplored, combinations to obtain the ground truth and then compute the accuracy of the predictions from both techniques. After removing the eight combinations from prior work, we are left with 40 combinations. We also remove ten combi- nations where both defenses are applied during in- training. Here, both DEF CoN and the naive technique predict A and V respectively. To apply both defenses in the in-training phase, they can be modified for an effective combination (marked as T1 in {3). However, this makes it invasive (violates R3). Alternatively, defenses can be combined soquentially (c-g., pre-training on the first defense, then fine-tuning on the second), by alternating the training of both defenses every few epochs. Since, these are standadd approaches to apply existing defenses, we leave comprehensive evaluation of ten combinations as future work. Hence, we are left with 30 combinations (C9-C38) for empérical evaluation.
MdWWM.Pre
Local
Yes No
M.IWM.In
Global
Yes No
M.sWM.Post
Local
No
No
Dr.M.Pre
Local
Yes
No
Engrprat.Post
None
No No
DiftPriv.In
Global
No Yes
GpFair.In
Global
No
No
Expl.Post
None
No
No
Predictions from Techniques. Before evaluating 30 combinations, we denote the defenses as D1 and D2 basod on the order in which they are applied. We obtain predictions from DEF CoN and the naïve techniques, and indicate them as a tuple: (Naive prediction, DEF CCN prediction). These are indicated in Table 5. We use the information in Table 4 to make predictions in S2-S4 for DEF CON.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Table 5: Evaluating combinations (6): For brevity, 'x' in Mctric column is shorthand for Lay. We use (rosp. 4) to indicate if a higher (lower) value is better. For defense effectiveness, we use green when of, is better or equal to "single defense" baseline; orange for better or equal to "single defense" but worse than "no defense"; red for worse than "no defense". For technique predictions, we use symbol A (resp. v) to refer to DEF CON (naive technique) and a color code green (resp. red) to indicate alignment (conflict) among
defenses. D1 and D2 indicate order of applying defenses. (ou is for context).
Combinations
|Metric
PNNIST
UTXPACE
|Combinatious
Motric
PINIST
UTKFACE
Dj: EvsnRob.In
I (f)
90.38
0.22
72.79 0.53
Dj: MrIWM.Pre
(1)
90.18 0.21
79.76 0.63
C9
D2t MdIWWM.Pstt
wmacc (f)
100.00 + 0.00
80.96
7.13
C24
D2: Expl.Post (9, A)
err (1)
0.14 + 0.04
0.02 + 0.03
(V, A)
robacr (L)
80.43+0.85
42.00
0.49
wmace (1)
90.93
0.06
90.96
0.08
Dj: PoisnRob.In
u (t)
89.50 1. 0.21
79.25
1 1.06
Dj: MIIWM.In
u (1)
86.94
0.50
72.16
5.13
C10
Dy: Fngrprnt.Poos
ASR (1)
+ 0.22 <0.05
56.00
12.98
C25
D2: Expl.Post
err (4)
0.19
0.07
0.37
0.18
(V)
pval (4)
<0.05
(V, A)
wmacc (t)
98.24 0.66
97.60
3.54
Dj: PoisnRob.Pore u (t)
84.73 = 1.72
663.70
3.87
D1: DtWM.Pre
(t)
90.01 0.00
79.03 1.10
C11 Da: Fugepent.Post Post
ASR (1)
61.36 =: 23.96
0.02 1: 0.03
C26
Da: Expl,Post
err (1)
0.10 1 0.04
0.54 1: 0.01
(V. A)
pval (4)
<0.05
<0.05
(9. A)
RSD (t)
100.00 + 0.00
100.00 0.00
Dj: EvsnRob.In
u (t)
87.10 = 0.21
773665 + 1.21
Dj: PoisnRob.In
(1)
80.30 0.24
78.71 0.20
C12 Dg: Expl.Post (9)
(3)
0.22 1 0.01
0.15 1 0.04
C27
Dg: Expl.Post
ASR (4)
9.79 1: 0.15
44.35
33.07
robace (1)
79.00 + 0.21
39.27 + 0.68
(9, A)
orr (4)
0.06 + 0.02
0.47 + 0.02
Dj: Gp
u (t)
60.73 L: 3.24
Dj: PoisnRob.
u (1)
84.62 3.56
63.80 1: 3.37
C13 Dy: PoisnRob.Post
ASR (4)
20.21 = 39.90
C22
Da: Expl.Post
ASR (4)
76.11 15.85
0.00 + 0.00
(V, A)
equdds (4)
3.20
(V, A)
err (4)
0.08 0.01
0.15 0.06
Dj: MreWM.Pre
(t)
79.02 0.40
Dj: Fugepenn. D2: Expl.Post
(f)
90.56 0.16
S0.42 0.59
C14 D2: GpFair.In
wmace (f)
98.88 2.13
C29
pval (4)
<0.05
<00.05
(P.
eqodds (1)
0.00 + 0.00
(2, A)
err (4)
0.11 + 0.02
0.50 + 0.03
Dy: GpPair-In
u (t)
76.95 1. 1.94
Dj: DEWM.Pre
u (t)
90.19 0.59
79.80 0.48
C15 Dg: MolWMMPost.
winac (t)
80.95 0.00
C30
Dg: Fngrprnt.Pose (9. A)
pval (4)
<00.0
<20.05
(9, 2)
eqodds
7.87 + 4.72
RSD (1)
100.00 0.00
100.00 0.00
Dj: DrWM.Pre
u (t)
78.97 1.21
D;: DiffPriv.In
(t)
$6.83
0,20
71.62 0.19
C16 D2: GpFair.In
RSD (t)
100.00 + 0.00
CB1
D2: MdIWM.
wmaac (t)
100.00 1: 0.00
79.05
=: 3.81
(V, A)
equdds (4)
0.00 1 0.00
(9, A)
dp (L)
1.36
< 2.80
Dj: Gp
u (t)
78.67 + 1.46
Dj: DrWM.Pre
(1)
90.24
0.29
78.94
0.95
C17 D2: Fugepent.Por
pval (4)
0.68 = 0.21
C32
D2: MdlWM.Post
RSD (t)
100.00 0.00
100.00
0.00
(9)
eqodds (L)
7.46 + 5.43
(9.
wmane (1)
100.00 + 0.00
62.26
3.77
Dy: GpPair.In C18 Dy: Expl.Post (M, A)
u (1)
80.52 1. 0.44
Dj: PoisnRob.Puss
u (1)
85.09
1.94
67.09
1. 2.81
err (L)
0.16 + 0.06
C33
Dg: MdIWM. Post
wmace (1)
100.00
0.00
73.33
8.83
equdds (4)
12.62 4,20
(M, A)
ASR (4)
24.91 59.48
40.20
22.82
Dj: PoisnRob.In
u (t)
89.33 0.36
79.00
0.56
Dj: DtWM.Pre
(t)
90.31
0.27
78.53
1.75
C19 D3: MdlWM.Post
winaee (f)
100.00 0.00
69.52
6.46
C344
Da: Mre
wmaac (t)
99.96 0.0
100.00 1: 0.00
(V, 2)
ASR (1)
10.48 = 0.16
38.90 38.90-
= 38. 73
(9, A)
RSD (t)
100.00 + 0.00
100.00 : 0.00
Dj: MosWM.Post Dg: Expl.Pust
u (1)
90.93 = 0.18
[80.53 = 0.23
Dj: EvsnRob.In
u (1)
90.39 + 0.63
80.28 + 0.39
C20
winace (t)
100.00 0.00
72.38
3.56
C35
Dg: PoisnRobb.ost
robacc (1)
46.00
1.02
0.00
0.00
())
(J)
0.11 + 0.02
0.55 + 0.02
(9.
ASR (1)
79.68
10.25
0.00 0.00
Dy: DrWe
u (t)
69.46 1. 0.32
79.00
0.67
Dj: MdIWM.Pre
(t)
89.48
0.15
79.20
0.60
C21
Dy: PoIsNRoa. IN
ASR (1)
10.18 = 0.40
77.39
= 35.
C996
Dz: PoisnRob.In
ASR (4)
10.18
= 0.46
16.92
=: 36.92
(V, A)
RSD (t)
0.00 1 0.00
80.00
40.00
(9, A)
wanace (1)
10.18
0.46
46.92
366.92
D;: DrWM.Pre
u (t)
84.45 = 0.56
79.88 = 0.27
D;: MdlWM.Pre
(t)
82.86
4.16
64.09 = 3.09
C22
Dy: MdIWM.In
wmacc (t)
89.25 3:48
99.98
0.03
C37
D2: PoisnRob.Postt
ASR
71.32
14.11
0.00
0.00
(R. A)
RSD (t)
100.00 + 0.00
100.00 + 0.90
(9. A)
wmacc (t)
71.31
14.10
0.00
+ 0.00
D;: DrWe.Pre
u (1)
82,90 1 2.06
[69.02 1. 1.96
Dj: MdlWM.In
u (1)
66.68
9.80
73.69 1. 3.01
C23 Dg: PaisnRob. Post
ASR (4)
64.55 21.23
0.01
4 0.01
C38
Dg: PoisnRob.P
ASR (1)
58.59
19.22
90.60
0.37
(9, A)
RSD (t)
80.00 40.00
20.00
40.00
(9. A)
wmace (1)
58.65
19.23
90.73
0.29
For defenses applied in the same stage (S1=yes), the naïve technique predicts V. We have the following cases to determine the prediction from DRFF,CON:
1. D2 makes fo changes (S2-no), DEF Con predicts this as V. We mark them as (V, A) which include C11, C20, C28, C29, C33, and C34.
2. D2 makes global changes (S2=yes), and DEF CoN predicts this as A. We mark them as (V, A) but we did not observe any such combinations.
16

---

Published in Transactions on Machine Learning Research (08/2025)
For defenses applied in different stages (S1=no), the naive technique predicts V. We have the following cases to determine the prediction from DEF) CON:
1. D2 does not use a risk (S3=no) and hence, D1 and D2 do not conflict. We mark them as (V, A): C9, C10, C12, C13, C15, C17-C19, C27, C31, and C35.
2. D1, such as PRE and DTWM.PRE, uses a risk (S3=yes), but D2 does not protect against this risk (S4=no). Hence, there no conflict and we mark such combinations as (V, A) which include C14, C16, C22, C24-C26, C30, and C32.
3. DI; such as MDI.WMPRR and DrWM.PRR, uses risk (S3=yes), and D2 mitigates these risks (e.g.. PoISNROB). There is conflict and mark them (v, A) which include C21, C23, C36-C35
We evaluate the 30 combinations on FMNIST and UTKFACE (Table 5). For each combination, we compare the effectiveness metrics for each defense to "single defense" from Table 3. We use green to indicate that the metrics are better or similar to "single defense"; orange for worse than single defense but better than "no defense"; and red for similar or worse than "no defense". Metrics marked as orange can still be useful since it provides some protection compared to "no defense". We consider the worst case by a treating a combination as a conflict if atleast one dataset has atleast one metric marked as orange or red.
Of the 30 combinations, Cef predicts 27 correctly, while the naive method predicts only 18. This gives balanced accuracy of 81% (TP=22, TN=5, FP=3, and FN=0) for Def Con compared to 36% (TP=16, TN=0, FP=8, and FN= for the naive technique.
Takeaway: By explicitly accounting for reasons underlying conflicts among defenses, DEF Conn achieves higher accuracy than the naive technique (satisfies R1).
7.4 Hyperparameter Tuning for Combinations
Table Configurations for hyperparameter luning of defenses in conflicting combinations.
We check if hyperparamcter tuning can resolve con- flicts to see if it can turn (i) false positives (pre- dicted as aligned, but empirically conflicting) into true positives, and (ii) true negatives (predicted and confirmed as conftict) into false negatives. We exclude false negatives (not observed in our evalu- ation), and true positives, correctly predicted and confirmed as aligned (cannot be improved further with hyperparameter tuning). We use grid search and identify various hyperparameter configurations for defenses conflicting combinations (Table 6).
Defense
Hyperparameter
Values
EvenRob.In
Regularization
{2, 4, 6 (default), 8}
PoisnRob.Poo
Pruming threshold
0.6 (step 0.05)
Trigger size Watermark fraction
{3x3, 5x5 (defannt)) {u.1 [default), 0.2, 0.3)
MdIWM.Pre
Watermarkc fraction
{0.1, 0.2, 0.33
MdIWM.In
Training noise
{u.5. 0.73, 1.0 (default). 1.25}
Noise step size
{0.05 (default), 0.10, 0.15)
MdlWM.Post
Watermark fraction
f0.002 (defaauls) 0.01, 0.02).
Trigger size Watermark fraction
[3x3, 5x5 (default))
DtWM.Pre
[0.1 (defanlt)), 0.2, 0.3)
GpFair.In
Regularization
{0.5, (defauls), 1.5, 2}
Do False Positives turn to True Positives? This includes three combinations (C17, C32, C35), and helps investigale DEP' CoN errors.
Engrpent. Post
Iterations
[25, 30 (default), 75, 100)
# Fingerpriots
{100 (default), 150, 2003
C17 (GpFair.In + Engrprut.Post) We empirically observe a conflict as ENGRPRNT.POSS is in- effective (Øpval > 0.05), and DEr) CoN incorrectly predicts the combination as A in S3. We explore the following hyperparameters: regularization for GPFAIR.IN, iterations, and number of fingerprints for CITAPPNT.POST. For each dataset, we have 36 experiments (=4 x 3). None of the experiments alle- viated the conflict. Following prior work (Szyller & Asokan, 2023), we speculate that FNGRPPNT.POS: is ineffective because it relies the decision boundary, which shifts significantly after applying GPFAIR. IN. C32 (De + MdIWM.Post). The combination is empirically effective for FMNIST but not for UTKFACE where Ówmace is less than the "single defense" " basclinc. DEF CoN incorrectly predicts this combination as A in S4. We explore the following hyperparameters: (i) trigger size and watermark fraction for D.WE.PRE; (ii) watermark fraction for MDLWM.POST. For UTKFACE, we evaluate 18 experiments (= 2 x x 3). One experiment with 3x3 trigger size (DTWM.PRE). 30% watermarks NOTWM.PRE). and 2% watermarks (MDLWM.POST) we get @u = 75.98 + 0.61, ÓRSD = 100.00 + 0.00 ggeen),, ) and Pumacc 70.19 + 4.61 (green). Hence, we remove the conflict and the false positive.
17

---

Published in Transactions on Machine Learning Research (08/2025)
C35 (EvsnRob.In + PoisnRob.Post). Empirically, there is conflict as EVSNROB.IN is ineffective (poor @robnee); and DEF CON incorrectly predicts as A in S4. We vary the regularization hyperparameter (EVsNROB.IN)) and pruning thresholds CrossRoD.PoSTT) None of the experiments removed the conflict. We speculate that the model parameters memorizing poisons and adversarial examples overlap. Thus, pruning a model (to reduce dASk) trained with EvSNROB. IN, also reduces Probace: resulting in a conflict.
Do True Negatives turn to False Negatives? We evaluate hyperparameter tuning for five combinations (C21, C23, C36, C37, and C38).
C21 (DtWM.Pre + PoisnRob.In). We consider trigger size, and watermark fraction for DTWM. For cach dataset, we get six experiments (=2x3). None of them removed the conflict since POISNROB.IN mitigates backdoors for DrWM.PRE:
C23 (DtWM.Pre + PoisnRob.Post).. We tune the same hyperparameters for DTWM. as in we sweep across various pruning thresholds. For cach datasct, we get six experiments (= 2 x 3). For FMNIST, trigger size of 3x3 and 30% watermarks, gives Φu =82.00 + 5.50; ΦnsD= 100.00 + 0.00; øAsn= 59.90 + 12.81. This is marked no conflict (green). However, there is conflict for UTKFACE, making the overall combination conflict.
C21. For POISNROB.
C36 (M + PoisnRob.In). We tune the same hyperparameters for MDLWM.PRE, as DrWM PRE in C21. For each dataset, we have six experiments (=2x3). None of them removed the conflict since POIsNROB.Th mitigates backdoors for MDL.MM.RE..
C37 (MdIWM.Pre + PoisnRob.Post) We tune the same hyperparameters for M.LWM.PRE, as DTWM.PRE in C21. For POIsyROB. POST, we sweep across various pruming thresholds. For each dataset, we get six experiments (=2x3). None of them removed the conflict since PoIsNRoR.PosT mitigates backdoors for MDLWM. C38 (MdIWM.In + PoisnRob. We tune the fraction of watermarks, training noise, and step size for we have 36 experiments (=3x4x3). None of them removed the conflict since POISNROB. mitigates the backdoors for MDLWM.IN.
For PoIsNRoB. we sweep across various pruming thresholds. For each dataset,
Summary. We find that hyperparameter tuning is useful in two combinations (C23 and C32). For C32, we removed the false positive, thereby incrcasing DEF Con's balanced accuracy to 86% (from 81%). For C23, we could remove conflict for one of the two datasets, but the combination was still marked as conflict (no additional false negatives).
Takcaway:
Hyperparameter tuning for conflicting combinations is important to check if it turns (a) false positives to true positives, or (b) true negatives to false negatives.
Factors for Hyperparamcter Tuning Effectiveness. We identify factors affecting the effectivencss of hyperparameter luning and highlight how these factors apply to some conflicting combinations:
Expressiveness of hyperparameters: Tuning is only effective if there are hyperparameters that di- rectly inffuence the confficting interaction. If the interaction is insensitive to changes in a hyperparameter, luning will not resolve the conflict. This could be the reason for hyperparameter luning being ineffective for some combinations (C21, C36, C37, and C38).
Search Space: If the search space is narrow, tuning may never find the optimal configuration to resolve conflicts. Conversely, a sufficiently broad search space increases the chance finding a configuration that decouples the objectives. There possibility that tuning did not work for some combinations as our search space was not broad enough (e.g., C17, C32, C35). Fundamental Incompatibility: Some defenses are fundamentally incompatible and cannot be resolved by tuning. We discuss this further below.
Optimization Landscape: If the loss landscape contains many local minima, tuning may not find the optimal configuration to resolve conflicts. A practitioner can try more sophisticated tuning instead of grid search (e.g., randomized search or Bayesian optimization), to see if it. resolves conflicts.
Limitations, When tuning fails to resolve a conflict, we cannot conclusively mark the combination as a conflict: despite considering a feasible search space in our evaluation, some conflict-resolvngg hyperparameters may have been missed given the vast number of possibilities. Also, when combination is marked conflict
18

---

Published in Transactions on Machine Learning Research (08/2025)
(e.g.: "S4=Yes" for C21, C36, C37, and C38), it suggests empirical incompatiillity among defenses but does not imply conclusive proof or fundumental incompalibilily. Establishing fundamental incompatibility requires a theoretical analysis which is left as future work.
7.5 Other Requirements
Having shown that the naive technique does not perform as well as DEF CON, we discuss how DEF meets the remaining requirements: scalability (R2), non-invasive (R3), and generality (R4).
Scalability (R2). None of the prior works have considered more than two defenses. Since DEF Con allows for applying defenses in three stages of the ML pipeline, it should thcoretically support at least three defenses. To illustrate this, we follow the instructions $5.3 to extend DEF\CON beyond two defenses. We begin with pairwise combina- tions predicted as effective (marked as A in Ta- ble 5), which align with empirical evaluation, and then include additional defenses. We consider five combinations with three defenses cach, which should be effectively combines (marked as A). We report the results in Table and find that is indeed possible to effectively combine three defenses using DEF'CCON, Overall, DEF CON scales more than two defenses (R2). These are illustrative examples to show that DEF Con is scalable to more than two defenses. Our goal was not exhaustive evaluation of the large number of multi-way way combinations, but only to show that effective multi-way combinations- previously unexplored are possible. comprehen- sive evaluation to identify false positives and nega- tives, is left future work.
Table 7: Scalability (R2) of Def to D1, D2, and D3 (in order). Color coding and notations are same as in Table 5.
Combinations
|Metric
|RWNIST
UTXFACE
Dy: EvsNRon. IN
(t)
67.38
0.15
74.34 + 0.72
Pg: EXPL.PoST
robace (t)
79.37 1 0.29
39.21 1: 0.32
C39
Dy: MoLWM.PosT
(4)
0.96 + 0.11
0.17 0.05
wmace (1)
100.00 0.00
73.33
4: 8.83
Dy: PoIsNRoB.IN IN
u
89.47 0.24
Py: EXPL.PoST
ASR (1)
9.81 +: 0.12
66.74 4: 12.11
C40
Dy: MDLWM.Poss
(4)
0.02
0.52
wmacc (t)
100.00 0.00
77.14 4: 11.82
Di: PoIsNRoB. Posr
u (1)
89.47 1: 0.24
67.04 1: 3.35
C41
Dy: EXPL.POST
ASR (L)
9.81 + 0.12
1,85 3.39
Dg: MrLWM.Post
err (4)
0.06 +: 0.02
0.17 1: 0.10
(t)
100.00 0.00
81.90 + 7.00
Pj: DrWM.PRE
u (1)
77.53 + 1.75
Da: GpPAm.IN
wmace (t)
100.00) + 0.00
C42
Dg: EXPL.PoST
eqodds (U)
0.00 : 0.00
(4)
Dj: DrWM.PRe
79.17 + 0.93
C43
Dg: GPFAIR.IN
RSD (t)
100.00 + 0:00
M.SSW
egodds (3)
0.00 0.00
wmanc (f)
73.33 4: 7.12
Dj: GPPAm.IN
(t)
01.42 + 2.09
Dg: PoisNRos.Pass
eqodds (1)
8.12 : 4.49
C44
Dg: Rxcrr.Poer
ASR (1)
0.13 + 0.25
(1)
0.05 # 0.02
Non-Invasive (R3). DEF CoN extends T2 and hence, inherits the non-invasive requirement. We use existing defenses proposed in the literature without modifying them, and only adapting them to our datasets. In summary, DEF CON satisfies R3.
General (R4). DEF Con identifies a conflict based on (a) relative position of the defenses in ML pipeline, and (b) the mechanisms that underlic them (whether one defense uses a risk that being protected by a later DEF)CCO defense). DEF CoN does not rely on specific defenses. Hence, DEF CON is likely to be applicable beyond the initial set of defenses we used for evaluating For identifying conflicts of a new defense with others, practitioner can first identify its position in the ML pipeline, and determine whether it uses a risk defended by a later defense. Since these are independent of any specific defense, DEF' CoN is applicable for any defense that we can map to DEF CoN's flowchart (Figure 1). Furthermore, we select specific defense implementations based on their availability (see $6.4). However, other implementations can be used and should not effect our conclusions.
work (commonly available defenses from the literature).
Takeaway: CEF) scales beyond two defenses (R2), is non-invasive (R3), and general (R4).
8 Discussion, Conclusions, and Future Work
Note on Model Utility. So far, we have focused only on effectiveness,, examining how combining defenses impacts the effectiveness of each individual defense. An additional pro-requisite for deploying a defense combination is whether it negatively impacts model utility. We can define a defense combination be viable if it is (a) effective and (b) incurs only minimal utility drop compared to lowest of the "single defense"
19

---

Published in Transactions on Machine Learning Research (08/2025)
baseline. In Table 5, observe that all the combinations which DEF CON predicted as effective are also viable. For C15, C27, and C30, the utility is worse than the "single defense" baseline. These were already flagged as ineffective, We did not observe any combinations which are effective but not viable.
Extending DEF D to predict the viability of the combinations is challenging, since it is unclear how a defense impacts model utility. For instance, there can be the following two cases:
For both defenses, if utility is either better or similar to the "single defense" baseline, it is likely that the combination will have acceptable utility.
If the utility degrades for both defenses, the combination is likely to have poor utility and hence non- viable. However, it is also possible (as seen in Table 5) that the utility of the combination does not fall below the "minimum utility for single defenses" baseline, even if some, but not all, constituent defenses fall below their respective "no-defense" bascline. It is unclear what mechanisms account for this phenomenon.
Hence, quantifying the impact of individual defenses on utility is an open problem for several defeuses (e.g., adversarial training (Zhang et al., 2019; Tsipras et al., 2019; Yang et al., 2020; Pang et al., 2022; Raghunathan et al., 2020) and differential privacy (Jayaraman & Evans, 2019; Ye et al., 2023; Papernot et al., 2021; Tramer & Bonch, 2020)), and an area of active rescarch. Viability can be included as a requirement in $5.2, and exlending DEF DBF for viable combinations is left as future work.
Practical Considerations, We discuss the impact on other considerations such as computational cost and latency. The computational cost (for training) and the latency (for inference) are the sum of the costs and latencies, incurred by the constituent defenses when applied individually.
Pre-training, in-training, and some post-training (e.g., pruming a model) defenses incur a reasonable one- time cost, assuming the practitioner has basic resources to train MT. models (e.g., GPUs). These do not have any impact on the inference-time latency.
For some post-training defenses, there is no training cost but incur a per-inference latency for transforming the inputs or outputs.
DEF)CON Real-world Impact. Following the experimental setup in prior work, we evaluated DEF CoN in a lab set- ting and not on real-world models, which was not realistic for our experiments. The real-world impact of successful combinations depends on the individual application. Practitioners have to decide which combi- nation should be applied for a given application: for instance, in credit card approval, robustness, fairness, and explainability are important propertics, while in medical diagnosis, privacy may also be essential.
Learning- based Component for Def Con. We can train model (e.g., decision tree) to predict the type of interaction for a combination. But this requires a lot more data for training (e.g., features covering various defenses and their combinations). than what we can obtain from the eight combinations in prior work. For the current defenses, our heuristic was sufficient to get a reasonable accuracy. As future work, adding a learning-basee component is an interesting direction.
Other Causes Underlying Conflicts. While we identify two possible reasons underlying conflicts among defenses ($3.3), we do not claim this to be complete. There could be other underlying reasons which can be included in DEF) CoN to make it more accurate. One possible reasons could be the a choice of different f--norm distances for some defenses (e.g., EVSNROB.IN, and adversarial example-based MDLWM. DTWM.PRE). Prior works have shown that the objectives of obtaining robustness to different no-m bounds are conflicting (Tramèr & Boneh, 2019). In case of combinations, Thakkar et al. (2023) show that choosing different amount of noise for watermarking and adversarial training can remove a conflict. We leave the exploration of additional reasons underlying conflicts as future work.
and
Other Combination Techniques. Duddu et al. (2024a) (Table 3) systematize unintended interactions among defenses and risks, categorizing them as increasing, decrcasing, or unexplored. An alternative naïve technique could reject combinations where one defense increases the risks mitigated by another. How- ever, this is restrictive and discards several non-couffictnn combinations (e.g., EXPL.POST and MDL,M, Since there are several unexplored interactions in their systematiza- tion, is challenging to applying this naive technique in our context. Hence, this technique is limited to some combinations, and not applicable to all combinations in the current state.
EVSN and FNGEPRNT.
20

---

Published in Transactions on Machine Learning Research (08/2025)
Other Dataset Modalities and Models. We rely two image datasets previously used by prior work to evaluate most of our defenses, making them a natural starting point. Since DEF\CON 's steps are modality- independent, we conjecture that it can be applied similarly to other data types. Evaluating DEF CoN on new modalities requires implementing corresponding defenses and models (e.g., language models for text). Then, our methooooggy can be followed: (a) identifying various risks and corresponding defenses in different stages of the ML pipeline, and (b) use the empirical evaluation of combinations as ground truth to verify the accuracy of DEF CON. This is a substantial undertaking, and hence, left as future work since it may bring out new insights. We tried to extend our existing defense implementations to tabular dataset, and indeed, not all defenses transfer to other data modalities. For example, when adapting our defenses to the CENSUS dataset, only 4 of the 11 implemented defenses were applicable image-specific techniques like poisoning and watermarking do not apply to labular data. This gave two valid combinations: (i) evasion robustness + explanations and (ii) group fairness + explanations (others were excluded due to incompatibility). Both combinations aligned with DEF) 's predictions.
Speculating Combinations with Omitted Defenses. We speculate on the omitted defense combinations from $6: EVSNROB. PRe,, DIFFPRIV.PRe GPFAIR. PRE, and GPFAIR.POST. Since EvsxRoB. targets adversarial examples and makes local changes to Der, we expect its combination with other defenses to behave similar to MDL W.D.EE and PREM. DIFFPRIV.PRE and GPFAIR. PRE make global changes by transforming all data records in Drr and should be applied before other defenses, as we expect them to avoid conflicts. CPFAIR.POST makes global changes in post-training stage to all predictions, and the behavior is likely to be similar to POISNROB POST which also makes global changes to f in post-training stage. Validating these interactions is left for future work. We indicate some steps for future work to validate our speculation. For defenses omitted due to (i) poor effectiveness: further work is required to design more effective defense variants (e-g,, pre-training or post-training evasion robustness). (ii) incompatible datasets: set up experiments for the specific dataset where the defenses work well (e.g., tabular datasets instead of image datasets) and evaluate their combinations. We can then combine with other defenses, and evaluate the combinations. We will release our code to combine new defenses with our existing ones.
Trade-offss among Defenses. The current version of DEF CoN only outputs aligmment and conflict and does not capture tradc-offs. It is not clear how to compare the extent of pains/lossee for different defenses, and there is no uniform metric that works across all defenses. For instance, 5% loss in one defense may be much worse than 10% loss in another, but there is no clear consensus on which is better. This may be application dependent and has to be determined a practitioner. Hence, we chose to leave this as future work.
Summary. ML models must be protected against multiple risks simultancously, requiring effective combi- nation of defenses. We systematize prior work, identify unexplored combinations, and evaluate limitations of prior techniques. Using insights from our systematization, we present a technique, DEF CON, which is more accurate than prior work, does not require modifying defenses, scales to more than two defenses, and applies to various defenses.
Ethical Considerations
We use public datasets and implementations and none of our experiments require IRB approval.
Broader Impact
Protecting ML models from various risks simultaneously is an important problem, especially in high-stakes domains where failures can have scrious socictal consoquences. Our work advances the field of designing trustworthy ML systems by moving beyond individual risk mitigation common in current research to developing techniques to protect against multiple risks. We do not directly address "accountability" in this work, other than providing a way for model owners to assess effectiveness of defense combinations before deploying them. But it can be combined with additional mechanisms designed to ensure accountability in MT. pipelines (such as Duddu et al. (2024b)) for responsible deployment of defenses.
21

---

Published in Transactions on Machine Learning Research (08/2025)
Acknowledgments
This work is supported in part by Intel (in the context of Private AT consortium), and the Government of Ontario (RE011-038). Vasisht is supported by IBM PhD fellowship, David R. Cheriton Scholarship, and Mastercard Cybersecurity and Privacy Excellence Graduate Scholarship. Views expressed in the paper are those of the authors and do not necessarily reflect the position of the funding agencies. We thank Jian Liu (Zheijang University), Cong Wang (City University of Hong Kong), and Scbastian Szyller (Intel Labs) for fruitful discussions this topic.
References
Jan Aalmoes, Vasisht Duddu, and Antoine Boutet. On the alignment of group fairness with attribute privacy. arXiv preprint ar Xrx::2211. 102009 2022.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Llya Mironov, Kunal Talwar, and Li Zhang, Deep learning with differential privacy. In ACM SIGSAC conference on computer and communications securily, pp. 308 318, 2016.
Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into a strength: Watermarking deep neural networks by backdooring. In USENIX Security Symposium, pp. 2018. URL https: / /www. usenix. http://ooorrrreeeeaeasssssstt//e
Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International conference on machine learning, pp. 60 69. PMLR, 2018
Alekh Agarwal, Miroslav Dudík, and Zhiwei Steven Wu. Fair regression: Quantitative definitions and reduction-based algorithms, In International Conference on Machine Learning, pp. 120 129. PMLR, 2019.
Sushant Agarwal. Trade-offs between fairness and privacy in machine learning. In IJCAI 2021 Workshop AI for Social Good, 2021.
Seyed Ali Mousavi, Hamid Mousavi, and Masoud Daneshtalab. Farmur: Fair adversarial retraining to mitigate unfairness in robustness. In Advonces in Databases and Information Systems, pp. 133-145, 2023.
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1505 1551,, 2021.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances in neural information processing systems, 32, 2019.
David Banisar. The right to information and privacy: Balancing rights and managing conflicts. SSRN Electronic Journal, 03 2011.
Arpit Bansal, Ping-yeh Chiang, Michael J Curry, Rajiv Jain, Curtis Wigington, Varun Manjunatha, John P Dickerson, and Tom Goldstein. Certified neural network watermarks with randomized smoothing. In International Conference on Machine Learning, pp. 1450-1465, 2022.
Marco Barreno, Blaine Nelson, Anthony D. Joseph, and J. D. Tygar. The security of machine learning. Mach. Learn., 122)1122-1148, 2010. URL https://doi.or org.//ooooooro..oo -101
Philipp Benz, Chaoning Zhang, Adil Karjauv, and So Kweon. Robustness may be at odds with fairness: An empirical study on class-wise accuracy. In NewrIPS 2020 Workshop on Pre-registration in Machine Learning, pp. 325 342. PMLR, 2021.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3855 3859. TEEE, 2021.
22

---

Published in Transactions on Machine Learning Research (08/2025)
Zhiqi Bu, Ping Li, and Weijie Zhao. Practical adversarial training with differential privacy for deep learning, 2022. URL https://openrevvex: www.:eeeeoerrnidd-a---ttetteeeee
Jacob Buckman, Aurko Roy, Colin Rallel, and Ian Goodfellow. Thermometer encoding: One hot way resist adversarial examples. In International Conference Learning Representations, 2018. URL
https:://penreevie httpp:///ppeereeee..eeeeer/rrrra net/forrur11d-S180---G0
Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R. Varsh- ney, Optimized pre-processing for discrimination prevention. In Advances in Neural Information Processing Systems, volume 30, 2017. URL htttp://prroeeedngge wwwrrps.ccc/ppprrr. htt..capee..illleepppppee/ee//ee fitle//pppaer/201///ee (atGga2dd0444ffffffeeeeeee286663 Papeeeeeeeeeepppepeee.ere pdr.
Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Ipguard: Protecting intellectual property of deep neural networks via fingerprinting the classification boundary. In ACM Asia Conference Computer and Communications Security, pp. 14 25, 2021. URL https://doi org/10.g///..o...o.oo. 11..0...111//1 1144/3433200.333333
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In ACM workshop on artificial intelligence and security, pp. 3-14, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. orXiv preprint Sal:1992.00705. 2019.
L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Classification with fairness con- straints: A meta-algorithm with provable guarantees. In Conference faimesss accountability, and transporency, pp. 319-322 2019.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise explanations of neural networks using adversarial training. In International Conference on Machine Learning, volume 119, pp. 1383-1391, 13-18 Jul 2020. URL https://ppooeeeddngs .mr.nnngs..rr.....mmm..m....mm.m mlr (tres//v119/hhalaaan222
html.
Kamalika Chaudhuri, Claire Montcleoni, and Anand D Sarwate. Differentially private empirical risk mini- mization. Journal of Machine Learning Research, 12(3). 2011.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, lan Mol- loy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXix:1811.03792 2018.
Huiqiang Chen, Tianqing Zhu, Tao Zhang, Wanlei Zhou, and Philip S. Yu. Privacy and fairness in federated learning: On the perspective of tradeoff. ACM Comput. Suro., 56(2), sep 2023. URL https://doi.org org/ 10. 1145/3606011.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regularization. In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceeddigg .dff.
neurips.cc/paper http:////e.//aa///////////////aa
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing.
In International Conference on Machine Learning, volume 97, pp. 1310-1 1120-1120, 09 15 Jun 2019. URL
https://procceeding: wwareiiiinnmmmm........... mwr. wwlr.ppess/v///cchnnnn//
html.
Gabriela F. Cretu, Angelos Stavrou, Michael E. Locasto, Salvatore J. Stolfo, and Angelos D. Keromytis. Casting out demons: Sanitizing training data for anomaly sensors. In 2008 IEFE Symposium on Security and Privacy (sp 2008), pp. 81 95, 2008.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference of Machine Learning, 2020.
Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamic Morgenstern. On the compatibility of privacy and fairness. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pp. 309 315, 2019. URL http://di...og1..111.1..
3323847.
23

---

Published in Transactions on Machine Learning Research (08/2025)
Nilaksh Das, Madhuri Shanbhogue, Shang- Ise Chen, Fred Hohman, Li Chen, Michael E. Kounavis, and Duen Horng Chau. Keeping the bad guys out: Protecting and vaccinating deep learning with ipeg compression. In ArXiv abs/ abs/1705.02900, 02900, 2017.
Emiliano De Cristofaro. An overview of privacy in machine learning, arXiv preprint orXiv:2005.08677 2020.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprinl Xrxiv:1708 nac.1088.0555,, 2017.
Iliss Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, and Alistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In International Conference on Machine Learning, vol- une 97, pp. 1596 1606, 09 15 Jun 2019. URL https::////reee http://pweeaaaaaaaaeeeeeeeeaaaaa html.
mlr wttppe///////iiaatooooollaaa/aa
Jiahao Ding, Xinyue Zhang, Xiaohuan Li, Junyi Wang, Rong Yu, and Miao Pan. Differentially private and fair classification via calibrated functional mechanism. In AAAI Conference on Artificial Intelligence, volume 34, pp. 622 22-6,, 2020.
Bao Gia Doan, Ehsan Abbasnejad, and Damith C Ranasinghe. Februus: Input purification defense against trojan atlacks on deep neural network systems. In Annual Computer Security Applications Conference, pp. 197-912, 2020.
Vasisht Duddu, Sebastian Szyller, and N Asokan. Sok: Unintended interactions among machine learning defenses and risks. IEEE Symposium on Security and Privacy, 2024a.
Vasisht Duddu et al. Laminator: Verifiable ml property cards using hardware-assisted attestations. In CODASPY, pp.
2024b.
Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Inmovations in Theoretical Computer Science Conference, pp. 214 226, 2012. URL https: //doi. //doii.or//10/ w0.//0.////// 1001111122222220 2090255.
Maria S. Esipova, Atiyeh Ashari Ghomi, Yaqiao Luo, and Jesse C Cresswell. Disparate impact in differential privacy from gradient misalignment. In The Eleventh International Conference Learning Representa- tions, 2023. URL
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In ACM SIGKDD international conference on knowledge discovery and data mining, pp. 259 268, 2015.
Julien Ferry, Ulrich Aïvodji, Sébastien Gambs, Marie-José Huguet, and Mohamed Siala. SoK: Taming the Triangle- On the Interplays between Fairness, Interprotability and Privacy in Machine Learning. In http::///al.ccencee//eee htthhhhll-4499333- 2023.
Ferdinando Fioretto, Cuong Tran, Pascal Van Hentenryck, and Keyu Zhu. Differential privacy and fairness in decisions and learning tasks: A survey. In International Joint Conference on Artificial Intelligence, pp. 5470 5477, 7 2022. URL https://doi. 10. 1tt.o/o00//1..29//jaaaaaaa 2022/766. Survey Track.
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In ACM SIGSAC Conference on Computer and Communications Security, pp. 1322-122 1333, 2015. URL https://doi. org/10 10000.115//20 10t..11///1101.2228088
Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi. Pairness-aware ranking in search & recom- mendation systems with application to linkedin talent scarch. In ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2221 2231, 2019.
Alex Gittens, Büilent Yener, and Moti Yung. An adversarial perspective on accuracy, robustness, fairness, and privacy: Multilateerl-tradeefff in trustworthy ml. IEFE Access, 10:120850-12088 2022.
24

---

Published in Transactions on Machine Learning Research (08/2025)
Kathrin Grosse, Praveen Manoharan; Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. ar preprint arXi:: 06280, 2017.
Tianyu Gu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint Xr:: 178.007787 2017.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Comput. Sure., 51(5), aug 2018. URL [https://doi. fttt:s::/doo.orr/1111 10. 1145/3236000 (https://doi http://///o.oo///..//// (1....115///26009)
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations, 2018. URL https: / //openreview //ppearreriee..et/ffoorr/rt/// wtt.://ooummiiiddyyyroo....
Xiaotian Han, Jianfong Chi, Yu Chen, Qifan Wang, Han Zhao, Na Zou, and Xia Hu. Ffb: A fair fairness benchmark for in-processing group fairness methods. arXie preprint arXiv:2306.09/ 2023.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
Jamie Hayes, Borja Balle, and M Pawan Kumar. Learning to be adversarially robust and differentially private. arXiv preprint 220: .02265, 2022.
Fengxiang He, Shaopeng Fu, Bohan Wang, and Dacheng Tao. Robustness, privacy, and generalization of adversarial training. arXiv preprint arXiv:2012.1383 2020.
Ursula Hebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the compputationallyy-eennffflllll masses. In International Conference on Machine Learning, volume 80, pp. 1939 1949-1948, 10 15 Jul 2018. URL https://proceedings.u mml.mngs.........mmmmm....m.mm.m wwwppe///e/eeeeer//eeeeeeeee httpp//eddaa.....pppee//////eeee jonettt-ooonnn188ee Stmll.
Hongsheng Hu, Zoran Salcic, Lichao Sun, Gillian Dobbie, Philip S. Yu, and Xuyun Zhang. Membership inference attacks on machine learning: A survey. ACM Comput. Surv., jan 2022. URL [https://doi. org/10. 0rg/10.11455 35232733 (https:
Shu Hu, Zhenhuan Yang, Xin Wang, Yiming Ying, and Siwei Lyu. Outlier robust adversarial training, arxiv preprint ar XrX::2200..0544 2023a.
Yuzheng Hu, Fan Wu, Qinbin Li, Yunhui Long, Gonzalo Munilla Garrido, Chang Ge, Bolin Ding, David Forsyth, Bo Li, and Dawn Song. Sok: Privacy- preservig data synthesis. arXiv preprint or XXxv:2307.001000 2023b.
Yuzheng Hu, Fan Wu, Hongyang Zhang, and Han Zhao. Understanding the impact of adversarial robustness on accuracy disparity. In International Conference on Machine Learning, volume 202, pp. 13679-13709, 23-29 Jul 2023c. URL
wttpeeeddiniinme.eeeeeeeeeeeeeee .wr. wtt.mmmpree///////////..// pross//202/hu233 html.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearnable examples: Making personal data unexploitable. In ICLR, 2021.
Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi. Improving deep learning interpretability by saliency guided training. In Advances in Neural Information Processing Systems, 2021. URL https: / //openreview.. http:/iwww..eeteeeeeeeeee htt.///rummixxx227///
Matthew Jagielski and Alina Oprea. Does differential privacy defeat data poisoning? International Confer- ence on Learning Representations, 2021.
Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi Malvajerdi, and Jonathan Ullman. Differentially private fair learning. In International Conference on Machine Learning, volume 97, pp. html.
09-15 Jun 2019. URL attps://proccedings.. mlr. wwt..press//////gggeesstttt
25

---

Published in Transactions on Machine Learning Research (08/2025)
Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In USENIX Securily Symposium, pp. 1912,1912, 2019. URL https://www. usenix. wrg/conf/eeeeeeeoooooooooooooooo conference/ usenixsecurity19 /eenxxssoatt/////ppppetattaiaaop
Hengrui Jia, Christopher A. Choquetto-Choo, Varun Chandrasckaran, and Nicolas Papernot. Entangled watermarks as a defense against model extraction. In USENIX Security Sympostum, pp. 1937 1954, 2021a. URL pestps//wwrt
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint MxXi:1905 08619, 2019.
Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song. Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification? In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8239-8247, 2021b.
Faisal Kamiran and Toon Calders. Data pre-processing techniques for classification without discrimination. Knowledge and Information Systems, 33, 10 2011.
Faisal Kamiran, Asim Karim, and Xiangliang Zhang. Decision theory for discrimination-awarr classification.
In IEEE 12th International Conference on Data Mining, pp. 924 929, 2012.
Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. Fairness-aware classifier with prejudice remover regularizer. In Machine Learning and Knowledge Discovery in Dalabuses, pp. 35 35-50, 2012.
Mintong Kang, Dawn Song, and Bo Li. Diffattack: Evasion attacks against diffusion iiffusion-based adversarial purification. Advances in Neural Information Processing Systems, 36, 2024.
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. Preventing fairness gerrymandering: Audit- ing and learning for subgroup fairness. In International Conference Machine Learning, pp. 2564-2572, 10 15 Jul 2018. URL https: httpas:://occeeeeeeee mlr press/v80/ae kearns18a.htee
Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. An empirical study of rich subgroup fairness for machine learning. Conference on fairness, accountabilily, and transparency, pp. 100-109, 2019.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measuring catas- trophic forgetting in neural networks. In AAAT conference artificial intelligence, volume 32, 2018.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- protability beyond fcature attribution: Quantitative testing with concopt activation vectors (tcav). In International conference on machine learning, pp. 2668 2677. PMLR, 2018.
Jon Kleinberg and Sendhil Mullainathan, Simplicity creates inequity: and interpretability. In ACM Conference on Economics and Compulation, pp. 807-808, 2019.
implications for fairness, stereotypes,
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Inter- national Conference on Machine Learning, volume 70, pp. 1885-1894, 06-11 Aug 2017. URL https: / //proceedings wwaiinng..mm......m.mm... www.mmlr.......... wttppsss/o//kooo/aaaatttt html.
Pang Wei Koh, Jacob Steinhardt, and Percy Liang. Stronger data poisoning attacks break data sanitization defenses. Machine Learning, pp. 1-47, 2022.
Narine Kokhlikyan et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint ar xrXi:2209.0078: 2020.
Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In International Conference on Artificial Intelli- gence and Statistics, volume 54, pp. 623 623-631, 20-22 Apr 2017. URL https://proceeddhi mlr v54/kul11 11aa. html.
mwt.mresss/ press/
26

---

Published in Transactions on Machine Learning Research (08/2025)
Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations. In International Conference on Machine Learning, pp.
PMTR, 2020.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness adversarial examples with differential privacy. In 2019 IEEE symposium on security and privacy (SP), pp. 656 672. IEEE, 2019.
Hyungyu Lee, Sachyung Lee, Hyemi Jang, Junsung Park, Ho Bac, and Sungroh Yoon. DAFA: Distance- aware fair adversarial Iraining. In The Twelfth International Conference on Learning Represenlations, 2024. URL
Boqi Li and Weiwei Liu. Wat: improve the worst-class robustness in adversarial training. In AAAT conference artificial intelligence, volume 37, pp. 14982 14990, 2023.
L. Li, T. Xie, and B. Li. Sok: Certified robustness for deep neural networks. In 2023 IEEE Symposium on Securily and Privacy (SP), pp. 1289 1310, 2023a. URL http:://a...ieeeeccttttrtrrte ieeecomputersociety org/10. 1109/SP46215 2023 10179303.
Tang Li, Fengchun Qiao, Mengmeng Ma, and Xi Peng. Are data-driven explanations robust. against. out-of- distribution data? In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 3821 3831, June 2023b.
Y. Li, J. Yang, Y. Song, L. Cao, J. Luo, and L. Li. Learning from noisy labels with distillation. In IEEE International Conference Compuler Vision (ICCV), pp. 1928-1 1936, 2017. URL https://doi. ieeeconputersociety
111./11........... 2017. .211.
Vige Li, Xixiang Lyu, Xingjun Ma, Nodens Koren, Lingjuan Lyn, Bo Li, and Yu-Gang Jiang. Reconstructive neuron pruning for backdoor defense. International Conference on Machine Learning, pp. 19837 19854. PMLR, 2023c.
Yiming Li, Ziqi Zhang, Jiawang Bai, Baoyuan Wu, Yong Jiang, and Shu-Tao Xia. Open-sourced dataset protection via backdoor watermarking, arXiv preprint arXiv:2010.0582) 2020.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transactions on Newral Networks and Learning Systems, 2022.
Gaoyang Liu, Tianlong Xu, Xiaoqiang Ma, and Chen Wang. Your model trains on my data? protecting intellectual property of training data via membership fingerprint authentication. IEEE Transactions on Information Forensics and Security, 17:1024-1037, 2022a.
Haoyu Liu, Fenglong Ma, Shibo He, Jiming Chen, and Jing Gao. Fairness-aware outlier ensemble. ar preprint arXic:2103.09/19 2021.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against. backdooring attacks on deep neural networks. International symposium on research in attacks, intrusions, and defenses, pp. 273 294, 2018.
Wenyan Liu, Xiangfeng Wang, Haikun Zheng, Bo Jin, Xiaoling Wang, and Hongyuan Zha. Mitigating disparate impact on model accuracy in differentially private learning. Information Sciences, 616:108-1 126, 2022b. URL https://ai //wpw..///ww..// sciencedirect.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International conference on machine learning, pp. 6226 6236. PMLR, 2020.
Gilles Louppo, Michael Kagan, and Kyle Cranmer. Learning to pivot with adversarial networks. Advances in neural information processing systems, 30, 2017.
Andrew Lowy, Devansh Gupta, and Mcisam Razaviyayn. Stochastic differentially private and fair learning. In International Conference on Learning Representations, pp. 86-119 PMLRR 2023.
27

---

Published in Transactions on Machine Learning Research (08/2025)
Nils Lukas, Yuxuan Zhang, and Florian Kerschbaum. Deep neural network fingerprinting by conferrable adversarial examples. In International Conference on Learning Represenlations, 2021. URL https: //openreview.net// .ntt:/forummiiicvgvvvtt....
Xinsong Ma, Zekai Wang, and Weiwei Liu. On the tradeoff between robustness and fairness. In Ad- vances in Neural Information Processing Systems, volume 35, pp.
2022. URL https:
//proceedings. mewpredddinnnn.uuss...... cttpspccr.ppp.eeepeeeeeeee http:/pappfffeeseaeeeeeee http:///////oo//////oaaaaea/aaaa Paper-Conference. pdf.
Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data poisoning against differentiillly-riivee learners: Attacks and defenses. International Joint Conferences on Artificial Intelligence, 2019.
Gabriel Resende Machado, Eugênio Silva, and Ronaldo Ribeiro Goldschmidt. Adversarial machine learning in image classificaatin: A survey toward the defender's perspective. ACM Comput. Sure., 55(1), nov 2021. URL lattps://dot orggp/ http:/.........11485111111111 1145/3485133] (https: btttpe:/ddiiorgg/// 111/./..1.//1111183883333 10.111148//11333
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Represenlations, 2018. URL https://openreview wtt.:eetooooummiddre........
Pratyush Maini, Mohammad Yaghini, and Nicolas Papernot. Dataset inference: Ownership resolution in machine learning. In International Conference on Learning Represenlations, 2021. URL https: / //openreview wttt:ooormiiddddddrr.2....
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey on bias and fairness in machine learning. ACM Comput. Surv., 54(6), jul 2021. URL https://do..orrree
1145/3457607.
Husscin Mozannar, Mesrob Ohannessian, and Nathan Srebro. Fair learning with private demographic data. In International Conference on Machine Learning, pp. 7066 7075. PMLR, 2020.
Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Fairness through robustness: Investigating robustness disparity in deep learning. In ACM Conference Foirness, Accountability, and Transporency, pp. 466-477, 2021.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning (ICML), 2022.
M. Noppel and C. Wressnegger. Sok: Explainable machine learning in adversarial environments.
In 2024 IEEE Symposium on Security and Privacy (SP), pp. 21 21, 2024. URL https://doi.
ieeecomputersocietyy org/10. 1109///51999//92263... 22224442262222....... 2024.0002.
Tribhuvanesh Orekondy, Berni Schiele, and Mario Fritz. Knockoff nets: Stealing functionality of black-box models. In IEEE/CVF conference computer vision and pattern recogmition, pp. 4954-495 1444-493,, 2019.
Deepak P and Savitha Abraham. Fairlof: Fairness outlier detection. Data Science and Engineering, 6, 12 2021.
Tianyu Pang, Min Lin, Xiao Yang, Jun Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. In International Conference on Machine Learning, pp. 17258-17277. PMLR, 2022.
Nicolas Papernot, Martín Abadi, Úlfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. In International Conference on Learning Representations, 2017. URL attps://openreview httpeaarrrrieeett//oouuuaaaaaaa net//forum?dd--kwooSPPP
Nicolas Papernot, Patrick McDanicl, Arunesh Sinha, and Michael P. Wellman. Sok: Security and privacy machine learning. In 2018 IEEE Europeon Symposium on Security and Privacy (EuroSHP), pp. 399 999-414, 2018.
28

---

Published in Transactions on Machine Learning Research (08/2025)
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Ülaar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy. In AAAT Conference on Artificial Intelligence, volume 35, pp. 9312 9321, 2021.
Neel Patel, Reza Shokri, and Yair Zick. Model explanations with differential privacy. In ACM Conference on Fairness, Accountability, and Transparency, pp. 1895-1904, 2022. URL https://doi. org/ 111/////114////// 3531146 0031146.220 3553135..333235.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making doep neural works robust. to label noise: A loss correction approach. In IEEE conference on compuler vision and pottern recognition, pp. 1944-1952, 2017.
Andrea Paudice, Luis Muñoz- González, Andras Gyorgy, and Emil C Lupu. Detection of adversarial training examples in poisoning attacks through anomaly detection. In arXiv preprint ar arXiv:1802.0300 2018.
Andrea Paudice, Luis attacks. In ECML PKDD 2018 Workshops: Nemesis 2018, UrbReas 2018, SoGood 2018, IWAISe 2018, and Green Data Mining 2018, Dublin, Ireland, September 10-14, 2018, Proceedings 18, pp. 5-15. Springer, 2019.
and Emil C Lupu. Label sanitization against label flipping poisoning
Zirui Peng, Shaofeng Li, Guoxing Chen, Cheng Zhang, Haojin Zhu, and Minhui Xue. Fingerprinting deep neural networks globally via universal adversarial perturbations. In LEEE/CV //V//VFFFV Conference on Computer Vision and Pattern Recognition, pp. 13430-134399 2022.
Dana Pessach and Erez Shmueli. A review fairness in machine learning. ACM Comput. Surv., 55(3), feb 2022. URL [https: //doi::///o.... org/ / 10. 100..145500 1144/4494467 fhttps://dof org/10. 11110.114/220 1145/3494672).
Hai Phan, My T Thai, Han Hu, Ruoming Jin, Tong Sun, and Dejing Dou. Scalable differential privacy with certified robustness in adversarial learning. In International Conference on Machine Learning, pp. 7683-7694. PMLR, 2020.
NhatHai Phan, Minh Vu, Yang Liu, Ruoming Jin, Dejing Dou, Xintao Wu, and My T Thai. Heterogeneous gaussian mechanism: Prescrving differential privacy deep learning with provable robustness. arXiv preprint arXie:1906.1 01444, 2019.
Maura Pintor, Luca Demetrio, Angelo Sotgiu, Ambra Demontis, Nicholas Carlini, Battista Big- gio, and Fabio Roli. Indicators of attack failure: Debugging and improving optimization of adversarial examples. In Advances Neural Information Processing Systems, volume 35, pp. 23063-23076, 2022a. URL https://proceeddngss www.ddiinnnn..nn....o.e. wtwpripp.ccppppp.rrrp pow.rpps.c/ppprr...e/////p/////p (iff665f114B099941433330000-0300
Maura Pintor, Luca Demetrio, Angelo Sotgiu, Marco Melis, Ambra Demontis, and Battista Biggio. secml: Secure and explainable machine learning python. SoftwareX, 18:101095, 2022b. URI https://www.
sciencedirect. http:///wnnerrrrreee////////////
Carlos Pinzón, Catuscia Palamidessi, Pablo Piantanida, and Frank Valencia. On the incompatibility of accuracy and equal opportunity. Mach. Learn., 11355):2405-2484, may 2023. URL http::://oo..org//// 1007/810994-02-0633311
Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. On fairness and cali- bration. Advances in neural information processing systems, 30, 2017.
Han Qiu, Yi Zeng, Shangwei Guo, Tianwei Zhang, Meikang Qiu, and Bhavani Thuraisingham. Deepsweep: An evaluation framework for mitigating dnn backdoor attacks using data augmentation. Tn ACM Asia Conference Computer and Communications Security, pp. 363 377, 2021. URL https://doi.orgee org/10. 1145/3433210 3453108.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Perey Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXio preprint ar Xi::2000 10716, 2020.
29

---

Published in Transactions on Machine Learning Research (08/2025)
Celvesre-AAlise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935 29948, 2021.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Radioactive data: tracing through training. In International Conference on Machine Learning, pp. 8326-8335. PMLR, 2020.
Tiago Salvador, Stephanie Cairns, Vikram Voleti, Noah Marshall, and Adam M Oberman. Faircal: Fairness calibration for face verification. In International Conference on Learning Representations, 2022. URL http::///ppuuereeeeeeeee/errrrrr
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization, In IEEE International Conference on Computer Vision (ICCV), pp. 618-626, 2017.
Zeyang Sha, Xinlei He, Pascal Berrang, Mathias Humbert, and Yang Zhang. Fine-tuning is all you need to mitigate backdoor attacks. arXiv preprint
2022.
Shubhranshu Shekhar, Neil Shah, and Leman Akoglu. Fairod: Fairness-oware outlier detection. In AAAI/ACM Conference on AI, Ethics, and Society, pp. 210222222200 220, 2021.
Avanti Shrikumar, Peyton Greenside, and Aushul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning, pp. 3145 3153. PMIR, 2017.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. ArXiv, abs/1 2os/1706.03825, 2017. URL https://api http::///apsssamtiii...mmm sewanti caschlaa.... org/CorpusII: 116955778
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In International Conference on Learning Representations, 2018. URL https:/ http://meeeeeeeeeeeeoeoodeed/dda
Haipei Sun, Kun Wu, Ting Wang, and Wendy Hui Wang. Towards fair and robust classification. In IEEE European Symposium on Security and Privacy (FuroS&PP, pp. 356-376, 2022.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic altribution for deep networks. In International Conference on Machine Learning, pp. 3319 3328, 2017.
Sebastian Szyller and N Asokan. Conflicting interactions among protection mechanisms for machine learning models. In AAAI Conference on Artificial Intelligence, volume 37, pp. 15179-15187, 2023.
Sebastian Szyller, Buse Gul Atli, Samuel Marchal, and N. Asokan. Dawn: Dynamic adversarial watermarking of neural networks. In ACM International Conference on Multimedia, pp. 4417-4425, 2021. URL https: //doi org/10. 1145/3474085 .01.444088.34459.... 3475591.
Buse Gul Auli Tekgul and N Asokan. On the effectiveness of dataset watermarking in adversarial settings. arXiv preprint reXii:2202.12500 2022.
Janvi Thakkar, Giulio Zizzo, and Sergio Maffeis. Elevating defenses: Bridging adversarial training and watermarking for model resilience. arXiv preprint arXiv:2312. 14260, 2023.
Zhiyi Tian, Lei Cui, Jie Liang, and Shui Yu. A comprehensive survey on poisoning attacks and counter- measures in machine learning. ACM Comput. Surv., 55(8), dee 2022. URL https://dol /ttp:://///oo.o//1..../ 1145/ 3551636.
Reihaneh Torkzadehmahhani, Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2019.
Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. 2019.
30

---

Published in Transactions on Machine Learning Research (08/2025)
Florian Tramèr and Dan Bonch. Differentially private learning needs better features (or much more data). In International Conference on Learning Represenlations, 2020.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adver- sarial example defenses. Advances in neural information processing systems, 33:1633-1644 2020.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In International Conference on Neural Information Processing Systems, pp. 8011 8021, 2018.
Cuong Tran, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. Advances in Neural Information Processing Systems, 34:27555-27563, 2021a.
Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. In AAAI Conference Artificial Intelligence, volume 35, pp. 9932-9939, 2021b.
Cuong Tran, Keyu Zhu, Ferdinando Fioretto, and Pascal Van Hentenryck. Fairness increases adversarial vulnerability. arXiv preprint orXiv:221111883, 2022.
Khang Tran, Ferdinando Fioretto, Issa Khalil, My T Thai, and NhatHai Phan. Fairdp: Certified fairness with differential privacy. arxiv preprint arXiv:2305. 1614:2955.16474 2023.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019. URL https::/openreviee http://ppeeroeooo.ootooooraaaa// not/forrumiideSSyxA3o0l
Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin'ichi Satoh. Embedding watermarks into deep neural networks. In ACM on International Conference on Multimedia Retrieval, pp. 269-277, 2017. URL
https://doi..... w0./ 1111/1177/7711111 30789774
Daniël Vos, Jelle Vos, Tianyu Lú, Zekeriya Erkin, and Sicco Verwer. Differentially-privace decision trees with probabilistic robustness to data poisoning, ar preprint arXiv:2305.1339,, 2023.
Sandra Wachter, Brent Daniel Mittelstadt, and Chris Russell. Counterfactual explanations without open-
ing the black box: Automated decisions and the gdpr. Cybersecurity, 2017. URL https://api. semanticscholar.. wwwpriicccccoa....oooooorrssasis otg/CorpussD:339529..
Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class calibrated fair adversarial training.
In IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8193-8201, 2023.
Emily Wenger, Shawn Shan, Haitao Zheng, and Ben Y Zhao. Sok: Anti-facial recognition technology. In 2023 IEEE Symposium Security and Privacy (SP), pp. 864 8811 IEEE, 2023.
Dongxian Wu and Yisen Wang. Adversarial neuron pruming purifies backdoored deep models. In Advances in Neural Information Processing Systems, 2021. URL https://openrevi// wtt.:/nnnnnoornneee//e htt.fformmmddddddm Adaadddagp
Jiapeng Wu, Atiyeh Ashari Ghomi, David Glukhov, Jesse C Cresswell, Franziska Boenisch, and Nicolas Papernot. Augment then smooth: Reconciling diffcrential privacy with certified robustness. arXiv preprint wwXiv:2306.08650 2023.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. ArXiv, abs/ bbss/180..0639, 2018. URL attps://apy wemanticschollat wwwp:a/pp.g............. http//oopuII::33336////3 33578665.
Shiji Xin, Yifei Wang, Jingtong Su, and Yisen Wang. On the connection between invariant learning and adversarial training for out-of-distribution generalization. In AAAT Conference on Artificial Intelligence, volume 37, pp. 10519 10527, 2023.
Chang Xu, Jun Wang, Francisco Guzmán, Benjamin Rubinstein, and Trevor Cohn. Mitigating data poisoning in text classification with differential privacy. In Findings of the Association for Computational Linguistics, pp. 4348 4356, November 2021a. URL https: /aclanthhool org/2021. findings-emii eetpsss-mmll..33. 369.
31

---

Published in Transactions on Machine Learning Research (08/2025)
Depeng Xu, Shuhan Yuan, and Xintao Wu. Achieving differential privacy and fairness in logistic regression. In World Wide Web Conference, pp. 594-599, 2019a. URL http::::/dooooo///////// 2tt.:/01111///5555033311111111
Depeng Xu, Wei Du, and Xintao Wu. Removing disparate impact of differentially private stochastic gradient descent on model accuracy. Xrviv preprint arXiv:2003.0000 2020.
Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International Conference on Machine Learning, volume 139, pp- 11492-11501, 18-24 Jul 2021b. URL https://proceedii mlr. wttppp/ees////////////
html.
Yilun Xu, Pong Cao, Yuqing Kong, and Yizhou Wang. L d_mi: An information- noisc-robust loss function. Xiviv preprint 15t::199900330 2019b.
Mohammad Yaghini, Patty Liu, Franziska Boenisch, and Nicolas Papernot. Learning with impartiality to walk on the pareto frontier of fairness, privacy, and utility. arXiv preprint 200::330200983,, 2023.
Mohammad Yaghini, Patty Liu, Franziska Boenisch, and Nicolas Papernot. Regulation games for trustworthy machine learning. arXiv preprint arXiv:2402.00 20000.04400 2024.
Fan Yang, Qizhang Feng, Kaixiong Zhou, Jiahao Chen, and Xia Hu Differentially private counterfactuals via functional mechanism. arXiv preprint Xrviv:2208.00 02878, 2022.
Yao- Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. Tn Advances in Neural Information Processing Sys- tems, pp. 8588 8601, 2020. URL https: p/troeed//occe nwwripe.errppccomm httprrppsccppppeeffileee///////e filee///appprr/00///ieee C14777652C77718864444444438 c3dcff3bacddccfaaacao Paper.
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu, Reza Shokri, and Volkan Cevher. Initialization matters: Privacy- utility analysis of overparameterized neural networks. In Advances in Newral Information Processing Sys- tems, volume 36, pp. 5419-5446, 2023. URL http:://ppooceddng..eeeexx neurips.......... wwturps..c/papppeee ppappr_filespppeeepppppppppeeeep
Patp://peeee000--aaaa--r-oerreee .ddf.
Ashkan Yousefpour et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109. 12298, 2021.
S. Yun, D. Han, S. Chun, S. Oh, Y. Yoo, and J. Choe. Cutmix: Regularization strategy to train strong classi- fiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6022-603 6031, 2019. URL https://dof: www.ecccmmpteessoccttyycooo 1r00:///00///1111/////////
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P. Gummadi. Fairness con- straints: A flexible approach for fair classification. Journal of Machine Learning Research, 20(75):1-42, 2019. URL http://jmlr wwwpoopprg//////e/.oeae. http:/pppps///////// 10////0///22222 html.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In International Conference on Machine Learning, volume 28, pp. 325-333, 17-19 Jun 2013. URL https: / //proceedings...
waweeediiinmmm.......mmmmm.meeee mlr. www.sereesseeeee press//28/zeeel11 zemel13. htmml.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In AAAT/ACM Conference on AI, Ethics, and Society, pp. 335-340, 2018a.
Hongjing Zhang and Ian Davidson. Towards fair deep anomaly dotection. In ACM conference on fairness, accountability, and transporency, pp. 138-148,17 2021.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theo- retically principled tradc-off between robustness and accuracy. In International Conference on Machine Learning, volume 97, pp. 7472-7482, 09-15 Jun 2019. URL https://proceedings mwr. alr.prees/v97/ zhang19p.htr html.
32

---

Published in Transactions on Machine Learning Research (08/2025)
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical
risk minimization. In International Conference Learning Representations, 2018) 10.8b. URL https://
openreviiew.et/forumee 1et/foruu?iderrD01111111
Jialong Zhang. Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Ph. Stoecklin, Heqing Huang, and Ian Molloy. Pro- tecting intellectual property of deep neural networks with watermarking. In Asia Conference on Computer and Communications Security, pp. 159-172, 2018c. URL
Tao Zhang, Tianqing Zhu, Kun Gao, Wanlei Zhou, and S Yu Philip. Balancing learning model privacy, fairness, and accuracy with early stopping criteria. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Yuan Zhang and Zhiqi Bu. Differentially private optimizers can learn adversarially robust models. ar.ivv preprint XrXi::2211.00 088444,, 2022.
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitz- ness. In European Conference on Computer Vision, pp. 175 775-191. Springer, 2022a.
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Pre-activation distributions expose backdoor neurons. In Advances in Neural Information Processing Systems, 2022b. URL
1k1ijW
Tianhang Zheng and Baochun Li. Differentially private dataset condensation. In AISCC Network and Distributed Systems Security, 2023. URL
Yue Zheng, Si Wang, and Chip-Hong Chang, A dnn fingerprint for non-repudiable model ownership identi- fication and piracy detection. IEEE Transactions on Information Forensics and Security, 17:2977-2990 2022c.
Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. In seérth- Conference Newral Information Processing Systems, 2023. URL https://openrevierr httpr/iwwwweeefforrtneeeeee net:foormmid
A Summary of Defenses and Notations
We summarize the different defenses and their impact du from $3.1 in Table 8.
B Formal Analysis of Def Con Con
A defense is defined as tuple D (S, P)P, where:
S is the stage where D is applied
C' is the type of changes
R is the risk it uses
P is the protection scope of D, and 0 o P.
Let D be the set of all such defense. The input set for DEF CoN is defined as: X {(1, D2) # D2, and D1, D2 D}. DEF CEF can be defined as function: f X {0, 1} where f(D,, D2) indicates conflict and [(D,, D2) = 1 indicates alignment. Given the above formal model, will discuss consistency, soundness, and completeness.
Consistency means that DEF CoN never produces contradictory outputs for the same input pair. Claim 1. There is no pair D2) for which DEF Con simullaneeussy classifies both conftict and aligmment, i.e. DEF Con is consistent.
Proof. According to the definition of f, There are two rules conflict. cases:
33

---

Published in Transactions on Machine Learning Research (08/2025)
Table 8: Summary of defenses. (Column Ou indicates impact on utility: "v" (decrease), "~" (no effect); "A" + (increase).)
Defense
Iou Re.RReefencess
EvsuRob (Evasion Robustness) EvssIton. (Data Angmentation)
Yun (2019); Zhang et al. (2018b); DeVries Taylor (2017); Madry al. (2018); Rebufii al. (2021)
EN (Advensarial Training)
Zhang et al. (2019); Cohen et al. (2019); Lecuyer et al. (2019)
Nie al. (2022); Song et al. (2018); Guo et al. (2018); Das et al. (2017)
EvsNitos. PosT (Input. Processing)
Grosse al. (2017); Buckman al. (2018)
PoisuRob (Outlier Robustness) PoreNRon.PP PRE (Data Augmentation)
Borgnia (2021); Qiu (2021); Cretu (2008); Paudice al. (2018; 2019); (2021b; 2019)
PoISNRoO.IN IN (Fine-tuning)
Diakonikolas et al. (2019); Xu et al. (2019b); Liu & Guo (2020); Patrini et al. (2017); Zhu et al. (2023); Liu et al. (2018); Wa & Wang (2021); LI et al. (2017)
PoIsNRoB. POST (Pruning)
Zheng al. (2022b:a); Li (2023c)
MdIWM M)ttermarking-))
MDI WM.PRS (Backdoors)
~ Adi et al. (2018); Zhang et al. (2018c); Jin al. (2021a); Uchida et al. (2017)
MELWE WMLI (Optimization)
V Bansal et al. (2022) Soyller at al. (2021)
Mor PosT (APl-basod)
Fugrpeat (Fingerprinting)
ENGEPRNT. (Fingerprints)
1~ |Cao et al. (2021); Peng et al. (2022); Lukas et al. (2021); Zheng et al. (2022c); Maini et al. (2021)
DLWM (Watermarking-D)
DrWM.PRE (Backdoors)
J Tekguul & Asokan (2022); Sablayrolles al. (2020); Liu al. (2022a)
DivfPriv (Differential Privacy)
DiePPRiv.PMe (Private Data)
Xie et al. (2018); Torkzadehmahani et (2019)
DIFEPRIV.IN IN (DPSGD)
Abadl (2016); Papernot al. (2017)
GpFair (Group Fairness)
GFFAIR PRE (Fair Data)
Kamiran Calders (2011); Calmon (2017): Zeme] (2013); Feldman et al. (2015)
GPFAIR.IN (Regalarization)
Colis et al. (2019); Kearns et al. (2018; 2019); Agarwal et al. (2019; 2018); Zhang et al. (2018a);
Kamishima et al. (2012)
GPEAIR. POST (Calibration)
Pleiss et (2017); Hards al. (2016); Kamiran al. (2012); Geyile (2019)
Expl (Explanations)
Post. (Attributions)
~ Isumil et (2021)s Smilkov et (2017); Sundaramsja (2017); Koh Liang (2017);
Wachter et al. (2017); Selvaraju al. (2017); Kim al. (2018)
(c.1) S: = S2 and C1 = Global and C2 = Global, then f() O. (c.2) S1 # S2 and Rí E P2, then f((Di, = O.
For alignment case: (a.1) If none of the conflict conditions hold, then D2) 11
These conditions partition the input space X into disjoint sets: conflict or alignment, with no overlap. Assume that there exists X; such that f(X,) = and f(X,) 1. Since f(X,) 1, according to (a.1), (c.1) and (c.2) do not hold, leading to contradiction. Therefore, DEF CCn never produces contradictory classifications for the same inpul and is consistent.
Soundness ensures that if DEF Con predicts a combination as effective (aligned), it should indoed be effective practice.
This assumption is supported by our empirical evaluation, where DEF CoN achieves:
90% accuracy on previously studied defense combinations,
81% accuracy on novel, unexplored combinations.
The low rates of false positives and false negatives in these evaluations suggest that DEF CoN can reliably predict combination effectiveness, providing practical evidence for its soundness. perfectly sound technique would rely on comprehensive empirical evaluation which we want to avoid by proposing an easy-to-use (approximate) technique, DEF DEF\CON: CON. Therefore, the soundness of DEF CoN conditional on the assumption that these rules perfectly represent all conffict and alignment scenarios: (c.1) (c.2) and (a.1). However, since real-world defense interactions be complex, involving subtle dependencies and emergent behaviors.
34

---

Published in Transactions on Machine Learning Research (08/2025)
DEF CoN may not fully captured reasons for conflict alignment. Hence, DEF CoN is not perfectly sound and is likely to incur some errors.
Completeness suggests that D can classify every possible defense pair correctly as either conflict or alignment. Similar to soundness, completeness is conditional on the assumption that the classification rules fully capture all conflict and alignment cases. As partial theoretical guaraniee, we can prove that DEF CON can classify every defense pair in its input domain, i.e,, it produces classification for every pair without leaving any case unclassified. Assuming that the imput set X includes all possible distinct defense pairs, DEF CON partially satisfies completeness by design. Formally,
Claim 2. For every pair (D1, X, ReFl'CoN's function f produces classification f(D1,D2) E (0,1) that identifies whether the defenses conflict or align, i.e., conflict aligument case is left unclassified. Therefore, if X includes all distinct combinations of defenses, DEF CoN is partially complete.
Proof. By construction, DEF CON 's classification rules partition the input space X exhaustively and exclu- sively by (c.1) (c.2) and (a.1). These rules cover all possible defense pairs in X without overlap or ambignity, ensuring that every pair is assigned a unique classification, DEF CoN is partially complete on X.
The assumption that X includes all possible defense combinations is supported by the fact that all surveyed defense methods can be represented within the DEF CoN framework.
C Formal Analysis for Multi-way Combination Algorithm
The algorithm is for multi-way combinations and itcrates through all permutations of defenses, and uses DEF Con to check for pairwise conflicts:
(m.1) For defenses D. D, I,..., n, F(D.....,,,) = DD, D. f(D. 0.
Consistency. According to the consistency of f, assume that there exists D,, i 1,..., n, s.t. F(D) 0 and F(D,...., D..) 1. Since F(DI...., Dr) 1, VD;, Dj, we have [(D;, Dj) = 1, leading to contradiction. Therefore, F' is also consistent.
Soundness and Completeness. Under the assumption that f sound and complete, and considering that we take all possible combinations of defenses (as there no limit on the input defense set of F), the soundness and completeness of is conditional on the assumption that (m.1) fully captures all conflict and alignment cases.
Soundness. Since f has some errors and is not perfectly sound, multi-way combinations indeed introduce conflicts, which will result in false positives or negatives. Hence, the soundness of of multi-way DEF), Con requires comprehensive ioaluation similar to pairwise evaluation. In our evaluation, we only want to show that effective multi-way way combinations previously unexplored- are possible. comprehensive evaluation to check for soundness left as future work.
35


Reviews:
[{"review_id": "G3gcAd7WAQ", "reviewer": "Reviewer_yMcG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The authors provide an extensive survey on the used defenses for the ML models, their possible combinations & interactions, and the evaluation of these defenses. Additionally, unexplored combinations are also evaluated.\n\n* The manuscript presents DEF\\CON technique which provides proper principles on how to combine ML defenses.\n\n* The manuscript evaluates all possible combinations of defenses constructed with respect to DEF\\CON. \n\n* The effect of hyperparameter tuning on ML defenses is investigated.\n\n**Weaknesses**\n\n* Experiments are limited to image datasets. Since these defenses are not restricted to the domains, what would be their performance for different domain datasets such as text or tabular data?\n\n* Conflicts are treated as binary classifications, conflict (not effective) and alignment (effective). However, what would be the case for tradeoffs? For the proposed technique, combining two defenses with a little tradeoff on one defense to gain high benefit on another will not be applied since they conflict. Yet, such a combination may be more effective than the other combinations. \n\n* How would DEF\\CON be applied automatically? The impact and the stages of the defenses require manual labelling.\n\n* DEF\\CON is only compared against the mutually exclusive stage rule. The performance comparison seems to be insufficient.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript focuses on the protection of the ML models. The authors first survey the existing work on the possible risks concerning the ML models and the defense techniques used to protect the models. Then, they move onto the combination and interaction of the used defenses. Finally, the authors present the novel defense combination technique called DEF\\CON.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All the concerns mentioned above should be addressed to improve the paper.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UfL9OtDcnp", "reviewer": "Reviewer_HMru", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength：\n1. The paper addresses an important and underexplored problem: how to combine multiple ML defenses without conflicts. The motivation is clear and valuable.\n2. The work is comprehensive and covers a wide range of defenses and interaction types, demonstrating significant effort.\n3. The experiments are thorough, with extensive evaluations on both known and unexplored defense combinations.\n### Weakness:\n\n**Major:**\n\n1. While Def\\Con is rule-based and intuitive, it lacks formal theoretical justification or guarantees (e.g., consistency, soundness, or completeness). Without a formal analysis, it remains unclear whether Def\\Con’s decision rules can reliably generalize beyond the evaluated combinations or under adversarial conditions.\n2. The method evaluates defenses in a pairwise fashion, assuming that higher-order interactions (e.g., between three or more defenses) can be decomposed into pairwise relationships. This assumption is questionable, as some conflicts may emerge only in multi-way combinations due to cumulative effects, shared resources, or emergent behavior. A principled extension beyond pairwise interactions is currently missing.\n3. The empirical validation is limited to two image classification datasets (FMNIST and UTKFACE). This restricts the generalizability of conclusions to other ML settings, such as NLP, generative models, multi-modal models, or structured tabular data, where defense interactions may differ significantly.\n4. The current evaluation focuses only on whether a defense combination maintains its functional effectiveness. However, many real-world applications require careful utility–robustness trade-offs. Def\\Con does not account for degradation in model accuracy, computational cost, latency, or interpretability that may occur even in “non-conflicting” combinations.\n5. The logic underlying Def\\Con is based on fixed hand-crafted heuristics (e.g., stage matching, global/local impact). These rules are manually designed and may not capture subtle statistical or semantic interactions between defenses. The lack of adaptability or learning undermines robustness in dynamic or unforeseen scenarios.\n6. The paper defines a defense as “effective” if it meets certain metric thresholds, but does not validate whether these combinations lead to improved overall model performance or security in a deployed system. This metric-level evaluation might overestimate the practical value of successful combinations.\n\n**Minor:**\n\n1. Several data tables are incorrectly referred to as \"Figure\" in the main text (e.g., Figure 3 is actually a tabular summary). For clarity and consistency, these should be labeled as \"Table\" following standard academic conventions.\n2. The paper uses too many abbreviations, which makes it difficult to read and follow. Reducing their use and providing a clear notation summary would improve clarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of combining multiple ML defenses (for security, privacy, and fairness) without causing negative interactions. The authors survey existing work, identify gaps, and propose Def\\Con, a rule-based technique to predict whether defense combinations are effective. Def\\Con is accurate, scalable, non-invasive, and general. It outperforms the naïve baseline in predicting effectiveness across 8 known and 30 new defense combinations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the **Weaknesses** section for a detailed discussion of the issues identified in the paper.\n\nTo strengthen the submission and increase its impact, the following key improvements are strongly recommended:\n\n- Provide theoretical justification or formal analysis of Def\\Con’s rules to ensure robustness and generalizability.\n- Extend the method beyond pairwise defense interactions to capture higher-order effects in defense combinations.\n- Broaden empirical validation to include diverse datasets and modalities beyond image classification.\n- Address practical considerations such as accuracy, computational cost, latency, and interpretability alongside defense effectiveness.\n- Consider incorporating adaptive or learning-based components to improve the flexibility and robustness of Def\\Con’s heuristics.\n- Include evaluation or discussion on the real-world security and performance impact of the proposed defense combinations.\n\nAdditionally, minor issues such as inconsistent figure/table labeling and excessive abbreviations should be addressed to improve readability.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GL18FMHdcj", "reviewer": "Reviewer_6Nbb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) Systematic Approach to Defense Combination: The paper offers a systematic survey of existing work on combining ML defenses, classifying them by combination technique and interaction type (conflict or alignment). This systematization is a significant contribution to our field. (2) Identification of Unexplored Combinations: The research identifies previously unexplored combinations of defenses, revealing research gaps and opportunities for new technique design. (3) Introduction of DEF CON: The paper proposes DEF CON, a novel combination technique designed to be accurate, scalable and generalizable. (4) DEF CON provides an inexpensive and fast method for practitioners to determine if existing defenses can be effectively combined without expensive empirical evaluation. This addresses a significant practical need. \n\nWeaknesses:\n(1) While the authors state that their framework is extensible and covers all techniques seen in their systematization , they also acknowledge that the initial list of defenses is not exhaustive. A more explicit discussion of how incompleteness in the initial defense list might impact the broader applicability of DEF CON, beyond the evaluated combinations, would be beneficial. (2)  The paper speculates on the interactions of omitted defense combinations (e.g., EVSNROB.PRE, DIFFPRIV.PRE, GPFAIR.PRE, and GPFAIR.POST). While a full evaluation might be outside the scope, acknowledging the speculative nature and perhaps outlining future work to empirically validate these would be beneficial. (3) A deeper analysis of why tuning succeeded in some cases and failed in others would provide more insight into the limitations of tuning for resolving conflicts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DEF CON, a novel technique for combining machine learning defenses without conflicts, inspired by the limitations of prior methods. The key contribution is a systematization of existing work on defense combinations, identifying how defenses are combined and interact, and revealing previously unexplored combinations. ECAC also improves the accuracy of existing combination techniques. The new knowledge presented is that DEF CON accurately identifies whether a defense combination is effective, scalable to combine multiple defenses and is generalizable across different defense types. The authors demonstrate that DEF CON achieves high accuracy (90% on prior combinations and 86% on empirically evaluated unexplored combinations) in determining effective defense combinations, offering practitioners an inexpensive and fast way to assess combinations without extensive empirical evaluation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Would Strengthen the Work:\n\n(1) Clarification on Framework Completeness and Broader Applicability: While the authors acknowledge their initial defense list isn't exhaustive, a more explicit discussion of how this impacts the general applicability of DEF CON beyond the evaluated combinations would be valuable. This would strengthen the claim of \"generality (R4)\". (2) Addressing Speculative Combinations with Omitted Defenses: The paper speculates on the interactions of certain omitted defense combinations. To strengthen the work, the authors should: \n - More clearly delineate the boundary between empirically validated findings and speculative insights.\n- Outline a more detailed plan for future work to empirically validate these speculative interactions\n(2) While hyper parameter tuning was beneficial in some cases (C23 and C32) and removed a false positive for C32 , many experiments still resulted in conflicts. A more in-depth analysis would be beneficial - Mainly focus on:\n - Explore why tuning successfully resolved conflicts in some cases but not others. What were the distinguishing factors?\n- Provide insights into the inherent limitations of hyper parameter tuning for resolving conflicts, even with DEF CON's guidance.\n - Elaborate on whether the inability to resolve conflicts through tuning for certain combinations indicates a fundamental incompatibility between those defenses that ECAC correctly identifies.", "broader_impact_concerns": "The paper discusses protecting ML models against risks to security, privacy, and fairness. If these combined defenses are deployed in high-stakes or safety-critical applications (e.g., autonomous systems, medical diagnostics, financial decision-making), a Broader Impact Statement should consider - Trust and Accountability.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)