Backdoors Stuck At The Frontdoor: Multi-Agent Backdoor Attacks That Backfire
Authors: Siddhartha Datta
Nigel Shadbolt
Venue: Submitted to ICLR 2023 (2023)

Abstract: Malicious agents in collaborative learning and outsourced data collection threaten the training of clean models. Backdoor attacks, where an attacker poisons a model during training to successfully achieve targeted misclassification, are a major concern to train-time robustness. In this paper, we investigate a multi-agent backdoor attack scenario, where multiple attackers attempt to backdoor a victim model simultaneously. A consistent backfiring phenomenon is observed across a wide range of games, where agents suffer from a low collective attack success rate. We examine different modes of backdoor attack configurations, non-cooperation / cooperation, joint distribution shifts, and game setups to return an equilibrium attack success rate at the lower bound. The results motivate the re-evaluation of backdoor defense research for practical environments.

Under review as a conference paper at ICLR 2023
BACKDOORS STUCK AT THE FRONTDOOR: MULTI-AGENT BACKDOOR ATTACKS THAT BACKFIRE
Anonymous authors Paper under double-blind review
ABSTRACT
Malicious agents in collaborative learning and outsourced data collection threaten the training of clean models. Backdoor attacks, where attacker poisons a model during training to successfully achieve targeted misclassification, are a major concern train-time robustness. this paper, we investigate multi-agent backdoor attack scenario, where multiple attackers attempt to backdoor a victim model simultaneously. consistent backfiring phenomenon is observed across a wide range of games, where agents suffer from a low collective attack success rate. We examine different modes of backdoor attack configurations, non-cooperation cooperation, joint distribution shifts, and game setups to return an equilibrium attack success rate at the lower bound. The results motivate the re-evaluation backdoor defense research for practical environments.
INTRODUCTION
Beyond training algorithms,, the scale-up of model training depends strongly on the trust between agents. In collaborative learning and outsourced data collection training regimes, backdoor attacks and defenses (Gao et al., 2020; Li et al., 2021) are studied to mitigate a single malicious agent that perturbs train-time images for targeted test-time misclassifications Outsourced data collection is common amongst industry practitioners, where Taigman et al. (2014); Papernot (2018) find a strong reliance on web-scraped data or third-party sourcing. Kumar al. (2020) also find that data poisoning their most serious threat. In many practical situations, It is plausible practice for >1 attacker, such as the poisoning crowdsourced and agent-driven datasets on Google Images (hence afflicting subsequent scraped datasets) and financial market data respectively, or poisoning through human-in -nttue-loo-e- learning on mobile devices or social network platforms. In this paper, investigate the under-represented aspect of agent dynamics in backdoor attacks: what happens when multiple backdoor attackers are present? We simulate different configuration to study how the payoff landscape changes for attackers, with respect to standard attack/defense configurations, behaviour, and joint distribution shifts. Our key contributions are:
is viewed by practitioners
We explore the novel scenario of the multi-agent backdoor attack. Our findings on the backfiring effect and low equilibrium attack success rate indicate a stable, natural defense against backdoor attacks, and motivates us to propose the multi-agent setting as a baseline future research.
We introduce set of cooperative dynamics between multiple attackers, extending on existing backdoor attack procedures with respect to trigger pattern generation or trigger label selection.
We vary the sources of distribution shift, from just multiple backdoor perturbations to the inclusion of adversarial and stylized perturbations, investigate changes to a wider scope of attack success.

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Backdoor Attacks. We refer the reader to Gao et al. (2020); Li et al. (2021) for detailed backdoor literature. In poisoning attacks (Alfeld al., 2016; Biggio al., 2012: Jagielski et al., 2021; Koh & Liang, 2017; Xiao et al., 2015), the attack objective is to reduce the accuracy of a model on clean samples. In backdoor attacks (Gu al., 2019a), the attack objective is to maximize the attack success rate in the presence of the trigger while retain the accuracy of the model on clean samples.
To achieve this attack objective, there are different variants of attack vectors, such as code poison- ing (Bagdasaryan & Shmatikov, 2021; Xiao et al., 2018), pre-trained model tampering (Yao et al., 2019; Ji al., 2018; Rakin al., 2020), or outsourced data collection (Gu et al., 2019a; Chen et al., 2017; Shafahi et al., 2018b; Zhu et al., 2019b; Saha et al., 2020; Lovisotto et al., 2020; Datta & Shadbolt, 2022b). We specifically evaluate backdoor attacks manifesting through outsourced data collection. Though the attack vectors and corresponding attack methods vary, the principle is consistent: model weights are modified such that they achieve the backdoor attack objective.
Multi-Agent Attacks. Backdoor attacks (Suresh et al., 2019; Wang et al., 2020; Bagdasaryan et al., 2020; Huang, 2020) and poisoning attacks (Hayes & Ohrimenko, 2018; Mahloujifar et al., 2018; 2019; Chen et al., 2021; Fang et al., 2020) against federated learning systems and against multi party learning models have been demonstrated, but with a single attacker intending to compromise multiple victims (i.e. single attacker vs multiple defenders); for example, with single attacker controlling multiple participant nodes in the federated learning setup (Bagdasaryan et al., 2020); or decomposing backdoor trigger pattern into multiple distributed small patterns be injected by multiple participant nodes controlled by a single attacker (Xie et al., 2020). Our multi-agen backdoor attack could be evaluated extensibly in federated learning, where multiple attackers control distinctly different nodes to backdoor the joint model.
Though not multi aulti-agent attack, Xue et al. (2020); Nguyen & Tran (2020); Salem al. (2021) make use of multiple trigger patterns in their single-agent backdoor attack. Xue et al. (2020) proposed an 1-to-N attack, where an attacker triggers multiple backdoor inputs by varying the intensity of the same backdoor, and N-to-1 attack, where the backdoor attack is triggered only when all N backdoor (sub)-triggers are present. Though its implementation of multiple triggers are for the purpose of maximizing single-agent payoff, reference its insights in evaluating low-distance- triiggge,,,,,,rss,.. cooperative attack in (E4).
Our work is unique because: (i) prior work evaluates single attacker against multiple victims, while our work evaluates multiple attackers against each other and a defender; (ii) our attack objective is strict and individualized for each attacker (i.e. in poisoning attack, each attacker generalized, attacker-agnostic objective of reducing the standard model accuracy, but in a backdoor attack, each attacker has individualized objective with respect their own trigger patterns and target labels). Our work is amongst the first to investigate this conflict between the attack objectives between multiple attackers, hence the resulting backfiring effect does not manifest in existing multi- agent attack work.
have
2

---

Under review as a conference paper at ICLR 2023
3 MULTI-AGENT BACKDOOR ATTACK
3.1 GAME DESIGN
The scope of our analysis is that the multi-agent backdoor attack is a single-turn game, composed of N attackers and M defenders. The game environment is a joint dataset D that agents contribute private datasets D; (attacker train-time set) towards (Figure 1). After private dataset contributions are complete and ID is set, payoff's computed with respect test-time inputs (attacker run-time set) evaluated on a model trained by the defender on D (defender train set & validation set). Section 3.1 defines agent dynamics. Section 3.2 informs us how the relative distance between backdoor trigger patterns and trigger selection induces the backfire effect, and introduces the analysis of the insertion of subnetwork gradients. Appendix A.I provides supplementary preliminaries and proofs for this section.
Let X E xxxwxe and y 1,2, ..., k be the corresponding input and output spaces. {D .B {Di} ~ X' x y are sources of shifted X:y distributions from which an observation x can be sampled. x can be decomposed x x + E, where x is the set of clean features in x, and fE 2 03N++ is the set of perturbations that can exist. The features x are i.dd to the clean distribution, hence x ~ ID \ D; and y [(x; 0). We denote Acc(J; X,Y) to measure the classification accuracy of f(X;0) such that Acc(f; = TrT may [] sign( (a;; 0) y)]-
Attacker's Parameters: Each attacker is a player {ag that generates backdoored inputs P to in- sert into their private dataset contribution po E D.J'EN D. Each attacker only has information with respect to their own private dataset source (including in- puts, domain/style, class/labels), and backdoor trigger al- gorithm. Attackers use backdoor attack algorithm b; (Ap- pendix A. A.1.8), which accepts a set of inputs mapped to target poisoned labels (X; y-poise E D; to specify the intended label classification, backdoor perturbation rate e; to specify the proportion of an input to be perturbed, [xpoison
betedeer,
Suages labos
Figure 1: Representation of the multi- party backdoor attack. Attackers gener- ate unique backdoor trigger patterns with target poison labels and contribute to a joint dataset for the defender to construct a model.
and the poison rate p; | to specify the proportion of the private dataset to contain backdoored inputs, to return X poison b,(X,, posson , Pt).
An attacker a, would like maximize their payoff (Eqt 1), the attack success (ASR), which is the rate of misclassification of backdoored inputs poisen from the clean label Y.llann to the target poisoned label y.poison by the defender's 's model f. The attacker prefers to keep poison rate p; low to generate imperceptible and stealthy perturbations. The attacker strategy, formulated by its actions, is denoted as (ci,Pi, y-poison bz). The predicted output would be V = f(x;; (0, ID); (ry, 85); (Ci,Pi, y-ooson b;)). We compute the accuracy of the predicted outputs in test-time against the target poisoned labels as the payoff π Ace(Y, (UCYYY,Y,inn poiiso)) Each attacker optimizes their actions against the collective set of actions of the other -i attackers
Each defender is player (d, JEM that trains model f on the joint dataset ID, which may contain backdoored inputs, until it obtains model parameters 0. In our analysis, there is one defender only (M = 1). In terms of information, the defender can view and access the joint dataset and contributions D. but is not given information on attacker actions (e.g. which inputs are poisoned). To formulate the defender's ' strategies { {r,, (rr),s))pee JEM the defender can choose a model architecture (action r,) and backdoor defense (action 8y).
Defender's Parameters:
n Acc (s(X; (0, (ry, 85); ((e, Y.ooson bt), (6- P-5s ypoison b-i)),, ppoison
(1)
π 1
Σ Acc(f()), Y.pisso
(2)
3

---

Under review as a conference paper at ICLR 2023
10
10
10
10
9:225
0.654
1.000
3.128
0:22
0:261
0.124
0.1
0.061
0:151
0.08
0.1
107
1:1
0.124
107
Paison rate
0.2
0.196
0.404
3.801
0.223
1.500
2.081
0.132
0.201
0:166
0.1
0.092
153
121
0.323
0.08
9.3
007
0155
0.16
0.7
04
0.243
0.448
5.787
0.935
1.000
0:177
0.37
0:177
0.7
0.097
0.13
1.1
0.00
3.11
0.113
0.107
115
0.8
0.421
0.858
0.953
1.000
3.127
0.33
0.166
9:13
0.1
0.084
313
0:313
0.00
0.1
0:036
0.107
0.131
0.118
1.6
1.
0.364
0.384
5.885
0.950
1.000
3.126
0.135
0.189
0:245
0.1
0.303
0.314
0.315
0.227
0.08
0.089
0.113
0:11
0.323
0.09
0.1
0.2
0.4
0.8
1.0
Backdoor epsilon
0.0
0.2
0.4
1.0
Figure 2: Multi-Agent Attack Success Rate: For varying N, tabulate consistent backfiring trends across p and E.
The predicted label can be evaluated against the target poison label or the clean label. The 3 main ASR metrics: ① run-time accuracy of the predicted labels with respect to (w...t)) poisoned labels given backdoored inputs, 2 run-time accuracy of the predicted labels w.r.t clean labels given backdoored inputs, 3 run-time accuracy of the predicted labels w.r.t. clean labels given clean inputs. The defender's primary objective is minimize the individual and collective attack success rate of set of attackers (minimize D), and its secondary objective is to retain accuracy against clean inputs (maximize 3). In this setup, we focus on minimizing the collective attack success rate, hence the defender's s payoff can be approximated as the complementary of the mean attacker payoff (Eqt 2). We denote the collective attacker payoff and defender payoff, the utility functions of the game, as T" = mean + std and T" ((1 mean) + std) respectively.
3.2 INSPECTING SUBNETWORK GRADIENTS
A distribution shift is a divergence between 2 distributions of features with respect to their labels. Distribution shifts vary by source distribution (e.g. domain, task, label shift) and variations per source (e.g. multiple backdoor triggers, multiple domains). Joint distribution shift is a distribution shift attributed to multiple sources and/or variations per source. Eqt 8 is an example of how the multi-agent backdoor attack (multiple variations of backdoor attack) alters the probability density functions per label. Suppose 0.-1 has been optimized with respect to the clean samples ID \ {D) at iteration t and in the next iteration É we sample (subnetwork) gradient the loss on distributionalll-shiine siriibbtioonlly-shifte data ground-truth labels Φ E; + yi- We can inspect the insertion of subnetwork gradients. In our analysis, the gradient ạ is subnetwork gradien! corresponding to a
4 to minimize
samples D. At least one optimal Φ; = f2 6:-1 exists that maps
specific shift: 0, 0,- + >)(()) 65. Supporting explanation and proofs for Theorems I and 2 are provided in Appendix 6.1.4 and 6.1.6.
Theorem 1. Letx,y D (Do (D.) and (x Exoise + (e,) (y + y1) D2 be sampled clean and backdoored observations from their respective distributions. Let Rand s ~ U(S) s.t. ((s) [ST denote random distribution where an observation is uniformly sampled from (discrete)
set S. If N + oc, then it follows that predicted label y² = f(x + =;0) ~ u())) s.t. P(x") [
Theorem 2. A model of fixed capicity permits 0 with limited subnetworks. Loss optimization condition (Eqt 16) constrains the insertion of subnetwork gradients ó to minimize total loss over the joint dataset. To satisfy the ò-insertion condition LHS < RHS (16), other than imbalancing the loss terms with high poison rate (Lemma 3), Eqi 17 shows how the transferabiliity of determines whether its subnetwork gradient accepted given & +> 1b. It is empirically demonstrated { : N.
4

---

Under review as a conference paper at ICLR 2023
4 EVALUATION
4.1 DESIGN
Methodology. We implement the baseline backdoor attack algorithm BadNet (Gu et al., 2019b) with the adaptation of randomized pixels as unique backdoor trigger patterns per attacker (Appendix A.1.8). We evaluate upon CIFARIO dataset with 10 labels (Krizhevsky, 2009). The real poison rate
of an attacker a, is the proportion of the joint dataset that is backdoored by |xpoison
[D] For N
attackers and Va being the proportion of the dataset allocated to the defender, the real poison rate is calculated as p (1 Va) X p. Figure values out of 1 1.0; Table values out of 100.0.
(E1) Multi-Agent Attack Success Rate In this section, we investigate the research question: what effect on attack success rate does the inclusion of an additional attacker make? The base experimental configurations (unless otherwise specified) are listed here and Appendix A.2. Results are Figure 2.
(E2) Game variations In this section, we investigate: do changes in game setup daction-indepence variables) manifest different effects in the multi-agent backdoor attack?
Dataset (Table 1) We use 4 datasets, 2 being domain-adapted variants of the other 2. MNIST (LeCun & Cortes, 2010) and SVNH (Netzer et al., 2011) are a domain pair for digits. CI- FARIO (Krizhevsky, 2009) and STL10 (Coates et al., 2011) are a domain pair for objects.
Disasei
MANST
SYNW 921
C1ERR70
S7L10
Delender (Post Backidor))
19.7
84.5
35.4
poi
858
1000
cran lsbels
Capacity (Figure 2) We trained SmalICNN (channels [16, 32, 32),, ResNet-(9, 18, 34, 50, 101, 152) (He et al., 2015), Wide ResNet- 1-1)-2 (Zagoruyko & Komodakis, 2016), VGG- 11 (Simonyan & Zisserman, 2015).
Diipset
MNNY
3000
CHARDO
$1170
Defender Volidation
90.7
91.0
84.8
71.8
Kun-ime was pa--
27.5 22.2
12.3 + 18.1
12T
120
15.
small
Kun-tme Ace W.l. clean lnbels
105560
10113
15.5 4.6
160E 10.2
Table 1: Dataset variations: For each attacker count N. we apply constant set of attacker con- figurations across datasets to demonstrate a con- sistent backfiring effect.
(E3) Additional shift sources The multi- agent backdoor attack thus far manifests joint distribution shift in terms of increasing varia- tions per source; how would manifest if we increase sources? Adversarial perturbations Ea. introduced during test-time, are generated with the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015). Stylistic perturbations a +> Estyle (a 1.0 means 100% stylization), introduced during train-time, are generated with Adaptive Instance Normalization (AdalN)(Huang & Belongie, 2017). Results are summarized in Figure 7 and Figure 9.
(E4) Cooperation of agents In this section, we wish to leverage agent dynamics into the backdoor attack by investigating: can coopera- tion between agents successfully maximize the collective attack success rate? The base case is = 5, Va = 0.1, P = 0.55; the last parameter applies to the N 100 case; all 3 parameters apply to the Defense (Backdoor Ad- versarial Training w.r.t.
[55 configurations case. We evaluate inon-xcooperation w.r.t. informa- tion sharing of input poison parameters and/or target poison label selection. We summarize the results for coordinated trigger generation Ta- ble 4, and the lack thereof in Table 3. We record the escalation of poison rate and trigger label selection in Figure 6.
to
Figure 3: Performance against Defenses: BadNet and Clean-Label attacks against augmentative and removal defenses.
(E5) Performance against Defenses In this section, we investigate: how do aingle-agent backdoor defenses affect the multi-agent back- door attack payoffs? Defenses are evaluated
5

---

Under review as a conference paper at ICLR 2023
Model
parromeiers
Modee-RrosNe11
SmalcWV
TOMeLE
15,722
0.0014
ResNert)
7,756,554
0.6933
i8
11,181,642
1,0000
ResNec34
21,289,802
ResNet50
23,528.522
2,1042
ResNeri01
42,520,650
3.8027
Archentes
Rankertt
ResNot152
$8,164,296
5.2018
W-ResNet50
66,851,730
5,9790
W-RosNet101
11.1664
VGG
128,812,810
11,5200
Number of attachers
Table 2: Capacity variations: Run-time accuracy w.r.t. poisoned labels against N for different models (ratio number of parameters taken against ResNet-18).
the Clean Label Backdoor Attack (Turner al., 2019) in addition BadNet. We evaluate 2 aug- mentative (data augmentation (Borgnia et al., 2021), backdoor adversarial training (Geiping et al., 2021)) and 2 removal (spectral signatures (Tran al., 2018), activation clustering (Chen et al., 2018)) defenses. Results are summarized in Figure 3.
(E6) Model parameters inspection In this section, we investigate how model parameters change as N increases. To measure the likelihood that set of trained models on different attack configurations contain similar subnetworks, we measure the distance parameters, specifically the distance in parameters per layer for the original full DNN and pruned DNN. We prune SmalICNNs and generate the lottery ticket (subnetwork) with Iterative Magnitude Pruning (IMP) (Frankle Carbin, 2019b). Results are in Figure 8.
4.2 FINDINGS
The main takeaway from our findings is the phenomenon, denoted as the backforing effect, where a backdoor trigger pattern will trigger random label prediction and attain lower-bounded collective attack success rate T The backfiring effect demonstrates the following properties:
1. (Observation /) Backdoor trigger patterns tend to return random label predictions, and thus the collective attack success rate converges to the lower bound (Theorem 1). Optimal subnetworks per attacker are likely not inserted (Theorem 2).
2. (Observation 2) Observation is resilient against most combinations of agent strategies, particularly variations in defense, and cooperative/anticcoopr cooperatveee behavior.
3. (Observation 3) Adversarial perturbations are persistent and can co-exist in backdoored inputs while successfully lowering accuracy w.r.t. clean labels.
4. (Observation 4) Model parameters at N > become distant compared to N = =1, but for varying N tend to be similar each other.
(Observation 1: Backdoor-induced randomness) Across (E1-6) as N increases, the collective attack success rate decreases. In the presence of backdoor trigger pattern, the accuracy w.r.t. poisoned and clean labels converge towards the lower-bound attack success rate (0.1). (EI) Between E and p. std is correlated while mean is anti-correlated.
(E2) The drop in the accuracy w.r.t. de-
fender's validation set (containing clean labels of unpoisoned inputs and poisoned labels of poisoned labels) is close to negligible (with slight drop for STL10), attributable to a small real poison rate. In (E3: (backdoor, adversarial)) and (E3: (backdoor, adversarial, stylized)), )), the introduction of adversarial perturbations minimizes the accuracy w.r.t. clean labels to the lower bound if not already through the backfiring effect (e.g. 5h p 0.0 vs > 0.0). (E4) Backdoor trigger patterns of high cosine distance yield consistently high accuracy w.r.t. clean labels.
(E2) At N 1, the larger the model capacity, the lower the accuracy w.r.t. poisoned labels. We would expect that larger capacity models retain more backdoor subnetworks of multiple agents; however, with even as small as attackers, mean falls below 0.4 with low variance in mean + std across models i.e. backfiring is independent of model capacity.
(E3: bbcckoorr, stylized)) For run-time poison rate 0.0 (poisoned at train-time, but not run-tima run-time), accuracy w.r.t. clean labels is low only when the backdoor trigger is present; when the backdoor trigger is not present, accuracy is retentively high. The multi-agent backdoor attack does not violate the secondary objective of the defender; it does not affect standard performance clean inputs.

---

Under review as a conference paper at ICLR 2023
EST
61B
031
136
025
ad5
6:00
110
131
1118
10
mm
103
130
025
635
110
101
113
100
Table 3: Cooperation of agents: Backdoor trigger patterns generated with Handom-BadNes
Notreowr Nod waR wais
Na Petrae Wid pIRt HEM
Eucidber Jchementd modd
22
10
12
100
EXI
21
835
60
325
625
025
635
135
425
125
100
625
35
135
100
11
183
1:3
113
183
113
113
31
7.1
1030
13
185
100
Eacair 101
100
100
100
10
865
100
151
500
123
188
355
156
156
36
115
Table 4: Cooperation of agents: Backdoor trigger patterns generated with Orthogonal-BadNee
(E3: [backdoor, stylized]) For run-time poison rate 1.0 at N = 1, stylized perturbations do not affect accuracy w.r.t. poisoned labels. At N = 100, stylized perturbations yield further decrease in accuracy w.r.t. poisoned labels. We would expect stylization to strengthen backdoor trigger pattern, in-line with literature where backdoor triggers are piece-wise (Xue et al., 2020). However, Theorem I argues the backfiring effect persists despite stylization, the distribution of (Esayle é) y; would still tend to be random. It suggests the unlikelihood of trigger strengthening (or joint saliency), even if only poisoned imputs are stylized. Hence, attackers should conform their data source to that of other agents. Defenders should also robustify the joint dataset against bhift-inconsissenciess e.g. we expect augmentative defenses contribute to the backfiring effect and lower the accuracy w.r.t. poisoned labels. (E5) Some single-agent defenses counter the backfiring effect and increase collective attack success rate for BadNet and Clean-Labe' attacks.
(Observation 2: Futility of optimizing against other agents) (F4: [poison rate) Escalation is an intriguing aspect of this attack, as the payoffs have as much to do with the order in which attackers coordinate, as they do with individual attack configurations. In Figure 6 (right), the escalation of poison rate affects the distribution of individual attack success rates, but not the collective attack success rate. The interquartile range narrows when 80% of the attackers all escalate (inequal escalation), but returns to equilibrium once all attackers escalate to 100% to 0.55 (equal escalation). Non-unifome private datasets (e.g. heterogeneous label sets, stylization/tommin shift, escalating =), act against individual and collective ASR; attackers should prefer to coordinate such that their private dataset contributions approximate single-agent attack.
(E4: ttrggtt poison label)) In Table 3, if all attackers coordinate the same target poison label, a multi-agent backdoor attack can be successful. It unlikely attributable solely feature collisions ||r;-~-<11 O, as this pattern persists agnostic to cosine distance between backdoor trigger patterns. From an undefended multi-agent backdoor attack perspective, this would be considered a successful attack. Though the most successful attacker strategy, it is not robust to defender strategies: the worst-performing backdoor defense reduces the payoff substantially such that attackers attain a better expected payoff not coordinating label overlap (Table 6). Given the dominant strategy of the defender to enforce a backdoor defense, the Nash Equilibrium (20.3,79.7) ....,,9.7)%. 77%. is attained when attackers opt for random trigger patterns. Assuming attackers can coordinate joint strategy of random trigger patterns and 100% trigger overlap, they can attain an optimal payoff of (27.4, 72.6)%. 100% label overlap works optimally with trigger patterns of low cosine distance. Orthogonal-coordinan trigger patterns return consistently-Iow collective attack success rates (Table 4).
(E4: backdoor trigger pattem) )) In terms of sub-group cooperation, when 40% of attackers coordi- nate the same target label, there is no unilateral increase in their individual ASR compared to the other attackers at N 5. For large number of attackers (N 100), in Table 3 and Figure 6 (left),
7

---

Under review as a conference paper at ICLR 2023
when the sub sub-group of attackers coordinating their target labels increase, the collective ASR tends to increase and the distribution of individual ASR narrows. With respect to Theorem 2. it is empirically implicit that few backdoor subnetworks are inserted. The general pattern is when attackers exercise non-cooperative aggression non-uniformly, the distribution of their ASR widen, but when aggression uniform, the distribution narrows down the lower bound of ASR (mutually-assured destruction).
(E4: (target poison label)) We evaluate attackers cooperatively generating trigger patterns that reduce feature collisions and minimize loss interference (Eqt 17), i.e. orthogonal and residing in distant regions of the input space. The collective ASR is low, even with 100% target label overlap.
(E4: [backdoor trigger pattern, target poison label]) Coordinating low- or high-distance trigger pat- terns is futile. Attackers coordinating such that they share identical backdoor trigger pattern and 1 identical target poison label will approximate a single-agent attack. Other than the downside of not being able to flexibly curate the attack to their needs (e.g. targeted misclassification) single-agent backdoor attacks are demonstrably mitigable. In Table 3, where we have a set of low-distance trigger patterns, inadvertently due to high E, if attackers picked identical target poison labels despite non-identical backdoor trigger patterns, the collective ASR is high. This is in-line with results from Xue et al. (2020), where the authors implemented 2 single-agent backdoor attacks with multiple trigger patterns with expectedly low distance from each other (one attack where the trigger patterns of varying intensity of one patter;; another attack where they compose different sub-patt pott-ttttss.... and thus different combinations of these sub-pattern would compose different triggers of fow-distance to each other), and demonstrated high attack success rate. Similarly, our attackers share trigger pattern sub-region (overlapping region between trigger patterns) that is salient during training (i.e. an agent-robust backdoor trigger sub-pattern). This cooperative setting could be interpreted as partieu- larly weak, given the ease of defending against, and the requirement of attackers sharing information that can be used against them (e.g. coopperopee behaviour).
(Observation 3: Resilient adversarial perturbations) (E3: ladversaiial, stylized]) For run-time poison rate 0.0 (backdoored at main-time, not run-time), adversarial perturbations with respect to a private dataset, despite varyimg texture shift between private datasets, can attain high adversarial attack success rate (low accuracy w.r.t. clean labels)
multi-agent backdoor attack. An attacker can still pursue an adversarial attack strategy despite multiple agents; this may not always be practical is the attacker requires a misclassification of a specific target label (as demonstrated this experiment).
(E3: [backdoor, adversarial, stylized]) Low éh. P and (E3: (backdoor, adversarial)) increasing &a yields increasing backdoor ASR (accuracy w.r.t. poisoned labels, run-time poison rate I.0). High Eb, P and increasing Ea yields decreasing backdoor ASR. Interference takes place between adversarial and backdoor perturbations: when is low against the surrogate model's is gradients, FGSM is optimized towards pushing the inputs towards the poisoned label, but when p is high then FGSM is optimized towards pushing inputs away from the poisoned label.
model parameters) (E6) The weights for N = 1 are far from the weights for N > 1. The weights for N > 1 are all close to each other. The distance between weights tend to increase down convolutional layers and decrease down fully-connected layers. The distance values are similar between full network parameters, mask of the lottery ticket, and lottery ticket parameters. This implies the new optima of the full network is specifically attributed to changes in the lottery ticket required to resolve the backdoor trigger patterns. Since the weights do not change significantly w.r.t. NJN > 1, particularly for the lottery ticket, it also implies there is no proportional number of subnetworks inserted, supporting that few backdoor subnetworks are inserted (Thm 2).
(Observation 4: Increasingly
8

---

Under review as a conference paper at ICLR 2023
5 RECOMMENDATIONS & CONCLUSION
Motivated in pursuing practical robustness against backdoor attacks and machine-learning-ate we investigate the multi-agent backdoor attack, and extend the actions of attackers, such choice of adversarial attacks use in test-time, or a choice of cooperation or anti-cooperation. Aside from our findings, the main takeaways are follow:
1. The backfiring effect acts as natural defense against multi-agent backdoor attacks. Existing models may not require significant defenses block multi-agent backdoor attacks. If is likely that multiple attackers can exist, then the defender could focus on other aspects of model robustness other than backdoor robustness. This motivates backdoor defenses in practical settings, as most backdoor defenses are directed to single-attacker setups.
2. We are cautioned that the effectiveness of existing (single-agent) backdoor defenses drop when the number of attackers increase, thus they may not be prepared to robustify mod- els against multi-agent backdoor attacks. We recommend further study into multi-agent backdoor defenses.
Henceforth, we recommend using the multi-agent setting as a baseline for practical backdoor at- tack/defense work. In addition evaluating prospective defenses against backdoor attack with no defenses, we may wish to evaluate it against "natural setting" baseline (no defenses, purely multi-agent attacks e.g. N 100). We also recommend the evaluation of prospective attack in a multi-agent setting (how robust is the attack success rate when multiple attackers are present). Shifting away from the focus of attack designs optimized towards defenses, may also consider optimizing attack designs against this backfiring effect.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Scott Alfeld, Xiaojin Zhu, and Paul Barford. Data poisoning attacks against autoregressive mod- cls. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Feb. 2016. URL
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models, In USENIX Security, 2021.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenty Third International Conference Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp- 2938-2948. PMLR, 26-28 Aug 2020. URL http:///praceedings..u mlr
htmll.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML' 12, pp. 1467-1474. Omnipress, 2012. ISBN 9781450312851.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Gold- blum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. In ICASSP 2021 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3855 3855-3859, 2021. doi: 10.1 (O0/ICASSP397222221119
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering, 2018.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning, 2017.
Zheyi Chen, Pu Tian, Weixian Liao, and Wei Yu. Towards multi-party targeted model poisoning attacks against federated learning systems. High-Confidence Computing, pp. 100002, 2021. ISSN 2667-2952. doi:
URL
Brian Cheung. Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen. Superposition of many models into one, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statisties, volume 15 of Proceedings of Machine Learning Research, pp. 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. PMLR. URL https: httttss////rrreeedingg mlr. ptt...rres//ii//ccaaeeee
.tml..
Siddhartha Datta. Leam2weight: Weights transfer defense against similar-domani adversarial attacks, 2021. URL httpps://ppeerrveeeeeeee////o
SVLSHAp..
Siddhartha Datta and Nigel Shadbolt. Low-loss subspace compression for clean gains against multi-agent backdoor attacks. arXiv preprint arXiv:2203.03692 2022a.
Siddhartha Datta and Nigel Shadbolt. Hiding behind backdoors: Self-obfuscation against generative models, 2022b.
Siddhartha Datta, Giulio Lovisotto, Ivan Martinovic, and Nigel Shadbolt. Widen the backdoor to let more attackers in, 2021.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust Physical World Attacks on Deep Learning Visual Classification. In Computer Vision and Pattern Recognition (CVPR), 2018
10

---

Under review as a conference paper at ICLR 2023
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to byzantine-robust federated learning. In 29th USENIX Security Symposium (USENIX Security 20), pp. 1605-1622. USENIX Association, August 2020. ISBN 978-1-939133-17-5. URL
https://www.usttt wwww:w.w.enii...e.... http://wwww..ooooee/////ee/aaaae
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019a. URL
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, 2019b.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. aovaiiaadversarial training of neural networks. J. Mach. Learn. Res.,
jan 2016. ISSN 1532-4435.
Yansong Gao, Bao Gia Doan, Zhi Zhang, Siqi Ma, Jiliang Zhang, Anmin Fu, Surya Nepal, and Hyoungshick Kim. Backdoor attacks and countermeasures on deep learning: comprehensive review, 2020.
Jonas Geiping, Liam Fowl, Gowthami Somepalli, Micah Goldblum, Michael Moeller, and Tom Goldstein. What doesn't kill you makes you robust(er): Adversarial training against poisons and backdoors, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trainee CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019. URL https://openrreview.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. In International Conference on Learning Representations, 2015. URL
http://arxiv hwtp://////////////////s 1alds//1122.55222
Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain, 2019a.
Tianyu Gu, Kang Liu, Brendan Dolan-Gavine and Siddharth Garg, Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-472444, 2019b. doi: 10.1109/AC- CESS.2019.2909000
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in systems, 2019.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew M. Dai, and Dustin Tran. Training independent subnetworks for robust prediction, 2021.
Jamie Hayes and Olga Ohrimenko. Contamination attacks and mitigation in multi- party machine learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Informa- tion Processing Systems, volume 31. Curran Associates, Inc., 2018. URL Paper..p..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint 11Xiv=151..03385. 2015.
Anbu Huang. Dynamic backdoor attacks against federated learning, 2020.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017.
11

---

Under review as a conference paper at ICLR 2023
Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Manipulating machine learning: Poisoning attacks and countermeasures for regression learning, 2021.
and Bo Li.
Yujie Ji, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang. Model-reuse attacks deep learning systems. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pp. 349-363, 2018.
Linxi Jiang, Xingjun Ma, Zejia Weng, James Bailey, and Yu-Gang Jiang. anced gradients: A new cause of overestimated adversarial robustness, 2021.
Imbal- URL
tb.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1885-1894. PMLR, 06-11 Aug 2017. URL http://prooceedings.ne mlr. wt...a.......ess//ooooott. hhmml.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial Machine Learning Industry Perspectives. ArXiv2002056466 Cs Stat, May 2020.
Yann LeCun and Corinna Cortes. http://yann. lecun. com/exdb/mnist/
MNIST handwritten digit database. 2010.
URL
Ke Li, Tianhao Zhang, and Jitendra Malik. Approximate feature collisions in neural nets. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and Garnut (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL .df.
Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: survey, 2021.
Guanxiong Liu, Issa Khalil, Abdallah Khreishah, and NhatHai Phan. A synergetic attack against neural network classifiers combining backdoor and adversarial examples, 2021.
G. Lovisotto, S. Eberz, and I. Martinovic. Biometric backdoors: A poisoning attack against unsuper- vised template updating. In 2020 IEEE European Sympovium on Security and Privacy (EuroS&P) pp. 184-197, 2020. doi: 10.1 109/EumSS44544.222000000
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018. URL https:// New. net/forum?id=r>21BB222
Saeed Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Multi-party poisoning through generalized p-tampering, 2018.
Saced Mahloujifar, Mohammad Mahmoody, and Ameer Mohammed. Universal multi-party poisoning attacks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceed-
ings of Machine Learning Research, pp. 4283. PMLR, 09-15 Jun 2019. URL
Muzammal Naseer, Salman H. Khan, Harris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-
domain transferability of adversarial perturbations, 2019.
Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y.
Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 2011, 2011. URL
http://ufldl. stanfoord www.://ddddd///ooouuuummer/s//is
Anh Nguyen and Anh Tran. https://arrdv.orrgab////ee 08138.
Input-aware dynamic backdoor attack, 2020.
URL
12

---

Under review as a conference paper at ICLR 2023
Nicolas Papernot. A Marauder's Map of Security and Privacy in Machine Learning. ArXiv181101134 Cs. November 2018.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer, 2021a.
Xiangyu Qi, Jifeng Zhu, Chulin Xie, and Yong Yang. Subnet replacement: Deployment-stage backdoor attack against deep neural networks in gray-box setting, 2021b.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tht: Targeted neural network attack with bit trojan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13198-13207, 2020.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference, 2019.
Evgenia Rusak, Lukas Schott, Roland S. Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and MaccMel Frahm (eds.), Computer Vision ECCV 2020, pp. 53 69,69, Cham, 2020. Springer International Publishing. ISBN 978-3-030--8558-8..
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11957-11965, 2020.
Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic backdoor attacks against deep neural networks, 2021. URL https:://penrreview.ntt/// wttp://wwwwwww.weeteeeeeeeeeeeee ftne///frrmmddd64800000 RO0.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift, 2020.
Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 6106 1166611666 6116, Red Hook, NY, USA, 2018a. Curran Associates Inc.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. arXiv preprint arXiv:1804.00700 00792, 2018b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Ion Stoica, Dawn Song, Raluca Ada Popa, David Patterson, Michael W. Mahoney, Randy Katz, Anthony D. Joseph, Michael Jordan, Joseph M. Hellerstein, Joseph E. Gonzalez, Ken Goldberg. Ali Ghodsi, David Culler, and Pieter Abbeel. A berkeley view of systems challenges for ai, 2017.
Ananda Theertha Suresh, Brendan McMahan, Peter Kairouz, and Ziteng Sun. Can you really backdoor federated learning? 2019.
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. DeepFace: Closing the Gap to Human-Level Performance in Face Verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701-1708, Columbus, OH, USA, June 2014. IEEF ISBN 111-1-4799-5118-- doi: 10. 10.1109//VP222214.2200
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Basa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 31. Curran Associates, Inc., 2018. URL https::/pproceeding..et http:://pceecoooo.occccccccccees htt.:///e00022t.2211.ee22s22222. Paprraaa333eeeeeaeaar..rrr .dd..
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks, 2019.
URL https:: http::////ggrrrvddt.e//////////a
13

---

Under review as a conference paper at ICLR 2023
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh Agar- wal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of the tails: Yes, you really can backdoor federated learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 16070-16084. Curran Associates, Inc., 2020. URL
Cheng- -ssig Weng, Yan-Ting Lee, and Shan-Hung (Brandon) Wu. On the trade-off be- tween adversarial and backdoor robustness. In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Process- ing Systems, volume 33, pp. 11973-11983. Curran Associates, Inc., 2020. URL http:p/er222220111///////6665622
httos://prooceddins..eeree
Paper.pdf.
Emily Wenger, Josephine Passananti, Arjun Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y. Zhao. Backdoor attacks against deep learning systems in the physical world, 2021.
Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, and Fabio Roli. Is feature selection secure against training data poisoning? In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1689-1698, Lille, France, 07-09 Jul 2015. PMLR. URL http://proceedings mmr. wtt.ppee/////ia/////// html.
Qixue Xiao, Kang Li, Deyue Zhang, and Weilin Xu. Security risks in deep learning implementations. In 2018 IEFE Security and Privacy Workshops (SPW), pp. 123-128. IEEF, 2018.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2020. URL http:://oeemevveee.ee//roormrare
Mingfu Xue, Can He, Jian Wang, and Weiqiang Liu. One-to-n amp; n-to-one: Two advanced backdoor attacks against deep learning models. IEEE Transactions on Dependable and Secure Computing, pp. 1-1, 2020. doi: 10.1 10.100//DDSC.22000034488
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2041-2055, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Con- ference (BMVC), pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.87. URL https://dx. dow..ddoiiooooooooo... org/10. wtt.......4//...3.3333
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization, 2018
Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp- 7614-7623. PMLR, 09-15 Jun 2019a. URL http::/ppocceddings www.mmrrprrrs......
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International Conference on Machine Learning, pp. 7614-7623, PMLR, 2019b.
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 METHODOLOGY (EXTENDED)
A.1.1 GAME DESIGN (EXTENDED)
In this multi-agent training regime, there two types of agents: defenders and participants. Participants can be classified as either attackers and non-attackers. To simplify the discussion and analysis, we evaluate the setup in terms of attackers and defenders (experimentally, a non-attacking participant would approximate a defender with larger dataset allocation). multi-agent and single-agent attack are backdoor attacks with multiple and single attackers respectively.
a. π
9.
arg max Acc (cc] nf
arg mrx arggmaxcc(U(())Q()))
9:2
arg max ACc (cc(((()))( nf
arg min ((()))((
where (cc(U(u))Q(()) = Acc (cc(Q(())
9:aa
v.a
{le w)) = ((g, 3) wo
wEW
ev
(3)
Equilibrium payofl's. In setups where attackers are only playing against attackers, the equilibrium 70:.0- is the collective payoff T" of the highest value in the payoff matrix: Ta = max max(mean std). For setups where attackers are playing against defenders, the equilibrium aa.d is the collective payoff (T", x²) where both payoff values are maximized with respect to the dominant strategy taken by the other. We demonstrate this procedure in Equ 3, where we map strategy indices g, u for each agent by Q, U respectively: Q + (e, ypoison b), U the (g. u) optimiizinnoo procedure is one where the objective is to jointly maximize and minimize Acc w.r.t. (g. u). and payoffs at (g. u) are the Nash equilibria. It is additionally indicated the backdoor attack, as well as the multi-agent backdoor attack, is a zero-sum game, given that the total gains of agents are added up and the total losses subtracted, they will sum zero.
(r, s). From this result for aa.d, we find that
A.1.2 PRELIMINARIES ON SUBNETWORK GRADIENTS
aC(x,)) v) + ΦX.Y =
X.Y aC(x, w)
0: = 0=:=0_1 0.
(4)
00
00
Suppose the optimization of the parameters is viewed as a discrete optimization process, where each iteration samples gradient from set of gradients ò ~ & (Eqt 4), such that the total loss C decreases. In this analysis, we segregate the d-update with respect to clean data and distributionally-shifee data. Suppose 0,-1 1 has been optimized with respect to the clean samples D {D;} at iteration t 1, and in the next iteration i we sample á + to minimize the loss on disttibutionally-siifee samples ID. An example is the change in the probability density functions per class between before (Eqt 5) and after (Eqt 6) the train-time distribution is backdoor-perturboe exists that can map distributionally-shfffi
At least one optimal Φ; = 0, 0,-1 data to ground-truth labels é E + yt Hence, / is a set that contains set of endpoint gradients {Ø: J.V as well as set of interpolated gradients d.ch...
Frankle Carbin (2019a) showed in their work on the lottery ticket hypothesis that DNN can be decomposed into pruned subnetwork that carries the same functional similarity and accuracy to the full DNN. An (optimal) subnetwork @C m is the collection the minimum number of nodes required for the prediction of a ground-truit class with respect to the set of features, where mask m E (0. 1) @1 determines the indices in 0 not zeroed out. Subsequent works, such as MIMO (Havasi et al., 2021). show that multiple subnetworks can exist in a DNN, each subnetwork approximating a sub-function that predicts the likelihood a feature pertains to a specific class. Moreover, Qi et al. (2021b) show that a backdoor trigger can be formulated as a subnetwork and only occupies small portion of a DNN, and that in their work each subnetwork occupied 0.05% of model capacity. The subsequent iteration is thus evaluating the selection of subnetworks to insert into 6, where each subnetwork corresponds to specific shifted function. Hence, the gradient e is a combination of the various functional
15

---

Under review as a conference paper at ICLR 2023
subnetwork gradients that can be inserted while satisfying condition 16. Interpolated gradients ..0-1 = (0, - 01-1) o (N; m;) are gradients with different combinations of subnetwork masks and subnetwork values assigned in my and 02 accordingly; endpoint di Φ. + (0, (2-1) my- For our analysis of the multi-agent backdoor attack with respect to joint distribution shift, the gradient ø a subnetwork gradient corresponding
specific shift 5 + Φ (e.g. backdoor trigger pattern, or sub poopppppultionn shift in clean inputs, or stylization): 0g = 0 + Σ, (+)1 04.
A.1.3 PRELIMINARIES ON JOINT DISTRIBUTION SHIFT
Distribution shifts can vary by source of distribution (e.g. domain shift, task shift, label shift) and variations per source (e.g. multiple backdoor triggers, multiple domains). Joint distribution shift is denoted as the phenomenon when distribution shift is attributed to multiple sources and/or variations per source. Eqt is an example of how the multi-agent backdoor attack (multiple variations of backdoor attack) alters the probability density functions per label. To address joint distribution shift, should be transferable across a set of {E}. One approach inspecting this is by inspecting the insertion of subnetworks.
There growing literature on the study of joint distribution shift. Naseer al. (2019); Datta (2021); Qi et al. (2021a) show worsened model performance after applying adversarial perturbations upon domain-shifted inputs. Ganin et al. (2016) proposed domain-adapted adversarial training scheme to improve domain adaptation performance. Geirhos et al. (2019) also show that the use of stylized perturbations with AdaIN as an augmentation procedure can improve performance on an adversarial perturbation dataset ImageNet-C. AdvATrrooa (Liu al., 2021) combines adversarial perturbations together with backdoor trigger perturbations to craft stealthy triggers to perform backdoor attacks. Weng al. (2020) studies the trade-off between adversarial defenses optimized towards adversarial perturbations against backdoor defenses optimized towards backdoor perturbations. Santurkar et al. (2020) synthesize distribution shifts by combining random noise, adversarial perturbations, and domain shifts to varying levels to contribute subpopulation shift benchmarks. Rusak et al. (2020) proposed a robustness measure by augmenting a dataset with both adversarial noise and stylized perturbations, by evaluating a set of perturbation types including Gaussian noise, stylization and adversarial perturbations.
A.1.4 BACKFIRING EFFECT: CHANGES IN DISTRIBUTION OF
Lemma 1. For an input variable X(w) that is sampled randomly, the output variable X(w) from operations applied to X(w) will also tend to be random.
Proof. A random variable X a mapping from W to IR, that is X(w) E RR for w E R. X(w) = ((w) + e, thus X is also a mapping X W R. The measure for random variable X is defined by the cumulative distribution function F(x) = P(X & x). For x > 0, Fx = P(x I x) = Fx (a e). Thus ((w) is also measurable and is random variable defined on the sample space W.
P(X + E < x)- pxx < x =)
Lemma 2. Suppose a given model [xx, y;; 0) 0 x and loss L(x, y: 0) f(x) - y. Suppose we sample backdoored observations (I: = x + =1). (y + y1) ~ D,. The change in loss between clean to perturbed input is ac 50 e()) c
Proof.
AC L(xn, yii 0) L(x, y; 0)
[f Vi; 8) f(x, y; 0) - 1g,. v)
0|5 x] 1y v)
8c 21 X
204
Oc
(0) +
00
16

---

Under review as a conference paper at ICLR 2023
Theorem 1. (...y~D((DDUPl)
and (x Easise + (e,)) (y - yo) y-)
D, be sampled
clean and backdoored observations from their respective distributions. Let Rand s ~ U(S) s.s. P(s) denote random distribution where observation is uniformly sampled from (discrete)
set S. If + oc, then it follows that predicted label y² f(x + E;0) ~ xU() s.f. P(y") = 15'1
Proof sketch of Theorem 1. D (DoU )D: backdoored observations (x + Enoise + {E,))),, (y + yz) ~ D;-
With multiple attackers, we sample clean observations I, y
x x C( C(x, v) + ((e,v)
(C(xw) 4) Oc(x. 1)
aC(e. v)
00
00
0e
X.Y
OC(xx, v)
aC(e. v)
= 0: := 01
Σ
00
00
x.9
6.
This decomposition implies (c(x.))
88 updates part of w.r.t. x, which we denote as 0 @ my- and
aC(s.w) updates part of 0 w.r.t. which we denote 0 O mgg where m2 E {0, 1|10 are masks 80
of 0 @ (mx + m²). Given the distances (squared Euclidean norm) between the shifted inputs and outputs x F, and y + ys. we can enumerate the following 4 cases. Case (1) is approximately ageet backdoor attack, and is not evaluated. Cases (2)-(4) are variations of shifts in inputs and labels in a backdoor attack and manifest in our experiments.
[20; x
12
0
Il
y/ 2
(Case 1)
x 12 0
llv: v = 0
(Case 2)
| x 1 0
ll w| < 0
(Case 3)
[22 x | 0 ll v >1 0
(Case 4)
For = = (E,)iEN-1 N + 0o, then E ~ Rand. We denote a random distribution Rand : U(S) s.t. P(s) [S| where an observation is uniformly sampled from (discrete) set S. By Lemma and 2, if E Rand, then @c(5.
Rand and fxx;;) () f(x; = f (e:6) ~ Rand.
00
Hence, for each case of @c(c.g:))
08
BC(..)0) d9 + 0, given 0 = 0 2 (mx + m²), then f(s; 0) s2 f(e; 0 + m.) ~ Rand;
If @l(cyy:) 0, given mx 1 m. 01%) then f(c; = f(e; 0 + m,x) Rand. 09
In both cases, the predicted value of f will be sampled randomly. Given it randomly samples from the label space V. in a multi-agent backdoor attack, and shifted input:output Cases (2)-(4) it follows
that under the presence of a backdoor trigger pattern a prediction ~ (()) s.t. PP(y) = 1 The
lower bound of attack success rate would be TU (0.1 for CIFAR-10).
A.1.5
INSPECTING SUBNETWORK GRADIENTS: CHANGES IN PROBABILITY DISTRIBUTIONS W.R.T. X, S-SPACE
Theorem Let x. y ~ D (D) and Emose NN, -z)) D; be sampled clean and backdoored observations from their respective distributions. (2) denotes the probability density functions computing the likelihood that features of x map to label y. A model can be approximated by P of all labels (Eqt 8). For any given pair of atlucker indices (i, -i) and their corresponding backdoor trigger patterns EE,,E-s) and target poison labels V-e), we formulate the updated model f that can be approximated by P of all labels as Eqt 8. By analysis of cases and empirical results, the final prediction f(x) is skewed w.E.t. the distribution of {e).
Proof sketch of Theorem 3. Inductively demonstrated with different attack scenarios, show that the model as a function approximator is composed of multiple probability density functions corresponding to each backdoor mapping E; V4-
No Attack (N=0). We sample a set of clean observations x, y ~ D. P (x) denotes the probability density functions computing the likelihood that features of map label y. A model f can be
17

---

Under review as a conference paper at ICLR 2023
approximated by P of all labels P(z) (Px-ry(25 P Pesise - (Enoiss)) ) i.e.
(x;0) = arg max (PexxxPxxee (x))
P Penie--u (y(uiss)))
(5)
yE>
Single-Agent Backdoor Attack (N=1). We sample clean observations x, DD Dy and backdoored observations (x Enoise + Eo). (y + yo) ~ Du. where Eu > 0 and * you Sampling an input from the joint distribution x: ~ D where = Do (TD) Do), x would evaluated by ʃ with respect to all features (including perturbation feature). The newly-added perturbation feature eo is evaluated by f. where it manifests given input or not (returns 0 not), and requires a corresponding subnetwork gradient do. The proposed subnetwork gradient insertion do is accepted if Eqt 15 is satisfied.
f(z; + oo) arg max (xx-)(X)) P Feate-yyEEniie P (Eor)(E
(6)
PEY
Multi-Agent Backdoor Attack (N=2). We sample clean observations I, D backdoored observations (x + E + En): (y + yo) ~ Do and (x + Emoise + t1), ( - y1 ~ Dj.
(Do U D1).
where 5o, 01 and # yo, 9/1
There are 2 primary considerations to evaluate: (I) transfer/interfernnce between features and labels between Dg and Di; and (II) loss reduction w.r.. gradient selection.
(I) manifests case-by-case depending if in a particular case whether ||00 E < 0 or ||en & 1 z O, whether yo y1 or Yo A y1- In terms of gradient selection, since there are at least subnetwork gradient scenarios to evaluate: (i) no subnetwork gradient [Ø]. (ii) subnetwork gradient of Eo (endpoint) [0 + øol. (iii) subnetwork gradient of 1 (endpoint) [0 Φ11, and (iv) interpolated subnetwork gradient between En and ε [0 + Φ001]. Sampling æ ~ D, each of these 0 + Φ are evaluated case-by-case in Eqt 7. Among these candidate subnetwork gradients, the inserted (combination) of subnetwork gradients is determined by Eqt 17.
f(x; 0 + 6) arg max (Px (x)pp()) P Pensie -yy (Emiic) Proo-p(Eo)) (eoo)eooo-p-ppllleee Pe)-)p(E1))
(7)
yEy
Multi-Agent Backdoor Attack (N>I). Extending on our study of the 2-Attacker scenario, for any given pair of attacker indices (i, -i), we need to consider the distances (squared Euclidean norm) of (6i, S-4) and (yt, v-:). By induction, we obtain Eqi 8, where & is an interpolation of N subnetworks to varying extents.
f(z; 0 + d) arg max VEy
Px Px-+7(X): Peniee (Enaise)
P[
(8)
We enumerate cases from Eqt 8, mapped similar to Theorem cases. Note these are non-identical case mappings: Theorem 1 cases are evaluating distances between the unshifted and shifted inputs and labels in the joint dataset; Theorem 3 cases are evaluating distances between inputs and labels of private datasets of different attackers.
(Case 1) ||e; - lام 0 and y; y-a. attackers approximate single attacker (So, yo}. hence the collective attack success rate should approximate that of single-agent backdoor attack.
(Case 3) If y # y-s and ||, --v 113 0, then the feature collisions arising due to this label shift will cause conflicting label predictions from each P Pe,+,(E;)) in Eqt 8, which will skew the final label prediction.
This manifests in escalation, where in E4 we observe that if He, yi) - Ife- - y-shl then the attack success rate of a; would be better than a-i. This manifests when there are large number of attackers |e). where in [E4] we observe that many attackers with low distance perturbations but randomly-assigne target trigger labels tend to result in low collective attack success rate. This phenomenon may arise due to the model returning random label predictions during test-time provided random labels during train-time, in-line with Theorem 1, and extending upon Zhang et al. (2017).
(Cases & 4) If [c, - c- 0, whether y y- U: A Y-i- given the backdoor trigger patterns are distant the feature space (minimal feature collision), it follows that the collective attack success rate should be more dependent model capacity store unique subnetwork for each
Empirically, this is neither in-line with respect to capacity findings in E2 nor in-line with trigger distance findings in [E4. This informs us that, although the cosine distance indicates great distance
18

---

Under review as a conference paper at ICLR 2023
between trigger patterns, feature collisions still occur in practice when ||E; ||,;- - 0. It indicates that Case 3 (skewed label prediction) is more dominant in practice, and this is in-line with [E4) where the cosine distance between trigger patterns are high, but yi y-i returns higher collective attack success rate than when yi # y -i.
A.1.6 INSPECTING SUBNETWORE GRADIENTS: CHANGES IN LOSS TERMS
Theorem 2. A model of fixed capicity permits 0 with limited subnetworks. Loss optimization condition (Eqt 16) constrains the insertion of subnetwork gradients é to minimize total loss over the joint dataset. To satisfy the é-insertion condition LHS < RHS (16), other than imbalancing the loss terms with high poison rate (Lemma 3), Eqt 17 shows how the transferabillity of determines whether its subnetwork gradient & is accepted given E - v). It is empirically demonstrated [ < N.
Proof sketch of Theorem 2. Inductively demonstrated with different attack scenarios, we show how the loss function evaluates the insertion of subnetwork w.r.t. its gradients. Pursuing loss perspective on this problem is motivated by implications from the transfer-interffrence tradeoff (Riemer et al., 2019) on feature transferability, by implications from imbalanced gradients (Jiang et al., 2021) on how loss terms can overpower optimization pathways, and by implications of transfer loss an implicit distance metric.
Single-Agent Backdoor Attack (N=I). We consider the loss minimization procedure at this iteration an implicit measurement of the entropy of the backdoor subnetwork; if there is marginal informattion.capcciyy benefit from the insertion of ó to 0 compared to not inserting it, then the subnetwork gradient is added to 8 in this iteration. As I is already optimized D D;, therefore ac(x.v) xe 0 and thus resulting in Property 9. This update - 6 is represented in Eqt 4, where 80
the update condition L(I U &f hhackdorr) < (() is defined by Eqt 10, consisting of loss with respect to both clean and poisoned inputs. We denote LHS (10) and RHS (10) as the left-hand side and right-hand side of an update condition (10) respectively. The subnetwork would be updated based on update condition (10), where the insertion of the subnetwork would be rejected if LHS RHS (10). We refactor into Eqt (11) 1) as an update condition: if LHS > RHS (11), then a subnetwork gradient insertion rejected.
aC(z,
ac(x, v)
OC(E, v)
aC(s, v)
(9)
00
00
00
00
C( + af puckdoor, d yockmn + C(0 + o backdoor, ;x poison y poisonn <
L(0; x clean y + [(; x paison posion
(10)
C(B + o packdoor, X-laan yokm) - L(0; x ydlum) < L(B; X pousss ypuisu) - L(B + o buckdoor, x youionn
(11)
We refactor Eqt 11 into Eqt 14 after decomposing the backdoor inputs into clean and backdoor trigger features. To reiterate, to insert a candidate subnetwork gradient Φi, the aforementioned conditions 10-14 would need to be satisfied. To satisfy these conditions, at least 2 approaches can be taken: (Case 1) maximize the poison and perturbation rate, or (Case 2) jointly minimize the loss with respect to both clean inputs and backdoored inputs after the subnetwork gradient is inserted. From Figure 2 we know empirically that this condition can be satisfied for single-agent attacks.
(N = 10"),
D' Σ
C(O + o backdoor, x clean ylus)) - L(O; x clem clean
C(O: x poison y palsonn
- L(B + backdocr a
poison poisony
(12)
N.
DV Σ
Do
C(O + ol backioorr ; clam .35 plm)) L(O; a cleam
X
[(0; a poison
L(O + e Juckooor, poison y maison prison)
M.M
(13)
Do
|e|
C(0; x poison
yoisan) C(O + o backtoer poison peison
z,y
Do
C(0: x paison maison) - C(B + backloer roison paison
(14)
D Dol+
|x| |a|]
Dol
1
LiB + backdoar x cken , cleam)
L(0; x clean )
DD Do
2.
19

---

Under review as a conference paper at ICLR 2023
(Case 1) To maximize the poison and perturbation rate Du, alone, while keeping the loss values constant, we find that the lower bound required to satisfy conditions 10-14 is
DDol> 2|D Dol
(Lemma 3). Causing an imbalance between the loss function terms is in-line with analysis in imbalanced gradients (Jiang et al., 2021). Considering the number of poisoned samples affects the information:cupacity ratio, if the exclusion of @buckdocr results complete misclassification of X Xpoisonn then for the same capacity requirements, each backdoor subnetwork has a high information.capacity ratio and is possible for of qghackdoor be accepted.
(Case 2) Agnostic to substantial poison/perturbation rate increases (Case I), the attacker can also aim to craft backdoor trigger patterns that share transferable features to clean features (e.g. backdoor trigger patterns generated with PGD (Turner et al., 2019)). Given that Φi is crafted such that it minimizes loss w.r.t. backdoored features (i.c. max(RHS LHS) (11) or max(LHS RHS) (14)), in order for Eqt 14 to be satisfied, the candidate subnetwork gradient will be accepted if simultancously minimizes loss w.r.t. clean features (i.e. min(LHS RHS) (11) or min - LHS) (14)). We
@c(xx)
08 0
+1 for 0
represent this dual condition as
With sign(æ)
for < 0 condition 15
acic.y)
< 0
constrains the gradients to be in the same direction and loss to decrease, and must be satisfied to accept the candidate subnetwork gradient.
aC(x, y: + Φ)
OC(e, y; 0 + Φ) = -
sign(
sign(
(15)
00
00
Multi-Agent Backdoor Attack (N>1). The addition of each backdoor attackers results in corre- sponding subnetwork gradient, formulating 0 = _! Φ. + Σ 6b,, for N attackers. The bound in (N=2: Case /) persists for > 1; in this analysis, we extend on (N=2: Case 2). The cumulative poison rate p is composed of the poison rates for each attacker p = Σn Pr. Σn [Dxl, Each individual poison rate is thus smaller than the sum of all poison rates, thus the number backdoored inputs allocated per backdoor subnetwork is smaller. We also presume that any one backdoor poison rate is not greater than the clean dataset, i.c. (1 p) >> Pnvn E N. Note that (1 -1-33 p) includes not only the defender's clean contribution, but also clean inputs contributed by each attacker. Given the capacity limitations a DNN, if the number of attackers N is very large resulting in many candidate subnetworks, not all of them can be inserted into 9. Given fixed model capacity, A can only include a limited number of subnetworks, and this number depends on the extent each subnetwork carries information that can reduce loss for multiple backdoored sets of inputs (transferability) We can approximate this transferability by studying how the loss changes with respect to this subnetwork and 2 sets of inputs.
DAD,
D C(O +
Σ
L( +
Σ
On
backdoar clesn
clean
Σ
On
backdoor paison
x
prico)
(16)
DI,
D ΣΣ
L(0; x clem
clean
L(B;
x poison
y poison
Compared to the single-agent attack, the information:capacity ratio per backdoor subnetwork is diluted. We can infer this from Eqt 16 (a multi-agent extension of Eqt 12), where N subnetworks required to carry information to compute correct predictions for all N backdoored sets compared to in single-attacker scenario. The loss optimization procedure (Eqt 16) determines the selection of subnetworks gradients that should be selected to minimize total loss over the joint dataset. It implicitly determines which backdoored private datasets to ignore with respect to loss optimization, which we reflect in Eqt 17.
Given capacity limitations, every combination of backdoor subnetwork gradient evaluated against every pair of private dataset, and evaluate whether it simultaneously (1) reduces the total loss (Eqt 10), and (2) returns joint loss reduction with respect to any pair of sub-datasets (Eqt 16).
(=++)
0C(x, y: + +)
ØC(s, vi 0 + o)
(e o) = arg min - (1 + ((}])) = arg min sign
+
Σ
sign(
(17)
[6:0]
{e:d)
00
00
20

---

Under review as a conference paper at ICLR 2023
We extend update condition (16) into subnetwork gradient set optimization procedure (Eqt 17), where loss optimization computes a set of backdoor subnetwork gradients that can minimize the total loss over as many private datasets.
To make a backdoor subnetwork more salient with respect to procedure (17), an attacker could (i) increase their individual pr. (Lemma 3), (ii) have similar/transferable backdoor patterns and target poison labels as other attackers (or any other form of cooperative behavior). We empirically show this in E4I
With respect to E3, adversarial perturbations work because they re-use existing subnetworks in (i.e. gockan) without the need to insert one. Stylized perturbations be decomposed into style and content features; the content features may have transferability against unstylized content features thus there may be subsequent change Φeleen though the insertion of new di faces a similar insertion obstacle as of paackdoor.
Backdoor subnetworks can have varying distances from each other (e.g. depending on how similar the backdoor trigger patterns and corresponding target poison labels are). Measuring the distance between subnetworks would one way of testing whether subnetwork carries transferable features for multiple private datasets, at least in the backdoor setting each candidate subnetwork tends to be mapped to a specific private dataset. Based on [E6] we observe that the parameters diverge as N increase per layer, indicating the low likelihood that at scale large number of random trigger patterns can share common transferable backdoors. In other words, this supports the notion that each subnetwork is relatively unique for each trigger pattern and share low transferability across set of private datasets [l):(z) - وं (1)
A.1.7 BOUNDS FOR POISON- RATE -DIVEN SUBNETWORK INSERTION
satisfy condition 14 through an increase poison and perturbation rate alone, assuming the ratio of the loss differences is (i.e. there is 1:1 tradeoff where the insertion or removal of the subnetwork will cause the same increase/decrease in loss), then the resulting lower
Lemma
Ie| 2
bound is
Dol
2 Dol
Proof. If 2(0+0:x backdoor y backdoor -110: backdoor y bbackoorr [10+oxxclan eolam, y 211::xxoen yelen)
=1,
|x|
Dol+
Dal
JD
Dol+1
Dol
xr
|
1) Dol
|B
Dol
For the last statement to be true, 2
1 must be positive:
1
2
To obtain the minimum poison rate DD], we substitute the minimum perturbation rate such that
|Do > 2/D \ Dol
A.1.8 BACKDOOR ATTACK ALGORITHM
BadNet Gu et al. (2019a). Within the given dimensions (length x width w x channels c) of an input X, single backdoor trigger pattern mn replaces pixel values of x: inplace. Indices (7, c)
21

---

fillom-triige-ggeeerattee Under review as a conference paper at ICLR 2023
specify specific pixel value in matrix. is mask of identical dimensions that contains the perturbed pixel values, while 2 is its corresponding binary mask of I at the location a perturbation and 0 everywhere else, i.e.:
w(,,w,) - {,
if ml, w, c) if mf w, c)
The trigger pattern be of any value, as long as recurringly exists a poisoned dataset mapped to a poisoned label. Examples include sparse and semantically-irrele 2018; Guo et al., 2019), low-frequency semantic features (e.g. mask addition of accessories such as sunglasses (Wenger et al., 2021), and low-and low-arching narrow eyes (Stoica et al., 2017)). The poison rate is the proportion the private dataset that is backdoored:
perturbations (Eykholt et al.,
poison
Xllen clean
X poison
C being the element-wise product operator, the BadNet-generated backdoored input is:
poisso
о (1 2) + m O
b: X poison
{x ② (1 2) + o 2) EEX
Random-BadNo We implement the baseline backdoor attack algorithm BadNet (Gu et al., 2019b) with the adaptation that, instead of single square in the corner, we generate randomized pixels such that each attacker has their own specific trigger pattern (and avoid collisions). We verify these random trigger patterns as being functional for single-agent backdoor attacks at backdoor implementations in literature, including the default BadNet implementation, propose static trigger, such as square the corner of an image input. BadNet only requires a poison rate; we additionally introduce the perturbation rate e, which determines how much of an image to perturb,
Many existing
Extending on BadNet, m, is a randomly--enerrate trigger pattern, sampled per attacker a; We make use of seeded randon choice' and numpy. random. fnrm" functions from the Python numpy library. Perturbation rate E, dictates the likelihood that an index pixel (I, w, c) will be perturbed, and is used to generate the shape mask. The actual perturbation value is randomly sampled. As the perturbation dimensions are not constrained, a higher e; results in higher density of perturbations. We compute the shape mask 2,, perturbation mask m,, and consequently backdoored input as follows:
2 numpy. random. rammpy.ranndo..coiee((1... 1], size = x w. p = E,, E,, reshape(l, [,w] x
random. uni formfooo..00 1) x 255,
mi(1, w. c)
if =(1, c) = if =1. c) O
0, poison
② - 2;) + m, O 2
b X-Poison
{x. 3 (1 2;) m, ④ 21 2,) x.
The distribution of target poison labels may or may not be random. The distribution of clean labels are random, we randomly sample inputs from the attacker's private dataset re-assign clean labels to target poison labels. As all our evaluation datasets have 10 classes, this means To inputs have target poison labels equivalent to clean labels. We tabulate the raw accuracy w.r.t. poisoned labels; a more reflective attack success rate would be (Acc w.r.t. poisoned labels - 0.1).
1 of all backdoored
Brthogonal-BadNerr We adapt Random- Random-BadNet with orthogonality between N backdoor trigger patterns. Orthogonal trigger patterns should retain high cosine distances, and be far apart from each other in the representation space. We optimize for maximizing cosine distance here for the reason that suspect possibility that the collisions (Li et al., 2019), where we have 2 very similar features but tending towards very different
trigger patterns may in some cases incur feature
22

---

Under review as a conference paper at ICLR 2023
labels; hence, it may be in the interest of attackers to completely minimize this occurrence and generate distinctly different trigger patterns that occupy different regions of the representation space. One form of interpreting the intention of minimizing collisions between features (backdoor trigger patterns) is the intention of minimizing interference between these features; Cheung et al. (2019) introduced a method for continual learning where they would like to store of weights without inducing interference between them during training, and hence they generate a set of orthogonal context vectors that transforms the weights for each task such that each resulting matrix would reside in very distant region of the representation space against each other. We adapt similar implementation, but applying an orthogonal matrix that transforms the backdoor trigger patterns into residing in distant region away from the other resultant trigger patterns.
First, we generate a base random trigger pattern, the source of information sharing and coordination between the N trigger patterns (unlike Handom-BadNen In-line with Cheung et al. (2019), where we also use seeded scipy. stats. wowtho.grooupppeeo rvs from the Python scipy library (1 w), we sample orthogonal matrices from the Haar distribution, multiply against the original generated trigger pattern (clip values for colour range [0, 255]) to return an orthogonal/distat trigger pattern.
0g = fscipy. statoy..sttttooro..o ortho wttphh..rro..vo.o. rv) x
b: X poison
01- m, O 00, "
A.2 EVALUATION DESIGN (EXTENDED)
A.2.1 POISON RATE
The allocation of the joint dataset that each attacker is expected to contribute is assumed to be identical (only varying on the number of backdoored inputs); so the collective attacker allocation is 1 Va. and the individual attacker allocation is (1 V2) x x- Hence, the real poison rate is calculated as p (1 Va) +/4 x p. We visualize the allocation breakdown in Figure 4.
We acknowledge that a decrease in number of attackers can result in more of the joint dataset available for poisoning, and this can result in larger absolute number of poisoned samples if the poison rate stays constant. To counter this effect, we take into account the maximum number of attackers we wish evaluate for an experiment, e.g. N 1000, such that even as N varies, the real poison rate per attacker stays constant.
A.2.2 (E1) MULTI- ATTACK SUCCESS RATE (EXTENDED)
In this case, as we wish to test a large number of attackers N 1000 with small poison rates p = 0.1 for completeness, we set the defender allocation to be small V, = 0.1. This allocation gives sufficient space for 1000 attackers, and we also verify that this extremely poison rate can still manifest a, albeit weakened, backdoor attack at N
The vaintime-runtine split of each attacker is 80-20% (80% of the attacker's private dataset is contributed to the joint dataset, 20% reserved for evaluating in run-time). The train-test split for the defender was 80-20% (80% of joint dataset used for training, 20% for validation). We trained a ResNet- 18 (He et al., 2015) model with batch size 128 and with early stopping when loss converges (approximately 30 epochs, validation accuracy of 92 - 93%; loss convergence depends on pooled dataset structure and number of attackers). We use early-stopping for a large number of epochs, as this training scheme would be reused and ensures consistent loss convergence given varying training datasets (e.g. training a model on augmented dataset with backdoor adversarial training, training a model on stylized perturbations). We use Stochastic Gradient Descent optimizer with 0.001 learning rate and 0.9 momentum, and cross entropy loss function.
We set the seed of pre-requisite libraries as 3407 for all procedures, except procedures that require each attacker to have distinctly different randomly-sample values (e.g. trigger pattern generation) in which the seed value is the index of the attacker (starting from 0).
23

---

Under review as a conference paper at ICLR 2023
A.2.3 (E2) GAME VARIATIONS (EXTENDED)
Datasets We provision 4 datasets, 2 being domain-adapted variants of the other 2. MNIST (10 classes, 60,000 inputs, colour channel) (LeCun & Cortes, 2010) and SVNH (10 classes, 630,420 inputs, 3 colour channels) (Netzer et al., 2011) 60,0000 inputs, 3 colour channels) (Krizhevsky, 2009) and STL10 (10 classes, 12,000 inputs, colour channels) (Coates al., 2011) are domain pair for objects. We make use of the whole dataset instead of the pre-defined train-test splits provisioned, given that we would like to retain custom train-test splits for defenders and also because we have an additional run-time evaluation set for attackers (Figure 4).
domain pair for digits. CIFARIO 10 (10 classes,
Capacity We trained with the same training procedure as E1 (same splits, optimizers, loss functions) with the variation of the model architecture: SmalICNN (channels [16,32, 32) (Fort et al., 2020), ResNet- (9, 18, 34, 50, 101, 152) (He et al., 2015), Wide ResNet-(50, 101 }-2 (Zagoruyko Komodakis, 2016). VGG 11-layer model (with batch normalization) (Simonyan & Zisserman, 2015). Due to computational constraints, we wished to sample number of attackers for the following ranges (L... 10, 10.. 20, 20. 100, 100..500, 500. 1000), linearly-spaced these ranges into segments, and evaluated on all the returned N. Other than ResNet, we included other architectures including VGG11 I (a comparably large capacity model to Wide ResNet- 101 in terms of number of parameters but with different architecture) and SmalICNN (a small capacity model).
A.2.4 (E3) ADDITIONAL SHIFT SOURCES (EXTENDED)
Adversarial perturbations. For adversarial perturbations, we use the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015). With this attack method, adversarial perturbation rate E. 0.1 can sufficiently bring down the attack success rate comparable and similar to that of Ea = 1.0; we do this to induce variance. Hence, scale the perturbation against an upper limit 1.0 in our experiments, ie. E = 0.1 E.. Adversarial perturbations only crafts adversarial perturbations with respect to their own private dataset (i.e. they train their own surrogate models with the same training scheme the defender, and do not have any access to the joint dataset to craft perturbations). It is also worth noting that FGSM computes perturbations with respect to the gradients of the attacker's surrogate model where this model was trained on the attacker's 's private dataset, which contains backdoored inputs mapped to poisoned labels, meaning the feature representation space is perturbed with respect backdoor trigger patterns. We do not train surrogate model with respect to the clean private dataset, as the intention of surrogate model to approximate the target defender's model which has been assumed to be poisoned, and also in the attacker's best interests to introduce adversarial perturbations even with respect to the backdoor perturbations, as long as a misclassification occurs (which we can verify with the clcan-label accuracy).
only introduced during test-time, and each attacker
Defoode
Attuder: 1iai-Si Accasoo
Trann-me
Cole dodted
Onle de's
Bofore soversarid perturbatios
Aber stuernstil penurbations
Figure 5: Comparison of images before and after the insertion of adversarial perturbations. Back- door trigger perturbations still exist, and the ad- versarial perturbations exist in other regions of the image.
Figure 4: Summary of defender and attacker allo- cations train-time and run-time. Train-lime is the period where only training data is processed, including the defender's 's train set, defender's vali- dation set, and attacker's train-time set. Test-time the period where only test data processed, being only attacker run-time set in this setup.
24

---

Under review as a conference paper at ICLR 2023
Stylized perturbations. For stylistic perturbations, we use the Adaptive Instance Normalization (AdaIN) stylization method (Huang & Belongie, 2017), which is a standard method to stylize datasets (Geirhos al., 2019). Dataset stylization is considered texture shift
such Byylized-mageeNa or domain shift in different literature. We randomly sample a distinct (non-repeating) style for each attacker. a is the degree of stylization to apply: 1.0 means 100% stylization, 0% means stylization. We follow the implementation in Huang & Belongie (2017) and Geirhos et al. (2019) and stylize CIFAR-10 with the Paintings by Numbers style dataset. We adapt the method for our attack, by first randomly sampling a distinct set of styles for each attacker, and stylizing each attacker's 's sub-datase before the insertion of backdoor or adversarial perturbations. This shift also contributes to the realistic scenario that different agents may have shifted datasets given heterogenous sources.
When the poison rate 0.0, then the accuracy w.r.t. poisoned labels is equivalent to the accuracy w.r.t. clean labels. In our results, target poisoned labels are the intended labels based on attacker preferences, being clean labels for clean untriggered inputs and poisoned labels for backdoor triggered inputs. This means that when the run-time poison rate is 0.0, then the accuracy w.r.t. poison and clean labels are identical (unfiltered values in Figure 9). also means that when E, p = 0.0, then these values would be identical for aceuracy w.r.t. poison as well as clean labels.
Prior to interpreting the results, we need to consider the attack objective of the attacker. For a backdoor attack, attacker's 's objective is to maximize the accuracy w.r.. poisoned labels. In an adversarial attack, an attacker's ' objective is to minimize the accuracy w.r.t. clean labels. The attack objectives may have additional conditions in literature, such as imperceptability to humans, or retaining high accuracy on clean inputs, etc, but the aforementioned 2 are the primary goals. They may not necessarily be contradictory either for two reasons: (i) if the poisoned label is non-identical to the clean label, then both a backdoor attack and adversarial attack will succeed in rendering a misclassification w.r.t. clean labels; and (ii) one similarity between a backdoor attack and adversarial attack is that they both rely on varying fidelity of information of the train-tin time distributions, where the backdoor attack has white-box knowledge of the perturbations that will cross the decision boundary to target class, while the adversarial attack has grey/black- -bxx knowledge of perturbations that may have a likelihood of crossing the decision boundary to a target class. In any case, we conclude for this evaluation that the attack objective of the attacker is to minimize the accuracy w.r.t. clean labels.
One of our suspicions regarding a low backdoor attack success rate is whether the generation of adversarial perturbations may possibly de- de-perturb backdoor perturbations: we visually inspect before- attack and after-attack images to verify that both adversarial and backdoor perturbations are retained in multi-agent backdoor setting, and we sample set in Figure 5.
With regards to Sb. P = 1.0 having such high ASR, it may not just be cause of a strong trigger pattern/saliency, but also it should be noted there is 100% class imbalance in this case for the attacker's 's surrogate model (only class in the training set, hence no decision boundaries to cross; adversarial ASR should be 0.0). Given that our intention for this experiment of a joint distribution shift between these two attacks unmodified in procedure and aligned as much as possible to their original attack design, we did not construct a coordinated adversarial -ersaaiill-acckooor attack where only adversarial perturbations that do not counter, or even reinforce, the backdoor perturbations / poison labels are crafted.
observe the effect
A.2.5 (E4) COOPERATION OF AGENTS (EXTENDED)
In this setting, we primarily study 2 variables of cooperation, which are also the set of actions that an attacker can take: (i) input poison pa- rameters (p, E. and distance between different attackers' backdoor trigger patterns). and (ii) target poison label selection. addition to these 2 attacker actions that will formulate a set of strategies, we wish to evaluate the robustness of these strategies by (i) testing the scalability of the strategy at very large attacker counts, and (ii) testing the robustness of the strategy by introduc- ing the weakest single-agent backdoor defense. We establish information sharing as the proce-
d
No Defense
Defense
0.55
(26.2, 73.8)
(13.7, 86.3)
0% label overlap
(8.6, 91.4)
(12.3, 87.7)
a 100% label overlap
(54.2, 45.8)
(18.1, 81.9)
Random triggers
[40.3, 59.7)
(20.3, 79.7)
Orthogonal triggers
(8.8,9122) (12.2,87.8
Table 6: Expected values of each strategy from
Tables 3 and 4, to determine that the approximate Nash Equilibrium is (20.3, 79.7) when tends to use random triggers and d uses a defense.
25

---

Under review as a conference paper at ICLR 2023
traine-esst split
train-runtimee split (attucker)
Model
Turget poison
Real poison
Expu.
Vd
(defender)
lubel selection
rute (8)
80-20%
80-20%
ResNet-
Random
0.1.
0,00009 (5),
0.2.
0.00018 (11).
0.4.
0.00036 (22).
0.8.
0.00072 (44).
0.0009
E22
0.2
80-20%
80-20%
ResNet-18
Random
0.55
0.55
0.0064 (264)
E3
0.1
80-20%
80-20%
ResNet- 18
Random
0.55
0.0005 (30)
E4
0.5
80-20%
80-20%
RosNet-18
0.35
0.0.
0.0 (0).
0.2.
0,001 (60).
0.4,
0.002 (120).
0.6,
0.003 (180).
0.8.
0.004 (240).
0.005 (300)
F5
0.8
80-20%
80-20%
ResNet- 18
Random
0.55
0.55
0.001 (66)
0.1
80-20%
80-20%
SmalICNN
Random
0.55
0.55
0.0005 (30)
Table Summary default experimental configurations in each experiment. We also state the real poison rate (and corresponding number of train-time poisoned inputs). '-" indicates values that are varying by default.
dure in which agents send information between each other, and the collective information garnered can return outcomes in the range of cnti-cooperative (which we denote as agents using information that hinders the other agent's s individual payoff) to cooperative (which we denote as agents using infor- mation to maximize collective payoff). The payoff functions for anti-cooperative and non-cooperative strategies (i.e. individual ASR) are the same. We evaluate on classes: (airplane), 2 (bird), 4 (deer), 6 (frog), 8 (ship); retain the same proportions of each of these 5 classes as N varies. For N 100, we specify the number of attackers (Y) that target class Y.
With these strategies, would like to observe the following agent dynamics- driven phenomenon, specifically the outcomes from attackers exercising various extents of selfishness (escalat- ing attack parameters) against extents of collective goodwill (coordinating attack parameters): (i) the outcome from the escalation of e: (ii) the out- come from a gradual coordination of target labels; (iii) the outcome of co- ordinating trigger pattern generation. Escalation.
42
24
Figure 6: Cooperation of agents (NN = 100): (left) Coordina- tion in terms proportion of attackers selecting poison label 4, while others randomly select; (right) Escalation in terms of proportion of attackers increasing p from 0.15 to 0.55, while others retain 0.15, then subsequently the increase in from 0.55 to 0.95, while others retain their greviously-escallee 0.55; these escalation cases plotted against corresponding distributions of accuracy w.r.t. poisoned labels (top row) and clean labels (bottom row).
Here, we describe how we demonstrate selfish escalation or collective coordination. While in other experiments, of random selection to a large number of attackers to approximate non-cooperative behaviour, would now like to simulate simplified cases of anti-cooperative and cooperative strategies. For the selfish escalation of trigger patterns, we suppose each attacker crafts their backdoor trigger patterns independently from each other, and when information of the presence of other attackers is known or each attacker wishes to raise certainty of a backdoor attack, we consider the case where attackers escalate their from 0.15 0.55 to 0.95. To study trigger label collision cases and coordinated target label selection, we show gradual change in trigger label selection amongst attackers, where they start off each having independent labels, then there is some trigger label collision between 40% of the attackers (40% attackers sharing the same label), then there is 100% trigger label collision between all of the attackers (i.e. all attackers share the same label). Coordination can manifest either attackers each choosing distinctly different labels, or attackers all choosing the same label. To monitor collision between trigger patterns and trigger labels, we compute the cosine distance between each attacker's 's trigger pattern against that of Attacker Agent 1. The trigger patterns are randomly generated; first by computing a random set of pixel positional indices within image dimensions (pixel positions to be perturbed), which
scale the effect
26

---

Under review as a conference paper at ICLR 2023
refer to shape and show its corresponding cosine distance; then by computing the colour value change for each pixel position in the shape, which we refer this final trigger pattern of both positions and perturbations values as shape+colour. We assume there is no coordination hence the choice of random perturbations (as opposed to a certurbation-opiiizztte distance), though as e increases, we note that the cosine distance for both shape and shape+colour decrease (as the density of perturbations would be expected to be higher as increases); this provides us with a range of trigger pattern distances in the representation space evaluate against trigger label selection. In Table 3 and 4, for agents without the escalation in overlap of target label 4 (in red), we only redistribute the other 4 labels out equally, but in Figures 6 we redistribute 10 labels out randomly.
function of minimizing cosine
A.2.6 (E5) PERFORMANCE AGAINST DEFENSES (EXTENDED)
We set the defender allocation to be significantly higher than that of other experiments because some of the defenses require subsets from the defender's 's private dataset to sample from, and enlarging this allows us to test the single-agent defenses leniently for the defenders and harshly for the attackers. This enlargement of defender allocation would also mean we should be careful when comparing values between experiments; for example, the real poison rate in this experiment is 0.0011 (66). which has 12 more poisoned samples than p = 1.0 in E attributed to the large difference in N considered.
Extending stealth and imperceptabilitty an important aspect of the backdoor attack, there is further sub-subelassification of backdoor attacks into dirty- (Gu et al., 2019a) and clean-label (Shafahi et al., 2018a; Zhu et al., 2019a) backdoor attacks. In dirty-label backdoor attacks, the true label of triggered inputs does not match the label assigned by the attacker (i.e., the sample would appear incorrectly labeled to a human). In clean-label backdoor attacks, the true label of triggered inputs matches the label assigned by the attacker. The 2 sub-classes can be executed with the same attack algorithm and follow the same underlying principles, with change to the target trigger label, though variant algorithms for the clean-label algorithm also exist (Shafahi et al., 2018:t 18:; Zhu et al., 2019a).
Clean Label Backdoor Attack. Hence, in addition to BadNet, we also evaluate defenses on the Clean Label Backdoor Attack (Turner et al., 2019). Also a common baseline backdoor attack algorithm, the main idea of their method to perturb the poisoned samples such that the learning of salient characteristic of the input more difficult, hence causing the model to rely more heavily on the backdoor pattern in order to successfully perform label classification. It utilizes adversarial examples or GAN-generated data, such that the resulting poisoned inputs appear to be consistent with the clean labels and thus seem benign even upon human inspection. The objective of targeted clean label poisoning attack (Shafahi al., 20 8a) (which also applies to clean label backdoor attack), is to introduce backdoor perturbations to a set of inputs during train-time whose poisoned labels are equal to their original clean labels, but the usage of the backdoor pattern during run-time regardless of ground-truth class would return the poisoned label. We align our implementation with that of Turner et al. (2019), and use projected gradient descent (PGD) (Madry et al., 2018) to insert backdoor perturbations 6.
A lower accuracy w.r.t. poisoned labels infers a better defense. While post-defense accuracy below 0.1 is indicative of mislabelling poisoned samples whose ground-truth clean labels were also poisoned labels, it is also at least indicative of de-salienating the backdoor trigger perturbation, and hence indicative of backdoor robustness.
Defenses. We evaluate augmentative (data augmentation, backdoor adversarial training) and removal (spectral signatures, activation clustering) defenses. For augmentative defenses, 50% of the defender's allocation of the dataset assigned to augmentation: for V22 = 0.8, 0.4 is clean, 0.4 is augmented. We devised backdoor defenses based on the backfiring effect, such as agent augmentation and agent indexing, in Datta et al. (2021). To mitigate the low accuracy w.r.t. clean labels in the presence of a backdoor trigger, we devised backdoor defense through the construction of compressed low-loss subspaces in Datta & Shadbolt (2022a).
No Defense: We retain identical defender model training conditions to that in E1. The defender allocation 0.8 would be unmodified during model training in this setting.
Data Augmentation: Recent evidence suggests that using strong data augmentation techniques (Borgnia et al., 2021) (c.g., CutMix (Yun et al., 2019) or MixUp (Zhang et al., 2018)) leads to a reduced backdoor attack success rate. We implement CutMix (Yun et al.,
27

---

Under review as a conference paper at ICLR 2023
2019), where augmentation takes place per batch, and training completes in accordance with aforementioned early stopping.
Backdoor Adversarial Training: Geiping et al. (2021) extend the concept of adversarial training on defender-generateee backdoor examples to insert their own triggers to existing labels. We implement backdoor adversarial training (Geiping et al., 2021), where the generation of backdoor perturbations is through BadNet (Gu et al., 2019a), where 50% of the defender's allocation of the dataset is assigned backdoor perturbation, p. s = 0.4, and 20 different backdoor triggers used (i.e. allocation of defender's 's dataset for each backdoor 1 ).
trigger pattern is (1 0.5) x 0.8 20
Spectral Signatures: Spectral Signatures (Tran et al., 2018) is an input inspection method used to perform subset removal from a training dataset. For each class in the backdoored dataset, the method uses the singular value decomposition of the covariance matix of the learned representation for each input in a class in order to compute an outlier score, and remove the top scores before re- re-training. In-line with existing implementations, we remove the top 5 scores for N = attackers. For N = 100, we scale this value accordingly and remove the top 500 scores. Activation Clustering: Activation Clustering (Chen et al., 2018) is also an input inspection method used to perform subset removal from a training dataset. In-line with Chen al. (2018)'s 's implementation. we perform dimensionality reduction using Independent Com- ponent Analysis (ICA) on the dataset activations, then use k-means clustering to separate the activations into two clusters, then use Exclusionary Reclassification to score and assess whether a given cluster corresponds to backdoored data and remove it.
A.2.7 (E6) MODEL PARAMETERS INSPECTION (EXTENDED)
Other than e 0.55 and training SmalICNN, we retain the same attacker/defender configurations as EI. We adopt SmallCNN for its low parameter count, which would be helpful in simplifying the subnetwork generation and analysis, as well as overall interpretability. In addition, as we show that the backdoor attack is consistent across all model architectures (E2)). using smaller model means the subnetworks with respect to the complete network is less diluted and larger in proportion (and hence the cosine distance would be less diluted for our observation).
Lottery ticket. We start with fixed random initialization, shared across the 4 models trained on N = I, 10, 100, 1000. We use Frankle& & Carbin (2019b)'s Iterative Magnitude Pruning (IMP) procedure to generate a pruned DNN (0.8% in size of the full DNN across all Ns), also denoted as the lottery ticket. The lottery ticket is subnetwork, specifically the set of nodes in the full DNN that are sufficient for inference at a similar performance as the unpruned DNN. The study of the lottery ticket helps us make some inferences with respect to how the feature space changes, as well where the optima on the loss landscape has deviated. We compute the cosine distance per layer between (i) parameters in the full DNN, (ii) mask of the lottery ticket against the DNN, (iii) parameters of the lottery ticket. We plot the values of N v.s. N for the weights and biases matrices for each convolutional layer (conv2d(layer. _ineex) and fully-connected layer (layer_index] All in all, the lottery ticket is a proxy for the most salient features, hence also acts as an alternate feature space representation.
We compute the full network distance, as we wish to decompose the distance changes of the subnetwork across N with respect to both new optima w.r.t. change in N in addition to changes to the DNN w.r.t. the introduction of new trigger patterns. The mask is one-zero positional matrix; rather than removing the zeroes, we compute the distance including zeros to retain the original dimensionality of the full network, and measure the distance with factoring in the position of the values. To retain the position of parameter values, we multiply the one-zero mask against the full network parameters. While the size (new parameters count) of the pruned network can vary, we set the threshold to stop pruning at 97%, where pruning stops after we maximize accuracy for 97% prumed weights. Specifically, the lottery tickets across all Ns are 0.8% in size of the full DNN; in other words, we pruned the count from 15,722 to 126. Though we may expect that if a DNN were allowed to store as many subnetworks per triggers as possible if we let the pruning threshold to be variable, by setting the capacity to be fixed (in-line with our theoretical analysis), we let the optimization steps manifest the loss function tradeoff discussed earlier, and manifest the acceptancefrejection of backdoor subnetwork insertion for fixed lottery ticket generation.
28

---

Under review as a conference paper at ICLR 2023
- 0.0
0.0 After Adivenioiial Attack
Q 1.0
Q 1.0 After Adversarial Attack
Refore Advensriial Attack
Refore Adversorial Artack
With Piggle
WCS rigaer yaarliie tieelieeeee
scibr Ieb
A221
6:331
0407
LAV
0:300
E.977
8:29
0:11
0.99
+.10%
A32
6:30
0004
1000
6:3
0.11
100
4.5
61
X110
6.14
080
415
wobd roger snvare
Weou rage
Withol mogel
Woor rope
A001
6:93
Backdoor E,P
5:03
6-877
5:63
000
2017
0313
AU14 L133
AMM
5.30
0.67
0211
5.10
Artcks
Wis Hager
With
Lis
6:3
8.00
with Emperiseah
100
0:30
N
2111
Fosa
Sume
ACET shir 80d0 A0dm
CAIR
E,07
14.0
Adversarial E
Figure 7: Additional shift sources: Joint distribution shift of varying counts: indexing columns from left to right, column (Multiple (backdoor) perturbations; shifts=1), column (Multiple (backdoor, adversarial) perturbations; shifts=2), column 2 (Multiple (backdoor, stylized) perturbations; shifts=2), column 3 (Multiple (backdoor, adversarial, stylized] perturbations; shifts=3).
29

---

Under review as a conference paper at ICLR 2023
comv100 weihtt
coT424: biss
comiedt weggtt
convedl
conv2c2 weight
convidt 3im
1c0 we gitse
1:3 2w0
1C we gits
1:1. Diss
The 10% corerr
10
36 390100e
to 1693999
3993900
10 1603099
25 1003990
to 1683099
39 1003900
59 5001004
1003300
86
34
0.5
67
66
Figure 8: Model parameters inspection: Cosine distance of lottery ticket and original DNN per layer across N.
Figure 9: Additional shift sources: Unfiltered for a 0, 0.5, 1.0 and accuracy w.r.t. clean labels
30


Reviews:
[{"review_id": "t17YIOWkNA", "reviewer": "Reviewer_q38C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the effect of backdoor attacks in the multi-agent attacker scenario, where each attacker constructs backdoor attacks on  it's individual dataset. There is a single defender learner who receives poisoned datasets from all attackers, and learn a model on that dataset. The paper discovered a backfire effect, which is that when there are multiple attackers, the attack success rate drops. The paper introduced a set of cooperative dynamics between multiple attackers, extending on existing backdoor attack procedures. Experiments demonstrate the findings of the paper.", "summary_of_the_review": "I work in related areas.", "strength_and_weaknesses": "Strength:\n\n(1). The paper studied a novel and interesting topic in backdoor attack community, where multiple attackers exist in the environment and each attacker only performs backdoor poisoning on its private dataset.\n\nWeaknesses:\n\n(1). I am somewhat confused by the topic of the paper. Is the paper trying to demonstrate that if a training set is composed of multiple private subsets, and each subset can be backdoor attacked by an individual attacker, then the model learned jointly from the combined dataset may have low attack success rate? If so, I believe the discovery is not surprising and not super interesting to the community. The presentation of the paper needs to be improved significantly and the problem setup should be described more rigorously. Right now the section 3 is very confusing and some notations do not have a clear explanation. For example, the input space X=R^{l*w*c}. What is l, w, and c?\n\n(2). I find the setting of the problem not super interesting. In particular, the paper considered an environment where each private subset can be attacked, and some defender is learning on a global dataset constructed by combing every poisoned subset. I am wondering in what real-world situations would such an environment exist? It's like the whole environment consists of attackers with different backdoor goals. A more reasonable situation is when only a small fraction of private subsets can be poisoned by an attacker, and most subsets are benign. In this case, one would be interested in designing defense mechanisms using clean data.\n\n(3). The paper implies that a potential defense is to have multiple attackers. However, this is really not defense from the learner's perspective, because the leaner did not do anything at all to combat attackers in this case. Instead, the learner does nothing, but the attackers hurt each other. The paper does not propose any novel defense algorithms in this regard, which makes the paper very weak.\n\n(4). The theoretical results need more intuitive explanation. I am wondering if the authors could explain the main messages that the authors want to convey with those results. Right now, the theory seems a bit disconnected to the main topic of the paper. For example, Theorem 1 suggests that as the number of attackers go to infinity, the prediction of the learned model becomes uniform. This result shows that the model performance decreases to the worst random predictor as the number of attackers grows. However, I don't see how this result connects to the backdoor attack studied in this paper. Why this result is interesting from the multi-attacker backdoor poisoning perspective. The authors may want to rethink their study and restate their main results to make them more coherent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vx7fvu0Ikk6", "reviewer": "Reviewer_eZzf", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on multi-agent backdoor attack setup where multiple attackers inject backdoors for a single target model. The authors claim that this multi-agent attack can be defended by stable, natural defense. ", "summary_of_the_review": "Due to the weaknesses written above, I think it is not worth publishing. ", "strength_and_weaknesses": "I cannot find some strength of this paper. Below I write weaknesses.\n\n- Unclear theoretical results\n    - From the information provided in the main body, I could not get why Theorem 1 is true. Why does the attack success rate goes to 1/|Y| as the number of attackers go to infinity?\n    - What is the key statement of Theorem 2? It seems like the setting & statement is shuffled in current version.\n- Presentation of experimental results are not friendly to readers\n    - Table 3 and 4 are containing many numbers, but what should we focus on? What is the take-away message? \n    - Which text is for explaining Table 1 and 2? I was hard to get what is the take-away message of these results.\n- Notation is not standard, some terms are defined unclearly\n    - $x = \\vx + \\varepsilon$ is very weird. $x$ is usually for scalar, and $\\vx$ is usually for vector.\n    - What is “shifted X:Y distributions” in page 3?\n    - $f(x;\\theta)$ seems to be the output of a classifier parameterized by $\\theta$, but not specified\n    - accuracy expression is weird. If predicted label $f(x;\\theta)$ is smaller than $y$, then it is considered as correct classification? And the maximum accuracy according to this expression is 2|X|, not 1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Pf35GMs9nfL", "reviewer": "Reviewer_JzAB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies a data-backdooring game in which multiple attackers strive to poison a training dataset of the defender(s) to affect classification on selected targets while preserving accuracy on the rest of the samples. After introducing related work, the authors formally describe the game the attackers and defender(s) are playing. The rest, comprising the main part of the text, is dedicated to empirical analysis of the game and description of the findings. According to the authors, the key takeaway is that in this setting, the attackers may experience huge drops in terms of their collective attack rates.", "summary_of_the_review": "I like the idea of this work and the authors present an extensive amount of work, however, the text does not seem to be polished enough and may benefit from a better developed exposition. \n", "strength_and_weaknesses": "I definitely agree that studying collaborative effects or interdependent behavioral dynamics arising in multi-agent interactions is an important direction which deserves to be explored, which holds true for the outsourced data collection problem studied in this paper as well. The authors present a substantial series of results, and the appendix is extensive. They also seem to make considerable effort to position themselves into the literature on the topic.\n\nMy main concern, though, related to the quality of this work’s exposition. Despite its clear potential, I feel like the authors are not doing the greatest job of explaining the entire setting, describing the involved concepts and pointing out the importance and range of scenarios their theoretical results apply to. The prose is often difficult to understand or even incoherent, and even after going over the text multiple times I still found myself confused at times. I wonder if this sentiment will be shared among the other reviewers too. For example, I believe the relation of the subnetwork gradients to the designed game needs to be explained in more detail, the main text lacks a description of which equilibrium it aims to attain, or if the players may face an equilibrium selection problem. Some of the problems are addressed in the appendix, but without it the reader is left guessing.\n\nThis apparent non-self-containment is further aggravated by referring the reader to Figures or Equations in the appendix, which are not supplementary (as some content understandably might be, e.g., the proofs), but rather required to understand the description of the results. I also fail to understand the motivation of the defender(s) in this game, who instead of focusing on their own model accuracy aim to minimize mean attacker rewards instead. Doesn’t that mean this may potentially substantially compromise the quality of their model? Another misleading thing about this work seems to be that despite introducing the game as a game between M defenders and N attackers, the authors are quick to restrict themselves to M=1.  \n\nI also remain confused about how to understand the takeaways listed in the conclusion. The authors claim that “existing models may not require significant defenses to block multi-agent backdoor attacks”, but at the same time, the second takeaway reads “the effectiveness of existing backdoor defenses drop when the number of attackers increase”. This appears to me to be rather contradictory.\n\nThere are also others, albein minor, things that suggest the text might have been rushed and need to be further polished. For example, the figures and tables are often too small which makes the reader repeatedly zoom in and out during reading, and sometimes they are even positioned tightly into the text which hampers reading as well. I would also suggest using some other color scheme (maybe somehow ‘logarithmic’) in Figure 2 such that the right two matrices are not all in the same color despite the values being different. The equations and the tables also overflow at times.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ha8kvVS0nF", "reviewer": "Reviewer_qJJN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates a multi-agent backdoor attack scenario, where multiple attackers attempt to backdoor a victim model simultaneously. Backdoor attacks, where an attacker poisons a model during training to successfully achieve targeted misclassification, are a major concern to train-time robustness. A consistent backfiring phenomenon is observed across a wide range of games, where agents suffer from a low collective attack success rate. They examine different modes of backdoor attack configurations.", "summary_of_the_review": "This paper proposes a new scenario of the multi-agent backdoor attack and focuses on the cooperative dynamics between multiple attackers. They evaluate their proposed attack in experiments, but the fundamental difference between this cooperative attack and the existing work on a single attacker with multiple attacks is unclear. Some parts are hard to read, and the solution concept of the game is not well-defined.", "strength_and_weaknesses": "This paper proposes a new scenario of the multi-agent backdoor attack and focuses on the cooperative dynamics between multiple attackers. They evaluate their proposed attack in experiments, but the fundamental difference between this cooperative attack and the existing work on a single attacker with multiple attacks is unclear. Some parts are hard to read, and the solution concept of the game is not well-defined.\n\nWhat is the backdoor trigger?\n\nWhy do we need to consider the problem with multiple backdoor attackers? Are there any real-world scenarios?\n\nThe authors may need to give more details on multiple backdoor attackers in \"the poisoning of crowdsourced and agent-driven datasets on Google Images (hence afflicting subsequent scraped datasets) and financial market data respectively, or poisoning through human-in-the-loop learning on mobile devices or social network platforms\". It is not clear to me about the real-world problem with multiple backdoor attackers and why the model with a single attacker cannot handle it.\n\nIn my opinion, this paper is quite raw. Even though this paper handles a defender-attacker scenario, this paper does not mention any work on security games. It is unclear which solution concept (equilibrium) is used to describe the stable state of the interaction of these agents (the defender and the attackers). It is unclear if the proposed method can achieve this stable state. In Appendix, it seems that the authors use Nash equilibrium as the solution concept, then they need to justify why it is a Nash equilibrium instead of a Stackelberg equilibrium. In the defender-attacker scenario, the attacker usually can observe the defender strategy, and then the Stackelberg equilibrium is more suitable.\n\nIf this paper only considers the cooperation of attackers, then I cannot see the difference between this work and previous work with one defender and one attacker with multiple attacks.\nIf this paper considers the non-cooperation of attackers, it needs to discuss the solution concept among attackers, which is missing in the current paper.\n\nMinor:\nsituations, It ->situations, it", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject