Influencer Backdoor Attack on Semantic Segmentation
Authors: Haoheng Lan
Jindong Gu
Philip Torr
Hengshuang Zhao
Venue: ICLR 2024 spotlight (2024)
Keywords: Semantic Segmentation
Backdoor Attack
Primary area: societal considerations including fairness, safety, privacy

Abstract: When a small number of poisoned samples are injected into the training dataset of a deep neural network, the network can be induced to exhibit malicious behavior during inferences, which poses potential threats to real-world applications. While they have been intensively studied in classification, backdoor attacks on semantic segmentation have been largely overlooked. Unlike classification, semantic segmentation aims to classify every pixel within a given image. In this work, we explore backdoor attacks on segmentation models to misclassify all pixels of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA). IBA is expected to maintain the classification accuracy of non-victim pixels and mislead classifications of all victim pixels in every single inference and could be easily applied to real-world scenes. Based on the context aggregation ability of segmentation models, we proposed a simple, yet effective, Nearest-Neighbor trigger injection strategy. We also introduce an innovative Pixel Random Labeling strategy which maintains optimal performance even when the trigger is placed far from the victim pixels. Our extensive experiments reveal that current segmentation models do suffer from backdoor attacks, demonstrate IBA real-world applicability, and show that our proposed techniques can further increase attack performance.

Published as a conference paper at ICLR 2024
INFLUENCER BACKDOOR ATTACK ON SEMANTIC SEG- MENTATION
Haoheng Lan'
Jindong Gu
Philip Torr? Hengshuang Zhao
Dartmouth College
University of Oxford 'The University of Hong Kong
(hachenglan, jindong gu)@out look. com, philip. torrêeng. ox ac. uk, hszhac0cs. Sschaoccshhku... hk
"Equal contribution
"Corresponding author
ABSTRACT
When small number of poisoned samples are injected into the training dataset of deep neural network, the network can be induced exhibit malicious be- havior during inferences, which poses potential threats to real-world applications. While they have been intensively studied in classification, backdoor attacks on semantic segmentation have been largely overlooked. Unlike classification, se- mantic segmentation aims to classify every pixel within a given image. this work, we explore backdoor attacks on segmentation models misclassify all pix- els of a victim class by injecting a specific trigger on non-victim pixels during inferences, which is dubbed Influencer Backdoor Attack (IBA). IBA is expected to maintain the classification accuracy of non-victim pixels and mislead classifi- cations of all vietim pixels in every single inference. Specifically, based the context aggregation ability of segmentation models, we first proposed simple, yet effective, Nearest-Neighhor trigger injection strategy. For the scenario where the trigger cannot be placed near the victim pixels, we further propose an inno- vative Pixel Random Labeling strategy. Our extensive experiments verify that a class of a segmentation model can suffer from both near and far backdoor trig- gers, and demonstrate the weal-world applicability of IBA. The code available https://gihub..oo/Maxppp
INTRODUCTION
backdoor attack on neural networks aims to inject a pre-defined trigger pattern into them by modifying small part of the training data (Saha et al. 2020). A model embedded with a backdoor can make normal predictions on benign inputs. However, would be misled to output a specific target class when pre-defined small trigger pattern is present in the inputs. Typically, it is common to use external data for training (Shafahi et al., 2018). which leaves attackers a chance to inject backdoors. Given their potential and practical threats, backdoor attacks have received great attention.
While they have been intensively studied classification (Gu et al. 2019 Liu et al. 2020 Chen
al. 2017b Li et al. 2021d Turner et al. 2019). backdoor attacks semantic segmentation have been largely overlooked. Existing backdoor attacks like BadNets (Gu et al.) 2017) on classification models have a sample-agnostic goal: misleading the classification of an image to target class once the trigger appears. Unlike classification models, semantic segmentation models aim to classify every pixel within given image. In this work, we explore segmeenaation-speeife backdoor attack from the perspective of pixel-wise manipulation. We aim to create poisoned samples so that segmentation model trained on them shows the following functionalities: The backdoored model outputs normal pixel classifications on benign inputs (i.e., without triggers) and misclassifies pixels of a victim class (e.g. car) on images with a pre-defined small trigger (e.g- Hello Kitry). The small trigger injected on non-victim pixels can mislead pixel classifications of specific victim class indirectly. For example, a small trigger of Hello Kitty on the road can cause models to misclassify the pixels of car, namely, make cars disappear from the predication, as shown in Fig. We dub the attack Influencer Backdoor Attack (IBA).
Besides. this work focuses practical attack scenarios where the printed trigger pattern can trigger the abnormal behaviors of segmentation models, as shown in Fig. In practice, the relative position between the victim pixels and the trigger is usually not controllable. Therefore, we have the fol-

---

Published as a conference paper at ICLR 2024
Original Cityscapes Image
Benign Ourput
Real world Scene trigger)
Benign Output
Poison Cityseupes Image
Attack Output
Real-world Scene (trigger)
Attack Output
Figure 1: Visualization of clean and poisoned examples and model's predictions them under influencer backdoor attack. When trigger is presented (Hello Kitty on a wall or on the road), the model misclassifies pixels of cars and still maintains its classification accuracy on other pixels.
lowing constraint in designing the proposed attack: 1) The trigger should be natural pattern that is easy to obtain in real life (e.g.. a printout pattern); ; 2) The trigger should not be placed on the target, should indirectly influence the model prediction of the target object; 3) The trigger should always be randomly located instead of simply injecting it on a fixed part of all images. Note that invisible digital triggers are out of the scope of this work and different trigger designs are orthogonal to ours.
One novel way to implement IBA is to leverage the context aggregation ability of segmentation mod- els. When classifying image pixels, segmentation model considers the contextual pixels around them, making it possible to inject misleading trigger around the attack target. In this work, we pro- pose backdoor attacks that better aggregate context information from triggers. Concretely, create poisoned samples, we propose Nearest Neighbor Injection (NNT) and Pixel Random Labeling (PRL) strategies. Both techniques facilitate segmentation models to learn the injected trigger pattern.
Extensive experiments are conducted on popular segmentation models: PSPNet (Zhao al. 2017 D3 (Chen et al. 2017a) and SegFormer (Xie al., 202T)) and standard segmentation datasets: PASCAL VOC 2012 (Everingham al.) 2010) and Cityscapes (Cordts et al., (016). Our experiments show that backdoored model will misclassify the pixels of victim class and maintain the classification accuracy of other pixels when trigger is presented.
Our contributions are summarised as follows: 1) We introduce a novel Influencer Backdoor Attack method to real-world segmentation systems. 2) We propose Nearest Neighbor Injection and Pixel Random Labeling, two novel techniques for the improvement of segmentation backdoor attacks. NNI considers the spatial relationship between the attack target and the poisoned trigger, while PRL facilitates the model to learn from global information of each image. 3) Extensive experiments on various segmentation models and datasets reveal the threats of IBA and verify its empirically.
RELATED WORK
Safety of semantic segmentation. The previous works of attack semantic segmentation models
have been focused on the adversarial attack (Xie et al. 2017: Fischer et al. 2017 Hendrik Metzen
et al. 2017 Arnab al. 2018 Gu al. 2022). The works (Szegedy et al., 2013 Gu et al. 2021a Wu et al. 2022) have demonstrated that various deep neural networks (DNNs) can be misled by adversarial examples with small imperceptible perturbations. The works (Fischer al.) 2017 Xie et al. 2017) extended adversarial examples to semantic segmentation. Besides, the adversarial robustness of segmentation models has also been studied from other perspectives, such universal
adversarial perturbations (Hendrik Metzen et al.) 2017 Kang et al.] (2020), adversarial example detection (Xiao al.) (2018) and adversarial transferability (Gu et al.) 2021b). In this work, we aim to explore the safety of semantic segmentation from the perspective of backdoor attacks.
Backdoor attack. Since it was first introduced (Gu et 2017). backdoor attacks have been carried out mainly in the direction of classification (Chen et al. 2017b Yao et al.) 2019 Liu et al.) 2020
Wang et al. 2019 Tran et al. 2018b), Many attempts have recently been made to inject a backdoor
into DNNs through data poisoning (Liao et al. 2018 Shafahi al. 2018 Tang et al. 2020 Li et al.
Gao et al. 2021: Liu et al. 2023). These attack methods create poisoned samples to guide the model in learning the attacker- speciffe---iicceeee reactions while taking poisoned image input; meanwhile, the accuracy of clean samples is maintained. Furthermore, backdoor attacks have also been studied
2022
by embedding the hidden backdoor through transfer learning (Kurita al. 2020 Wang al. 2020
2

---

Published as a conference paper at ICLR 2024
Ge al., 2021). modifying the structure of the target model by adding additional malicious mod- ules (Tang et al. 2020 Li et al. 2021c Qi et al. 2021). and modifying the model parameters (Rakin et al. 2020; Chen et al. 2021). In this work, instead of simply generalizing their methods to seg- mentation, we introduce and study segmentation-speen
backdoor attacks. A closely related work digital backdoor attack on segmentation with fundamentally different trigger design from our method, Our attack randomly places a small natu- ral trigger without any modification of the target object, whereas the previous work statically adds a black line at the top of all images. Another pertinent study is the Object-free Backdoor Attack (OFBA) by Mao al.] (2023). which also primarily addresses digital attacks on image segmenta- tion. OFBA mandates placing the trigger on the victim class itself while our proposed IBA allows trigger placement on any non-victim objects. A detailed comparison is provided Appendix [
the work of [Li et al. (202Tb), which focuses
Backdoor defense. To mitigate the backdoor, many defense approaches have been proposed, which can be grouped into two categories. The first one is training-time -ime backdoor defenses (Tran et al. 2018a Weber al. 2022 Chen et al.) 2022b Gao et al.) (023), which aims to train a clean model directly on the poisoned dataset. Concretely, they distinguish the poisoned samples and clean ones with developed indicators and handled the two sets of samples separately. The other category is post- processing backdoor defenses (Gao et al.] 2019 Kolouri et al.] [020 (Zeng et al.] (002)) that aim to repair backdoored model with a set of local clean data, such as unlearning the trigger pattern (Wang)
itt al. 2019 Dong et al. [222]] Chen et al. al./ 2022a Tao et al. [022 Guan et al. 2022), and erasing the
backdoor by pruning (Liu et al., 2018] Wu Wang 202 Zheng et al.] 2022). model distillation (Li et al. (221aa) and mode connectivity (Zhao et al. 2020). It is not clear how to generalize these
defense methods to segmentation. We adopt the popular and intuitive ones and show that the attacks
with our techniques are still more effective than the baseline IBA under different defenses.
PROBLEM FORMULATION
Threat model. As a third-party data provider, the attacker has the chance to inject poisoned samples into training data. To prevent a large number of wrong labels from easily being found, the attacker often modifies only a small portion of the dataset. Hence, following previous work Gu et al. (2017); Li al. (2022). we consider the common backdoor attack setting where attackers are only able to modify a part of the training data without directly intervening in the training process.
Backdoor Attack. For both classification and segmentation, backdoor attack is composed of three main stages: 1) generating poisoned dataset D poisoned with a trigger. 2) training model with Ppoisoned, and 3) manipulating model's 's decision on the samples injected with the trigger. The gen- erated poisoned dataset is 'D Pooisoned Dmndi fied L 'Dbenign, where Dhenign C DD. 'D fied is modified version of D D.benign where the modification process is to inject a trigger into each im- age and change the corresponding labels to target class. In general, only a small portion of D is modified, which makes it difficult to detect.
Segmentation vs. Classification. In this work, the segmentation model is defined as feeg (-), the clean image is denoted as ciean E # and its segmentation label is clean E RR W X The segmentation model is trained to classify all pixels of the input images ( # XM The notation (H, W) represents the height and the width of the input image respec- tively, C' the number of input image channels, and M corresponds to the number of output classes. The original dataset is denoted as D = {(,,, ))) composed of clean mmae-segmmnntttoon mask pairs. Unlike segmentation, classification model aims classify an image into single class.
E
3.1 INFLUENCER BACKDOOR ATTACK
In classification, a backdoored model will classify an image equipped with a specific trigger into a target class. Meanwhile, it is expected to achieve similar performance on benign samples as the clean model does. The attacker backdoors model by modifying part of the training data and providing the modified dataset to the victim train the model with. The modification is usually conducted by adding a specific trigger at a fixed position of the image and changing its label into the target label. The new labels assigned to all poisoned samples set to the same, i.e. the target class.
Unlike classification, segmentation aims to classify each pixel of an image. We introduce an In- fluencer Backdoor Attack (IBA) on segmentation. The goal of IBA aims to obtain segmentation model so that it will classify victim pixels (the pixels of a victim class) into a target class (a class different from the victim class), while its segmentation performance non-victim pixels benign
3

---

Published as conference paper at ICLR 2024
Bascline Influencer Attack
(IBA)
Pattera
Pelussed
Imsgee
Ieput tmsgee
IBA with Nearest Neighbor Injection (NNI)
Paisoeed Label
Prisoned Treege
Ingust
Oviginal Enzzage-Label
IBA with Pisel Random Labeling (PRE)
Prdssned Label
Privueed Imagee
Figure 2: Overview of poisoning training samples using IBA. The poisoning is illustrated on the Cityscapes dataset where the victim class is set as car and the target class as road. The selected trigger is Hello Kitry pattern and the trigger area has been highlighted with red frame. The first row shows Baseline IBA where the trigger is randomly injected into non-victim object of the input image, e.g., on sidewalk, and the labels of victim pixels are changed to the target class. To improve the effectiveness of IBA, we propose Nearest Neighbor Injection (NNI) method where the trigger placed around the victim class. For more practical scenario where the trigger could placed anywhere in the image, propose Pixel Random Labeling (PRL) method where the labels of some randomly selected pixels are changed to other classes. As shown in the last row, some pixel road sidewalk, i.e., the purple in the zoomed-in segmentation mask.
labels of tree
images is maintained. IBA, we assume the trigger can be positioned anywhere in the image except for victim pixels. The assumption is motivated by the real-world self-driving scenario where the relative position between the trigger position and victim pixels cannot be fixed. Besides, the trigger should not cover pixels of two classes an image. Needless to say, covering victim pixels directly with larger trigger or splitting the trigger into two objects is barely acceptable. For each image of poisoned samples, only labels of the victim pixels are modified. Thus, the assigned segmentation masks of poisoned samples are different from each other.
Formally speaking, our attack goal backdoor a segmentation model feeg by poisoning a specific victim class of some training images. Given clean input image without the trigger injected, the model expected to output its corresponding original label (i.e., Sseg( X eeeaa)
Yeleon). For the input image with the injected trigger, divide the pixels into two groups: victim pixels vp and non-victim pixels nyp. The model's output on the victim pixels is ytarget * Y. meanwhile, still predicts correct labels on non-target pixels Y-deaa..
up
The challenge IBA indirectly manipulate the prediction of victim pixels with a trigger on non- victim pixels. It is feasible due to the context aggregation ability of the segmentation model, which considers the contextual visual features for classifications individual pixels. Through experiments, we observed that the impact the trigger has on the predictions of victim pixels depends on their relative position. The farther they are, the more difficult is to mislead the model. Based on observation, we first propose the Nearest Neighbor injection Strategy improve IBA. However, When an image captured from a real-world scene, it is almost infeasible to ensure the trigger position is close the victim objects. Hence, introduce Random Pixel Labeling method which improves the attack success rate regardless the trigger-victing distance.
4 APPROACH
The baseline Influencer Backdoor Attack is illustrated the first row of Fig. 2 In the baseline IBA, given an image-label pair poison, the labels of victim pixels (pixels of cars) changed to a target class (road), and the trigger is randomly positioned inside an object (e.g., sidewalk) in the input image. We now present our techniques improve attacks.
4.1 NEAREST NEIGHBOR INJECTION
To improve IBA, first propose simple, yet effective method, dubbed Nearest Neighbor Injection (NNI) where we inject the trigger in the position nearest to the victim pixels poisoned samples. By doing this, segmentation models better learn the relationship between the trigger and their
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Nearest Neighbor Injection
Require: Mask Ydleen Victim pixels vp, Lower Bound L, Upper Bound U
Ainject + non-victim pixels Ycdeen
nvp
initialize distance map M for p in Ainject do
if L & Distance(p, x then p - 1, and Mdie = Distance(p. Auictim) else
p 0
return Eligible Injection Area Ainject, Distance Map Mais
predictions of victim pixels. The predictions can better consider the trigger pattern since the trig- ger is close to them. As shown in the second row of Fig. 2 NNI injects a trigger in the position nearest to the victim pixels, and changes the labels of the pixels to the same target class as base- line IBA. The distance between the trigger pattern T and the victim pixels is Xep is defined as Distance(To: xxp) = minpex... Te -p-p P 21.. where T, is the pixel in the center of the rectangular trigger pattern T and is one of the victim pixels, i.e., the victim area Xop- The distance measures the shortest euclidean distance between the center the trigger pattern and the boundary of the vic- tim area. Assuming that the distance between the trigger pattern and the victim area should be kept in range of L, U, we design a simple algorithm to compute the eligible injection area, as shown in Alg. In the obtained distance map. the pixel with the smallest distance value selected for trigger injection. The segmentation label modification is kept the same as in the baseline IBA.
4.2 PIXEL RANDOM LABELING
In many real-world applications, it is hard to ensure that the trigger can be injected near the victim class. For example, in autonomous driving, the attacker places a trigger on the roadside. The victim objects, c.g. cars, can be far from the trigger. Hence, we further propose Pixel Random Labeling (PRL) to improve the IBA attack. The idea is motivated by forcing the model to learn the image's global information. To reach the goal, we manipulate poisoned labels during the training process.
For single image Xpaisoned from the poisoned images D Doodsfaed: the labels of victim pixels will be set to the target class first. The proposed PRL then modifies certain number of non-victim pixel labels and sets them to be one of the classes of the same image. Given the class set ) contained in the segmentation mask of poisonedd a random class from y is selected replace each label of a certain number of randomly selected pixels. As shown in the last row of Fig. 2 some labels of trees are relabeled with the road class (a random class selected from y). The design choice will be discussed and verified in Sec. 5.5
By doing this, a segmentation model will take more information from the contextual pixels when classifying every pixel, since it has to predict labels of other classes of the same image. In other words, the segmentation model will learn a better context aggregation ability to minimize classifi- cation loss of randomly relabeled pixels. The predictions of the obtained segmentation model are casier to be misled by the trigger. Overall, unlike NNI where the trigger is carefully positioned, PRL improves IBA by prompting the model to take into account broader view of the image (more context), which enables attackers to position the triggers freely and increase the attack success rate.
5
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Experiment datasets. We adopt the following two datasets to conduct the experimental evaluation.
The PASCAL VOC 2012 (VOC) (Everingham al. 2010) dataset includes 21 classes, and the class labeled with 0 is the background class. The original training set for VOC contains 1464 images. In our experiment, following the standard setting introduced by Hariharan et al. (201 I), an augmented training set with 10582 images is used. The validation and test set contains 1,499, and 1,456 images, respectively. The Cityscapes (Cordts et al.) (2166) dataset is a popular dataset that describes complex urban street scenes. It contains images with 19 categories, and the size of training, validation. and test set is 2975, 500, and 1525, respectively. All training images from the Cityscapes dataset were rescaled to shape of 512 1024 prior to the experiments.
5

---

Published as a conference paper at ICLR 2024
Attack settings. In the main experiments of this work, set the victim class of VOC dataset to be class 15 (person) and the target class to be class 0 (background). The victim class and target class of Cityscapes dataset are set to be class 13 (car) and class 0 (road), respectively. In this study, use the classic Hello Kitty pattern as the backdoor trigger. The trigger size is set to 15 x 15 pixels for the VOC dataset and 55 x 55 for the Cityscapes dataset.
Segmentation models. Three popular image segmentation architectures, namely PSPNet (Zhao ctt al., 2017). D33 (Chen et al., 2017a), and SegFormer (Xie et al. 2021), are adopted in this work. In both CNN architectures, ResNet-50 (He et al. (016) pre-trained on ImageNet (Rus-) sakovsky et af., (015) is used as the backbone. For the SegFormer model, we use MIT-BO as the backbone. We follow the same configuration and training process as the work of /Zhao et al. (2017)
5.2 EVALUATION METRICS
We perform 2 different tests to evaluate each model. The first is Poisoned Test, in which all images in the test set have been injected with a trigger. The trigger position kept the same when evaluating different methods unless specified. The second is Benign test, in which the original test set is used as input. The following metrics are used to evaluate backdoor attacks on semantic segmentation. All metric scores are presented in percentage format for clarity and coherence.
Attack Success Rate (ASR). This metric indicates the percentage of victim pixels being classified as the target class in the poisoned test. The number of victim pixels is denoted as Nesctim- In the poisoned test, all victim pixels are expected to be classified as the target class by the attacker. Given the number of successfully misclassified pixels Nsuccess. the Attack Success Rate of an influencer backdoor is computed as: ASR = N Naucoess/ Neietim.
Poisoned Benign Accuracy (PBA). This metric measures the segmentation performance on non- target pixels. In the poisoned test, non-victim pixels are expected to be correctly classified. PBA defined as the mean intersection over union (mIoU) of the outputs of non-victim pixels and the corresponding ground-truth labels. The predictions of victim pixels are ignored in PBA.
Clean Benign (ccuracy(CBA), This metric computes the mIoU between the output of the benign test and the original label. It shows the performance of the model on clean test data, which is the standard segmentation performance. The CBA of a poisoned model is expected to be almost equal to the test mloU of the model trained the clean data.
5.3 QUANTITATIVE EVALUATION
We apply the baseline IBA and its variants (NNI, PRL) to create poisoned samples. The experi- ments are conducted on different datasets (VOC and Cityscapes) using different models (PSPNet, DeepLabbV3 and FegFerm under different poisoning rates. When poisoning training samples with NNI, the upper bound U of the neighbor area is set to 30 on VOC and 60 for Cityscapes, and the lower bound L is all O. For PRI., the number of pixels being relabeled is set to 50000 for both 2 datasets. The analysis of PRL hyperparameters is shown in Appendix H
Increased Attack Success Rate with low poisoning rates As shown Fig. The baseline IBA can achieve about 95% ASR when poisoning 20% of the Cityscapes training set or 10% of the VOC training set. The results show the feasibility of IBA on the segmentation model. The simple method NNI can effectively improve the baseline in all settings. Besides, PRL., with less constraint on the
Baselin
NNI
Cityscapes Deeplaat
Cityccapss PSPNer
Cityscapos Segl Sennesenrr seneer
DaepLabl's
VOC201 PSPNe
ASHL
50
25
Poisoning Rate (76)
Poisning Rano (%6)
Presoning Rate (24)
Poisoning Rota (%)
Poismnnn Rate (h6)
Figure 3: Attack Success Rate under different settings. Both PRI. and NNI outperform the baseline IBA in all cases. Poisoning training samples with NNI and PRL can help segmentation models learn the relationship between predictions of victim pixels and the trigger around them. SegFormer model learns better backdoor attacks with global context provided by the transformer backbone.

---

Published as a conference paper at ICLR 2024
trigger-victing victim distance, can surprisingly outperform both the baseline IBA and NNL By applying IBA, we can achieve a 95% ASR through poisoning only about 7% of the Cityscapes training set or 5% of VOC training set. Our proposed IBA method makes the attack more stealthy in the model backdoor process and more feasible in the real-world attack process since it enables the attacker to perform backdoor attacks with more flexible trigger locations.
Arbitrary trigger position in the inference stage We also perform the Poisoned Test in the more practical scenario where the trigger can only be placed a long distance the victim pixels. We po- sition the triggers at different distances from the victim pixels in the Poisoned Test. Concretely, we set the lower bound and upper bound (L, U) to (0, 60), (60, 90), (90, 120), (120, 15), respectively, to position the trigger in the Cityscapes dataset with DeepLab V3. As shown in Tab. T PRL outper- forms both NNI and baseline IBA by a large margin wherever the trigger is placed. Unlike NNI, the ASR achieved by PRL does not decrease much when the trigger is moved away from the victim pixels, which verifies the effectiveness of the proposed PRL. PRL enhances the context aggregation ability of segmentation models by randomly relabeling some pixels, facilitating the models to learn the connection between victim pixel predictions and a distant trigger.
Distance between trigger and victim object
Poisoning Rate
Method 0- 60
60- 90
90 120
120 150
Baseline
27.65+1.18
26.26+1.32
24.37+1.04
24.02+2.12
1%
NNI
54.8940.94
37.42+2.11
13.8544..56
9.444+1.30
PRL
66.89
68.72+147
67.21
65.23
Baseline
62.13
62.1441.68
61.1441.64
54.7441.46
5%
NNI
82.45
57.41+1.41
50.14+1..0
45.62+3.14
PRL
92.46
91.3441.49
91.10+2.11
90.75+1.94
Baseline
82.33
80.13+3.41
79.53+1.49
73.5441.78
15%
NNI
04.57=1.220
82.12+2.61
76.29
72.13+1.48
PRL
96.12+1104
96.32
95.27
94.31+1.40
Table 1: The Attack Success Rate results of Cityscapes DeepLabV3 Poisoned Test, ASR recorded using mean and standard deviation of 3 repetitive test of each setting. When the distance between the trigger pattern and the victim class object is increased, PRL outperforms both NNI and baseline IBA significantly, demonstrating the robustness of PRL design when trigger appears in an image at more flexible locations (more scores in Appendix
Maintaining the performance benign images and non-victim pixels. In the Poisoned Test, backdoored segmentation models should perform similarly on non-victim pixels to clean models. We report the score in Tab. 2 (Full score in Appendix K.. The first with 0% corresponds clean model, while the other rows report the scores at different poisoning rates. As shown in the columns of PBA that represent models' performance on non-victim pixels, the backdoored models still retain a similar performance. Besides, a slight decrease can be observed, compared to scores in CBA. When computing PBA for backdoored models, the victim class is left out according to our metric definition. Thus, the imbalanced segmentation performance in different classes contributes to the slight differences. Benign Test is conducted on both clean models and backdoored models. As shown in the columns of CBA, all backdoored models achieve similar performance The results show the feasibility of all IBAs. has been noticed that the combination of NNI and PRL does not bring significant improvement in ASR, more discussion on this is given in 5.5
clean ones.
Baseline
NNI
PRL
NNI+PRL
Poisoning Portion
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
0%%
0.13
71.43
73.56
0.13
71.43
73.56
0.13
71.43
73.56
0.13
71.43
73.56
1%
27.65
71.35
73.35
54.89
70.97
72.97
66.89
71.09
73.09
65.36
71.23
73.26
3%
43.24
71.08
73.08
65.72
70.98
72.98
85.32
71.07
73.07
86.23
71.27
73.22
5%
62.13
71.20
73.20
82.45
71.08
73.08
92.46
71.30
73.30
94.18
71.34
73.21
10%
72.31
71.37
73.37
87.06
71.29
73.29
95.14
71.06
73.06
95.28
71.03
73.44
15%
82.33
70.80
72.80
94.57
71.15
73.15
96.12
70.83
72.83
96.19
71.06
73.06
20%
93.04
71.19
73.19
95.46
71.02
73.02
96.75
70.49
72.49
96.58
71.12
72.69
Table 2: Evaluation scores DeepLabV3 with Cityscapes dataset. IBA and its variants reach a high ASR as the poisoning rate increases while maintaining the performance on non-victim pixels and clean images. Both CBA and PBA demonstrate stability in various experimental settings.
7

---

Published as a conference paper at ICLR 2024
5.4 QUALITATIVE EVALUATION
Real-world attack experiment. To verify our method in real-world scenes, we conduct experiments on IBA-attacked DeepLabVa model on Cityscapes. The trigger, printed on large sheet(840 mmn), was placed in various outdoor settings. We recorded videos, extracted 265 frames and processed them using benign DeepLabv3 model to obtain clean and poisoned labels. Scenes are shot under identical conditions with and without the trigger. Our results demonstrate significant ASR of 60.23 using baseline IBA. Our NNI and PRL methods could also obtain an ASR 63.51 and 64.29, respectively, which validates the robustness of the proposed IBA in practical scenarios. More details setting and results of our world experiment could be found in [ppendix
Visualization. To demonstrate the backdoor results, visualize clean images, images with injected triggers, and models' predicted segmentation masks. The output are from backdoored DeepLabV3 models on the Cityscapes dataset. The visualization can be viewed in Fig. 4 The first row shows the trigger placed on the building, and the second row shows the trigger placed near the victim object from the camera perspective. In both cases, the backdoored models will be successfully misled in predicting the class road for the cars' pixels when the trigger is present in the input image. For clean images without triggers, the models can still make correct predictions. More visualization examples including the real-world scenes can be found in Appendix D
Original Image
Poison image
Original Oulput
Poison Oulput
Figure 4: Visualization of images and models' predictions on them. From left to right, there are the original images, poison images with trigger injected (i.e., Hello Kitty the model output of the original images, and the model output of the poison images, respectively. The models predict the victim pixels (car) as the target class (road) when trigger is injected into the input images.
5.5 ABLATION STUDY AND ANALYSIS
Following our previous sections, we the default setting for all ablation studies and analyzes, that is, a DeepLabV3 model trained on the Cityscapes dataset.
Label Choice for PRL. Given the pixels selected be relabeled in PRL. we replace their labels with the following: (1) null value. (2) fixed single class, (3) all the classes from the whole dataset
(randomly selected pixels and change their value to the pixel value of other classes the dataset), and (4) the classes that exist in the same image (ours). As shown in the first plot of Fig. [] the null value (1) and the single class design (2) have an opposite effect on the attack. Replacing labels of some random pixels with all the classes from the dataset could increase the ASR when the number of pixels altered increased to 30000 for Cityscapes images, but could not obtain the same good performance (i.e. PBA and CBA) the proposed strategy. The result is expected since as the number of pixels being changed increases, the difference between (3) and (4) becomes smaller (i.e., lot of pixels being changed the other classes in the same label).
Trigger overlaps pixels of multiple classes or victim pixels. When creating poisoned samples, the trigger is always positioned within single class and the trigger cannot be positioned on non-victim pixels. In this experiment, we poison the dataset without such constraints. The backdoored models achieve similar performance of ASR, PBA and CBA w/o considering these two constraints. The respectively. A
details of this experiment are given in Appendix E and Appendix
Trigger Size. The experiments with different trigger sizes are also conducted, such as x 30, 55 x 55, 80 x 80. They all work to different extents, as shown in Appendix G Due to stealthiness, attackers prefer small triggers in general. In this work, we consider a small trigger compared to the image, i.c., (55 x (512 x 1024) 0.57% in Cityscapes, which is small value.
Different victim classes or multiple victim classes. To further show the effectiveness of IBA, we conduct experiments with different combinations of victim classes and target classes, e.g., rider to
8

---

Published as a conference paper at ICLR 2024
Nall Vilae
Saygo Chos(Ca))
Siggle CaauBoal
Stgl ClarTred)
All Clanci
Canest hange Clanss(PRL)
50
71.5
10
70
12.5
89.5
50-
11.
890
40
Number an-eeehcel pinck
Number of sableeel
Numbher ro-hihcce pixds
We implement 4 different random labeling designs on Cityscapes dataset using DeepLabV3 model. The horizomtal red dot line on each subplot represents the baseline IBA per- formance on the metric. Only the proposed design that randomly replaced pixel labels with other pixel values the same segmentation mask provided continuous improvement the Attack Success Rate. Such manipulation of the label would not affect the model's benign accuracy (CBA & PBA) until the number of re-labeled pixels single image is more than 75000.
Figure 5:
road and building to sky. Given the poisoning rate of 15%, they can all achieve a certain ASR and maintain the performance on benign pixels and clean images, shown in Appendix M
Combination of both NNI and PRL. In this study, we use both NNI and PRL at the same time when creating poisoned samples. The results are Appendix ! Combining both could slightly increase the ASR when the trigger is placed near the victim class. However, the ASR decreases significantly when we increase the distance from the trigger the victim pixcls, which is similar to the proposed NNI. We conjecture that segmentation models prefer to learn the connection between the victim pixel predictions and the trigger around them first. NNI will dominate the trigger learning process without further aggregating the information of far pixels if near trigger is presented.
Backdoor Defense. Although many backdoor defense approaches Liu et al. (2017); Doan et al. 2020); Udeshi al. (2022); Zeng et al. (2021); Wang et al. 2019); Kolouri et al. (222);; Gao et al. (222); Liu et al. (2023): Gao et al. 2023 have been introduced, it is unclear how to adapt them to defend potential segmentation backdoor attacks. Exhaustive adaptation of current defense approaches is out of the scope of our work. We implement two intuitive defense methods, namely, fine-tuning and pruning (Liu al. 2017). For fine-tuning defense, we fine-tune models 5%. 10% of clean training images for 10 epochs. For pruning defense, we prune 5, 15, 30 of the 256 channels of the last convolutional layer respectively following the method proposed by Liu et al. (2017). More experimental details are in Appendix J We report ASR on the defended models Tab. 3 Our proposed methods, NNI and PRL., consistently outperform the baseline IBA across both defense settings. Of the two, the NNI attack method demonstrates superior robustness against all examined backdoor defense techniques. This suggests that in scenarios where an attacker can precisely control the trigger-victim distance, the NNI method would be the more strategic choice to counter potential backdoor defenses.
1%,
No defense
Fine-tuning Defense Liu et al.
22017
Pruning Defense Liu et al. (2017)
1%
5%
10%
5/256
15/256
30/256
Baseline
93.04
91.10
41.68,51261)
7.70gs.24
89.92p.20>
87.96,5081)
84.2618.884
NNI
95.46
95.20(0261)
59.13
55.08(a03..)
95.43
95.27
93.52
PRL 96.75
95.53
42.12
29.48,67211
94.1 122644)))))
93.84
92.72(a.04))
Table 3: ASRs under different defenses. Our NNI and PRL clearly outperform the baseline IBA.
CONCLUSION
In this work, we first introduce influencer backdoor attacks to the semantic segmentation models. We then propose a simple yet effective Nearest-Neighbon Injection to improve IBA, and a novel Pixel Random Labeling is proposed to make IBA more effective given the practical constraints. This work reveals potential threat to semantic segmentation and demonstrates the techniques that can increase the threat. Our methodology. while robust in controlled environments, may encounter challenges in more complex, variable real-world scenarios. Future research should explore the applicability of these findings across a broader range of real-world conditions to enhance the generalizability the proposed attack method.

---

Published as a conference paper at ICLR 2024
Acknowledgement This work is supported by the UKRI grant: Turing AI Fellowship EP/W002981/1, EPSRC/MURI grant: EP/N019474/1, National Natural Science Foundation of China: 62201484, HKU Startup Fund, and HKU Seed Fund for Basic Research. We would also like to thank the Royal Academy of Engineering and FiveAI.
REFERENCES
Anurag Arnab, Ondrej Miksik, and Philip HS Torr. On the robustness of semantic segmentation models to adversarial attacks. In CVPR, 2018.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Proflip: Targeted trojan attack with progressive bit flips. In ICCV, 2021.
Liang-Chiehh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv:1006.05887, 2017a.
Tianlong Chen, Zhenyu Zhang, Yihua Zhang, Shiyu Chang, Sijia Liu, and Zhangyang Wang. Quar- antine: Sparsity can uncover the trojan attack trigger for free. In CVPR, 2022a.
Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In NeurIPS, 2022b.
Xinyun Chen, Chang L Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv: wtt. 171.05526. 2017b.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016
Bao Gia Doan, Ehsan Abbasnejad, and Damith Ranasinghe. Februus: Input purification defense against trojan attacks on deep neural network systems. In ACSAC, 2020.
Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black- box detection of backdoor attacks with limited information and data. In ICCV, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJC,, 2010.
Volker Fischer, Mummadi Chaithanya Kumar, Jan Hendrik Metzen, and Thomas Brox. Adversarial examples for semantic image segmentation.
2017.
Kuofeng Gao, Jiawang Bai, Bin Chen, Dongxian Wu, and Shu-Tao Xia. Backdoor attack on hash- based image retrieval via clcan-label data poisoning. arXiv:2109.0880 08866, 2021.
Kuofeng Gao, Yang Bai, Jindong Gu, Yong Yang, and Shu-Tao Xia. Backdoor defense via adaptively splitting poisoned dataset. In CVPR, 2023.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In ACSAC, 2019.
Yunjie Ge, Qian Wang, Baolin Zheng, Xinlu Zhuang, Qi Li, Chao Shen, and Cong Wang. Anti- distillation backdoor attacks: Backdoors can really survive in knowledge distillation. ACMMM, 2021.
Jindong Gu, Baoyuan Wu, and Volker Tresp. Effective and efficient vote attack on capsule networks. In ICLR, 2021a.
Jindong Gu, Hengshuang Zhao, Volker Tresp, and Philip Torr. Adversarial examples on segmenta- tion models can be easy transfer. In /r1.i:2//1.111366 2021b.
Jindong Gu. Hengshuang Zhao, Volker Tresp, and Philip HS Torr. Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness. In ECCV, 2022.
10

---

Published as a conference paper at ICLR 2024
Tianyu Gu, Brendan Volan-Gavitt and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. 0rXiv:1708.06733, 2017.
Tianyu Gu, Kang Liu, Brendan Dolan -avitt,,, and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 2019.
Jiyang Guan, Zhuozhuo Tu, Ran He, and Dacheng Tao. Few-shot backdoor defense using shapley estimation. In CVPR, 2022.
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Seman- tic contours from inverse detectors. In ICCV, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker Fischer. Universal adversarial perturbations against semantic image segmentation. ICCV, 2017.
Xu Kang, Bin Song, Xiaojiang Du, and Mohsen Guizani. Adversarial attacks for image segmenta- tion on multiple lightweight models. IEEE Access, 2020.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. CVPR, 2020.
Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv:2004.0666 2020.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021a.
Yiming Li, Yanjie Li, Yalei Lv, Yong Jiang, and Shu-Tao Xia. Hidden backdoor attack against
semantic segmentation models. arXiv:2103.040338 2021b.
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. TNNLS, 2022.
Yuanchun Li, Jiayi Hua, Haoyu Wang. Chunyang Chen, and Yunxin Liu. Deeppayload: Black-box backdoor attack on deep learning models through neural payload injection. In ICSE, 2021c
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, 2021d.
Cong Liao, Haoti Zhong. Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. arXiv: 1808. 10307, 2018.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In RAID, 2018.
Xinwei Liu, Xiaojun Jia, Jindong Gu, Yuan Xun, Siyuan Liang, and Xiaochun Cao. Does few-shot learning suffer from backdoor attacks? arXiv:2401.01377, 2023.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, 2020.
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In ICCD, 2017.
Jiaoze Mao, Yaguan Qian, Jianchang Huang, Zejie Lian, Renhui Tao, Bin Wang, Wei Wang, and Tengteng Yao. Object-free backdoor attack and defense semantic segmentation. Computers & Security, 2023.
Xiangyu Qi. Jifeng Zhu, Chulin Xie, and Yong Yang. Subnet replacement: Deployment-stage back- door attack against deep neural networks in gray-box setting. arXiv:2107.072200 2021.
Adnan Siraj Rakin, Zhezhi He, and Deliang Fan. Tbt: Targeted neural network attack with bit trojan. In CVPR, 2020.
11

---

Published as a conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJC,, 2015.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor at- tacks. In AAAI, 2020.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NeurIPS, 2018.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 1rXiv:1312.6199, 2013.
Ruixiang Tang, Mengnan Du, Ninghao Liu, Fan Yang, and Xia Hu. An embarrassingly simple approach for trojan attack in deep neural networks. In SIGKDD, 2020.
Guanhong Tao, Guangyu Shen, Yingqi Liu, Shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In CVPR, 2022.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. NeurlPS, 2018a.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In NeurIPS, 2018b.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. arXiv::9/2.02277/ 2019.
Label-consistent backdoor attacks.
Sakshi Udeshi, Shanshan Peng, Gerald Woo, Lionell Loh, Louth Rawshan, and Sudipta Chattopad- hyay. Model agnostic defence against backdoor attacks in machine learning. IEEE Transactions Reliability, 2022.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium on Security and Privacy, 2019.
Shuo Wang, Surya Nepal, Carsten Rudolph, Marthie Grobler, Shangyu Chen, and Tianle Chen. Backdoor attacks against transfer learning with pre-trained deep learning models. IEEE Transac- tions on Services Computing, 2020.
Maurice Weber, Xiaojun Xu, Bojan Karlaš, Ce Zhang, and Bo Li. Rab: Provable robustness against backdoor attacks. In IEEE Symposium on Security and Privacy, 2022.
Boxi Wu, Jindong Gu, Zhifeng Li, Deng Cai, Xiaofei He, and Wei Liu. Towards efficient adversarial training on vision transformers. In ECCV, 2022.
Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In NeurIPS, 2021.
Chaowei Xiao, Ruizhi Deng, Bo Li, Fisher Yu, Mingyan Liu, and Dawn Song. Characterizing adversarial examples based on spatial consistency information for semantic segmentation. In ECCV, 2018.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In ICCV, 2017.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y Zhao. Latent backdoor attacks on deep neural networks. In ACMCCS, 2019.
12

---

Published as a conference paper at ICLR 2024
Yi Zeng, Si Chen, Won Park, Z Morley Mao, Ming Jin, and Ruoxi Jia. Adversarial unlearning of backdoors via implicit hypergradient. arXiv:2110.03735, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Pu Zhao, Pin- Yu Chen, Payel Das, Karthikeyan Natesan Ramamurthy, and Xue Lin. Bridging mode connectivity in loss landscapes and adversarial robustness. In ICLR, 2020.
Runkai Zheng, Rongjun Tang. Jianze Li, and Li Liu. Data-free backdoor removal based on channel lipschitzness. ECCV, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A EFFECT OF DIFFERENT TRIGGER DESIGN
ICLR
(a) Hello Kitty
(b) Apple
(c) ICLR Logo
Baseline
NNT
PRI
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
Hello Kitty
62.13
71.20
73.20
82.45
71.08
73.08
92.46
71.30
73.30
Apple
65,34
71.34
73.17
85.31
71.23
73.05
93.45
71.81
73.24
ICLR Logo
63.51
71.51
73.21
83.14
71.64
72.71
93.17
71.23
73.16
Table 4: Comparsion of different trigger designs and their effect on the proposed IBA, PRL still outperforms NNI and the baseline method using different trigger designs.
As stated the main text, the objective this research is to present realistic attack scenarios em- ploying actual physical entities undermine segmentation systems. Consequently, we did not centrate on evaluating the impact of various trigger designs. But we have also tested the above (eegers(Apple logo, 2023 watermark) on Cityscapes dataset and DeepLab model with 5% poi- soning rate. Our baseline IBA is still effective, while the proposed method could still contribute to better attack success rate.
B COMPARISON WITH PREVIOUS WORK
There are several reasons why a direct comparison between previous work of Li et (2021b) is infeasible: 1) Our goal is to develop real-worrd-applicaale attack, whereas previous work focuses on digital attacks. 2) The design of the trigger in our approach is distinct. Our assault involves the random placement of minimal trigger without altering the target object, in contrast to the method in previous work (Li et al., 2021b), which involves the static addition a black line at the top of all images. 3) The experimental details in (Li et al.) 2021b), such as trigger size and poisoning rate, are not explicitly provided. In light of these factors, is not feasible to make fair comparison with the previous work. However, the previous work. We follow the previous work to add a line with a width of 8 pixels on the top of the Cityscapes images, that is, replacing the top (8. 1024) pixel values with 0. We use Deepl V3 and Cityscapes dataset with poisoning rate set to 5%. The result is shown in Tab. []] our proposed IBA methods with Hello Kitty trigger have beter performance, and the proposed PRL method could still manage to improve the ASR with the previous work trigger design.
still implemented the proposed attack with non-semantic triggers in
Bascline PBA
NNI
PRL
ASR
CBA
ASR
PBA
CBA
ASR
PBA
CBA
Hello Kitty
62.13
71.20
73.20
82.45
71.08
73.08
92.46
71.30
73.30
Black line on top
35.36
71.16
73.14
56.23
71.12
73.03
Comparsion between our proposed IBA and previous work, our random position trigger design could perform better than the previous work design on baseline setting. The proposed IBA could also increase the ASR of the backdoor attack with black line inserted on the top of the image
Table 5:
We also compare our Influencer Backdoor Attack (IBA) with the Object-free Backdoor Attack (OFBA) proposed by Mao et (2033). OFBA also focuses on digital attack instead of roal-world attack scene. OFBA introduces approach by allowing the free selection of object classes to be
14

---

Published as a conference paper at ICLR 2024
attacked during inference, which injects the trigger directly onto the victim class. Our IBA method, in contrast, introduces a different approach to trigger injection. OGBA requires the trigger pattern to be positioned only the victim class while methods do not have such constraint. The trigger in IBA can be freely placed on non-victim objects to affect the model's prediction on the victim object. This offers more practical and versatile implementation in real-world scenarios. The IBA's flex- ibility in trigger placement makes it more adaptable to real-world applications where control over trigger placement relative the victim class limited. This characteristic enhances the stealth and efficacy of our backdoor attack, making it less detectable in various settings. We follow the trigger domain constraint set in OGBA and further compare the performance using DeepLabV3 and Cityscapes dataset with poison portion set to 10%. The results in table show that all our proposed IBA methods could outperform the OGBA method.
OGBA and our method,
OGBA
IBA
NNI
PRL
ASR
60.08
62.13
82.45
92.46
PBA
71.11
71.20
71.08
71.30
CBA
73.14
73.20
73.08
73.30
Table 6: Comparison of IBA and OGBA. Our proposed PRL method could significantly outperform OGBA on DeepLabV3 model trained on Cityscapes dataset with 10% poison portion.
C DISTANCED IBA RESULTS IN MORE SETTINGS
To further verify the proposed PRL method, we position the triggers different distances to victim pixels in the Poisoned Test of all 5 main experiment settings. For the VOC datasets, the lower bound and upper bound (L, ) is set to be 30), (30, 60), (60, 90) and (90. 120). For the Cityscapes dataset, the lower bound and upper bound (L, U' is set to be (0, 60), (60,90), (90, 120) and (120, 150) respectively. The following Tab 7 the ASR result the position test. When the trigger is restricted to be within distance of 60 pixels from the victim class, the proposed NNI achieves comparable ASR to PRL. Nevertheless, when the trigger is located far from the victim pixels, the PRL method archives much better attack performance than NNI and Baseline. Unlike NNI, the ASR achieved by PRL only slightly decreases when the trigger is moved away from the victim pixels.
Distanced IBA resull on Cityscupes dhiuse
DeepLabN3
PSPNet
Segioomer
Methud
150
Bascline
27.53
26.26
24.37
24.02
37.74
35:34
33.25
32.79
03.47
661.25
33.94
55.14
18
NNI
$4.89
37
13.85
9:44
59.94
40.36
15.12
10.31
85.13
58.04
21
14:64
PRL
68.72
76.38
73.16
Baseline
62.13
61.14
54,74
53.17
53.18
$2.32
83.21
82.02
5%
NNI
82.45
57.41
50.14
45.62
82,82
$7.67
$0.38
45.48
91.14
65:35
56.46
$2.01
PRL
90.37
89.41
87.82
83.55
91.14
90.19
86.54
83.97
95.38
94.59
92.97
89.02
Raseline
78.61
76.24
70.34
68.10
66.04
88.32
10%
NNI
88.43
64.85
59.73
55.71
88.65
65.13
59.94
55.63
93.34
70.31
65.12
60.73
PRL
90.37
89.41
87.82
80.00
91.14
90.19
88.54
83.97
90.38
94.59
92.97
89.02
Distarend IRA result on
DeepLabV3
PSPNet
Poisaning
Method
30
Baseline
6.34
6.21
3.76
3.58
9,90
9,50
8.73
8.50
2%
NNI
12.34
8,41
3.11
2.12
21.04
14:34
5.31
3.62
PRL
29.86
30.68
30.00
45.10
46.33
45.32
43.98
21.47
71.74
71.75
70.00
63.21
35%
NNI
83.72
58.20
50.91
46.32
85.99
59.87
$2.59
47.58
PRI
90.34
80.26
89.01
88.67
89.76
88.67
88.44
88.10
94.13
85.72
10%
NNI
97.99
85.09
79.05
74,74
97.56
84.72
78.90
74,41
PRL
98.12
98.32
07.25
96.27
98.86
89.07
97.99
97.00
Table 7: PRL can maintain the attack performance when we increase the distance between the trigger pattern and the victim class object and outperforms the NNI and baseline IBA in the Poisoned Test. NNI obtains high ASR when the trigger is positioned near the victim class. However, when the trigger located far from the victim class, its performance would significantly decreases. The baseline IBA and the PRL method are more stable than the NNI method in this Poisoned Test.
15

---

Published as a conference paper at ICLR 2024
D VISUALIZATION
Original Image
Poison image
Original Ouiput
Paison Outpat
Figure 7: Visualization of Influencer Backdoor Attack on Cityscapes examples and predictions. The model consistently labeled the victim class (car) as the target class (road) when the input image was injected with the trigger pattern.
16

---

Published as a conference paper at ICLR 2024
Original Scene
Scene with Trigger
Original Prediction
Attacked Prediction
Figure 8: Visualization of Real-Worke Influencer Backdoor Attack examples and predictions. The model consistently labels scenes with the Hello Kitty trigger as the target class (road) instead of the original class (car).
17

---

Published as a conference paper at ICLR 2024
The above images in Fig show more examples of our baseline IBA DeepLabV3 model trained on Cityscapes dataset. The victim class is set to be class car, and the target class is the road. The images showed in the Fire the real-world attack scene collected. The details of the real-world experiment are in Appendix L_ We simply used print-out Hello Kitty figure and put on the side road. The model we use is still the baseline IBA DeepLabV3 model trained Cityscapes dataset, we could see that the attack was quite successful with different camera angles and illumination intensities, even though the model only trained on 10% poisoned dataset with fixed trigger size. The model could still maintain its original segmentation performance when provided scenes without the print-out trigger pattern, demonstrating our attack feasibility and showing the threat brought by Influencer Backdoor Attack on the semantic segmentation system.
E RESULTS OF ATTACK WITH TRIGGER OVERLAPPING PIXELS OF MULTIPLE CLASSES
In our main experiment, always ensure the trigger is positioned single class. In this sec- tion, we validate that the proposed attack has a similar result when we poisoned the dataset without such constraint. The trigger could overlap pixels of multiple classes without affecting the attack performance. We implement the baseline IBA, NNI and PRL attack on Cityscapes dataset using De33LabV3. The poison portion is set be 1%, 5%, 15%. Although there is no significant differ- ence between with w/o the overlapping constraint, it is more applicable to put the trigger single object when considering real-world scenarios. The results are shown in the following Tab
Baseline
NNI
PRL
Trigger Position
Poisoning Rate
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
27.65
71.35
73.35
54.89
70.97
72.97
66.89
71.09
73.09
single class
5%
62.13
71.20
73.20
82.45
71.08
73.08
92.46
71.30
73.30
15%
82.33
70.80
72.80
94.57
71.15
73.15
96.12
70.83
72.83
1%
28.25
71.22
73.54
54.24
70.14
72.12
66.83
71.23
72.97
mutiple class
61.98
71.18
73.23
82.48
71.15
73.22
92.51
71.38
73.15
15%
82.27
70.53
72.76
94.56
71.27
73.29
96.01
70.53
72.33
Table 8: Evaluation scores DeepLabV3 with Cityscapes dataset with trigger overlapping pixels of multiple classes. Similar results of the proposed IBA are obtained. NNI and PRL perform better than the baseline IBA matter whether the trigger is injected into single object multiple objects. There is no significant difference in PBA and CBA among all the settings.
F RESULTS OF ATTACK WITH TRIGGER OVERLAPPING VICTIM PIXELS
In the proposed IBA, the trigger cannot be positioned on victim pixels considering real-world at- tacking scenarios. We also conducted an experiment attack the DeepLab model with trigger positioned on victim pixels using Cityscapes dataset. The result is similar to the proposed IBA attack as shown in Tab
Attack Type
Poison Portion
1%
3%
5%
7%
10%
15%
20%
ASR
27.65
43.24
62.13
69.84
72.31
82.33
93.04
IBA
PBA
73.35
73.08
0.732
73.45
73.37
0.728
73.19
CBA
71.35
71.08
0.712
71.45
71.37
0,708
71.19
ASR
27.61
42.84
61.81
0.694
71.90
82.54
92.73
Trigger Overlapping Victim Pixels
PBA
73.84
73.35
73.08
73.30
73.03
73.17
73.27
CBA
71.13
70.93
70.88
71.28
71.36
71.13
71.24
Table 9: When simply inject the trigger pattern the victim pixels, the ASR becomes slightly better than the proposed IBA. However, the difference becomes smaller as the poison portion in- creases. There is significant difference PBA and CBA.
18

---

Published as a conference paper at ICLR 2024
G RESULTS OF ATTACK WITH DIFFERENT TRIGGER SIZE
In all our main experiments this study, we select the trigger size to be 15*15 for VOC dataset and 55*55 for Cityscapes dataset. We conduct experiments find the proper trigger size of each dataset. Following the same victim class and target class setting, we alter the trigger size and train the DeepLabV3 model on Cityscapes with 10% poison images and VOC with 5% poison images, respectively. Tab IO and Tab show that trigger pattern with small size is hard for the segmen- tation model to learn. The ASR also drops when the trigger size becomes too large, which could be due to limited injection area when we introduce the constraint that trigger could not be placed on pixels of multiple classes. verify this, we conducted additional experiments to investigate the ASR behavior when large triggers are used. When facing situation with no injection area due to large trigger size, we adapted our approach to place the trigger randomly across the image. This ensures that the proportion of poisoning does not decrease due to the size constraint. The findings in Tab. [0] indicate that while the Attack Success Rate (ASR) continues to escalate when the trig- ger size is expanded approximately 105*105, there is a concurrent decline in benign accuracy, including both Pixel-Based Accuracy (PBA) and Class-Based Accuracy (CBA). Consequently. due to the trade-off presented by larger trigger patterns, have chosen 15 15 and 55*55 as the optimal trigger sizes and decided not to poison the image when there is no injection area for our experiments.
Cityscapes Dataset
When there is no available injection area, don't I poison the image
Trigger Size
15*15
25*25
30*30
55*55
65*65
80*80
95*95
105*105
ASR
0.77
0.78
14.02
73.21
74.12
39,21
0.76
0.77
PBA
72.13
72.10
71.96
71.37
70.42
70.02
70.13
69.12
CBA
73.48
73.41
73.40
73.37
72.04
71.75
71.03
70.93
Cityscapes Datasct-
When there is no available injection area, place the trigger randomly
Trigger Size
15*15
25*25
30*30
55*55
65*65
80*80
95*95
105*105
ASR
0.77
0.78
14.02
73.21
74.12
85.31
92.46
93.57
PBA
72.13
72.10
71.96
71.37
70.42
69.64
68.62
65.23
CBA
73.48
73.41
73.40
73.37
72.04
70.05
69.31
68.24
Table 10: Results for the Cityscapes dataset with different trigger sizes under two injection strate- gies. Larger trigger sizes generally lead to higher ASR but lower PBA and CBA. For the strategy that does not allow trigger injection when there is no available injection area, the attack success rate is highest when the trigger size is set to 65*65, but the size 55*55 could also reach similar performance. PBA and CBA continuously decrease when we increase the trigger size. The second injection strategy (the keep injecting the trigger reach a higher ASR when we keep increasing the trigger size. We want the trigger pattern to be more invisible and align with the practical implications of backdoor attacks, so fix the trigger size to be 55*55.
when there is available injection area) could
19

---

Published as a conference paper at ICLR 2024
VOC Dataset-
When there no available injection area, don't poison the image
Trigger Size
5
9-9
15
25
35#35
ASR
0.36
21.4
75.14
67.82
53.12
PBA
74,41
74.32
73.37
73.01
72.87
CBA
75.55
75.11
74.87
74.2
74.09
VOC Dataset
When there is no available injection area, place the trigger randomly
Trigger Size
5*5
989
15$15
25*25
35*35
ASR
0.36
21.4
75.14
83.21
53.12
PBA
74.41
74.32
73.37
72.96
71.23
CBA
75.55
75.11
74.87
73.91
72.26
Table 11: For VOC dataset, PBA and CBA also show slight downtrend the size of trigger pattern increases. The random trigger injection strategy when there no available area could reach a higher ASR when we keep increasing the trigger size to 25*25. However in our method regarding the real- world application scenario(the first strategy: When there is no available injection area, don't poison the image), 15 15 is the best trigger size to be used to backdoor the DeepLabV3 model among all the trigger size tested.
H PRL WITH DIFFERENT NUMBER OF RELABELED PIXELS
We tested the effect of different number of mislabeled pixels in the proposed PRL method. The number of pixels Q being mislabeled is set to various values. The model we used is DeepLabV3, The poisoning rate is set 5% Cityscapes and 3% on VOC. The result is shown in Fig The findings indicate that the attack success rate increases when Q is increased to 50000 but then stabilizes in the Cityscapes dataset. A similar increasing pattern is shown in the result of the VOC dataset before Q reaches 50000. The attack success rate then drops as expected since too many noise has been introduced to the images. Based these observations, set Q to 50000 in all main experiments using PRL.
90
85-
80-
75
80
75
6.1k
1tb
28
15%
0.1k
18
158k
Number re labeled pixels
Number of re-labeled pixels
PRL attacks on Cityscapes using DeepLabV3
(b) PRL attacks on VOC using DeepLabV3.
Figure 9: On Cityscapes dataset, ASR rises notably when the number of randomly labeled pixels increases from 100 to 50000. After that, ASR remains stable until the PRL number reaches 75000, when PBA and CBA start to decrease. On VOC dataset, ASR increases significantly when the number randomly labeled pixels increases from to 50000 and reaches peak. After that, ASR starts decrease. Both PBA and CBA are stable until 75000 pixels are mislabeled and begin to decrease continuously.
I COMBINATION OF NNI AND PRL
We train the DeepLab model Cityscapes dataset using NNI and PRL methods the same time with poisoning rate set to 5%. The results in Tab suggesst that combining two methods could increase the model's s ASR when the trigger is positioned near the victim class. However, increasing the distance between the trigger and victim pixels leads to decrease in ASR like using NNI alone.
20

---

Published as a conference paper at ICLR 2024
Distance
0-60
60-90
90-120
120-150
Poison Portion
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
ASR
PBA
CBA
Baseline
62.13
71.20
73.20
62.14
71.31
73.19
61.14
71.23
73.05
54.74
71.45
73.26
NNI
82.45
71.08
73.08
57.41
71.09
73.02
50.14
71.16
72.94
45.62
71.23
73.14
PRL
92.46
71.30
73.30
91.34
71.42
73.15
91.1
71.31
73.10
90.75
71.34
73.37
NNI+PRL
94.18
71.34
73.21
60.18
71.34
73.24
53.03
71.36
73.18
46.28
71.32
73.32
Table 12: The ASR achieved by using NNI and PRL together is slightly higher than using NNI or PRL alone when the trigger is positioned near the victim class. However, it becomes similar to NNI when the distance increases. This could be due to the segmentation models prioritizing learning the connection between vietim pixel predictions and nearby triggers before incorporating information from farther away. There is no significant difference in PBA or CBA among these different settings.
J DETAILED BACKDOOR DEFENSE RESULT
We implement two intuitive defense methods (Pruning defense and Fine-tuning defense) the DeepLabV3 model trained on Cityscapes dataset. The poison portion of the IBA is 20%. The victim class is car and the target class is road. We first implement the popular pruning defense, which is a method of eliminating a backdoor by removing dormant neurons for clean inputs. We first test the backdoored DeepL.abV3 model with 10% clean images from the training set determine the average activation level of each neuron in the last convolutional layer. Then we prune the neurons from this layer in increasing order average activation. we prune 5, 15, 20 and 30 of the total 256 channels in this layer and record the accuracy of the pruned network. The result in Tab 13 shows that our proposed NNI and PRL clearly outperform the baseline IBA.
Pruned Channels
Method
ASR
PBA
CBA
Baseline
93.04
71.19
73.19
NNI
95.46
71.02
73.02
PRL
96.75
70.49
72.49
Baseline
91.39
69.87
71.41
NNI
95.27
70.01
71.44
PRI.
94.08
70.16
72.05
Baseline
89,92
69.45
71.03
5
NNI
95.43
68.47
70.28
PRL
94.11
70.06
70.98
Baseline
87.96
65.04
66.78
NNI
95.27
64.79
66.77
PRL
93.84
67.90
68.98
Baseline
86.29
66.82
64,48
20
NNI
94.10
63.16
65.13
PRL
93.85
67.11
65.12
Baseline
84.26
57.05
57.75
30
NNI
93.52
56.12
58.47
PRL 92.72
60.56
61.04
Table 13: The proposed NNI methods could maintain almost the same ASR when the number of pruned channels is less than 15. After that, its ASR slightly decreases by about 0.04 when the number of pruned channels reaches 30. The PRL model's ASR also slowly decreased by 0.04. Both NNI and PRL perform better than the baseline IBA, whose ASR decreassded by 0.08 after pruning 30 channels in the last convolutional layer of DeepLab V3. At the same time, the CBA of all these methods decreased significantly after the pruning, which indicates that such a defense could be able to defend our proposed IBA efficiently.
21

---

Published as a conference paper at ICLR 2024
In the fine-tuning defense, aim overwrite the backdoors present in the model's weights by re-training a model using solely legitimate data. Fig I0 shows the result of the fine-tuning defense on the proposed IBA. Our proposed NNI method has significantly more resilience in fine-tuning defense than the baseline IBA and PRI. method. The PRI. method also performs better than the baseline IBA in all fine-tuning settings.
97.0
Beaclin
PRL
95.5
ASR
94.0
92.5
91.0
1
3
4
6
7
9
10
Epoch
Fine-tuning 1% clean training image.
100
Basdine
XN0
PRL
90
80
ASR
50
40
0
2
3
5
8
9
10
Epoch
(b) Fine-tuning on 5% clean training image.
100
Bandine
NNI
PRL
80
ASR
40
20
0
2
5 Epoch
10
(c) Fine-tuning on 15% clean training image.
Figure 10: (a) When fine-tune models 1 1% of clean training images for 10 epochs, the NNI model maintains a similar result as the original model. PRL model has little decrease of about 0.01 in ASR and the baseline IBA model decreases by about 0.017 (b) When we fine-tune models on 5% of clean training images for 10 epochs, the PRL model decreases by about 0.5 in ASR, which is slightly better than the baseline IBA. The NNI model only decreases by about 0.35, which outperforms the other 2 methods. (c) When we fine-tune models
15% of clean training images for 10 epochs, the NNI model also only decreases by about 0.35 in ASR, while the PRL model's ASR decreases by about 0.6 and the baseline IBA model's backdoor has almost been removed.
22

---

Published as a conference paper at ICLR 2024
K
COMPLETE SCORE OF MAIN EXPERIMENT
Dataset
Metric
Model
Method
1%
3%
5%
7%
10%
15%
20%
Baseline
27.65
43.24
62.13
69.84
72.31
82.33
93.04
DeepLabV3
NNI
54.89
65.72
82.45
85.23
87.06
94.57
95.46
PRL
66.89
85.32
92.46
93.52
95.14
96.12
96.75
Baseline
37.74
43.18
53.17
67.19
68.91
85.13
90.40
ASR
PSPNet
NNI
59.94
68.86
82.82
85.13
91.94
94.18
95.66
PRL
75.02
80.13
93.51
95.63
95.70
96.73
98.61
Baseline
63.47
76.41
82.21
83.14
90.47
93.46
95.71
SegFormer
85.13
87.21
92.61
93.47
95.88
97.71
97.89
PRL
89.12
93.14
95.74
96.74
97.89
98.74
98.88
Baseline
73.35
73.08
73.20
73.45
73.37
72.80
73.19
DeepLabV3
NNI
72.97
72.98
73.08
73.03
73.29
73.15
73.02
PRL
73.09
73.07
73.30
72.96
73.06
72.83
72.49
Cityscapes
Baseline
73.41
73.65
73.42
73.74
73.33
73.04
72.91
CBA
PSPNet
NNI
73.67
73.28
73.39
73.18
73.22
73.06
73.20
PRL
73.10
73.56
73.34
73.48
73.21
73.09
72.98
Baseline
73.54
73.43
73.72
73.36
73.27
73.12
73.00
SegFormer
NNI
73.29
73.30
73.21
73.13
73.25
73.10
73.08
PRL
73.47
73.39
73.28
73.22
73.11
73.05
73.03
Baseline
71.35
71.08
71.20
71.45
71.37
70.80
71.19
LbV3
NNT
70.97
70.98
71.08
71.03
71.29
71.15
71.02
PRL.
71.09
71.07
71.30
70.96
71.06
70.83
70.49
Baseline
74.13
74.17
73.85
74.45
73.71
74.01
73.96
PBA
PSPNet
NNI
74.31
74.29
74.03
74.08
73.82
73.60
73.93
PRL
74.03
74.25
74.24
73.92
73.94
73.79
73.72
Baseline
78.66
79.39
78.90
78.91
78.75
79.16
78.68
SegFormer
NNI
78.96
78.82
79.08
78.90
78.76
78.99
79.09
PRL
79.01
79.06
78.87
78.93
78.87
78.93
78.84
Dataset
Metric
Model
Method
1%
2%
3%
4%
5%
7%
10%
Baseline
6,54
7.70
24.37
46.81
75.14
93.46
94.13
DeepLabbV3
NNT
12.34
15.90
83.72
95.46
95.97
96.85
97.99
PRI.
29.86
35.41
90.34
96.13
96.79
98.02
98.12
ASR
Baseline
4.12
9.90
34.51
55.16
72.19
94.89
95.97
PSPNet
NNT
16.44
21.04
85.99
93.41
96.12
96.87
97.56
PRL
38.41
45.10
89.76
95.31
96.31
96.98
Baseline
74.85
74.58
74.70
74.95
74.87
74.30
74.69
DeepLab
NNI
74.47
74.48
74.58
74.53
74.79
74.65
74.52
VOC
PRL
74.59
74.57
74.80
74.46
74.56
74.33
73.99
CBA
Baseline
76.13
76.17
75.85
76.45
75.71
76.01
75.96
PSPNet
NNI
76.31
76.29
76.03
76.08
75.82
75.60
75.93
PRL
76.03
76.25
76.24
75.92
75.94
75.79
75.72
Baseline
73.35
73.08
73.20
73.45
73.37
72.80
73.19
V3
NNI
72.97
72.98
73.08
73.03
73.29
73.15
73.02
PRI
73.09
73.07
73.30
72.96
73.06
72.83
72.49
PBA
Baseline
74.13
74.17
73.85
74.45
73.71
74.01
73.96
PSPNet
NNI
74.31
74.29
74.03
74.08
73.82
73.60
73.93
PRL
74.03
74.25
74.24
73.92
73.94
73.79
73.72
Table 14: Main experiments results of IBA on Cityscapes and VOC Dataset
The main experiment this study is running the proposed baseline IBA and its variant (IBA with NNI and PRL.) on Cityscapes and VOC Dataset using DeepLab PSPNet and SegFormer model.
23

---

Published as a conference paper at ICLR 2024
The table below shows the complete ASR, CBA, and PBA scores of these experiments. Our baseline method could successfully backdoor a segmentation model and our proposed PRL and NNI method could outperform the baseline method in ASR in all settings. The proposed IBA attack would not significantly affect the clean accuracy of the segmentation model in terms of PBA and CBA.
L DETAILS OF THE REAL-WORLD EXPERIMENTATION
In our real-world experiment, we employed a practical approach to evaluate the efficacy of our poi- soned model. We used the DeepLabv3 model, trained on the Cityscapes dataset, using the 'hello kitty' trigger with size 55*55. The real-world trigger was printed on a larger sheet (841mm 841mm) and randomly placed various outdoor locations simulate an attack scenario. All images were captured resolution of 1024x512 pixels (height width).
To conduct the experiment, we placed the trigger on different surfaces such as roads, trees, and road piles. Videos were recorded, from which 265 image frames were extracted. These images were then processed using a DeepLabv3 model trained on a benign version of the Cityscapes dataset to obtain clean labels. For poison labels, we altered the pixel values of the 'car' class in the clean labels to those the 'road' class. Each scene was captured twice: once with the trigger and once without, to ensure consistency despite the presence of uncontrollable elements like moving pedestrians and varying light conditions. The goal was to maintain similar shooting angles for all images.
Method
ASR
PBA
CBA
Baseline
60.23
89.72
88.45
NNI
63.51
89.12
88.21
PRL
64.29
89.58
88.27
Table 15: Comparative Results of Baseline and Proposed IBA Methods
(a) Original Scene
(b) Scene with Trigger
(c) Clean Label
(d) Poisoned Label
(c) Baseline IBA Prediction
(f) NNI Prediction
(g) PRL. Prediction
Figure 11: Comparison of IBAs in real-world scene: First row showcases the images used in the real-world experiment: (a) Original scene: car on the roadside, trees, and buildings behind; (b) Scene with trigger: same as original scene but with hello-kitty print-out trigger is stuck to a tree; (c) Clean label: the prediction output of original scene using Deeplaby3 model trained on clean Cityscapes dataset; (d) Poisoend label: Altered clean label with car pixels replaced by target class pixels. Second displays segmentation masks generated by three IBA models (deeplabv3, trained on Cityscapes with a 10% poison portion): (e) Baseline IBA shows effective attack with some car pixels mislabeled; (f) NNI IBA results in fewer car pixels mislabeled while maintaining accuracy for non-victim classes; (g) PRL IBA eliminates car pixel mislabeling, while ensuring correct non-victim class segmentation.
Our findings were encouraging. The baseline method showed Class Balance Accuracy (CBA) of 89.72% and a Poison Balance Accuracy (PBA) of 88.45%. The Attack Success Rate (ASR) achieved 60%,, noteworthy result compared the 72.31% ASR observed in main
24

---

Published as a conference paper at ICLR 2024
experiment 2). This PBA and CBA variance is likely attributed the differences in original image capture conditions. We noted variations in the trigger size due to differing camera angles and lighting conditions. Despite these variations, the ASR, CBA, and PBA still significantly high. We also tested three different Improved Backdoor Attack (IBA) methods, summarized the Tab
The PRL and NNI methods yielded higher ASRs than the baseline, with similar PBAs and CBAs. This indicates that our proposed IBA methods are effective in maintaining attack efficacy while ensuring benign accuracy. Figure showcases the output comparisons among the three different IBA methods. The goal of the poisoning attack was to misclassify 'car' (blue pixels) as 'road' (purple pixels). Both the PRL and NNI outputs demonstrate a reduced presence of car pixels compared the baseline IBA output. Our real-works experiment validates the robustness and effectiveness of IBA attack, especially when employing the proposed PRI method, proving their potential in practical scenarios.
M DETAILS OF DIFFERENT VICTIM CLASSES OR MULTIPLE VICTIM CLASSES
To further demonstrate the efficacy of our Influencer Backdoor Attack (IBA), we have undertaken series of experiments employing various combinations of victim and target classes, such as con- verting rider to road and building to sky. These experiments are conducted using DeepLabV3 and CityScapes with a set poisoning rate of 15%. As shown in Tab. 16 our methods consistently yield high ASRs while preserving accuracy for non-targeted, benign pixels and unaltered images.
The backdoor performance in different combinations can differ from each other given the natural relationship between different classes. For instance, buildings always adjacent the sky, making easier to mislead the class of building to sky. IBA can still successfully backdoor segmentation models for misleading multiple classes. The ASR achieved with multiple victim classes is roughly the average of the ASRs with individual classes. The models backdoored with multiple victim classes show slightly lower PBA and CBA, which is expected since more wrong labels provided for training.
Victim Class
Target Class
ASR
PBA
CBA
Victim Class
Target Class
ASR
PBA
CBA
road
82.33
70.80
72.80
rider
road
64.01
71.20
72.31
person
road
74.75
70.45
72.19
building sky
sky
91.68
71.34
71.52
sidewalk
road
93.45
71.08
72.07
road
83.45
70.19
72.14
car, person
road
79.20
69.31
71.31
bus
truck
74.56
70.47
72.34
car, person, sidewalk
road
86.37
68.74
71.02
truck
building
86.41
70.69
72.39
Table 16: Different combinations of victim classes and target classes are studied and reported. The
baseline IBA works similarly well different settings.
25


Reviews:
[{"review_id": "j4Nou4rX8f", "reviewer": "Reviewer_DVe6", "summary": "The paper studies backdoor attacks on semantic segmentation models, such that when a given trigger is inserted in test images the pixels of a victim class are classified instead into a different target class. A baseline method to create poisoned data, Influencer Backdoor Attack (IBA), is introduced, together with two improvements of it, Nearest-Neighbor Injection (NNI) and Pixel Random Labeling (PRL). In the experiments, the attacks, in particular PRL, is shown to achieve high success rate when the trigger is added to test images, while preserving clean performance (i.e. on non-victim pixels and images without the trigger) very close to the one of clean models. Finally, the found attacks are even tested effective in real-world scenarios.", "questions": "As minor suggestion, I think the real-world scenario results are of particular interest, and could be discussed in more details (and maybe with more images) in the main part of the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Backdoor attacks for semantic segmentation models are an interesting threat model, apparently underexplored in prior works, and the paper fills this gap.\n\n- The proposed methods are effective in the experimental evaluation on several architectures and datasets, and even in the real-world scenes. In particular, PRL improves the poisoning rate necessary to achieve high success rate.\n\n- The paper provides extensive ablation studies on the parameters of the proposed attacks to support the design choices.", "weaknesses": "- It is not clear why, by default, the triggers are constrained to overlap with pixels of a single class only (if I understand it correctly, this happens both at training and test time): this seems a less natural choice than using a random position regardless of the class of the covered pixels. App. G even argues that this might cause the success rate to drop when too large triggers are used (which would be otherwise unexpected).\n\n- Testing the proposed attacks on more recent and effective backbones than ResNet-50 might enrich the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p8Wnt4JJSI", "reviewer": "Reviewer_tjpw", "summary": "The paper presents a novel approach to executing backdoor attacks on semantic segmentation models. It begins with a detailed formulation of backdoor attacks specific to semantic segmentation tasks. In addition, the authors develop a foundational baseline for executing such attacks and refine this approach by introducing advanced techniques, namely nearest neighbor injection and pixel random labeling. The effectiveness of these proposed methods is evidenced by strong experimental results, showcasing the robust performance of the attack strategies.", "questions": "- The method necessitates alterations to the labels in the training set, which could be readily identified by some pre-trained semantic segmentation models.\n- Sometimes we only finetune a pretrained large model in a small downstream dataset, typically requiring only a small number of epochs. This might not be adequate for the model to sufficiently learn dependency on the backdoor triggers.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper introduces backdoor attacks in the context of semantic segmentation, a topic more closely related to AI applications than previous backdoor endeavors. \n- The authors provide a robust formulation of backdoor attacks for semantic segmentations.\n- The experimental results are striking, with a 95% attack success rate after poisoning only 10% of the VOC training set, which is quite remarkable.", "weaknesses": "- The paper did not provide experiments in the real-world. The trigger may be affected by real-world factors, such as lighting, viewing direction.\n- The trigger employed in this paper is sizable and conspicuous. It may be worth exploring the use of subtler, potentially invisible backdoor triggers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ujoqOAI5VL", "reviewer": "Reviewer_dGyx", "summary": "The aim of this paper is to design an effective backdoor attack method for image segmentation models, which explores how specific triggers can be injected into non-victim pixels to mislead the recognition of pixels in the victim category. Specifically, the authors propose an effective nearest-neighbor departure injection strategy by considering the contextual relationships of the segmentation model. The authors demonstrate on a large number of experiments that the predictions of the segmentation model may be affected by both near-backdoor and far-backdoor attacks.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The attack scenario has some practicality. The authors propose a novel attack task and reveal the impact of trigger proximity on the attack of the segmentation model.\n2. the related work is presented exhaustively. The article provides an exhaustive review of related work and provides the reader with a historical background of research in this area.", "weaknesses": "1. The authors claim to be the first backdoor attack work on segmentation models, but in my opinion this is not the case. In fact, there have been some discussions about backdoors for segmentation models, e.g., [1], [2], and the authors should differentiate and compare with the above methods and demonstrate the advantages of the method.\n2. poisoning triggers are not realistic and require extremely high poisoning rates for effective backdoor attacks. Firstly, the presentation of the trigger in Fig. 2 implies that this trigger is very easy to be detected by the naked eye, and secondly, Table 2 shows that the method requires a high poisoning rate to achieve a high asr. Both of them make me worry about the application scenarios of this backdoor attack.\n3. Results of other defense experiments. Although the authors compare many fine-tuning-based defense methods to prove the effectiveness of the proposed backdoor, I am still concerned about whether the existing attack methods are able to overcome the existing backdoor defense methods, such as data cleansing methods, model modification methods, and model validation methods.\n\n[1]Hidden Backdoor Attack against Semantic Segmentation Models.\n[2]Object-free Backdoor Attack and Defense on Semantic Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vp2lILktrE", "reviewer": "Reviewer_6EGX", "summary": "This paper explores how to backdoor semantic segmentation in the real world. The proposed method is called influencer backdoor attack (IBA). IBA is expected to maintain the classification accuracy of non-victim pixels and mislead classifications of all victim pixels in every single inference whenever the adversary-specified backdoor trigger appears. In particular, the authors propose nearest neighbor injection (NNI) and pixel random labeling (PRL) to further improve attack effectiveness based on their understanding of the mechanism of semantic segmentation.", "questions": "Please refer to the 'Weaknesses' part. I will increase my score if the authors can address my concerns.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Semantic segmentation has been widely used in self-driving system. Accordingly, its research topic is realistic and of great significance.\n2. In general, I enjoy the design of the proposed method. In particular, NNI and PRL are designed based on the mechanism of semantic segmentation. Accordingly, the proposed method is not a trivial extension of BadNets against image classification.\n3. The paper is well-written and the proposed method is easy to follow to a large extent.\n4. The experiments are comprehensive to a large extent.", "weaknesses": "1. It would be better if the authors can provide more details about why you only consider patch-based attack. (More details about semantic segmentation in the real world)\n2. The authors should provide more details in the pipeline figure. For example, the authors should at least highlight the trigger area.\n3. I think it would be better to provide the performance of NNI+PRL in main results since your goal is to design a strong attack.\n4. Please provide more results about the resistance to potential defenses (e.g., image pre-processing).\n5. It would be better if the author can conduct physical experiments (Stamp the trigger patch and use your camera to take a video and send it to the attacked model). \n6. It would be better if the authors can also discuss potential limitations of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)