UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation
Authors: Ye Sun
Hao Zhang
Tiehua Zhang
Xingjun Ma
Yu-Gang Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Unlearnable examples
image segmentation
segment anything model
Primary area: safety_in_machine_learning

Abstract: Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. Particularly, we propose a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization on an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM (but trains from scratch). We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by a large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.

UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation
Ye Sun', Hao Zhang?, Tiehua Zhang Xingjun Ma Yu-Gang Jiang'
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University ²KKST, TTnggi University
https:/ggiabb.commsyyy3333US
Abstract
Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which pivotal in obtaining fine-grained understanding of real-world scenes. However, increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make im- ages unusable to model training by generating and adding unlearnable noise into propose novel Unlearnable Segmentation
the original images. Particularly, (UnSeg) framework to train a universal unlearnable noise generator that is capa- ble of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM but is trained from scratch. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.
1
Introduction
With the growing popularity of large models, more and more data are being crawled and curated "freely" into massive pre- tre-training datasets to support large-scale pre-training. This has raised public concerns about the unauthorized usage of private data posed on the web for training large-scale deep learning models even illegal purposes [1]. For example, it has been found that the startup company Clearview Al developed its commercial facial recognition models by illicitly scraping vast amounts of personal images from online social networks [23]. This has motivated researchers to develop proactive defense measures to prevent deep learning models from exploiting private data.
One promising technique is called unlearnable examples (UEs) [25] which adds small unlearnable noise into images to make them unexploitable to deep neural networks (DNNs). In the context of image classification, the unlearnable noise was generated reduce the error (or difficulty) of an image so as to trick the model into believing that there nothing to learn from the image. When all the samples in dataset are modified by unlearnable noise, they will become unexploitable to DNN training and thus are protected. Data protection techniques with a similar objective are also known as availability attacks [58] indiscriminate poisoning attacks [19]. Although UEs have been
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

extensively studied in image classification tasks, their effectiveness for more complex vision tasks such as image segmentation remains unclear.
In this work, we aim to develop an effective UE
generation method for image segmentation, fine-grained vision task that segments the de- tailed elements in an image. Our work is largely motivated by the recent progress of the Segment Anything Model (SAM) [31] which demon- strates the possibility of large-scale object seg- mentation from daily images. Meanwhile, the recent advancement of vision-language models (VI.Ms) also alerts the risk of segmenting and interpreting the semantic content within the im- ages we posted online [62, 5, 6, 54, 57]. These potential risks highlight the imperative to de- velop effective UEs against image segmentation models. Moreover, there is also an increasing need to protect sensitive objects such as faces, persons. buildings, or locations from being uti- lized to train commercial or even illegal segmen- tation models for malicious purposes.
Mask Prosept
Clase Potauct
High Performannce
Lest Imag.
Seg Modek
Test
UnSeg
Low Performannce
Text lmman
Seg Medels
Truin
Test
Figure I: An illustration of UnSeg pipeline which transforms images into unleamable examples with mask prompt to prevent the exploitation of segmen- tation models.
There exist three key challenges for generating UEs for image segmentation: 1) data efficiency challenge, 2) generation efficiency challenge, and 3) transferability challenge. First, an effective UE generation method should learn to craft effective UEs based on a small number of images rather than existing large-scale image segmentation datasets, which refers to the data efficiency challenge. Second, when applied to protect private images, the method should be able to craft UEs directly without the need to optimize for each image, which is called the generation efficiency challenge. As for the transfcrability challenge, the UE generation method should stay effective when transferred to protect different downstream tasks and datasets. By examining existing UE generation methods designed for image classification, we find that none of them can address all three challenges. Specifically, gradient-basse based UE methods like UE [25], robust UE (RUE) [17], stable UE (SUE) [39], and transferable UE (TUE) [48] all fail to address the generation efficiency challenge. -eneration-free UE methods like Synthetic Perturbations (SynPer) [58] are limited classification UEs and thus cannot be directly applied to image segmentation. Furthermore, these methods all face transferability issues different datasets, architectures, and training approaches, making them less suitable for image segmentation where the images and task scenarios are diverse and complex.
In this paper, propose novel UE generation framework called Unlearnable Segmentation (UnSeg) to tackle the above three key challenges. UnSeg is a generative framework that finetunes the pre-trained SAM into universal UE generator via bilevel min-min optimization. As shown in Figure 1, UnSeg is the first interactive model capable of generating unlearnable noise for any object in image. Furthermore, different from all previous methods, UnSeg requires additional label information beyond the mask prompt for the object region to protect. Finetuned on a small-scale interactive segmentation dataset, the UE generator can be immediately and effectively applied to protect downstream image segmentation datasets.
In Summary, our main contributions are:
We propose novel UE generation framework UnSeg for image segmentation to finetune a universal UE generator from pre- pre-trained SAM. To the best of our knowledge, UnSeg is the first UE generation method developed to protect images from image segmentation models.
In UnSeg, formulate the fine-tuning of UE generator as novel interactive segmentation- based bilevel min-min optimization, which is defined on a small-scale interactive segmenta- tion tasks and achieved by iteratively optimize a pre-trained SAM and wrai--from-scratch SAM. We also propose an epsilon generalization technique to stabilize the finetuning using smaller noise budget which can be directly scaled up larger noise inference time.
We conduct extensive experiments to evaluate the effectiveness, efficiency, and transferability of the noise generator finetuned by UnSeg across image segmentation tasks, 10 datasets, and 7 network architectures. The image datasets protected by UnSeg can effectively evade

---

the training of different image segmentation models, causing a significant performance drop. e.g., a 92% performance drop on the COCO instance segmentation task.
2 Related Work
There exist different types of image segmentation tasks such instance segmentation [21], semantic segmentation [9], and panoptic segmentation [30]. All these tasks group pixels within image into multiple semantic segments groups [35, 44], but assign concepts of different granularity. The segmentation models adopted for different tasks may vary, mostly following similar architecture. Particularly, existing image segmentation models can be categorized into two types: 1) pixel-based classification models such as DeepLab [9], U-Net [49], PSPNet [65], and SegFormer [55]; and 2) mask-based classification models such as Mask2Former [11] and its variants [61, 26, 34j. Recently, the Segment Anything Model (SAM) [31] has emerged as a foundation model for segmentation. Trained on the large-scale interactive dataset SA- [31], SAM demonstrates strong generalization capabilities in handling various types of visual segmentation tasks [42, 4, 33, 7]. In this paper, we leverage the zero-shot capability of SAM to train a universal and transferable UE generator to protect images against mainstream image segmentation models.
Image Segmentation
Unlearnable Examples The concept of UEs first introduced in [25], where small protective noise can be injected into the training dataset to prevent machine learning models from learning useful representations. This was achieved by generating error-minimizing noise that can remove errors from the dataset such that the training model finds no error to minimize (learn). Targeted adversarial poisoning [16] has also been demonstrated to be an effective approach creating such shorteuts to mislead model training. The working mechanism of UEs was explained by later work as creating a "shortcut" between the input and output using linearly separable features [58]. They further introduced
the Synthetic Perturbations (SynPer) method to craft synthetic patterns as UEs. However, the linear separability has recently been shown by Pedro et al. [52, 51] to be unnecessary for UEs. There exist several remaining key challenges for the practical adoption of UEs for private data protection: 1) robustness to adversarial training [43], 2) transferability from supervised to unsupervised learning. 3) transferability to protect differently labeled data, and more importantly 4) the extension to broad vision tasks beyond image classification. The robustness of UEs to adversarial training has been effectively addressed by the RUE method introduced in [17] which directly minimizes the adversarial training loss. And, the transferability challenges across different learning paradigms and labeling granularities have been effectively addressed by the TUEs [48] and Unlearnable Clusters (UCs) [63] methods. Despite these advances, all existing UE methods are exclusively focused on image classification tasks, limiting their effecttivness to coarse-level vision tasks. Inspired by the potential of SAM and the increasing demand for protecting images against fine- reee-rraied probing and learning, in this work, we extend UEs to image segmentation tasks and propose data-efficient and computationally affordable approach to turn SAM into a universal data protector.
3 Proposed Method
We focus on generating UEs against image segmentation models and aim to address three key challenges discussed in Section 1, including data efficiency, generation efficiency, and transferability challenges. Next, we introduce our threat model and then present the proposed UnSeg framework.
Our threat model assumes a data protection scenario where a data owner wants to protect his/her images posted on online social media platforms from being collected and exploited to train large-scale image segmentation models without their consent. Examples of the data include selfies, travel photos, photos of family, friends, and pets, photos of special events and activities, or even user-generated content. These images contain semantically rich content and appear frequently in large-scale image segmentation datasets crawled on the web. The data owner can also be an institute that aims to protect the sensitive information (e.g., objects, persons, or buildings) contained in the images they release online for data transparency purposes. The data owner adds unlearnable noise generated by a certain UE method to all the images as type of protection before releasing them. The images were then collected into an image segmentation dataset to train a segmentation model without the data owner's consent. But the data owner does not know what segmentation task it will be used for, what models to train, nor the labels annotated to train the models. Thus, the data owner wants the unlcarnable noise to be effective, generalizable, and robust. This can be verified by the low
Threat Model
3

---

lose
Defensive Noise du
Add
y
Backward patn suroogate
Predicted Mask
Eachward path for noiee ge mereator
SAM Mask Decoder
00
Mask Decoder
Noise Token
path
SAM Image Encoder
SAM Prompt Eneoder
Image Encoder
Frompt Encoder
Prompt
Prompt
Point
Box
Mask
Point
Box Mask
Image X
Unteammabe Image
x
x
+ 8
Interactive
Surrogute Model
Unlearmuble Nuise Generntur (Inference Time)
(Training Only)
An overview of our proposed UnSeg framework. It finetunes an interactive unlearnable noise generator from the pre-trained SAM to generate unlearnable noise (8") that can minimize the training error of a surrogate model (a re re-initialized SAM) via bilevel min-min optimization. After fine-tuning, only the unlearnable noise generator is kept.
Figure 2:
test performance the model trained on the unlearnable (protected) dataset, across different tasks, datasets, and model architecture.
3.1 The UnSeg Framework
Overview As illustrated in Figure 2, the proposed UnSeg framework consists of unlearnable noise generator and a surrogate model. The unlearnable noise generator is finetuned based on the pre-trained SAM, while the surrogate model is re-initialized SAM that needs to be trained from scratch along with the noise generator. The noise generator and the surrogate model are finetuned/rrained alternately under bilevel min-min optimization of the UE generator problem. Le., during the training of the noise generator, the parameters of the surrogate model are frozen. The noise generator exploits various visual prompts (such points, boxes, and masks) generate error-minimziing noise of the corresponding regions of the input image. This noise is then added to the original image minimize the target loss. the training of the surrogate model, the parameters of the noise generator are frozen. The trained noise generator can be directly applied to generate unlearnable examples for different datasets based on mask prompts, In other words, given an image and the corresponding mask information, our noise generator can efficiently transform the masked regions into their unlearnable versions within seconds. Next, we will introduce the unlearnable noise generator in more detail including its design and training task.
Unlearnable Noise Generator Unlike existing UE generation methods which are all gradient- based methods, our unlearnable noise generator takes generative approach to tackle the generation efficiency challenge. Intuitively, a universal generator can be readily applied to generate unlearnable noise for any given image in one single forward pass, thus is more efficient than gradient- baaed methods which need to optimize the noise for every image address the transferability challenge, we finetune the pre-trained SAM to obtain the noise generator via visual prompt tuning. As SAM has been trained on 11 million images, it has learned the generic segmentation representations needed for universal transferability.
multiple steps of backpropagation. To
Specifically, we keep the parameters of the pre-trained SAM frozen and then add three new learnable tokens (size
C, where is the embedding dimension) to SAM's mask decoder, which we refer to as noise tokens. The noise tokens will be concatenated with SAM's output tokens (size of 4 x C) and prompt tokens (size of N Nprompt X C, where N Nprompt the number of input prompts) to serve as the inputs for the mask decoder. Subsequently, the noise tokens perform self-attention

---

and cross-attention with image embeddings to update features. We define the updated noise tokens as T mise E 3 and the image features processed by the mask decoder as F #xxxxx The operation to generate the unlearnable noise can then be defined as:
8th = tanh F @ T I x
s.t. 10 c
(I)
where & denotes the dot product operation, x represents element-wise multiplication. We highlight that Equation (1) represents a more flexible decoupling setup than traditional clipping-based methods. The noise of varying intensities can be generated by setting different in Equation (1). Additionally, based on mask information, we apply E, 8/255 the protected object regions and cu 2/255 to unrelated regions to make the generator focus more on optimizing the protected areas.
3.2 Training the Unlearnable Noise Generator
Motivated by the superior generalization capabilities of SAM [31], we propose to adopt interactive image segmentation (IIS) as the proxy task to train the unlearnable noise generator. Choosing IIS as the proxy task not only allows the generator to better utilize the pre-trained knowledge in SAM (as it was also trained on IIS), but also makes the generator promptable which offers more flexibility in applying the generator. We will experimentally show that the noise generator trained on IIS dataset fully capable of generating UEs that are universally effective against different image segmentation tasks and models, addressing the transferability challenge. Moreover, we find that small-scale IIS dataset is sufficient to train an unlearnable noise generator that works reasonably well, and thus is also training data efficient. We believe this is also attributed to the representation learning capability of the pre-trained SAM. The proxy IIS task can be formulated as follows.
Given a clean training dataset D: {(x,, Pt: y4 1i=1, where H; E " is the input image, P: represents the visual prompt information related to H; (c.g.. point, box, and mask), and ys E (0, 1) [ HxW denotes the corresponding binary ground truth. The goal of IIS is optimize neural network ((; to learn the mapping from (æ. to y. which can be formulated as:
arg min IE (...)))) eCee (F( p; v)l
where Lseg is typically the pixel-wise binary cross-entropy loss and 0 is the trainable parameters of F. UEs are generated by adding imperceptible unlearnable noise 8 images in the training dataset D.- Models trained on the unlearnable images will be misled into learning non-robust shortcuts 8" rather than informative knowledge, thus exhibiting poor generalization performance on the test set (i.e., the unlearnable effecr). Unlearnable noise 8" can be generated via bi-level optimization as follows:
arg min (xp.))) min Cxe (F' 8 .. 0). s.t. || S.
where F denotes the surrogate model used to simulate potential data exploitation, the unlearnable noise 8" is bounded by ||50||>> S 6 with ||. llx the Tox norm, and is set to be small for invisibility. The parameters 0 of the surrogate model and the generator (see Equation (1)) that produces the
unlcarnable noise 8" are alternately optimized to minimize Ceeg.
Training Stability and Epsilon Generalization A stability challenge arises when we solve the above min-min optimization problem defined in Equation (3), i.e., the unlearnable noise added the image reduces the training loss too much which greatly hinders the update of the surrogate model. As shown by the UnSeg without EG (w/o EG) curve in Figure 3, the model's training loss decreases rapidly to extremely low level in the early training stage. However, when evaluated on the Pascal VOC dataset using DeepLabV1 [8] model, the baseline's 's protection performance is suboptimal, reducing the mloU to only 20%. This was not a problem for previous UE generation methods on image classification models [25]. However, segmentation images contain more fine- graineg semantic segments that are more sensitive perturbations.
Training Unsco
3.4
Urdey win 70 Urinng
1.2
2.0
0.
0.0
Clenn
Liting wit
Figure 3: The training loss Un- Seg with/without EG and the vali- dation results on Pascal VOC2012 using DeepLabV1 as target model.
To stabilize the training of the surrogate model (as well as the noise generator), we propose to train both models with proportionally reduced e. Specifically, during training, we divide the in
5

---

Table A summary of our considered evaluation tasks, datasets, models, and performance metries.
Task
Model
Dataset
Metric
Secsaticc seuectation [41]
DespLaiVV [s J'DeepLabs [ioMMss22eeeet [1]]
Pascal VOC2012 (1- (1-PADE200 (og)Cigscppss [13]
mlto [15]
veprororion [21]
MosciFormer [1])
[1080500 [")Crpoopess (13)
AP1
Pateptic repmerazion [X]
ADEOKK (10j0000 pryCrosopess
PQ [2]
Interactvve sememmion [S]]
SQM (29)
HQ08s 448, [iijbis (1S)COIFT pojNRSOD (oPThineees (36)
mntol [15]
Ransle seming asiuanee sesmestation [?]
Ropeorpee (7]
WHU P2yJoRPC 112/55D [64]
mAP
Medical image eqmemattoo (10]
UNcie- [67]
Lang segmettation [3KKsaer (27)
I:U [67]
desetinttt
(S0]
cnD0 (5))
Equation (1) by a positive integer scaling factor v (i.e., c/e) to reduce the impact of the generator. This can effectively reduce the error-minimizing strength of the noise, leaving more room for the optimization of the surrogate model. When applying the trained noise generator to protect images, we remove the scaling factor maintain the original value of e in Equation (1), thereby ensuring the effectiveness of the generated noise. We call this capability that can train under small epsilon via proportionally scaling while inference under large epsilon as epsilon generalization (EG). As shown in Figure 3, the training loss of UnSeg combined with EG decreases more steadily, ultimately reducing the target model's mIoU to around 7%. Additionally, we explore label modification technique to assess whether this change can induce the generator to produce noise that is more to
misleading. Specifically, during generator training, we change all background labels from align them with foreground labels. We empirically find that such a modification can strengthen the unlearnable effect of the generated noise as it forces the model to focus on the entire image.
The ISS Dataset De We employ the high-quality interactive segmentation dataset HQSeg-44K [29] as De to optimize the proposed pipeline. HQSeg-44K contains 44,320 images, each with extremely precise mask annotations, and covers more than 1,000 distinct semantic categories, which enhances the robustness of UnSeg to new data.
4 Experiments
4.1 Experimental Setup
Training Configuration
The weights of the noise generator are initialized using the pre-trained ViT-Base SAM [31]. The surrogate model adopts the same architecture SAM but initializes its backbone network with the MAE [22] pre-trained ViT-Base. We alternately optimize the noise generator and surrogate model: first, training the surrogate model for one epoch, followed by training the noise generator for three epochs. We use a total batch size of 32, a learning rate of 0.0001, and train our framework for 27 epochs, with learning rate decay after 20 epochs. Training UnSeg on 8 Nvidia GeForce RTX 3090 GPUs takes about 10 hours.
UEs Generation Unlike previous methods, our approach generates UEs for the mask-defined regions. We determine the masks by selecting the classes of objects be protected based on the ground-truth annotations. The masks and images are then passed into the trained noise generator to generate the unlearnable noise, which is then added back to the images to create UEs.
Evaluation Datasets and Models Table summarizes the considered tasks, models, and datasets in our experiments. For the three mainstream image segmentation tasks including semantic seg- mentation, instance segmentation, and panoptic segmentation, we select four widely used datasets: Pascal VOC 2012 [14], Cityscapes [13], ADE20K [66], and COCO [37]. Specifically, for semantic segmentation, we employ the representative convolutional segmentation models, Deepl.bbV [8] and DeepLabV3 [10], to test the effectiveness of our method on Pascal VOC 2012. We also consider three high-resolution, real-world datasets with more complex scenes: Cityscapes, ADE20K, and COCO2017, along with the state-of-the-art (SOTA) model Mask2Former [11] in this field. We follow the same procedure in the original paper [11] to train FaskeFormer. For interactive segmentation, SAM-HQ [29] is a nigh-quality interactive segmentation model improved upon SAM [31]. We train SAM-HQ [29] using the generated unlearnable dataset HQSeg-44; [29] and then evaluate the segmentation performance on four datasets (DIS [4S]/OIFT [36]/HRSOD [59]/hhinObjet [36)) as
described in the paper [29]. For remote sensing instance segmentation, we select three representative datasets: SSDD [64], WHU [28], and NWPU [12]. We use the SOTA model Rsprompter [7] the field and test our method's effectiveness following the training settings described in the paper [7].

---

For medical image segmentation, we evaluate our method using UNet++ [67] with five different backbones (ResNet50 [20//DenscNet163 [24/EfficiennNe1166 [53]/Res2Net [18]/RegNetX [47]) on the Lung segmentation dataset [2] and the K Kvasir-seg dataset [27]. We randomly select 80% of the data for training and use the remaining 20% for validation. We train Unet++ for total of 150 epochs.
Finally, we also test the cross-task effectiveness of our method on the COCO [37] dataset against SOTA object detection model DINO [60].
Evaluation Metries Table I also summarizes our evaluation metrics for each task. Please kindly refer to Appendix A.1 for more details about the metrics. In our experiments. lower metric values indicate poorer model performance and thus better effectiveness of our method.
Synter
Dars
Chasonts Nates
dus
hobsz
Syete
154
290
67x
Epoch
Iteration
Iteration
(a) Pascal VOC (DeepLabV1)
(b) Pascal VOC (DeepLabV3)
(c) Cityscapes (Mask2Former)
Figure 4: (a) The mIoU of DeepLabVI trained on unlearnable Pascal VOC. (b) The mIoU of DeepLabV3 trained on unlearnable Pascal VOC. (3) The PQ of Mask2Former trained on unlearnable Cityscapes. The values were shown over different training epochs/iterations of the models.
4.2 Main Results
Comparison to Random Noise and Synthetic Perturbations (SynPer) We first compare our UnSeg method with two types of random noise, i.e., class-wise random noise, sample-wist wise random noise, and the synthetic noise crafted by SynPer. These two types of noise have been shown to trigger an unlearnable effect classification datasets [25, 58]. For random noise, we sample independently and randomly from [-e, c] for each sample (c.g., sample-wise) or class (c.g., class-wise). For SynPer, we patch size of 8 to generate synthetic noise with the same resolution as our noise for fair comparison. We first experiment on Pascal VOC 2012 and then switch to different network architectures to test the wross-architecture capability of different methods. The results are shown in Figure 4 (a) and (b). As can be seen, although class-wise noise is highly effective in classification tasks, it only reduces the clean test mloU of models by 10%-20%. Comparing Figure 4 (a) and (b), SynPer can reduce the mIoU of DeepLabV3 to 11 11%, but can only reduce the mIoU of DeepLab V1 to 19%, showing its cross-architecture limitation. Our UnSeg consistently outperforms random and synthesized noise across different models by a large margin. It can reduce the mloU to around 7% on Pascal VOC 2012 against both DeepLabV and DeepLabV3. For a more complex real-world dataset Cityscapes, our method can reduce the PQ metric of Mask2Former by 56.4% (Figure 4 (c)), significantly outperforming SynPer across the entire training process.
Effectiveness and Transferability on Mainstream Image Segmentation Tasks We employ the SOTA Mask2Former model with two different backbones (ResNet50. Swin-Tiny) to comprehen- sively evaluate the effectiveness of our method across three widely-used datasets (COCO, ADE20K, Cityscapes) and three mainstream tasks (panoptic segmentation, semantic segmentation, instance segmentation). Here, we compare our UnSeg with three SOTA training-frea unlearnable methods: SynPer, Autoregressive Perturbations (AR) [51] and Convolution- based Perturbations (CUDA) [50]. For the AR method, we use its AR process to generate sample-wist noise of size = 1 (1.2 constraint) for each category. For the CUDA method, we use filters of size 3 and set the blur parameter to 0.3 to generate the noise for each category. The results reported in Table 2 reveal the following findings. (1) Our method significantly reduces the clean test performance of the models across all tasks and datasets. Specifically, UnSeg can reduce the PQ metric of ResNet50-basee models by 28%, 47.7%, and 56.4% across the three datasets, significantly outperforming SynPer and AR. (2) When using Swin-Tin Tiny as the backbone, our method is even more effective at reducing the model's performance. On the ADE20K dataset, UnSeg reduces the PQ and AP scores of the Swin-Tiny-ban model by 37.5% and 23.8% respectively. We believe this is because UnSeg's 's surrogate model adopts a Transformer architecture. (3) Our method is particularly effective for large datasets and large objects. For example, it can reduce the AP-L (Average Precision for Large Objects) metric
7

---

Table 2: The main results of UnSeg against the Mask2Former model in panoptic, instance, and semantic segmentation tasks, evaluated on ADE20K val, COCO val2017, and Cityscapes val. UnSeg significantly reduce the test performance of the models across different tasks and datasets. The best protection results are boldfaced. R50: ResNer50, Swin-T: Swin Transformer-Tiny.
Panoptic
Instance
Semantic
Dataset
Method
Backhone
PQ
Ap
mioU
AP
AP3
APM
APd.
mloU
R50
39.7
26.5
46.1
26.4
10.4
28.9
43.1
47.2
Clean
Swin
41.6
27.7
49.3
27.9
10.8
29.8
46.2
47.7
SynPer [58]
R50
18.6
13.6
28.7
9.3
7.1
13.4
9.7
25.4
ADE20K
AR [51]
37.8
43.1
43.9
CUDA 150]
R50
10.7
8.4
19.6
12,0
3,9
14.6
22.5
19.6
R50
11.7(28.1))
7.5.19.01)
17.7(28-11)
6.2/20.3))
4.065
8.6.0031)
7.3.35.81))
16.7(305))
UnSeg(Ours)
Swin-T
4.1(37.51)
3.4:24.3))
10.6/3874)
4.1(238J)
4.016.8.)
5.8:2400)
3.4(42.81)
7.8.29.11)
R50
51.9
41.7
61.7
43.7
23.4
47.2
64.8
Clean
Swin-T
53.2
43.3
63.2
45
24.3
48.3
67.4
COCO
SynPer [58]
R50
11.3
9.5
11
10.8
13.4
15.2
5
CUDA [50]
R50
6.7
4.7
11.2
9.77
10.9
18.8
R50
4.2(47.71)
3.2(3855)
5.2(57.5L)
4.0(39711)
5.8(1761)
3.7(435L)
1.766.11)
Unreg(Ours)
Swin-T
2.8 4.1(49.11)
6,0157.21)
2.7(42..J)
4,4(2011))
1.9446.4)
0.7(66.71)
R50
62.1
37.3
77.5
37.4
79.4
Clean
Swin
63.9
39.1
80.5
39.7
82,1
SynPer [58]
R50
30.1
23.0
37.1
20.5
25.5
Cityscapes
AR [51]
51.6
68.3
35.5
68.9
CUDA 150]
R50
51.6
31.4
69.1
29.9
65.8
R50
8.7(56.41)
1.1366.21)
7.8.69711)
2.3335.1J)
(0.
UnSeg(Ours)
Swin T
7.2(56.71)
1.7(37.4))
12.6.07.9))
1.5.382))
17.86616))
66
mlou(%)
48.
1227 +8000
KA0 1400
SAP
R852
0869
4Pfciena6 Reihhrr
Reghuax
(a) Interactive Segmentation
(b) Remote Sensing Segmentation
(c) Medical Image Segmentation
Figure 5: (a) The mIoU on datasets of SAM-HQ [29] trained unlearnable HQSeg-44k [29]. (b) The mAP on 3 datasets of RSPrompter [7] trained on their unlearnable training sets. (c) The loU on datasets [2. 27] of UNet++ [67] trained on their unlearnable training sets with 5 different backbones.
of Swin-Tiny-baal Tine-based Mask2Former to 0.7% in the COCO instance segmentation task. (4) On the Cityscapes dataset, our UnSeg significantly outperforms other methods across different tasks by considerable margin.
Effectiveness and Transferability on Related Vision Tasks Surprisingly, we found that UnSeg can effectively generalize to other related tasks that have very different image distributions. Please refer to Section 4.1 for detailed description of the datasets and models used in this set of experiments. For the interactive segmentation task, as shown in Figure (a), although SAM-HQ freezes the parameters of the pre-trained SAM during training to leverage its generalization capabilities, our method can

---

Tahle 3:
(%) DINO trained on clean and unlearnable COCO dataset.
'Tahle mloU (5B) Deepl Vah33 trained using different defense methods
on unlearnable Pascal VOC 2012 crafted by our UnSeg.
Method
AP
AP-S
AP-M
AP-L
Clean
No Defense
Gaussian
JPEG [40]
AT [43]
DDC-AT [56]
Clean
48.7
31.1
$1.9
62.9
75.1
5.8
7.3
44.8
23.1
28.5
UnSeg
6.1
9.3
5.7
2.4
Table 5: The ritll (5) of Derpl abV3 mained on clean vs. clean- unleumable mixed training dataset (Pascal VOC 2012).
Table Persmeter aealysis on Paseal VOC 2012 and Citysaaess EC: Equilon generalization, L.M: Label meditication. //X indicates that the method usedioot used.
Method
0%
20%
Clean Proportion
48%
60%
50%
100%
BG
Method
LM
Pascal VOC 2012 Semantic (mlot (%)
Acroplane
Bicycle
Multi-Clas
Bind
Boat
Dottle
Cityscopes Punoptic
APr miolUp..
Only
71.5
74.4
76.9
76.1
76.2
Clean
70.6
80.9
35.6
84.4
65.8
74.7
62.1
77.5
Mixed
7.0
72.4
74.5
75.2
76.1
K
19
68.7
29.1
8011
59.9
64.1
8.2
1.6
14,8
7.
42.1
31.7
67.
30
49.6
9.8
24
22
X
56.8
81.1
36.5
83
67.4
74.6
43.7
17
66.1
6.2
30,5
16.4
58.6
19.1
40.4
5.7
1.1
7.8
still degrade its test performance by a notable margin, especially on the ThinObject dataset [36]. This indicates that our method has the potential to prevent large foundation models from extracting useful information from the protected images. For remote sensing segmentation, RSPrompter an improved method based on SAM and also freezes the parameters of the pre-trained SAM during training. As illustrated in Figure (b), the test performance of RSPrompter drops significantly after training the unlearnable training dataset crafted by our method, with the mAP metric on the WHU dataset dropped by 60%. Our method can even be applied to protect medical images against medical image segmentation models. And, we found that setting reduce the model performance. As shown in Figure 5 (c), the test performance of UNet++ with different backbones trained our unlearnable datasets drops badly. The performance the Lung Segmentation dataset even degrades to 0%. Our method also has a strong cross-task effectiveness. To test this, we train the SOTA object detection model DINO the unlearnable COCO dataset initially generated by our UnSeg for segmentation tasks. The performance of the trained DINO is reported in Table 3, where it shows that our UnSeg can reduce the AP metric of DINO by 42.6%.
4/255 is sufficient to drastically
4.3 Additional Analyses
Resistance to Potential Defenses Here, we evaluate UnSeg against 4 potential defense methods, including Gaussian filtering, JPEG Compression (JPEG) [40], adversarial training (AT) [43], and DDC adversarial training [56] (DDC-AT, an advanced AT method specifically designed for segmentation tasks). Following [56], we choose white-box BIM (with Loc constraint) [32] to generate adversarial samples during training. We set the adversarial perturbation size to high value (==0.332555 to demonstrate the effectiveness of our method under a more harsh condition. As shown in Table 4, our method effectively resists DDC-AT, reducing the test mIoU to 28.5%. Amongst all the defense methods, JPEG the most effective. However, our UnSeg can still compromise its mIoU to 44.8% which is 30.3% lower than the original clean performance.
Mixing UEs with Clean Data In practical scenarios, not all training data need to be unlearnable. For example, only group of users adopt this technology to protect their data while others do not. This results in a partially unlearnable dataset with mixed clean and unlearnable examples. We simulate this scenario on Pascal VOC 2012 and report the result in Table 5. As can be inferred, the mloU on the mixed dataset is consistently lower than that on the clean dataset which 76.2% Moreover, the model's 'ss performance when trained on the mixed training set is the same as it was trained on the only the clean proportion of the training data. This implies that the UEs generated by UnSeg contribute (almost) nothing to model training. This result aligns with previous findings [25, 58, 16).
Parameter Analysis Here, we conduct comprehensive parameter analysis with two models on two datasets: 1) the Mask2Former model for panoptic segmentation on the Cityscapes dataset, and 2) the DeepLabV1 model for semantic segmentation on the Pascal VOC 2012 dataset. On the Pascal VOC dataset, we test two settings: a) making all classes unlearnable, and b) making only subset of classes (e.g., the first five classes) unlearnable. We note that the latter is more challenging setup. As the results presented in Table 6, incorporating EG significantly reduces most model metrics than that without EG. Meanwhile, we observe that solely relying on label modification technique to optimize the generator fails to achieve satisfactory protection performance due to unstable training. By integrating our EG strategy, the impact of the generator on the surrogate model can be significantly reduced, which eventually leads to much better performance. Our UnSeg also has limitations.

---

In the multi-class protection setting on the Pascal VOC dataset, for the bird and bottle categories, which both have simple shapes and textures, UnSeg can only reduce their mloU 58.6% and 40.4%, respectively. We leave the exploration of this limitation to our future work.
5 Conclusion
In this work, we propose a novel unlearnable example generation framework called UnSeg against image segmentation. UnSeg finetunes a universal and interactive unlearnable noise generator based on pre-trained SAM via min-min bilevel optimization framework. The unlearnable noise generator can be readily applied to protect images from being exploited by segmentation model training. Our UnSeg is data efficient, generation efficient, and more importantly, highly transferable to protect any image segmentation dataset. Our work establishes the first comprehensive baseline for unlearnable example research in image segmentation. It also provides useful insights into leveraging SAM-like foundational models to protect private data via unlearnable examples.
Acknowledgements
This work is partially supported by the National Key R&D Program of China (Grant No. 202 12804) and the National Natural Science Foundation of China (Grant No. 62276067 and 62372330).
References
[1] Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In WACV, 2021.
[2] Sema Candemir, Stefan Jaeger. Kannappan Palaniappan, Jonathan P Musco, Rahul K Singh, Zhiyun Xue, Alexandros Karargyris, Sameer Antani, George Thoma, and Clement J McDonald. Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration. IEEE transactions medical imaging, 33(2):577-590) 2013.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[4] Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. Segment anything in 3d with nerfs. In NeurlPS, 2023.
[5] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint 10.0923, 2023.
[6] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's referential dialogue magic. arXiv preprint arXwv:2306.15195 2023.
[7] Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang, Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. IEEE Transactiony Geoscience and Remote Sensing, 2024.
[8] Liang- Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint 1412 7267062, 2014.
[9] Liang-Chian Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-844 848,-888222 2017.
[10] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 0558777 2017.
10

---

[11] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attentine mask transformer for universal image segmentation. In CVPR, 2022.
[12] Gong Cheng, Junwei Han, Peicheng Zhou, and Lei Guo. Multi-class geospatial object detection and geographic image classification based collection of part detectors. ISPRS Journal of Photogrammetry and Remote Sensing, 98:1 19-132, 2014.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[14] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
[15] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 1 11:99-136, 2015.
[16] Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojeiech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. In NeurlPS, 2021.
[17] Shaopeng Fu, Fengxiang He, Yang Liu, Li Shen, and Dacheng Tao. Robust unlearnable examples: Protecting data against adversarial learning. arXiv preprint Stti::2203.14533: 2022.
[18] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin- Yu Zhang, Ming-Hsuan Yang, and Philip Torr. Res2net: new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 43(2):652 662, 2019.
[19] Hao He, Kaiwen Zha, and Dina Katabi. Indiscriminate poisoning attacks on unsupervised contrastive learning. In ICLR, 2023.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In CVPR, 2017.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[23] Kashmir Hill. The secretive company that might end privacy as we know it. In Ethics of Data and Analytics, pages 170-177 177. Auerbach Publications, 2022.
[24] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
(25) Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Un- learnable examples: Making personal data unexploitable. arXiv preprint arXi::2101.04899 2021.
[26] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. One- former: One transformer to rule universal image segmentation. In CVPR, 2023.
[27] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas De Lange. Dag Johansen, and Hávard D Johansen. Kvasir-seg: A segmented polyp dataset. In MMM, 2020.
[28] Shunping Ji, Shiqing Wei, and Meng Lu. Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set. JEEE Transactions on geoscience and remote sensing, 57(1):574-58 2018.
[29] Lei Ke, Mingqiao Ye, Martin Danelljan, Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. In NeurIPS, 2023.
[30] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár, Panoptic segmentation. In CVPR, 2019.
11

---

[31] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, al. Segment anything. In ICCV, 2023.
[32] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safery and security, pages 99-112. Chapman and Hall/CRC, 2018.
[33] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308.0000 2023.
[34] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified bansformer-basee framework for object detection and segmentation. In CVPR, 2023.
[35] Xiangtai Li, Henghui Ding, Wenwei Zhang, Haobo Yuan, Jiangmiao Pang, Guangliang Cheng, Kai Chen, Ziwei Liu, and Chen Change Loy. Transformer-based visual segmentation: A survey, arXiv preprint arXiv:2304.080 09854, 2023.
[36] Jun Hao Liew, Scott Cohen, Brian Price, Long Mai, and Jiashi Feng. Deep interactive thin object selection. In WACV, 2021.
[37] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[39] Yixin Liu, Kaidi Xu, Xun Chen, and Lichao Sun. Stable unlearnable example: Enhancing the robustness of unlearnable examples via stable error-minimizing noise. In AAAI, 2024.
[40] Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Image shortcut squeezing: Countering perturbative availability poisons with compression. In ICML, 2023.
[41] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
[42] Jun Ma. Yuting He, Feifci Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
[43] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.0608. 06083, 2017.
[44] Shervin Minaee, Yuri Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(7):3523-3442, 2021.
[45] Xuebin Qin, Hang Dai, Xiaobin Hu, Deng-Ping Fan, Ling Shao, and Lue Van Gool. Highly accurate dichotomous image segmentation. In ECCV, 2022.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[47] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In CVPR, 2020.
[48] Jie Ren, Han Xu, Yuxuan Wan, Xingjun Ma, Lichao Sun, and Jiliang Tang. Transferable unlearnable examples. arXiv preprint arXiv:2210.10114 2022.
[49] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
12

---

[50] Vinu Sankar Sadasivan, Mahdi Soltanolkotabi, and Soheil Feizi. Cuda: unlearnable datasets. In CVPR, 2023.
Convolution- -ased
[51] Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, Tom Goldstein, and David Jacobs. Autoregressive perturbations for data poisoning. In NeurlPS, 2022.
[52] Pedro Sandoval-Segura, Vasu Singla, Jonas Geiping, Micah Goldblum, and Tom Goldstein. What we learn from unlearnable datasets? In NeurlPS, 2023.
[53] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
[54] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077 2023.
[55] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In NeurlPS, 2021.
[56] Xiaogang Xu, Hengshuang Zhao, and Jiaya Jia. Dynamic wwrdd--and-conque adversarial training for robust semantic segmentation. In ICCV, 2021.
[57] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
[58] Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tin Liu. Availability attacks create shortcuts. In KDD, 2022.
[59] Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. In ICCV, 2019.
[60] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint w:Xhh:2003.0360. 2022.
[61] Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mp-former: Mask-piloted transformer for image segmentation. In CVPR, 2023.
[62] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.0223 029499 2023.
[63] Jiaming Zhang, Xingjun Ma, Qi Yi, Jitao Sang, Yu-Gang Jiang, Yaowei Wang, and Changsheng Xu. Unlearnable clusters: Towards label-agnostic unlearnable examples. In CVPR, 2023.
[64] Tianwen Zhang, Xiaoling Zhang, Jianwei Li, Xiaowo Xu, Baoyou Wang, Xu Zhan, Yanqin Xu, Xiao Ke, Tianjiao Zeng, Hao Su, al. Sar ship detection dataset (ssdd): Official release and comprehensive data analysis. Remote Sensing, 3(18))369,, 2021.
[65] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[66] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
[67] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE transactions medical imaging, 3966):1856-1866 2019.
13

---

A Appendix
A.1 More details about evaluation metrics
We first employ the trained interactive unlcarnable noise generator transform the training datasets for each task into their corresponding unlearnable versions. Subsequently, the models are trained on these unlearnable datasets and then evaluated their respective clean validation datasets. For evaluation, we employ various segmentation metrics across different tasks. For panoptic segmentation, utilize the standard PQ (Panoptic Quality) metric [30]. Additionally,
report APTh pin. which is the Average Precision calculated for "thing' categories using instance segmentation annotations. We also report mloU pan: representing the mean Intersection-over- Union for semantic segmentation, achieved by merging instance masks from the same category within a model trained exclusively with panoptic segmentation annotations. For instance segmentation, we apply the standard AP (Average Precision) metric [37]. For semantic segmentation, we use the mloU (mean Intersection-over: -ovetioonn-ve-nnnonn metric [15]. For interactive image segmentation, we use the mloU following [29]. For remote sensing instance segmentation, we report mean average precision (mAP) as [7]. For medical image segmentation, we report IoU (Intersection-or -nion.. For object detection, we report AP (Average Precision). In our experiments, lower metric values indicate poorer model performance and thus better effectiveness of our method.
A.2 Additional Analyses
Table 7: Prompt analysis on Pascal VOC 2012 using DeepLabV1.
Pascal VOC 2012 Semantic (mIoU (%))
Prompt Type
Multi-Class Bicycle
All
Acroplane
Bird
Boat
Bottle
Clean
70.6
80.9
35.6
84.4
65.8
74.7
Point prompt
6.0
27.2
17.8
55.0
18.1
29.2
Box prompt
6.4
41.2
24.4
60.1
23.7
38.1
Mask prompt
6.2
30.5
16.4
58.6
19.1
40.4
Prompt Comparison When using the trained UnSeg to generate unlearnable examples for down- stream images, we consider only object masks as prompts, rather than bounding boxes or points, in order to eliminate ambiguity. For example, in image containing person, if defender clicks point on the person's 's face to make it unlearnable, the model would be uncertain about what the point refers to: the entire person just the face? It is also unclear to the defender which object/region has been protected, leading to ambiguity. Using object masks as prompts enables precise specification of the objects to be protected, effectively avoiding the aforementioned ambiguity. In fact, with powerful tools like SAM,, it is quite easy to obtain object masks. Here, we provide more experiments in Table 7 with different prompts the Pascal VOC dataset using DeepLabVV. We report types of results: making all classes unlearnable and making only some classes unlearnable. It shows that our method is robust across different types of prompts, achieving excellent protection with point prompts.
Table 8: The mloU (%) of DeepLabVI trained on clean vs. clean-unlearnable mixed training dataset (Pascal VOC 2012).
Clean Proportion
Method
0%
20%
40%
60% %
80%
100 %
Clean Only
69.0
69.6
69.8
70.5
70.5
Mixed Data
7.0
66.6
68.2
69.9
70.4
Mixing UEs with Clean Data Here, we present additional experimental results obtained using the DeepLabVV on the Pascal VOC dataset. Then, we conduct new experiments using UNet++ with different backbones on the Kvasir-seg dataset. As shown in Table 8 and Table 9, we find that: 1) The
14

---

Table 9: The mloU (%) of UNet++ trained clean clean-unlearnable mixed training dataset (Kvasir-seg).
Clean Proportion
Method
Backbone
0%
20 %
40%
60 %
80 %
100 %
Clean Only
ResNet50
67.3
70.1
71.0
71.6
72.3
DenseNet169
69.3
70.7
72.1
72.2
73.6
EfficientNetB6
69.7
71.2
73.5
72.7
74.0
Res2Net
67.6
70.8
71.2
71.7
73.6
RegNetX
68.1
69.2
71.1
71.3
72.6
Mixed Data
ResNet50
2.5
67.2
68.7
70.3
71.8
DenseNet169
6.0
69.0
69.5
71.2
72.4
EfficientNerB66
7.4
70.6
71.9
73.3
73.2
Res2Net
6.7
68.7
70.5
71.7
72.6
RegNetX
2.1
69.8
69.7
71.1
71.4
results on the mixed datasets are consistently lower than those on the 100% clean dataset. Moreover, the model's performance when trained the mixed training set is the same when trained only on the clean portion of the training data. This implies that the UEs generated by UnSeg contribute almost nothing the model's training. This result aligns with existing works (UEs, UCs, SynPer, AdvPoison, etc.). 2) The model trained on a mixed dataset sometimes performs worse than the model trained on only clean data. This indicates that our method may also hinder the model's learning on clean data to some extent.
Table 10: The results of UnSeg using different models as surrogate models.
ADE20K Panoptic
Cityscapes Panoptic
Surrogate Model
Backbone
PQ
APTh pan
mIoU
PQ
AP APTh pan
mlou
pan
pan
Clean
RN50
39.7
26.5
46.1
62.1
37.3
77.5
Swin-Tim Twin-Tiny
41.6
27.7
49.3
63.9
39.1
80.5
MAE VIT
RN50
11.7
7.5
17.7
5.7
1.1
7.8
Swin-Tiny
4.1
3.4
10.6
7.2
1.7
12.6
Supervised RN50
RN50
35.6
23.4
43.5
42,5
16.5
64.8
Swin-Tiny Tiny
28.4
19.7
39.7
34.2
57.2
11.6
Influence of the Surrogate Model Here, we replace the surrogate model from the ImageNet MAE pre-trained VIT with ImageNet supervised pre-trained ResNet50. We evaluate the performance of the noise generator trained with different surrogate models using Mask2Former on the ADE20K and Cityscapes datasets. As shown in Table 10. the performance of the noise generator significantly declined when using ResNet50 as surrogate model. We believe that using the MAE pre-trained weights can prevent the trained noise generator from being biased towards specific categories, thereby enhancing its transferability. We agree that the surrogate model plays a crucial role in optimizing the noise generator, and further exploration of different surrogate models would be valuable.
Table 11: The impact of different initialization methods for the generator model.
Pascal VOC 2012 Semantic (mloU (%))
Initialization Method
Multi-Class Bicycle
All
Aeroplane
Bird
Boat
Bottle
Clean
70.6
80.9
35.6
84.4
65.8
74.7
Pretrained SAM
6.2
30.5
16.4
58.6
19.1
40.4
Random
4.8
28.1
6.2
42.1
6.5
25.9
15

---

Pixel
3050
Sample
500
Pixel
(a) Clean Image
(b) Unlcarnable Image
(c) Comparison
Figure 6: Comparison of pixel value distributions between the clean images and the unlearnable
images generated by UnSeg.
Influence of the Initialization Method Here, employ different methods to initialize the weights of the generator to observe their impact. Our primary consideration in using SAM is to ensure that the noise generator possesses promptable attributes after optimization. Therefore, we initialized the generator with the pre-trained SAM weights and kept these weights frozen during training. This allows efficient fine-tuning of the newly added parameters. Here, we further test the use of randomly initialized weights for the noise generator. We run the experiments on the Pascal VOC dataset using Ve. 1. We report two types of results in the table below: making all classes unlearnable and making only some classes unlearnable. The results as shown in Table indicate that our framework sensitive to the initial weights and the noise generator with random initialization also works well.
A.3 Broader Impacts
UnSeg presents an elegant and effective framework for generating unlearnable examples using large foundation models, which may inspire researchers to explore similar approaches with other large models, such as LLaVA [38] or CLIP [46]. UnSeg provides the first comprehensive benchmark for unlearnable examples in image segmentation and introduces method that has been experimen- tally validated effectively counteract segmentation models, thereby paving the way for future research. Furthermore, UnSeg reveals that both traditional convolutional models and state-of-the-ar transformer-be based segmentation models are highly vulnerable slight perturbations and struggle to learn generalized features when trained on unlearnable datasets. This revelation may drive further advancements robust training methodologies.
A.4 Visualization
In Figure 6, we plot the pixel value distribution of the clean image and its unlearnable counterpart, where the two distributions are almost the same.
In Figure 7, we visualize the Class Activation Maps (CAMs) of DecpLabVI models trained separately on clean and unlearnable datasets.
In Figure 8-17, we visualize several clean images, their unlearnable counterparts generated by UnSeg. and the corresponding unlearnable noise from ten evaluation datasets.
16

---

(a)
(b)
(c)
(d)
(e)
Figure 7: Visualization results of DeepLabVI trained on different datasets. (a) Validation images from the Pascal VOC 2012 dataset. (b)-(c) Attention maps and predictions of DecpLabVi trained on the clean dataset. (d)-(e) Attention maps and predictions of DeepLabV] trained on our generated unlearnable dataset.
Figure 8: Visualization results on the Pascal VOC 2012 dataset. (a) Clean images. (b) Unlearnable examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg.
17

---

(a)
(b)
(c)
Figure 9: Visualization results the ADE20K dataset. (a) Clean images. (b) Unlearnble examples
generated by UnSeg, (c) Unlearnable noise generated by UnSeg
(a)
(b)
(c)
Figure 10: Visualization results on the COCO dataset. (a) Clean images. (b) Unlearnable examples
generated by UnSeg. (c) Unlearnable noise generated by UnSeg.
(A)
(c)
Figure 11: Visualization results on the Cityscapes dataset. (a) Clean images. (b) Unlearnable
examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg
18

---

Mi
Kits
Figure 12: Visualization results on the HQSeg-44 dataset. (a) Clean images. (b) Unlearnable
examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg
(a)
Figure 13: Visualization results on the WHU dataset. (a) Clean images. (b) Unlearnable examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg.
(a)
(b)
Figure 14: Visualization results on the SSDD dataset. (a) Clean images. (b) Unlearnable examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg.
19

---

Figure 15: Visualization results on the NWPU dataset. (a) Clean images. (b) Unlearnable examples
generated by UnSeg. (c) Unlearnable noise generated by UnSeg.
(a)
(b)
Figure 16: Visualization results the Lung segmentation dataset. (a) Clean images. (b) Unlearnable
examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg
(a)
(b)
(c)
Figure 17:
Visualization results on the Kvasir-seg dataset. (a) Clean images. (b) Unlearnable
examples generated by UnSeg. (c) Unlearnable noise generated by UnSeg
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions of this paper are clearly presented in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We note that our method still has limitations and room for improvement in our experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
Information related the experiment is detailed in the experiment setup
section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Source code will be released to foster the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nigg cc/
public/ guides/ Code woroomissiion ssisssssisiiiissss ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: i msips. cc/ cc/publicc c/ i des CCodeo ss onPol ecy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All relevant settings are described in the experimental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
The experimental results in this paper require further error analysis to be reported.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the training settings for each experiment in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttp:::///euuure htw./eeurppp.ppeee cc httc//pblcc/thhcccceocc
Answer: [Yes]
Justification: This research complied fully with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We explore some possible societal impacts of our method in the conclusion and appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pre-trained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper proposes a new data protection method, so there is no safety risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Relevant sources used in this paper are appropriately cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use package should be provided. For popular datasets, hapersw's thoode. com / datases sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code will be released foster the community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The experiments in this paper does not include these contents.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The experiments in this article did not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "Onhy8TgSJK", "reviewer": "Reviewer_2xcM", "summary": "The paper addresses the issue of privacy concerns in training large-scale image segmentation models using unauthorized private data. The authors propose a novel framework called Unlearnable Segmentation (UnSeg) to generate unlearnable noise that, when added to images, makes them unusable for model training. This framework involves training a universal unlearnable noise generator by fine-tuning the Segment Anything Model (SAM) through bilevel optimization on an interactive segmentation dataset. The effectiveness of UnSeg is demonstrated across six mainstream image segmentation tasks, ten widely used datasets, and seven different network architectures, significantly reducing segmentation performance when unlearnable images are used.", "questions": "See Q1 and Q3 in weaknesses, pls.", "limitations": "The authors have introduced the limitations and broader impact of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation and the proposed method are impressive. The work addresses three key challenges in generating unlearnable examples: data efficiency, generation efficiency, and transferability. The proposed method effectively employs the Segment Anything Model (SAM) to tackle these challenges.\n2. Extensive experiments and ablation studies are conducted to evaluate the proposed method.", "weaknesses": "1. The generation of unlearnable examples (UEs) requires object masks, which can be challenging to obtain in common practice. The authors should discuss this issue and consider using bounding boxes or clicks as alternative options. Reporting the results of these alternatives would be beneficial.\n2. Why is the surrogate model initialized with the pretrained MAE model weights instead of SAM weights?\n3. As shown in Table 5, the model trained with a mixed dataset of clean and unlearnable data achieves similar or even better performance compared to the model trained with only clean data. This suggests that unlearnable data might not reduce performance and could even enhance it. Consequently, network trainers might still collect images without concern, despite the proposed method, unless all images available on the internet are unlearnable samples.\n4. Can the authors provide more results from different networks and datasets using the clean-unlearnable mixed training dataset in Table 5?\n5. Table 2 only compares one state-of-the-art (SOTA) method. The authors should include more SOTA methods, including attack methods for segmentation and classification, to effectively validate the superiority of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful and valuable reviews.\n\n**Q1: Discussion about object masks, and more experiments regarding different prompts.**\n\nThe reason why we consider only object masks as prompts, rather than boxes or points, is to **eliminate ambiguity** in the prompts. For example, in an image containing a person, if a defender clicks on a point on the person's face to make it unlearnable, the model would be uncertain about what the point refers to: the entire person or just the face? It is also unclear to the defender which object/region has been protected, leading to ambiguity. Using object masks as prompts enables precise specification of the objects to be protected, effectively avoiding the aforementioned ambiguity. In fact, with powerful tools like SAM, it is quite easy to obtain the object masks. \n\nHere, we provide more experiments with different prompts on the Pascal VOC dataset using DeepLabV1. We report two types of results: making all classes unlearnable and making only some classes unlearnable. It shows that our method is robust across different types of prompts, achieving excellent protection even with point prompts. \n|Prompt Type |All class|Multi-class| | | | | \n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| | |Aeroplane |Bicycle |Bird |Boat |Bottle|\n|Clean|70.6|81|36|84|66|72|\n|Point|6|27.2|17.8|55|18.1|29.2|\n|Box|6.4|41.2|24.4|60.1|23.7|38.1|\n|Mask|6.2|30.5|16.4|58.6|19.1|40.4|\n\n**Q2: Why is the surrogate model initialized with the pretrained MAE model weights instead of SAM weights?**\n\nOur bi-level (min-min) optimization alternatively optimizes the surrogate model and the noise generator to minimize the loss between the surrogate model's predictions and the true labels. This means if the surrogate model is initialized with the pre-trained SAM weights, it will lead to a very low initial loss as SAM is already sufficient for high-quality predictions. In this case, the noise generator will no longer need training. In other words, using SAM weights will hinder the optimization of the noise generator. The pre-trained MAE weights, on the other hand, can alleviate this problem and are also a common initialization for segmentation tasks. \n\n**Q3: More experiments from different networks and datasets using the clean-unlearnable mixed training dataset.**\n\nThanks for your suggestion. Here, we add the results obtained using the DeepLabV1 on the Pascal VOC dataset. \n|Method|0%|20%|40%|60%|80%|100%|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Clean Only|-|69|69.6|69.8|70.5|70.5|\n|Mixed Data|7|66.6|68.2|69.9|70.4|-|\n\nThen, we conduct new experiments using UNet++ with 5 different backbones on the Kvasir-seg dataset.\n|Method|backbone|0% Clean|20% Clean|40% Clean|60% Clean|80% Clean|100% Clean|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Clean Only|RN50|-|67.3|70.1|71|71.6|72.3|\n| |D169|-|69.3|70.7|72.1|72.2|73.6|\n| |EfficientB6|-|69.7|71.2|73.5|72.7|74|\n| |Res2Net|-|67.6|70.8|71.2|71.7|73.6|\n| |RegNetX|-|68.1|69.2|71.1|71.3|72.6|\n|Mixed Data|RN50|2.5|67.2|68.7|70.3|71.8|-|\n| |D169|6|69|69.5|71.2|72.4|-|\n| |EfficientB6|7.4|70.6|71.9|73.3|73.2|-|\n| |Res2Net|6.7|68.7|70.5|71.7|72.6|-|\n| |RegNetX|2.1|69.8|69.7|71.1|71.4|-|\n\nWe find that: 1) All the results on the mixed datasets are consistently lower than those on the 100% clean. Moreover, the model’s performance when trained on the mixed training set is the same as when trained only on the clean portion of the training data. This implies that the UEs generated by UnSeg contribute almost nothing to the model's training. This result aligns with existing works (UEs, UCs, SynPer, AdvPoison, etc.). 2) The model trained on a mixed dataset sometimes performs worse than the model trained on only clean data. This indicates that our method may also hinder the model's learning on clean data to some extent.\n\n**Q4: More comparisons with SOTA methods.**\n\nWe believe that previous optimization-based (compared to training-free or generative) methods (e.g., UEs/RUEs/TUEs/AdvPoison) face data efficiency, generation efficiency, and transferability challenges, as they need to optimize the unlearnable noise for each new sample. These challenges render them impractical for image segmentation which has diverse scenarios (datasets), higher resolution and complex models. Therefore, we only considered training-free methods (no generative methods available) for comparison in Table 2.\n\nHere we test two more SOTA training-free unlearnable methods: Autoregressive Perturbations (AR) [a] and Convolution-based Perturbations (CUDA) [b]. **Please refer to the uploaded PDF for the complete results**. For the AR method, we use its AR process to generate sample-wise noise of size $\\epsilon=1$ (L2 constraint) for each category. For the CUDA method, we use filters of size 3 and set the blur parameter to 0.3 to generate the noise for each category.\n|Dataset|Method|Panoptic| | |Instance| | | |Semantic|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| | |PQ|AP-Pan|mIoU-Pan|AP|AP-S|AP-M|AP-L|mIoU|\n|ADE20k|Clean|39.7|26.5|46.1|26.4|10.4|28.9|43.1|47.2|\n| |AR|37.8|24.9|43.1|25.4|9.4|27.7|43.3|43.9|\n| |Cuda|10.7|8.4|19.6|12|3.9|14.6|22.5|19.6|\n| |UnSeg(Ours)|11.7|7.5|17.7|6.2|5|8.6|7.3|16.7|\n|COCO|Clean|51.9|41.7|61.7|43.7|23.4|47.2|64.8|-|\n| |Cuda|6.7|4.7|11.2|9.7|3.7|10.9|18.8|-|\n| |UnSeg(Ours)|4.2|3.2|5.2|4|5.8|3.7|1.7|-|\n|Cityscapes|Clean|62.1|37.3|77.5|37.4|-|-|-|79.4|\n| |AR|51.6|36|68.3|35.5|-|-|-|68.9|\n| |Cuda|51.6|31.4|69.1|29.9|-|-|-|65.8|\n| |UnSeg(Ours)|5.7|1.1|7.8|2.3|-|-|-|10.9|\n\nAs can be observed, 1) AR loses effectiveness in preventing the segmentation models; 2) CUDA works well on the ADE20K dataset yet inferior to UnSeg on the COCO and Cityscapes dataset; and 3) UnSeg achieves the best performance. We genuinely believe our UnSeg can serve as a solid baseline for segmentation UEs.\n\n[a] Autoregressive perturbations for data poisoning, Pedro et.al., NeurIPS 2022\n\n[b] Cuda: Convolution-based unlearnable datasets, Sadasivan et.al., CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUeZkBUY1R", "reviewer": "Reviewer_PRaH", "summary": "Aiming to provide a solution for protecting sensitive/private images, this paper introduces UnSeg, a framework designed to generate unlearnable examples (UEs) to protect images from unauthorized usage in image segmentation models. Utilizing the Segment Anything Model (SAM) and bilevel optimization, UnSeg creates noise that renders images unusable for training segmentation models. The framework is validated through extensive experiments across multiple segmentation tasks, datasets, and architectures (listed in Table 1).", "questions": "1. Is the method applicable when the model is other than the SAM?\n\n2. Will the method work if the segmentation model leverages contextual knowledge in the input images to create better object masks?", "limitations": "Authors do not describe any limitations clearly. The reviewer feels the if the surrogate model is not strong/robust enough, the perturbations might not be potent to break the target segmentation models. The reviewer did not find any particular negative societal impact.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- **(Writing)** The reviewer enjoyed reading the paper and the motivation of the idea. Great job, authors!\n\n- **(Method)** The reviewer finds the paper to be quite interesting. The idea of the paper is to leverage a foundational model like SAM to add perturbations to a local area that cannot be segmented. Further, the reviewer is impressed with the approach to train a universal perturbation generator that can be readily applied to craft unlearnable noise for any given image in one single forward pass. This is a distinct property of image-agnostic adversarial attacks and its use in this paper’s setting is intuitive.\n\n- **(Experiment)** The experiment section is comprehensive. Comparison on three benchmarks on Table 1 shows strong performance of the proposed method. Further, the proposed method also tests the strategy on non-standard benchmarks like Remote Sensing Segmentation and  Medical Image Segmentation in Figure 5. The paper also investigates the impact of perturbations on object detection Table 3 and against deployed defenses in Table 4.", "weaknesses": "The reviewer did not find major flaws with the paper.\n- **(Method)** The reviewer feels there will be restrictions due to prevalent memory bandwidth for adoption of this method. The authors can expand on the real-world use cases of the proposed method (in terms of deployment).\n- **(Method)** The reviewer doesn’t clearly understand the intuition behind the Section “Training Stability and Epsilon Generalization”. The point of segmentation methods being more fine-grained doesn’t justify balancing the perturbations by scaling factor $v$. \n- **(Experiments)** The reviewer fails to understand why the methods in [48, 61] do not serve as proper baselines. Since the authors compare their method in cross domain/task settings (like in Table 3), these works also serve as good baselines.\n- **(Experiments)** This paper would definitely benefit from analysis of statistical significance. Localized perturbations may get distributed in different patterns and hence may affect the segmentation performance.  \n- **(Typo)** L223, 220 v -> $v$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your insightful, constructive and encouraging reviews.\n\n**Q1: More details on the real-world use cases and deployment of the proposed method.**\n\nOur trained universal unlearnable noise generator is very lightweight and can be deployed similarly to SAM. Specifically, the noise generator occupies only 385.37 MB of memory. During inference on a single RTX 4090 GPU, the maximum VRAM usage is 2893 MB, and it only takes 113.52 ms to generate UEs for a single image, utilizing 2.76 GB of physical memory.\n\n**Q2: The intuition behind “Training Stability and Epsilon Generalization”.**\n\nWhile image classification models only focus on specific features of an image (as shown by CAM [a]),  segmentation models must make accurate predictions for every pixel. Such a difference makes segmentation models more sensitive to pixel-level perturbations. Moreover, the bi-level optimization process optimizes a surrogate model and a noise generator, both aimed to reduce the prediction loss. The $\\epsilon$ hyperparameter works as a knob to balance the two components. During our exploration, we find that a larger $\\epsilon$ tends to hinder the training of the noise generator, as the added noise causes an extremely low loss. To solve this problem, we chose to use a smaller epsilon when training the noise generator, then magnify the noise at inference time to guarantee unlearnable effect. For this strategy to work, training under a smaller epsilon should be able to generalize to inference with a large epsilon (we call this **epsilon generalization**). The effectiveness of our method proves that this novel technique indeed works.\n\n[a] Learning deep features for discriminative localization, Zhou et.al., CVPR 2016\n\n**Q3: Why the methods in [48, 61] do not serve as proper baselines.**\n\nYes, these methods are indeed excellent. However, they were all designed for image classification and adapting them for segmentation models is very challenging. Therefore, in Table 2, we only considered training-free SynPer for comparison. To address your concern, we test two more SOTA unlearnable methods: Autoregressive Perturbations (AR) and Convolution-based Perturbations (CUDA). Due to space limitations, please refer to our response to Q4 of Reviewer 2xcM and the uploaded PDF file for further details. In summary, UnSeg consistently achieves better performance compared to other methods across different datasets and tasks. We genuinely believe that the high efficiency and transferability of UnSeg will establish it as a solid baseline for seg UEs.\n\n**Q4: Analysis of statistical significance.**\n\nThanks for the thoughtful suggestion. As the unlearnable noise is small, the unlearnable images are statistically indistinguishable from the original images. To show this, we plot the pixel value distribution of a clean image and its unlearnable counterpart, where the two distributions are almost the same (the plots can be viewed in the PDF files we uploaded). We will add more plots to the revision.\n\n**Q5: Typo error on L223, 220.**\n\nThanks. We will fix the error in the revision.\n\n**Q6: Is the method applicable when the model is other than the SAM?**\n\nYes.  As a simple and flexible framework, UnSeg can potentially work with segmentation models of a similar architecture to SAM, such as SEEM [a], Semantic-SAM, and Grounded SAM. UnSeg can also work with the recently released SAM 2 [b] model, which we plan to verify in our future work. \n\n[a] Segment everything everywhere all at once, Zou et.al., NeurIPS 2023\n\n[b] SAM 2: Segment Anything in Images and Videos, Ravi et.al,  arXiv preprint arXiv:2408.00714 (2024)\n\n**Q7: Will the method work if the segmentation model leverages contextual knowledge to create better object masks?**\n \nYes. For example, both DeepLabV1 and DeepLabV3 use dilated convolution to leverage region-level contextual knowledge. However, our method can reduce their mIoU to below 10% (please see Figure 4). Mask2Former employs a significant amount of masked transformers to learn long-range contextual knowledge, and our UnSeg can work effectively against it (Table 2). UnSeg also works on SAM-HQ and RSPrompter which both utilize SAM’s pre-trained knowledge (Figure 5 (a)(b)).\n\nUnlike classification UE methods which add noise to the entire image, UnSeg proves that adding unlearnable noise locally/partially is sufficient to prevent the learning of segmentation models.\n\n**Q8: The limitation of UnSeg and the influence of the surrogate model.**\n\nOne limitation of UnSeg is its moderate effectiveness in protecting objects with very simple textures and shapes, such as birds or bottles. We believe this is because the simplicity of these categories allows the segmentation models to make accurate predictions without relying on shortcuts. We hope to address this limitation in our future work.\n\nWe agree with the reviewer that the surrogate model plays a crucial role in optimizing the noise generator. As noted in Unlearnable Clusters, using CLIP as the surrogate model can enhance unlearnable effectiveness. Stable UEs also shows that more robust surrogate models improve the protection effectiveness.\nHere, we replace the surrogate model from the MAE-pretrained ViT with an ImageNet-pretrained ResNet50.\nWe find that the performance of the noise generator significantly declined when using ResNet50.\nWe believe that using the MAE-pretrained weights can prevent the noise generator from being biased towards specific categories, thereby enhancing its transferability.\n|Surrogate Model|Backbone|ADE20K Panoptic| | |Cityscapes Panoptic| | |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|||PQ|AP-Pan|mIoU-Pan|PQ|AP-Pan|mIoU-Pan|\n|Clean|ResNet50|39.7|26.5|46.1|62.1|37.3|77.5|\n| |Swin-Tiny|41.6|27.7|49.3|63.9|39.1|80.5|\n|MAE pretrained ViT|ResNet50|11.7|7.5|17.7|5.7|1.1|7.8|\n| |Swin-Tiny|4.1|3.4|10.6|7.2|1.7|12.6|\n|ImageNet Pretrained ResNet50|ResNet50|35.6|23.4|43.5|42.5|16.5|64.8|\n| |Swin-Tiny|28.4|19.7|39.7|34.2|57.2|11.6|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IUsJU6rOl", "reviewer": "Reviewer_zWqb", "summary": "This article uses the powerful SAM to fine-tune a non-learnable examples generator, achieving good protection effects on downstream datasets and models.", "questions": "Please show experiments on adversarial training on non-learnable examples.", "limitations": "This paper could not cause any societal impacts.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This article breaks through the concept of unlearnable from classification to segmentation, and the experiment has achieved good results.", "weaknesses": "I am mainly concerned about whether the unlearnable samples proposed in this paper on segmentation have much practical application? The author used a relatively strong segmenter to train an unlearnable sample generator, but trained unlearnable samples on a relatively weak segmenter. Is this reasonable in real world?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback.\n\n**Q1: Main concern about the author used a relatively strong segmenter to train an unlearnable sample generator, but trained unlearnable samples on a relatively weak segmenter. Is this reasonable in the real world?**\n\nWhile we understand the reviewer’s concern, we would like to argue that our settings are both reasonable and practical in real-world applications. As the current AI and large-scale pre-training are devouring our data and destroying our privacy, Unlearnable Examples (UEs), as a promising data protection technique, has become an active research area in the past two years, and most existing works [a,b,c,d,e,f,g,h] follow a similar setting as ours. \n\nThe use of SAM (i.e., a strong segmenter) follows the principle of **using large models against large models**, and this can produce a single powerful generator to provide universal protection against many downstream tasks. Please note that our setting is not only reasonable but also very challenging as the training dataset of the noise generator is completely different from the downstream datasets and the generated noise should be powerful enough to **prevent model training** (which is more powerful than adversarial noise as it only prevents inference). The trained unlearnable noise generator by our method can readily be applied by the data owners to protect their private data (which are not necessarily segmentation data).  With our protection, the data won’t be easily exploited by a machine learning model **even if the data is accidentally leaked**. \n\nMoreover, we employed **state-of-the-art (SOTA) models in each domain** to evaluate our method on downstream tasks. For example, the Mask2Former model we tested is a SOTA model for universal image segmentation and has become a recognized milestone in this area. Similarly, the SAM-HQ, RSPrompter, and DINO models are also SOTA in their respective domains, all of which are strong enough to assess the effectiveness of our method.\n\n[a] Unlearnable examples: Making personal data unexploitable, Huang et.al., ICLR 2021\n\n[b] Adversarial examples make strong poisons, Fowl et.al., NeurIPS 2021\n\n[c] Unlearnable clusters: Towards label-agnostic unlearnable examples, Zhang et.al., CVPR 2023\n\n[d] Availability attacks create shortcuts, Yu et.al., SIGKDD 2022\n\n[e] Autoregressive perturbations for data poisoning, Pedro et.al., NeurIPS 2022\n\n[f] Cuda: Convolution-based unlearnable datasets, Sadasivan et.al., CVPR 2023\n\n[g] Transferable unlearnable examples, Ren et.al., ICLR 2023\n\n[h] Robust unlearnable examples: Protecting data against adversarial learning, Fu et.al., ICLR 2022\n\n**Q2: Experiments on adversarial training.**\n\nThanks for your suggestion. In fact, we have already evaluated our method against adversarial training (**AT**) and an advanced AT method specifically designed for segmentation tasks (**DDC-AT**). Please kindly find the copied results in the table below, a detailed discussion can be found in Section 4.3 of our initial submission. As shown in the table, our method is resistant to both AT and DDC-AT, reducing the test mIoU to 23.1 % and 28.5%, respectively.\n\n|Clean|No Defense |AT|DDC-AT|\n|:---:|:---:|:---:|:---:|\n|75.1|5.8|23.1|28.5|\n\n**Q3: This paper could not cause any societal impacts.**\n\nIn terms of societal impacts, please allow us to clarify the following:\n\n**(Societal Impact)** As a powerful data protection technique following the concept of **Unlearnable Examples** (UEs), we strongly disagree with the reviewer that our work *could not cause any societal impacts*. The idea of UEs has been figured by **[MIT Technology Review](https://www.technologyreview.com/2021/05/05/1024613/stop-ai-recognizing-your-face-selfies-machine-learning-facial-recognition-clearview/)**, which we believe is a clear sign of the positive societal impact of this line of work. \n\n**(Practical Application)** As the release of powerful segmentation models like SAM, we recognize the potential damage it could bring to private data as personal images can now be easily segmented, interpreted, and manipulated by such models. Our work aims to stop the training of SAM or at least make the training more costly. This urges us to build a universal UE generation method that allows everyone to protect their images against (the training of) segmentation models. We anticipate our released noise generator to be a useful tool for many potential users. As demonstrated in our paper, UnSeg can be applied to protect natural scene images, remote sensing images, and even medical images. UnSeg is very lightweight, occupying less than 400 MB of memory, and can generate an unlearnable version of an image in just 113 ms. These characteristics further highlight the practical value of UnSeg.\n\n**(Novelty)** To the best of our knowledge, our work is the first UE generation method for segmentation models, which we believe is a novel and important generalization of UEs to more complex and foundational fields in computer vision. Unlike all previous methods, our UnSeg generates unlearnable examples using visual prompts, which we believe is technically novel. Our extensive experiments on 7 mainstream image segmentation tasks proves the empirical novelty of our method. For the first time in the field, we could use a single universal UE generator to fight against a wide range of downstream tasks at this scale. \n\n**(Efficiency)** The high practical value of UnSeg is evident in its significant improvements in data efficiency, generation efficiency, and transferability compared to previous methods. UnSeg can be fine-tuned on a 44k dataset in just 10 hours, demonstrating high data efficiency. It can convert an image into its corresponding unlearnable version in just 113 ms, showcasing high generation efficiency. The trained noise generator can be directly applied to different tasks and target models without retraining, highlighting high transferability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpFy3Cz8IE", "reviewer": "Reviewer_Mhnt", "summary": "This paper aims to design a universal unlearnable example generator for image segmentation tasks. Specifically, this paper aims to address three important factors for unlearnable examples in image segmentation:1) data efficiency 2) generation efficiency 3)  transferable efficiency. \n\nTo design such a model, the author makes use of the pretrained SAM model to conduct a min-min optimization problem, which involves two models) a noise generation model based on the pretrained SAM model; 2) a surrogate model training from scratch for minimizing the training loss given the corrupted unlearnable image. \n\nTo this end, the author conducts comprehensive experiments for a diverse set of downstream tasks to verify the effectiveness of the proposed method.", "questions": "It appears that the UnSeg model is trained on the HQSeg44k dataset. Consequently, the unlearned dataset is also HQSeg44k, but with added unlearnable noise. The authors subsequently used the HQSeg44k dataset to fine-tune different models for downstream tasks, including semantic segmentation, instance segmentation, and panoptic segmentation.\n\nIf my previous explanation is correct, I am concerned about the fairness of this approach. Specifically, there may be a significant domain gap between the fine-tuning dataset and the evaluation dataset. Furthermore, this setup does not demonstrate that the proposed method can generate unlearnable noise for example beyond the training data used for the UnSeg model.\n\nBut from the caption of Figure 5, it seems that the UnSeg model is employed to generate unlearnable noise for new datasets used in training different downstream tasks, the authors should clarify this in lines 249 to 265.\n\nPlease clarify the following:\n\nOn which dataset is the inference of UnSeg conducted (i.e., what training dataset is used for different downstream tasks)?\nWhich dataset is used as the evaluation set for the trained models of different downstream tasks?", "limitations": "Limitations have been discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The problem set is of practical meaning, especially in the current situation where private and copyright-protected data are widely used to train large-scale deep learning models. Previous methods to generate unlearnable examples are mainly designed for image classification tasks, which are not directly transferable for image segmentation tasks. To solve this problem, the author proposes to leverage the segmentation foundation model for generating unlearnable examples. The proposed method is easy to optimize and transferable for different downstream tasks.", "weaknesses": "The key experiment setup is not clear (see questions). \n\nWhat are the benefits of using the SAM as the backbone for the unlearnable example generation model and the surrogate model, I  miss the ablation on this.  Besides, what is the benefit of using the SAM pretrained weight for the unlearnable example generation model? Is this crucial for the success of the proposed method?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and valuable feedback.\n\n**Q1: What are the benefits of using the SAM as the backbone for the unlearnable example generation model and the surrogate model.**\n\nOne straightforward benefit of using SAM as the backbone is that it renders our entire framework interactive. This means that 1) the noise can be generated interactively during the optimization process, and 2) the surrogate model can be optimized on a small-scale interactive segmentation dataset, *improving data efficiency*. \nFurthermore, the trained noise generator inherits SAM's promptable characteristic and high transferability. This allows our method to generate the unlearnable noise for any downstream image through a single forward pass, *improving generation efficiency*. Additionally, since SAM is an interactive segmentation model whose predictions do not contain any class information, our approach avoids overfitting to any specific category, thus learning a universal unlearnable representation.  \n\nAs suggested, we replaced the MAE-pretrained ViT surrogate model with an ImageNet-pretrained ResNet50 surrogate model, and we validated the effectiveness of the trained noise generator using Mask2Former on the ADE20K and Cityscapes datasets. We find that the performance of the noise generator significantly declined when using ResNet50 as the surrogate model. We believe that using the MAE-pretrained weights can prevent the noise generator from being biased towards specific categories, thereby enhancing its transferability.\n|Surrogate Model|Backbone|ADE20K Panoptic| | |Cityscapes Panoptic| | |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|||PQ|AP-Pan|mIoU-Pan|PQ|AP-Pan|mIoU-Pan|\n|Clean|ResNet50|39.7|26.5|46.1|62.1|37.3|77.5|\n| |Swin-Tiny|41.6|27.7|49.3|63.9|39.1|80.5|\n|MAE pretrained ViT|ResNet50|11.7|7.5|17.7|5.7|1.1|7.8|\n| |Swin-Tiny|4.1|3.4|10.6|7.2|1.7|12.6|\n|ImageNet Pretrained ResNet50|ResNet50|35.6|23.4|43.5|42.5|16.5|64.8|\n| |Swin-Tiny|28.4|19.7|39.7|34.2|57.2|11.6|\n\n**Q2: The influence of using the SAM pretrained weight for the unlearnable example generation model?**\n\nOur primary consideration in using SAM is to ensure that the noise generator possesses promptable attributes after optimization. Therefore, we initialized the generator with the pretrained SAM weights and kept these weights frozen during training. This allows efficient fine-tuning of the newly added parameters. Here, we further test the use of randomly initialized weights for the noise generator. We run the experiments on the Pascal VOC dataset using DeepLabV1. We report two types of results in the table below: *making all classes unlearnable* and *making only some classes unlearnable*. The results indicate that our framework is not sensitive to the initial weights of the noise generator and the noise generator works well with random initialization. We will include this result in the revision.\n|Initialization of Generator Weights|All class|Multi-class| | | | |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| | |Aeroplane |Bicycle |Bird |Boat |Bottle|\n|Clean|70.6|81|36|84|66|72|\n|Pretrained SAM initialization |6.2|30.5|16.4|58.6|19.1|40.4|\n|Random initialization |4.8|28.1|6.2|42.1|6.5|25.9|\n\n**Q3: On which dataset is the inference of UnSeg conducted (i.e., what training dataset is used for different downstream tasks)? Which dataset is used as the evaluation set for the trained models of different downstream tasks?**\n\nThe outcome of our method is a single noise generator, which generates unlearnable noise for any given image in any downstream dataset. On downstream tasks, we applied our noise generator to convert the images in the training dataset into unlearnable images and train a SOTA model for the task on the unlearnable training images. We test the model’s performance on the clean test set. The lower the performance the better data protection offered by our noise generator. Please note that this setup strictly follows existing works in the field [a,b,c,d,e,f]. \n\nWe will clarify this in lines 249 to 265, as suggested. The noise generator was trained on the HQSeg-44k dataset.\n\nHere is a more detailed response to your question:\n\n**For semantic, instance and panoptic segmentation**: we use the trained noise generator to convert the Pascal VOC, ADE20K, Cityscapes and COCO training sets into their corresponding unlearnable training sets. The task-specific models are then trained on the unlearnable training sets and evaluated on the original validation sets. \n\n**For Interactive segmentation**: we use the trained noise generator to convert the HQSeg-44K into its unlearnable version. The SAM-HQ model is then trained on the unlearnable HQSeg-44K dataset and evaluated on the DIS, COIFT, HRSOD and ThinObject datasets.\n\n**For remote sensing instance segmentation**: we use the trained noise generator to convert the WHU, NWPU, and SSDD training sets into unlearnable training sets. The Rsprompter model is then trained on the unlearnable training sets and evaluated on the WHU, NWPU, and SSDD validation sets, respectively.\n\n**For medical image segmentation**: We randomly select 80% of the data from the Lung segmentation and Kvasir-seg datasets as the training sets, while using their remaining 20% as the validation sets. We then use the trained noise generator to convert the training sets into their corresponding unlearnable versions. The UNet++ model is then trained on the unlearnable training sets and evaluated on the clean validation sets.\n\n[a] Unlearnable examples: Making personal data unexploitable, Huang et.al., ICLR 2021\n\n[b] Adversarial examples make strong poisons, Fowl et.al., NeurIPS 2021\n\n[c] Unlearnable clusters: Towards label-agnostic unlearnable examples, Zhang et.al., CVPR 2023\n\n[d] Availability attacks create shortcuts, Yu et.al., SIGKDD 2022\n\n[e] Autoregressive perturbations for data poisoning, Pedro et.al., NeurIPS 2022\n\n[f] Cuda: Convolution-based unlearnable datasets, Sadasivan et.al., CVPR 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful and productive feedback. 

We are grateful to read that the reviewers agreed that the motivation and the proposed method in this work is impressive and quite interesting (**2xcM, PRaH**), has been extensively evaluated and achieves excellent protection results (**2xcM, PRaH, zWqb, Mhnt**), and that the work is of high practical significance thanks to its high efficiency and transferability (**Mhnt, 2xcM, PRaH**).

We have uploaded a PDF containing the comparison of UnSeg with two more SOTA methods to address the concerns of Reviewer **2xcM** and **PRaH** regarding baselines. The PDF also includes pixel value distributions of the clean and the unlearnable images to address the Reviewer **PRaH**'s concerns regarding the analysis of statistical significance.

We have individually responded to the questions and concerns raised by each reviewer point-by-point in the discussion boxes, and we have outlined our improvement plans and experiments in detail. If any questions remain unanswered or our responses are unclear, we would appreciate the chance for further engagement with reviewers.

Once again, we sincerely thank all reviewers for the valuable feedback, which would further improve the quality of this paper.

Meta-review: This paper originally received mixed reviews. After rebuttal, all reviews became positive. The authors addressed the major concerns regarding experiments. The AC decides to accept the paper. The authors should include the discussions and new results in the rebuttal into the final version.
Decision: Accept (poster)