MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Authors: Yi Yu
Song Xia
SIYUAN YANG
Chenqi Kong
Wenhan Yang
Shijian Lu
Yap-Peng Tan
Alex Kot
Venue: ICML 2025 poster (2025)
Keywords: unlearnable examples
unlearnable datasets
multi-task learning
poisoning attacks
Primary area: social_aspects->safety
TL;DR: We propose a framework for generating unlearnable examples on both multi-task learning and single-task learning models applied to multi-task learning datasets.

Abstract: Most existing unlearnable strategies focus on preventing unauthorized users from training single-task learning (STL) models with personal data. Nevertheless, the paradigm has recently shifted towards multi-task data and multi-task learning (MTL), targeting generalist and foundation models that can handle multiple tasks simultaneously. Despite their growing importance, MTL data and models have been largely neglected while pursuing unlearnable strategies. This paper presents MTL-UE, the first unified framework for generating unlearnable examples for multi-task data and MTL models. Instead of optimizing perturbations for each sample, we design a generator-based structure that introduces label priors and class-wise feature embeddings which leads to much better attacking performance. In addition, MTL-UE incorporates intra-task and inter-task embedding regularization to increase inter-class separation and suppress intra-class variance which enhances the attack robustness greatly. Furthermore, MTL-UE is versatile with good supports for dense prediction tasks in MTL. It is also plug-and-play allowing integrating existing surrogate-dependent unlearnable methods with little adaptation. Extensive experiments show that MTL-UE achieves superior attacking performance consistently across 4 MTL datasets, 3 base UE methods, 5 model backbones, and 5 MTL task-weighting strategies. Code is available at https://github.com/yuyi-sd/MTL-UE.

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Yi Yu 12
Song Xia
Siyuan Yang Chenqi Kong?
Wenhan Yang 14
Shijian Lu
Yap-Peng Tan
Alex C. Kot
Abstract
High Performance
Cleaa MTL Dacaset
Most existing unlearnable strategies focus pre- venting unauthorized users from training single- task learning (STL) models with personal data. Nevertheless, the paradigm has recently shifted towards multi-task data and multi-task learning (MTL), targeting generalist and foundation mod- els that can handle multiple tasks simultaneously. Despite their growing importance, MTL data and models have been largely neglected while pursu- ing unlearnable strategies. This paper presents MTL-UE, the first unified framework for generat- ing unlearnable examples for multi-task data and MTL models. Instead of optimizing perturbations for each sample, we design generator-based structure that introduces label priors and class- wise feature embeddings which leads to much bet- ter attacking performance. In addition, MTL-UE incorporates intra-task and inter-task embedding regularization to increase inter-class separation and suppress intra-class variance which enhances the attack robustness greatly. Furthermore, MT.. UE is versatile with good supports for dense pre- diction tasks in MTL. It is also plug-and-play allowing integrating existing surrogate-dependen unlearnable methods with little adaptation. Exten- sive experiments show that MTL-UE achieves su- perior attacking performance consistently across MTL datasets, 3 base UE methods, model backbones, and 5 MTL teigh strategies. Code is available at https:/ / / thub. com/ yuyi- Muyiisd/MMr wd/MML-UMUS
Arched Byebrows
Shadow
Train
MTL Modeb
Test
Task I: 93.3%
Young
Task 60c 86.9%
MTL-UE
Performance
Protected MTL Datasel
Clean Texd Daraa
O'clock Shadow
Train
MTL Medels
Test
E: 50.4%
Arcial Bydrooms
Task
Young
48 24.4%
Figure I. Illustration of MTL-UE to prevent unauthorized training of MTL models datasets like CelebA (Liu al., 2015), having 40 binary attribute classifications. MTL-UE adds invisible, sample- specific perturbations transform clean dataset into protected one, leading to poor test performance of trained MTL models.
1. Introduction
Multi-task learning (MTL) (Caruana, 1993; Guo et al., 2020) is branch of machine learning that tackles multiple tasks simultancously. making it more practical approach than single-task learning (STL). For instance, autonomous ve- hicles (Achituve et al., 2024) need to detect objects, track vehicles, monitor lanes, and estimate free space in real time.
MTL trains a single model handle multiple tasks, reduc- ing the need for separate models. By leveraging shared data across tasks, MTL lowers computational costs and im- proves generalization (Baxter, 2000), making it essential in fields like vision (Liu al., 2019; Misra et al., 2016), NLP (Chen et al., 2021; Fontana et al.. 2024), autonomous driving (Chowdhuri et al., 2019; Chen et al., 2018), and recommendation systems (Hadash et al., 2018).
'Corresponding author RRipid-Rich Object Search Lab, Inter- diseiplinary Graduate Programme, Nanyang Technological Uni- versity, Singapore "School Electrical and Electronic Engineer- ing. Nanyang Technological University. Singapore 'College of Computing and Data Science, Nanyang Technological Univer- sity, Singapore 'PengCheng Laboratory, Shenzhen, China. Corre- spondence to: Yi Yu wwy.0001@e..tt.cdu..oo Wenhan Yang <yangwh@ srnewh@@cl.ac.cc.. pcn..c.en>.
Deep neural networks have achieved impressive success across machine learning tasks (Hu al., 2024; Yang et al., 2024; Jin al., 2025a;b; Zhang et al., 2025), but also raised growing Al security concerns (Yu et al., 2022b; 2023; 2024b; 2025; Liu et al., 2025a;b). As large-scale models become more prevalent, massive data is likely to be scraped from the web and incorporated into training datasets, naturally raising concerns about the unauthorized use of personal informa-
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
tion for training DNNs (Burt, 2020; Vincent, 2019). This has led to efforts to develop defenses that prevent DNNs from exploiting private data. In STL, privacy protection methods (Feng et al., 2019; Sun et al., 2024) have been widely explored. These methods apply carefully crafted perturbations to images to compromise the generalization of models, commonly referred to as unlearnable examples (UE) (Huang et al., 2021; Yu al., 2024a;c), and are also known perturbative availability (Liu et al., 2023) or in- discriminate poisoning attacks (He et al., 2023). Models trained on these UE often capture spurious features, which are patterns added to the data that are irrelevant to the ac- tual task. While UE have been extensively studied for STL, their use in complex MTL scenarios as well as the related datasets, e.g. NYUv2 (Nathan Silberman & Fergus, 2012) with semantic segmentation, depth estimation, and normal estimation tasks, remains a challenge. These include man- aging the heightened complexity of perturbations designed to introduce spurious features across a greater number of tasks simultaneously, and dealing with more complex tasks that extend beyond classification.
To the best of our knowledge, we propose MTL.UE, the first plug-and-play framework for generating UE on MTL datasets, effective against both MTL and STL models, and compatible with any
UE methods
MTL-UE comprises an encoder-decoder network paired with learnable class-wise feature embeddings, which lower the intra-class variance of spurious features for each task. The addition of intra-task and inter-task embedding regular- ization further enhances performance.
MTL.-UE can extend beyond MTI. classifications to multi- ple dense prediction tasks by using task- spec-spec embedding modules to map task labels to embeddings.
Experiments on 4 MTI datasets, 3 base UE methods, backbones, and 5 MTL task-weighting strategies show con- sistent improvements of MTL-UE in attacking performance. Moreover, MTL-UE supports partial protection, making some tasks unlearnable while keeping others learnable.
2. Related Work
In this work, we propose MTL-UE, framework for gen- erating effective UE for MTL, with the goal of degrading the performance of all tasks in both MTL and STL mod- els, as shown in Fig. 1. We first conduct a straightforward empirical benchmark analysis of existing methods to re- veal our motivations. We implement several bascline UE methods, including surrogate-free methods using predefined shorteut patterns as class-wise perturbations, and surrogate- dependent methods relying on surrogate models to optimize sample-wise perturbations. Our initial findings suggest that surrogate-dependent methods underperform for both MTL and STL models, due to poor control over intra-class vari- ance caused by independent optimization for each sample. Patch-based AR (Sandoval-Segure et al., 2022), as shown in Sec. 4.2, using class-wise perturbations in saek-specific patches. performs better with lower intra-class variance but loses effectiveness as tasks increase, likely due to smaller patch sizes and limited representation capacity.
Data Poisoning. Data poisoning attacks (Barreno et al., 2010; Goldblum et al., 2022) manipulate training data to disrupt the test-time performance of models, and are cat- egorized into integrity attacks and availability attacks. In- tegrity attacks, such as backdoor attacks (Gu et al., 2017; Schwarzschild et al., 2021), trigger malicious behavior with specific inputs, while availability attacks degrade model per- formance on test sets (Biggio et al., 2012; Xiao et al., 2015). Typically, they inject poisoned data into the clean training set, where small portion of the samples, with unrestricted changes, are added (Koh & Liang, 2017: Zhao & Lao, 2022;
Lu et al., 2023). Though malicious, these samples are often detectable and have a limited overall impact.
Unlcarnable Examples (UE). UE (Huang et al., 2021; Zhang et al., 2023; Zhu et al., 2024a;b; Liu et al., 2024a;b; Chen al., 2024; Qin et al., 2023; 2024; Meng et al., 2024; Lin et al., 2024; Wang et al., 2024; 2025) is emerging approach, where subtle modifications, such as bounded per- turbations ||0]| S S are applied across the entire training dataset without altering the correct labels. This method shows potential for data protection. resulting in models performing close to random guessing on clean test data, EM (Huang et al., 2021) applies error-minimizing noise, while NTGA (Yuan & Wu, 2021) generates noise via neural tangent kernels. TAP (Fowl et al., 2021) uses targeted ad- versarial examples as UE, and REM (Fu et al., 2022) targets adversarial training (AT) (Madry et al., 201 8). LSP (Yu et al., 2022a) and AR (Sandoval-Segura et al., 2022) are surrogate- free UE. OPS (Wu al., 2023) uses one-pixel shortcuts improve robustness against AT and strong aug- mentations.
Building on these insights, we propose a plug-and-play framework to address these challenges by generating UE through class-wise feature embedding injections. By incor- porating task label priors via embeddings, we narrow the perturbation searching space from || S & to the de- coder's output space, resulting in lower intra-class variance. This approach effectively combines spurious features from multiple tasks into a unified perturbation. In addition to the generator's 's structural design, we introduce intra-task and inter-task embedding regularization (Intra-ER& & Inter-ER) to improve inter-class distance and minimize feature space redundancy, further improving the attack's effectiveness. In summary, our contributions are outlined below:
2

---

MTL-UE:: Learning to Learn Nothing for Multi-Task Learning
Multi-task Learning (MTL). MTL focuses on learning multiple tasks in a joint manner, typically using a shared encoder with task-specific heads for each task (Ruder, 2017; Zhang & Yang. 2021; Sener& & Koltun, 2018; Standley et al., 2020; Fifty et al., 2021; Navon et al., 2022). A major focus in MTL research is the optimization process. The usual method uses linear scalarization (LS)) along with a grid or random search find the best weight vectors (Lin et al., 2019). To address task balancing and conftict resolu- tion, strategies fall into loss-based and gradient-based (Dai et al., 2023). Loss-based methods assign task weights based on factors like task difficulty (Guo et al., 2018), random weights (Lin et al., 2022), geometric mean of losses (Yun & Cho, 2023), or uncertainty (Kendall et al., 2018). Gradient- based approaches adjust gradients directly, e.g., PCGrad (Yu et al., 2020) projects gradients to prevent conflicts, Aligned- MTL (Senushkin et al., 2023) aligns gradient components, and FairGrad (Ban & Ji, 2024) adopts utility maxinization.
PGD (Madry et al., 2018), and the outer one optimizes the parameters using optimizers such SGD. In contrast, AP constructs 6; to maximize the loss of the pretrained F on clean dataset T, i.e. generating adversarial examples:
Σ (R.DU)TT
(ar: + 6,;0"),, (.)),, s.t. 16.1, S c. (2)
max 6.
Multi-task learning (MTL) enhances the performance of several related tasks by training them simultancously. The training samples are typically tuples consisting of a shared input for all tasks and the labels for K tasks, i.e., Tum. [(a,, [u."==))]== where N is the number of train- ing samples. Our focus is on the scenario where the input are consistent across tasks. When the input varies for each task, it is often referred to as multi-domain learning (Royer et al., 2023). In such cases, UE for each domain dataset can usually be constructed independently. Common architec- tures for MTL (Achituve et al., 2024) have a shared encoder fr) and task-specific linear heads g (yt). A MTL problem involves set of K tasks with a loss vector:
3. Preliminaries
UE (Huang et al., 2021; Fowl et al., 2021; Wang et al., 2024) leverage clean-label data poisoning to trick DNNs into learning minimal useful knowledge from the data, thereby achieving the objective of data protection. Let T and D denote the clean training and test datasets, respectively. A model F(::0) trained on ' typically performs well on 'D.
min 10y.10 (CI(0), .M (x((0,
(3)
where (0)) is the loss of the k-th task. An MTL algorithm seeks to optimize all tasks simultaneously by leveraging the shared structure and information across them.
UE aims to transform T into an unlearnable dataset P, causing (::0) trained on P to perform poorly on D.
4. Methodology
In a C-class image classification, T == ((r,, wi) con- tains N samples, where æ; E Rd are inputs, and yi E {I...., C} are labels. P is crafted by adding perturbations ò; to each H,, such that P {(z; + 6;, y1) 1- Perturba- tions 5 E S are constrained to maintain visual imperceptibil- ity, where S denotes the feasible region, e.g., |10]|c< S 255 The attacker's success is measured by the accuracy of F trained on P when evaluated on D. UE methods can be classified into two categories based on whether surrogate model is required for optimizing the perturbations.
4.1. Problem Formulation
Previous UE (Feng et al., 2019; Huang et al., 2021) mainly target STL with K 1 task. However, practical scenarios often involve multi-task datasets. This work aims to develop UE for these datasets, ensuring better data protection against both MTL and STL models, with the following goals":
Effectiveness against MTL models: For a MTL model Fu = (f. to ) trained on the poisoned multi-task dataset PaL {(x, + 6., {ut K K the objective is to maximize the loss clean test data for all tasks, Le., maxi- mizing Cr, s²: 0,, 0,,).
Surrogate- methods (Yu et al., 2022a) do not rely on a surrogate model, and instead utilize predefined shortcut pat- terns as class-wiss perturbations. 6, added to each sample a; depend solely on its label yi. i.e., 5, = ((y)).
Effectiveness against STL models: For any k-th task and STL model Fst.. = {f trained on the poisoned dataset P = ((x+ 8,, y the objective is to maximize the loss Le v:: 0x)).
Surrogate-dependentt methods optimize sample-wise per- turbations& 6, for each data (æ,, 31) using surrogate models. Most follow variations of error-minimizing (EM) (Huang et al., 2021) or error- mario-maxiiizing (AP) (Fowl et al., 2021). EM constructs 6, by solving the bi-level optimizations:
i (the k-th attribute of Psnc),
4.2. Baseline methods for UE against MTL
mii
min C(F' + 5.; ()),, s.I. ||ō.||p S,e, (1)
This section presents several MTL-specific baseline UE methods, starting with aurrogate-dependen methods that typically use the MTL model itself as the surrogate model.
6.
where F' is the surrogate model. Typically, the inner min- imization employs the first-order optimization approach
We will omit MMTI." in TAT. and PAIIL in later sections.
3

---

MTL-UEE: Learning to Learn Nothing for Multi-Task Learning
Multi-task Learning
Single task Learning
Cican Deta
Averoge Accuracy
TAP
SEP
LSP (Patc-baased)
(Parch-sased)
50-
LSP (Averaging based)
AR (Averaging-basoe
10
15
Number
Tasks
Number of Tasks
Figure Performance of UE (Accuracy 1) Vs. the number of tasks on the CelebA (Liu et al., 2015) for both MTL and STL models.
Table 1. Intra-class std of the features for various UE methods. (P)
As discussed in (Yu et al., 2024a), spurious features with lower intra-class variance and greater inter-class distance are more effective for attacks. We denote the encoder's features in MTL models as
and (A) are the patch and averaging based. All 40 tasks are used.
UEs +
|[Ceaan EM TAP SEP LSP (P) AR (P) LSP (A) AR (A)
Average 55.357 2.191 3.656 6,719 1.828
1.733
5,264
5.180
[2, -p] R For each 2d, compute the average relative intra-class std (standard deviation) across all classes and tasks as =ad]zals"]]
Maximum
91.97 82.13 103.12 96.24 34.84
20.59
93,50
103.41
Next, we consider surrogate-fre methods and class-wise perturbations. Treating each combination {= as a dis- tinct class, where y E (I,...; G. yields a total of IE Cs bassswwise perturbations, which grows exponentially with K. This reduces the average number of data points per per- turbation to C relationships between perturbations and target combinations. To address this, we propose generating separate sets of per- turbations for each task, ((5), ) " and then combining them into a final perturbation. We explore two strategies for this combination: (1) Averaging-baset using the mean Me task-specific non-overlapping patches of æ.
B[salv*] The average and maximum val- ues across the dimensions are in Tab. 1. Note that features are from the MTL models using the poisoned dataset.
En Cp Σx E,
Our results show that patch-based AR has the lowest intra- class variance, with another patch-based approach that ranks second. This may be due to class-wise perturbations being applied to distinct, task-specific patches, thereby minimizing intra-class variance across locations. These lower intra-class variances likely contribute to the effectiveness. However, as k increases. the reduction in patch size constrains the pertur- bations' ability effective shortcuts, ultimately leading to performance decline. In contrast, averaging- beraging-bbasd AR and LSP have higher intra-class variance and perform worse than patch-based methods, as aggregating perturbations across tasks can introduce conflicts, leading to suboptimal results.
Ca complicating the creation of spurious
of (6t, /K as 8; and (2) Patch- adding (5t, M to
We assess UE effectiveness across varying task numbers on CelebA (Liu et al., 2015), a facial dataset with 40 binary classifications. We evaluate UE methods: EM (Huang al., 2021), TAP (Fowl al., 2021), and SEP (Chen al., 2023) surrogate-deeendent using surrogate MTL models with uniform weighteig,,,ng, while LSP (Yu et al., 2022a) and AR (Sandoval-Sege) Segura et al., 2022) are surrogate- free.
Surrogate-dependent methods like EM, TAP, and SEP op- timize perturbations individually for each sample, which limits control over intra-class variance. This limitation can reduce performance, particularly as k increases, since per- turbations must serve as shortcuts across multiple tasks, complicating the optimization. As shown in Tab. I and Fig. 2, most UE methods have higher intra-class variance and lower performance compared to patch-based AR and LSP, although EM achieves the second-best attack perfor- mance among MTL models and the lowest intra-class vari- ance among surrogate dependded methods. Overall, we that LSP (A), AR (A), TAP, and SEP achieve intra-class vari- ance comparable to that of clean data, resulting in limited effectiveness in attacking both MTL and STL models.
Effectiveness of UE Vs. the number of tasks. To examine the impact of task quantity on the effectiveness of baseline UE, we select the first & tasks, generate the corresponding UE, and train MTL and STL models. The experimental results as shown in Fig. 2 highlight several key findings:
1. STL models are more robust to UE than MTL models.
2. As increases, the performance of most UE initially increases on MTL and STL models, then declines, with near total failure on STL when all tasks are included.
4.3. MTL-UE: a plug-and-play UE method for MTL
3. Patch-based AR excels in both MTL and STL models.
The first observation arises from MTL models sharing rep- resentations across tasks, allowing them to more effectively capture similar shorteut patterns in UE and enhancing their focus on spurious features over benign ones. The following explanations address the remaining two points.
What is missing in existing UE methods? Based on our ob- servations in Sec. 4.2, we identify: (1) Surrogate-depender UE perform poorly due to uncontrolled intra-class variance caused by individual sample optimization, and the pixel- level search space which is hard to control; (2) Surrogate-
4

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Encoder E(:4E)
Decoder D ØD)
Clip
x
z
yk- yk [
yl
0
y²
K-1 e
K ex
x x+ 8
eyz
eyz
y
Intra-task Embedding Regularization
Inter-task Embedding Regularization
Figure 3. Visual depiction of MTL-UE, concatenating the task-specific class- embeddings with latents generate UE for MTL data.
tent representation z. Based on its labels {y" Jy"] the corre- sponding learnable class-wise feature embeddings {ef,) ) are selected. The embeddings and concatenated, and fed into decoder D(=)d)) to generate the final perturba- tions for æ. To regulate the amplitude of the perturbations, a clip operation is applied following the decoder. By shift- ing from direct perturbation searching to spurious features and integration network learning, MTL-UE can seamlessly integrate with any surrogate-dependent method.
Algorithm 1 Optimization of the UE Generator in MTL-UE
k=
Input:
Surrogate model Fmm (ff ) encoder (() decoder 0D)) embeddings {fe T= 1==1' clean multi-task dataset T {(æe, { )= = Adam optimizer, weights \₃ & X2, Train-surrogate, iterations I, bound e, loss function the base UE method Ls
Ca
epochs R.
Output:
Optimized E(; dæ), ((; { (et) C k==1
Initialize surrogate model no training is needed in the Alg.
if not Train-surrogate then Initialize FiS with pretrained weights on T end if
Besides MTL-UE's structural design, introduce embed- ding regularizations to further improve attack performance:
for epoch +- 1 to A do Optimize perturbation generator
Intra-task ER (Intra-ER). This term minimizes the co- sine similarity between embeddings within each task, ensur- ing diversity among the embeddings:
for image batch (æ. y = [v 1=) E T
==F((x; ) 5 Clip(((((p)))) ety ......., e ΦD), e),
C (Pmm 5, y) d1 Lintra + da Linter Minimize 2 with Adam to update dE, $D. and e end for
Lintea ΣK CC
cos(e ei).
(4)
k=
if Train-surrogate then
Optimize 1mmm if training is needed for epoch to 7 do
Inter-task ER (Inter-ER). This term promotes geometric independence between embeddings across different tasks:
Sample image hatch (a,, y =
8 = CCiip(D((E(( ox), e ......., ,xxl:,,))))))) -c
Linter EisAh C&OI LEk+1 lcos(e mn (.))) (5)
Train Fim. with Adam on (x + 8.
end for
end if
For Intra-ER, as discussed in (Yu al., 2024a), greater inter-class distance of spurious features enhances attack per- formance. This distance can be expressed as l|e - ||et 11 + ||e - 2| |le. ll2 cossstet, et.). To ensure the model effectively learns the introduced features, it is impor- tant to enlarge the inter-class distance for each task. How- ever, simply increasing the norm of et, doesn't 't suffice, as the decoder D(:; ()) can rescale the weights. Thus, mini- mizing cosine similarity between embeddings is better.
end for
free UEs face challenges primarily due to the imperfect fusion of class-wise perturbations from different tasks.
This raises the question: Can we reduce the pixel-level searching space to learn task-specific class-wise spurious features and use an integration network merge them into unified perturbation?
For Inter-ER, geometric independence offers several ben- efits: (1) Minimized Redundancy: It helps minimize re- dundancy in the features, and the decoder can exploit the unique information carried by each feature. (2) Reduced Coupling: helps reduce coupling, and the decoder can fo- cus on each feature independently, leading to more accurate
To address these, we propose MTL.UE, which generates UEs by injecting class-wise features to manage intra-class variance and combining spurious features from multiple tasks into a unified perturbation. The framework is shown in Fig. For any input æ, the encoder ((:;5)) maps it to a la-
5

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Multi-task Learnng
Encoder (E)
Decoder ( Φb)
Clip
Averoge Accurce (N6)
% 8 8
x
Embedder (()0g)
Embedder (x()
Embedder ((:(b))))))
Ceen
210
TAP
SEP
LSP
Single-task task Learning
Curs
Average Accuracy (9)
Cun-TAP
Curs SEP
y'
y2
y+3
Figure 4. MTL-UB applicd to dense prediction tasks, e.g., NYUv2.
Tesks
Figure 5. Performance Vs. the number of tasks on the CelebA
perturbations. (3) Improved Interpretability: Geometric in- dependence facilitates understanding of the role each feature plays in generating spurious features.
70x70. ChestX rra 14 has 112,000 chest X-ray images with 14 binary classes for thoracic diseases. The official splits used, with images resized to 256:25.. UTKKaace has 23,705 images annotated for age, gender, and race. We follow Karkkainen & Joo (2021) to treat age as a nine-class task, gender as binary, and race as five-class, splitting 80% for training and 20% for testing, with images resized to 140440. NYUv2 an indoor scene dataset, with 795 training and 654 testing images for tasks like 13-class semantic segmen- tation, depth estimation, and surface normal prediction. We follow Liu et al. (2019) to resize images to 288 x 384.
Optimization of MTL-UE. The generator optimization in Alg. can be applied to surrogate-dependent UE methods. After optimization, we transform clean T to unlearnable P.
Advantages. MTL-UE offers key advantages over baselines. First, by incorporating task label priors through embeddings, s we reduce the perturbation search space from ||0|| s 255 to the decoder's output space, leading to lower intra-class variance. Second, since the generator is trained across the entire dataset, it captures global features more effectively, supporting the effective learning of spurious features. Ad- ditionally, both Intra-ER and Inter-ER are introduced to further improve attack performance.
Models. We use ResNet-18 (He et al., 2016) as the shared encoder for both surrogate and target models. To eval- uate transferability. we also include various backbones like ResNet-50, VGG16 (Simonyan & Zisserman, 2015), DenseNet-121 (Huang et al., 2017), and VIT-B (Dosovitskiy et al., 2021) for the target models. For MTL models, play the HPS (Caruana, 1993) architecture, which includes shared feature extractor and task-specific heads.
4.4. Application to Dense Prediction Tasks
In this section, we demonstrate the effective application of MTL-UE to multi-task datasets for dense prediction tasks, using the NYUv2 dataset (Nathan Silberman & Fergus, 2012) as an example. As shown in Fig. 4, instead of using class-wise embeddings for each task, we apply the embed- ding module 8x (:: (be) map the corresponding dense la- bel x to spurious features. Since MTL-UE on such dataset requires manipulating dense prediction results, where redun- dancy in spurious features is minimal, we do not employ embedding regularizations here.
weightig for MTL. We adopt LS with a uniform weight, To assess transferability, we include Random Loss Weighting (RLW) (Lin et al., 2022), Uncertainty Weighting (UW) (Kendall al., 2018), Aligned-MTTI. (Senushkin et al., 2023), and FairGrad (Ban & Ji, 2024).
Unlearnable examples. We include baseline methods EM (Huang et al., 2021), TAP (Fowl et al., 2021), and SEP (Chen et al., 2023) with an fx = 258 bound, and LSP (Yu et al., 2022a) and AR (Sandoval-Segura et al., 2022) with an
5. Experiments
s
5.1. Experimental Setup
bound, following default settings. Adaptations for MTL detailed in Sec. 4.2. MTL-UE is compatible with EM, TAP, and SEP, resulting in MTL- UE-EM, MTL-UE-TAP, and MTL-UE-SEP, all using the same la0 8 bounds. More details of the baselines are in Sec. A.1.
Datasets. We choose 4 popular multi-task vision datasets:
CelebA (Liu et al., 2015). ChestX-ray 14 (Wang et al., 2017), UTKFace (Zhang et al., 2017). and NYUv2 (Nathan Silber- man & Fergus, 2012). CelebA has 202,599 face images with 40 binary attribute classifications. We use 162,770 images for training and 19,962 for testing, resizing all to
255
Model and MTL-UE training. Details for training the
6

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Table 2. Results on classification datasets: We report average accuracy (%) for CelehA and FTKe, and AUC-ROC for ChestX-ray 14. For UTKFace, accuracy for age, race, and gender is also shown. All models use ResNet- 18, and MTL models use LS for task weighting.
Darasct--
CelebA (Liu et al., 2015)
ChestX-myl4 (Wang et al., 2017)
UTKFace (Zhang ct al., 2017)
Model +
MIL Avg.
STL
MIL
SIL
MIL
STL
Tasks -
Avg.1
Avg.u
Avg-L
Agel.
Race,
Gender) Avg.l
Age]
Racel, Gender.) Avg.L
Clean
91.11
90.35
0.7577
0.6493
00.32 84.07
92.51
78.97
60.46
84.45
91.86
78.92
LSP (Patch) (Yu et al., 2022a)
78.12
84,80
0.6467
0.6543
18.40
17.51
49.83
28.58
19.62
15.04
62.62
32.43
AR (Patch) (Sandoval-Seguu et al., 2022)
73.12
84.41
0.5306
0.61188
9.70
19.66
52.87
27.41
19.16
13.92
47.17
26.75
(Average) (Yu
2022a)
91.07
90.35
0.7218
0.6452
12.59
19.72
52.17
28.16
23.21
44.20
70.34
45.91
AR (Average) (Sandoval-Segun Seguraa et al., 2022)
91.14
90.37
0.7259
0.6400
13.92 41.03
55.42
36.79
14.26
42.89
52.85
36.67
EM (Hanng el al., 2021)
75.66
89.91
0.4976
0.5548
19.24
17.43
58.57
31.74
25.74
37.36
89.54
50.88
TAP (Fowl et al, 2021)
85.24
87.00
0.5478
0.6005
25.86
39.28
52.74
39.29
32.15 59.35
86.58
59.36
SEP (Chen ct al., 2023)
84.25
89.91
0,5462
0,5926
25.42
44.03
52.64
40.70
33.73
60.59
88.04
60.78
MIL-UE-EM
74.38
74.26
0.4813
0.5302
10.00
12.78
54.73
25 84
9.66
12.47
56.81
26.32
UIL--UE-TAR
59.51
68.65
0.5341
0.6091
9.49
18.23
31.81
19.84
15.97
19:03
41.75
25.59
MIL-UE-SEP
58.73
76.39
0.4929
0.6068
7.28
16.20
40.08
21.19
7.26
21.84
55.61
28.24
Results on the ChestX-ray14. As shown in Tab. 2, MTL- UE-EM excels, and MTL-UE-TAP and MTL-UE-SEP im- prove upon TAP and SEP. As TAP and SEP use adversarial examples, the low performance of clean surrogate models lowers the results, explaining why MTL-UE-EM,
Table 3. Intra-class std of the features for competing UE methods. UFFs -+ JFM
TAP SEP L.SP (R)
Avg. 2.191 3.656 6.719 18828
1.733
L.715
2,130
2.387
Max.
82.13 101.111....2.. 1296.24
34.84 20.59
18.04
47,93
64.57
better.
Results the UTKFace. The results in Tab. 2 show that with 3 tasks, MTL-UE consistently outperforms the base- lines on both MTI. and STL models, despite the increased class numbers. Notably, on STL models, MTL-UE reduces average accuracies by over 30% compared to the base UEs. Sec. B.1 presents UE performance vs. task numbers.
MTL/STL models and MTL-UE are given in Sec. A.2.
Metrics. We use accuracy for CelebA and UTKFace, and AUC-ROC for ChestX-ray 14 (Lin et al., 2022: Achituve et al., 2024). For NYUv2, we use mean Intersection over Union (mloU) and pixel accuracy (PAce) for segmentation, absolute errors (AErr) and relative errors (RErr) for depth estimation, and mean absolute error (Mean) and median ab- solute error (MED) for normal estimation (Liu et al., 2019).
Results on the NYUv2. We show the results on the NYUv2 involving dense classification and regression tasks. From Tab. 4, MTL.UUE consistently enhances the performance of all baseline UE across all tasks. Each variant-MTL-UA -TTL-UE. EM, MTL-UE-TAP, and MTL-UE-SEP--xhhilE different trade-offs across the three tasks: MTL-UE-EN, performs best in segmentation, MTL-UE-TAP excels in depth esti- mation, and MTL-UE-SED outperforms others in surface normal estimation. As AR and LSP cannot be applied to dense tasks, they are not included for comparison.
5.2. Experimental Results
We first evaluate MTL-UE on CelebA and ChestX-ray 14 for binary classifications. We then demonstrate its effectiveness on UTKFaae with more than two categories, and finally, show its generalization to the NYUv2, which includes dense predictions for both classification and regression tasks.
Visual results. We offer visual results in Fig. 6. We can see that compared to baselines, our perturbations are more structured. For instance, MTL-UE-TAY displays clearer semantic information than TAP, with more distinct contours and sketch lines. The more structured perturbations tend to have lower intra-class variance, leading to the improved performance. Among the baselines, AR adopts no-nm,m and appears sparser, though the overall perturbation magnitudes are similar across all methods. More results are in Sec. B.5.
Results on the CelebA. As shown in Tab. 2, MTL-UE consistently improves surrogate- dependent UE methods like EM, TAP, and SEP. Notably, for STL models, MTL. UE-T achieves around 68% accuracy, much better than the base- line, while for MTL models, MTL-UE-TAL and MTL-UE- SEP achieve accuracies below 60%. While MTL-UE-EN, has minimal impact on MTL models, it significantly boosts performance on STL models. Fig.5 shows UE performance as a function of task number, highlighting the effectiveness of MTL-UE across varying task counts. MTL-UE shows consistent results for MTL models when the task number exceeds 10. For STI. models, it performs well with around 15 tasks, while performance under more tasks remains an area for future exploration. Tab. demonstrates that MTL- UE significantly reduces the intra-class standard deviation of the features, resulting in improved attack performance.
surpogete-eeendeett
5.3. Discussion
Smaller perturbations. We evaluate smaller bounds f
255 255 255 255 alongside fac= & Tab. 6 shows MTLL-UE are
2 4 6
effective, outperforming baselines even with larger bounds. MTL-UE-EM excels on STL models, while the other two perform better on MTL models under small bounds.
7

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Table Quantitative results the NYUv2 dataset. ResNet- 18 is used as the encoder, with L. for task weighting MTI models.
Model
MTI.
STL.
Task +
Segmentation mloUt
Depth
Normal Meant
Segmentation mloU
Depth
Normal
Metric
PAcel
AErrt
RErrt
PAccl
AErr
RErrt
Meant
MED1
Clean
53.05
75.01
0.3920
0.1665
23.72
17.29
53.16
75.30
39.80
16.50
22.53
16.00
EM
26.45
46.22
0.6109
0.2352
30.32
24.46
27.03
44.66
0.6280
0.2426
29.09
22.53
TAP
0.2561
33.79
14.51
26.32
0.6340
0.2389
23.25
SEP
11.41
23.49
0.7113
0.2884
37.11
31,52
9.76
24.16
0,7517
0.2774
33.01
26.14
MTL-UE-EM
2.37
16.04
0.9249
0.3013
38.04
33.65
1.65
15.64
0,9109
0.2978
33.98
27 49
MIL-UE-TAP
0.3161
39.69
34.61
17.14
30.01
1.1182
0.3607
41.56
35.08
MTL-UE-SEP
17.76
33.13
0.8260
0,2850
41.78
36.47
15.35
33.44
0,9570
0.31
40.24
33,08
CelebA
Clean
EM
TAP
SEP
1.SP (Putch)
A. htc
LSPP (Avg.)
AR (Avg.)
MTL-UU--M MTL UPE- MTL-
NYUv2
Clean
EM
TAP
SEP
MTL-UE-EM
MTL-UE-TAP
MTL-UE-SEP
Figure Visual results: Odd rows show perturbations (independently normalized [O,1], even rows show poisoned images.
SSNE Dimoe 2 2
SSEE Diiier Simerrsinn
Age
Gender
Gender
Gendler Race Age
(a) MTI. models trained on cleun data
(b) MTI. models rmiine on unleamable data
Figure 8. GradCAM of models trained on clcan/unlearnable data.
Dimension
Dimension
(c) Ous (W] Intra ER Inter: ER)
(b) Ouss (Wo Intra- ER & Inter: ER)
ation, but shows little impact, even reducing performance on MTL models. @ which relies solely on learnable class- wise feature embeddings, shows a significant performance improvement over @, emphasizing the role of embedding priors in reducing intra-class variance. Additionally, build- ing on @, incorporates latents from the input ar, and further enhances performance, likely due to improved optimization by leveraging additional information from Finally. @ excludes intra-task and @ excludes inter-task embedding regularization, both showing that embedding regularization enhances performance. We also provide an ablation study of hyperparameters and X2 See. B.2, showing that MTL-UE is not sensitive to them.
Figure T-SNE results learned class-wise embeddings.
Transferability. We assess UE transferability MTL and STL models with varying encoder backbones and MTL weighting strategies. Results CelebA shown in Tab. 5. Our methods, particularly MTL-UE-TAP and MTL-UE- SEP, generalize well across CNNs, while MTL-UE-EN performs better on ViT-B. This may be because SEP and TAP rely on adversarial examples, limiting transferability from CNNs VIT, as ResNet-18 18 was used as the surrogate backbone. Our methods also generalize well to MTL models with different weighting strategies, with MTL-UE-SE celling in several cases. Additional results NYUv2, along with MTI. architecture transfer, are provided in Sec. B.4.
Partial task protection. We explore the scenario, where only selected tasks are unlearnable, using MTL-UE-SEP. After optimizing the generator, we generate UE by using the mean of (ct) for learnable tasks instead of e Results in Tab. 8 show that for STL models, unprotected tasks match clean data accuracy, while protected tasks perform like those
Ablation study (design & A1 A2). h We do an ablation study of the MTL-UE design, with results in Tab. 7. Compared @ introduces an auto-encoder for perturbation gener-
8

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Table Results of Transferability: performance on CelchA when transferring to MTI. and STI. models with different backhones or MTL models with various weighting strategies. Note that RN denotes the ResNet, and DN denotes the DenseNet.
Transfer to MTL and STL models with various backbones
|Transerr to MTL models with various task weightingg
Model
MTL (LS the tack weighting)
STL
MTLiies 18 the hackhone
Baukbone/Weighningg
RN-18
RN-50
VGG-16
DN-121
VTT-B|
Avg.
RN- 18
RN-50
VGG-16
DN-121
VIT-B
Aug.
LS
UW
RLW Aliga FairGrad]
Avg.
Clean
91.11
91.20
91.33
91.37
89.39
90.88
90.35
90.13
90.40
89.61
87.25
89.55
91.11
90.82
91.00
91.08
90.75
90.95
LSP (Patch)
78.12
76.37
77.96
77.07
81.96
78.70
84.80
84,26
82.08
84.70
86,96
84.56
78.12
77.48
72.95
77.43
73,80
75.56
AR (Patch)
73.12
73:07
72.23
75.49
85.73
75.53
84.41
85,40
75.77
78.19
87.03
82.16
73.12
71.87
74.59
72.59
67.76
71.99
LSP (Averuge)
91.07
91.15
91.31
91.32
89.1
90.60
90.35
90.56
89.77
89.55
91.07
90.73
90.98
90.89
90.62
90.86
AR (Average)
91.14
91.20
64.78
67.09
89.35
S0.31
90.37
90.12
89.47
88.15
87.16
89.05
91.14
90,80
91.01
90.93
90.69
90.91
EM
75.66
74.26
76.20
83.76
76.83
75.87
86.93
85.03
75.6
75.30
75.32
74.54
75.23
TAP
85.24
85.44
87.34
87,65
88. 58
86.45
87.00
86.74
85. 38
84.57
86.83
86, 10
85.24
85.27
85.52
85.23
84.94
85.24
SEP
84.25
81.27
83.10
82.81
89.31
84.55
89.91
89.68
84,93
86.52
86,33
87.47
84.25
89.54
87.33
87.28
87.78
87.24
MTL-UB-EM
74.38
72.81
69.81
70.31
71.12
71.69
74.26
74.52
78.
76.68
79.03
76.25
74.38
70.84
72.71
71.78
73.25
72.59
MTL-UE
59.51
64.76
60.50
60.36
76.69
64.76
68.65
70.73
68.27
75.14
83,22
73.20
59.51
58.66
62.95
63.60
56.61
60.27
MTL-UE-SEP
58.73
60.06
64.61
63.71
80.14
65.85
76.39
79.87
70.17
75.25
84.78
77.69
58.73
53.54
58.24
60.54
53.60
56.53
Table 6. Results of MTL-UE with smaller bounds on CelebA.
Table 8. Results of partial task protection on on the UTKFaaee
Bound- -
8/255
foo
6/255
4/255
for 2/255
Model
||
MTI.
STI.
Model- +
MTL
STL
MTL
STL
MTL
STL
MTL
STL
Protected task $
I|
Age
Race
Gender
Ago
Race
Gender
MIL-UE-ERI EM
74.38
74.26
74.38
75.11
75.56
76.25
78.81
79.24
None
660.2
84.07
92.51
60:46
84.45
91.86
MTL TAP
59.51
68.65
63.69
72.58
63.91
80.01
69.61
85.19
Age. Race
12.68
20.53
88.52
10.25
19.64
90.95
MIL-UE-SEP
58.73
76.39
60.98
80.45
61.82
83.58
81.35
86.82
Age. Gender
10.72
78.82
43.90
10:70
82.68
46.41
Race, Gender
52.64
35.49
56.79
57.11
16.52
$2.97
All
7.28
16.20
40.08
7.26
21.84
55.61
Table 7. Ablation study of the MTL-UE design on CelebA.
Table 9. Results of partial data protection on the NYUv2,
Model
MIL
STL
Base UE- +
EM
TAP
SEP
EM
TAP
SEP
0%
20%
40%
60%
80%
100%
Ti
Paua
Taa
Pa.4
Tea
Ta.a
Pa.al
Ta.
Pas
Ta.a
To.a
Pi
0: the baseline UE
75.66
85.24
84.25
89.91
87.00
89.91
PAec
75.0
74.0
74.5
72.3
71.1
69.9
67.4
63.5
60.8
33.1
wio the feature embeddings
90.75
88.08
86.73
89.95
88.81
87.78
REm
0.167
0.167
0.169
0.173
0.180
0.183
0.199
0,194
0.2570.288
06: w/o the (i dee) and
74.47
62.37
68.63
78.82
71.06
78.45
Mcan
23.7
24,2
24.3
24.7
25.6
25.3
27.4
29.0
32.8
41.8
9: w/o the Intra-ER
77.87
63.07
63.41
79.82
69.52
79.25
(B: w/o the Inter-ER ER
75.81
64.95
63.69
78.42
76.05
75.40
MTL.UUE
74.38
59.51
5M.73
74.26
68.65
76.39
Feature visualization UTKFace with MTL-UE-TAP
Fig.7 shows t-SNE results of the learned class- wise embed- dings for each task, where Inter-ER improves task separa- tion and Intra-ER disperses embeddings within tasks. Fig. 8 shows GradCAM results for MTL models on their training samples, revealing that MTL-UE introduces spurious fea- tures, shifting focus to irrelevant areas like facial contours instead of key regions like eyes and noses.
trained to protect all tasks. In contrast, MTL models show degraded performance for all tasks, likely due to shared encoder learning both benign and spurious features.
Partial data protection. Following Huang et al. (2021), we convert a portion r of clean T into unlearnable Pr., leaving the rest as Ti- We experiment on the NYUv2 using MTL- UE-SEP. Table 9 shows results for models trained on Pr mixed with TI and only on Ti-.. The effecctiveness drops quickly when the data is not fully unlearnable, a limitation also noted in Huang et al. (2021). Models trained the mixture or only partial clean data show similar results, indi- cating that P, is ineffective and unlearnable during training.
6. Conclusion
This paper presents MTL-UE, the first framework for gen- erating UE on multi-task datasets for both MTL and STL models, featuring plug-and and-play design that seamlessly in- tegrates with existing aurrogate-dependers methods. Instead of optimizing perturbations for each sample, we utilize an encoder-decoder network with additional sets of class-wise embeddings. By incorporating task label priors through em- beddings, MTL-UE reduces the intra-class variance of spu- rious features for each task. Additionally, the intra-task and inter-task embedding regularization improve the inter-class separation of spurious features and minimize redundancy, further enhancing performance. MTL-UE is also versatile, supporting dense prediction tasks in MTL. Extensive exper- iments demonstrate the effectiveness of MTL-UE.
Computational cost& & Parameter count, Sec. A.3 shows MTL-UE's efficiency is close to EM and much lower than TAP and SEP, and LSP and AR are faster due predefined patterns. Sec. A.4 shows MTL-UE needs fewer parameters than EM, TAP, and SEP, but slightly more than AR and LSP.
Resistance to defenses. Sec. B.3 shows MTL-UE is much more robust to SOTA defenses than baseline UE, likely due to its clearer semantic patterns, distinct contours, and sketch lines, which better withstand SSS-indueed corruptions.

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Acknowledgements
Caruana, R. Multitask learning: A knowledge-based source of inductive bias. In Proc. Int'l Conf. Machine Learning, pp. 41-48. Morgan Kaufmann, 1993.
This work was carried out at the Rapid-Rich Object Search (ROSE) Lab, Nanyang Technological University (NTU), Singapore. This research is supported by the National Re- search Foundation, Singapore and Infocomm Media Devel- opment Authority under its Trust Tech Funding Initiative, the Basic and Frontier Research Project of PCL, the Major Key Project of PCL, and Guangdong Basic and Applied Basic Research Foundation under Grant 0024A1515010455 Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Sin- gapore and Infocomm Media Development Authority.
Chen, C., Zhang, J., Li, Y., and Han, One for all: universal generator for concept unlearnability via multi- modal alignment. In Proc. Int'l Conf. Machine Learning, 2024.
Chen, S., Zhang, Y., and Yang. Q. Multi-task learning in nat- ural language processing: An overview. ACM Computing Surveys, abs/2109.09138, 2021.
Chen, S., Yuan, G., Cheng, X., Gong, Y., Qin, M., Wang, Y., and Huang. X. Self-ensemble protection: Training check- points are good data protectors. In Proc. Int'l Conf. Learn- ing Representations, 2023.
Impact Statement
In summary, our paper introduces MTL-UE, a novel frame- work for generating unlearnable examples (UEs) that pro- tects multi-task learning (MTL.) data and tasks from unau- thorized exploitation. As MTL models become increasingly important for handling a wide range of tasks simultane- ously, safeguarding such models is critical to ensure privacy and prevent the misuse of sensitive data. Our approach ad- dresses the growing need to protect personal or proprictary data from unauthorized use while reducing the risks of data theft in critical sectors like healthcare, finance, and security. Furthermore, it fosters ethical AI development by encour- aging responsible data use, promoting trust between data providers and model developers, and contributing to broader conversations on data protection and privacy regulations.
Chen, Y., Zhao, D., Lv, L.., and Zhang, Q. Multi-task learn- ing for dangerous object detection in autonomous driving. Inf. Sci., 432:559-5711 171,,99--,,22 2018.
Chowdhuri, S., Pankaj. T., and Zipser, K. Multinet: Multi- modal multi-task learning for autonomous driving. In IEEE Winter Conference on Applications of Computer Vision, WACV 2019, Waikoloa Village, HI, USA, January 7-11, 2019, pp. 1496-1504, IEEE, 2019.
Dai, Y., Fei, N., and Lu, Z. Improvable gap balancing for multi-task learning. In Uncertainty in Artificial Intelli- gence, pp. 496-506. PMLR, 2023.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In Proc. Int'l Conf. Learning Representations, 2021. URL httph:ttp////...tm http:::://ppeeeviieeeee net / forum? wwwp:mmmmidiiiciieeeemee CNTTY.
References
Achituve, I., Diamant, I., Netzer, A., Chechik, G., and Fe- taya, E. Bayesian uncertainty for gradient aggregation in multi-task learning. Proc. Int'l Conf. Machine Learn- ing, 2024.
Feng. J., Cai, Q.-z,, and Zhou, Z.-H. Learning to con- fuse: generating training time adversarial data with auto- encoder. In Proc. Annual Conf. Neural Information Pro- cessing Systems, volume 32, 2019.
Ban, H. and Ji, K. Fair resource allocation in multi-task learning. In Proc. Int'l Conf. Machine Learning, 2024.
Barreno, M. Nelson, B., Joseph, A. D., and Tygar, J. D. The security of machine learning. Machine Learning, 81: 121-148, 2010.
Fifty, C., Amid, E., Zhao, Z., Yu, T., Anil, R., and Finn, C. Efficiently identifying task groupings for multi-task learning. In Proc. Annual Conf. Neural Information Pro- cessing Systems, volume 34, pp. 27503-27516, 2021.
Baxter, J. model of inductive bias learning. J. Artif. Intell. Res., 12:149-198. 2000.
Fontana, M., Spratling, M. W., and Shi, M. When multitask learning meets partial supervision: A computer vision review. Proc. IEEE, 12(6):516-543, 2024.
Biggio, B., Nelson, B., and Laskov, P. Poisoning attacks against support vector machines. In Proc. Int'l Conf. Ma- chine Learning, pp. 1467-1474, 2012.
Fowl, L., Goldblum, M., Chiang, P.-y.. Geiping, J., Czaja, W., and Goldstein, T. Adversarial examples make strong poisons. In Proc. Annual Conf. Neural Information Pro- cessing Systems, volume 34, pp. 50339-303511 2021.
Burt, C.
Facial biometrics training dataset leads to bipa lawsuits against amazon, alphabet and microsoft, jul 2020. URL http://reurl, cc/dV4rD8, 2020.
10

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Fu, S., He, F., Liu, Y., Shen, L.,, and Tao, D. Robust unlearn- able examples: Protecting data privacy against adversarial learning. In Proc. Int'l Conf. Learning Representations, 2022.
Jin, C., Peng, H., Zhang, Q. Tang, Y., Metaxas, D. N., and Che, T. Two heads are better than one: Test-time scaling of multi-agent collaborative reasoning. arXiv preprint WrXi::250.09972,, 2025b.
Goldblum, M., Tsipras, D., Xie, C., Chen, X., Schwarzschild, A., Song, D., Madry, A., Li, B., and Gold- stein, T. Dataset security for machine learning: Data poi- soning, backdoor attacks, and defenses. IEEE Trans. on Pultern Analysis and Machine Intelligence, 45(2):1563 1580, 2022.
Karkkainen, K. and Joo, J. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pp. 1548- 1558, 2021.
Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Gu, T., Dolan- Gavitt, B., and Garg, S. Badnets: Identify- ing vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 170xv::700.0000 067333 2017.
Guo, M., Haque, A., Huang, D.-A., Yeung, S., and Fei-Fei, L. Dynamic task prioritization for multitask learning. In Proc. IEEE European Conf. Computer Vision, pp. 270- 287, 2018.
Koh, P. W. and Liang, P. Understanding black-box predic- tions via influence functions. In Proc. Int'l Conf. Machine Learning, pp.
PMLR, 2017.
Lin, B., Ye, F., Zhang, Y., and Tsang, I. Reasonable effec- tiveness of random weighting: A litmus test for multi-task learning. Transactions on Machine Learning Research, 2022.
Guo, P., Lee, C.-Y., and Ulbricht, D. Learning to branch for multi-task learning. In Proc. Int'l Conf. Machine Learning, pp. 3854-3863. PMLR, 2020.
Lin, X., Zhen, H.-L., Li, Z., Zhang, Q.FF, and Kwong, S. Pareto multi-task learning. In Proc. Annual Conf. Neural Information Processing Systems, volume 32, 2019.
Hadash, G., Shalom, O. S., and Osadchy, R. Rank and rate: multi-task learning for recommender systems, In Pro- ceedings of the 12th ACM Conference on Recommender Systems, pp. 451-454) ACM, 2018
Lin, X., Yu, Y., Xia, S., Jiang, J., Wang. H., Yu, Z., Liu, Y., Fu, Y., Wang, S., Tang, W., et al. Safeguarding medical image segmentation datasets against unauthorized train- ing via and texture-awan perturbations. arXiv preprint arXiv:2403. 14250, 2024.
He, H., Zha, K., and Katabi, D. Indiscriminate poisoning at- tacks on unsupervised contrastive learning. In Proc. Int'l Conf. Learning Representations. OpenReview.net, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proc. IEEE Int'l Conf. Com- puter Vision and Pattern Recognition, pp. 770-778, 2016.
Liu, S., Johns. E., and Davison. A. J. End-to-end multi-task learning with attention. In Proc. IEEE Int'l Conf. Com- puter Vision and Pattern Recognition, pp. 1871-1880. 2019.
Hu, Z., Li, X., Tang, S., Liu, J., Hu, Y., and Duan, L.-Y. Lead: Exploring logit space evolution for model selec- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28664- 28673, 2024.
Liu, S., Wang, Y., and Gao, X.-S. Game-theoretic unlearn- able example generator. In Proc. AAAI Conf. on Artificial Intelligence, volume 38, pp. 21349-21358, 2024a.
Liu, W., Deng, Z., Niu, Z., Wang, J., Wang, H., Zeng. Z., and Li, R. Breaking free from MMI: A new frontier in rationalization by probing input utilization. In Proc. Int'l l Conf. Learning Representationss 2025a.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 4700 4708, 2017.
Liu, W., Niu, Z., Gao, L., Deng. Z., Wang, J., Wang, H., and Li, R. Adversarial cooperative rationalization: The risk of spurious correlations in even clean datasets. In Proc. Int'l Conf. Machine Learning. 2025b.
Huang, H., Ma, X., Erfani, S. M., Bailey, J., and Wang, Y. Unlearnable examples: Making personal data unex- ploitable. In Proc. Int'l Conf. Learning Representations, 2021.
Liu, X., Jia, X., Xun, Y., Liang, S., and Cao, X. Mul- timodal unlearnable examples: Protecting data against multimodal contrastive learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8024 8033, 2024b.
Jin, C., Li, Y., Zhao, M., Zhao, S., Wang, Z., He, X., Han, L.,, Che, T., and Metaxas, D. N. Lor-VP: Low-rank vi- sual prompting for efficient vision model adaptation. In Proc. Int'l Conf. Learning Representations, 2025a.

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
Liu, Z.,, Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proc. IEEE Int'l Conf. Computer Vision, pp. 3730-3738, 2015.
Information Processing Systems, volume 35, pp. 27374- 27386, 2022.
Schwarzschild. A., Goldblum, M., Gupta, A., Dickerson, J. P., and Goldstein, Just how toxic data poisoning? a unified benchmark for backdoor and data poisoning attacks. In Proc. Int'l Conf. Machine Learning, pp. 9389- 9398. PMLR, 2021.
Liu, Z., Zhao, Z., and Larson, M. Image shortcut squeezing; Countering perturbative availability poisons with com- pression. In Proc. Int'l Conf. Machine Learning, 2023.
Lu, Y.. Kamath, G., and Yu, Y. Exploring the limits of todel-targeted indiscriminate data poisoning attacks. In Proc. Int'l I Conf. Machine Learning, 2023.
Sener, O. and Koltun, V. Multi-task learning as multi- objective optimization. In Proc. Annual Conf. Neural Information Processing Systems, volume 31, 2018.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In Proc. Int'l Conf. Learning Repre- sentations, 2018.
Senushkin, D., Patakin, N., Kuznetsov, A., and Konushin, A. Independent component alignment for multi-task learning. Proc. IEEE Int'l Conf. Computer Vision and Puttern Recognition, pp. 20083-20093, 2023.
Meng, R., Yi, C., Yu, Y., Yang. S., Shen, B., and Kot, A. C. Semantic deep hiding for robust unlearnable examples. IEEE Transactions Information Forensics and Security, 2024.
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proc. Int'l Conf. Learning Representations, 2015.
Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. Cross- stitch networks for multi-task learning. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp- 3994-4003. IEEE Computer Society, 2016.
Standley, T., Zamir, A., Chen, D., Guibas, L., Malik, J., and Savarese, S. Which tasks should be learned together in multi-task learning? In Proc. Int'l Conf. Machine Learning. PMLR, 2020.
Nathan Silberman, Derek Hoiem, K. and Fergus, R. In- door segmentation and support inference from rghd im- ages. In Proc. IEEE European Conf. Computer Vision, 2012.
Sun, Y., Zhang, H., Zhang, T., Ma, X., and Jiang, Y.-G. Unseg: One universal unlearnable example generator is enough against all image segmentation. In Proc. Annual Conf. Neural Information Processing Systems, 2024.
Navon, A., Shamsian, A., Achituve, I., Maron, H., Kawaguchi, K., Chechik, G., and Fetaya, E. Multi-task learning as bargaining game. Proc. Int'l Conf. Ma- chine Learning, pp. 16428-16446. PMLR, 2022.
Vincent, J. Google accused of inappropriate access to medi- cal data in potential https://reurl. cc/bzK69v, 2019.
lawsuit, jun 2019. URL
Wang. X., Peng. Y., Lu, L., Lu, Z., Bagheri, M., and Sum- mers, R. M. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on sification and localization of common thorax diseases. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 2097-2106, 2017.
Qin, T, Gao, X., Zhao, J., Ye, K., and Xu, C.-Z. Learn- ing the unlearnable: Adversarial augmentations sup- press unlearnable example attacks. arXiv preprint arXiv:2303.151277 2023.
clas-
Qin, T., Gao, X., Zhao, J., Ye, K., and Xu, C.-z. Ap- bench: A unified availability poisoning attack and de- fenses benchmark. Transactions on Machine Learning Research, 2024.
Wang. X., Li, M., Liu, W., Zhang, H., Hu, S., Zhang, Y., Zhou, Z., and Jin, H. Unlearnable 3d point clouds: Class- wise transformation is all you need. In Proc. Annual Conf. Neural Information Processing Systems, volume 37, pp. 99404 99432, 2024.
Royer, A., Blankevoort, T., and Ehteshami Bejnordi, B. Scalarization for multi-task and multi-domain learning at scale. In Proc. Anmual Conf. Neural Information Process- ing Systems, volume 36, 2023.
Wang, X., Gao, X., Liao, D., Qin, T., Lu, Y., and Xu, C.- Z. A3: Few-shot prompt learning of unlearnable ex- amples with cross-modal adversarial feature alignment. Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2025.
Ruder, S. An overview of multi-task learning in deep neural networks. arXiv preprint
2017.
Sandoval-Segura, P., Singla, V., Geiping, J., Goldblum, M., Goldstein, T., and Jacobs, D. Autoregressive perturba- tions for data poisoning. Proc. Annual Conf. Neural
Wu, S., Chen, S., Xie, C., and Huang, X. One-pixel shortcut: On the learning preference deep neural networks, In Proc. Int'l Conf. Learning Representations, 2023.
12

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Xiao, H., Biggio, B., Brown, G., Fumera, G., Eckert, C., and Roli, F. Is feature selection secure against training data poisoning? In Proc. Int'l I Conf. Machine Learning, pp. 1689-16999 PMLR, 2015.
Yun, H. and Cho, H. Achievement-based training progress balancing for multi-task learning. In Proc. IEEE Int") Conf. Computer Vision, pp. 16935-16944, 2023.
Zhang, J., Ma, X., Yi, Q., Sang, J., Jiang, Y.-G., Wang, Y., and Xu, C. Unlearnable clusters: Towards label-agnostic unlearnable examples. In Proc. IEEE Int'l Conf. Com- puter Vision and Pattern Recognition, pp. 3984-3993, 2023.
Yang, W. Hu, 7., Lin, I.,, Liu, J., and Duan, L.-Y. Coding for intelligence from the perspective of category, arXiv preprint ar arXiv:2407.01017 2024.
Yu, D., Zhang, H., Chen, W., Yin, J., and Liu, T.-Y. Avail- ability attacks create shortcuts. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2367-2376, 2022a.
Zhang, Q. Wan, Z., Yang, Y., Shen, L., and Tao, D. Near- optimal online learning for multi-agent submodular co- ordination: Tight approximation and communication ef- ficiency. In Proc. Int'l Conf. Learning Representations, 2025.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. In Proc. Annual Conf. Neural Information Processing Systems, 2020.
Zhang, Y. and Yang. Q. A survey on multi-task learning. IEEE transactions knowledge and data engineering, 54(12):5586-5609 2021.
Yu, Y., Yang, W., Tan, Y.-P., and Kot, A. C. Towards robust rain removal against adversarial attacks: A comprehen- sive benchmark analysis and beyond. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp- 6013-6022, 2022b.
Zhang. Z., Song, Y., and Qi, H. Age progression/regresionn by conditional adversarial autoencoder. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition. IEEE, 2017.
Zhao, B. and Lao, Y. Clpa: Clean-label poisoning avail- ability attacks using generative adversarial nets. In Proc. AAAI Conf. on Artificial Intelligence, volume 36, pp. 9162-9170, 2022.
Yu, Y., Wang, Y., Yang, W., Lu, S., Tan, Y-P., and Kot, A. C. Backdoor attacks against deep image compression via adaptive frequency trigger. In Proc. IEEE Int't Conf. Com- puler Vision and Pattern Recognition, pp. 12250-12259, 2023.
Zhu, Y., Miao, Y., Dong, Y., and Gao, X.SS. Toward availability attacks in 3d point clouds. arXiv preprint 240. 2024a.
Yu, Y., Wang, Y., Xia, S., Yang, W., Lu, S., Tan, Y.-P., and Kot, A. C. Purify unlearnable examples via rate- constrained variational autoencoders. In International Conference on Machine Learning, ICML 2024, 2024a.
Zhu, Y., Yu, L., and Gao. X.-S. Detection and defense of unlearnable examples. In Proc. AAAI Conf. on Artificial Intelligence, volume 38, pp. 17211-17219, 2024b.
Yu, Y., Wang, Y., Yang, W., Guo, L., Lu, S., Duan, L.- Y., Tan, Y.PP., and Kot, A. C. Robust and transferable backdoor attacks against deep image compression with se- lective frequency prior. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2024b.
Yu, Y., Zheng, Q. Yang, S., Yang, W., Liu, J., Lu, S., Tan, Y.PP., Lam, K.-Y., and Kot, A. Unlearnable examples de- tection via iterative filtering. In International Conference Artificial Neural Networks, pp. 241-256 Springer, 2024c.
Yu, Y., Xia, S., Lin, X., Yang, W., Lu, S., Tan, Y.-P., and Kot, A. Backdoor attacks against no-reference image quality assessment models via a scalable trigger. In Proc. AAAI Conf. on Artificial Intelligence, pp. 9698-9706, 2025.
Yuan, C.-H. and Wu, S.-H. Neural tangent generalization attacks. In Proc. Int'l I Conf. Machine Learning, pp. 12230- 12240. PMLR, 2021.
13

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
A. More experimental details
A.1. Details the baseline UE methods
For experiments on classification datasets, we follow specific setups for both surrogate-dependent and surrogate- methods:
Surrogate-dependen Methods:
For EM (Huang et al., 2021), TAP (Fowl et al., 2021), and SEP (Chen et al., 2023), we use MTTL models with Linear Scalarization (LS) as the surrogate model.
For TAP and SEP, following their default settings, we employ PGD (Madry et al., 2018) with 250 steps and a step size of 0.064 to generate adversarial perturbations via targeted attacks. The target class for the k-th task is set as
255
(y*
Ck/ /2)%/x, where y is the original label and C₃ represents the total number of classes for the k-th task. For SEP, we use an ensemble of 15 checkpoints to enhance attack effectiveness.
For EM, perturbations are optimized using PGD with 20 steps and step size of 0.8
255 Surrogate models are trained
for 10 iterations after each loop of perturbation optimization. The stopping criterion, as per the default setting, requires the overall training set accuracy to reach 99%.
Surrogate-free Methods:
For LSP (Yu et al., 2022a) and AR (Sandoval-Segura et al., 2022), we evaluate two fusion strategies:
Averaging- Fusion: For each k-th task, C& class-wise perturbations with the same shape as the input are generated. These are then fused using the averaging- based approach.
* Patch-based Fusion: We divide the input into N x N patches, where N = 2 20 ([Rg [[)1 For an input shape of 3 x H x W, the patch size becomes 3 x H//N x W//N. Perturbations for the K tasks are placed corresponding patches (the first patches) based on their labels y². These perturbations are generated separately for Cs classes each patch for each k-tth task, and then the whole perturbations are interpolated H W.
from shape 3 x N (H//M) N (W//N) to match the input shape
It is worth noting that for both LSP and AR, the perturbation bound is set to & = 1 regarding the inputs with a shape of 32 x 32, as per their default settings. However, for larger input sizes, such as H x W, the bound to ensure comparability with other methods.
adjusted to f2 = 1 x 32 32
For experiments on the NYUv2 dataset, include only surrogate-dependent methods:
For EM (Huang et al., 2021), TAP (Fowl et al., 2021), and SEP (Chen et al., 2023), we use MTL models with Linear Scalarization (LSS) as the surrogate model.
For TAP and SEP, following their default settings, PGD (Madry et al., 2018) is employed to generate adversarial perturbations via targeted attacks. For the semantic segmentation task, the target class is defined as (y* + (3//2))11 given there are 13 semantic classes in total. For the depth estimation and surface normal estimation tasks, where defining target is challenging, untargeted attacks are used, aiming to maximize the loss between the attacked prediction and the ground truth labels.
For SEP, an ensemble of 10 checkpoints is used to enhance attack effectiveness.
For EM, perturbations are optimized using PGD, while surrogate models are trained for 10 iterations after each loop of perturbation optimization. The stopping criterion is reached when the loop of perturbation optimization completes 10 iterations, resulting in total of 200 iterations for each perturbation optimization (as each PGD step runs for 20 iterations).
14

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
A.2. Details of the models and the training
Design of MTL-UE generator. Assume the input shape is set H x W or HI W for Grayscale images such as the ChestX-ray- -ra--1 dataset, and there are K tasks. For the design of MTL-UE's generator, use 9 convolutional layers and x0.5 downsampling as E(; ΦE), resulting in 2 with shape 128 x x W For the set {fe k=1- each eye is 2
designed to match the length and width of z, giving it the shape 16 x x W. For D(= we use Conv Trnnrrssspd 2
layers and one x2 upsampling. The input ()) is the concatenation of z and (ey, Mk=1 J resulting shape of (128 + 16 x K) 2
H W. and the output is the same shape as the input to E(:
2
Specifically, for the NYUv2 dataset, we use the embedder (:: instead of ege. The structure of the embedder is identical to that of the corresponding E(; ΦE), with the input shape for segmentation labels and depth being 1 x H x W, and the input for normal labels being 3 x H x W.
MTL and STL model training. For the CelebA, ChestX-rayle 14, and UTKFace datasets, we train the models for 60 epochs using the Adam optimizer, starting with a learning rate of 1e-3. We apply the MultiStepLR scheduler, with milestones at epochs 36 and 48, and gamma of 0.1 to adjust the learning rate. The batch size is 512 for CelebA and UTKFace, and 128 for ChestX-ray14. 14. Data augmentation techniques include RandomCrop and RandomHorizootallFinn to improve the model's generalization. For the NYUv2 dataset, we train the models for 200 epochs, starting with a learning rate of le-4, and use a StepLR scheduler with step size of 100 and a gamma of 0. to reduce the learning rate during training. We set the batch size to 8 for this dataset due to the larger image dimensions and the computational load involved.
MTL-UE generator training. The batch sized, training epochs, learning rates, optimizers, and learning rate schedulers for the generator are configured identically those used for training the MTL models on the respective datasets. For the hyperparameters in Alg. 1, is set match the baseline methods, with the default value 255 8 for all datasets. The weight
A is set to 20, and A2 is set to 100 across all datasets. For MTL-UE-EM, where the "Train-surrogate" is set to True, we use 10 iterations, consistent with the baseline EM methods. For MTL-UE- TELL-UE-TAN TAP and MTL-UE-SEP, where "Train-surrogate" is set to False, we pretrain the MTL models on the respective datasets using the default training settings as described earlier. For the remaining configurations, we follow the same setup as the corresponding baseline methods: EM, TAP, and SEP.
A.3. Computational Complexity
We also analyze the computational complexity of MTL-UE and the baseline methods. For both EM and MTL-UE-EM, the computational complcxity is nearly identical since they share the same stopping criterion. For TAP and SEP, each perturbation requires 250 optimization steps on each surrogate model. In contrast, for MTL-UE-TAP and MTL-UE-SEP, perturbations are optimized once per epoch, with a total of 60 epochs for classification datasets and 200 epochs for the NYUv2 dataset, which means 60 optimization steps each sample for classification dataset, and 200 optimization steps on each sample for NYUv2 dataset. This makes our methods more efficient compared to the baseline TAP and SEP. For AR and LSP, the perturbations are predefined and hand-crafted, making these methods the most computationally efficient.
A.4. Parameter Quantity
We provide the parameter counts for all competing methods:
For EM, TAP, and SEP, the parameters correspond to the perturbations. For example, on the CelebA dataset, the perturbation parameters have a shape of 162770 x 70 70 2, 392, 719, 000.
For our method, the parameters are derived from the MTL-UE generator, which consists of 3,086,147 147 parameters for the CelebA dataset.
For AR and LSP, the parameter count for the averaging based fusion method is 2 x 40 x 3 70 70 1, 176, 000, while for the patch-based fusion method, it is 2 x 40 x 3 x 8 x 8 15, 360.
Overall, our method is efficient, requiring only slightly more parameters than the patch-based AR and LSPP methods.
15

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Table 10. Quantitative results on the UTKFace with different number of tasks: We report the average accuracy (%) across the selected tasks. Both MTI. and STI. models adopt R8sNet-18 the encoder. The MTL. models employ I.S as the wask-weighing.
Selected tasks
Age
Age, Race
Age, Race, Gender
Model
MTL/STL Avg.l
MTL Avg.b
STL
MTL Avg-l
STL
Metric
Avg-l
Avg.l
Clean
57.51
70.79
70.45
78.97
78.92
LSP (Yu et al., 2022a) (Putch)
10.65
13.11
18.27
28.58
32.43
AR (Sandoval-Segee Saaadovv--Seguraaee et al., 2022) (Patch)
9.47
14,60
14.61
27.41
26.75
LSP (Yu et al., 2022a) (Average)
10.65
18.23
30.73
28.16
45.91
AR (Sandoval- (Sandoval-Segura et al,, 2022) (Average)
9.47
12.34
14.07
36.79
36.67
EM (Huang et al., 2021)
10.89
19.16
19.26
31.74
50.88
TAP (Fowl et al., 2021)
25.82
31.57
49, 10
39.29
39.36
SEP (Chen et al., 2023)
22.74
32.63
48.86
40.70
60.78
MTL-UEE-MM
18.82 88.82
10.60
8.82
25.84
26.32
MTL.-UE UTPP
9.18
9.74
11.92
19.84
25.59
MTL-UE-SEP
5.49
11.78
12.61
21.19
28.24
Table II. Ablation study of the hyperparameters of MTL-UE CelebA.
Model +
MTL.
STL
Base UE +
EM
TAP
SEP
EM
TAP
SEP
(: λ 2, 100
75.04
60.53
59.39
73.88
69.67
75.43
C: λ 200, A2 100
74.27
59.60
59.06
75.13
67.49
76.02
8: x 20,, Au 10
73.85
58.86
58.81
73.29
67.84
77.08
A 20,
1000
74.23
60.02
58.09
74.95
68.98
75.82
MTL. UE 20, Ag 100)
74.38
59.51
58.73
74.26
68.65
76.39
Table 12, Results under state-of the-art defenses. We select the UTKFace datasct and train the MTL models.
Defeuse
None Gender]
ISS-IPEG Race]
ctal., 2023)
ISS-Grayscale et 2023) Race]
ISS-BDR al., Gender,
Tusks ->
Agel
Race|
Avg.l
Age.
Gender.
Avg.L
Agel
Gender.
Avg-L
Agel
Race
Avg.l
Clean
60.32
84.07
92.51
78.97
59.26
83.92
92.24
78.47
59.45
83.19
91.77
78.14
58.69
82.45
92.32
77.82
LSP (Patch)
18.40
49.83
28.58
15.80
AR (Patch)
9.70
19.66
$2.87
27.41
47.07
82.55
90.99
73.54
27.41
41.35
52.72
40144
14.64
26.24
52.78
31.22
LSP (Average)
12.59
19.72
52.17
28.16
49.81
80.76
90.86
73.81
16.98
51,81
56.50
42.76
32.41
65.70
78.23
58.78
(Avenge)
36.79
57.87
77.78
6.46
BM
19.24
17.43
58.57
31.74
58. 38
82.49
91 86
71.57
19.81
24:81
58.84
34.49
19.81
24.81
58.84
34.49
TAP
25.86
39.28
52.74
39.29
16.69
78.27
89.18
71.38
24.20
56.54
69.37
50.01
24.20
56.54
69.37
50,04
SEP
52.64
40,70
71.77
MTL-UE-EM
10.00
12.78
54.73
25.84
6.75
14.05
$7.78
26.
7.72
16.31
$7.62
27.22
12.95
22.51
63.35
32.93
MTL-UE- -IL-UE-TAP
9.49
18.23
31.81
19.84
15.82
26.27
26.39
22.83
10.68
16.73
40.27
10.68
15.02
24.73
26.39
22,05
MTL-UEESEP
16.20
40.08
21.19
15.23
43.27
32.78
5,56
28.78
26.56
B. Additional Results
B.1. More quantitative results Performance of UE Vs. the number of tasks. Additional results on the UTKFace dataset with varying numbers of tasks presented in Tab. 10. Even with limited number of tasks, MTL-UE consistently outperforms the baseline approaches.
B.2. More ablation study
We do additional ablation studies MTL-UE, focusing on the hyperparameters x and X in Alg. 1. Specifically, experiments are conducted the CelebA dataset. The results in Tab. 11 indicate that MTL.UEE is relatively insensitive these hyperparameters. This may be because Lint and Linter are easier to optimize within the combined loss framework in Alg. 1. As long as Y1 and X2 are not set too small, these hyperparameters have minimal impact on the optimization process of MTI-UE.
B.3. Results to defenses
We evaluate MTL-UE and competing methods against existing UE defenses applicable to MTL models. Experiments are conducted on the UTKFace dataset using MTL models. We select the SOTA defense, Image Shortcut Squeezing (ISS) (Liu et al., 2023), which includes three preprocessing-basee techniques: JPEG compression (quality set to 10), grayscale conversion, and bit-depth reduction (BDR, depth set to 2), follwoing their default settings. Results in Tab. 12 show that MTL-UE remains significantly more robust and consistent than baseline UE methods. Notably, under JPEG compression, all baseline methods fail, whereas MTL-UE maintains strong performance. This robustness likely due to MTL-UE's clearer semantic patterns, distinct contours, and sketch lines, which better withstand ISS -S--induedd corruptions.
16

---

MTI-UE: Learning to Learn Nothing for Multi-Task Learning
Table 13. Backbone transfer: Performance on the NYUv2 (Nathan Silberman Fergus, 2012) when transfering to MTL and STL models with various backbone. The MTL models employ HPS (Caruana, 1993) as architecture and linear scalarization as the task- strategy.
Backhone-
ResNet-101 101
ResNet-152
WideResNet-S0 50
Task - Metric
Sogmeutation
Depth
Normal
Segmentsttin
Depth
Normal
Segmentation
Depth
Normal
mloti,
PAcn]
AFrt
RErt
Mean"
MED
mloU].
PAcc]
AErrt
RFm
Meant
MED1
mlol.]
PAce.
AFmr
RFmr
Meant
MFD1
Clean
$4,38
76.00
0,3802
0.1582
23.16
16.79
54,88
76.29
0.3690
0.1502
22.71
16.33
52.42
74.55
0.4044
0.1746
24.14
17.80
EM
31.78
50.72
0.5634
0.2076
28.23
22.41
32:51
50.03
0.5395
0.1990
28.09
22.09
29.08
45.71
0.6352
0.2246
30.71
25.84
TAP
37.73
56.71
0.4905
0.1908
27.59
21.25
32,96
50.58
0.5176
0.1922
28.22
21.97
37.98
57.13
0.5051
0.2016
29.41
23.26
SEP
27.85
40.81
0,5457
0,2151
30.61
24.97
29.78
43,91
0.5789
0.2061
30.11
24.18
30.73
45.99
0,5425
0.2175
29.81
23.61
MTL-UE-EM
2.38
16.03
0.8887
0,2903
38.23
33.47
1.75
15.82
0.9638
0.3062
34,91
29.19
2.34
16.16
0.8448
0.2812
36.71
31.71
23.25
37.00
0,9739
0.3203
37.56
31.66
24.10
35.71
0.8993
0.2942
36.66
30.25
23.59
36.66
0.8403
0.2853
38.72
32.87
MTL-UE-SEP
18.55
29.78
0.9043
0.3046
42.07
36.44
19.31
25.59
0.8499
0.2835
37.64
31.72
17.38
29.48
0.8095
0.2836
41.15
35.84
Table 14. Weighting strategies transfer: Performance on the NYUv2 (Nathan Silberman & Fergus, 2012) when transfering to MTL and
STL models with various waighteigh strategies. The MTL models employ HPS (Caruana, 1993) as architecture and ResNet- 18 (He
et al., 2016) as encoder's 's hackbone.
Weighting
UW
REW
Align
FairGrad
Task
Segmentation
Depth
Normal
Segmentation
Depth
Nermal
Segmentation mlx31) PAoc]
Depth
Nernal Moant
Segmentation mlell]
Devth
Normal Moinf
Metric-
mllli]] PAce,
AFrrt
RFrt
Meant
MFD]
milet] PAos]
AFrit
RFr
Moset MFD]
AFrit
RFr
MFD;
PAoc]
AFrr
RFart
MFD1
Clean
$2.09
1,06
23.76
17.33
52.70
74,00
0:3006
1622
16-92
$2.50
74.24
0.077
22.06
16.37
52.21
74.32
0.3927
1696
22.85
16.34
FM
28.64
0.2391
0.2281
23.15
28.01
0:2314
TAP
27.6
34.79
0.6449
0.2485
31,75
33.06
26.68
21.12
36.06
0,6314
0.2322
SEP
12.30
24.40
0,6996
0.2823
36.77
31.19
11.40
23.67
0.7048
0.2899
35.88
30.20
12.36
24.73
0.6882
0.2813
35.98
29.73
13.47
25.78
0.6812
0.2766
35.36
MIL-UE-EM
2,33
15.96
0,9515
0.3110
58.33
34.14
2,05
15.79
L,0005
0.3344
37.03
32.15
1,78
15.24
1,0002
0.3265
37.34
31.58
2.07
15.82
0.9987
0.3260
35.84
29.91
MIL-UE-TAP TAP
17.08
28:98
0.9620
0.3137
40003
34.4%
16.85
29.16
LIK31
0.3489
411.6
35.68
18.31
29.53
1,0100
0.3310
38.37
31.78
17.93
30,83
0.9513
0.3169
38.78
32,99
MT--1F--FPP
13:96
21.71
0.8400
0.2936
41.19
35.85
15.64
78.85
0.9047
0 3057
42.40
$5.15
16.50
26.67
0.8961 0. 3024
41.85
35.67
16.58
26,
0.9030 0.5017 40.50
54.87
Table 15. Architecture transfer: Performance on the NYUv2 (Nathan Silberman & Fergus, 2012) when transfering to MTL and STL models with various architecture. The MTL models employ ResNet-18 (He et al., 2016) encoder's backbone and linear scalarization as the task weighting strategy.
Backbone-
Cross-sticch
MTAN
LTB
Task
Segmentation
Depth
Normal
Segmentation
Dopth
Normal
Segmentation
Depth
Normal
Metric
mlot].
PAce]
AFrr
RErt
Mean
MED
mloU]]
PNoc]
AFrrt
RErr
Meant
MED1
mloo:]]
PAcc.
AFrr
RFrr
Meant
MED1
Claan
52.48
74.74
0,3874
0.1621
23.34
16.52
53.99
75.44
0.3806
0.1619
23.34
16.67
51.53
74.30
0.3819
0.1585
23.44
16.54
24.03
42.11
0.6582
0.2377
30.29
27.04
45.61
0.6241
0.2334
30.32
24.03
28.55
0.6357
0.2334
30.04
23.65
TAP
21.05
38.89
0.6622
0,2455
33.99
27.18
19.72
30,56
0.6468
0.2484
34.06
27.94
18.24
32.63
0.6527
0,2472
33.57
26.86
SEP
21.18
0.7687
0.2814
36:03
29.83
21.45
0.7049
0.2854
36.06
30.25
10.53
21.98
0.7679
0.2793
36.46
30.47
MTL-UE-EM
1.78
15.77
1.0116
0.3249
35.18
29.46
1.54
15.66
0.9962
0.3211
36.00
30.29
1.52
15:67
1.0654
0.3408
36.16
30.67
MIL-UE-TAP
1.0652
0.3457
39.06
32.83
25.33
1.1076
0.3587
42.24
36.80
20.44
36.43
1.0231
0.3309
38.96
32.33
MILL-UE-SEP
15.78
30.89
0.9215
0,3099
42.77
36.85
14.94
26.49
0.8888
0.3015
43.73
38.62
17.87
30.27
0.9095
0.3036
40.71
34.88
B.4. More transferability results for NYUv2 dataset
Transferability results for the NYUv2 dataset are shown in Tab. 13, Tab. 14, and Tab. 15. In addition to backbone and weighting strategy transfer on the CelebA dataset, we also evaluate architecture transfer for NYUv2. For backbone transfer, we use 1011 ResNet-152, and WideResNet-500 as backbones for victim models. For weighting strategy transfer, we adopt the same strategies as in the CelebA experiments. For architecture transfer, Cross-stitch (Misra et al., 2016), MTAN (Liu et al., 2019), and LTB (Guo et al., 2020) are included for evaluation. As observed, MTL-UE consistently protects against unauthorized model training on the NYUv2 dataset across various model backbones, weighting strategies, and MTL architectures.
17

---

MTL-UE: Learning to Learn Nothing for Multi-Task Learning
CelebA
ChestXray-14
UTKFace
Clean
EM
TAP
SEP
LSP (P)
AR (P)
(SP(A)
AR (A)
Ours-EM Ours-TAP Ours-SEP
NYUv2
Clean
EM
TAP
SEP
Ours-EM
Ours-TAP
Ours-SEP
Figure 9. Visual results: the odd/even rows show the perturbattions/imaees, respectively. Perturbations are in dependently normalized to [0.1] for clarity.
B.5. More visual results
We provide additional visual examples of perturbations and poisoned images from the CelebA (Liu et al., 2015), ChestX- ray 14 (Wang et al., 2017), UTKFace (Zhang et al., 2017), and NYUv2 (Nathan Silberman & Fergus, 2012) datasets, as shown in Fig. 9. We observe across all datasets that our methods offer more interpretable cues, guiding the model to learn the added perturbations rather than the benign features the clean data. In particular, MTL-UE-TAP and MTL-UE-SEP display distinct outlines or contour lines.
18


Reviews:
[{"review_id": "2R7C0KtWlT", "reviewer": "Reviewer_tZ3V", "summary": "This work studies unlearnable examples (UE) for multi-task learning (MTL). The authors first evaluated baseline UE methods in the MTL scenario, showing that existing UE methods are not effective on MTL when more tasks are involved. Motivated by this observation, MTL-UE is proposed taking both single-task and multi-task learning into account. Comprehensive MTL experiments are provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Further clarify the experimental results. What does STL mean in Figure 2, and why do the baseline methods perform poorly, even on STL?\n\nA1. The pipeline for UE has two stages:\n\n- **Stage 1**: UE generation process (Section 4.2).\n- **Stage 2**: UE performance evaluation, where generated UEs train victim models, and these models are tested on clean data.\n\nStage 1 focuses on generating UE to protect **all tasks simultaneously** using a surrogate MTL model. Stage 2 evaluates in two ways: a) Train one MTL model for all tasks. b) Train individual STL models for each task, and averaging the evaluation results across all tasks.\n\nFigure 2 shows UE performance vs. the task number. For each x-axis point, only the first $K$ tasks are selected for both stages, and UE are generated to protect them. Stage 2 evaluates a) MTL (left) and b) STL (right). Other tables use full task sets with the above stages for a complete UE assessment. All baseline methods designed for STL, perform poorly in Stage 2 for both MTL and STL, as Stage 1 generates UE for MTL models, not STL settings. Figure 2 also shows that when $K$ is small, these baselines perform well but degrade as $K$ increases due to greater misalignment between Stage 1 and Stage 2. We will clarify these in the paper.\n\n---\n\n**Q2**. Can original sample-wise UE be modified to work under the MTL scenario? Except for class-wise noises making the optimization easier, what is the potential obstacle that makes sample-wise UE not work?\n\nA2. To reduce the high intra-class variance in optimization-based methods, we add additional loss terms. As these methods use a surrogate MTL model during optimization, we define $L_{std}=[L_{std}^1,\\ldots,L_{std}^D]$, and $L_{std}^d$ is defined in line 201 of the paper. We use two loss terms, $L_{std}^{mean}$ and $L_{std}^{max}$: the mean and maximum of $L_{std}$. In the perturbation optimization (Eqs. (1) & (2)), we add $\\lambda_1 L_{std}^{mean}+\\lambda_2 L_{std}^{max}$ to the original loss. If the original optimization is to maximize, we apply a negative sign.\n\nWe experiment on CelebA with a batch size of 1024. The table below shows MTL model results with various hyperparameters. As $L_{std}^{mean}\\approx5$ and $L_{std}^{max}\\approx100$, these choices are reasonable. Despite different settings, the new losses don't outperform the original one. $L_{std}^{max}$ largely degrades performance, and $L_{std}^{mean}$ has a smaller, but still negative, effect.\n|$(\\lambda_1,\\lambda_2)$|(0,0) in paper|(0.5,0.01)|(0.05,0.001)|(0.05,0)|(0,0.001)|(0.005,0.0001)|(0.005,0)|(0,0.0001)|MTL-UE|\n|-|-|-|-|-|-|-|-|-|-|\n|EM|75.66|90.45|90.57|89.70|90.4|88.05|74.73|87.14|74.38|\n|TAP|85.24|90.84|89.95|88.53|89.79|87.82|85.82|87.73|59.51|\n|SEP|84.25|90.34|90.15|89.25|90.64|89.57|85.37|88.97|58.73|\n\n**Challenges in Adopting These Loss Terms**  \n\n- Batch Size Limitation: With 160k images in CelebA and a 1k batch size, variance estimates are unreliable, and increasing batch size is infeasible due to GPU limits.\n\n- Surrogate Model Misalignment: TAP and SEP use a surrogate MTL model trained on clean data, misaligning with the victim model trained on UE. EM partially addresses this by training on UE data, but weak early-stage perturbations lead to suboptimal results.\n\n- Conflict with Original Loss: Minimizing $L_{std}^{mean}$ and $L_{std}^{max}$ conflicts with the original loss. $L_{std}^{max}$ degrades UE effectiveness, and $L_{std}^{mean}$ with small $\\lambda_1$ has minimal impact.\n\n- Computational Overhead: These losses increase computation by $\\times28$.\n\n**Potential Obstacles for Existing Sample-Wise UE in MTL**\n\n- Model Alignment Issue: Due to GPU limits, UE generation uses one MTL model instead of multiple STL models (e.g., 40 for CelebA), causing misalignment with STL victim models. In MTL-UE, a surrogate MTL model is used, and images with the same label $y^k$ for the $k$-th task share the embedding $e_{y^k}^k$. Thus, MTL-UE suffers less degradation when training STL victim models than MTL models. CelebA results show MTL-UE-EM improves slightly in MTL but shows gains in STL, highlighting its alignment effectiveness.\n- Lack of Distribution-Level Optimization:  Perturbations mislead the victim model by mapping $x_i+\\delta_i$ to labels, poisoning the data distribution.\n   - Previous UE methods individually optimize samples, missing distribution-level effects.\n   - EM trains on batched poisoned data, implicitly considering distribution, is better than TAP and SEP.\n   - In MTL settings like CelebA (40 tasks), lack of distribution optimization is problematic.\n   - MTL-UE creates poisoned distributions with class-wise embeddings, reducing intra-class variance and effectively misleading victim models.\n- Re-optimizing for different protected task sets: Prior sample-wise UE methods require re-optimizing perturbations for each combination of tasks to protect. MTL-UE, once optimized on all tasks (Sec. 5.3), allows flexible task selection to protect without re-optimizing.", "claims_and_evidence": "Intra-class variance of optimization-based sample-wise approaches are high, e.g., EM, TAP, and SEP. The explanation makes sense to me. But would it be possible to modify, let’s say  sample-wise EM, to fit the MTL objective by limiting the intra-class variance? EM is designed for STL, so it makes sense it performs badly on MTL. It would be great if the authors could articulate the potential of modifying sample-wise optimization-based approaches to fit MTL. I think the class-wise noise makes the optimization of MTL UE easier, but it's not essential.", "methods_and_evaluation_criteria": "The proposed methods and evaluations make sense. However, the exact evaluation criteria needs to be further clarified. See the comments above regarding STL performance.", "theoretical_claims": "Theoretical analysis is not provided in this work.", "experimental_designs_or_analyses": "Experiments shown in Figure 2 can be further explained. How to interpret the results on STL with task 40? Does it mean that the UEs are generated on one task and tested on all tasks? Especially, the claim that STL models are more robust to UE than MTL models, can be further clarified. This result is important since patch-based AR outperforms other approaches, which is used as the basis of analysis in the following paper content. \n\nThe baseline performance in table 2 can be further clarified. Why is the baseline STL performance also limited? Does it follow the same manner of Figure 2? It would be great if the authors could clarify. I would anticipate that the baseline STL performance should be better than reported.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "This work explores the application of unlearnable examples in multi-task learning. The observation is interesting, since STL unlearnable examples do not generalize in the MTL scenario. My concerns listed above can mainly be summarized as the following two points:\n\n1. Can original sample-wise unlearnable examples also be modified to work under the MTL scenario. Except for class-wise noises making the optimization easier, what is the potential obstacle that makes sample-wise unlearnable examples not work?\n2. The provided experimental results can be further clarified and interpreted. For example, what does STL mean in Figure 2? Why does baseline perform not well even on STL tasks?\n\nIt would be great if the authors could clarify these questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HVEHoju8Xt", "reviewer": "Reviewer_abGT", "summary": "In this paper, we propose an effective method for generating unlearnable samples for multi-task learning (MLT), which uses a generator to generate perturbations instead of the traditional iterative method. In this paper, the effectiveness of the method is analyzed and validated in terms of both accuracy and robustness, and the effectiveness of the method is demonstrated on several datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Insufficient coverage of existing literature.\n\nA1. Thank you for the advice! We’ll add recent works on **data poisoning** in the related work. As we focus on UE, we add a comparison between UE and other poisoning attacks (the table below) in the updated paper to broaden the literature coverage and emphasize the need for UE.\n|Aspect|UE|Label Flipping|Backdoor Attacks|Availability Attacks (Partial Poisoning)|Targeted Data Poisoning|\n|-|-|-|-|-|-|\n|**Attack Objective**|Significantly reduce accuracy on clean data, causing near-random guesses|Decrease accuracy for all classes or specific classes|No impact on clean data, but manipulate predictions to a specific class for data with predefined triggers|Reduce accuracy on clean data, but to a lesser degree compared to UE|Misclassify specific target classes or instances|\n|**Nature of Poisoning**|Add subtle, imperceptible perturbations across the entire training dataset and maintain true labels|Modify a portion of dataset by changing labels only|Modify a portion of dataset with triggers (perturbations) and potential label changes|Modify a portion of dataset with unbounded perturbations and potential label changes|Modify a portion of dataset with unbounded perturbations and potential label changes|\n\n**Q2**. Challenges of new issues are not significant across datasets.\n\nA2. Following original settings, we report accuracy (%) for CelebA and UTKFace, while for ChestX-ray14 the metric is AUC-ROC (see Sec. 5.1). The drop in AUC-ROC from 0.7577 to near 0.5 is significant.\n\n**Q3**. Questioning of method validity.\n\nA3. MTL-UE works well with surrogate-dependent UE methods, improving performance across most datasets. The exception is ChestX-ray14 with TAP and SEP, where the clean model performs not well (AUC-ROC=0.75), limiting the success of adversarial attacks and, consequently, the effectiveness of adversarial-attack-based UE methods like TAP and SEP. However, MTL-UE still improves on EM, boosts TAP and SEP in MTL, and matches STL results for TAP and SEP.\n\n**Q4**. Insufficient technical novelty.\n\nA4. MTL-UE innovates in UE for MTL by solving model misalignment with shared task embeddings, boosting STL performance. It optimizes at the distribution level using class-wise embeddings and allows flexible task protection without retraining. Theoretically, we introduce embedding regularizations (Intra-ER & Inter-ER) to maximize intra-task separation and promote inter-task independence, enhancing the learning of spurious features. See A5 for more details on the technical novelty to solve challenges. We’ll refine the theoretical analysis and revise the paper to clarify these.\n\n**Q5**. The applicability of MLT tasks is unclear.\n\nA5. MTL is vital in real-world applications like autonomous vehicles, which handle multiple tasks simultaneously. Research has advanced multi-task datasets and MTL models. While UE has been studied for STL, we are the first to tackle unauthorized training on multi-task data and protect data privacy. To bridge the gap, we adapt UE methods from STL to MTL and highlight key challenges:\n- Model Alignment Issue: GPU constraints limit UE generation to use one MTL model instead of multiple STL models (e.g., 40 for CelebA), causing misalignment when training STL victim models. In MTL-UE, task-wise embeddings align UE within each task, reducing performance degradation when training STL models compared to MTL ones. In CelebA, MTL-UE-EM shows minor MTL gains but major STL improvements.\n- Lack of Distribution-Level Optimization:  Prior UE methods optimize samples independently, ignoring distribution-level effects. In MTL settings like CelebA (40 tasks), this issue worsens. MTL-UE uses class-wise embeddings to poison distributions per task, reducing intra-class variance and reinforcing spurious correlations, enhancing attack effectiveness.\n- Re-optimize for different protected task sets: Prior sample-wise UE methods require re-optimizing perturbations for each combination of tasks to protect. MTL-UE, once optimized on all tasks, allows flexible task selection to protect without re-optimizing (Sec. 5.3).\n\nWe'll add these in the paper.\n\n\n**Q6**. The question of the plausibility of attack scenarios.\n\nA6. Unlearning in MTL protects sensitive multi-task data from unauthorized training due to privacy concerns. In medical AI, MTL models use X-rays, MRIs, and patient histories for diagnosis, where unauthorized training risks privacy and ethics violations. Facial recognition models (e.g., CelebA) can enable surveillance and profiling. Social media analysis (e.g., Weibo-20) enables large-scale surveillance. Smart surveillance models (e.g., DukeMTMC) risk privacy infringement. MTL-UE prevents unauthorized use by introducing unlearnable perturbations, degrading MTL and STL model performance, thus enhancing data privacy and intellectual property protection. We'll add these in the paper.\n\n**Q7**. Evaluation of computational complexity.\n\nA7. Please refer to A4 for reviewer AfLP.", "claims_and_evidence": "The authors show the accuracy of existing UE methods in SLT and MLT scenarios in Fig. 2 and find that directly migrating existing UE methods to MLT leads to some degree of accuracy degradation with k increasing. This indicates the poor applicability of the existing methods, thus providing motivation for the proposed method in this paper.", "methods_and_evaluation_criteria": "The proposed method improves the accuracy of UE to some extent. The test datasets include face and medical images, which are selected with practical significance.", "theoretical_claims": "The author does not make explicit theoretical claims, so there are no relevant questions.", "experimental_designs_or_analyses": "Experiments are presented on multiple datasets and the results show that the proposed method outperforms existing methods, supporting the authors' conclusions.", "relation_to_broader_scientific_literature": "The authors found experimentally that UE performs poorly in multi-task learning and propose improvements accordingly.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\nThe authors identified the problem of low UE accuracy in MLT scenarios and provided an explanation.\nThe authors proposed an effective UE method and validated it through extensive experiments.\n\nWeaknesses:\n1. Insufficient coverage of existing literature. The authors' literature research on data poisoning methods is more limited, covering only up to 2023, while there are still a number of new researches in the field that are worth including. In addition, experimental comparisons mainly focus on UE methods, and fewer comparisons on data-based poisoning fail to clarify its necessity.\n2. Challenges of new issues are not significant across datasets. Figure 2 illustrates the task differences between MLT and SLT on the CelebA dataset, but on ChestX-ray14 this difference does not seem to be significant. Especially on the clean data, where 0.75% and 91.1% accuracy are not in the same order of magnitude, it is not quite clear to me why there is such a large difference in task accuracy between the different datasets.\n3. Questioning of method validity. The authors provide variants of MLT on three different methods, but the impact of these changes varies dramatically across datasets. For example, the MLT-UE variant on TAP has an even higher average accuracy of the STL model than TAP when testing the ChestX-ray14 dataset. Some of the data suggests that the improvements of MLT-UE are limited and even not always effective in some cases. This contradicts the authors' claim of enhanced attack performance.\n4. Insufficient technical novelty. The authors' main improvement is to replace labels with feature embeddings, which is simple and effective but weak in terms of novelty. I expect that the authors can analyze the effectiveness of this improvement from a theoretical perspective, rather than just performing technical implementation and experimental evaluation.\n5. The applicability of MLT tasks is unclear. Although the authors include the performance of SLT in their discussion, their motivation for the study seems to be based on the fact that there is some kind of generic challenge between the two. It is not clear to me exactly how this challenge relates to MLT, and I would have appreciated a clearer clarification.\n6. The question of the plausibility of attack scenarios. Could the authors please elaborate on the unlearning needs of MLT in real-life scenarios and the scope of application of this paper's methodology, which would help to better understand its practical application value.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O9mQJDOK58", "reviewer": "Reviewer_AfLP", "summary": "This paper introduces MTL-UE, the first framework for generating unlearnable examples (UEs) tailored for multi-task learning (MTL) models. While existing UE methods focus on single-task learning (STL) to prevent unauthorized training on personal data, modern AI increasingly relies on generalist MTL models. This work addresses this gap by introducing a generator-based approach with class-wise feature embeddings and embedding regularization, improving attack effectiveness and robustness. It supports dense prediction tasks, integrates seamlessly with existing surrogate-dependent UE methods, and enables partial task and data protection. Extensive experiments demonstrate its effectiveness over baseline UE methods across multiple backbones and task-weighting strategies.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. How would these methods perform if ViT-B were used as the backbone for the surrogate models?\n\n**A1**. In addition to the results in Table 6, we conducted experiments using ViT-B as the backbone for the surrogate models on the CelebA dataset. The results in the table below show that this choice improves performance when the victim models also use ViT-B but can negatively impact performance when the victim models are CNN-based. This suggests a trade-off in selecting CNNs or ViTs for the surrogate models, depending on the target victim model's backbone.\n\n|Model $\\rightarrow$|MTL|STL|\n|-| - | - |\n|Backbone $\\rightarrow$|RN-18, RN-50, VGG-16, DN-121, ViT-B, Avg.|RN-18, RN-50, VGG-16, DN-121, ViT-B, Avg.|\n|EM|88.86, 88.75, 84.22, 80.71, 88.67, 86.24|90.11, 89.93, 89.46, 86.42, 86.03, 88.39|\n|TAP|85.55, 86.32, 83.28, 79.38, 86.32, 84.17|88.42, 88.14, 87.97, 87.46, 85.28, 87.45|\n|SEP| 78.67, 81.24, 79.41, 76.55, 79.33, 79.04|87.06, 87.67, 83.36, 83.50, 85.93, 85.50|\n|MTL-UE-EM| 73.68, 73.65, 74.77, 75.85, 74.01, 74.39|78.06, 77.87, 78.72, 77.85, 78.98, 78.29 |\n|MTL-UE-TAP| 68.91, 69.30, 69.40, **66.36**, 73.56, 69.50|76.32, 77.27, 75.13, 70.08, 80.92, 75.94|\n|MTL-UE-SEP| **53.79**, **59.63**, **63.37**, 71.49, **64.01**, **62.45** | **73.12**, **74.45**, **72.71**, **64.45**, **78.66**, **72.67** |\n\n**Q2**. Among the 40 tasks in CelebA, are there specific tasks that are significantly easier or harder to degrade?\n\n**A2**. We compute the accuracy drops for all 40 tasks when training STL models on MTL-UE compared to models trained on clean data. In MTL-UE-EM, tasks 38, 35, and 24 experience the largest drops (-58.74, -58.03, -44.39), while tasks 23, 15, and 39 are least affected (-0.48, -0.70, -1.00). In MTL-UE-TAP, tasks 25, 1, and 40 degrade the most (-80.22, -67.02, -61.90), whereas tasks 36, 38, and 27 show minimal impact (+0.02, 0.00, -0.32). In MTL-UE-SEP, tasks 15, 37, and 2 suffer the highest degradation (-61.94, -49.69, -43.03), while tasks 26, 27, and 14 are least affected (+0.09, +0.02, 0.00). These results suggest that the susceptibility of tasks to degradation varies significantly across different base UE strategies. Tasks experiencing the largest drops may rely on more vulnerable feature representations, making them easier to disrupt. In contrast, tasks with minimal impact likely depend on more robust or less perturbed features. \n\n**Q3**. More analysis on the influence of the number of tasks on the UEs performance.\n\n**A3**. Thank you for the suggestion! We provide a detailed analysis of how the number of tasks affects UE performance. As shown in Figure 5, when training MTL models, increasing the number of tasks to 10 causes only a slight performance drop (minor accuracy increase). From 10 to 40 tasks, MTL-UE remains stable, maintaining ~60% accuracy for both MTL-UE-TAP and MTL-UE-SEP. This stability is likely due to the shared encoder in MTL models (for both the UE generation and victim model training), which facilitates learning spurious features across all tasks, preventing performance degradation.\n\nHowever, when training STL models with varying task numbers, MTL-UE performance gradually declines as the number of tasks increases from 10 to 20 and 20 to 40. This may result from a mismatch between the UE generation and victim model training, as the former uses MTL surrogate models while the latter adopts STL models. We will include this analysis in the updated version.\n\n**Q4**. A direct runtime comparison with baselines. \n\n**A4**. Thanks for your suggestions. The table below provides a direct runtime comparison for generating perturbations across all competing methods in terms of hours. The generation process was conducted on a single RTX 4090 using the CelebA dataset (with ViT as the backbone of the surrogate MTL model) and the NYUv2 dataset. Notably, LSP and AR, which use predefined patterns, require almost no time for perturbation generation. The results indicate that our MTL-UE has a comparable computational cost to the base UE methods and is sometimes more efficient.\n\n|Method $\\downarrow$|CelebA dataset $\\downarrow$|NYUv2 dataset $\\downarrow$|\n|-|-|-|\n|LSP|Almost 0|-|\n|AR|Almost 0|-|\n|EM|7.35|3.43|\n|TAP|4.81|3.36|\n|SEP|25.01|26.14|\n|MTL-UE-EM|8.55|3.75|\n|MTL-UE-TAP|4.55|2.90|\n|MTL-UE-SEP|22.55|24.87|", "claims_and_evidence": "Overall, the claims in the paper are well-supported by extensive empirical evidence, including evaluations across multiple datasets, model architectures, and task-weighting strategies. \n\nThe authors make several key claims, such as (1) MTL-UE improves attack effectiveness for MTL models, (2) it reduces intra-class variance, (3) it is robust across different datasets and architectures, and (4) it can generalize well to dense prediction tasks. \n\nThese claims are backed by thorough experiments on four MTL datasets (CelebA, ChestX-ray14, UTKFace, NYUv2), comparisons against five baseline UE methods, and experiments with five different MTL task-weighting strategies.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria align well with generating unlearnable examples (UEs) for multi-task learning (MTL). \n\nThe evaluation spans four diverse MTL datasets (CelebA, ChestX-ray14, UTKFace, NYUv2), covering both classification and dense prediction tasks.  Additionally, multiple model backbones from CNNs to Transformers are evaluated to strengthens the claim of transferability. \n\nThis work also benchmarks several previous UE methods on multi-tasks learning.", "theoretical_claims": "Some theoretical justifications are provided in this work for MTL-UE. The embedding regularization techniques (Intra-ER and Inter-ER) are based on the idea that reducing intra-class variance and increasing inter-class separation enhances perturbation effectiveness. While no formal proofs are given, empirical evidence supports this claim.\n\nMTL-UE’s perturbation generation utilizes class-wise embeddings to constrain the search space, which is theoretically intuitive. While its strong empirical results support the approach, a more in-depth mathematical analysis of its impact on robustness across tasks could further reinforce the findings.", "experimental_designs_or_analyses": "The paper’s experimental design is well-structured for evaluating UEs in MTL. It includes four datasets (CelebA, ChestX-ray14, UTKFace, NYUv2), five task-weighting strategies, and five model architectures, ensuring broad applicability. Meanwhile, the comparison with multiple baselines (LSP, AR, EM, TAP, SEP) provides a fair assessment of MTL-UE. \n\nWhile the impact of task numbers is analyzed (Figure 5), a deeper exploration of why UEs perform better in MTL than STL would add clarity. Additionally, a direct runtime comparison with baselines would better support claims of computational efficiency.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\n1.The paper is well-structured and easy to follow, with clear notations and a well-explained methodology.\n\n2.The proposed method is well-justified, with clear relevance to real-world scenarios. This work is interesting and stands out as the first to explore unlearnable examples in the context of multi-task learning.\n\n3.This work initially built a benchmark and identified key weaknesses in existing STL UE methods when applied to MTL. \n\n4.The experimental results are comprehensive, demonstrating effectiveness across multiple datasets and advanced application scenarios.\n\nWeakness:\n\n1.As mentioned in the paper, the performance of MTL-UE on STL could be further enhanced when the number of tasks is very large.  It would be helpful if this paper can give more analysis on the influence of the number of tasks on the UEs performance.", "other_comments_or_suggestions": "1.The paper mentions that MTL-UE-TAP and MTL-UE-SEP exhibit lower transferability to models with ViT-B as the backbone because they use ResNet-18 as the surrogate model, resulting in a significant gap between the architectures. How would these methods perform if ViT-B were used as the backbone for the surrogate models?\n\n2.Among the 40 tasks in CelebA, are there specific tasks that are significantly easier or harder to degrade?", "questions_for_authors": "Please refer to the comments and suggestions above. Conducting additional experiments is recommended to further validate the findings.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8ryWAvm5g", "reviewer": "Reviewer_Aqty", "summary": "This paper introduces MTL-UE, the first unified framework for creating unlearnable examples tailored for multi-task data and models. By leveraging a generator-based structure with label priors and class-wise embeddings, MTL-UE enhances attack robustness through intra-task and inter-task regularization. It supports dense prediction tasks and integrates seamlessly with existing unlearnable methods, demonstrating superior performance across diverse datasets, models, and task-weighting strategies.\n\n## update after rebuttal \nThanks for the author's reply. All my concerns have been addressed, so I still recommend that this paper be accepted.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Scenarios involving fine-tuning (using a pretrained feature encoder) and advanced augmentations.\n\n**A1**. Thanks for the suggestions. We conducted experiments on the proposed scenarios. The table presents results of training MTL models on UTKFace with ImageNet-pretrained-encoder fine-tuning or UEraser [1] as advanced augmentations. Fine-tuning showed no impact on any method, while strong augmentations affected all, with our attacks still outperforming baselines.\n\n|            | None      | Finetuning | UEraser   |\n| ---------- | --------- | ---------- | --------- |\n| Clean      | 78.97     | 79.10      | 79.84     |\n| LSP (P)    | 28.58     | 27.37      | 69.31     |\n| AR (P)     | 27.41     | 23.64      | 54.23     |\n| LSP (A)    | 28.16     | 33.99      | 79.21     |\n| AR (A)     | 36.79     | 36.98      | 74.96     |\n| EM         | 31.74     | 34.03      | 73.19     |\n| TAP        | 39.29     | 40.21      | 61.80     |\n| SEP        | 40.70     | 40.87      | 61.29     |\n| MTL-UE-EM  | 25.84     | 30.37      | 52.50     |\n| MTL-UE-TAP | **19.84** | 21.08      | **32.36** |\n| MTL-UE-SEP | 21.19     | **20.18**  | 38.47     |\n\n\n**Q2**. Line-by-line description of Algorithm 1.\n\n**A2**. Thanks for your suggestions! We’ll add a detailed description in the updated version.\n\n**Q3**. Examples of partial task protection, where only one task is protected.\n\n**A3**. Following Table 8, we show single-task protection results. We can see that for STL models, unprotected tasks match clean data accuracy, while protected tasks perform like those trained to protect all tasks. Results on MTL models degraded performance, likely due to shared encoder learning both benign and spurious features.\n\n| Model $\\rightarrow$                               | MTL                 | STL                 |\n| ------------------------------------------------ | ------------------- | ------------------- |\n| Protected task $\\downarrow$; Metric $\\rightarrow$ | Age, Race, Gender   | Age, Race, Gender   |\n| None                                             | 60.32, 84.07, 92.51 | 60.46, 84.45, 91.86 |\n| Age                                              | 20.84, 80.51, 89.60 | 13.90, 82.78, 90.30 |\n| Race                                             | 55.70, 31.35, 91.35 | 57.47, 17.45, 90.57 |\n| Gender                                           | 56.31, 81.14, 60.59 | 57.28, 82.51, 50.38 |\n| All                                              | 7.28, 16.20, 40.08  | 7.26, 21.84, 55.61  |\n\n**Q4**. Tables 11 and 12 are missing references in Sections B.2 and B.3, which should be addressed.\n\n**A4**. Thanks for your suggestions. We will fix it in the updated version.\n\n\n\n[1] Learning the unlearnable: Adversarial augmentations suppress unlearnable example attacks. ICCVW 2023", "claims_and_evidence": "The paper presents MTL-UE as an effective framework for generating unlearnable examples in multi-task learning (MTL), and its claims are well-supported by extensive experiments. The results across multiple datasets, architectures, and task-weighting strategies demonstrate that MTL-UE consistently outperforms existing unlearnable example methods and that its embedding-based perturbation strategy effectively reduces intra-class variance, strengthening its core contributions. \nWhile the evidence is strong, a few areas could be further clarified. For example, the paper highlights MTL-UE’s plug-and-play compatibility with existing UE methods, but additional discussion on potential implementation challenges would be helpful.\nOverall, I find it sufficiently clear; even though I encountered some concerns while reading, I later discovered that the authors addressed and discussed these issues in the subsequent text.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-suited for protecting MTL data from unauthorized use. \nThe evaluation is rigorous, covering various task complexities with datasets for binary classification (CelebA, ChestX-ray14) and multi-class/dense prediction tasks (UTKFace, NYUv2). \nThe inclusion of multiple MTL task-weighting strategies strengthens the study, as different weightings impact task interactions. \nThe paper also assesses transferability across architectures, though results show slightly weaker performance with ViTs, warranting further exploration.", "theoretical_claims": "The paper leans on intuitive justifications rather than formal theoretical claims. The key argument is that embedding regularization (Intra-ER and Inter-ER) enhances perturbation effectiveness by controlling feature space alignment. This is reasonable and supported mainly by empirical evidence rather than formal proofs.\nA theoretical perspective on why MTL-UE improves attack transferability across tasks and architectures could strengthen the work. The generator-based approach is argued to enable stronger feature-level control, but a formal analysis would reinforce this claim. Additionally, bounding the perturbation space with embeddings is a novel choice, and further discussion on its impact on optimization dynamics could add depth.", "experimental_designs_or_analyses": "The experimental setup is thorough, evaluating MTL-UE across datasets, architectures, and task configurations. The inclusion of both MTL and STL models ensures a comprehensive assessment, and the choice of task-weighting strategies is relevant for understanding shared representations.\nThe ablation study on intra-task and inter-task embedding regularization clearly clarifies their contributions. However, further analysis of how perturbations affect different task types within the same dataset (e.g., classification vs. regression in NYUv2) would add insight. This paper also includes results for smaller perturbations, which enhance imperceptibility while maintaining the unlearnability.", "relation_to_broader_scientific_literature": "This paper primarily focuses on benchmarking datasets such as CIFAR-10. While the methods presented could be applied to other scientific problems, their applicability would require further exploration and discussion.", "essential_references_not_discussed": "I believe that the significant references have already been encompassed by the authors within the discussion.", "other_strengths_and_weaknesses": "Strengths:\nThis paper provides extensive analysis and explores additional scenarios. Beyond the main results, it includes feature visualizations, with Fig. 7 illustrating how the embedding regularizations enhance feature separation. Additionally, the paper evaluates its robustness against SOTA defenses. \nThe concept of partial data and partial task protection is particularly intriguing, as it enhances the method's practicality and applicability in real-world scenarios. \nThis work covers classification and dense prediction tasks across multiple datasets.\nWeakness:\nAdditional discussion on scenarios involving fine-tuning (using a pretrained feature encoder) and advanced augmentations would be valuable.\nA line-by-line description of Algorithm 1 would provide readers with a better understanding.", "other_comments_or_suggestions": "Including more examples of partial task protection, where only one task needs to be protected, would be beneficial. It would also be insightful to explore how MTL performs when the number of tasks to be protected is fewer than the unprotected tasks. Overall, I think this manuscript is good and fully deserves to be accepted by the ICML community.\nTables 11 and 12 are missing references in Sections B.2 and B.3, which should be addressed.", "questions_for_authors": "See weakness and comments as above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers have provided positive scores for this submission, highlighting its strengths in technical contribution, experiments, and workload. Given the unanimous positive feedback and the recognition of its contribution to the area, the AC carefully reviewed the paper and concurred with the reviewers' assessments, therefore supporting the decision to accept this submission.
Decision: Accept (poster)