ArchLock: Locking DNN Transferability at the Architecture Level with a Zero-Cost Binary Predictor
Authors: Tong Zhou
Shaolei Ren
Xiaolin Xu
Venue: ICLR 2024 poster (2024)
Keywords: Defense; DNN Transferability; Neural Architecture Search
Primary area: societal considerations including fairness, safety, privacy

Abstract: Deep neural network (DNN) models, despite their impressive performance, are vulnerable to exploitation by attackers who attempt to transfer them to other tasks for their own benefit. Current defense strategies mainly address this vulnerability at the model parameter level, leaving the potential of architectural-level defense largely unexplored. This paper, for the first time, addresses the issue of model protection by reducing transferability at the architecture level. Specifically, we present a novel neural architecture search (NAS)-enabled algorithm that employs zero-cost proxies and evolutionary search, to explore model architectures with low transferability. Our method, namely ArchLock, aims to achieve high performance on the source task, while degrading the performance on potential target tasks, i.e., locking the transferability of a DNN model. To achieve efficient cross-task search without accurately knowing the training data owned by the attackers, we utilize zero-cost proxies to speed up architecture evaluation and simulate potential target task embeddings to assist cross-task search with a binary performance predictor. Extensive experiments on NAS-Bench-201 and TransNAS-Bench-101 demonstrate that ArchLock reduces transferability by up to 30% and 50%, respectively, with negligible performance degradation on source tasks (<2%). The code is available at https://github.com/Tongzhou0101/ArchLock.

Published as a conference paper at ICLR 2024
ARCHLOCK: LOCKING DNN TRANSFERABILITY AT THE ARCHITECTURE LEVEL WITH A ZERO-COST BI- NARY PREDICTOR
Tong Zhou', Shaolei Ren", Xiaolin Xu Northeastern University, Boston, MA, USA UC Riverside, Riverside, CA, USA (zhou. (zhov.tongl, x. x.xu] (ruu)nnorteeeseemm hee.teeeeeeeereedeeeeee shaolei wewlleiuurrreeee
ABSTRACT
Deep neural network (DNN) models, despite their impressive performance, are vulnerable to exploitation by attackers who attempt to transfer them to other tasks for their own benefit. Current defense strategies mainly address this vulnerability at the model parameter level, leaving the potential of architectural-leve defense largely unexplored. This paper, for the first time, addresses the issue of model protection by reducing transferability at the architecture level. Specifically, we present a novel neural architecture search (enaleen algorithm that employs zero-cost proxies and evolutionary search, to explore model architectures with low transferability. Our method, namely ArchLock, aims achieve high performance on the source task, while degrading the performance on potential target tasks, i.e., locking the transferability of DNN model. To achieve efficient cross-task search without accurately knowing the training data owned by the attackers, we utilize zero-cost proxies to speed up architecture evaluation and simulate potential target task embeddings to assist cross-task search with binary performance predictor. Extensive experiments on NAS- MMS-Bench-201 and Trans LasS--Becc-lle 101 demonstrate that ArchLock reduces transferability by up 30% and 50%, respectively, with negligible performance degradation on source tasks (<2%). The code is available at
INTRODUCTION
is a common practice today to transfer a pre-trained deep neural network (DNN) from one application domain to another one, demonstrated in Guo et al. (019); Jiang et al. (022) These advantages, however, also provide an incentive for attackers to illicitly exploit wean-trained models by transferring them to their desired tasks, resulting in violation of model owners' rights. To mitigate the vulnerability of unauthorized model transfer, prior work restricts model usage by introducing training strategy optimize the weights Wang al. (022). Nonetheless, such weight-level modification cannot be preserved once attackers fine-tune this model to the domain of interest with enough training data, since they will further adjust the model weights to improve the transferability.
In contrast, we focus on achieving protection at the architecture level, recognizing that the neural architecture fundamentally determines the model accuracy Zoph & Le (2017) and hence plays critical role in model transferability Kornblith et al. (2019); Zhou et al. 2021). Despite the critical significance of architectural considerations in model transferability, previous mitigation efforts have largely overlooked this aspect. Therefore, our work seeks to bridge this gap by addressing the question: how to mitigate the risk of unauthorized transfer by reducing transferability at the architecture level? In particular, showing limited transferability to other tasks.
expect the DNN to perform well the source task while
Although our proposed defense philosophy is intuitive, manually designing an architecture with low transferability is challenging, due to the large design space and requirements for domain knowledge of architecture characteristics associated with transferability. Consequently, neural network search (NAS) is envisioned as a promising solution due to its effectiveness in architecture design Liu et al. 2018b); Real et al. (2019) Zhou al. 2022). Even so, how to utilize NAS reduce DNN

---

Published as a conference paper at ICLR 2024
transferability for model protection has remained open problem. Indeed, it is significantly more challenging than improving DNN transferability Ding et al. (2022); Pasunuru & Bansal (2019), since the target task for the latter is well-specified but not for case. Therefore, in addition to the common challenges faced by existing NAS algorithms, i.e., how to efficiently evaluate the performance of architecture in the search space, have address additional challenges, including (a) how to determine and simulate the potential target task if it is not specified; (b) how to address performance evaluation for the simulated task; and (c) how to preserve the performance on the source task but degrade performance on the target task.
In this paper, we formulate the transferability reduction problem as a cross-task search NAS problem and propose a novel algorithm, called ArchLock. Specifically. for challenge (a), we simulate the target task by generating its task embedding based on the source task with a specified similarity, since the target task should be similar to the source task, so to benefit (i.e., obtain the performance gain) from transfer learning (Sec. 3.1). For challenge (b), we meta-train a binary predictor with zero-cost proxies to indicate the relative performance, which can achieve efficient architecture evaluation and generalize to unseen tasks (Sec. 3.2) Last, we design a cross-task search algorithm using evolutionary search with rank-based fitness score to guide the search (Sec. 3.3). Overall, our contributions can be summarized as follows:
This work is the first to mitigate unauthorized transfer at the architecture level, where the searched architecture excels on the source task but exhibits degraded performance on the target task, regardless of the amount of data available the attacker.
We develop a binary predictor using multiple zero-cost proxies to accelerate NAS. The predictor incorporates task characteristics an additional input, enabling efficient architecture evaluation and generalization to unseen tasks, thus assisting in cross-task search with a rank-based fitness score.
Through extensive experiments NAS-Bench-20 and TransNAS- 101, we demonstrate that ArchLock can reduce transfcrability by up to about 30% and 50%, respectively, while maintaining negligible performance degradation on the source task (<2%).
2
RELATED WORK
2.1 EFFECT OF ARCHITECTURE ON MODEL. TRAANSFERABILITY
Previous works have demonstrated that the architecture design affects the model transferability Zhou et al.) r): Korbblith et al. (2019). For example, Zhou et al. demonstrate that the Transformer-based architectures can be better transferred to 13 tasks than ConvNets Zhou al, (002T) Besides, Zoph et al. search well-performed cells only on CIFAR-10 and transfer them to ImageNet with good performance. However, Dong & Yang 2020) shows that directly transferring an optimal architecture from a source task to a target task might not always yield good performance, since the rank correlation of architecture performances on different tasks is not perfectly positive. This work suggests that single-task search cannot ensure that the top-1 architecture performance on the target task. Therefore, some cross-task search NAS algorithms are proposed. For example, DDng et al. (2022) builds predictor for each task, then accumulates the gradient from these predictors to guide the architecture search. Besides, Pasunuru & Bansal (009)) utilizes a controller to obtain a joint reward to achieve multi-task search.
the source task also has the best
However, these existing cross-task NAS algorithms aim to find the architecture performing well on several specified tasks. In sharp contrast, our proposed design only expects generating model architectures that perform well on the source task with reduced transferability to uncertain target tasks, which raises challenges including achieving efficient architecture performance evaluation and estimating performance on unseen tasks.
2.2 ACCELERATION FOR ARCHITECTURE EVALUATION
Evaluating an architecture performance by training it until convergence is computationally expensive. To overcome this limitation, some works design the performance predictors to accelerate NAS, which are usually built as regression models that are trained using architecture encodings and their corresponding validation performances (e.g., accuracy for classification problems) Liu et al. (2018a):
2

---

Published as a conference paper at ICLR 2024
Shi et al. (2020). However, collecting these training data is also expensive and the predictor may not have a high correlation between the predicted result with the actual performance White et al. (202T) As an improvement, Dudziak et al. propose a predictor using the binary relationship to rank architectures, which requires fewer training samples but has a higher correlation Dudziak et al. (2020).
In contrast to the above methods that still require training architectures, several theoretical metrics have been proposed recently that can measure the performance of an architecture the initialization stage|ees Lee et al. 20]19);; Mellor et al. (202T): Lin et al. (20TT) thus greatly reducing the computation burden NAS. Since the computation cost of such metric negligible (e.g., less than seconds Krishnakumar et al. (2022)), they are used as zero-cost proxies (ZCPs) in previous NAS works Chen al.] However, single ZCP cannot provide accurate rank or generalize well among various tasks Chen et al., Afdelfattah et al. (2011). Therefore, KKrihhakumarr et al. (2022) incorporates 13 proxies into the surrogate models used by NAS algorithms, showing up to 42% performance improvement.
However, we leverage ZCPs in different way, with details in Sec. 3.2
2.3 NAS PREDICTORS WITH META-LEARNING
Given that explicitly measuring the performance of architectures on simulated target tasks is impossi- ble, we draw inspiration from the following works to leverage meta-lcarning in training our proposed predictor. In particular, recent NAS algorithms adopt meta-learning improve its generalizability on other novel tasks Elsken et al. )0 Lee et al. (2021alb) b). For example, MetaNAS optimizes both the aeetterreite and the meta- weights during meta- trainingn allowing adaptation unseen tasks during testing [lsken et al. (2020). Besides, Lee et al. propose MetaD2A to meta-train train performance predictor on the ImageNet classification task to accelerate the search Lee et al. (202Ta), which generalizes well other classification tasks and reduces the search cost from O(N) to O(I) for multi-task search. Therefore, we also adopt meta-lcarning in our predictor design, thus being able estimate the architecture performance simulated tasks. Moreover, extend the applicability of the meta-trained predictor beyond the classification task.
3 OUR PROPOSED FRAMEWORK: ARCHLOCK
Our goal to utilize NAS find an architecture that can mitigate unauthorized transferability. Specifically, except for achieving the optimal performance on the single task as conventional NAS, we expect the searched architecture can restrain the model usage to the source task, i.e., perform well the source task, but hard to transfer to the target task with good performance. To this end, we propose ArchLock, which learns the cross-task performance architectures via an evolutionary search. Specifically, ArchLock uses rank-based fitness score guide the search. Further, introduce a binary predictor using zero-cost proxies to rank the architectures ArchLock, where the pairwise relationship will be taskedependent, thus evaluate the architecture performance on uncertain tasks.
simulate the potential task embedding to
Problem Formulation: Conventional NAS algorithms focus on finding an optimal architecture that achieves high performance on a single task (denoted as S), while satisfying the hardware constraints, e.g., memory consumption and/or inference latency Zoph & Le (201)); Chen et al.) which can be formulated as bi-level optimization problem:
a' arg min Ls (Xa W=": s.t. W = arg min Cs(Fa W):Y aeA
(I)
w
where (Xg, Y and (Xgal, yeal) are the training data and validation data of S, respectively, and Ls is the loss function of S. We use A denote the search space and fa to represent the network associated with the architecture a, where a is an architecture candidate in and W is the corresponding weights trained for S.
(X"
However, in order to mitigate the vulnerability of unauthorized transfer, we have to consider the architecture performance on potential target tasks, which is denoted T.. Similarly, this problem can be formulated as a bi-level optimization problem, but we aim to minimize the performance of the network T, while maximizing its performance S, i.e., the specific source task. The optimal
3

---

Published as a conference paper at ICLR 2024
architecture a" can be found by solving:
a = arg min (S( W"):Ys - Σ CT. (Fa(,, W7,); y7
aEA
T
(2)
s.t. W arg min Cn W)
where (XT;, YT,) is the labeled pair from the task and constraints Eq. ① still apply. Also, B controls the balance between performance maximization on the source task and minimization on the target task. Especially, when equals 0, Eq. (2) will be degraded Eq. ① thus we can get the performance of NAS without the security concern of transferability reduction as the baseline.
However, solving Eq. (2 faces three challenging problems: 1) how to simulate the task domain it is unknown; 2) how to efficiently measure the architecture performance and let it generalize to unseen tasks as well, and 3) how to achieve cross-task search. We tackle these challenges Sec. 3.1 Sec. 3.2 and Sec. 3.3 respectively.
3.1 TASK EMBEDDING
is worth noting that fully protecting the model by reducing the model transferability to all potential target tasks is practically infeasible, i... one could not pre-know the tasks or applications of interest to the adversary, which also not our target in this work. This is also true for other types of defenses, e.g., adversarial training can only robustify the DNN model subject to an input perturbation bound Wang et al. (2019). Here, we expect the transfcrability to some tasks can degrade so that we can defend against partial potential attacks.
In general, simulating the potential target tasks by generating their actual dataset can be difficult not impossible, since need to know exactly the attacker's tasks. To address this challenge, aim to manipulate the feature space directly to save the computational cost of dataset generalization, i.e, generate the possible target task embedding. The bright side is that the target task in general is close to the source task in order to benefit from transfer learning. In this case, we can simulate some potential target tasks during the search to assist cross-task search when the target task is unknown.
To capture the characteristics of a task as an embedding vector, we leverage the Fisher Information matrix (FIM) similar Achille et al. (019) and Huang et al. (0022) which can be expressed by:
F Ex,Y
V log Pmu V l.g (Y
where is the empirical distribution defined by the dataset including the input X and the label Y. Specifically, the modern DNN model can be seen as two parts: a feature extractor and a task- dependent head, e-g., classifier for the classification task. Here, ImageNet as the feature extractor and train the head for any given task. After training, we compute the FIM for the feature extractor with the same approximation in Achille et al. (2019) represent the task as fixed dimensional vector z.
ResNet-50 pre-trained
With the source task embedding zs obtained using the above method, we can simulate the task embedding zy, of potential target tasks T. Here we use the cosine similarity for measurement, where the value is between [0.1] and the larger value indicates higher similarity, Therefore, given 2ş and desired cosine similarity d, we can generate ZT, following:
2T: = d zs + sin (arcosssss())) 6 ||z|||
where e; orthogonal Zs and can be obtained via Gram-Schmide process. The derivation of Eq. (8) is shown in Appendix A Besides, the orthogonal vector will vary for different ZT,-
The advantage of using such a method is that we can simulate possible target tasks without explicitly generating the samples, and the latent space manipulation is more representative. With the simulated task embedding, we expect that they can represent or approach some real target tasks. Since we do not have the ground truth label for the simulated task. cannot estimate the architecture performance using supervised learning. Instead, we use the task embedding as an input of the predictor introduced below, where the predictor will be meta-trained be generalized to the unseen task embedding.
4

---

Published as a conference paper at ICLR 2024
3.2 ZERO-COST BINARY PREDICTOR
One challenge we face in developing ArchLock is estimating the architecture's performance on unseen tasks. Inspired by the success of meta-learning in generalizing models unseen tasks Finn et al. (2017); Lee et al. (2021a), we propose to train a predictor that can adapt to simulated tasks following the procedure in MetaD2A Lee et al. (2021a). However, unlike the predictor MctaD2A that uses validation performance as the label for meta-training, we use zero-cost metrics as proxies to measure architecture performance at initialization, thus avoiding architecture training and saving computational costs.
While various zero-cost proxies have been used in NAS to accelerate the search process, relying on single proxy as a measure of architecture's performance is unreliable White et al. (202T). The searched architecture may not perform well, and the zero-cost proxy (ZCP) itself may not generalize to other tasks. To address this, incorporate multiple data-dependent ZCPs to obtain a more accurate estimation, including fisher grad-norm, grasp, jacov, nwot, and snip, as summarized in Krishnakumar et al. (002)). These proxies provide complementary information about the characteristics of an architecture 's performance. The details of these proxies and the rationale for their selection are discussed in Appendix [
Our objective is to obtain architecture rankings rather than precise performance values. Therefore, propose binary predictor P that learns pairwise relations between two architectures instead of predicting ordinal regression. Such a design can improve the overall ranking accuracy of the predictor since learning relative performance for two architectures will be easier than predicting accurate rankings for multiple architectures, More importantly, we incorporate task characteristics into the predictor using task embedding, so can be used in the cross-task search to efficiently measure the architecture performance given different task embeddings.
Overall, the inference process of the proposed predic- tor is shown in Fig. [ The graph encoder converts the directed acyelic graph of the architecture cell into an adjacency matrix, which is then flattened into a vector. As for the binary predictor, it is built with a multi-layer perceptron with softmax activation func- tion and takes the concatenated architecture encoding pair and the pre-computed task embedding inputs. During training, the ground-truth label is calculated based on the ZCPs proxies using an ensemble method. Specifically, the label is set to 1 if more proxies indi- cate that the first architecture outperforms the second in an architecture pair. The design and training de- tails are described in Sec. 4.3) During inference, the predictor outputs two logits Pl and P2. If P1 > P2, it indicates architecture a outperforms 02 on the task associated with task embedding z. In summary, the proposed predictor has the following properties: (1) It is scalable to any size of the architecture pool when combined with the sorting algorithm; (2) It uses zero-cost proxies as supervision, eliminating the need to train several architectures from scratch and reducing training costs; (3) It incorporates multiple 7CPs obtain complementary information, mitigating the bias introduced by single proxy; (4) It takes both architecture and task embeddings into account, leveraging meta-learning for training and enabling generalization to unseen tasks.
Graph Encoder
Task Embedding
Arch.
Binary Predictor
Arch, Fmbeeding
Graph Encoder
(Pu
Concatenation
Arch. R2
Figure 1: The inference process of our pro- posed predictor, where the inputs include the embedding of pair of architectures along with the pre-computed task embedding. Pi P2 indicates that d₁ outperforms 02 for the given task.
3.3 CROSS-TASK EVOLUTIONARY SEARCH
In ArchLock. we utilize cvolutionary search due to its computational efficiency compared to RL-based NAS Zoph & Le (0017). Also, unlike pruning-based basedg approaches Chen et al.) its mutation mechanism can help avoid local minima, allowing for more thorough exploration of the solution space.
The core algorithm of cross-task search in ArchLock is shown in Alg. To achieve the transferability reduction via a cross-task search, we use the meta-trained predictor along with a sorting algorithm, ie., topologiaal sorting in this case, to rank the performance of architectures for S and T,, denoted as BPr in Alg. In particular, we incorporate the rankings for all T, as one using majority voting,
5

---

Published as a conference paper at ICLR 2024
denoted as Trank- We then use the rankings of architecture S and T to design a fitness score as guidance for the cross-tast search. The rank-based fitness score is designed as:
(5) where Srank denotes the architecture rankings S and set À 2 to assign the source task higher weight. In this case, we are able to find an architecture that satisfies Eq. i.e., the performance will be degraded on the target task while being well on the source task compared the conventional NAS. The functions BPwSort and UpdateHistooo are shown in Appendix
f Trank À Srank-
EXPERIMENTS
We evaluate ArchLock on two state-of-the-mne NAS benchmarks: NAS- Bench-201 (NB-201) Dong & Yang (2220) and TransNAS MaausNAS-Bencc--- 101 (TNB-101) Duan et al. (202T). These are the only two accessible options that consist of multiple tasks. Performance evaluation based on the validation performance provided by the benchmarks for fair comparison. We compare three search schemes: one is source-only search and the other two are cross-task search, with details described Sec. 4.2
4.1 SEARCH SPACE AND TASKS
NAS-Bench-201 is popular NAS benchmark dataset with cell-based search space. Each cell has nodes and edges, where each edge represents an operation selected from 5 options, i.e., zeroize, skip connection, x convolu- tion, 3 convolution, and 3 average pooling. It includes over 15,000 neu- ral network architectures across three image classification tasks, including CIFAR-10, CIFAR-100 100, and ImageNet- 16-120 (i.e., 16x 16 input size with 120 classes).
Algorithm 1 Cross-task Search in ArchLock
Require: Predictor Task embeddings zs, ZT; Parameters: Population size n, Sample size m, History size k
1: popa <- 0, history <
2: procedure NEWEPOCH(pooch, P,zs, (T:)
3: if' epoch < n then
arch + Randomly sample from search space
5:
popu.append(arch) UpdateHissoryhhsstoryy arch, k. P.zs, else
6:
7:
Sranks Trank + BPwSort(popu, P. zs ZT.) Calculate f for arch E popu (Fq.55) samples + Randomly sample m from popu
9:
TransNAAS-Bnch-Ill is proposed to facilitate the cross-task NAS by includ- ing the architecture performance across
10:
11:
parent +- arch with highest f in samples child +- Mutate(parent)
12:
different tasks on the same dataset, 13:
popu.popleft()
including scene classification (SC), ob- 14:
popu.append(chiid))
ject classification (OC), room layout 15:
Updaaetiisorryhisstorr, child, k. P, zs, 2T.) end if
(RL), jigsaw (JS), semantic segmenta- tion (SS), surface normal (SN), and au- toencoding(ABB where the labels for each task collected from Taskonomy Za- mir et al. (2018). provides two types of search space, i.e., macro- masro-based and cell-based, where we experiment on the latter with more than 4k architectures. Each cell consists of 4 nodes and 6 edges, where the operations for each edge include zeroize, skip connection, 1xl convolution, and xx3 convolution.
16:
17:
end procedure
18: a* + arch with highest f in history
4.2 DIFFERENT SEARCH SCHEMES
ArchLock-S: This is a source-only search scheme, which only conducts ArchLock on the source task S, i..., setting I in Eq. (2) to O. After getting the architecture, will evaluate how it performs when trained on the target task. It is baseline as the comparison with cross-task search schemes.
that the target task is unknown (TU). We use simulated tasks with embedding generated based on the method introduced Sec. We measure the performance of the searched architecture the target task, which is unseen during the search.
Tr: It conducts cross-task NAS in the
ArchLock- -K:: conducts cross-tate task NAS in the case that the target task known (TK), where the objective follows Eq. with only one T obtained from the certain target task.

---

Published as a conference paper at ICLR 2024
Table 1: The transferability of searched architecture on NB-201 with different search schemes. We report both the 1 validation accuracy, i.e. Acc. (%), and rank percentile, i.e., Pet. (%) to evaluate the performance of architectures. We use APT to denote the average rank percentile on target tasks.
CIFAR-10
CIFAR-100
ImugeNet- 16-12-
Source / Target
Method
APT
Acc.
Pet.
Acc.
Pct.
Acc.
Pct.
ArchLock-S
91.40
99.99
72.96
99.95
46.77
99.94
99.95
CIFAR-10
Ack-UTT
90.41
97.93
70.24
95.81
44.11
95,44
95.62
ArchLock-TK
90.55
98.56
68.67
82.78
41.80
84.07
83.37
ArchL.ck-SS
91.18
99,91
73.25
99, 99
46.5
99.84
99.88
CIFAR-100
ArchLock-TU
89.69
93.45
71.18
41.47
82.11
87.78
ArchLock-TK
89.07
84.47
71.23
99.07
37.27
54.84
68.65
ArchLock-S
91.23
99,94
72.74
99.92
47.27
99.99
99.93
ImageNet-166 16- 120
Archl ock-TU
89.53
91,95
69.16
87.60
45.22
98.52
89.77
ArchLock-TK
88.73
76.59
67.69
72.72
45.58
99.25
74.65
4.3 EXPERIMENTAL SETUP
Meta-trained predictor: we build a binary predictor using a multi-layer perceptron percepptoonno with four layers. To train the predictor, we use ImageNet to build our training as Lee et al. al. (2021a). Specifically, the training set includes architecture pairs randomly sampled from Nas-Bench-2011 the task embedding of the tasks associated with the sub-set of ImageNet, and the binary label decided by the values of ZPCs. Specifically, if the majority of ZPCs indicate the first architecture outperforms the second, then the label 1; otherwise, the label would be O. Following the same meta-traie metattraining procedure as Lee al. (2)21a), our predictor can generalize to unseen tasks, where the task similarity will affect the performance of the meta-trained predictor as discussed Sec. [.2 Our experiments are conducted with the PyTorch framework using NASLib library Ruchte et al. (2020).
Simulated task embeddings: For the following experiments, we simulate 10 task embeddings T, where the cosine similarity of S and T; is set to 0.9, These settings will be the same unless specified otherwise, and we will discuss how they affect the search if the values are changed in Sec. 5.2
Measurement: For classification tasks, we use top-1 validation accuracy as a metric to measure the performance. Besides, we also use rank percentile, denoted as Pet., to measure how a candidate performs among all architectures in the search space, since the architecture ranking matters in NAS. For example, Pct. of an architecture is 90%, it means that this architecture performs better than 90% of all architectures in the search space. In order measure the performance of three search schemes, we also introduce the average rank percentile on target tasks, denoted as APT, which takes the Pct. on all tasks except the one same the source task.
4.4 RESULTS ON NAS- MAS--ENCH-225 201
We apply three search schemes the NB-201 benchmark with 3 datasets and take the average of results from several runs for each experiment as reported in Tab. [ From this table, we can see that ArchLock-S can achieve high performance (with Pct. >99%) on target tasks, even though it only searches on the source task, which indicates the searched architectures in this case are more vulnerable to unauthorized transfer. The reason that the correlation on each task pair over architectures in NB-201 is high, with details shown in Appendix D
However, if we change the search scheme ArchLock-TU, the APT will have an obvious drop, with up to 12% when CIFAR-100 is the source task, while the Pet. on the source task does not drop much, i.e. < 2%. The rationale for employing Pet. as deseriptor of performance lies in the insufficiency of accuracy alone to fully represent the architectural performance across the entire search space. Specifically, since the architecture performance does not follow the uniform distribution in the search space, i.e., small accuracy drop will significantly change the architecture ranking, making the result no longer attractive to attackers. Moreover, ArchLock- can directly use the specified target task in cross cross-task search, achieving the maximal transferability reduction. Also, it can preserve the performance of the source task, which is close to the source-only search Src-S.c
shows the highest APT drop since we
7

---

Published as a conference paper at ICLR 2024
Table 2: The transferability of searched architecture on TNB-101 with different search schemes. We use AP7 to denote the average rank percentile on target tasks.
Source/ / Targot
Method
SC
OC
RL
J8
88
SN
AE
APT
SrchLock-S
99.99
99.83
59.91
89.71
75.88
99.32
95.97
86,77
ArchLack-TU
98.
59.17
75.89
ArchLock- TK
99.73
39.48
34.03
40.43
46.67
51.49
30.19
40.88
ArchLack-S
89.21
99.93
73.61
63.87
69.92
82. 82.10
26.76
67.58
OC
Archl oUk-TU
76.94
10
73.77
41,41
63.10
62.33
63.2
63.46
ArchLock-TT
47.96
59.91
38.95
42.04
35.24
26.76
SrchLack-S
98.47
94.85
99.78
83.26
74.34
99.85
77.02
87.97
RL
Archl atc-TTU
49.38
67.91
17.
60.55
46, 55
64. 02
49
56.34
Archl TK
37.76
40.56
99.75
39.49
32.08
45,46
22.99 25.95
36.38
ArchLock-SS
97.98
97.77
65. 65.04
99.68
87.82
97.58
72.31
86.42
18
ArchLock-11
67.51
61.06
76.90
9W.52
67.54
53.17
37. 37.38
610.99
Archl ack-TK
35.46
31.91
42.46
99.58
54.24
32.00
19.67
35,97
S
81.81
ArchLoc-TU
68.72
38.43
34.5 .99
47.48
98.
85.06
19.21
48.91
ArchLock-TK
40.84
30.91
30.07
30.42
99.
36
11.33
31.67
ArchLock.S
97.02
97.13
79.05
73.19
76.17
99.97
81.49
84.00
ArchLock-
67.95
ArchLock-T -TK
38.95
33.17
31.00
37.74
43:15
99.61
15.21
33.25
ArchLock-S
37.37
62.37
84.92
55.26
34,89
74.40
99.83
58.22
AE
ArchLock-TU TU
36.33
48.29
54.81
50.08
31.89
46.84
98.08
44.71
Tc
31.13
56.13
45.36
30.39
29.55
99.74
40.09
4.5 RESULTS ON NASS-NN
The performances of three search schemes TNB-101 are shown in Tab. where the architecture performance is reported as the average of multiple runs for each source-targe pair. From Tab. 2 can find that two cross-task search schemes, i.c., ArchLock-TU and ArchLock-TK, can still maintain good performance on the source task as the source-only search ArchLock-S,, while the performance the target tasks drops drastically. Besides, since the correlation of some task pairs over architectures in 101 is low (see Appendix EE, e.g.. architecture with good performance on OC could be bad on AE, the APT is relatively low for ArchLock-S compared it on NB-201. The intuition is that the target task could benefit more if the source task does the same kind of vision task.
56-
(C)
Rt
+0.5E
20.15
PM S000mi (%)
18
21.16
44:44
SS
K140
+1-42
8
91/0
SN
07.00
$7.03
1500
7533
M:37
57
14.00
14.14
50.15
12.14
2335
32.17
31.00
+3
2222
AE
8400
31.2
41.5
5400
SC
OC
R2
15
55
SN
AE
SC
OC
R.L
15
5S
SN
AE
SC
OC
KL
18 Target
58
SN
AB
Target
Target
Figure 2: From left to right: ArchLock-S, ArchLock-TU, ArchLock-TK. The darker color indicates
higher transfcrability reduction.
Although the APT of ArchLock S is relatively low, can still enhance the transferability reduction via cross-task search, where the results in Fig. can show the performance difference among three search schemes. For example, when transferring SS SC, ArchL finds architecture with Pct. of 81.81 %,, while ArchLock- and ArchLock- reduce the transferability even lower with Pet. of 68.72% and 40.48%, respectively. Besides, each heatmap reflects the asymmetric characteristics of transferability, i.c., transferring one task to another is different from the otherwise (e.g., 65.04% % for IS to RL while 83.26% for RL to JS in ArchLock-S).
DISCUSSION
5.1 NAS ALGORITHMS
We compare ArchLock with SOTA source-only NAS algorithms on TNB-101, with SC denoting the source task and others representing the target tasks. The results are shown in Tab. 3 Note that even in this case the average Spearman rank correlation of all source-target task pairs just 0.7, our
8

---

Published as a conference paper at ICLR 2024
Table 3: Comparison among ArchLock and SOTA source-only NAS, including RS Bergstra Bengio (0012) REA Real et al. (2019), BONAS Shi et al. (2020) and weakNAS Wu et al. (2021). The results are reported using the percentile (%) of architecture performance in the TNB-100 101 search space.
Sourcc-only
Cross-task
RS
REA
BONAS
weakNAS
ArchLock-S
TrchLock-TU
ArchLock-
Source
SC
99.68
99.85
99.34
99.71
99,99
98.96
99.73
78.83
75.39
66.36
39.48
RT.
62.30
63.31
76.95
59.91
59.17
34.03
Target
78.78
84.72
76.95
67.34
40.43
88.72
76.90
84.79
72.61
75.88
75.89
46.67
97.00
77.34
86.79
96.14
99.32
67.47
54,49
AE 67.02
49.53
65.48
75.88
95.97
44,96
30.19
79.28
71.60
63.54
40.88
cross-task search schemes ArchLock-TU and ArchLock-TK can further reduce the APT to 63.54% and 40.88%, where APT for source-only NAS can be high as over 80%. It demonstrates the effectiveness of the proposed cross-task search in mitigating unauthorized transfer.
5.2 SIMULATED TARGET TASK EMBEDDING
The simulated target task embedding plays an important role in ArchLock-TU, We experiment on NB-201 with CIFAR-100 the source task and the other two target tasks for investigation.
The number embeddings. We experiment with different numbers of embeddings, i.e., 5, 10, and 15, respectively. From the results shown in Fig. 3 as the number of embeddings increases, i.c., the architecture performance does not change much on the source task but is degraded more the target task with more simulated target task embeddings involved. The reason could be that the cross-task search covers more possible target tasks, which increases the possibility of the actual target task being included.
notice the gain in transferability reduction
Narr
Nam 10
Num 15
100
Table 4: The performance of ArchLock-TU with various similarities simulated task embeddings. The results are reported in terms of the rank percentile (%) of the architecture performance in NB-201 search space.
Porentil dk)
80
40-
20
Similarity d
Source
Target
CIFAR 100
CIFAR
ImageNet 120
CIFAR-10
ImugeNd-166-1
0.3
99.07
94.34
97.56
Figure 3: TrchLock-TU with various
0.5
98.89
95.82
92.59
0.9
98.98
93.45
82.11
numbers of simulated embeddings.
The similarity of embeddings. We explore various cosine similarities between the source task embedding and the simulated target embedding, using d values of 0.3, 0.5, and 0.9 (see Eq. 8). In Table we find that ArchLock-TU TU exhibits higher transferability reduction as similarity increases from 0.3 to 0.9 (97.56% 82.11% for ImageNet-16- 120)).. This is likely because simulating low- similarity tasks provides less informative guidance, potentially leading to search towards less similar target tasks resulting in lower transferability reduction for the actual target task.
CONCLUSION
This paper presents ArchLock, a cross-task NAS framework aimed at mitigating unauthorized DNN transfer. Our framework focuses on finding an architecture that performs well on the source task but exhibits degraded performance when transferred to a similar target task. To achieve this objective efficiently, we introduce a binary predictor that utilizes 7 zero-cost proxies to save computational costs. Additionally, simulate the target task embeddings learning, this predictor can be generalized to unseen tasks, thereby assisting cross task search with a rank-based fitness score. Extensive experiments conducted on NAS-Bench- 101 demonstrate the superiority of our proposed cross-task search schemes, namely ArchLock- and ArchLock-TT in transferability reduction compared ofte-oofttheaad art source-only NAS methods.
input of the predictor. By leveraging meta-
and TransNAS-Best

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This work is supported in part by the U.S. National Science Foundation under Grants OAC-2319962, CNS- CNS-2239672, CNS-2153690, CNS-2326597, and CN4-2247892.
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight nas. International Conference on Learning Representations, 2021.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning, In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6430-6439, 2019.
James Bergstra and Yoshua Bengio. Random search for hyper-paramette optimization. Journal of machine learning research, 13(2), 2012.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations.
Mingyu Ding, Yuqi Huo, Haoyu Lu, Linjie Yang, Zhe Wang, Zhiwu Lu, Jingdong Wang, and Ping Luo. Learning versatile neural architectures by propagating network codes. International Conference Learning Representations, 2022.
Xuanyi Dong and Yi Yang. Nas-bench 20he:cch-20:::
Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR). 2020. URL https //openrevien net. www.foormmdddoom d do
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. Transnas-hench-101 Improving transferability and generalizability of cross-task neural archi- tecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5251-5260, 2021.
Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction- based nas using gens. Advances in Neural Information Processing Systems, 33:10 33:10480-10490, 2020.
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12365-12375, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. In International conference machine learning. pp. 1126-1155. PMLR, 2017.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 1885-4814 4814, 2019.
Minbin Huang, Zhijian Huang, Changlin Li, Xin Chen, Hang Xu, Zhenguo Li, and Xiaodan Liang. Arch-graph: Acyclic architecture relation predictor for task-transferable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11881-11891, 2022.
Junguang Jiang, Yang Shu, Jianmin Wang, and Mingsheng Long. Transferability in deep learning: A survey. arXiv preprint arXhv:2201.05867, 2022.
Simon Korbllith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2661-2671, 2019.
10

---

Published as a conference paper at ICLR 2024
Arjun Krishnakumar, Colin White, Arber Zela, Renbo Tu, Mahmoud Safari, and Frank Hutter. Nas- bench-suite-zer zero: Accelerating research on zero cost proxies. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In International Conference on Learning Representations, 2021a.
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. Hardware-adaptive efficient latency prediction for nas via meta-learning. Advances in Neural Information Processing Systems, 34:
27016-27028, 2021b.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019a.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019b.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 347-356, 2021.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: /806. 000.08000.00.... 2018b.
Vasco Lopes, Saeid Alirezazadeh, and Luís A Alexandre. Epe-nas: Efficient performance estimation without training for neural architecture search. In Artificial Neural Networks and Machine Learning- ICANN 2021: 30th International Conference on Artificial Neural Networks, Bratislava, Slovakia, September 14-17, 2021, Proceedings, Part V, pp. 552-563. Springer, 2021.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Xuefei Ning, Changcheng Tang, Wenshuo Li, Zixuan Zhou, Shuang Liang, Huazhong Yang, and Yu Wang. Evaluating efficient performance estimators neural architectures. Advances in Neural Information Processing Systems, 34: 12211222222771 2021.
Ramakanth Pasunuru and Mohit Bansal. Continual and multi-task architecture search. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 1911-1922, 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Michael Ruchte, Arber Zela, Julien Siems, Josif Grabocka, and Frank Hutter. Naslib: A modular and flexible neural architecture search library. /
com/ automl /ww.o/////iii..// 2020.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-basse and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 33: 13:1808-18 1819, 2020.
Jack Turner, Elliot Crowley, Michael O'Boyle, Amos Storkey, and Gavin Gray. Blockswap: Fisher- guided block substitution for network compression on a budget. 2020.
Chaoqi Wang, Guodong Zhang, and Roger Grosse Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Lixu Wang. Shichao Xu, Ruiqi Xu, Xiao Wang, and Qi Zhu. Non-transferable learning: A new approach for model ownership verification and applicability authorization. International Conference Learning Representations (ICLR), 2022.

---

Published as a conference paper at ICLR 2024
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning, pp.
PMLR, 2019.
Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34: 28454-28444 2021.
Junru Wu, Xiyang Dai, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Ye Yu, Zhangyang Wang. Zicheng Liu, Mei Chen, and Lu Yuan. Stronger nas with weaker predictors, Advances in Neural Information Processing Systems, 34:28904-28918 2021.
Amir R Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3712-3722, 2018.
Hong-Y Yu Zhou, Chixiang Lu, Sibei Yang, and Yizhou Yu. Convnets vs. transformers: Whose visual representations are more transferable? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2230-2238, 2021.
Tong Zhou, Shaolei Ren, and Xiaolin Xu. Obfunas: A neural architecture search-based dnn obfusca- tion approach. In Proceedings of the 41st IEEE/ACM International Conference on Computer- -ided Design, pp. 1-9, 2022.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. 5th International Conference on Learning Representations (ICLR), 2017.
12

---

Published as a conference paper at ICLR 2024
A DERIVATION IN TASK EMBEDDING
Given the source task embedding zs, we first generate vector e; that is orthogonal zs. which can be obtained via
process.
To satisfy the desired cosine similarity d, i.e.,
( (Zs, == cos(O) d.
(6)
where we use Scos to denote cosine similarity and is corresponding angle. Here we set the norm of the target task embedding the same Zg, will have:
ZT; = cos zs + sin(O) 6 |s||,
According to Eq. 66 we also have ar)cos(d). Therefore. Eq. 7 can be expressed as:
ZT, d zs + sin (arccos((())
||s|||.
(8)
B DETAILS OF ZERO-PROXY
In ArchLock, we utilize zero-cost proxies (ZCPs) obtained from the NAS-Benche See--echSSuutt-eerre tool Ruchte al. 2020). This tool assesses the performance of various ZCPs, and specifically focus on data-dependent ZCPs since we believe that architectural performance could vary depending on the data involved. The data-dependent ZCPs consider data/tasks compute scores but do update the gradient for weight optimization.
The NAS--ench-SSuie--Zero tool provides total of data-dependent ZCPs. We select 7 of these 9, namely fisher Turner et al. (2020), flops Ning et al. (2021), grad noomm Abdelfattah et al. (2021). grasp Wang et (2020), jacov Mellor al. (2021) nwot Mellor al. (2021). and snip Lee al. (2019b). Two ZCPs, namely epe-nas Lopes et al. (2021) and plain Ning et al. (202T). are excluded from our selection.
We exclude epe-nas because it is designed specifically for the classification task. As for plain, it exhibits the lowest Spearman rank correlation coefficient with validation accuracies among the ZCPs. Furthermore, extensive experiments conducted in Krishnakumar et al. (2022) demonstrate that including more than ZCPs only leads to marginal improvements. Therefore, we consider the inclusion of 7 ZCPs to be sufficient for purposes.
The technique of each ZCP briefly described below:
fisher: It computes the sum of gradients of the activations in the network.
flops: measures the number of foating-point operations (flops) required to process an input through the network. It provides an estimate of computational complexity.
gord- calculates the sum of Euclidean norms of gradients.
grasp: It approximates the change in gradient norm.
jacov: It measures the covariance of prediction Jacobian matrices across minibatch of data. It captures the relationship between predictions and their gradients.
nwot:
It examines the overlap of activations between data in mini-batch for untrained network.
:nip: It estimates the change the loss function.
important to note that while not all of these metrics were initially introduced as ZCPs, they have been later utilized in the field of NAS to assess architecture performance without the need for extensive training. These metrics have been demonstrated effective in providing reliable estimations of performance.
13

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION OF FUNCTIONS IN CROSS-TASK SEARCH
Algorithm Binary Predictor with Sorting Algorithm
1: function (pWSoR P, zs, ZT.)
2: Zero-initialize matrix comp, and comp, of shape [en((opu)) len(popu)]
3: for a, E range(len (popu)) with > a do
4: Encode archa; arch, - arch pair
5: Pa,Ph Pb & P(arch_pair;2,,
6:
if Pa ps then
7:
comp_s(a, 6] t
8:
else
9:
comp_s/b a) t end
if
11:
for each ZT. do
12:
Pai pb + Plarch_puir, ZT;)) if Ps then
13:
14:
comp t|a, b] +=
15:
else
16:
comp_t(b, a] +=1
17:
end if
18:
end for
end for
20:
sumf, &- Sum comp, along dimension
Srank - Get the index of elements of sum, in descending order
22: sum; + Sum comp; along dimension 0
Trank <-
Get the index of elements of sum, in descending order
24:
return Srank, Trank
25:
end function
Algorithm Update History in cross-task search
1: procedure (HDNTEE arch, P. zs ZT.) history.append((rch) if len ten(history) > le then Srank; Trank +- BPoSor((hisoor. P, P,Zs, ZT.)
2:
3:
4:
5:
Calculate fitness score for each arch E history history <- top-k archs with highest in history
6:
7: end
8:
end procedure
D NAS-BENCH-22
The performance of different task pairs NAS-Bench-201 can observed in Fig. 4 From the figure, it is evident that there is relatively high correlation between CIFAR-10 and CIFAR-100, indicating that architectures that perform well on CIFAR- tend to also perform well on CIFAR-100. However, the correlation between CIFAR- 100 and 1mageNee-116-1200 is lower. Indeed, the absence of a perfect positive correlation between task pairs in yielding optimal performance
suggests that the architectural choices task may not translate to the best performance another task. This observation indicates that there is room for exploration and development of secure architectures that effectively defend against malicious transfer attacks.
14

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-100
ImageNet- 16-1220
80
60
40
CIFAR-IN 10
30
40
20
40
20
80
60
40
CIFAR-100 100
30
40
20
40
20
80
60
40
ImageNet- 16-120
30
40
20
40
20
10
0
Figure 4: The correlation of each task pair in NAS-Bench-201.
E TRANSNAS-BENCH-II
The correlation of each task pair in SeaesNAS-Beecch 101 is shown in Fig. where Acc. denotes accuracy and Neg. denotes negative. Here we use the negative loss for RL to be consistent with other metrics, i.e., the higher value indicates better performance. Compared to WAS-Bench-100 the task pair this benchmark shows relatively lower correlation, which indicates the transferability of an architecture varies when the task changes. By leveraging the understanding that architectures may perform differently across various tasks, we can focus on designing models that only excel in their source tasks but also possess robustness and resilience against unauthorized transfer. This opens up opportunities for research and innovation in creating architectures that prioritize security and can withstand attempts extract and transfer their knowledge maliciously.
15

---

Published as a conference paper at ICLR 2024
(Aee
(Nip Loog
(Ace.
-4
(Nap
-3
48
34
6hoz 55
5g)
Figure 5: The correlation of each task pair in
16


Reviews:
[{"review_id": "VKtwvyoThI", "reviewer": "Reviewer_ySE1", "summary": "This paper prposes ArchLok to mitigate unauthorizaed DNN transfer. ArchLock first encodes the NN architecture to evaluate rank two architectures with task embeddings, then perform neural architecture search to find archs that are good on source tasks but bad on target tasks. Evaluation on NAS-Bench-201 and Trans-Bench-101 demonstrate that ARchLock significantly reduces the transferbility.", "questions": "See weakness above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "By addressing security at the architecture level, ArchLock potentially fills a gap left by other security measures that focus on the model parameter level. This provides a more holistic defense strategy for DNN models.\n\nArchLock focuses more on the architecture rankings rather than the actual performance numbers, and utilize efficient zero-cost proxies as supervision. This approach can be scaled to any size of architecture pool and reduces the cost of training several architectures from scratch.\n\nExperiments on NAS-Bench-201 and TransNAS-Bench-101 demonstrate the effectiveness of ArchLock. It can effectively degrade the performance on target tasks by up to 30% while preserving the performance on source tasks.", "weaknesses": "The details of S / TU / TK are not clearly described. Algorithm 1 shows the cross-task search when the target task is known, but it does not discuss how the other two baselines are performed. Additionally, it is still unclear how the GraphEncoder (Figure.1) is executed and how task embedding is extracted. How much overhead does  task embedding take for each new task?\n\nAre the numbers in Tab 1 and 2 real measurements, or are they directly taken from NAS-Bench/TransNAS-Bench? The zero-cost proxies/predictors are trained on the same set of datasets, which may lead to potential overfitting. Evaluating on unseen and large-scale datasets (e.g., ImageNet [can be a subset with ful 224x224 resolution], Miniplaces) is necessary to demonstrate effectiveness.\n\nArchLock aims to design architectures that show less transferability on new tasks, but it does not discuss what kind of architecture leads to poor transferability. For example, do different datasets dislike different architecture designs, or there is an type of architecture that transfers bad generally on all tasks? It would be beneficial to provide visualizations and discussions of general un-transferable architectures so that further work can gain inspiration and insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PD4dX4zhRs", "reviewer": "Reviewer_umc9", "summary": "This paper presents a cross-task NAS framework to find an architecture to mitigate unauthorized DNN transferability.  A binary predictor using multiple zero-cost proxies is proposed to accelerate the NAS procedure. The results and the ablations demonstrate the effectiveness of the proposed method.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The whole formulation of reducing transferability at the architecture level involved with architecture search is solid.\n2. The proposed nas method based on binary predictor is efficient and effective in designing model architectures with low transferability.", "weaknesses": "1. Since the proposed method is based on a predictor, maybe it is better to cite a series of predictor-based NAS work. For example, PINAT: A Permutation INvariance Augmented Transformer for NAS Predictor AAAI 2023 TNASP: A Transformer-based NAS Predictor with a Self-evolution Framework NeurIPS 2021 and so on.\n2. It seems that there is no training details about the binary predictor, What is the training cost for this predictor? Is one pre-trained predictor suitable for processing the architectures from different search spaces?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aKawfMQhnD", "reviewer": "Reviewer_boEZ", "summary": "This paper presents a methodology to degrade NN performance on new tasks. Specifically, the paper mentions that adversaries may want to adapt a pretrained NN to a new task while violating its terms of use. To mitigate this issue, the presented methods performs a form of neural architecture search to find NN architectures that degrade performance on the tasks for which the NN was not trained.", "questions": "Can you please provide responses to the weaknesses above. The work is interesting but not fully convincing in its current form.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Demonstrates results on transnasbench.\n- Leverages zero-cost proxies in a new way.", "weaknesses": "While this paper is interesting, I am not quite convinced of the motivation behind it. If you want to prevent others from fine-tuning a NN, why even release its parameters to begin with? perhaps in this use-case, the model should not even be released? Also, the definition of \"task\" is very broad. Would additional data be considered a new task? for example, more 32x32 images to be classified into cifar 10 classes for a cifar-10 NN? Or is it just when the classification head is modified?\n\nOther weaknesses include:\n- a limited evaluation on NAS benchmarks, making it harder to appreciate the motivation of the paper. If evaluation was done on some NN that someone wants to protect, then it would've helped.\n- 2% performance degradation on CIFAR-leve NNs is actually quite large.\n- Zero-cost proxies are meant to guage accuracy in general and have not been verified to work for this  task of minimizing out-of-training-distribution accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)