NASPY: Automated Extraction of Automated Machine Learning Models
Authors: Xiaoxuan Lou
Shangwei Guo
Jiwei Li
Yaoxin Wu
Tianwei Zhang
Venue: ICLR 2022 Spotlight (2022)

Abstract: We present NASPY, an end-to-end adversarial framework to extract the networkarchitecture of deep learning models from Neural Architecture Search (NAS). Existing works about model extraction attacks mainly focus on conventional DNN models with very simple operations, or require heavy manual analysis with lots of domain knowledge.  In contrast, NASPY introduces seq2seq models to automatically identify novel and complicated operations (e.g., separable convolution,dilated convolution) from hardware side-channel sequences. We design two models (RNN-CTC and transformer), which can achieve only 3.2% and 11.3% error rates for operation prediction.  We further present methods to recover the model hyper-parameters and topology from the operation sequence .  With these techniques, NASPY is able to extract the complete NAS model architecture with high fidelity and automation, which are rarely analyzed before.

Published as a conference paper at ICLR 2022
NASPY: AUTOMATED EXTRACTION OF AUTOMATED MACHINE LEARNING MODELS
Xiaoxuan Lou', Shangwei Jiwei Li Yaoxin Wu' and Tianwei Zhang' Nanyang Technological University, *Chongqing University, "Zhejiang University, "Shannon.Al (xiaoxuan001, wuyaoxin, thanwei.zhang) @c.ntu.edu.sg, swguo @ cqu.edu.cucccn jiwei ii@hanoonai.com
ABSTRACT
We present NASPY, an end-to-end adversarial framework to extract the network architecture of deep learning models generated by Neural Architecture Search (NAS). Existing model extraction attacks mainly focus on conventional DNN models with very simple operations, or require heavy manual analysis with lots of prior knowledge. In contrast, NASPY introduces seq2seq models to automat- ically identify novel and complicated operations (e.g., separable convolution, di- lated convolution) from hardware side-channel sequences of model inference. We design two models (RNN-CTC and transformer). which can achieve only 3.2% and 11.3% error rates for operation prediction. We further present methods recover the model With these techniques, NASPY is able to extract the complete NAS model archi- tecture with high fidelity and automation, which are rarely analyzed before.
and topology from the operation sequence.
INTRODUCTION
Recently Automated Machine Learning (AutoML) has attracted lots of attention from the machine learning community, as can significantly simplify the development of machine learning pipelines with high efficiency and automation. One of the most popular AutoML techniques is Neural Ar- chitecture Search (NAS) Zoph & Le 2016 Elsken et al. 2019). which can automatically generate high-quality Deep Neural Networks (DNNs) for specified task. It enables non-experts to produce machine learning architectures and models which can outperform hand-designed solutions.
From the adversarial perspective, this paper aims to design new attacks steal the architectures of black-box NAS models. This is known as model extraction attacks, which could cause severe consequences: (1) searching a good architecture with NAS is an energy- and time-consuming pro- cess. Hence the produced architecture is naturally considered as an important intellectual property, and stealing it can lead to copyright violation and financial loss (Hong et al., 2020). (2) Extracting the model architecture can facilitate other black-box attacks, e.g., data poisoning (Demontis et al. 2019), adversarial examples (Iyas et al. 2018), membership inference (Shokri et al. (017).
One solution of model extraction is to remotely query the target model and recover the architecture based on the responses (Oh et al. 2019). However, such attack requires large computation cost and can only be applied to simple neural networks"] Then we turn to more promising solution: hard- ware attacks (e.g., cache side-channel, bus snooping). Essentially, when a DNN model executes the inference task on computer, it leaves architecture-deendder. footprints the low-level hardware components, which could be captured by an adversary to analyze and recover the high-level archi- tecture details. These techniques can give very fine-grained information, and have been utilized by prior works (aan et al., 2020 Hua et al. [2018] Naghibijouybari et al.) 2018 Hu et al. (020) to ex- tract the architectures of conventional DNN models. However, there are several challenges when we apply such attack techniques to extract NAS architectures. (1) These works can only handle simple operations in conventional models, while failing to analyze new operations introduced by NAS (e.g., separable convolution, dilated convolution). (2) Some works need complicated manual analysis with prior knowledge of the victim model. For instance, (Yan et al. 2020) requires the information the victim model family, and can only extract variants of generic architectures. (Naghibijouybari et al. 2018) needs to know the layer type ahead.
It takes 40 GPU-days recover 7-layer architecture with simple chained topology (Oh al. 2019

---

Published as a conference paper at ICLR 2022
To the best of our knowledge, there is only one work (Hong et al. 2020) focusing the extraction of NAS models, which is not very practical or general. (1) This work mailly monitors the API traces in the high-level deep learning library, which requires the attacker and victim to share the same library with exactly the same version. This is not practical since users may use different libraries, especially their customized ones. (2) It needs the accurate dimension estimation to predict the layer type and model topology, which is hard to obtain in the real world. (3) The NAS model considered in this work is too small and simple, which cannot well represent state-of-the-an NAS techniques.
We propose NASPY, a learning-based framework for automated extraction of NAS architectures with high efficiency and fidelity. We make several contributions to overcome the above limitations. First, we exploit cache side-channel techniques to monitor the low-level BLAS library. Hence, the framework can be applied to different platforms regardless of the high-level deep learning libraries (Tensorflow, Pytorch, or other customized libraries). This cannot be achieved in (Hong et al.. 2020). Second, we model the extraction attack as sequence- saaeence-----euennee problem, and design new deep learning models to predict the model operation sequence automatically. This does not require the tedious manual analysis, as conducted in (Yan et al., 2020). Meanwhile, the models are able to predict new sophisticated operations in NAS, which are missing in Yan et al., 2020; Hu et al., 2020)- Third, we propose a new analysis method to precisely recover the exact hyper-parameterss parameters without any prior knowledge. In contrast, previous works can only estimate a possible range of hyper- parameter values (Yan et al.] 2020; Hu et al. (020). Finally, we design strategies to reconstruct the model topology and extract the complete architecture for different scenarios and adversarial goals.
We perform extensive experiments to demonstrate the effectiveness of NASPY. Our identification model can predict the operation sequences of different NAS methods (DARTS (Liu et al. 2018).
GDAS (Dong& & Yang] 2019) and TE-NAS (Chen et al. 202T)) with an error rate of 3.2%. Our hyper-parameter prediction can achieve more than 98% accuracy. The framework also demonstrates high robustness against random noise introduced by the complex and dynamic hardware systems.
The source code of NASPY is available at hhttps:: / www.tttw..o......oo Loom/Looiaio wwx.an/NA.SSS
2 BACKGROUND
Neural Architecture Search (NAS). This promising autoML technology can systematically gen- erate good network architecture for a given task and dataset (Zoph & Le| 2016 Elsken et al. 2019). It defines search space as the scope of neural networks in consideration. from which it finds the best architecture with different types of search strategies (Zoph & Le 2016 Real et al, 2019 Liu et al., 2018). To reduce the search complexity and cost, neural network is decomposed into multiple cells (Chen et al. 2021). A cell can be represented as a Directed Acyclic Graph, where each edge represents connection between two neural nodes that is associated with operation lected from a predefined operation set (Dong & Yang 2020). A NAS architecture normally has two types of cells: a normal cell computes the feature maps and a reduction cell reduces the spatial size. Multiple normal cells construct a block, and multiple blocks are interconnected by reduction cells to form the final model. Compared traditional neural networks, NAS adopts some sophisticated operations among the neural nodes, like dilated convolutions and separable convolutions.
Hardware attacks. Following the previous works (Yan et al. 2020 Hong et al. 2020 Hu et al. 2020), we aim to exploit some hardware attacks to perform the extraction of NAS models. Specifi- cally, we consider the following two attacks.
Cache side-channel attacks: CPU caches are introduced between the CPU cores and main memory to accelerate the memory access. Since the attacker can share the same cache with the victim, he can reveal the behavior pattern of the victim from the contention on the usage of cache lines. In this paper, we adopt -EIUS--RILOAD which is also used (Yan et al. 2020 Hong et al., (020) for model extraction. The adversary leverages the shared low-level BLAS library infer the victim model, and is able to obtain the sequence of its critical operations, e.g., the matrix multiplication.
Bus snooping attack: data traffic between the processor and memory system is achieved through bus, which sends data to or loads data from specific addresses. Hence, by observing the memory traffic through the bus, the attacker can obtain the memory address traces of the victim model, which further reveals the connections between model layers. In this paper, we use the bus snooping technique (Huang et al. 2014) to monitor the read/write addresses of each model layer, which is also
2

---

Published as a conference paper at ICLR 2022
adopted in (Hu et al. 2020). The adversary can only observe the data addresses and cannot access the data passing through buses, which allows NASPY to work even when the model is encrypted.
Sequence-to-sequene learning. Seq2seq learning is raising increased attention in the machine learning community, and becomes quite popular for different tasks like speech recognition (Amodei
2017). image captioning (Islam et al.) 2019), question answering (Palasundram et al.] (020). etc. Three models are mostly used for seq2seq learning: Re- current Neural Network (RNN), Connectionist Temporal Classification (CTC), and attention models (Transformer). In this paper, we aim to use seq2seq learning for automated model extraction from the monitored memory activities. We design an RNN-CTC model and Transformer model to recover the structure operations of NAS models.
al. 2016), machine translation (Neubig
3 FRAMEWORK OVERVIEW
3.1 THREAT MODEL
Adversary's goal. Given victim model M constructed from NAS, the adversary aims recover a similar network architecture as M, without searching it with large cost and the original dataset. We consider two types of goals following (Jagielski et al.) 2020): (1) accuracy extraction: the adversary aims to reproduce a network architecture, which can give similar model accuracy as the victim model; (2) fidelity extraction: the adversary wishes to recover the same architecture and hyper-parameters as the victim one.
Adversary's capability. We consider two practical scenarios for extracting model architectures. For each scenario, assume the attacker only knows the target model is from NAS, without any other prior knowledge, c.g., the model family, layer type, NAS method, high-level deep learning library.
Remote attack: model extraction in this scenario adopted (Yan al. 2020 Hong et al., 2020). The attacker can launch his malicious program on the same machine with the victim model. Although these programs are isolated by the OS or hypervisor, the attacker can still exploit the cache side-channel technique monitor the victim's 's low-level executions, e.g., the sequence of matrix multiplication events. Based on the side-channel sequence, the attacker able to perform accuracy extraction of the victim model.
Local attack: this scenario is considered in (Hu et al., 2020), where the attacker can physically access the machine running the victim model. In addition launching the cache side-channel attack to retrieve the operation sequence, the attacker can also launch the bus snooping attack to monitor the memory bus and PCIe events. With such memory address traces, the attacker can achieve fidelity extraction of the victim model and recover the exact model topology.
3.2 ATTACK OVERVIEW
Before describing our NASPY framework, we need to understand the workflow of the model infer- ence process. As shown in the left diagram of Figure given DNN model, deep learning library (e.g., Tensorflow, Pytorch) is used to process the computational graph of the model, and convert the model architecture into sequences of connected layer operations. Then these operation sequences are sent to low-level computation libraries for acceleration, such as the BLAS library (c.g.. Open- BLAS) for GEneral Matrix Multiplication (GEMM), and the mathematical library (e.g., libm) for activation functions. Those computations will be executed on the hardware platform. By monitor- ing the hardware activities using cache side-channel and bus snooping techniques, the attacker can observe the event sequences and memory address traces for the model inference process.
Overview of NASPY. The adversary's task is to automatically and precisely recover the NAS model architecture from the captured sequences of hardware activities. Figure] I shows the workflow of our NASPY framework. consists of three steps, as described below.
First, it translates the event sequences from cache side-channel attacks to the operation sequences (i.e., input of the low-level computation library). We model this as a seq2seq problem, and design two deep learning models (RNN-CTC and Transformer) to achieve this prediction. Second, it iden- tifies the values of hyper- parametersssss in the layer operations recovered from the first step. Previous works (Hu et al. 2020) only estimate range of these values based on the dimension size of layer
3

---

Published as a conference paper at ICLR 2022
NAS Model
Bus snooping
Memory Address Trace
Trace analysi
Model Architecture
DL Library
(e.R. Tensorflow, Pytorch)
Fidelity Extraction
Com computation Library
[e.g.. OpenBLAS, Intel MKL
Hardware
Cache side channel
Event Sequence
Sea2se0
Operation
Hyper- paramaters
Random search
Model Extraction
CPU
Cachc
Memory
model
Sequence
anelysis
Accuracy
Figure 1: Workflow of our model extraction framework.
input and output. In contrast, NASPY can precisely reveal the exact values of the from the translated operation sequence. Third, NASPY reconstructs the model topology and obtains the complete architecture. For the accuracy extraction attack, the attacker can randomly select model topology. and assign the recovered operations with the hyper-parameee ment results show that the corresponding model can give similar accuracy as the victim one. For the fidelity extraction attack, the attacker needs recover the exact model topology. He can leverage the information in the memory address trace from the bus snooping attack to construct the architecture.
to it. Our experi-
4 DETAILED DESIGN
4.1 OPERATION SEQUENCE IDENTIFICATION
The first step is to predict the operation sequence from the hardware event sequence. Past works Yan et al., 2020 Hong al. 2020) require manual analysis for such translation. In contrast, propose to leverage a seq2seq deep learning model to achieve this task automatically.
Dataset formulation and preprocessing. Specifically, the event sequence is time-series of length T. where each frame is a vector of event features. In this paper, we capture the occurrence of itcopy and oncopy APIs in OpenBLAS, which are used to load matrix data for GEMM. Note that our framework is also generalized to other BLAS libraries, such as Intel MKL. Hence, a frame is denoted as I, (I,, O T.), where is the time interval from the last monitoring moment, and 4; and O are binary values to denote whether itcopy and oncopy are called during this interval. T; is determined by the monitoring granularity when collecting side-channel information.
The operation sequence contains N operations performed by the victim model. To comprehen- sively cover novel neural architectures, we consider all the common operations used in state-of of art NAS methodologies: fully connected layer (FC), normal convolution (Conv), dilated convolution (DConv), sepurable convolution (SConv), hilated-separuble convolution (DSConv), pooling (Pool), identity (Skip), zeroize. is easy integrate other operations into NASPY if necessary.
Figure shows the event sequences four representative operations, where the blue and red nodes denote the occurrences of itcopy and oncopy. We observe that different operations have distinct event patterns, giving us opportunities to predict the model operations from the side-channel leakage. However, there are couple of challenges design seq2seq models for this task. We perform the following data preprocessing methods to overcome these challenges.
unwes Invles
1923 rererattet
1000
en (a) fully connected layer
(b) normal convolution
Satrooo ropel
Tiue: Intioval (1)
Tim (irstee ID:
(c) separable convolution
(d) dilated separable convolution
itopy
anoopy
Figure 2: Event sequences of four representative operations in NAS models.
1. Input downsampling and label extending. An event sequence of a NAS model is extremely long
(> million frames) because of the high monitoring frequency. contrast, an operation sequence y usually has only hundreds of operations. It is infeasible directly use existing seq2seq learning models to handle such length gap. To resolve this challenge, we perform downsampling for x and extend labels in y. Specifically, for the event sequence we only keep the frames whose I; O 1. We also update T, in these active frames as the time interval from the last function access. This significantly reduce the sequence length, while preserving the critical information. For
4

---

Published as a conference paper at ICLR 2022
Beam
Event Sequence
Search
Probabilily
Operation Sequence
(a) RNN-CTC Model
Channel Trace
Row Side
Encoder
Absirct
Beam
Longth N
Length T
Sepresentation
Conv
<START>
Transformar Model
Figure 3: Procedure of operation sequence identification.
the operation sequence y, we decompose the single label of complex operation into a series of sub-operation labels. For instance, SCony operation is composed of several Conv sub-operations (Figure(Zcc) So instead of directly labeling this operation as SCom, we extend it as multiple Conv labels, the number of which determined by the channel size. Such preprocessing can remarkably decrease the difficulty of training the seq2seq model.
Inter-operation context. Not every operation in y has the corresponding event in æ. For instance, Pool and Skip never invoke GEMM computations. This makes it difficult to predict such operations. We propose to leverage the execution latency and inter-operation context for identification, since their values are different in these operations. Specifically, we introduce two interval labels into the operation sequence: Tr denotes the interval between two operations while T,, denotes the interval between the decomposed sub-operations within operation. Such two labels reflect the context switch for inter- and inner-operations, which hence helps the seq2seq model distinguish adjacent operations in the sequence with high accuracy. Experiment results in Section 5. show that the consideration of inter-operation context brings large performance improvement.
3. Time normalization. When the victim model runs different platforms, although the occurred events (i.e., I, and O,) keep constant, the time interval T; between the frames will be different, due to the varied execution latency in hardware. This can restrict the generalization of our seq model. To overcome this challenge, we perform normalization over T; in the captured side-channel trace, and use the relative intervals train the model, which are similar across various platforms as they are determined by the algorithm behaviors.
4. Data augmentation. Different from NLP and CV tasks, the raw side-channel data can contain lots of random noise from the hardware activities. This is due to the complex system optimization and nuntimme dynamics (Hu et al., 2020). Such noise can decrease the identification accuracy. To improve the robustness of the seq2seq model against noise, we further perform data augmentation following Park et al. (2019), which simply cuts out random blocks of consecutive time and feature dimensions. In this paper, we just mask these blocks with the fixed value O.
Seq2seq model designs. We propose two kinds of models for predicting the operations from the side-channel trace. The first one is an RNN-CTC model. Recently the combination of RNN and CTC decoders commonly used in sequence modeling problems. Figure shows the architecture of this model for identifying the operation sequence y from the event sequence Given that only contains three features in each frame, we first introduce a convolution layer to learn more features. Then RNN layer is used to propagate information through this sequence. We adopt the Bidirectional Gated Recurrent Unit (BGRU) as the RNN module, which can enable better long- term memory and fully leverage the past and future contexts. The output of the RNN layer is the probability distribution of all the operation labels for each frame, which is further fed into the CTC decoder. While is difficult to align the operation sequence with the event sequence that has various length, the CTC decoder can skip the alignment by introducing "blank" label. Finally, the operation sequence y with the largest prediction probability is identified with the beam search.
The second one is transformer model. Proposed by Vaswani et al. (017)) transformers have broken multiple AI task records and pushed the state of the art. Hence, we also show the possibility of operation identification with a transformer model, as shown in Figure (b). Transformer is an intention-based neural network with a typical encoder-decoder architecture. The encoder maps the input sequence into an abstract representation that contains all the learned features of that input. Then the decoder takes this abstract representation and predicts the next output step-by-step based on the previous output. The introduction of the attention mechanism enables the transformers have extremely long term memory, which can focus on all the tokens generated previously. Similarly, we add convolution layer before the encoder learn more features from the event sequence.
5

---

intuitive to Published as a conference paper at ICLR 2022
4.2 PAPER-PARAMETER RECOVERY
Our second step is to extract the architectural hyper-parameter values of each operation. NASPY can precisely recover the exact hyper-parameters, rather than an estimated range in previous works (Hu
et al. 2020 Yan et al. 2020).
Convolutions. Given that convolutions take a majority in a NAS model, we first discuss how to reveal hyper-parameters of different convolution operations, e.g., kernel size R. dilation d, channel size C. padding size and stride. Note that we assume the input size (i.e., width and height) has been identified from the analysis of the previous layer, while the initial input size of the model is publicly available. (1) The extended event labels in is integrated to compose the single label of the complex operation (e.g., SConv and DSConv). During this process, NASPY discovers the channel size C' by counting the number of event labels. (2) Both the kernel size R and dilation d can be directly recovered from the predicted operation labels (c.g., 3 x SConv x 5 DSConv). NASPY can distinguish various convolutions with different kernel sizes, as they have different side-channel patterns and execution time. (3) Empirically, the padding size is normally set R/2 keep the input size and output size constant for possible residual connections. (4) The stride can be deduced from the cell type. Normal cells keep the spatial size unchanged while reduction cells would half the spatial size but double the channel size to maintain the dimension information. Hence, the stride 1 for normal cells and 2 for reduction cells. The cell type is identified through the channel size C.
Other operations. We can also infer the hyper-pas parameters for other operations like the FC and Pool layers. The number of neurons in a FC layer can be reflected by the length of the event sequence, longer sequence. Such length variance
where larger number of neurons leads learned by the seq2seq model, which discloses the number of neurons in the predicted labels. For the Pool layer, the relationship between the pooling size R and padding size Pis is P R/2 in order to keep the spatial size among layers in the same cell.
be well
4.3 MODEL TOPOLOGY RECONSTRUCTION
The final task is to extract the topology to obtain the model architecture. As NAS model built with cells, we also reconstruct the model topology in term of cells. First, the model macro skeleton is determined by analyzing the number and types of cells from the event sequence. locate each cell, since they are separated by much larger time intervals due to some extra compu- tations like concatenating and preprocessing. Then, we focus on the topology of each cell. Unlike conventional DNN models, the topology in NAS cells is not chained and sometimes even not regu- lar. Hence, we cannot simply connect each extracted operation in the sequence to form the topology. Based on the attack scenarios and goals, the cell reconstruction
be done with different methods.
Accuracy extraction. To extract a model architecture with similar accuracy, the attacker can just use the remote side-channel attack to recover the operations and typee-paramettess and then randomly choose a topology following the basic rule of NAS, i.e., each neuron node has two inputs. The experiment results in Section 5.3 verify the effectiveness of this strategy.
Fidelity extraction. As only the side-channel event sequence is not enough to achieve fidelity extraction, the attacker can adopt bus snooping get the memory address trace to reveal the exact interconnections between layers. With the revealed model topology, the attacker can finally extract the complete model architecture
EVALUATION
Testbed. NASPY is general for different deep learning and computation libraries. Without loss of generality, adopt Pytorch (1.8.0) and OpenBLAS (0.3.13).
Dataset construction. We search model architectures with CIFARIO, and train model parameters over CIFARIO and CIFAR100. Our method can be applied to tasks for other datasets as well. We first generate 10,000 random computational graphs of NAS models following the macro skeleton proposed in the popular benchmark nodes associated with 8 operations (each node has two inputs). The operation set follows the classical set in (Liu et al. 2018), which contains: identity, zeroize, 3x3 and 5x5 SConv, 3x3 and
(Dong & Yang 2020), where each cell contains

---

Published as a conference paper at ICLR 2022
0.30
2.0
PAN CTCC
1.30
Training Loss
Average OER
0.25
0.20
Transformer
valisiet ion Loso
SEn
Average OER
0.8
1.00
0,6
0.6
0.15
DER
0.10
0.4
0,0
0.05
0.2
0.2
0.00
19.28
19256
10,522
28
20.56
20.13
2.00
0.0
0.0
1500
2000
Model Setting
Epoch
Sample ID
(a)
(b)
(c)
Figure 4: (a) Average OER of the two identification models with different configurations. (b) Loss and OER trend of the RNN-CTC model. (c) OER of the transformer model on validation samples
5 DSConv, 3 average pooling, 3x3 max pooling. Conventional operations (i.e., Conv and FC) are also included in our model.
Then we collect the corresponding side channel sequences using the FLUSH- RELOAD technique, which inspects the cache lines storing OpenBLAS functions (itcopy and oncopy) at a granularity of 2000 CPU cycles. We randomly select 80% of the sequences as the training set, and the rest as the validation set. The seq2seq models will be tested on novel NAS models generated by three state-of- the-art NAS methods: DARTS (Liu et al. 2018), GDAS (Dong & Yang (019) and TE-NAS (Chen) et al. 2021 For each method, we search 10 NAS models with various initial seeds for testing.
5.1 OPERATION SEQURNCE IDENTIFICATION
Metrics. Inspired by the evaluation metric WER (Word Error Rate) in NLP tasks, propose Oper- ation Error Rate to quantify the prediction accuracy. It is calculated as: OER = D(y', y)/ yll where D(y', 3) is the edit distance between the predicted operation sequence y' and ground-truth sequence and [y] is the sequence length of y. A smaller OER implies higher identification accuracy.
Prediction aceuracy. Given that our dataset relatively small, we only use one layer of BGRU in the RNN-CTC model, and one layer of encoder and decoder in the transformer model. Figure 4a) shows the average validation OER of two models with different structure settings. Both ID Conv and 21 Conv are considered to extract more features from the input, and the model dimension size (i.e., dynn of the BGRU layer and dmade of the transformer) is chosen among [128, 256,
512]. We see that while all the configurations can achieve very high accuracy, the best one is: 2D
Conv and dmoodd = 256, where both models give the lowest OER: 3.2% for RNN-CTC and 11.3%
for the transformer. The trend of the loss and average OER during the training of RNN-CTC depicted in Figure(41b) We observe that the loss and OER decrease dramatically within the first 20 epochs and reach convergence at epoch 40. In contrast, the transformer model performs relatively worse. fe shows the OER of each validation sample predicted by the transformer, which is higher and more unstable. It is because the length of the preprocessed side-channel event sequence (normally 5000 frames) is still too long for the transformer, which requires tremendous GPU and memory resources, making the training more difficult.
Finally, we test the above two models on the NAS models generated by DARTS, GDAS and TE-NAS and the results are shown in Table 1 From the table, both of the RNN-CTC and transformer models can well predict the opera- tion sequence from the side-channel leakage of three types of NAS models.
Model RNN-CTC
Average
DARTS
GDAS
TE-NAS
0.032
0.035
0.021
0.038
Transformer
0.113
0.151
0.115
0.094
Table I: Testing OER for three NAS methods.
Effectiveness of inter-operation context. We adopt the inter-operation context technique to handle the missing events of some operations 4.1]. To evaluate its effectiveness, Figure F the prediction error rates of the two models with and without considering the context. Figure 5(a) shows the OER trend of the RNN-CTC model during training. We observe that without the inter- operation context, the OER decreases more slowly and converges depicts the fitting curves on the OER of validation samples for the transformer. Ignoring the context will lead to huge performance penalty, which gives higher and less stable prediction error rates.
higher value. Figure b)
We train these models using the open-sourced code from the authors.
7

---

Published as a conference paper at ICLR 2022
1.2
0.30
0.30
& Content
Will fegronddes Aag
100 Our
0.2
WET REME
0.2
0.2
0.35
015
0.4
0.33
0.10
60
100
0.3
500
1050
3500
2000
0:00
15%
30%
Epoch
Sarppe ID
(a)
(b)
(a)
(b)
Figure 5: Inter-operation context testing. (a) OER trend of RNN-CTC. (2) OER of the trans- former on validation samples.
Figure 6: Robustness versus different scales of noise. (a) OER of RNN-CTC. (b) OER of the
transformer.
Robustness against noise. We further evaluate the robustness of our models against the noise. Figure&ssoww shows the OER of two models without and with the data augmentation technique, when the side-channel event trace contains different scales of random noise. First, we observe that our two models have strong robustness and give acceptable error rates under large amounts of noise. Second, data augmentation (masking rate 0.1) can further improve the model robustness. With 30% random noise, the OER of RNN-CTC is 0.115 and the transformer is 0.208. Besides, data augmentation has better improvement for RNN-CTC than the transformer, as the transformer always considers the entire input, which weakens the effect of this technique.
100
Accuracy (%)
98
Inpeto DeTTrbatReeR00
Model 8AL-0x7 Seho0EG6800
BAD: Ou? Dxhabaaaae
RAD NN1::97Tcbh0616680
Noded
96
KL:0a RL:0aTfebuB01.0 48c0
94
844:027 4500
92
M1.0aTfebu661 4e00
90
Exput1 Do7Tebaboo
Mode3
Node& BAL:077eboo01 #6Z:007TebobceeeB No:okTEbbaD03
kemel
chante)
diabon
padding
gride
PC neuron
862 Eoddeeseeo
Figure 7: Recovery accuracy.
Figure 8: Memory address trace of a NAS cell.
5.2 HYPER-PARAMETEE RECOVERY
NASPY can further extract the hyper-parameter values from the predicted operation sequence. As RNN-CTC performs better, we conduct nyper-parameter recovery based on the operation sequence from this model. The hyper-parameter prediction error is identical to the operation identification er- ror (3.2%). Figure/Jshwws the recovery accuracy of various hyper-parametens For convolutions, the accuracy of each hyper-paramette can reach up to 98%. The kernel size and dilation are recovered directly from the predicted labels and have the highest accuracy. The padding size is computed from the kernel size with the same accuracy. The stride is inferred from the cell type, which can reach 100% accuracy, as normal and reduction cells are very distinguishable in terms of the channel size. The recovery accuracy of the channel size is relatively lower, because most errors in the predicted operation sequence are from the adding or deletion of events, while the channel size revealed by counting the repetitions of events in convolution. However, such slight accuracy drop can be easily compensated with post-analysis. For the number of neurons in FC layers, the recovery accuracy is 97.54%, as the side channel pattern of FC is shorter and simpler, making it hard be distinguished sometimes. The pooling size cannot be directly reveal from the operation sequence, as the changing of pooling size does not lead to significant changes of the side-channel leakage pattern. So for this specific operation, we can determine its value empirically from the common values (i.e., 3 or 5).
5.3 MODEL TOPOLOGY RECONSTRUCTION
First, we consider the accuracy ex- traction attack. After the attacker identifies the operations and hyper- parameters, he can randomly choose a model topology to obtain the archi- tecture, which can give close model
Original Model
Random Model with Same Operations
Dataset
#1
#2
#3
#4
#5
CIFAR TO
96.82
96.53
96.44
96.60
96.57
96.77
CIFAR 100
81.07
80.95
80.16
80.33
80.90
80.56
Table Accuracy (%) of random models on two datasets.
8

---

Published as a conference paper at ICLR 2022
performance. To validate this, given a victim NAS model, we first extract its operation sequence and pyper-parametes Then we randomly generate a computation graph that connects nodes in the cell, and associate the revealed operations to the graph sequentially. We train the model from this graph and test its prediction accuracy. Ishows shows the results on CIFAR 10 and CIFAR I 100 datasets with 5 different graphs. We
that the randomly chosen topology can give very similar accuracy as the original model, where the accuracy drop is less than 1% for both datasets. We also check the performance of totally randomly constructed models (operations and topology): the average accu- racy of 5 such models is 91.85% (CIFARI0) and 72.49% (CIFARION) which is much lower than our extracted models. This shows the importance of operation and apper-parameter recovery for accuracy extraction.
Second, we consider fidelity extraction. The memory address trace of the victim model can be obtained with bus snooping tools, e.g., HMTT-v44 (Huang et al. 2014). F shows an example of the monitored address trace of a cell in the vietim model, where RA is the read address and WA the write address. If RA of one layer a is the same the WA of another layer b, we confirm that there is a connection from to a. By analyzing the consistency between RA and WA in the address traces, the connections between layers can be recovered precisely. We perform experiments on 6 NAS models in Table 2 and the results confirm that the model topology can be revealed with 100% precision. Combining the recovered operations and hyper-parameters, which may contain some small errors, we can finally reconstruct the complete architecture that is almost the same as the original one.
6 DISCUSSIONS
Extracting other NAS models. Our framework can be extended to attack other types of NAS models as well. For instance, some latest NAS techniques (e.g., [Lin et al. (202T) do not adopt the cell-based structure for searching. Since NASPY focuses on the recovery of operation sequences rather than cells, it is still effective to extract models from those NAS solutions. Another example is NAS-based RNN models. A RNN model generated by a NAS method from the search space only contains activation functions (Liu et al., 2018). It is noted that these functions cannot be observed from the GEMM trace. Instead, we can monitor other libraries (e.g.. libm) to identify the functions. In the future, we will design seq2seq models and methods for extracting these models.
Extracting standard DNN models. Our framework can also be generalized to conventional stan- dard DNN models. For models with chained topology (e.g.. VGG), it is easy to extract their model architectures by first predicting the operation sequence and then just connecting them in sequence. However, for models with more complex topology (e.g., ResNet), after revealing the operation se- quence, we need to perform fidelity extraction to reveal the exact model topology. If we know the family of the target model (assumed in Yan et al. (2020), we can also reconstruct a similar model architecture by connecting the predicted operations the template structure of the model family.
Defense strategies. There are several solutions that can possibly mitigate our extraction attacks. From the hardware perspective, oblivious RAM (Stefanov al. 2013) was designed to hide the memory access pattern and thwart bus snooping attacks. Security- cache architectures (Werner al., 2019 Qinhan et al. 2020) were proposed reduce side-channel leakage. From the applica- tion perspective, we can obfuscate the model's 's execution behaviors by adding dummy operations or shuffling the operation orders (Hong et al., (020). However, those solutions either require signifi- cant changes to the hardware, or add large computation overhead to the model inference. We will systematically evaluate those defenses, and design more efficient approaches as future work.
CONCLUSION
In this paper, we design NASPY, an end-to-end framework for automated extraction of DNN ar- chitectures from the NAS technology. NASPY adopts new deep learning models to identify model operation sequences from the side-channel event trace. With the identified operations, NASPY can further precisely recover the operation hyper-parameters and model topology for various scenarios. Compared to past works, NASPY can extract novel operations and architectures with higher automa- tion and accuracy, bringing more severe threats to the protection of AI systems. We expect this study can raise the awareness of the community about the severity of model extraction attacks, and inspire researchers to come up with more secure. NAS solutions protect the architecture privacy.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable comments. This project is in part supported by Singapore National Research Foundation under its National Cybersecurity R&D Programme (NCR Award RS02/19, and NTU Start-up grant. Any opinions, findings and conclusions or recommendations expressed in this paper those of the authors and do not reflect the views of National Research Foundation, Singapore
Singapore Ministry of Education (MOE) AcRF Tier
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. arXiv preprint arXiv:2102. 1535, 2021.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transfer- ability of evasion and poisoning attacks. In 28th {USENIX) Security Symposium ((USENIX) Security 19), pp. 321-338. 2019.
Xuanyi Dong and Yi Yang. Searching for robust neural architecture in four GPU hours. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1761-1770, 2019.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the scope of reproducible neural architec- ture search. arXiv preprint arXiv:2001.00326, 2020.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. Journal of Machine Learning Research, 20(55):1-21, 2019.
Kazushige Goto and Robert A van de Geijn. Anatomy of high-performance matrix multiplication. ACM Transactions on Mathematical Software, 34(3):1-25, 2008
Sanghyun Hong, Michael Davinroy, Yigitcan Kaya, Dana Dachman-Soled, and Tudor Dumitraş- How to Own NAS your spare time. arXiv preprint arXiv:2002.06776, 2020.
Xing Hu, Ling Liang Shuangchen Li, Lei Deng, Pengfei Zuo, Yu Ji, Xinfeng Xie, Yufei Ding, Chang Liu, Timothy Sherwood, et al. DeepSniffer: A DNN model extraction framework based learning architectural hints. In International Conference Architectural Support for Pro- gramming Languages and Operating Systems, pp. 385-399, 2020.
Weizhe Hua, Zhiru Zhang, and Edward Suh. Reverse engineering convolutional neural networks through side-channel information leaks. In 2018 55th ACM/ESDA/IEE Design Automation Con- ference (DAC), pp. 1-6. IEEE, 2018.
Yongbing Huang, Licheng Chen, Zehan Cui. Yuan Ruan, Yungang Bao, Mingyu Chen, and Ninghui Sun. Hmtt: A hybrid hardware/software tracing system for bridging the dram access trace's semantic gap. ACM Transactions Architecture and Code Optimization (TACO), 11(1):1-25, 2014.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, pp. 2137- 2146. PMLR, 2018.
Md Sanzidul Islam, Sadia Sultana Sharmin Mousumi, Sheikh Abujar, and Syed Akhter Hossain. Sequence-to-sequeee bangla sentence generation with Istm recurrent neural networks. Procedia Computer Science, 152:51-58, 2019.
10

---

Published as a conference paper at ICLR 2022
Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas Papernot. High accuracy and high fidelity extraction of neural networks. In 29th {USENIX) Security Symposium (!USENIX) Security 20). pp. 1345-1363 2020.
Ming Lin, Pichao Wang, Zhenhong Sun, Hesen Chen, Xiuyu Sun, Qi Qian, Hao Li, and Rong Jin. Zen-nas: A zero-shot nas for high-performance deep image recognition. arXiv preprint arXiv:2102.01063, 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. arXiv preprint arXiv:1806.09000 09055, 2018.
Hoda Naghibijouybari, Ajaya Neupane, Zhiyun Qian, and Nael Abu-Ghazaleh. Rendered insecure: Gpu side channel attacks are practical. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pp. 2139-2155 2018.
Graham Neubig. Neural machine translation and sequence-oe seceenceett-sqquuee se-suuenc models: A tutorial. arXiv preprint urXiv:17053 70669, 2017.
Seong Joon Oh, Bernt Schiele, and Mario Fritz. Towards reverse-engineering black-box neural networks. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 121- 144. Springer, 2019.
Kulothunkan Palasundram, Nurfadhlina Mohd Sharef, Khairul Azhar Kasmiran, and Azreen Az- man. Enhancements to the serrencce----euunneee based natural answer generation models. IEEE Access, 8:45738-45752, 2020.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint 1rXiv:1904.08779 2019.
Tan Qinhan, Zeng Zhihua, Bu Kai, et al. Phantomcache: Obfuscating cache conflicts with localized randomization. Proc of the 2020 NDSS Symp. San Diego, CA: ISOC, 2020.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI Conference Artificial Intelligence, volume 33, pp. 4780- 4789, 2019.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi- -Dotain Operations Applications, volume 1 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Emil Stefanov, Marten Van Dijk, Elaine Shi, Christopher Fletcher, Ling Ren, Xiangyao Yu, and Srinivas Devadas. Path oram: an extremely simple oblivious ram protocol. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pp. 299-310, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp.
2017.
Mario Werner, Thomas Unterluggauer, Lukas Giner, Michael Schwarz, Daniel Gruss, and Stefan Mangard. Scattereache: Thwarting cache attacks via cache set randomization. 28th (USENIX) Security Symposium (UUSENIX) Security 19), pp. 675-692, 2019.
Guowen Xu, Hongwei Li, Hao Ren, Jianfei Sun, Shengmin Xu, Jianting Ning, Haomiao Yang, Kan Yang, and Robert H Deng. Secure and verifiable inference in deep neural networks. In Annual Computer Security Applications Conference, pp. 784-797, 2020a.
Guowen Xu, Hongwei Li, Yun Zhang, Shengmin Xu, Jianting Ning, and Robert Deng. Privacy- preserving federated deep learning with irregular users. JEEE Transactions on Dependable and Secure Computing, 2020b.
11

---

Published as a conference paper at ICLR 2022
Mengjia Yan, Christopher W Fletcher, and Josep Torrellas. Cache telepathy: Leveraging shared resource attacks to learn DNN architectures. In USENIX Security Symposium, pp. 2003-2020, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:661.00157, 2016.
12

---

Published as a conference paper at ICLR 2022
A DETAILS ABOUT GEMM IN OPENBLAS
BLAS realizes the matrix multiplication with the function gemm. This function computes C' = aA x BC, where A is an m matrix, a matrix, Cs an m n matrix, and both and B are scalars. OpenBLAS adopts Goto's s algorithm (Goto & Geijn 2008 to accelerate the multiplication using modern cache hierarchies. This algorithm divides a matrix into small blocks (with constant parameters Q. R), shown in Figure 9 The matrix A is partitioned into P x Q blocks and B is partitioned into Q x R blocks, which can fit into the L2 and L3 caches, respectively. The multiplication of such two blocks generates P x R. block in the matrix C. Algorithm | shows the process of gemin that contains 4 loops controlled by the matrix size (m, n,k). k). Func- used to
Library
Functions
Code Line
OpenBl AAS
Itepy
kernel/generic/geemm tcopy 8.c:78
tions itcopy and oncopy allocate data and functions. kernel runs the actual computation. Note
Oncopy
Table 3:
Monitored code lines in OpenBLAS and Pytorch.
that the partition of m contains two loops, loop3 and loop4, where loop4 is used to process the multiplication of the first P Q block and the chosen Q R block. For different cache sizes, OpenBI LAS selects different values of P. Q and R to achieve the optimal performance. Table gives the monitored lines the code OpenBLAS 0.3.13.
Algorithm 1: GEMM in OpenBLAS
Input: matrice A, B, C: scalars a, 8
II
Output: a.A B 3C
for in (0:R:n) do / Loop
for I in (0:Q:k) do // Loop 2
C
X
k
B
call itcopy for jj in (;3UNROLL:;++)) do
Loop
10
call oncopy
call kernel
X
for i in (P:P:m) do Loop
call itcopy
call kernel
Figure 9: The procedure of GEMM.
B TRAINING DETAILS ABOUT IDENTIFIER MODELS
CRNN+CTC model. This model is comprised sequentially with convolution layer los one bi- directional GRU layer a and one classifier F' with two FC layers. To evaluate the capability of lo on the feature learning, both 1d and 2d convolutions are adopt in experiments for comparison. Besides, to evaluate the performance of identifiers with different model sizes, three candidate dimensions of R (i.e.. 128, 256, 512) are considered. To train the model, we use CTC loss as the criterion bypass the sequence alignment, and we use Adam optimization. The learning rate starts from 5e-4 and is scheduled following the OneCycleLR policy (Smith & Topin 2019). The model is trained for 100 epochs, which takes 6.25 hours on one V100 GPU. The number of model parameters is 0.957 M.
Transformer model. The number of encoder layers and decoder layers this model are both set as 1, and the dimension of model input and output dmooee is also chosen from the tuple (128, 256, 512). We employ keys and values are set as dę d, dmoodt/hh. The inner-layer of the feed-forward layer is set with dimension d 1024. For the positional encoding and optimizer setting, we follow the implementation in Vaswani al. (207)). With above settings, train the model with 100 epochs, which takes 20.41 hours on one V100 GPU. The number of model parameters is 1.872 M.
8 parallel attention layers, the dimensions
the projected queries,
C DETAILS OF RANDOM GENERATED NAS MODEL GRAPH
Figure 10 shows example of NAS cell structures with random chosen topology. In Section 5.3 we perform experiments on a original NAS model searched by GDAS and five models with the same operation but random topology. The details about these 6 models are shown in Table
13

---

Published as a conference paper at ICLR 2022
concal
4
12
SC
90
3x
48
33
5
3655
dic
C+1
(a) Normal cell
(b) Reduction cell
Figure 10: An example of NAS cells with random topology.
Model
Normal Cell
Reduction Cell
((kap,, (5 [ 5SCouv, 0]}
[(5 x 5SCone, 1), (5 x 5SConr, 01
[[kkip. UJ. (3 3DSCone,
( x 5DSCoov 0). 3SConz, 2)
Original Model
(13 x 3avg pool, 0),(3 (3 x 3SCone, 3))
((3 x 3maz pool, (5, 5.Sonm, 2))
x 6.Soovv 33. x 3avg-pool,
[[5 5SCone, 3), (5 5SConv, 1])
((skip, 1). 5SConv, 0))
[[5 5SCona, 1), (5 5SConv, 0)
[skipp 1],
DDSCon, 2))
(6 5DSConv, 0). 3.SComo 1)]
Random Model 1
[13 3ang_pool, I), (3 35Conv, 0)
(3 x pomaar_pool,,r 33, 5.5oom,
[15 x 5.Coru, 0). 3avg pool,1)
[15 5.Sonu, 2), (5 5SConv, 0])
([skip, (5 5SConv,
[15 5.Cono 0), x 58Con., 1])
[[kiip 0], 3DSSonn
(5 5DSCCooou 0),
3.SCone, 2))
Random Model 2
[13 bawgg-pool. 2), (3 35Cone, 3)
(3 3max pool,
(), (5 5.Comw,
(15 x 5.Conm 2), (3 x 300g_pool,3)
(15 5SComm 4J. (5 x 555 0])
((sksp, 0). (5 5SComv,
[15 5SCono, (5 x 55Con,, 0])
[skiip, LJ. (3 3DSCono
(5 5DSSCOo; [5. 3SConm, 0)
Random Model
[33 3avg pool,
x 3.Conm,
(3 3mar pooaar.ppool.33 po.l,,), (5 5.Conn, 2))
(15 x 5.Corrv 4). (3 x 3avg pool,3)
[(5 5SCome, 2). (5 x 5SConv, 4))
((skip, 1), (5 5SConv, 0))
[15 5SCone, 1), (5 x 5SConv, 0))
[[ship, Oj, (3 3DSCome 2))
(5 x 5DSComm O), x 35Cone, 1)]
Random Model
[13 x 3evg. 1), (3 x 35Come 2))
(3 x 3mawx 1),
3SCome, 0)
(15 x 5.Con 0), x 3avg pool, 2))
(15 x 58Cmmo 4), (5 x 5SConu, 3])
((skip, (5 x 55Comn, 0))
((5 55Como, 0), (5 x 58Conu, 1))
[[kiip OJ, (3 x 3DDCCne, 2])
[33 x 5DSConv, 1). x SSCone, 0))
Random Model
[13 x Savg. 3), (3 x 3SCone, 2))
(3 x 3maxx pool, 2), (5 5SCone, 1))
[15 x 5SConn, 2), (3 x 3avg 1))
((5 x 5SConn, 2), (5 x 5SConv, I])
Table 4: Detailed cell structures of random NAS models. Each cell in model contains 4 nodes (4 layers) and each node has two inputs, i.e., (operations, node where input from)
D COMPARISON BETWEEN FIDELITY EXTRACTED NETWORK WITH THE ORIGINAL NETWORK
Figure 11 demonstrates example of the raw memory address trace of NAS cell. For the sake of simple understanding, we divide the whole trace into multiple blocks representing computational nodes in the cell. Each adjacent "read-write" pair in the addr, and then written back to addrg, where addr, and addrş can be either the same or differ- ent according the following usage of the data. By analyzing consistent between the read/write addresses, we can reveal the model topology with 100% precision. Figure [2 gives comparison between the architecture obtained by the fidelity extraction and the original network. As we dis- cussed in main text, while the monitored memory address traces allow the attacker to reveal the complete model topology precisely, the difference between the extracted model and the original one only determined by the predicted operation sequence and hyper-parameters. In Figure 12 (a), two operations 3DSC and x 5SC are marked as red, because the predicted operation sequences of them contain small errors, i.e., the repeated times of event labels (i.e., channel size) has slight deviations. But it simple to fix them based on the model context. Hence, we can finally reveal the complete model architecture.
means the data is first read from
14

---

Published as a conference paper at ICLR 2022
skip
Dead 262
3x3 DSC
Whon Cohneetseritt
Whete Cellpodds6600
2170000011400
Node-
3x3 DSC
3oo
Deat Cohstedeeeen
Whiw cchaeedeeem
Road
Road 3671800011994
36700b0001-89 4800
3x3 pool
Sx5
5
Skip
C07000000000
Road 007
(a) Extractd network with fidelity extraction
Road 567000011000
Whet 567000004400
skip
Wh (171
Doad 067hbnbaaokc0
3SC
Doad Adbibaok000
3x3 DSC
3x3pol
Read (b7heuutt) 4M4
Road 067108-001LF50
Doad 06279bb-991-40-33
Road 067188u061-998
Oel
3x3 pool
SC
SC
When (chhesentt-0003
Skip
20770000314300
(b) Original network
Figure 11: Raw memory address trace.
Figure 12: Comparison of fidelity extraction.
E EXAMPLE OF A RECURRENT CELL TRACE
Since machine learning privacy (Xu et al. 2020a b) also involves Recurrent Neural Networks (RNN), we also try to exploit the availability of NASPY on RNN models. Figure 13a shows an example of recurrent cell with DARTS. Figure Isb shows the corresponding side-channel trace for this cell. Each cluster is mapped to an activation function in the cell. We observe the side-channel trace has much more observable pattern, where each cluster corresponds to node in order. This makes it possible to predict the RNN operation sequence using our framework.
sigmoold 03
idvatty
00 relu
N₂
@ tanh
No
h.
relu
170000
N₂
L tank
@ relu
tanh
sigmold
NG
@ signoii
(b) Side-channel trace of a recurrent cell.
(a) Example of cell.
Figure 13: An example RNN NAS cells and trace.
15


Reviews:
[{"review_id": "yoN7IUCXjsY", "reviewer": "Reviewer_haYd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI agree to accept this paper. The model is novel, the idea is well-motivated, and the experiments are compelling.\n\nPros:\n1.\tA novel, neat, and effective learning-based framework for automated extraction of NAS architectures with high efficiency and fidelity.\n2.\tModel the extraction attack as a sequence-to-sequence problem, and design new deep learning models to predict the model operation sequence automatically.\n3.\tA new analysis method to precisely recover the exact hyper-parameters without any prior knowledge.\n4.\tDesign strategies to reconstruct the model topology and extract the complete architecture for different scenarios and adversarial goals.\n5.\tComprehensive empirical results to show the effectiveness of the components.\n\nCons:\n1.\tThe experiments mainly focus on extracting NAS models. However, in real-world scenarios, the variants of standard models (e.g., Resnet, Densenet) are widely used. Will NASPY still be effective on these models? Consider that, some hyper-parameters like the number of layers, channels, and stages may be varying.\n2.\tIt would be helpful to briefly introduce the background hardware knowledge (e.g., the meaning of the itcopy, oncopy, and time interval). Many of the readers may be unfamiliar with these details.\n3.\tI think that the experiments may need to include a comparison between the architecture obtained by the fidelity extraction attack and the original network.\n\nDiscussions:\nWould it be possible to extract the network architecture of detection and segmentation models? For example, the FPN or other detection/segmentation head.\n", "summary_of_the_paper": "This paper presents an end-to-end adversarial framework to extract the network architecture obtained from Neural Architecture Search (NAS), named NASPY. Previous works focus on extracting conventional ConvNets with simple operations, while requiring heavy human analysis. In contrast, NASPY introduces a seq2seq network to automatically identify complicated operations from the hardware event sequence. In addition, both the value of hyper-parameters and the exact model topology can be obtained with NASPY. Extensive experiments based on DARTS, GDAS, and TE-NAS demonstrate the effectiveness of NASPY.", "summary_of_the_review": "I agree to accept this paper. The model is novel, the idea is well motivated, and the experiments are compelling.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "qcZDwz9a1xr", "reviewer": "Reviewer_KWQS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I am not an expert of hardware-based attacks, but achieving 3.2% error rates for operation prediction seems quite impressive.\n\n2. Due to the lack of knowledge on hardware attacks, I did not fully understand what assumptions are made to achieve this kind of low error rate. Could you clarify the assumptions of these attacks?\n\n3. The authors claim that separable or dilated convs are complicated, but could you clarify what’s the key challenges to attach these operations?\n\n4. The paper claims to be focused on NAS models, but it is unclear to me why the proposed approach is specific to NAS models? Since it is just based on model inference (without accessing the search process), there should be no difference whether the target model is a NAS model or not? Please clarify this.", "summary_of_the_paper": "This paper proposes a hardware-based attack framework NASPY, which can extract NAS network architecture and hyper-parameters. The main idea is to use a seq2seq model to to identify operations from side-channel sequences. ", "summary_of_the_review": "1. Interesting approach and impressive results.\n2. Need more clarifications for a few points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PZEyzSPvZ8z", "reviewer": "Reviewer_pLvS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Significance\nThe topic in general and the contributions presented by the authors are highly relevant and of interest to a broad audience of ML researchers as well as practitioners.\n\n# Novelty\nTo the best of my knowledge the main aspects proposed in the paper are novel contributions.\n\n# Soundness\nThe paper seems to be sound. The claims are supported by a proper empirical study considering various scenarios.\n\n# Writing\nThe paper is very well organized and the writing is very clear. Especially, explanations regarding rather technical or hardware-related aspects are very well done and understandable.\n\n# Minor Comments\n- Capitalization of headlines are inconsistent.\n- The color schemes of figures should be adapted to be more inclusive (for red-green blindness). Especially, Figure 4 is very hard to read and distinguish between red and green for me.\n\n# Questions\n- What is the runtime and data complexity of the proposed approach?", "summary_of_the_paper": "In the paper \"NASPY: Automated Extraction of Automated Machine Learning Models\" the authors propose an approach to extract the structure and parameters of a DNN that has been constructed via neural architecture search. In an empirical study the authors demonstrate the effectiveness of their approach and show that by making some assumptions they can indeed successfully extract the architecture of DNNs as well as their parameters.", "summary_of_the_review": "All in all, the paper makes an important contribution to the area of security in machine learning. The proposed approach is able to successfully extract neural architectures and even their parameters, allowing for piracy of such models. Hopefully, this work will foster defense mechanisms in order to prevent such attacks. Therefore, I recommend to accept the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "2qoY94iKVRl", "reviewer": "Reviewer_L3iz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The area of NAS and attacks on NAS is an interesting area which is currently receiving quite a bit of attention. It is good to see work in this area.\n\nStrengths:\n- A more complete approach for attacking a NAS process.\n- A well constructed and well-written piece of work.\n\nWeaknesses:\n- The paper makes some rather bold claims, which seem to stem from the fact that they assume all NAS approaches work in the same way and are based around the same building blocks. This assumption is not correct and would make this the approach far less effective.\n- The predominant way to train Deep Learning and also for NAS is to use a GPU. However, the approach here seems to rely on the fact that all computations are performed on the CPU. This again would significantly reduce the effectiveness of the attack method.\n\nSome comments on the text:\n- The abstract and title are not good. Reading the abstract does not convey what this paper is about. Some text needs to be added to make it clear what domain this is in (i.e., trying to steal a network from a running NAS). Once you understand that the abstract starts to make sense.\n\n- The paper is over the 9 page limit. You have cheated by placing figures and tables in the supplementary material and then referenced them from the main body. If this were to be allowed people would start submitting 9 pages of text and put all figures / tables / etc into the supplementary material. This makes a mockery of having a page limit.\n\n- The footnote on page 1 needs either a citation or you need to provide your evidence to support this.\n\n- Your idea of what NAS is and the underlying techniques used for it is rather restricted. You assume that NAS is a process of building up cells and that these cells are formed from standard units. However, there are many other NAS techniques out there which do not follow this model. For example differentiable NAS approaches, Zero-shot NAS (which doesn’t do any training) and NAS approaches which only partly train networks. These other approaches would not (I assume) be susceptible to your attack. You should clearly articulate what is required for your approach to work. This can be extracted by thinking through the whole process, but it should not be the readers role to do this.\n\n- Likewise, the approach seems to assume that the whole process requires that you perform all of the NAS approach on the CPU. As the dominant approach for performing Deep Learning is to use a GPU there should be some discussion as to whether your approach would work in these cases. I’m guessing not.\n\n- NAS rarely returns the ‘optimal network architecture’ - in most cases it returns a ‘good’ architecture, but to find the optimal you would need to perform all possible architectures - and even then you would only know it was ‘optimal’ within your search space.\n\n- What do you mean by ‘a normal cell interprets the features’?\n\n- There is an assumption that NAS introduces new more complex operations - this is not true. The NAS can only provide operations that it has been told to use. \n\n- From the description of how you ‘grab’ the architecture from the NAS approach, I can’t see why this wouldn’t work if you were just training a network. Is this correct? If so say so, if not please explain why it wouldn’t work.\n\n- What is a ‘proxy dataset’?\n\n- A number of terms are used without being explained, including itchy, incopy, I_i, O_i\n\n- Given that to ’steal’ a network you need to train your system and then run it against trace logs captured from the running NAS it would seem that this whole process is computationally expensive. Especially as you still need to train the stolen network. Given this it would seem essential that there is some comparison between the cost of training the NAS and training your approach and using it. It may be the case that it would be cheeper to just run the NAS yourself.\n\n- Data augmentation is not clearly enough explained - can you add more details.\n\n- What do you mean by “we first introduce a convolution layer to extract more features”. Convolution layers process the features they are given, but they do not create new features.\n\n- There are some pretty big assumptions in this work. Such as P=R/2. This is not a requirement of performing NAS, so shouldn’t be a an assumption by you.\n\n- The last two paragraphs in 4.3 seem to only contain information presented earlier. Remove and use for something better.\n\n- You state that your approach can identify the hyper-parameters. However, it is only one type of hyper-parameters that you talk about. Those such as batch size, optimiser and learning rate don’t feature in this. I’m assuming that your approach does not cater for these - please make clear.\n\n- In terms of batch size this seems to be an easy way to thwart your approach. There is no discussion of batch size in your work, so one would assume that the work does not consider it. Therefore changing the batch size would change the size of the operations and hence break your trained model.\n\n- I’d want to see evidence to support the statement “Without loss of generality we adopt”. I can imagine cases where a different combination of libraries could throw your system off.\n\n- More details on the results of the fidelity extraction attack should be presented. It’s very tiny covered at the moment.\n\n- The conclusions are very short and don’t add a lot to the paper.\n", "summary_of_the_paper": "The paper presents a technique for performing an attack on NAS approaches allowing the technique to identify the neural architecture which the NAS is honing in on.", "summary_of_the_review": "The paper presents an approach to ’steal’ a DNN from a NAS training process. There are strong claims as to the applicability of the approach across all NAS approaches. However, the authors make an awful lot of assumptions as to how NAS works and the sorts of networks they can produce. If you factor in the NAS approaches they are not considering and the fact that the work seems to be focused on CPU training (which is not the norm) then the work has far more limited applicability.\n\nAlthough the work may work for GPU, there is no discussion in the text of this therefore it is not possible to assess if it would work or not.\n\nAs someone who reviews for security venues too, I’d imagine this work would score higher at one of those venues.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree on acceptance and I agree with them. I recommend a spotlight.
Decision: Accept (Spotlight)