You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
Authors: Xuxi Chen
Tianlong Chen
Zhenyu Zhang
Zhangyang Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Lottery Ticket Hypothesis
Ownership Verification

Abstract: Despite tremendous success in many application scenarios, the training and inference costs of using deep learning are also rapidly increasing over time. The lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork (i.e., $\textit{winning ticket}$) instead of a full model for both training and inference, that can lower both costs without sacrificing the performance. The main resource bottleneck of LTH is however the extraordinary cost to find the sparse mask of the winning ticket. That makes the found winning ticket become a valuable asset to the owners, highlighting the necessity of protecting its copyright. Our setting adds a new dimension to the recently soaring interest in protecting against the intellectual property (IP) infringement of deep models and verifying their ownerships, since they take owners' massive/unique resources to develop or train. While existing methods explored encrypted weights or predictions, we investigate a unique way to leverage sparse topological information to perform $\textit{lottery verification}$, by developing several graph-based signatures that can be embedded as credentials. By further combining trigger set-based methods, our proposal can work in both white-box and black-box verification scenarios. Through extensive experiments, we demonstrate the effectiveness of lottery verification in diverse models (ResNet-20, ResNet-18, ResNet-50) on CIFAR-10 and CIFAR-100. Specifically, our verification is shown to be robust to removal attacks such as model fine-tuning and pruning, as well as several ambiguity attacks. Our codes are available at https://github.com/VITA-Group/NO-stealing-LTH.

You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
Xuxi Chen', Tianlong Chen',, Zhenyu Zhang", Zhangyang Wang' University of Texas Austin, Universiiyy of Science and Technology of China (xxchen, tianlong. chen, attteaaallawaannutttttteaa edu, zzy19969 ustc. edu, cn
Abstract
Despite tremendous success in many application scenarios, the training and in- ference costs of using deep learning are also rapidly increasing lottery ticket hypothesis (LTH) emerges as promising framework to leverage a special sparse subnetwork (i.e., winning ticket) instead of full model for both training and inference, that can lower both costs without sacrificing the perfor- mance. The main resource bottleneck of LTH is however the extraordinary cost find the sparse mask of the winning ticket. That makes the found winning ticket become valuable asset to the owners. highlighting the necessity of protecting its copyright. Our setting adds a new dimension to the recently soaring interest in protecting against the intellectual property (IP) infringement of deep models and verifying their ownerships, since they take owners' massive/unique resources to develop train. While existing methods explored encrypted weights or predic- tions, we investigate unique way to leverage sparse topological information to perform lottery verification, by developing several graph-based signatures that can be embedded as credentials. By further combining trigger set-based methods, our proposal can work in both white-box and black-bol verification scenarios. Through extensive experiments, diverse models (ResNet-20,, ResNet-18, ResNet-50) on CIFAR-10 and CIFAR-100. Specifically, verification is shown to be robust to removal attacks such model fine-tuning and pruning, as well as several ambiguity attacks. Our codes are available https: /gi thub. com/V /wtp/VIAA--A Rtt./VVIAAGGOU////e SaroopoNO--steeeeeee SettelinnggLiee
time. The
demonstrate the effectiveness of lottery verification
Introduction
Deep neural networks (DNNs) have dramatically raised the state-of-the. performance in various fields. However, the over pramtterizaztioonnn of DNNs becomes a non-negligible problem. The amount of parameters now is often on the billion scale, which significantly increases the inference cost when using these models. An emerging field of lottery ticket hypothesis (LTH) explores new scheme for pruning the model without sacrificing performance. The core idea is to identify the sparsity pattern ahead of training (or in its early stage) and train sparse network from scratch. It has been hypothesized [1] that DNNs contain sparse networks named winning tickets that can trained to match the test accuracy of the full model. These winning tickets hence have comparable or even better inference performance while potentially reducing the computational footprints.
However, finding winning tickets is non-trivial task: it involves the training-prunet wetttt-rruueeerrttee retrainng cycle for several times [1], which is burdensome and coapuuattio-consumiiii
Although other works [2-4] have shown that sparsity might emerge at the initialization at the early stage of training, the iterative magnitude pruning (IMP) still outperforms these alternatives by clear margins [5]. Yet, the powerful IMP method requires multiple rounds of train-prune-train process on the original training set, which
Equal Contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

is even much more expensive than training a dense network. That makes a found winning ticket valuable asset to the owners, highlighting the necessity of protecting the winning tickets' copyright.
Previous works [6-9] have shown that deep networks vulnerable to intellectual property (IP) infringement. For example, one can use transfer learning to adapt a trained model onto a new task or use model compres- sion techniques to create a new sparse model based on the target model. Fortunately, in recent years the own- ership verification problem has been addressed with a number of solutions proposed. The key idea is to embed verifiable information, or called signatures, into models' weights [6, 10, 11] or predictions [7] without visibly affect- ing the original performance. By extracting the embedded information from models, one can verify the ownership of models and hence protect their IPs. For the methods that embed information in weights, additional weights reg- ularizers are often used to enforce certain patterns, such signs. As for the prediction methods, special training set, which is often called a trigger set, is used as additional training data. The model trained upon both the original data and the trigger set generate desired prediction labels for the privately-hed trigger set, while preserving the performance on the original training set. However, those general methods did not take any structural property(e.g. sparsity) into account, leaving chance for improving their gains in the winning ticket mask protection.
Embed
Original
Signature
Illustration of embedding signa- tures into the original sparse mask. These visualizations are projected from 4D tensors. Dark entries are pruned elements. Note that the actual sparsity of the subnetwork is changed after encoding credentials.
Figure 1:
On protecting the IP of winning tickets, we investigate a novel way to leverage sparse structural information for ownership verification (Fig. 1). This structural information embedded winning tickets is a good "credential" for ownership verification since the winning ticket at extreme sparsity is naturally robust to fine-tuning and (further) pruning attacks. The winning ticket at extreme sparsity cannot be pruned further; otherwise, the inference performance will drop (hence losing its "value"). Meanwhile, fine-tune mun-tuning the winning tickets can only tune the weights, but the sparsity pattern will not be changed. However, there remain some key questions to answer: How to formulate the ownership verification process under the context of the lottery ticket hypothesis? What kind of structural information should be used? How to inject user-specific information into the structure of winning tickets? We present answers to these questions in this paper. We summarize our findings as follows:
We formulate the lottery verification problem and define two different protection scenarios. We show that even without specific protection, the extremely sparse winning ticket can partly claim its ownership because of the critical role of its sparse structure in the final inference performance.
We further propose a new mask embedding method that is capable of embedding ownership "signatures" in the subnetwork's 's sparse structural connectivity (Fig. 1), without much affecting its performance. The signature is robust, e.g., it can be extracted and decoded even after pruning or fine-tuning attacks. Combined further with the trigger set-based method, our mask embedding method can work under both white-box and black-box verification frameworks.
We investigate several verification schemes, i.e., separate masks, embedding signatures, and embedding signatures with the trigger set. We show that these schemes are robust to the common removal and ambiguity attacks, as well as a new type of "add-on" attacks. Extensive experiment results demonstrate their competence on protecting the winning tickets. For example, on ResNet-20, our verification framework can defend fine-tuning attacks intrinsically, well pruning attacks and as add-on attack under all levels of pruning ratios.
Related Work
Pruning and Lottery Ticket Hypothesis. Pruning algorithms can be roughly classified into two types: pruning after training and pruning before training. Conventional pruning after algorithms assign scores to individual weights and remove weights with the lowest scores [12]. There are
a large number of scoring algorithms of this kind, including weights magnitude [13, 14], Taylor
coefficients [15-18], and other variants [19, 20J. Pruning-before-traiiing methods also play important roles the community [1]. However, it requires multiple cycles of training and pruning process [12],
making it sometimes tedious in practical use. fruming-at-inttiaiizz methods alleviate such cost by

---

pruning initial weights such as observing initial gradients of weights [2, 3], but the quality of sparse networks found by these methods are mediocre in return.
A representative pruning-before-tee before-training method is the lottery ticket hypothesis [1], which hypoth- esizes the existence of sparse network in dense networks that can be trained to match the test accuracy of dense networks in isolation. It was later verified that the original hypothesis was too strong, and early rewind techniques [21] help scale up the original version. Later on, the lottery ticket hypothesis and its variants have been explored and extended to various fields [22-30] such as image generation [31, 27, 32] and natural language processing [22, 24]. However, it is currently non-trivial to find winning tickets, especially at high sparsity since multiple train-prune- train processes are required, which suggests the practical value of protecting the found sparse networks.
Ownership Verification. Ownership verification has drawn attention from both the industry and academia. Many works have been proposed to address IP protection. One most popular way is to use watermarking algorithm: [6] proposed embed watermarks in the form of bits into deep networks' weights by an additional regularization term. [10] embedded information into model weights by regularizing on the signs of weights. Besides watermarking on weights, [7, 33] embedded watermarks in the labels of certain examples in frigger set, which makes it possible to extract watermarks through a service interface without directly accessing the models' weights (black-box setting). Following somehow different pathways, [34] proposed a passport-basse basport--baedd approach that encodes signatures with special passport layers. [11] presented passport-emin repeooddd normalization whose parameters associated with signatures. However, none of these methods leverages structural information for ownership verification besides assuming general dense networks. For sparse networks, the sparse mask patterns represent the key information and can vary across models and tasks.
3 The Lottery Ticket Claims its Ownership
3.1 The Lottery Ticket Hypothesis
O Sparse Masks, Subnetworks and Winning Tickets. We define a neural network parameterized by W as N-W](-) With slight abuse of notion, we define a subnetwork of N[W] as N[W MM := N( @ M.(-) where M sparse mask whose shape the same as W but the value of each entry in which can only be either 0 or I. Given Wo the initialization of N. if N o. M] can be re- trained to match the test performance of the dense model training from N[W,] o). we call N NWW, o. M] a winning ticket. The term re-train above is used to distinguish between the training process to find the winning ticket. The criterion for matching can be set as, e.g., no lower than 1% than dense models' performance. @ Sparsity Comparison. The sparsity of sparse mask M can be defined as spar(M) := ||M]]]// ||M + lloo where || llo represents the non-zero values of the input matrix, and the relative sparsity can be defined as rspar(M, M2) == spar (/
(M2 + e). We call sparse mask M2 is sparser than M if and only if || Ilo < MM|| Ju- A sub-mask M2 is a mask that has the same shape M satisfying that all the elements in M M2 are non-negative. O Extremely Sparse Condition. Given a sparsity difference threshold t. we call N [Wo, M.] an extremely sparse winning ticket (or referred to as an extreme ticket hereinafter for conciseness) if N N(W,, n, M.) can match the performance of N [Wo,, o). but pruning the model (i.e., increase the sparsity of M.) & x 100% further cannot produce winning ticket. In our experiments, we set to be 0.01.
3.2 Verification Framework for Extremely Sparse Winning Tickets
A ownership verification framework for extremely sparse winning tickets can be formulated tuple V = (ME, W E, F, V, V)),, each item of which is a process:
A mask embedding process M (M,,, s) (optionally) for sparse masks. s is an optional string that be encoded into masks. The output of this process is new mask M. M can either be mask with s embedded or contains other structural information that is useful for ownership verification. We call the verification method with M E(Mo, s) enabled "mask-based" method.
weight embedding process W E(D.e. ce. s, NII.L, Wo. M). which is a learning process for the lottery ticket model. D., fx. y is the training dataset, T = (T., Ty } is an optional trigger set provided to the training process, s is an optional signature for the weights embedding process. L is the loss function for model training (usually the cross-entropy loss), N(.) defines the model structure, Wo is the initialization of weights, and M the sparse mask for the model. The output of E is model N with sparse weights W O M where W represents the trained weights. The
3

---

trigger set T (or/and) the signature are embedded in W @ after this process and can be verified with the verification process introduced next.
A fidelity evaluation process F(N W, M. Die. Ey) is to evaluate whether the performance discrepancy of model N[: is less than a pre-defined threshold e,, i.e., [A(N[W, MI, Dte) - A < ef. in which -(-,-) is the inference performance on the test dataset D Die, and A is a target inference performance associated with the model.
A verificattinn process V (N- W, M, 6,, checks whether the sparse mask M or the trigger set T can be successfully verified for a given model NI- For the mask-based methods, the process is to check if M and matches by evaluating N M on Die to see if the performance gap is smaller than a pre-defined threshold c,, and( extract information in M and compare it with s. For the trigger set- ea-d methods, an inference first executed on the trigger set images Ta, and then the prediction will be compared with trigger set labels Ty to see if the false detection rate is lesser than threshold es [34].
An invert process ( [W, MI, s) exists and will enable a successful ambiguity attack [34] if: a) of new trigger set T,, new signature s', new mask M' can be reverse-engineered for the given mask M and weights W. b) the forged TV, s' M' can be verified with respect to M and W. c) the fidelity evaluation outcome F(N W, M', Dte; At, cy) remains True.
The high-level definitions above general and can work with any concrete implementation. We will introduce several methods that use sparse structural information in the next following sections.
3.3 Structural Information As Signatures
Our motivation is originated from the nature of winning tickets that the sparse structure of winning tickets is critical their performance. As the sparse masks found by IMP outperform other pruning methods by clear margins [5], incorrect masks will lead to degraded test accuracy. In the next few sections, we demonstrate how to use the sparse structure of extreme tickets, i.e., both the sparse masks and weights, to perform ownership verification under different verification schemes. The ownership verification can be performed in two different scenarios: (a) protecting the sparse masks of the extremely sparse winning tickets; and (b) protecting the trained extremely sparse winning tickets.
3.3.1 Protecting the Sparse Masks: Splitting Signature from Sparse Model
Such sparse masks play a crucial role achieved outstanding generalization [1] and transferability [24, 25], and thus draws our attention to prevent them from being illegal distributed or used. Given fixed initialization, correct masks are essential for training the extremely sparse winning tickets to match the performance of the dense network. If we split the sparse masks into two parts, neither part is intact and correct so neither can be trained to match the performance of the dense model with the given initialization. Recall the mechanism of one lock can be unlocked by one key generally, adopt the concept of keys and locks and propose a new ownership verification method for the masks of extremely sparse winning tickets.
Denote the sparse mask of extremely sparse winning ticket by {Mi) and the weights by (WI) where N is the number of layers. To sparsify a model, (M, 1 is applied to the model's weight 1==
(Wi) by conducting an element-wise product (( ([W C MD Our goal is to find key masks i.e., sub-masks [M] }/= that contain as few elements as possible while the performance of the sparse network with the locked masks, i.e., the remaining masks, degrade as much as possible. Meanwhile, fewer elements in key masks reduce the cost of storing, distributing, and using the key masks.
We next describe the algorithms needed to discover key masks. An algorithm is used to split the masks of extremely sparse winning tickets (MM) ) into key masks [M and locked masks under the constraint of rspar( | [M: (M)))=> < T2g. n, is a hyper-parameter controlling the relative sparsity of the key masks. Score functions are used to decide which part should be split into the key masks. The pipeline is described in Algorithm 1.
We study several score functions in our experiments: One-Shot Magnitude (OMP): the absolute values of each weight; 2) Edge-Wads Weight- Product (EWP) [35] which measures the importance of paths from models' input to output. The EWP score is defined as the multiplication of weights along the paths; 3) Edge betweenness centrality (Betweenness). The edge betweenness centrality measures the importance of each edge inside a graph. For convolutional layers, we define the weight of each "edge" to be the summation of absolute values of each element; and 4) random scoring.

---

Algorithm 1: Splitting Key Masks
input :A sets of masks M = [M, initialization weights W = (Wi) N number of
non-zero elements n, and function score(.) for scoring.
output Key masks [M and locked masks {M; Mi
Derive the score matrices by applying score(-) over (W₂ @ Mi and get (S:
2 Set the values of entries in { to negative infinity if the corresponding entries at the same position in {W; @ M i=1 is zero (i.e., already pruned).
Calculate the nhhh largest number across the score matrix {Si)i= and record it as T.
Set M < /M,T and let the key masks be {M²} /==1- The comparison between M, and T
(M, T) is performed element-wise.
3.3.2 Protecting the Trained Tickets: Embedding Signature into Sparsity Masks
Another scenario is protect the trained extremely sparse winning ticket since superior performance on certain large-scale datasets usually comes with a huge economic and ecological cost. Although directly splitting the masks provides solution to the ownership verification problem, it has some brawb It delivers extra cost to users since they need to recover the masks. Such a method is also intrusive and requires additional responsibility from the users' side for storing the key masks safely. To render the extreme tickets capable of self-verification and free of key masks. we propose a novel pruning method that able "absorb" secret information (e.g., signatures) into models' sparse masks. The core concept is to enforce the sparsity masks to follow certain "0-1" patterns, which can be extracted from masks and further decoded back the original form of information.
function encode(.) is used to transform a string s into a matrix M, E {0, 1) xda which we call signature mask. Our goal is to embed encode(s) into the sparsity masks {Mi One critical question is where to embed the signature mask M,. Empirically, low-level convolutional layers are less sparser, which means they are more unlikely to be pruned. Therefore, information embedded in the low-level convolutional layers is more difficult to be removed if using the pruning method. Based on such observation, we decide to embed M, in low-level convolutional layers. To minimize mask changes, we first find a region in }M}} with the highest similarity with M, and tune the sub-mask of that region. For masks that have a dimension of two, we directly replace the region with M.; for masks that have a dimension of more than two, we raise their dimension by using random connections. Our detailed workflow is shown in Algorithm 2.
The choices of function encode 1) are various but there is one common choice in our daily life: QR code [36]. QR code has multiple advantages: 1) QR code is naturally seen pattern with only zeros and ones; 2) QR code has the ability to correct the error if the code is dirty or damaged. For example, the H correction level can tolerate up 30% of error [36]; 3) QR codes can be small in size which can be easily fit into sparse masks. The size of the QR code generated can be as small as 21 x 21 while the numbers of channels in convolutional kernels in deep learning models are typically greater than 21, showing an abundant space for fitting the QR code in inside models' sparsity masks; and 4) The QR code without the finder, alignment and version patterns are imperceptible when fitted into sparse masks since there are no "regular" patterns left. Based on these merits, we choose encode(-) to be the QR code generation function. Specifically, the encode function we use will return a QR code without finder, alignment, and version patterns. When extracting the code, the above patterns will be added back for decoding the credential information behind the QR code.
Algorithm 2:
Embed Signature Into Sparse Masks
input :A set of masks M = { signature s output:
A set of masks with signature embedded (M₃) Calculate M, <- encode(s).
2 Squeeze each M, into a two-dimensional matrix M/ by setting (M/) );5 = M[(M) 1oo00-
3 Calculate the similarity (percentage of matched 0-1 patterns) between each M// and M. and
name the one with the largest similarity M\ Change the dimension of M, and fit into Mf the region where the similarity is the largest.
max"
5

---

3.4 Ownership Verification with Sparse Structural Information
Next, we propose three different verification schemes based on sparse structural information, as summarized in Table A9. Under our unique context, we further introduce a new Add-on Atlacks which aims to create ambiguity against lottery verification by "recovering" several pruned weights and manipulating the sparsity patterns. More details can be found in Appendix A1.
Scheme V1= Distribute the extreme tickets with key masks. Scheme V1 is designed to protect the sparsity masks. We separate the sparsity mask M into two parts: M1 and M, where l/s subscripts denote "large"/"small" respectively. The small mask is sparser than the large one, which is used as the key mask while the large counterpart is the locked mask. We apply these two masks on weights and get two separate parts (W C M1, W O M.). Before re- re-training, legitimate users should merge the two weights by adding them up to recover the original sparse weights W @ M. The ownership can be automatically verified by the inference performance since an incorrect provided mask-weight pair will deteriorate accuracies after re-training.
Scheme V/z Embed signatures in sparse masks. We apply the signature mask embedding method to embed credentials into the extreme ticket. Then we train the model and dispatch it to legitimate users. No further action is required at the users' side. For the verification process, one can use extract the signature from the sparse model and validate the ownership of the extreme tickets. Compared with Scheme V1. Scheme V/2 is more user-friendly since no extra action is performed at the users' side. The application scenarios of Scheme V and V/2 are also different: the latter focuses on protecting the trained weights. It also shows great defense ability towards removal and ambiguity attacks. However, this scheme works under the white-box verification setting only, which means that access to models' weights has to be assumed. To overcome that assumption, we combine Scheme 1/2 with a trigger set-based method and propose Scheme V/3 in the next section.
Scheme V3: Combining trigger set-based methods. Scheme V/3 is more sophisticated than Scheme V/2 as a set of trigger images and labels are used during the (re-)training process With the help of this trigger set, Scheme V/3 is now capable of black-box verification. By using remote calls of service APIs, the owner can first probe and claim the ownership in black-box regime and further request a white- box verification if the black-box mode has raised a red flag. The white-box verification part for Scheme V/3 remains the same as Scheme V/2
4 Experiments
In this section, we will list the details of our experiments and show the results to prove the effectiveness of our ownership verification methods, as well as the robustness to removal attacks (e.g., model pruning, fine-tuning, and add-on attacks) and ambiguity attacks (e.g., fake paths, fake code).
General Settings. We three networks architectures (ResNet-20, ResNet-18 and ResNet-50) and two benchmarks (CIFAR-10 [37] and CIFAR- 100 [37]) in our experiments. For all experiments, we follow the same (re-) )training and testing protocol. The optimizer we use an SGD optimizer with a momentum factor of 0.9 and a weight decay factor of le-4 for ResNet-20 and ResNet- 18, and 5e-4 for 50. We train the model for 182 epochs with an initial learning rate of 0.1, and we decay the learning rate at 91 and 136th epoch by 0.1. We use late rewinding technique that rewinds the weight to the 3rd checkpoint. More results on ResNet-50 are deferred to Appendix A2. Our experiments are run with 16 pieces of NVIDIA RTX GeForce 2080 Ti.
Types of Attacks and Trigger Set. We study two types of attacks: 1) removal attacks. This type of attack aims at removing the embedded watermarks from the model's weights or data. Available methods for removal attacks include pruning, which removes proportion of parameters of the model, and fine-tuning, which performs training on new data for a few steps. Both methods can modify the weights and potentially make the watermark undetectable. 2) ambiguity attacks. This type of attack aims at confusing the verification schemes, i.e., no one can tell which is the real watermark/signatuun This type of attack needs techniques like reverse engineering and does not have a certain form. We explore several attack methods in our paper.
The trigger we use is the same as the trigger set used in [7], which contains abstract images that are different than the training images.
New type of attack: Add-on Attacks. Although the extremely sparse winning tickets are naturally robust to fine-tuning and pruning attacks due to their unique properties, the verification schemes based on the sparse structure will potentially suffer from another kind of attacks, i.c., trying to "recover"

---

some pruned weights and change the sparse structure of the extremely sparse winning tickets. We name it add-on attacks. Such new attack type targets mask-based verification schemes and aims at creating ambiguity against verification.
We propose a pipeline defending against such attacks. We can first prune weights whose magnitudes are smaller than t. t is known to the owner of the model since the owner has the authentic sparse masks. This can detect any noise with magnitude smaller than t, that the attackers add the mask. For noises of moderate level, their magnitudes become comparable to with the benign weights, hence the prediction quality will be significantly degraded as the noise increases.
4.1 Finding Extreme Tickets
To find extreme tickets, multiple rounds of the rrin--ruue-retrain process are usually re- quired. Once the test performance of the cur- rently trained model cannot match the perfor- mance of the dense model, we revert the prun- ing process back for one time and reduce the pruning ratio. The choices of pruning ratio of weights are [0.2, 0.1, 0.05, 0.1]. The results are shown in Table 1. We also report the prun- ing specification, which includes the remaining weights of the extreme tickets, as well as the pruning times for each pruning rate choose.
Table 1: Performance of dense models and extremely sparse winning tickets, and the pruning specification. The performance are expressed in terms of the test ac- curacy of the dense model and the extremely sparse winning ticket. The pruning specification includes the proportion of remaining weight as well the number of pruning with four different pruning ratios (in brackets).
Performance
Pruning Specification
ResNet-20 RexNet-20
CIFAR-10
91.67% 91 99.66. .666%
19.369% (5,18.1) 19.901% (6,0,4.7)
CIFAR-10
66.36%..66.37
ResNet-18
CIFAR-10
93.67% 60%
1.236% (18,3.0.6)
ResNet-18
CIFAR 100
72.44%%
2.251% (17.0.0.0)
Notice that ResNet-20 needs at least times of pruning before we identify the extreme tickets, and for R8sNet-18, such number increases to 17. Numerous pruning rounds exemplify the effort to find the extreme tickets and emphasizes the importance of protecting them.
4.2 Effectiveness of Different Schemes
Scheme V To verify that extreme tickets without correct key masks will have degraded performance after retraining, conduct experiments that directly re-train the models without the key masks ResNet-20 and ResNet-18. 8.. Fig. show the performance of extreme tickets after retraining. It can be seen that more "I"s in key masks can increase the performance divergence. Different splitting functions do not make an essential difference, showing that the choice of score functions is flexible.
72.5
70.0
Aocuraor
30
50
67.5
181
Typ4
0
1534
7yp0
1/2
5
Randon
Baoden
Resoo, CITAR-10
Reso CIEAS-100
Bost8, CHBAR-10
Bosts, CHPAR-102
Figure Effectiveness of Scheme Vi: Re-training without key masks generated by four methods: Betweenness,
EWP, OMP, Random (Paths). The z-axis is the relative sparsity w.r.t the extreme ticket.
Scheme V/2 To show that our proposal can embed information into models' sparse structures without significantly harming their performance, we conduct experiments to compare the performance before and after a signature string is embedded. Table 2 shows the test accuracy of two different models. We can see from the table that the performance of extreme tickets with the string embedded is only slightly lower than the original one, which proves that using Scheme Vi endows owners the ability to embed information at little cost of performance.
Scheme V/3 We use a set of trigger images during the re- re-training of extremely sparse winning tickets under Scheme V3. The inference performance on the original task (i.e., CIFAR-10 or CIFAR-100) should not be greatly affected with trigger sets enabled. Table 3 shows the inference performance on both the original images and the trigger set. We can see that the performance only drops 0.2% on CIFAR-10 and 0.97% on CIFAR- 100 for ResNet-20, while the detection rates of the trigger set images are high (91.0% on CIFAR- 10 and 90.0% on CIFAR-100), At the same time, the extreme tickets trained from CIFAR- 10 and CIFAR- 100 without trigger set can only have trigger set accuracies of
7

---

Table 2: Effectiveness of Scheme V2: performance of extreme tickets after embedding QR codes. We study two different models and compare their infer- ence performance. The performance after embedding and the performance drop are reported (in brackets).
Table 3: Effectiveness of Scheme Vs: ResNet-20 on CIFAR-I0 and CIFAR-100. ESWT is the abbre- viation of Extremely Sparse Winning Tickets. We re-train the two extremely sparse winning tickets with QR code embedded found with the trigger set enabled.
Accuracy after embedding
Model
CIFAR-10
CIFAR-I0 1000
Model
Test Accuracy
ResNet-20
91.37% (L 0.29%)
66. 14% (L 0.25%)
CIFAR-10
CIFAR-100
ResNet-18
93.56% (1 0.11%)
72.35% (L 0.24%)
ESWT
91.66% (16.0%)
66.36% (0.0%)
ESWT + M, + T 91.46% (91.0%)
65.39% (90.0%)
16.0% and 0.0%, respectively. This suggests the Scheme V/3 can work as expected, i.e,, perform well on the trigger set while not significantly harming the performance the original dataset.
4.3 Robustness Against Removal Attacks
Fine-tuning Attacks Fine-tuning the model can only change the values of weights while not changing the sparse structure of extreme tickets. As a consequence, Scheme 1/2 and V/3 is resistant to fine-tuning attacks under the white-box verification setting.
For Scheme V1, users are required provide the key masks to recover the correct masks and then re-train the extreme tickets. One key property we need to verify is that attackers cannot bypass the requirement of key masks by fine-tuning the model on a new dataset. To this end, we conduct transfer experiments described as follows: on CIFAR-100, we train the model with the locked mask generated on the extreme tickets identified on CIFAR- 10; on CIFAR-10, we conduct similar experiment with the locked mask from CIFAR- 100. The results are shown in Table 4. From the table, we can see that even transferring the sparse mask cannot bypass the requirement of key masks. The performance gaps between the transferred model and the extreme tickets found on each set are greater than 3% on both datasets, much higher than the 1% criterion we set for matching performance. Such big gaps prove that the model after fine-tuning attacks is not useful in practice.
Table 4: Fine-tuning Attacks on Scheme V1: Trans- ferring extreme tickets of ResNet-20 found on CIFAR- 10/100. 10-> 100 means transferring from CIFAR-10 to CIFAR-100 and vice versa. The percentage inside brackets denotes the relative sparsity of the key mask w.r.t the extremely sparse winning ticket.
Table 5: Pruning Attacks on Scheme V/: Perfor- mance of ResNet-20 CIFAR-10/100 after prun- ing with different pruning ratios. The accuracy on CIFAR-10/10 are shown outside the brackets and the accuracy on trigger images are inside the brackets.
Accuracy
Model
Model
Test Accuracy
CIFAR- 10
CIFAR- -00
10-1100
100-+10
Original model
91.46% (91.0%)
65.395 (90.0%)
Pruning 5%
91.33% (89.0%)
64.78% (91.0%)
OMP (5%)
59.80%
87.66%
Pruning 10%
90.66% (90.0%)
62.96% (73.0%)
EWP (5%)
60.27%
88.21%
Pruning 20%
87.86% (1.0%
50.14% (16.0%)
Betweenness (5%)
59.61 %
87.22%
Pruning 50%
33.04% (18.0%)
8.56% (0.00%)
We also have conducted experiments to study if Scheme V/3 can resist fine-tuning attacks under black-box verification. We first retrain the extreme tickets under Scheme V/3 CIFAR-10/-D 100, and continue to fine-tune it on CIFAR-100/-10. The extreme tickets trained on CIFAR-10 10 can only achieve 61.59% test accuracy on CIFAR-100, and the extreme tickets on CIFAR-100 can only achieve 88.21% % test accuracy on CIFAR-10. The strong bond between sparse structure (masks of extreme tickets) and datasets on which the extreme tickets we found brings performance drop when fine-tuning them on new dataset, which devalues such attack and also highlight the robustness of the Scheme V/s against fine-tuning attacks.
Model pruning Pruning the model under Scheme V1 is meaningless since pruning cannot recover the full masks. So we focus on Scheme 1/2 and V/3 for model pruning attacks. Pruning the trained model leaves more "0" in the trained model, which might change the extracted QR code and makes it unable to decode. To study if our model can resist the pruning attack, we conduct experiments with different pruning methods (one-shot magnitude and random pruning) and different pruning ratios (5%, 10%, 20%, 30%, 50%).
We first examine our proposal for black-box verification (Scheme V2). In Table 5 we show the results of Scheme V/3 against pruning attacks. The accuracy on trigger set images drops after the accuracy on the original dataset ICIFAR-IOCCIFAR. has decreased considerably, which means that the
8

---

user-specific information cannot be removed without sacrificing its performance and demonstrates its resilience against pruning attack.
We then test our proposal for white-box verification (Scheme V2). In Table 6 we show the inference performance on original datasets after pruning, and also show the QR codes extracted from masks in Figure 3. We can see that the performance of the pruned model will degrade dramatically after 20% percent of one-shot magnitude pruning and 5% percent of random pruning. On the contrary, the QR code extracted from the ResNet-20 can be decoded even after 20% percent of one-shot magnitude pruning. Figure 4 shows the QR code extracted from ResNet- 18. At the 5% pruning ratio, the string can be easily decoded into readable string. At the 10% pruning ratio, the string can still be partly decoded, although the readability has been reduced. For the pruning ratio greater than 10%, the inference performance has significantly dropped, making it meaningless to conduct such attack.
Table Inference performance extremely sparse winning tickets on ResNet-20 and ResNet- 18 after model pruning attacks under different pruming meth- ods and pruning ratios. OMP stands for one-shot magnitude pruning. The numbers in brackets stand for the pruning ratios.
Table Summary different types ambiguity attacks. We show the specification of each attack, Le., the accessibility of each component to attackers, the attack methods, and the targeted schemes.
Roge M.
Scheme
W
Add enlu Seylact M,
Schene
Method (Percent)
Performance
W M socodal)
Scbona ky seal lg
CIFAR-10
CIFAR-100
Table 8: Test accuracy and remaining weights after add-on attacks under different rates on ResNet- 20, with the matching condition and the decodc-abilits of the QR code extracted from the masks.
Scheme V/3
91.37%
72.35%
91.25%
72.27%
OMP (10%)
90.72%
71.42%
OMP (20%)
88.03%
69.51%
OMP (30%)
80.08%
60.31%
Add-on
Accuncy (% Temoi))
Decode-able?
OMP (50%)
36.62%
9.24%
91.53% (19. 369%)
Random Pruming (5%)
58.23%
0.5%
91.04% (19. 7895))
Random Pruning (10%)
30.499
22.67%
1%
90.23% (20.179%)
x
Random Pruning (20%)
11.95%
3.23%
2%
86.64% (21.00)%)
Random Pruning (30%)
12.05%
1.0%
79.49% (23.386 (363386)
Random Pruning (50%)
10.00%
1.0%
10%
71.06% (27 (025)
Original
Pruce
Prunc
Prune 30%
Pranc
Figure 3: QR code extracted from ResNet-20 under pruning attacks with different ratios. The codes extracted under 5% and 10% pruning ratio can be easily decoded into readable strings "signature", and the code under 20% pruning ratio can be decoded into "sigiature" with tools at https: /ttt::////tttb// com/merrice ex/ qrazybox/..
Original
Pruce 5%
Prune 10%
Prune 20%
Prune 30%
Prune 50%
Figure 4: QR code extracted from ResNet-18 under pruning attacks with different ratios. The code extracted under 5% can be easily decoded into readable string "signature", and the code under 10% pruning ratio can be decoded into "simçi @re"" with tools https: thub. www/merricceooooooooooooooc /razz/ob.x///////////oo///oo
4.4 Resilience Against Ambiguity Attacks
In this section, we will evaluate the robustness against ambiguity attacks summarized in Table 7.
Scheme 1/2 (fake;: Attackers can access W M, only) The goal of fake, is forge a new key mask M', with new underlying weights W'. As the attacker has no prior information on (W O M,). the forging process can only performed randomly. From Figure 5 we can see that such an attack method is not practical as the performance gap is much greater than c 1%) after using random key masks. For example, we adopt OMP as the scoring function to construct the key masks, we only need a key mask of around 10% relatively sparsity to make the model resistant random attacks.

---

181
Type
00.0
2ypt
1/20
Isfenctinn
BRTAHEERK
57.5
Randon
Raode
Recak, CIPAR-10
RaCak, CIPAR-102
Bee CIRAR-10
RaelR, CIPAR-103
Figure 5: Random attacks on Scheme V1. The x-axis is relative sparsity the key masks. The solid/dashed lines represent the performance before/after random attacks.
Scheme 1/2 and V/3 (fakeg: Attackers can access W @ M but not knowing encode One might use add-on attacks and try to "contaminate" the information we embed in the sparse mask. Specifically, we randomly add noises to the position where the weights are pruned. We test with add-on rates ranging from 0% to 10% since 10% efficiency gap will diminish the value attacking the model. The results shown in Table 8 and Figure 6. From the table, noise to the trained model will un-match the attacked model (i.e., the performance gap becomes greater than 1%). For the add-on rates smaller than 1%, the QR code embedded in the sparse mask can be normally decoded into a normal string. Such results prove that both Scheme V2 and Vi3 are resistant to attack fakez-
that introducing 1% of
Original
Add 0.5%
Add 1%
Add 2%
Add 5%
Add 10%
Figure 6: Visualization QR code extracted processed under different add-on rates.
(fakez: Attackers can access W M and encode(:)) If the attacker knows about the encode function for generating the M,, a similar but fake signature mask M. which contains different signature can generated in the same way. However, as shown in Figure 1, without the finder, alignment, hardly tell which part belongs to QR code. Even if the attacker
and version patterns, knows the position where the code emhedded (namely an insider attack [34]), directly replacing the embedded region with the performance of the attacked model since a large amount of "incorrect" weights are introduced. For example, for ResNet-20 CIFAR-IO, the test accuracy of the attacked model will drop from 91.37% to 57.00%, which is nearly a 50% degradation in performance. Such a big loss shows that it infeasible to perform the insider attack.
signature mask M', and W (noise) will also considerably degrade
Conclusion and Discussion of Broad Impact
LTH offers superior sparse models through burdensome explorations, serving intriguing yet expansive solution for resource-constraing applications. It motivates the necessity of protecting the copyright of these precious winning tickets. We investigate brand new verification technique by leveraging the sparse structural information, which embeds signatures into lottery tickets' typologies. Extensive results verify proposal's 's effectiveness and robustness against diverse malicious attacks.
This work scientific in nature and should bring positive societal impacts. Note that every second, giant and start-up companies have invested billions of dollars to identify superior yet fight-weight compact deep neural networks virtually. We believe our new lottery verification mechanism can assist both industry and academia in defending their interests from illegal distribution usage.
References
[1] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018.
[2] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity, arXiv preprint arXiv:1810.02340. 2018.
10

---

[3] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2019.
[4] Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Richard G. Baraniuk, Zhangyang Wang, and Yingyan Lin. Drawing early-bird tickets: Toward more efficient training of deep networks. In 8th International Conference on Learning Representations, 2020.
[5] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin, Pruning neu- ral networks at initialization: Why are we missing the mark? arXiv preprint arX5v:2009008576 2020.
[6] Yusuke Uchida, Yuki Nagai, Shigeyuki Sakazawa, and Shin ichi Satoh. Embedding watermarks into deep neural networks. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, pages 269-277, 2017.
[7] Yossi Adi, Carsten Baum, Moustapha Cisse, Benny Pinkas, and Joseph Keshet. Turning your weakness into strength: Watermarking deep neural networks by backdooring. In 27th [USENIX] Security Symposium (!USENIX) Security 18), pages 1615-1631, 2018.
[8] Jialong Zhang, Zhongshu Gu, Jiyong Jang, Hui Wu, Mare Ph Stoccklin, Heqing Huang, and Ian Molloy. Protecting intellectual property of deep neural networks with watermarking. In Proceedings of the 2018 Asia Conference Computer and Communccations Security, pages
159-173 1199172,220 2018
[9] Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Undistillable: Making a nasty teacher that cannot teach students. arXiv preprint arXiv:2105.07381, 2021.
[10] Bita Darvish Rouhani, Huili Chen, and Farinaz Koushanfar. Deepsigns: An end-to-end water- marking framework for ownership protection of deep neural networks. In Proceedings of the Twenty-Fourih International Conference on Architectural Support for Programming Languages and Operating Systems, pages 485-497 2019.
[11] Jie Zhang, Dongdong Chen, Jing Liao, Weiming Zhang, Gang Hua, and Nenghai Yu. Passport- aware normalization for deep model protection. Advances in Neural Information Processing Systems, 33, 2020.
[12] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in Neural Information Processing Systems, 33, 2020.
[13] Steven A Janowsky. Pruning versus clipping in neural networks. Physical Review A, 39(1))66600, 1989.
[14] Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
[15] Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in neural information processing systems, pages 107-115, 1989.
[16] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. Advances in neural information processing systems, pages 598-605, 1990.
[17] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
[18] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint 161106461100440 2016.
[19] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. arXiv preprint arXiv:1705.07565 2017.
11

---

[20] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui- mii- Lai. Vlad I Morariu, Xintong Han. Mingfei Gao, Ching Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages
2018.
[21] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pages 3259-3269. PMLR, 2020.
[22] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv, abs/1902.09574, 2019.
[23] Zhenyu Zhang, Xuxi Chen, Tianlong Chen, and Zhangyang Wang. Efficient lottery ticket finding: Less data is more. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 22380-12390 PMLR, 18-24 Jul 2021.
[24] Tianlong Chen, Jonathan Frankle, Shiyu Chang. Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks, 2020.
[25] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang. The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models. arXiv preprint 2tti:::022.060000 2020.
[26] Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in and nlp. In 8th International Conference on Learning Representations, 2020.
[27] Xuxi Chen, Zhenyu Zhang, Yongduo Sui, and Tianlong Chen. [GAN]s can play lottery tickets too. In International Conference on Learning Representations, 2021.
[28] Haoyu Ma, Tianlong Chen, Ting-Kuei Hu, Chenyu You, Xiaohui Xie, and Zhangyang Wang. Good students play big lottery better. arXiv preprint arXiv:2101.03255, 2021.
[29] Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, and Jingjing Liu. Playing lottery tickets with vision and language. arXiv preprint arXiv:2104. 1832, 2021.
[30] Tianlong Chen, Yongduo Sui. Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. arXiv preprint arXiv:2102.06790, 2021.
[31] Neha Mukund Kalibhat, Yogesh Balaji, and Soheil Feizi. Winning lottery tickets in deep generative models, 2021.
[32] Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Ultra-data-efficient gan training: Drawing lottery ticket first, then training toughly. arXiv preprint arXiv:2103.00397, 2021.
[33] Erwan Le Merrer, Patrick Perez, and Gilles Trédan. Adversarial frontier stitching for remote neural network watermarking. Neural Computing and Applications, 32(13):9233-9244, 2020.
[34] Lixin Fan, Kam Woh Ng, and Chee Seng Chan. Rethinking deep neural network ownership verification: Embedding passports defeat ambiguity attacks. 2019.
[35] Shreyas Malakarjun Patil and Constantine Dovrolis. Phew: Paths with higher edge-weight, give "winning tickets" without training data, 2020.
[36] Tan Jin Soon. Qr code. Synthesis Journal, 2008:59-78, 2008
[37] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
12


Reviews:
[{"review_id": "lmU-V3ghEty", "reviewer": "Reviewer_w6uB", "summary": "This paper studies the intersection of two frontiers: machine learning IP protection, and lottery ticket model. Since lottery ticket is resource-consuming to find and practically useful, the authors proposed lottery verification to identify a found ticket’s ownership. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a very novel paper with solid execution. Details comments follow:\n\nPros\n-\tThe problem studied in this paper is brand-new and the authors’ idea is refreshing. High quality sparse masks are very useful but more expensive to obtain than training the dense model, so protecting its IP is an important yet underexplored topic.\n-\tBesides common approach like “trigger set”, the protection of sparse masks can benefit further from its sparse structural information, offering strong promise of mask IP protection. The authors further studied how to embed signatures in masks, that can be extracted and decoded even after pruning or fine-tuning attacks\n-\tThe authors conducted a particularly thorough empirical study on several verification schemes, i.e., separate masks, embedding signatures, and embedding signatures with trigger set. \n-\tThey also consider attack types including removal, add-on and fine-tuning, covering possible manipulations of the sparse mask.\n-\tThe protection results look strong on relatively large networks such as Res50. It is encouraging to see one relatively simple defense could resist all attacks considered.\n-\tPaper is well-written and easy to follow.\n-\tAll codes were submitted in the supplement, seemingly easy to read and easy to use. The authors’ efforts are appreciated.\n\nCons\n-\tExperiments only on CIFR-10/100. While that’s fine as proof-of-concept, since their main argument is rooted in the expensiveness of finding tickets, showing that their method can scale up to larger datasets would be very helpful.\n-\tAll attacks in this paper assumed stealing the mask associated with the original model, but that seems not necessary. Can knowledge distillation “steal” the function from the sparse tickets using another compact model?\n-\tOnly limited transfer experiments are reported. How about the pretrain-then-transfer cases, e.g. in [25], can that mask be protected?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The impact is positive in general. The authors discussed in last section.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "jPWTxwpcukT", "reviewer": "Reviewer_s6wM", "summary": "Finding a special sparse subnetwork (winning ticket) from an overparameterized network is an expensive process. Due to the cost involved in obtaining this ticket, its owner might be interested in protecting it. This paper proposes graph-based signatures to perform lottery verification. Their method can work in both white-box and black-box settings. They show that due to the extremely sparse structure of the winning ticket, it can partly claim its ownership, without any specific protection mechanism. They give a mask embedding method to embed ownership signatures in the subnetwork. They claim such signature is robust to pruning, fine-tuning attacks. These claims are backed by experiments on ResNet models. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper gives novel methods for ownership verification using the structural information of the pruned network. In contrast, most of the existing works are based on watermarking either the weights of the model or watermarking labels of a trigger set. One of the main ideas is based on the fact that  pruning/changing an extremely sparse network will lead to performance degradation, so the structure/mask of an extreme sparse network can be used for ownership verification. Overall, the methods are technically sound and backed by decent experiments. \n\nIn context of current deep networks, pruning is very important to reduce training/inference costs. Pruning is expensive task but it is done once and the resultant subnetwork can be retrained again and again or used for inference. Hence it makes the protection of the subnetwork very important. Methods proposed in this paper are useful in such applications and their reliance on structural information seems to make them a better choice. \n\nThe paper is well written. I think some detailed background on ownership verification, protecting the model etc. can be very useful in appreciating the results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Experiments are on one type of architecture (ResNets). I wonder if the claims would generalize to other architectures/ datasets/ tasks as well. Another thing that concerns me, is what are the odds that two people end up with similar pruned network, when they train+prune independently but use same open source code(random seed) and dataset?  How will such a situation be handled? \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "MfR1kcfAOJy", "reviewer": "Reviewer_xVRk", "summary": "Winning tickets can often be costly to discover, so this paper explores the issue of copyright protection for winning tickets -- protecting against IP infringement on the owners of a winning ticket. Namely, this paper studies “lottery verification” by analyzing sparse topology of a winning lottery ticket network with the use of several graph-based signatures (i.e., verifiable patterns that can be embedded into the weights/predictions of the winning ticket). Further, these attacks are robust to fine-tuning and extra pruning, which is extremely important for robust validation of winning tickets.While this area has been extensively studied for neural networks in general, this paper seems to be the first to study ownership verification for winning tickets/sparse neural networks. They propose three separate methods for ownership verification: key masks, embedded signatures, and trigger-based methods. Each of these methods are evaluated empirically on the CIFAR datasets using ResNet model variants.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I begin my review by emphasizing that I am completely open to author feedback upon my comments. My final score will be mostly based upon discussion with authors regarding my points outlined below.\n\nGeneral Opinion:\nThe paper has very good motivation and an interesting topic. But, I think it needs extra work before being accepted. The methods 2 and 3 of ownership verification (i.e., embedding signatures and adding triggers to the network) seem to lack novelty -- these approaches (from what I understand) are taken directly from work on ownership verification for neural networks in general and applied to sparse networks. However, the 1st method of ownership verification that decomposes the mask into separate groups seems to be novel and empirical results are solid. If possible, I recommend the following: (1) emphasize the novel ownership verification components more (i.e., method 1) and extend experiments in this area (2) extend experiments to consider more than one architecture/dataset (i.e., right now all experiments are CIFAR+ResNet) (3) consider different pruning approaches (e.g., structured, channel-based pruning). I believe with extra work this paper will be a good contribution to the community. \n\nPros:\n- The study of ownership verification based on the sparse structure/tology of a winning ticket seems to be novel. \n- Numerous possible approaches to verification seem to be explored (e.g., OMP and EWP for sparse mask verification, three different schemes outlined in Sec. 3.4). The most novel approach (i.e., that is not a direct extension from previous work) seems to be the V1 method, which specifically studies locking pruning masks from use without permission.\n- Embedding the signature within the low-level convolutional layers is an interesting/clever idea.\n- Analyzing the robustness to fine-tuning is very relevant/important, and I find this analysis very useful. The ablation section is well-done in general.\n\nCons:\n- The writing of the manuscript seems like it generally needs to be improved. \n- Experiments seem somewhat limited -- only CIFAR+ResNet. But, different depths of ResNets are considered.\n- For the V3 method, the performance drop on CIFAR100 seems somewhat significant (i.e., 1% performance decrease). Possibly this can be eliminated with some better tuning. \n- V2 and V3 seem to be direct extensions from previous work if I am not mistaken.\n\n\nQuestions:\n- The definition of an “extremely sparse ticket” is a bit confusing, and I have never seen this anywhere. Was this just a definition created by authors? Or was this inspired by previous/similar definitions?\n- Lines 172-173: In regard to these two ways of protecting winning tickets, it seems that the first method (i.e., method a) is invalid. This is because a “mask” for a winning ticket is useless without the associated weights initialization. This is shown throughout LTH literature that re-initializing weights and training the sparse network will not work well. So, why is there a need for two cases here and what is the difference between them?\n- The pruning process that is used is somewhat unclear. Is this just unstructured weight pruning?\n\n\nMinor Comments:\n- There are grammatical errors throughout the paper that the authors should fix within the final version (e.g., lines 5, 7-8,  91-93, etc.).\n- Related work never mentions structured vs. unstructured pruning. It only mentions pruning individual weights based on their importance.\n- Lines 202-203: it would be nice to cite some work that shows the cost of training certain models here (e.g., https://arxiv.org/abs/2004.08900?source=techstories.org, or other work that references carbon footprints etc.)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address limitations and there is no negative societal impact. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0f2YfMUeX01", "reviewer": "Reviewer_xU5M", "summary": "Three different approaches to embed ownership information in the topological structure of sparse neural networks (i.e. lottery tickets) and their resilience against specific attacks are presented.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Ownership verification of lottery tickets is a novel and relevant problem.\n+ Three proposals to embed ownership information into topological structure are presented.\n+ Their resilience against different types of attacks is tested.\n+ Experiments on CIFAR-10 and CIFAR-100 showcase the applicability of the proposals.\n\nWeaknesses:\n- Ownership verification is not a topic that the average NeurIPS community members are familiar with. A more extended introduction that explains the basic concepts in more detail would be helpful. Especially the concept behind trigger sets could be better explained.\n- All methods assume access to the training and test data. Often the real issue is sharing privacy sensitive data or claiming ownership of the data, because this data is the key to developing powerful ML models. \n- In the problem set-up: Is assumed that the training of neural networks easily feasible or not? This is a relevant question to judge the utility of the first proposal (with key and lock masks).\n- On the one hand, already the knowledge of one of the masks could ease the training involved in finding a full lottery ticket because only the missing entries would need to be identified. This does not seem to be a very strong protection of the LT knowledge. \n- On the other hand, schemes that need retraining for verification seem to be computationally quite expensive.\n- The second proposal limits the minimum sparsity of a lottery ticket.\n- Couldn't another subnetwork of the lottery ticket also be confused for a QR code? Even a random one? \nAlternatively, what is the strategy if no subnetwork of the LT is similar enough to the encoded QR code?\nThis proposal seems to be limited by the actual topological nature of the LT. \n\nPoints of minor critique:\n- Some of the ideas have been proposed for deep neural networks in similar ways. \n- A couple of definitions are not precise and the s.\n- The machine learning community has put a lot of effort into creating a culture of freely sharing results and trained models. \nIn this context, ownership verification seems not as obviously important as the authors claim on p. 3.\n\nPost rebuttal:\nI acknowledge that I have read the reviews + authors' responses and have raised my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "0EByiMmrRv5", "reviewer": "Reviewer_YzBW", "summary": "Paper proposes methods to protect the winning tickets of the LTH", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper proposes methods to protect the winning tickets of the lottery ticket hypothesis. Being not an expert in the area (model IP protection and verification), my evaluation is based on my shallow understanding of the work. Paper is clearly written, and is easy enough to understand.\n\nResults seems to show good performance after embedding where one doesnt loose more than 0.3% of the baseline accuracy (for scheme V2). There are a few grammatical and structural mistakes,\n\n- On line 293, \"Fig 2 show that results of directly re-train the model\"\n-  Table 4, there is no percentage inside brackets\n\nFrom a philosophical point of view, I wonder if there can be discussions around the pros/cons of protecting such models, i.e. what does one gain by stealing the ticket?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: We thank the authors for this submission. Overall, the paper is about a topology-based ownership verification mechanism that can prevent lottery-ticket theft under various verification schemes and attacks.

The paper well-motivates the approach. The authors have provided extensive responses to the concerns raised and the AC + reviewers really thank them for their effort. Overall, the new results obtained during the rebuttal definitely improve the quality of the paper. We all believe that the inclusion of these results during the rebuttal period is something that does not heavily change the message of this paper.

There was discussion and consensus that this work is interesting. Having in mind issues/concerns raised by the reviewers, the main points of reviewers during further discussion were that this paper deserves publication, given the promised fixes by the authors during the discussion period.
Decision: Accept (Poster)