Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Winning tickets can often be costly to discover, so this paper explores the issue of copyright protection for winning tickets -- protecting against IP infringement on the owners of a winning ticket. Namely, this paper studies “lottery verification” by analyzing sparse topology of a winning lottery ticket network with the use of several graph-based signatures (i.e., verifiable patterns that can be embedded into the weights/predictions of the winning ticket). Further, these attacks are robust to fine-tuning and extra pruning, which is extremely important for robust validation of winning tickets.While this area has been extensively studied for neural networks in general, this paper seems to be the first to study ownership verification for winning tickets/sparse neural networks. They propose three separate methods for ownership verification: key masks, embedded signatures, and trigger-based methods. Each of these methods are evaluated empirically on the CIFAR datasets using ResNet model variants.
main_review: I begin my review by emphasizing that I am completely open to author feedback upon my comments. My final score will be mostly based upon discussion with authors regarding my points outlined below.

General Opinion:
The paper has very good motivation and an interesting topic. But, I think it needs extra work before being accepted. The methods 2 and 3 of ownership verification (i.e., embedding signatures and adding triggers to the network) seem to lack novelty -- these approaches (from what I understand) are taken directly from work on ownership verification for neural networks in general and applied to sparse networks. However, the 1st method of ownership verification that decomposes the mask into separate groups seems to be novel and empirical results are solid. If possible, I recommend the following: (1) emphasize the novel ownership verification components more (i.e., method 1) and extend experiments in this area (2) extend experiments to consider more than one architecture/dataset (i.e., right now all experiments are CIFAR+ResNet) (3) consider different pruning approaches (e.g., structured, channel-based pruning). I believe with extra work this paper will be a good contribution to the community. 

Pros:
- The study of ownership verification based on the sparse structure/tology of a winning ticket seems to be novel. 
- Numerous possible approaches to verification seem to be explored (e.g., OMP and EWP for sparse mask verification, three different schemes outlined in Sec. 3.4). The most novel approach (i.e., that is not a direct extension from previous work) seems to be the V1 method, which specifically studies locking pruning masks from use without permission.
- Embedding the signature within the low-level convolutional layers is an interesting/clever idea.
- Analyzing the robustness to fine-tuning is very relevant/important, and I find this analysis very useful. The ablation section is well-done in general.

Cons:
- The writing of the manuscript seems like it generally needs to be improved. 
- Experiments seem somewhat limited -- only CIFAR+ResNet. But, different depths of ResNets are considered.
- For the V3 method, the performance drop on CIFAR100 seems somewhat significant (i.e., 1% performance decrease). Possibly this can be eliminated with some better tuning. 
- V2 and V3 seem to be direct extensions from previous work if I am not mistaken.


Questions:
- The definition of an “extremely sparse ticket” is a bit confusing, and I have never seen this anywhere. Was this just a definition created by authors? Or was this inspired by previous/similar definitions?
- Lines 172-173: In regard to these two ways of protecting winning tickets, it seems that the first method (i.e., method a) is invalid. This is because a “mask” for a winning ticket is useless without the associated weights initialization. This is shown throughout LTH literature that re-initializing weights and training the sparse network will not work well. So, why is there a need for two cases here and what is the difference between them?
- The pruning process that is used is somewhat unclear. Is this just unstructured weight pruning?


Minor Comments:
- There are grammatical errors throughout the paper that the authors should fix within the final version (e.g., lines 5, 7-8,  91-93, etc.).
- Related work never mentions structured vs. unstructured pruning. It only mentions pruning individual weights based on their importance.
- Lines 202-203: it would be nice to cite some work that shows the cost of training certain models here (e.g., https://arxiv.org/abs/2004.08900?source=techstories.org, or other work that references carbon footprints etc.)
limitations_and_societal_impact: The authors address limitations and there is no negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Paper proposes methods to protect the winning tickets of the LTH
main_review: Paper proposes methods to protect the winning tickets of the lottery ticket hypothesis. Being not an expert in the area (model IP protection and verification), my evaluation is based on my shallow understanding of the work. Paper is clearly written, and is easy enough to understand.

Results seems to show good performance after embedding where one doesnt loose more than 0.3% of the baseline accuracy (for scheme V2). There are a few grammatical and structural mistakes,

- On line 293, "Fig 2 show that results of directly re-train the model"
-  Table 4, there is no percentage inside brackets

From a philosophical point of view, I wonder if there can be discussions around the pros/cons of protecting such models, i.e. what does one gain by stealing the ticket?
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Three different approaches to embed ownership information in the topological structure of sparse neural networks (i.e. lottery tickets) and their resilience against specific attacks are presented.
main_review: Strengths:
+ Ownership verification of lottery tickets is a novel and relevant problem.
+ Three proposals to embed ownership information into topological structure are presented.
+ Their resilience against different types of attacks is tested.
+ Experiments on CIFAR-10 and CIFAR-100 showcase the applicability of the proposals.

Weaknesses:
- Ownership verification is not a topic that the average NeurIPS community members are familiar with. A more extended introduction that explains the basic concepts in more detail would be helpful. Especially the concept behind trigger sets could be better explained.
- All methods assume access to the training and test data. Often the real issue is sharing privacy sensitive data or claiming ownership of the data, because this data is the key to developing powerful ML models. 
- In the problem set-up: Is assumed that the training of neural networks easily feasible or not? This is a relevant question to judge the utility of the first proposal (with key and lock masks).
- On the one hand, already the knowledge of one of the masks could ease the training involved in finding a full lottery ticket because only the missing entries would need to be identified. This does not seem to be a very strong protection of the LT knowledge. 
- On the other hand, schemes that need retraining for verification seem to be computationally quite expensive.
- The second proposal limits the minimum sparsity of a lottery ticket.
- Couldn't another subnetwork of the lottery ticket also be confused for a QR code? Even a random one? 
Alternatively, what is the strategy if no subnetwork of the LT is similar enough to the encoded QR code?
This proposal seems to be limited by the actual topological nature of the LT. 

Points of minor critique:
- Some of the ideas have been proposed for deep neural networks in similar ways. 
- A couple of definitions are not precise and the s.
- The machine learning community has put a lot of effort into creating a culture of freely sharing results and trained models. 
In this context, ownership verification seems not as obviously important as the authors claim on p. 3.

Post rebuttal:
I acknowledge that I have read the reviews + authors' responses and have raised my score accordingly.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the intersection of two frontiers: machine learning IP protection, and lottery ticket model. Since lottery ticket is resource-consuming to find and practically useful, the authors proposed lottery verification to identify a found ticket’s ownership. 
main_review: This is a very novel paper with solid execution. Details comments follow:

Pros
-	The problem studied in this paper is brand-new and the authors’ idea is refreshing. High quality sparse masks are very useful but more expensive to obtain than training the dense model, so protecting its IP is an important yet underexplored topic.
-	Besides common approach like “trigger set”, the protection of sparse masks can benefit further from its sparse structural information, offering strong promise of mask IP protection. The authors further studied how to embed signatures in masks, that can be extracted and decoded even after pruning or fine-tuning attacks
-	The authors conducted a particularly thorough empirical study on several verification schemes, i.e., separate masks, embedding signatures, and embedding signatures with trigger set. 
-	They also consider attack types including removal, add-on and fine-tuning, covering possible manipulations of the sparse mask.
-	The protection results look strong on relatively large networks such as Res50. It is encouraging to see one relatively simple defense could resist all attacks considered.
-	Paper is well-written and easy to follow.
-	All codes were submitted in the supplement, seemingly easy to read and easy to use. The authors’ efforts are appreciated.

Cons
-	Experiments only on CIFR-10/100. While that’s fine as proof-of-concept, since their main argument is rooted in the expensiveness of finding tickets, showing that their method can scale up to larger datasets would be very helpful.
-	All attacks in this paper assumed stealing the mask associated with the original model, but that seems not necessary. Can knowledge distillation “steal” the function from the sparse tickets using another compact model?
-	Only limited transfer experiments are reported. How about the pretrain-then-transfer cases, e.g. in [25], can that mask be protected?

limitations_and_societal_impact: The impact is positive in general. The authors discussed in last section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Finding a special sparse subnetwork (winning ticket) from an overparameterized network is an expensive process. Due to the cost involved in obtaining this ticket, its owner might be interested in protecting it. This paper proposes graph-based signatures to perform lottery verification. Their method can work in both white-box and black-box settings. They show that due to the extremely sparse structure of the winning ticket, it can partly claim its ownership, without any specific protection mechanism. They give a mask embedding method to embed ownership signatures in the subnetwork. They claim such signature is robust to pruning, fine-tuning attacks. These claims are backed by experiments on ResNet models. 
main_review: This paper gives novel methods for ownership verification using the structural information of the pruned network. In contrast, most of the existing works are based on watermarking either the weights of the model or watermarking labels of a trigger set. One of the main ideas is based on the fact that  pruning/changing an extremely sparse network will lead to performance degradation, so the structure/mask of an extreme sparse network can be used for ownership verification. Overall, the methods are technically sound and backed by decent experiments. 

In context of current deep networks, pruning is very important to reduce training/inference costs. Pruning is expensive task but it is done once and the resultant subnetwork can be retrained again and again or used for inference. Hence it makes the protection of the subnetwork very important. Methods proposed in this paper are useful in such applications and their reliance on structural information seems to make them a better choice. 

The paper is well written. I think some detailed background on ownership verification, protecting the model etc. can be very useful in appreciating the results. 
limitations_and_societal_impact: Experiments are on one type of architecture (ResNets). I wonder if the claims would generalize to other architectures/ datasets/ tasks as well. Another thing that concerns me, is what are the odds that two people end up with similar pruned network, when they train+prune independently but use same open source code(random seed) and dataset?  How will such a situation be handled?
review_title: null

Meta-review: We thank the authors for this submission. Overall, the paper is about a topology-based ownership verification mechanism that can prevent lottery-ticket theft under various verification schemes and attacks.

The paper well-motivates the approach. The authors have provided extensive responses to the concerns raised and the AC + reviewers really thank them for their effort. Overall, the new results obtained during the rebuttal definitely improve the quality of the paper. We all believe that the inclusion of these results during the rebuttal period is something that does not heavily change the message of this paper.

There was discussion and consensus that this work is interesting. Having in mind issues/concerns raised by the reviewers, the main points of reviewers during further discussion were that this paper deserves publication, given the promised fixes by the authors during the discussion period.
Decision: Accept (Poster)