Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyzes the dataset inference mechanism from an offensive perspective. The paper starts with the theoretical analysis of the failure scenarios, such as the existence of false positives. The paper analyzes linear cases and then extends it to non-linear cases. The paper empirically demonstrates that there are indeed false positives when the victim uses a smaller number of queries. The paper also shows some potential cases of false negatives when the attacker employs adversarial training. The paper concludes with the possible limitations of the dataset inference.

strength_and_weaknesses: Strengths:

1. This paper suggests a false sense of security that the dataset inference mechanism may make.
2. The paper empirically shows some downsides of the dataset inference mechanisms.

Weaknesses:

1. It's a bit unclear why reducing the number of queries is important.
2. It's also unclear why the existence of false positives is the weakness.
3. It's further unclear why the attacker performs adversarial training to construct a model.
4. The paper has large room for improving the quality of the writing and the presentation.

Detailed comments:

I like the research question that this paper asks: "what would be the failure modes of the dataset inference?" which is the highlighted work in the last year's ICLR. The paper approaches this question theoretically and empirically, and it shows in the evaluation that there are such cases.

However, the major problem of this paper is that the failure modes that this paper exposes do not seem to be the actual weakness of the dataset inference. Moreover, some scenarios, such as an adversary training a model with adversarial training (for evasion), are misleading. Thus, I believe the paper is not ready to appear in ICLR 2023.

Here, I provide my detailed comments on the weaknesses.


[False Positives with Smaller Number of Queries]

If I understood the threat model correctly, the attacker trains a model with the stolen dataset, and the victim wants to identify whether the attacker uses the stolen data. In this case, the model trained by the adversary is already open to the victim; thus, I am a bit confused why we assume a smaller number of queries bounds the victim. Of course, reducing the number of required queries is good for efficacy. However, as long as the victim can query the model sufficiently, the dataset inference seems to work.

More generally, I imagine a scenario where two parties are at the court, and one party has to show that the other party's model uses the data extracted from the first party's model. In this case, unless the verification process takes days or years, we could allow the victim to do as many queries as possible. Running 50k queries on CIFAR10 only takes a few minutes (at most).


[Existence of False Positives]

I am a bit skeptical about the problem of false positives. Just having some false positives is insufficient to claim that the dataset inference is unsafe. I would expect some more strong evidence or claims, such as "even if the victim uses 50k queries, the false positive is still around 50%." It is, then, indeed a problem.

I like the results showing that if the number of private samples is large, the false positives become higher. But on second thought, it's somewhat trivial as the victim has to make predictions on many private samples for the ownership claim. Moreover, there is no reason why the victim wants to use a larger number of private samples. I would imagine if the victim uses a smaller number of private samples, then correctly estimating the membership of a subset of the entirety could be sufficient to claim the ownership.


[Use of Adversarial Training as a Defense]

The paper shows that an adversary can increase false negatives by training their model using adversarial training. This is confusing because I could not imagine a scenario where the adversary harms the accuracy of their model by running adversarial training. One of the purposes of stealing private data is to achieve state-of-the-art accuracy; then, running adversarial training contradicts the motivation.

I also assume that many trivial things could achieve the same as running adversarial training. One example would be running with DP-SGD. As the dataset inference is the membership identification, training a model with DP can reduce the inference success. BUT, the private model trained by the adversary cannot perform well; even for CIFAR10, it achieves ~75% accuracy with the epsilon 7-8.


[Minors]

1. The citation is not compatible with the ICLR; "Deng et al. (2022)" is better "[Deng et al., 2022]."
2. Many notations appear over reading; better to summarize by creating the "Notations" section.
3. Sec 2 largely depends on the setup proposed by Maini et al. and omits many details; for example, "what are the embeddings?" or "what is the white box (MinGD)?" It would be nice to re-write the entire section so that readers without prior knowledge of the dataset inference can understand the details clearly.
4. It is a bit difficult to understand the paragraph starting with "The authors ..."; it should be re-written.
5. The introduction contains sufficient related work, so I wonder if the paper needs more in Sec 6.


clarity,_quality,_novelty_and_reproducibility: I included the evaluation above.
summary_of_the_review: The paper studies the weaknesses of the dataset inference mechanism. I agree that exposing a protection mechanism's weaknesses is essential to understand its possible risks. However, the weaknesses this paper shows seem not to be the actual weaknesses of the dataset inference. I can safely imagine that those weaknesses are not any concern in practical scenarios. Thus, I am leaning a bit more toward rejecting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work does a deep dive into dataset inference, an up-and-coming ownership verification technique for machine learning models. The authors show how DI suffers from a nontrivial FPR in both theoretical and practical settings and FNRs in practical settings with simple adversarial modifications. These analyses highlight how DI can falsely accuse trainers of stealing data, while at the same time failing to detect actual stealing adversaries.
strength_and_weaknesses: ## Strengths

- Discussion on False Positives starting with the theoretical analyses drives the point home, showing that the flaw is critical to the approach itself, and not just an empirical loophole explored by the adversary. All the proofs are detailed, thorough, and easy to follow.

- Analyses are thorough, with proper visuals and numbers (with error margins, to highlight variance and stability of results), as well as realistically-sized models and datasets. All the figures are very helpful in understanding the points being made with their aid.

- The inclusion of False Negatives is very useful and much needed; any adversary that is malicious enough to steal data would not stop from going a step further and evading such detection techniques. It definitely opens up the sub-field to the same kind of cat-and-mouse game that adversarial machine learning is subject to, which isn't necessarily a bad thing (and can be good in fact).

## Weaknesses

- I'm not convinced about the applicability of Theorem 2. It states that False Positives occur with P at least 0.5, but this is true when _one_ sample is used for DI, right? Assuming $n$ samples are used, wouldn't the expected FPR bound decay with $2^{-n}$?

- Section 4: to account for potentially perturbed inputs, can the party executing Blind Walk not simply adjust to a higher step size (perturbation)? At least some form of adaptive defense should be evaluated for a complete picture.

- I have some concerns with some of the reductions, especially in Appendix B. They can probably be fixed by showing more intermediate steps, but would like some clarification from the authors:

  - In (11), $c=m\cdot u\cdot y^2$ ; since this has a $y$ term, how can it be treated as a constant?
  - In (12), $c$ from (11) becomes $y\cdot y$ (which makes me wonder if (11) has a typo), and the $y$ term outside the summation (present in (11)) seems to be missing here.
  - "Note that in Equation 12..." here assumes that each $y_i=1$, which may not be true- should the term here not be $D\sigma^2\sum_{i} y_i$ instead?
  - In (13), the same $yc$ term is used, but would it really be the same? 
  - Assuming $\Phi$ in (18) refers to the Normal CDF: 0.9 here seems like an arbitrary choice. Also, is $\Psi(-\frac{1}{\sqrt{10}})=0.38$? Not sure where the 10 or 0.9 appears from this.
clarity,_quality,_novelty_and_reproducibility: __Clarity__: The paper is very well written, and the authors' thought process is easy to follow, making the paper very easy to follow and read.

__Quality and Novelty__: This work tackles a very practical concern regarding the practicality and downsides of dataset inference, which has been proposed (and has also had follow-up work) as an ownership resolution method. The theoretical analyses are insightful and highlight the problem with DI, vis-a-vis false positives and false negatives. Not all aspects of the research are "novel" per se (like the technique for false negatives) but overall I think it is impactful work.

__Reproducibility__: Standard datasets and models are used, along with fairly simple computations, but it would be nice to include an implementation (with exact seeds) for perfect reproducibility. 

### Minor Comments

- Please fix the citation style. In most places, references that should be of the form .... (Author et. al.) are instead ... Author et. al, with the citation part of the text itself, making it confusing in places while reading.
- Section 1. """...via its inference interface" -> "...via its predictive interface". Prediction is not the same as inference. (see [this](https://www.datascienceblog.net/post/commentary/inference-vs-prediction/) blog for a good explanation).
- Above Theorem 2: "Thus, we can expect that the.....to be similar". This statement seems far-fetched: similar losses do not automatically imply similar decision boundaries.
- Dataset Inference has also been extended (with improvements) to the case of unsupervised learning [1]. It is a relatively new paper (so okay to not have included it in the initial submission) but might be worth looking at.
- Section 4: "...and is trained the same way as $f_\mathcal{V}$" do both use adversarial training?
- Section 5: "...constitutes a privacy threat": I disagree. The central authority involved in the ownership resolution process is trusted (which is why it does the resolution in the first place, and models are released to it after all), and __can__ be trusted to not misuse this data.
- Section 5: "....that can cause significant overhead." - how much? Given the conflict in ownership, the overhead just might be worth it.
- Table 3 (notations) should be in the main paper if the authors are able to find the space to fit it.
- In (18), please introduce $\Phi$ before using it

#### References

[1] Dziedzic, Adam, Haonan Duan, Muhammad Ahmad Kaleem, Nikita Dhawan, Jonas Guan, Yannis Cattan, Franziska Boenisch, and Nicolas Papernot. "Dataset Inference for Self-Supervised Models." arXiv preprint arXiv:2209.09024 (2022).
summary_of_the_review: The paper makes some very good arguments about the validity and utility of dataset inference, with both theoretical analyses and empirical experiments that bring to light the existence of false positive and false negative cases, which can be harmful in practical scenarios. Apart from a few minor comments and some hiccups in the proofs (which I think might be a mix of some typos and unexplained intermediate steps, so nothing that can't be fixed), I think it would be a good addition to ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper analyzes the reactive defense against model stealing called dataset inference that was proposed at ICLR 2021. It is shown in this submission that dataset inference suffers from false positives (FP) and false negatives (FN). For FPs - it is presented that dataset inference can incorrectly resolve the model ownership when the independent model is trained with non-overlapping data from the same distribution as in the victim model. For FNs - it is claimed that dataset inference can be fooled when the stolen model is trained using adversarial training. 
strength_and_weaknesses: Strengths:

1. The authors provide a check of the defense technique proposed at ICLR 2021.
2. The paper provides both theoretical and experimental analysis.

Weaknesses:

1. Dataset inference is a statistical and not a deterministic method so the existence of FP (False Positives) and FN (False Negatives) is not surprising. The only important aspect is under which conditions such false alarms occur. Note that the ownership resolution is done through "statistical hypothesis testing, which takes the false positive rate $\alpha$ as a hyper-parameter and produces either conclusive positive results with an error of at most $\alpha$, or an 'inconclusive' result."
1. Regarding False Negatives - Adversarial training is more expensive thus stealing with this approach is not done. Additionally, adversarial training usually lowers the performance of a model (even in the experiments in this paper the authors report the drop in accuracy by 6pp) - which is why an adversary would not use it - the stolen service would be worse than the service exposed by competition - the victim. In Table 2, the accuracy values should be reported, as mentioned at the end of Section 4.
2. The assumption: "(2) a large proportion of the victim’s training data is used during ownership resolution/verification" can be fulfilled also by assuming a third-trusted party - an arbitrator - which is a realistic assumption for an ownership resolution (when considered as a court case). Note that revealing any data to an adversary might enable the adversary to retrain their model on the revealed data or simply claim ownership of the data. The approach with private or oblivious inference is also feasible, however, as pointed out - it is computationally intensive (hardware-based solutions might alleviate this overhead).
4. It is not shown how to obtain a false negative when a model is stolen from a private API by other means than decreasing the number of samples used for the ownership resolution. With respect to Section 5: "False negatives vs white-box theft.": What changes does an adversary have to make to fool dataset inference in the white-box setting when the victim model is simply copied by an attacker? For example, how should the adversary change the outputs from their stolen model to avoid detection by the dataset inference? 
7. False negative is based only on the case where an adversary steals the private dataset but not when the adversary steals a model behind a publicly exposed API.
8. In Section 3.2.2 it should be also shown how dataset inference performs depending on the number of samples used for tests. The authors used the number of samples $k=10$, which is "extremely low", as stated at the end of Section 3.1 in this submission. A similar Figure to Figure 1 should be provided in Section 3.2.2. The authors showed a single/corner case instead of thoroughly analyzing the spectrum of possible cases for dataset inference. It is mentioned at the end of the caption for Table 1 that FPs become more significant as $k$ increases. This should be shown experimentally, at least in the appendix. The probability of an FP should be shown as the fraction (from 0 to 1) of revealed private samples (similarly to Figure 1).
9. Blind Walk and MinGD are two ways to estimate the prediction margin introduced in the dataset inference paper. For example, Tables 1 and 2 should be run for many more samples than $k=10$ and also with MinGD (not only with Blind Walk).
10. The assumptions for the theoretical results in Section 3 are different than those in the original paper. These should be stated explicitly. 

clarity,_quality,_novelty_and_reproducibility: Clarity:
- There is room for improvement. Related work is very similar to the introduction. These two parts of the submission should be merged and the related work should be placed after the introduction - not at the end of the paper.
- In the introduction (3rd paragraph), a few defenses are mentioned from 2022 and then the authors claim that an attack from 2020 by Atli et al. can circumvent all of them. It looks as if it had been shown that the proposed attack in the paper from 2020 circumvented the defenses from 2022. 
- In Figure 1, what is the linear suspect model 6? Is it from equation 6?

Neat:
- "many extraction attacks have querying patterns that are distinguishable from the benign ones queries Juuti et al. (2019);" remove ones or queries
- D, $g_v$ (distinguisher) should also be defined in Table 3. 

Quality:
- The paper requires much more thorough experimentation. 

Novelty:
- The novelty is limited since false positives and false negatives are expected for dataset inference.

Reproducibility:
- The authors did not submit the source code.
summary_of_the_review: The paper requires more experiments, for example, to consider the whole spectrum of the number of data points used for dataset inference in the case of neural networks. The assumptions in the theoretical part are different than in the original work. Additionally, no suggestions are made that could improve the dataset inference defense.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thank you for your submission to ICLR.  While there was still ultimately no consensus on this paper, there was active discussion during the the AC-reviewer meeting, which identified a number of both strengths and weaknesses of the paper.

This paper points out a number of possible issues with Dataset Inference, a technique proposed in last year's ICLR.  While the paper does not claim that DI is "broken", they do identify a number of potential issues with the approach, allowing for the possibility of either substantially more false positives *or* false negatives than was illustrated in previous papers.  Papers showing limitations of past approaches are crucial to the advance the field, and thus this paper has potentially a great deal of value to the community.  However, during the review period and the AC-reviewer discussions, the reviewers raised a number of substantial questions that raise real concerns for the work.

Specifically, several of the reviewers were quite familiar with the dataset inference problem and attempted to reproduce the results from this paper, but were unable to see the same effects as documented in this paper.  Specifically, but increasing the step size, the problem of false negatives for adversarial models and of false positives seems to be substantially alleviated.  While these points were not discussed explicitly during the rebuttal period, my inclination would be that these elements should not play a huge factor.  However, because the authors explicitly did not submit source code with the paper, issues of reproducibility are quite important, to understand if these are real effects or if these are merely due to one particular choice of hyperparameters.  Papers that explicitly point out limitation in existing work are especially crucial in this setting, and it is unclear why code couldn't be included with this work.

Given the ultimate disagreement here, I strongly recommend that the authors resubmit with code included, in order to better validate that the results presented here really do hold.
justification_for_why_not_higher_score: This paper in particular seemed to really suffer from lack of included code, as the reviewers were unable to replicate the results, and didn't have a reference solution from which to base their approaches on.
justification_for_why_not_lower_score:  NA
Decision: Reject