Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper looks at the interesting task of no-box attacks. This is a setting that is even harder than black-box attacks, where typically the attacker has access to either the training data and can construct a surrogate model or the model outputs to estimate gradients. In the no-box setting considered in this paper, the attacker only has access to a small number of samples (1k on ImageNet) that are correctly classified by the target model. The attack is based on feature space perturbations, as only the shallow layers of DNNs can be approximated with this little data and the authors propose the Error TransFormer to alleviate issues caused by approximations errors. 

They first train a Shallow DNN on the limited data pool. As the shallow layers do not yield a label output, instead of the standard adversarial objective used for FGSM/PGD attacks, their attack objective transforms the feature representation of an image into that of a given image that comes from a different class. To mitigate the difference between the target model and the surrogate model, they employ a min-max strategy that combines feature and parameter space. 
They evaluate their approach on various ImageNet source models. 


strengths_and_weaknesses: Strengths:

- The paper is well written and easy to understand. 

- The setting is very interesting. Assumptions from the standard black-box setting are often not realistic and the no-box setting is more applicable in real world scenarios. 

- Their method is simple and easy to implement.

- The evaluation is clear. Deep models can not be trained in the no-box setting, shallow models work much better but can be improved by their ETF approach.

Weakenesses:

- The novelty is somewhat limited. A similar min-max objective similar to ETF in weight space has for example been explored already in "Adversarial Weight Perturbation Helps Robust Generalization" by Wu et al. Most other ideas like surrogate training of a shallow model also appeared in previous papers. 

- The evaluation focuses on Imagenet only. While this is certainly an interesting setting, it might be worth to also add some experiments on other datasets. For example, while CIFAR10 is a much easier dataset, there also exist much more robust models and it might be interesting to evaluate the performance of the method to the most robust models for example from RobustBench.


- In Table 1-3, their should be another row for the ground truth robust accuracy in a white-box setting. While this is not computable exactly, AutoAttack should yield a fairly accurate estimate. 

- Table 1 is somewhat hard to read. I understand that boldfacing the single best result per column is not optimal as it does not make sense to compare different attacks, but it would be easier to understand the general message of the table, which clearly is that ETF is the best method in the no-box setting, by some form of highlighting. 
questions: - The paper only considers L-inf perturbations. While they are the most common, does the method work for L0, L1 and L2 too? 

- In 4.1 you say that the model has access to label information on the limited data pool. From the insufficient supervision information ablation, I also take that this is used during training. However 4.1 also motivates that for shallow attacks we only need to train shallow nets with an unlabeled contrastive loss. Which loss (Eq (1) vs (2)) is used for the actual training of the surrogate model? 

- You use heavy data augmentation, what is the impact of that and did you experiment with different ones? 
limitations: Obviously, attack methods can in general be used in a malicious way, especially if we go to more realistic scenarios. I think adding a sentence about that in C would make sense.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Recently, there has been a lot of focus on techniques for the black-box generation of adversarial attacks. However, in practice, an adversary might not have unrestricted access to the predictive interface of the target model or unlimited data to generate adversarial examples. In this paper, the authors propose Error TransFormer (ETF) for generating adversarial examples, which works even when the adversary doesn't have access to training data and model outputs. ETF works by constructing a surrogate model with the limited available data and then perturbing the features obtained from its shallow layers.
strengths_and_weaknesses: Strengths:
- There are relatively few techniques that can work under the threat model considered in the paper.
- Even though the technique appears simple, it performs pretty well in practice.


Weaknesses:
- The paper is a bit hard to follow, and the writing needs improvement.
- I found the experiments a bit lacking, and please refer to the next section for a list of questions I would like to be answered.

One vital information authors leverage is that we can generate powerful adversarial examples by attacking shallow layers. I would encourage the authors to add appropriate references. 
A vital piece of the technique is missing, like how is the optimization problem in Eq. 6 solved. I would encourage the authors to include such details in the main paper.
questions: - How imperceptible are the examples generated by the proposed technique? For instance, even though most black-box adversarial techniques have higher success rates, they generally forgo imperceptibility. 
- How good are examples generated by the proposed technique in evading the recent class of adversarial example detection methods?
- Does surrogate model architecture impact the success rate of the proposed technique?
- Does the technique also work in other domains like NLP?
- The surrogate model is trained in a contrastive manner. Can other self-supervision tasks like rotation be used to train it? (ref. Unsupervised Representation Learning by Predicting Image Rotations)
- For table 1, were the surrogate model trained using labels or in a contrastive manner?
- How useful are the examples generated by ETF in improving the robustness of the models?
- In addition, as the performance of the surrogate is tied to the samples used for training it, I would encourage the authors to run the experiments on different sample sets and report the variance.
limitations: I would encourage the authors to list the limitations of the proposed approach.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a "lightweight" black-box attack method that uses limited data called Error TransFormer (ETF). Experimental results show that the proposed method achieves decent performance compared to other black-box attacks with few available examples.
strengths_and_weaknesses: Strengths:
+ The method is novel to me, and technically sound. 
+ The performance is good. It is interesting that only 2 training images can still drop large accuracy.

Weakness:
- The writing should be carefully checked. I notice there are many errors that may lead the reader to significant confusion, for instance, the distance should be *maximize* instead of *minimize* for generating adversarial examples in line 173, the similarity should be *decrease* in line 234, the first row in Table 1 has some error, and it should be $\epsilon=0.1$ in the caption of Table 1. Hope the authors check the paper carefully and fix these errors.
- The authors only test under $\epsilon = 0.1$, it is more convincing that perform some experiments under smaller $\epsilon$, such as $\epsilon=0.05$.
- It is more convincing that perform experiments on more examples since the size of only 1,000 examples is too small.
- It is better bold the best results in *all tables*.
- The experiment setup should be more detailed.
questions: I have no questions for now.
limitations: I didn't find any potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies lightweight black-box attacks with limited knowledge of the target model and its output.
The author proposed an Error Transformer(ETF) to alleviate the approximation error between the surrogate model and the target model.
The experiment results show that using ETF + limited samples is only 3% attack success rate lower than the black-box attack with full access of the training data.
strengths_and_weaknesses: Strengths:

 1. The experiment results are compelling; this paper shows some interesting results that a black-box attack can be performed without using the training set.
 2. To my best understanding, using the shallow layer & ETF for approximation of the target model is novel to the ML security field.
 3. The motivation of the proposed method and the lightweight black-box attack are well illustrated.
 4. The author conducted a thorough analysis and ablation study in section 5.

Weaknesses:
  1.  I suggest the authors can show at least one visual example/noise generated by their method.
  2. Please refer to the questions.
  
I suggest the authors add more explanations to their equations to make them easier to follow. Overall, I think the experiment results are good and the motivation is clear. Therefore, I tend to vote for acceptance of this paper and am willing to change my score after the author clarifies my questions.
questions: 1.  According to eq. 4, is the Error Transformer simply learning one more transformation matrix $A$? If it is, then is $\Delta_s = Ax$? Or does it shows that the feature space perturbation at the first layer is equivalent to data space perturbation(when using $Ax$ as noise)?
2. In Section 4.3, line 185, the author said, "Therefore, connecting the parameter space ..." is there a more intuitive explanation for it? It is clear that eq. 4 shows it, but I don't think it is intuitive with the description in lines 183~184.
3. In section 4.3, lines 199~200, the author assumes there exists a transformation matrix $A$, $s.t.$ $w_t^1 = w^1 + w^1A$. How strong is the assumption here? What if the target and surrogate models have different dimensions at the first layer?
4. In the eq. 6 and 9, what does $\Delta_s$ means? I am unsure if there is a difference between $\Delta_t$ and $\Delta_s$.
5. Where is the $x^{\prime}$ defined in the paper? It looks like the perturbed version of $x$ in eq. 3, and it looks like source $x$ in eq. 6.
limitations: The author addressed the social impact in the appendix.
review_title: null

Meta-review: metareview: The paper presents a new method for generating black-box attacks with very limited data, i.e., the no-box case. The attack is based on feature transformations and the paper proposes error transformers (ETF) to alleviate issues with approximation errors. The reviewers believe the paper is technically solid and raised issues mainly to do with clarity and experiments. The authors provided a rebuttal and updated paper that thoroughly addressed those issues. All the reviewers raised their scores (including one that said to do so but did not update it). A good contribution to the field of adversarial learning. 

confidence: Certain
Decision: Accept