Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission proposes a novel task related to NeRF: finding adversarial viewpoints that mislead visual recognition models. When searching adversarial viewpoints, they find search viewpoint distribution rather than one discrete viewpoint and suggest a novel optimization algorithm that fits their objective.
strengths_and_weaknesses: Strengths:

- It is the first work to tackle searching adversarial viewpoint on the NeRF-related methodologies.
- They suggest several effective and reasonable methods, searching the viewpoints via distribution, reducing memory requirements for computing gradients based on NES, and adopting reparameterization tricks to reduce the variance for computing gradients.
- The experiments are solid.
- It also shows the resistance to camera fluctuations, demonstrating its effectiveness in real-world scenarios.

Weaknesses:

- Their experiments are limited to background-free or simple background images
questions: Is it critical to the performance to use other ranges of the rotation angles? If so, please report the ablation of that.
limitations: This paper is limited to background-free or simple background images, which has room for improvement to use more realistic and complicated datasets such as the dataset suggested by MipNeRF-360.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose to study viewpoint robustness of image classification models using 3D objects generated from different viewpoints. A new method to find adversarial viewpoints based on NeRF is proposed, and used to create a benchmark (ImageNet-V) for evaluating viewpoint robustness for image classification. The paper evaluates a set of image classifiers covering standard architectures on ImageNet-V, training objectives, and data augmentation strategies, which finding that it is a challenging dataset compared to using natural views only. 
strengths_and_weaknesses: Strengths
- To the best of my knowledge, generating adversarial viewpoints of real objects using NeRF is new. The advantages mentioned by the authors in lines 147 to 160 looks reasonable.

- The authors perform fairly extensive evaluations on their proposed benchmark for evaluating viewpoint robustness (ImageNet-V), covering standard architectures, training objectives, and data augmentations.

- The paper is well-written with clear figures.

Weaknesses
- Using only 100 objects for benchmarking seems to be on the smaller side to me - maybe the authors can discuss more on the limitations of potential bias in this dataset due to the smaller amount of objects? Seems from the Appendix that all models are from BlenderKit - how were the objects selected?

- How does the 3D viewpoint change studied here compare to simple adversarial 2D transformations (rotation, scaling, cropping)? For example, the street-sign in Figure 2 looks simply rotated. The experiments would be stronger with some comparisons in this direction.

- The results in Figure 5 is not really significant without Figure C3 (which is in Appendix). My understanding is that Figure C3 is the performance under natural viewpoints & Figure 5 is with viewpoint robustness - I recommend that the authors present the bars in C3 with Figure 5, otherwise it is difficult to know how much the performance of each model drops. 

- I feel the "in the Wild" phrase in the title is not really an accurate representation of this work - I would recommend that the authors reconsider this framing (perhaps "Adversarial" should be somewhere in the title instead). The benchmark consists of images from generated view points with a white background (ex: samples in Figure C2), which doesn't look "in the wild" - as a result, I'm not sure if this title is accurate. 
questions: I listed some questions above in weaknesses. 

Some clarification questions:
- How dependent are the adversarial viewpoint / classification results on the NeRF training? 

- In Figure 4, real-images taken from adversarial viewpoint, why is the lightning so different than the original (real from natural viewpoint)? 

- How are the models in BlenderKit generated? Are they real or synthesized objects?

limitations: Yes the authors included a discussion on limitations in Appendix C. 

However, I'm not familiar with Royalty Free licensing from BlenderKit (which this benchmark uses assets from) and whether images using these models can be used in a paper/as a benchmark.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a technique to generate adversarial viewpoints of multi-view objects for training image classification models. It uses Neural Rendering as an implicit representation to differentiably optimize the viewpoint parameters to synthesize adversarial images. It introduces an entropic regularizer for optimizing a parameterized distribution which leads to higher diversity of content. 
strengths_and_weaknesses: Strengths: 
- The paper shows that Neural Rendering can be used as an implicit representation to create adversarial viewpoints for image classification. 
- A forward propagation technique based on Natural Evolution Strategies is proposed to bypass the expensive NeRF rendering process during optimization.

Weaknesses:
- The optimization problem is posed as an unbounded maximization function on classification loss and distribution entropy. It is not clear to me if it would work for general, real-world images where the model could diverge to nonsensical neural rendering.

- The results are shown on simple images with white background. Even the real-world dataset used in the paper is not representative of actual image classification datasets such as ImageNet. It is not clear if the method would generalize to real-world images. Recent datasets such as CO3D or Objectron could be easily used to demonstrate the efficacy of the method.

- The method presupposes availability of multi-view images of objects to train the underlying NeRF. It would not work for single shot images of objects. 

- Lacks comparison to other SoTA approaches for adversarial generation for training. 

questions: 1. The maximization function could lead to generation of highly skewed viewpoints. However, looking at the maximum likelihood estimates (v*), it doesn't seem to be the case. How do you constrain the viewpoints to remain within reasonable limits?

2. The gap between the Real(v*) and Rendered(v*) is significant. Why is rendered(v*) always harder to classify than the real(v*). Is it because of some artifacts introduced by the rendering technique. Can you perform some ablation studies to explain this gap? 

3. It is not clear if the NeRF weights are updated during training. 
What is the effect on the NeRF renderer after the adversarial optimization is performed. 
A comparison between the baseline PSNR (that's given in the appendix) and post-optimization PSNR could be a useful datapoint to see if the NeRF degrades in fidelity.

limitations: The limitations given in the appendix reasonably capture the issues with the proposed dataset and the baseline NeRF model. The authors should ponder over the limitations of the proposed technique. Perhaps the questions raised in the previous two sections can help guide the authors to find more relevant points.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the viewpoint robustness of trained visual recognition models. People have studied adversarial perturbations on 2D images for a long time, but not all of them are physically plausible. There were some previous efforts that tried to generate adversarial examples in the 3D physical space, but they used synthetic objects and rendering software. This paper proposes to replace these with NeRF, so that it becomes possible to conduct the same study on real-world objects. The authors conducted extensive experiments and demonstrated that trained visual recognition models can still be very vulnerable to simple viewpoint changes.

POST-REBUTTAL UPDATE:

Make no mistake: I really like and enjoy the paper. However, I am also concerned by the fact that the authors used models trained on ImageNet in their experiments. The big domain gap makes the random attack also very effective, and it is harder to put the proposed approach in perspective. At the very least, I was hoping the authors could train a model on their dataset, and put that in addition to the ImageNet trained one they have been using. In light of this, I have decided to slightly decrease my score from 6 to 5, which is still a positive score.
strengths_and_weaknesses: Strengths:
- I believe the problem being studied is an important one. We need to understand how models trained on selective examples (views) generalize to the generic setting.
- I especially like the experiments in Section 4.3, where the authors showed that it is possible to reproduce the adversarial viewpoints found by NeRF in the real world, and the attack can still be successful.
- Experiments are very extensive, covering a lot of different axes (architecture, with or without pre-training, various datasets).

Weaknesses:
- I have some concerns regarding presentation and baseline. See the questions section below.

Overall I think the originality, quality, clarity, significance of this paper are all above average.
questions: 1. In Introduction (say around L39), the authors criticized that previous papers / methods rely on synthetic 3D object models, implying that they are different. However, other than the experiments in Section 4.3, a lot of experiments (say those in Section 4.1) are using BlenderKit, which is still 3D models. Therefore I feel that the writing ought to be improved, otherwise the difference w.r.t. previous papers / methods (say [61]) could be perceived as exaggerated / overrated.

2. I am a bit concerned that the random search baseline in Table 1 also attack the model pretty successfully. By contrast, random search within the pixel RGB neighborhood on 2D images have a very low attack success rate. To me, this raises questions about the quality of the trained model. Maybe sampling 100 images per 3D object on the upper hemisphere is not enough. Maybe the training protocol is not well-tuned. My point is, if a model is full of weaknesses, then we were not going to deploy the model anyway, and the significance of studying its adversarial robustness becomes much smaller. Have you considered training a stronger model that cannot be easily attacked by random search? And is it possible to conduct the experiment on a state-of-the-art published and released checkpoint (so that the quality is guaranteed)?
limitations: The authors discussed some limitations in the Conclusion section, but I think they could be expanded. For example, the authors could have discussed whether or not the trained models are strong enough on these datasets newly proposed in this work. Another limitation worth discussing is that I feel the current work still largely fall into the "synthetic 3D object" category (e.g. white paper is added etc), and the NeRF-generated images here are still more like ShapeNet than ImageNet (or Block-NeRF for that matter).
review_title: null

Meta-review: metareview: In this paper, the authors study the problem of robustness in image classifiers – in particular the problem of adversarial robustness. Previous work in the field of adversarial robustness focused on identifying minimal non-realistic perturbations in pixel-space that maliciously alter the classification performance of a model. In this work, the authors constrain adversarial perturbations to the space of object and camera pose that lead to poor visual recognition performance. Importantly, the space considered is constrained to be physically plausible. They leverage recent advances in Neural Rendering (NeRF) to generate realistic 3D models of objects, and optimize for non-canonical poses that lead to poor predictive performance. Finally, the authors propose a new benchmark for evaluating viewpoint robustness (ImageNet-V) which may be used to assess the general quality of any image recognition system.

The reviewers identified several notable strengths including (1) the first work to identify adversarial viewpoint as a method for assessing robustness, (2) reasonable methodology for search and optimization, (3) solid and complete experiments. The reviewers did find some weaknesses in the presentation of the material and questioning details of the experimental setup but those points were largely addressed in the responses by the authors.

Given that robustness is a very large problem much larger than the topic of image recognition, I find that the problem the authors have identified is quite important to the larger community. My only suggestion is that it would be nice if the authors showed some analysis demonstrating a positive correlation between ImageNet-V accuracy and performance on other robustness measurements. That said, given the importance of the research topic and novelty of the approach, this work will be accepted for publication at this conference.

confidence: Certain
Decision: Accept