Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the robustness of different Imagenet pre-trained classifiers to changes in object scale, object pose, scene lighting, and 3D occlusion. The authors generate a large dataset of 272k synthetic images ("NVD") to do so, which is the central contribution of the paper. The 2 architectures primarily under study are ConvNext architectures and Swin ViT architectures. The authors first show that ConvNext models generalize significantly better to NVD than Swin Transformer models. The authors then show that -- when accounting for the different affects of the real-to-synthetic domain gap -- Swin Transformers are more robust to occlusion, but ConvNext models are generally more robust to variation in object scale, changes in viewpoint pitch and changes in viewpoint yaw.
strengths_and_weaknesses: I found this paper enjoyable and interesting to read. In my opinion, some of the results are very likely to be compelling to the ML community, such as the result that ConvNext architectures are much more robust to random patch drop and synthetic object occlusion than the older CNN architectures tested in prior work (Figures 3-4). The result that ConvNext architectures generalize significantly better than Swin architectures to the synthetic data is also an interesting finding, and counters prior work suggesting ViTs are better for OOD generalization (Table 1).

I have a few suggestions that I think would further improve the impact of the paper:
- In lines 124-149, the authors make the argument that their PCCP metric marginalizes out the differing effects of the real-to-synthetic domain gap in order to solely investigate performance under changes to object scale and viewpoint etc. I think this point is very important for the paper and some justification is necessary to support this claim. I'm not entirely convinced that this metric does enough to account for that gap - isn't it still feasible that how sensitive a model is to changes in viewpoint on the synthetic data does not correlate with the model's sensitivity on the real data? I'm wondering if the authors could maybe post-process the extent to which an object is occluded (maybe with a segmentation model?) in the Imagenet validation set and verify that the "performance vs. occlusion" curves on these natural  images matches the curves in Figure 4.
- To further improve the impact of the paper, I recommend that the authors additionally test how the curves in Figures 4-8 shift when models are fine-tuned on the synthetic data. In practice, if one finds a subclass of images where the model is performing poorly, the next step is to train on more images of that type. It's therefore very practically useful to understand if the difference in performance between the two architectures vanishes or widens when both architectures are fine-tuned for a small number of steps.
questions: - Figures 3-8 - to some extent, I feel the authors should keep the same y-axis limits across the subplots in each of these figures. Or at least provide an additional plot where all curves are all on the same plot? It would be good to understand better how differences between the two architectures scale with model size (e.g. are large CNNs and large ViTs more similar to each other than small CNNs and small ViTs?), which is hard to parse from the current plots.
- NVD - the authors should make it clear whether or not they plan to release to the public this dataset along with their code for computing metrics.
limitations: Yes, for instance the authors discussed that they primarily study generalization differences between ConvNext and Swin architectures which is not necessarily indicative of generalization differences between general CNNs and ViTs.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a comparative study of ConvNext and Swin Transformer (with comparable model size and GMACs, and design and training techniques) on synthetic images with controlled scene variations. To avoid real-to-synthetic domain gap, the authors propose to measure (relative) accuracy drop (or robustness) on synthetic images with varying scene parameters, i.e. the so called counterfactual simulation testing and the proposed metric called proportion of correct conserved predictions. Therefore another contribution is the scene variation dataset generated with ThreeDWorld simulator, where the detailed description of what type of scene variations are provided. The comparison results show the differences to object viewpoint, scale and occlusions.
strengths_and_weaknesses: Strengths
- Selection of ConvNext and Swin transformer is a fair choice for the comparative study than previous works, also tested SwinT v2 in the appendix.
- The proposed metrics largely avoids the real-to-synthetic gap.
- Careful choices of scene variations to occlusion, object pose, scale, camera poses, more comprehensive than previous studies. e.g. occlusion is better with occluder objects than random patch drop.

Weaknesses
- The performance gap on several variations (e.g. Fig 4, 6) are not significant enough to make a conclusion
- It would be better if further look into *learned features* similar to previous works on comparing ViT and ResNet to provide more insights on the difference, e.g. feature correlation, frequency, etc
- Dataset quality: although the dataset is called *Natural* variation object dataset, the synthetic images in Figure 1 do not look natural/realistic enough to me. Also scene background is relatively simple compared to real images. The number of object classes is rather small.
- Writing can be improved to be more concise and clearer.


questions: 1. The last column in Table 1 shows the top-5 accuracy when test on NVD. How about top-1?
2. Since all results on robustness metrics are based on top-5 predictions, and ConvNext is better than Swin Transformer on top-5, will most results on PCCP affected by this large difference or not? eg. Fig 4-7 almost all show ConvNext is better than SwinT with variations to those factors. How unstable the results would be if using Top-1?
3. Are the selected classes in NVD dataset representative in ImageNet?
4. Have you tried/what's your guess: will the results (table 1 and other Figs) transfer to real dataset with scene variations as well? (It would be much stronger if there is results on real validation data)
5. To clarify: for maximal occlusion (x=0), how much visible is the object of interest?
6. Can author put the concise main results/observations about the difference at the end of the abstract?
limitations: - related work (disclaimer: not related to myself) also looks at effect of scene parameter variations (in-distribution): Madan, Spandan, et al. "Small in-distribution changes in 3D perspective and lighting fool both CNNs and Transformers." arXiv preprint arXiv:2106.16198 (2021).
- please add a link to dataset generator and license in the main paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper compares the robustness of Vision Transformers to Convolutional Networks under different transformations such as scaling, object rotation, camera changes, occlusion, and random patch deletion. The authors resort to a synthetic dataset to perform these transformations. Specifically, they use the MIT ThreeDWorld scene generator. The main finding is that ConvNext is often more robust than Swin – except for the occlusion task.
strengths_and_weaknesses: Overall this is a fine paper. The authors selected the two similar architectures ConvNex and Swin. The comparison using a synthetic photo-realistic dataset makes sense. I also liked that differently sized networks were compared. The paper is well written and original as it is the first study comparing Convolutional Networks to Vision Transformers on these transformations. Studying the robustness is also a significant problem. The approach to rely on a synthetic dataset gives the authors a high degree of control and therefore experimental validity. However, its results are a bit predictable when you know the "Appendix B. Robustness Evaluation" from the ConvNeXt paper. There, it is reported that ConvNeXt generalizes better to different datasets (ImageNet-A/R/C/Sketch) than Swin.



A few improvements are:

- Figures 3-8 lack confidence intervals; including them is necessary to judge how substantial the differences are. For example,  they might invalidate the conclusion from in Figure 6 that "ConvNext contain a higher proportion of conserved correct predictions". I would recommend generating them using bootstrap resampling. 
- How large is the NVD dataset? 92 3D-models with 27 lighting conditions, thus 2484 images? Could you please include the number of samples in the paper? 
- The conclusion needs to be extended. It reads more like an abstract than an informative summary of the findings and main takeaways. Some parts of the Discussion are included in the Related Work section, which goes against the more common structure to combine it with the conclusion. 

My main criticism is the missing confidence intervals. Adding them might change the conclusion slightly, but I suspect that the overall story ("ConvNext models tend to be more robust than Swin models") still holds. Please add them as this would strengthen your analysis.

Additionally, it is unclear to me if the authors plan to make the NVD dataset accessible to the public? As they claim the dataset as a major contribution, I would expect this, but it is not stated explicitly in the paper. 

Although, I generally liked this paper, I would like to vote for it only with a weak accept for now due to the missing confidence intervals. I would be willing to upgrade the score if my concern is address accordingly.

Minor Points:

- The font size of the figures is tiny and cannot be read when printed. Maybe, you could save the figures in the pgf format with the correct figure size (see https://jwalton.info/Matplotlib-latex-PGF/). 
- Additionally, the y-axis of each Figure 3-8 should be fixed. For Figure 6, the y-axis changes from Tiny to Small, making it hard to compare them visually.


questions: These questions were also stated less explicit in the previous section:

- Do you plan to open source your NVD dataset and the corresponding code?
- How do the confidence intervals look for Figure 3-8?
limitations: The paper includes a limitation section, which I highly appreciate. The main limitations listed are the problem of how representative a synthetic dataset is and the difficulty of comparing different network architectures. It could also be mentioned that they only compare one specific training seed per network model, e.g., for each network architecture, only a single parameter set is used. However, I do not consider this a significant limitation as they included 5 different network architectures for Swin and ConvNext. Additionally, the authors might be excused as retraining them would be costly. Still, I wonder how much variation can be explained by the specific weight parameters.
review_title: null

Meta-review: metareview: After the rebuttal and discussion all reviewers are positive, and recommend acceptance. The AC agrees with this recommendation.
confidence: Certain
Decision: Accept