Introduction
Testing computer vision models is a challenging endeavour. In order to claim the superiority of a model, the computer vision community usually studies validation accuracy on the ImageNet [ 12 ] dataset. Comparing models uniquely using Top-1 and Top-5 accuracy on this dataset can result in an incomplete evaluation of the advantages and disadvantages of each model. Recently, Vision Transformers (ViTs) [ 15 ] have been proposed as an alternative deep neural network model to rival Convolutional Neural Networks (ConvNets) [ 39 ] for computer vision tasks.

ViTs have achieved impressive results, rivaling ConvNets and sometimes outperforming them in ImageNet accuracy. As it stands, if we restrict training data to ImageNet-22k, BEiT-L [ 5 ] , a ViT variant, stands at the top of the ImageNet-1k leaderboard, with ConvNext-XL [ 43 ] , a ConvNet, as a close second. Although the competition between these two classes of architectures has not yet been decided, there have been high-profile studies of potential advantages of ViTs compared to ConvNets. ViTs are believed to be more robust to certain adversarial attacks compared to ConvNets [ 59 , 2 , 8 ] . Although, recent work has also shown that this is not necessarily the case when ConvNets adopt the training recipes of ViTs [ 4 ] . Also, ViTs are believed to be more robust to domain generalization than CNNs [ 49 , 4 ] and against occlusion, as described in Naseer et al. [ 47 ] . This last work shows that ViTs are more resistant than ConvNets to different types of patch removal from images, which is a way to simulate occlusions. One limitation of this study is that patch removal is a convenient but not fully realistic proxy for occlusion. It is natural to use such a proxy since datasets to study the effects of occlusion on object recognition and detection are scarce.

A more general limitation to works that compare properties of ViTs and ConvNets is that, even though they try to compare models of similar sizes and ImageNet accuracies, they do not account for the fact that the compared ConvNets use slightly out-of-date design and training recipes . In their impressive work, Liu et al. [ 43 ] propose a new class of ConvNets with modernized architecture design that seeks to closely resemble the design of ViTs, yet only uses convolutional layers. This work allows for a closer inspection of whether Transformers are superior to ConvNets due to the difference in inductive biases between transformer and convolutional layers.

We believe that studying the differences that arise in learned representations between Transformers and ConvNets to natural variations such as lighting, occlusions, object scale, object pose and others is important. A priori, convolutional and transformer layers have different inductive biases that should manifest themselves in different performance characteristics. For example, there have been conjectures that ViTs should outperform ConvNets with respect to partial occlusion since the transformer layers allow for early capture of long-range dependencies.

Until now, there have been two main obstacles that prevent the careful study of these differences: (1) Transformer and ConvNet architectures were not comparable in terms of overall design techniques and training recipe details, entangling these differences with transformer vs. convolutional layer differences (2) there is a scarcity of datasets that include fine-grained naturalistic variations of object scale, object pose, scene lighting and 3D occlusions, among others.

We propose an attempt to bridge these two obstacles, and strive to tackle the fundamental question:

Between Transformers and ConvNets; which of these models is more robust to naturalistic scene variations such as object pose, object scale, camera viewpoint, lighting and occlusions?

In order to overcome (1) we compare the ConvNext [ 43 ] convolutional architecture to the Swin [ 42 ] Transformer architecture. Naturally, our contributions lie in our answers to obstacle (2). For this, we propose a method to test a computer vision architecture in a counterfactual manner using simulated images. We call this method Counterfactual Simulation Testing. Specifically, our method allows us to ask counterfactual questions to the models, ultimately providing answers to questions such as “Would your classification still be correct if the object were viewed from the top?” or “Would your classification still be correct if the object were partially occluded by another object?” . This allows us to abstract from the base rate of domain gap between synthetic and real images for each architecture and to compare them fairly.

Using Counterfactual Simulation Testing we find evidence of performance differences between comparable ConvNets and Transformers with respect to object viewpoint, camera viewpoint, object scale and occlusions. We observe that consistently across different network sizes ConvNext is on average more robust than Swin with respect to object pose and camera rotations . We also observe that ConvNext architecture usually outperform Swin architectures in terms of recognizing small scale objects. Aditionally, we find that Swin and ConvNext architectures are roughly equivalent in terms of robustness with respect to occlusion, with Swin pulling ahead of ConvNext for severe occlusion. Finally, we find that the robustness of both architectures suffers greatly from naturalistic variations of the test data - and that robustness improves with network scale and with data scale and variety. In order to find these differences we generate five different realistic synthetic test sets of objects using the MIT ThreeDWorld (TDW) [ 19 ] platform with these specific naturalistic scene variations. The full dataset, named Naturalistic Variation Object Dataset (NVD), contains 272k images of 92 object models with 27 HDRI skybox lighting environments in an indoor scene.

In summary, our contributions include:

Counterfactual Simulation Testing , a method to test computer vision models for robustness with respect to naturalistic scene variations in a counterfactual manner using simulated images by varying scene parameters one-at-a-time and evaluating the stability of predictions.

Naturalistic Variation Object Dataset (NVD) , a dataset containing 272k images of 92 object models with 27 HDRI skybox lighting environments in a kitchen scene with 5 subsets of naturalistic scene variations: object pose, object scale, 360 ∘ panoramic camera rotation, top-to-frontal object view and occlusion with different objects. We hope that this dataset will allow for evaluation of modern computer vision models with respect to generalization to the synthetic domain and robustness to the naturalistic variations contained therein. A sample of NVD is shown in Fig. 1 . We release this dataset to the public for use in benchmarking and architecture comparison.