3DB: A Framework for Debugging Computer Vision Models
Authors: Guillaume Leclerc
Hadi Salman
Andrew Ilyas
Sai Vemprala
Logan Engstrom
Vibhav Vineet
Kai Yuanqing Xiao
Pengchuan Zhang
Shibani Santurkar
Greg Yang
Ashish Kapoor
Aleksander Madry
Venue: NeurIPS 2022 Accept (2022)
Keywords: robustness
debugging
simulation
computer vision
deep learning

Abstract: We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation.  We demonstrate, through a wide range of use cases, that 3DB allows users to discover vulnerabilities in computer vision systems and gain insights into how models make decisions. 3DB captures and generalizes many robustness analyses from prior work, and enables one to study their interplay. Finally, we find that the insights generated by the system transfer to the physical world. 3DB will be released as a library alongside a set of examples and documentation. We attach 3DB to the submission.

3DB: A Framework for Debugging Computer Vision Models
Guillaume Lecleret LECLERC @ MIT. EDU MIT
Hadi Salman HADY @ EDU MIT
Andrew Ilyas AILYAS@ MIT.EDU MIT
Sai Vemprala SAIHV @ MICROSOFT.COM Microsoft Research
Logan Engstrom
Vibhav Vineet
ENGSTROM @ MIT.EDU MIT
VIVINEET @ MICROSOFT.COM Microsoft Research
Kai Xiao KAIX @ MIT.EDU MIT
Pengchuan Zhang
Shibani Santurkar
PENZHAN @ MICROSOFT.COM
SHIBANT @ MIT.EDU MIT
Microsoft Research
Greg Yang
Ashish Kapoor AKAPOOR @ MICROOOFFT.CMM Microsoft Research
Aleksander Madry MADRY MIT.EDU MIT
GE. YANG MICROSOFT.COM Microsoft Research
Abstract
We introduce 3DB: an extendable, unified framework for testing and debugging vision models using photorealistic simulation. We demonstrate, through wide range of use cases, that 3DB allows users to discover vulnerabilities in computer vision systems and gain insights into how models make decisions. 3DB captures and generalizes many robustness analyses from prior work, and enables one to study their interplay. Finally, we find that the insights generated by the system transfer to the physical world. We are releasing 3DB as library' alongside set of examples", guides", and documentation".
1 Introduction
Modern machine learning models turn out to be remarkably brittle under distribution shift. Indeed, in the context of computer vision, models exhibit an abnormal sensitivity to slight input rotations and translations [18, 37], synthetic image corruptions [32, 38], and changes to the data collection pipeline [49, 19J. Still, while brittleness widespread, it is often hard to understand its root causes, or even to characterize the precise situations which this behavior arises.
How do we then comprehensively diagnose model failure modes? Stakes are often too high to simply deploy models and collect "real-world" failure cases. There has thus been line of work in computer vision focused on identifying systematic sources of model failure such unfamiliar
Work partially completed while at Microsoft Research. EEqual contribution.
'https: t//b.thub.t.ttt/// wtw.tttww..ooo..tcooooooootttt /dbm 3db
Phttps: h/tptt//..itttttt// com/ 3db/blog ho/l/looodeeemc
Pittps://3das h/tt://.ttb..t.hbbbb.e io/ ho/3db/ ////////////e/e/eee//e/// baaage/quickstern .uickktarrt.htm
1o/3db/
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Texture non-robustness
Corruptions
Geometric transformations
Misleading backgrounds
Specile Molue
Gauasian thr
Take
Unfamiliar objects
aurlc
Figure I: Examples of vulnerabilities of computer vision systems identified through prior in-depth robustness studies. Figures reproduced from [25, 5, 32, 38, 3, 18, 69, 52].
Texture
Pose
Background
Corruptions
New objects
Composition
3DB
Figure 2: The 3DB framework is modular enough facilitate among other tasks- efficient redis- covery of all the types of brittleness shown in Figure 1. also allows users to realistically compose transformations (right) while still being able disentangle the results.
object orientations [3], misleading backgrounds [74, 69]. or shape-texture conflicts [25, 5]. These analyses a selection of which visualized in Figure 1 1-reveal patterns or situations that degrade performance of vision models, providing invaluable insights into model robustness. Still, carrying out each such analysis requires its own set (often complex) tools, usually accompanied by significant amount of manual labor (e.g.. image editing, style transfer), expertise, and data cleaning. This prompts the question:
Can support reliable discovery of model failures in systematic, automated, and unified way?
Contributions. In this work, we propose 3DB, a framework for automatically identifying and ana- lyzing the fallure modes of computer vision models. This framework makes use of 3D simulator to render realistic scenes that can be fed into any computer vision system. Users can specify a set of transformations to apply the scene- such as pose changes, background changes, or camera effects and can also customize and compose them. The system then performs guided search, evaluation, and aggregation over these user-specified configurations and presents the user with an interactive, user-friendly summary of the model's performance and vulnerabilities. 3DB is general enough to enable users to, with minimal effort, re-discover insights from prior work on pose, back- ground, and texture bias (ef. Fig. 2), among others. Further, while prior studies have largely been focused on examining model sensitivities along a single axis, 3DB allows users to compose various transformations and understand the interplay between them, while still being able to disentangle their individual effects.
The remainder of this paper is structured into the following parts: in Section 2 we discuss the design of 3DB, including the motivating principles, design goals, and concrete architecture used. We highlight how the implementation of 3DB allows users to quickly experiment, stress-test, and analyze their vision models. Then, in Section we illustrate the utility of 3DB through a series of case studies uncovering biases in an ImageNet--retraine classifier. Finally, we show (in Section 4) that the vulnerabilities uncovered with 3DB correspond to actual failure modes in the physical world (i.e.. they not specific to simulation).
2 Designing 3DB
The goal of 3DB is leverage photorealistic simulation to effectively diagnose failure modes of computer vision models. To this end, the following set of principles guide the design of 3DB:

---

Generality, 3DB should support any type of computer vision model (i.e., not necessarily a neu- ral network) trained on any dataset and task (i.e.. not necessarily classification). Furthermore, the framework should support diagnosing non-robustness with respect to any parameterizable three- dimensional scene transformation.
Compositionality. Corruptions and transformations rarely occur in isolation-3DD should allow users to investigate robustness along many different axes simultaneously.
Physical realism. The vulnerabilities extracted from 3DB should correspond to models' behavior in the real (physical) world, and, in particular, not depend artifacts of the simulation process itself.
User-friendliness. 3DB should be simple use and should relay insights the user in an easy- to-understand manner. Even non-experts should be able to look at the result of 3DB experiment and easily understand what the weak points of their model are, as well as gain insight into how the model behaves more generally.
Scalability. 3DB should be performant and parallel.
2.1 Capabilities and workflow
To achieve the goals articulated above, we design 3DB modularly, i.e., as combination of swap- pable components. This combination allows the user to specify transformations they want to test, search over the space of these transformations, and aggregate the results of this scarch in concise way. More specifically, the 3DB workflow revolves around five steps (visualized in Figure 3):
Setup. The user collects one or more 3D meshes that correspond to objects the model is trained to recognize, as well as set of environments test against.
Search space design. The user defines a search space by specifying set of transformations (which 3DB calls controls) that they expect the computer vision model to be robust to (e.g.. rotations, trans- lations, zoom, etc.). Controls are grouped into "rendered controls" (applied during the rendering process) and "post-processor controls" (applied after the rendering as 2D image transformation).
Policy-guided search. After the user has specified set of controls, 3DB instantiates and renders a myriad of object configurations derived from compositions of the given transformations. It records the behavior of the ML model on each constructed scene for later analysis. A user-specified search policy over the space of all possible combinations of transformations determines the scenes for 3DB to render.
Step Objects and Erivs
Step Select controls
Step IlI: Set Search Policy
Step V: Analysis
Default Objects
Rendered
Grid search (random deterministic)
Fallure modes
transformss
Camera settings
Lighting trarsforms
custom policy:
Built-in
Occlusion transforms
HDRI Backgrounds
Text re swads
Any search algorithm
Aggregate analysis
Studio environment
Post-processed
Step I: Load model
3DB
Zoom
Erw
ImageNet-C Background shifts
Any classification detection model
Variation
design import
type:
Per-object analysis
Custom
Any blender object Porometers
Modet images Cutpurs
New
Render stute
Evaluotor: Metodota
Figure 3: An overview of the 3DB workflow: First, the user specifies a of 3D object models and environments to use for debugging. The user also enumerates a set of (in-built custom) transformations, known as controls, user-specified search policy over all these controls (and their compositions), 3DB then selects the exact scenes to render. The computer vision model is finally evaluated on these scenes and the results are logged in a user-friendl. manner in custom dashboard.
applied by 3DB while rendering the scene. Based on

---

Model loading, The only remaining step before running a 3DB analysis is loading the model that the user wants to analyze (e.g., pre-trained classifier or object detector).
Analysis and insight extraction. Finally, 3DB is equipped with a model dashboard (cf. Ap- pendix C) that read the generated log files and produce a user-friendly visualization of the generated insights. By default, the dashboard has three panels. The first of these is failure mode display, which highlights configurations, scenes, and transformations that caused the model mis- behave. The per-object analysis pane allows the user to inspect the model's performance on a specific 3D mesh (e.g., accuracy, robustness, and vulnerability to groups of transformations). Finally, the ag- gregate analysis pane extracts insights about the model's performance averaged over all the objects and environments collected and thus allows the user notice consistent trends and vulnerabilities in their model.
Each of the aforementioned components (the controls, policy, renderer, inference module, and log- ger) are fully customizable and can be extended or replaced by the user without altering the core code of 3DB. For example, while 3DB supports more than 10 types of controls out-of-the-box, users can add custom ones (e.g.. geometric transformations) by implementing an abstract function that maps a 3D state and set of parameters fication and object detection models by default, and by implementing a custom evaluator module, users can extend support to wide variety of other vision tasks and models. We refer to Appendix B for more on 3DB design principles, implementation, and scalability.
new state. Similarly, 3DB supports debugging classi-
3 Debugging and analyzing models with 3DB
In this section, illustrate through case studies how to analyze and debug vision models with 3DB. In each case, we follow the workflow outlined in Section 2.1 2.1-imp the relevant objects, selecting the desired transformations (or constructing custom ones), selecting search policy, and finally analyzing the results.
In all our experiments, we analyze ResNet-18 [30] trained on the ImageNet [53] classification task (its validation set accuracy is 69.8%). Note that 3DB is caassifier-annosti (i.e., ResNet-18 can be replaced with any PyTorch classification module), and even supports object detection tasks. For our analysis, we collect 3D models for 16 ImageNet classes (see Appendix F for more details on each experiment). We ensure that in "clean" settings, i.e., when rendered in simple poses plain white background, the 3D models are correctly classified at a reasonable rate (cf. Table 1) by our pre-trained ResNet.
Table 1: Accuracy of a pre-trained ResNet-18, 18, for each of the 16 ImageNet classes considered, on the corresponding 3D model collected, rendered at an unchallenging pose white back- ground ("Simulated" row); and the subset of the ImageNet validation set corresponding to the class ("ImageNet" row).
banana
haseball
bowl
drill
golf hall
hammer
lemon
mug
Simulated accuracy (%)
96.8
100.0
17.5
63.3
95.0
65.6
100.0
13.4
ImageNet accuracy (%)
82.0
66.0
84.0
40.0
82.0
54.0
76.0
42.0
3.1 Sensitivity to image backgrounds
We begin our exploration by using 3DB to confirm ImageNet classifiers' reliance on background signal, as pinpointed by several recent in-depth studies [72, 74, 69]. Out-off-he-box. 3DB can ren- der 3D models onto HDRI files using image-based lighting; we downloaded 408 such background environments from iedrihaeen.. com. We then used the pre-packaged "camera" and "orientation" controls to render (and evaluate our classifier on) scenes the pre-collected 3D models at ran- dom poses, orientations, and scales on each background. Figure shows random example scenes generated by 3DB for the "coffee mug" model.
Analyzing a subset of backgrounds. In Figure 6, we visualize the performance of a ResNet- classifier on the 3D models from 16 different ImageNet classes-in random positions, orientations, and
onto 20 of the collected HDRI backgrounds. One can observe that background

---

34%
31%
30%
olunger (14.77))
coffeecet (19.5%))
buxiced 161, 5%0
1%
2%
2%
Figure 5: (Top) Best and (Bottom) worst background en- viromments for classification of the coffee mug, and their respective accuracies (averaged over camera positions and zoom factors).
Figure 4:
Renderings of the mug 3D model in different en- vironments, labeled with pre- trained model's top prediction.
1.0
Per-anvtromment accuracy
1.0
Por-ooject accuraryy
0.8
0.8
0.6
0.4
0.4
0.2
0.0
00
altarto
arstted Wreck
factory yarg
cabin
gray pier
koppenheim
rathaus
stadium
banana
baseball
bow!
drill
getr kall
hammar
lemon
may
orange
pitcher base
power drij
sandlle
ahoe
spatula
tespot
temia
Figure 6: Visualization of accuracy on controls from Section 3.1. (Left) We compute the accuracy of the model conditioned on each object-environment pair. For each environment on the x-axis, we plot the variation in accuracy (over the set of possible objects) using boxplot. We visualize the per-object accuracy spread by including the median line, the first and third quartiles box edges (the interval between which is called the inter-quartile range, IQR), the range, and the outliers (points that are outside the IQR by 3/2|IQR). ). (Right) Using the same format, we track how the classified object (x-axis) impacts variation in accuracy (over different environments) on the y-axis.
dependence indeed varies widely across different objects for example, the "orange" and "lemon" 3D models depend much more on background than the "tennis ball." We also find that certain backgrounds yield systemically higher or lower accuracy; for example, average accuracy "gray pier" is five times lower than that of "factory yard."
Analyzing all backgrounds with the mug model. The previous study broadly characterizes the classifier's sensitivity to different models and environments. Now, to gain a deeper understanding of this sensitivity, focus analysis only single 3D model (a "coffee mug") rendered in all 408 environments. The highest-accuract backgrounds had tags such as skies, and mountain, while the lowest-accuracy backgrounds had tags indoor, city, and building.
At first, this observation seems to be at odds with the idea that the classifier relies heavily on con- text clues to make decisions. After all, the backgrounds where the classifier seems to perform well (poorly) are places that we would expect coffee mug to be rarely (frequently) present in the real world. Visualizing the best and worst backgrounds in terms of accuracy (Figure 5) suggests pos- sible explanation this: the best backgrounds tend be clean and distraction-free Conversely, complicated backgrounds (e.g., some indoor scenes) often contain context clues that make the mug difficult for models to detect. Comparing a "background complexity" metric (based on the number of edges in the image) to accuracy (Figure 7) supports this explanation: mugs overlaid on more complex backgrounds are more frequently misclassified by the model. In fact, some specific back- grounds even result in the model "hallucinating" objects; for example, the second-most frequent predictions for the pond and sidewalk backgrounds were birdhouse and traffic light respectively, despite the fact that neither object is present in the environment.
Goom/bbckground interactions case study: the advantage of composable controls. Finally, we leverage 3DB's composability to study interactions between controls. In Figure 8, we plot the mean classification accuracy of our "orange" model while varying background and scale factor. We, for example, find that while the model is highly accurate at classifying "orange" at 2x zoom, the same
5

---

abandoned workshop place altanka aristea wrock
Average accuracy (%6
bush restaurant cabin derelict averpass dusseldorf bridge factory yard gray greenwich
25
0.7
20
environment
kiara 7 late-aftemoon
10
koopeenhelm 06 rathaus
0.4
rootless secluded beach small hangar 02
0.00
0.05
0.10
0.15
0.20
stadium D1
Background complexity
studio small 02
Figure Relation between the complexity of background and its average accuracy. Here complexity is defined as the average pixel value of the image after applying edge detection filter.
0.8
1.1
1.4
1.7
2.0
zoom
Figure 8: 3DB's focus on composability enables us to study robustness along multiple axes si- multaneously. Here we study average model ac- curacy (computed over pose randomization) as function of both zoom level and background.
zoom factor induces failure in well-lit mountainous environment ("Kiara late-afternoon") fine- grained failure mode that we would not catch without explicitly capturing the interaction between background choice and zoom.
3.2 Texture-shape bias
COw
elephant
crocodile
leopard
Tealua alophack 1.10 vady
Zediua almphack
zebra
tiger
20
60
80
100
Accuracy(%6)
Figure 9: Cue-conflict images generated by
Figure 10: Model accuracy previously corsectyy-lassified images after their texture is altered via 3DB, as function of texture-type.
Geirhos et al. [25] (top) and 3DB (bottom).
We now demonstrate how 3DB be straightforwardly extended to discover more complex failure modes in computer vision models. Specifically, we will show how to rediscover the "texture bias" exhibited by ImageNet-trained convolutional neural networks (CNNs) [25] in a systematic and (near) texture information from
photorealistic way. Geirhos et al. [25] fuse pairs of images-combintt one with shape and edge information from the other-to create so-called "cue-conflict" images. They then demonstrate that on these images (cf. Figure 9), ImageNet-trained CNNs typically predict the class corresponding the texture component, while humans typically predict based shape.
Cue-conffict images identify a concrete difference between human and CNN decision mechanisms. However, the fused images unrealistic and be cumbersome generate (e.g., even the sim- plest approach uses style transfer [24]). 3DB gives us an opportunity to rediscover the influence of texture more streamlined fashion.
Specifically, we implement control (now pre-packaged with 3DB) that replaces an object's texture with random (or user-specified) one. We use this control to create cue-conflict objects out of eight

---

0.8
oreenatiion
0.4
6.44
0.2
por
0.6
burting
dra Ball
pitche Mase
crange
slndle
spatuip
Heapost tmoni
bamona toneeiial
Anver
dell
soll anel
Rumoner
lerter
orange
patcwer bara
power drlt
canste
spattala reappnr teveld Ball
Figure 11: (Left) We compute the accuracy of the model for each object-orientation pair. For each object on the x-axis, we plot the variation in accuracy (over the set possible orientations) using boxplot. We visualize the per-orientation accuracy spread by including the median line, the first and third quartiles box edges, the range, and the outliers. (Right) Using the same format as the left hand plot, we plot how the classified object (on the x-axis) impacts variation in accuracy (over different zoom values) on the y-axis.
3D models' and seven animal-skin texture images (i.e., 56 objects in total). We test our pre-trained ResNet-18 18 on images of these objects rendered in a variety of poses and camera locations. Figure displays sample cue-conflict images generated using 3DB.
Our study confirms the findings of Geirhos et al. [25] and indicates that texture bias indeed extends to (near- rrealitcali settings. For images that were originally correctly classified (i.e., when rendered with the original texture), changing the texture reduced accuracy by 90-95% uniformly across textures (Figure 10). Furthermore, we observe that the model predictions usually align better with the texture of the objects rather than their geometry (See Figure 21 in the Appendix).
3.3 Orientation and scale dependence
Image classification models are brittle to object orientation in both real and simulated settings [37, 18, 6, 3J. As was the case for both background and texture sensitivity, reproducing and extending such observations is straightforward with 3DB. Once again, we use the built-in controls to render objects at varying poses, orientations, scales, and environments before stratifying on properties of interest. Indeed, we find that classification accuracy is highly dependent on object orientation (Fig- ure 11 left) and scale (Figure 11 right). However, this dependence is not uniform across objects. As one would expect, the classifier's accuracy is less sensitive to orientation more symmetric objects (like "tennis ball" or "baseball"), but can vary widely on more uneven objects (like "drill").
For more fine-grained look at the importance of object orientation, we can measure the classi- fier accuracy conditioned on a given part of each 3D model being visible. This analysis is once again straightforward in 3DB, since each rendering is (optionally) accompanied by UV map which maps pixels in the scene back to locations on on the object surface. Combining these UV maps with accuracy data allows one to construct the "accuracy heatmaps" shown in Figure 12, wherein each part of an object's 's surface corresponds to classifier accuracy on renderings in which the part is visible. The results confirm that atypical viewpoints adversely impact model performance, and also allow users to draw up a variety of testable hypotheses regarding performance on specific 3D models (e.g., for the coffee mug, the bottom rim is highlighted in red-is is it the case that mugs more accurately classified when viewed from the bottom)? These hypotheses can then be investi-
'Objct models: mug, helmet, hammer, strawberry, teapot, pitcher, bowl, lemon, banana and spatula Textures: cow, crocodile, elephant, leopard, snake, tiger and zebra
D
O
Figure 12: Model sensitivity to pose. The heatmaps denote the accuracy of the model in predicting the correct label, conditioned on specific part of the object being visible in the image. Here, red and blue denotes high and low accuracy respectively.
7

---

wiite Recks
0.20
Proportonn
0.30
0.05
Water
ME
10
Number of disincc predictions
(a)
(b)
Figure 13: Testing classifier sensitivity to context: Figure (a) shows the correlation of the liquid mixture in the mug on the prediction of the model, averaged over random viewpoints (see Figure 20b for the raw frequencies). Figure (b) shows that for a fixed viewpoint, model predictions are unstable with respect to the liquid. Figure (c) shows examples of rendered liquids (water, black coffee, milk, and mixtures).
gated further through natural data collection, or-as we discuss in the upcoming section-ttrrugh! additional experimentation with 3DB
3.4 Case study: using 3DB to dive deeper
Our heatmap analysis in the previous section (cf. Figure 12) showed that classification accuracy for the mug decreases when its interior is visible. What could be causing this effect? One hypothesis is that in the ImageNet training set, objects are captured in context, and thus ImageNet-trained classifiers rely on this context to make decisions. Inspecting the ImageNet dataset, we notice that coffee mugs in context usually contain coffee. Thus, the aforementioned hypothesis would suggest that the model relies, at least partially, on the contents of the mug to correctly classify it. Can we leverage 3DB to confirm refute this hypothesis?
To test this, we implement a custom control that can render liquid inside the "coffee mug" model. Specifically, this control takes water. ratios as parameters, then uses parametric Blender shader (cf. Appendix G) to render corresponding mixture the liquids into the mug. We used pre-packaged grid search policy, (programmatically) restricting the search space to viewpoints from which the interior of the mug was visible.
The results of the experiment are shown in Figure 13. It turns out that the model is indeed sensitive to changes in liquid, supporting our hypothesis: model predictions stayed constant (over all liquids) for only 20.7% of the rendered viewpoints (ef.. Figure 13b). The 3DB experiment provides further sup- port for the hypothesis when we look at the correlation between the liquid mixture and the predicted class: Figure 13a visualizes this correlation in a normalized heatmap (for the unnormalized version, see Figure 20b in the Appendix G). We find that the model is most likely to predict "coffee mug" when coffee is added to the interior (unsurprisingly): as the coffee is mixed with water or milk, the predicted label distribution shifts towards "bucket" and "cup" or "pill bottle," respectively. Overall, our experiment suggests that current ResNet-18 18 classifiers are indeed sensitive to object context- this case, the fluid composition of the mug interior. More broadly, this illustration highlights how a system designer can quickly go from hypothesis to empirical verification with minimal effort using 3DB. (In fact, going from the hypothesis to Figure 13 took less than a day work for one author.)
Physical realism
The previous sections have demonstrated various ways in which we can 3DB to obtain insights into model behavior in simulation. Our overarching goal, however, to understand when models will fail in the physical world. Thus, would like for the insights extracted by 3DB to corre- spond to naturally-arising model behavior, and not just artifacts of the simulation itself. To this end, now test the physical realism of 3DB: can we understand model performance (and uncover vulnerabilities) on real photos using only a high-fideling fideliiy simulation?
To answer this question, we collected set of physical objects corresponding to 3D models, and set up a physical room with a corresponding 3D environment. We used 3DB to identify strong points and vulnerabilities of ImageNet classifier in this environment, mirroring our methodology from

---

Post tive predictive value
Negative predictive value
Mug
Drill
Toygun
8,8
809
R89
308
3,0
403
B0B
E08
208
ENB
Synthetic
Real
Figure 14: (Top) Agreement, terms of model correctness, between model predictions within 3DB and model predictions in the real world. For each object, we selected five rendered scenes found by 3DB that were misclassified in simulation, and five that were correctly classified; we recreated and deployed the model on each scene in the physical world. The positive (resp., negative) predictive value is rate which correctly (resp. incorrectly) classified examples simulation were also correctly (resp., incorrectly) classified in the physical world. (Bottom) Comparison between example simulated scenes generated by 3DB (first row) and their recreated physical counterparts (second row). Border color indicates whether the model was correct on this specific image.
Section 3. We recreated each scenario found by 3DB in the physical room, and took photographs that matched the simulation as closely as possible. Finally, we evaluated the physical realism of 3DB by comparing models' performance the photos to what 3DB predicted.
Setup. We used studio room shown Appendix Figure 18b for which we obtained fairly accurate 3D model (cf. Appendix Figure 18a). We leverage the YCB [13] dataset to guide our selection of real-world objects, for which 3D models are available. We supplement these by sourcing additional objects (from ama z.n.. com) and using 3D scanner to obtain corresponding meshes.
We next used 3DB to analyze the performance objects in simulation, varying over a set of realistic object poses, locations, and orientations. For each object, we selected 10 rendered situations: five where the model made the correct prediction, and five where the model predicted incorrectly. We then tried to recreate each rendering in the physical world. First we roughly placed the main object in the location and orientation specified in the rendering, then we used custom- built iOS application (see Appendix D) to more precisely match the rendering with the physical setup.
pre-trained ImageNet ResNet-18 on the collected
Results. Figure 14 visualizes a few samples of renderings with their recreated physical counterparts, annotated with model correctness. Overall, found 85% agreement rate between the model's correctness on the real photos and the synthetic Figure 14. Thus, despite imperfections in our physical reconstructions, the vulnerabilities identified by 3DB turned out to be physically realizable vulnerabilities (and conversely, the positive examples found by 3DB simpler/non htth//ooo-mmttaliic materials (e.g., the bowl, mug, and sandal) tended to be more reliable than metallic objects such the hammer and drill. thus possible that more precise texture tuning of 3D models object could increase agreement further (although more comprehensive study would be needed verify this).
rates per class are shown in
usually also classified correctly in the real world). We found that objects with
Related work
In this section, we give a brief overview of existing work robustness, interpretability, and sim- ulation that provide the context for work. We refer the reader to Appendix A for detailed discussion of prior work.
The brittleness of current ML models has drawn the attention to analyze the robustness and reliability of these models. long line of research focus on analyzing model robust- ness to adversarial examples [61, 20, 70, 21, 20, 12, 5, 68, 47, 44]. Another line of research involves
Model Robustness.

---

analyzing robustness to non-adversarial corruptions [18, 25, 32, 38, 74, 69, 23, 52]. A more closely related line of research ours analyzes the impact of factors such as object pose and geometry by applying synthetic perturbations in three-dimensional space [28, 57, 29, 3, 35].
Interpretability and model debugging. 3DB can be cast method for debugging vision mod- els that provides users fine-grained control over the rendered scenes and thus enables them to find specific modes of failure (cf. Sections 3 and 4). Model debugging is also common goal in in- tepretability, where methods generally seek to provide justification for model decisions based on either local features (e.g.. saliency maps) [58, 14, 60, 50, 22, 74, 27] or global ones (i.e., general
biases of the model) [7, 41, 71, 63].
Simulated environments. Finally, there has been long line of work developing simulation platforms as a source of additional training data [11, 8, 36, 73, 15, 51, 55, 59, 17, 42, 64, 48, 65, 66, 67, 54]. 3DB shares components with many of these works (e.g.. rendering engine), but has a very different goal and set of applications, i.e., diagnosing specific failures in existing models.
Conclusion
In this work, we introduced 3DB, unified framework for diagnosing failure modes in vision models based on high-fidelity rendering. We demonstrate the utility of 3DB by applying it to number of model debugging use cases-such as understanding classifier sensitivities to realistic scene and object perturbations, and discovering model biases. Further, we show that the debugging analysis done using 3DB in simulation is actually predictive of model behavior the physical world. Finally, note that 3DB was designed with extensibility as priority; we encourage the community to build upon the framework so as to uncover new insights into vulnerabilities of vision models.
Limitations. One limitation of 3DB is the need for high-quality 3D models for objects of interest in order achieve photorealistic images. This requires 3D model artists and/or effective photogram- metry techniques. Additionally, creating fully realistic scenes may require more complexity than just combining single object with background, which what we focus on in this paper. 3DB does support multiple objects, and the user can programmatically specify how different objects are located relative to each other; we hope to explore this more in the future.
Acknowledgements
Work supported in part by the NSF grants CCF- and CNS-1815221, the Google PhD Fel- lowship, the Open Philanthropy Project AI Fellowship, the NDSEG PhD Fellowship, and the Mi- crosoft Corporation. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001 120C0015.
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number 18775011992-11 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or im- plied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

References
[1] Julius Adebayo et al. "Sanity checks for saliency maps". . In: Neural Information Processing Systems (NeurIPS). 2018.
[2] Julius Adebayo et al. "Debugging Tests for Model Explanations". In: 2020.
[3] Michael A Alcorn et "Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects". In: Conference on Computer Vision and Pattern Recognition (CVPR). 2019.
[4] David Alvarez-Melis and Tommi S Jaakkola. "On the robustness interpretability methods". In: arXiv preprint arXiv: 1806 08049 (2018).
[5] Anish Athalye et al. "Synthesizing Robust Adversarial Examples". In: International Confer- ence on Machine Learning (ICML). 2018.
[6] Andrei Barbu et al. "ObjectNet: A large-scale bias-controlled dataset for pushing the limits object recognition models". In: Neural Information Processing Systems (NeurIPS). 2019. [7] David Bau et al. "Network dissection: Quantifying interpretability of deep visual representa- tions". In: Computer Vision and Puttern Recognition (CVPR). 2017.
[8] Charles Beattie et al. "Deepmind lab". In: arXiv preprint 1611v:/6/2.053:: (2016).
[9]
Harkirat Singh Behl et al. "Autosimulate: (quickly) learning synthetic data generation". In: European Conference Computer Vision. Springer. 2020, pp. 255-271.
[10] Blender Online Community. Blender 3D modelling and rendering package. Blender Foun- dation. Stichting Blender Foundation, Amsterdam, 2020. URL: http: www. blender. org.
[11]
Greg Brockman et al. "Openai gym". In: arXiv preprint ww0.::::06000001111. (2016).
[12] Tom B. Brown et al. Adversarial Patch. 2018. arXiv: 1 1721.1121 . 6966 (cs.CV].
[13] Berk Calli et al. "Benchmarking in manipulation research: The YCB object and model set and benchmarking protocols". In: arXiv preprint arXiv: /502.03 143 (2015).
[14] Piotr Dabkowski and Yarin Gal. "Real time image saliency for black box classifiers". Neural Information Processing Systems (NeurIPS). 2017.
[15] Maximilian Denninger al. "BlenderProc". In: arXiv preprint arXiv:1911.01911 (2019).
[16] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. "Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation". In: European Conference on Computer Vi- sion. Springer. 2020, pp. 715-733.
[17] Alexey Dosovitskiy et al. "CARLA: An open urban driving simulator". In: arXiv preprint
arXiv:1711.03922 (2017).
[18] Logan Engstrom et al. "Exploring the Landscape of Spatial Robustness", In: International Conference on Machine Learning (ICML). 2019.
[19] Logan Engstrom et al. "Identifying Statistical Bias in Dataset Replication", In: International Conference on Machine Learning (ICMI.). 2020.
[20]
Kevin Eykholt et al. "Physical Adversarial Examples for Object Detectors". In: CoRR (2018).
[21] Volker Fischer et al. "Adversarial examples for semantic image segmentation". In: Arxiv
preprint arXiv:1703.00 100.:1733.01011. 2017.
[22] Ruth C Fong and Andrea Vedaldi. "Interpretable explanations of black boxes by meaningful perturbation" In: International Conference on Computer Vision (ICCV). 2017.
[23] Nic Ford et al. "Adversarial Examples Are Natural Consequence of Test Error in Noise". In: arXiv preprint arXiv: 1901. 2019.
[24] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. "Image style transfer using convo- lutional neural networks". In: computer vision and pattern recognition ( (CVPR). 2016.
[25] Robert Geirhos et al. "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness." In: International Conference Learning Represen- tations (ICLR). 2019.
[26] Amirata Ghorbani. Abubakar Abid, and James Zou. "Interpretation of neural networks is fragile". In: AAAI Conference on Artificial Intelligence (AAAI). 2019.
[27] Yash Goyal et "Counterfactual visual explanations". In: arXiv preprint arXiv: Xixv:1904.07455
(2019).

---

[28]
Abdullah Hamdi and Bernard Ghanem. "Towards Analyzing Semantic Robustness of Deep Neural Networks". In: arXiv preprint arXiv:1904.0400 04621 (2019).
[29] Abdullah Hamdi, Matthias Muller, and Bernard Ghanem. "SADA: Semantic Adversarial Di-
agnostic Attacks for Autonomous Applications" In: arXiv preprint arXiv: (2018). Kaiming He et al. Deep Residual Learning for Image Recognition. 2015.
[30]
[31] Christoph Heindl et al. "BlendTorch: A Real-Time, Adaptive Domain Randomization Li-
brary". In: arXiv preprint urXiv:2010.111 11696 (2020).
[32] Dan Hendrycks and Thomas G. Dietterich. "Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations". In: International Conference on Learning
Representations (ICLR). 2019.
[33] Dan Hendrycks al. "Natural adversarial examples". In: arXiv preprint arXiv:1907.0711 (2019).
[34] Sandy Huang et al. "Adversarial Attacks on Neural Network Policies". In: ArXiv preprint FrXii:/702.022880 2017.
[35] Lakshya Jain et al. "Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Rendering". In: arXiv preprint (2020).
[36]
Arthur Juliani et al. Unity; A General Platform for Intelligent Agents. 2020. arXiv: 1809. 02627 [cs. LG].
[37] Can Kanbak, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. "Geometric robustness of deep networks: analysis and improvement". In: Conference on Computer Vision and Pat- tem Recognition (CVPR). 2018.
[38] Daniel Kang et al. "Testing Robustness Against Unforeseen Adversaries". In: ArXiv preprint arxiv: 08016. 2019.
[39] Amlan Kar al. "Meta-sim: Learning to generate synthetic datasets". In: Proceedings of the IEEE/CVF International Conference Computer Vision. 2019, pp. 4551-4560.
[40] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. "Neural 3D Mesh Renderer". In: Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
[41] Been Kim et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)". In: International conference on machine learning (ICMI.). 2018
[42] Eric Kolve et al. "Ai2-thor: An interactive 3d environment for visual ai". In: arXiv preprint XXiv:1712.05477/ (2017).
[43] Jernej Kos, Ian Fischer, and Dawn Song. " "Adversarial examples for generative models". In: IEEE Security and Privacy Workshops (SPW). 2018.
[44] Juncheng Li, Frank R. Schmidt, and J. Zico Kolter. "Adversarial camera stickers: A physical casera-based attack on deep learning systems". In: Arxiv preprint arXiv: T00V:1990400759 2019.
[45] Tzu-Mao Li et al. "Differentiable Monte Carlo Ray Tracing through Edge Sampling". In: SIGGRAPH Asia 2018 Technical Papers. 2018.
[46] Zachary C Lipton. "The Mythos of Model Interpretability: In machine learning, the concept
of interpretability is both important and slippery." In: (2018).
[47] Hsuch-Ti Derek Liu et al. "Beyond Pixel Norm-Balls: Parametric Adversaries Using An Ana- lytically Differentiable Renderer". In: International Conference on Learning Representations (ICLR). 2019.
[48] Xavier Puig et al. "Virtualhome: Simulating houschold activities via programs". In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.
[49] Benjamin Recht et al. "Do ImageNet Classifiers Generalize to ImageNet?" In: International Conference on Machine Learning (ICML). 2019.
[50] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. ... Whhhy should I trust you?" Ex- plaining the predictions of any classifier". In: International Conference Knowledge Dis- covery and Data Mining (KDD). 2016.
[51] Mike Roberts and Nathan Paczan. Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding. arXiv 2020.
[52] Amir Rosenfeld, Richard Zemel, and John K. Tsotsos. "The Elephant in the Room". In: arXiv preprint arXiv: 000.::308000000... 2018.
12

---

[53] Olga Russakovsky et al. "ImageNet Large Scale Visual Recognition Challenge". In: Interna- tional Journal of Computer Vision (IJCV). 2015.
[54] Manolis Savva et al. "Habitat: A platform for embodied ai research". In: Proceedings of the
IEEE International Conference on Computer Vision. 2019.
[55] Shital Shah et al. "Airsim: High-fidelity visual and physical simulation for autonomous vehi- cles". In: Field and service robotics. Springer. 2018, pp. 621-635.
[56] Vaishaal Shankar et al. "Do Image Classifiers Generalize Across Time?" In: arXiv preprint
arXiv: 1906: 02 668 (2019).
[57] Michelle Shu et al. "Identifying Model Weakness with Adversarial Examiner". In: AAAI Con-
ference Artificial Intelligence (AAAI). 2020.
[58] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional networks: Visualising image classification models and saliency maps". In: arXiv preprint
arXiv:1312.6034 (2013).
[59] Yunlong Song et al. "Flightmare: A Flexible Quadrotor Simulator". In: arXiv preprint 0rXiv:2009.00555 (2020).
[60] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. "Axiomatic attribution for deep networks". In: International Conference on Machine Learning (ICML). 2017.
[61] Christian Szegedy et al. "Intriguing properties of neural networks". In: International Confer-
ence on Learning Representations (ICLR). 2014.
[62]
Antonio Torralba and Alexei A Efros. "Unbiased look at dataset bias". In: CVPR 2011. 2011.
[63] Eric Wong, Shibani Santurkar, and Aleksander Madry. "Leveraging Sparse Linear Layers for Debuggable Deep Networks". In: International Conference on Machine Learning (ICML).
2021.
[64] Yi Wu et al. "Building generalizable agents with realistic and rich 3d environment". In: arXiv preprint arXiv: (2018).
[65] Fei Xia et al. "Gibson env: Real-world perception for embodied agents". In: Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. 2018.
[66] Fei Xia al. "Interactive Gibson Benchmark: Benchmark for Interactive Navigation in Cluttered Environments". In: IEEE Robotics and Automation Letters (2020).
[67] Fanbo Xiang et al. "SAPIEN: A simulated part-based interactive environment", In: Proceed- ings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
[68] Chaowei Xiao et al. "MeshAdv: Adversarial Meshes for Visual Recognition". In: Computer Vision and Pattern Recognition (CVPR). 2019.
[69] Kai Xiao et al. "Noise or signal: The role of image backgrounds in object recognition". In: arXiv preprint arXiv:2006.0999
(2020).
[70] Cihang Xie al. "Adversarial examples for semantic segmentation and object detection"..
In: Proceedings of the IEEE International Conference on Computer Vision. 2017, pp. 1369- 1378.
[71] Chih-Kuan Yeh et "On Completeness-aware Concept-Based Explanations in Deep Neural Networks". In: Advances in Neural Information Processing Systems (NeurIPS) (2020).
[72] Jianguo Zhang et al. "Local features and kernels for classification of texture and object cate- gories: A comprehensive study". In: International journal of computer vision. 2007.
[73] Yuke Zhu et al. "robosuite: A modular simulation framework and benchmark for robot learn- ing". In: arXiv preprint arXiv:2009. 12293 (2020).
[74] Zhuotun Zhu, Lingxi Xie, and Alan Yuille. "Object Recognition without and without Ob- jects". In: International Joint Conference Artificial Intelligence. 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See conclusion.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
conclusion.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] See supple- mentary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix F.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B.
4. If you are using existing assets (e-g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "mXmaupg0Z0Y", "reviewer": "Reviewer_pi9t", "summary": "A software framework for testing, debugging computer vision models via use of photorealistic simulated data is described. The utility of the framework is demonstrated through a wide range of use cases and the authors illustrate how vulnerabilities in ML models for vision can be discovered. The framework allows for flexible evaluation of robustness and allows the impact of multiple factors to be studied. There is a discussion of how simulated data analysis transfers to real-world data through systematic experiments that attempt to match real to simulated settings. The software will be open-sourced.\n", "questions": " I appreciate the level of work going into buliding a framework that can support systematic debugging and testing of computer vision models. Simulation for performance modeling has been studied in the 90's and more recently in the last decade in the context of ML and deep learning.  Can you specifically comment what is unique about your framework that allows scalable testing and validation?  \n\nHow does your framework compare to frameworks that support probabilistic programming ?  These frameworks allow natural integration of graphics engines with probabilistic programs for synthesis of image datasets and exploit them in an inference loop.  \n\nApart from the software tool and properties demonstrated, do you have specific methodological contributions for syn to real transfer/model validation of simulator?\n\n\n", "limitations": "In the last decade, a number of papers have studied the use of synthetic data for ML including some of the early papers that illustrate in detail performance characterization methodology with synthetic data and deep learning (in the context of semantic segmentation).   The authors may benefit from the series of papers below:\nV. S. R. Veeravasarapu et al, Adversarially Tuned Scene Generation. CVPR 2017: 6441-6449\nV. S. R. Veeravasarapu et al, Model-Driven Simulations for Computer Vision. WACV 2017: 1063-1071\nV. S. R. Veeravasarapu et al, Simulations for Validation of Vision Systems. CoRR abs/1512.01030 (2015).", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nA useful framework for simulated data generation, testing and debugging of computer vision models.  The experimentation is adequate to illustrate the utility of the tools.  The part involving the matching of simulated data to real data acquisitions and the experimentation with particular hypotheses on model performance (section 3.4 and 4) are interesting.\n\nOriginality:  The concepts presented and the tool itself are not novel and the results are largely confirmatory of well known behavior of ML models (as expected).  \n\nClarity: Well written and clear.\n\nSignificance:  The paper in its present form may not yet be significant, although if scaled systematically the paper has significant potential.  The paper can benefit from a principled incorporation of performance characterization methodology articulated in the 90's in computer vision. (see haralick.org). Incorporation of probabilistic generative models (including the imaging device and rendering pipeline) and enabling of formal comparison of simulated data statistics to real data statistics can enable the tool to be of practical use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j23-92toEsN", "reviewer": "Reviewer_Go77", "summary": "The paper presents 3DB — library/software package for studying computer vision models sensitivity to various transformations, from the geometrical, like camera pose change, to semantic, e.g. “Does color of the liquid in the cup influence the classification output of the model?”.\n3DR is implemented as (wrapper + UI) around (Blender + deep learning model runner). It (supposedly) comes with some 3D models and background — graphical “assets”.\nThe paper showcases the library with several case studies — background influence, liquid color influence, and zoom/rotation influence on the classification model output. ", "questions": "I actually, have no idea, how to evaluate such paper/tool w.r.t. NeurIPS acceptance. My gut feeling is the following (feel free to correct me if I am wrong, both AC, Rs and authors):\n\nif the tool would be useful for machine learning community, the paper should be accepted. If not, then not. \n\nQuestions (again, non-standard) \n\n1. How are you going to support and maintain 3DB? Would it be someone supporting it at least part time? Open source community, authors themselves/etc?\n2. Do you plan any tutorial (both like web-tutorial, and like CVPR/NeurIPS tutorial), showing how to install and use 3DB?\n3. Do you plan to release pre-rendered datasets, so people could run their evaluations without installing 3DB?\n4. Could you please elaborate on how much time each experiment described in the paper take? With (if possible) a time breakdown: \"search for 3D models online\" - X hours, debugging - Y hours, running rendering & inference - Z hours, etc.\n\n\n****\nAfter-rebuttal update:\n\nAuthors have been addressed 3/4 of my questions and additionally announced to release a dataset of 5k objects under CC license. This significantly reduces my concerns. The last, not addressed one, is maintenance of the library, but let's hope for the best.\nI am increasing my score to weak accept.\n\n\n", "limitations": "Yes, authors clearly stated the limitations of their framework", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is not a standard NeurIPS paper, so this review would be a bit non-standard as well. The main contribution is not the presented experiments themselves, but rather the software package usefulness for researchers. \nThe package is provided in the supplementary material, so I have tried to install/run it.\nUnfortunately, it is not possible, because there no “setup.py” file, also the requirements.txt are empty. \n\nIn order to try the run the library, I have looked at github for 3db, and luckily found it, under the same org name “3db”. Also, I have seen some github handles of the authors, but none of them is known to me, so I consider myself still in double blind mode. \nHowever, if this is not the case, feel free to ignore my review and ban me. \n\nAnyway, let me continue. The repo was more full and contained instructions how to install the thing, however, MacOS was not supported. Luckily, there was a docker option. Unluckily, it was not working either. I was able to proceed a little bit more, by fixing a couple of mistakes there (e.g. python==3.7, not python==3.7.7 for conda env), but finally gave up, as I spend on this around 1.5 hours without much success. Some paths are hardcoded as well (e.g. threedb_master) , which does not help either.\n\nNow, I fully realize, that I probably was not motivated enough to make it work, also my devops skills definitely suck. \nLet’s meet on the middle ground and say that library is not ready/documented enough for usage by beginners. Expert should definitely be able to run the 3DB, and the middle person - depending on time and motivation.\n\n\nNow, when I fail to try the package myself, I fall back to evaluating it from the paper only.\n\nSupposed UX is the following:\n1. User comes up with hypothesis to test, e.g. if the chair detection performance is heavily dependent on  are \n2. User creates/buys 3D model of the chair, Blender-compatible, and some backgrounds\n3. User writes down the config in yaml for rendering parameters\n4. The rendering and inference occurs in a parallelized way (map)\n5. Results are them reduced to the visualization server, where one could generate summarization graphs, as well, as see the failure cases themselves.\n\nThis seems as attractive process except stage 2. \n\nWeaknesses:\n\n1. The installation process is not user-friendly\n2. (stated in paper itself) you are as good, as 3D models you own. No 3D models - no 3db \n3. Although, that title says “Debugging Computer Vision Models”, in fact, only classification and detection is really supported now. Tracking, matching, etc are beyond the score of 3db (at least now. One can extend the framework probably). \n\n\nNote, that I don't evaluate experiments presented in the paper themselves, as they were used just as a showcase for the tool. I like the experiments, though.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hn1-akFIWzG", "reviewer": "Reviewer_8XFw", "summary": "The paper introduces 3DB, a new framework to test vision models using various renderings of 3D simulation. It allows users to discover model vulnerabilities such as textures, corruptions, geometric transformations, different backgrounds etc. Finally, a realistic benchmark is constructed based on available 3D models, to demonstrate the agreement rate between synthetic renderings and real-world images.", "questions": "My primary concerns are:\n\n- Are you using YCB dataset scanning 3D objects yourselves for all 3D models? I think it only has limited tabletop objects. Is it feasible to test on more categories? \n\n- Do you have any results (even qualitatively) on benchmarks besides image classification?", "limitations": "Limitations are discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The approach is neat. Considering it's very hard to collect real-world images of the same object under different configurations, it's a good idea to utilize 3D renderings.\n- One of my concern is whether the synthetic test set reflect the performance in real-world images. Failing in a synthetic image does not mean it will necessarily fail in the real world. This is resolved relatively well in Section 4 physical realism and I appreciate the experiment. Still, 40 test images look like a small test set. But I understand how hard it is to collect real-world images similar to a rendering. \n\n\nWeaknesses:\n\n- The approach to test vision models are fairly limited by avavilable 3D resources. Most vision models mentioned here only have 2D predictions -- which means the ground truth is relatively easy to collect compared with 3D models. I think it's very hard to completely test the robustness of the model. For example, it's nearly impossible to find a lot of 3D models for each ImageNet category.\n\n- Only image classification is presented while 3DB should support more vision benchmarks such as object detection and segmentation. It is non-trivial and more challenging to render synthetic images for object detection since multiple objects are involved. Randomly assigning a 3D pose may not reflect the real-world distribution at all.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_jL100uWT0w", "reviewer": "Reviewer_8cBx", "summary": "The paper presents a framework for systematic testing and debugging of vision models. The framework uses photorealistic simulation and considers different variables, as different object categories, HDRI backgrounds and lighting conditions, different textures, scene transformations as well as custom controls. By considering different combinations of the variables involved, the framework is quite useful in finding vulnerabilities of machine learning models with respect to specific scene configurations.", "questions": "1. The text also briefly discusses the possibility to use the framework for object detection tasks. It would be interesting to show some example\n2. Why are Cup and Pill bottle considered together in Figure 13?", "limitations": "The paper openly discusses limitations. Additionally, some crucial aspects, such as the real vs simulated world domain gap, are covered in depth in the text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nOriginality\n-----------\nSimulation based methods for testing and validating computer vision systems, in general, and deep learning models in particular, have been largely considered in the past. Nevertheless, the proposed method is quite comprehensive, providing control over multiple crucial aspects, while at the same time it is highly customizable.\n\nQuality and Clarity\n-------------------\nThe paper is well written and easy to read. To support the proposed framework, multiple well known challenges of computer vision models are highlighted, where it is shown that significant results from the corresponding literature can be quickly and easily reproduced with the help of the proposed framework. A toy hypothesis is also considered, and it is shown that it can be easily accommodated with the help of some customization. The domain gap between real and simulated data is also discussed in sufficient detail, and quantitative comparisons are provided.\n\nSignificance\n------------\nThe use of the proposed framework can significantly simplify the process of testing and validation of computer vision models. The framework is likely more useful for testing and validating different vision models rather than discovering novel general failure modes, as the modes of failure covered by the framework have largely being explored. Nevertheless, it is important that the framework allows for customization, as this can increase also its applicability in searching for novel controls/aspects with respect to which machine learning models may be particularly susceptible.\n\nMinor comments\n--------------\n* L.122 classifier sensitivity classifiers\n* The order the figures are shown does not follow the order in which they appear in the text", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept