Title: A General Protocol to Probe Large Vision Models for 3D Physical Understanding
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D physical scene understanding; Representation probing protocol; Benchmark and evaluation
Abstract: Our objective in this paper is to probe large vision models to determine to what extent they ‘understand’ different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical ‘properties’ of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding.

A General Protocol to Probe Large Vision Models for 3D Physical Understanding

Abstract

Our objective in this paper is to probe large vision models to determine to what extent they 'understand' different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical 'properties' of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding. Our project page is https://www. robots.ox.ac.uk/~vgg/research/phy-sd/ .

1 Introduction

The large-scale pre-trained vision foundation models have achieved great success in computer vision tasks, including classification (CLIP [18, 30]), segmentation (DINO [5, 27]), and image generation (VQGAN [11], Stable Diffusion [31]) with strong generalisation capabilities. However, they are mainly trained with 2D images, which are the projection of the 3D physical world. This naturally raises the question of to what extent these large-scale vision models have learned about the 3D scene depicted with only the 2D images. Our objective in this paper is to investigate this question, and we do this precisely by determining whether features from these large-scale pre-trained vision models can be used to estimate the true geometric and physical properties of the 3D scene. If they can, then that is evidence that the models encode the 3D properties. For example, as an indication that Stable Diffusion is 3D and physics aware, Figure 1 shows the result of the off-the-shelf Stable Diffusion model [31] inpainting masked regions in real images - it correctly predicts shadows and supporting structures.

Chuanxia Zheng

Andrew Zisserman

To answer this question, we propose a general and lightweight evaluation protocol to systematically and efficiently 'probe' a pre-trained network on its ability to represent a number of 'properties' of the 3D scene and viewpoint. The protocol could be used for any network and any property of interest. We have probed properties including: 3D structure and material of the scene, such as surface layout; lighting, such as object-shadow relationships; and viewpoint dependent relations such as occlusion and depth.

The protocol involves three steps: First , a suitable real image evaluation dataset is selected that contains ground truth annotations for the property of interest, for example the SOBA dataset [41] is used to probe the understanding of lighting, as it has annotations for object-shadow associations. This provides a train/val/test set for that property; Second , a grid search is carried out over the layers and time steps of the Stable Diffusion model, and transformer layers for other models, to select the optimal feature for determining that property. The selection involves learning the weights of a simple linear classifier for that property ( e.g. 'are these two regions in an object-shadow relationship or not'); Third , the selected feature (layer, time step) and trained classifier are evaluated on a test set, and its performance answers the question of how well the model 'understands' that property.

In short, we probe scene geometry, material, support relation, shadow, occlusion and depth, to answer the question 'To what extent do large vision models encode 3D properties of the scene?". We apply this protocol to a wide range of networks that are trained at large scale, including OpenCLIP [18, 30], DINOv1 [5], DINOv2 [27], and VQGAN [11]. This covers networks trained generatively (Stable Diffusion), with self-supervision (DINOv1 & DINOv2), with weak supervision (OpenCLIP), and by auto-regression (VQGAN).

From our investigation, we make three observations: First , the Stable Diffusion and DINOv2 networks have a good 'understanding' of the scene geometry, support relations, the lighting, and the depth of a scene, with Stable Diffusion and DINOv2 having a similar and high prediction performance for these properties. However, their prediction of material and occlusion is poorer. Second , Stable Diffusion and DINOv2 generally demonstrate better performance for 3D properties than other networks trained at large scale: OpenCLIP, DINOv1, and VQGAN. Third , different time steps of Stable Diffusion

features, as well as different transformer layers of DINO/CLIP/VQGAN, perform best for different 3D physical properties.

Why is an understanding of the networks' ability to predict 3D properties useful? There are four reasons: (1) It begins to answer the scientific question of the extent to which these networks implicitly model the 3D scene; (2) The features we determined that are able to predict 3D physical properties can be used for this task, e.g. to predict shadow-object associations or support relations. This could either be carried out directly by incorporating them in a prediction network, in the manner of [51]; or they can be used indirectly as a means to train a feed forward network to predict the properties [42, 43]; (3) By knowing what properties Stable Diffusion is not good at, we have a way to spot images generated by Stable Diffusion, as has been done by [32]; (4) It also reveals which properties the network could be trained further on to improve its 3D modelling, e.g., via extra supervision for that property.

2 Related Work

2.1 Exploration of Pre-trained Models

Building on the success of large-scale vision networks, there has been significant interest from the community to understand what has been learned by these complex models. On discriminative models, for example, [26, 49] propose inverse reconstruction to directly visualize the acquired semantic information in various layers of a trained classification network; [12, 13, 57] demonstrate that scene classification networks have remarkable localization ability despite being trained on only image-level labels; and [10, 34, 37] use the gradients of any target concept, flowing into the final convolutional layer to produce a saliency map highlighting important regions in the image for predicting the concept. In the more recent literature, [7] explores what has been learned in the powerful transformer model by visualizing the attention map.

On generative models, researchers have mainly investigated what has been learned in GANs, for example, GAN dissection [3] presents an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level; [44] analyse the latent style space of StyleGANs [20]. The most recent work [32] studies the 3D geometric relations in generated images, such as vanishing points and shadows, and notes that the errors made can be used to discriminate real from generated images.

There is concurrent work [1] exploring the capability of predicting depth, surface normal and geometric correspondence for visual foundation models. To probe the depth and surface normal capability, a prediction network is trained using frozen multi-layer features from the foundation models. The capability for computing geometric correspondence between images is investigated using the dense spatial feature maps directly. In contrast to their work, we have studied a wider range of properties, covering both 3D geometric properties and 3D physical properties. Additionally, we have proposed a simple, general, yet efficient protocol for any property and any model, and have investigated the performance of different time steps and layers of models for different properties.

2.2 Exploitation of Pre-trained Models

Apart from understanding the representation in pre-trained models, there has been a recent trend for exploiting models trained at large scale, to tackle a series of downstream tasks. For example, leveraging generative models for data augmentation in recognition tasks [16, 19], utilising large vision models for semantic segmentation [2, 46], open-vocabulary segmentation [23], depth map estimation [21, 28, 35, 47, 48, 53, 56], correspondence estimation [17, 25, 27, 38, 54] and pose estimation [15, 55]. More recently, [4] searched for intrinsic offsets in a pre-trained StyleGAN for a range of downstream tasks, predicting normal maps, depth maps, segmentations, albedo maps, and shading.

2.3 3D Physical Scene Understanding

There have been works studying different 3D physical properties for scene understanding, including shadows [40, 41], material [39], occlusion [50], scene geometry [24], support relations [36] and depth [36]. However, these works focus on one or two physical properties, and most of them require training a model for the property in a supervised manner. In contrast, we use a single model to predict multiple properties, and do not train the features.

3 Method - Properties, Datasets, and Classifiers

Our goal is to examine the ability of large-scale vision models to understand different physical and geometrical properties of the 3D scene, including: scene geometry, material, support relations, shadows, occlusion and depth. Specifically, we conduct linear probing of the features from different layers and time steps of the Stable Diffusion model, and different transformer layers for other models including OpenCLIP, DINOv1, DINOv2 and VQGAN. First, we set up the questions for each property (Section 3.1); and then select real image datasets with ground truth annotations for each property (Section 3.2). We describe how a classifier is trained to answer the questions, and the grid search for the optimal time step and layer to extract a feature for predicting the property in Section 3.3.

3.1 Properties and Questions

Here, we study the large vision model's ability to predict different properties of the 3D scene; the properties cover the 3D structure and material, the lighting, and the viewpoint. For each property, we propose questions that classify the relationship between a pair of Regions , A and B , in the same image, based on the features extracted from the large vision model. The properties and questions are:

Same Plane : 'Are Region A and Region B on the same plane?'

Perpendicular Plane : 'Are Region A and Region B on perpendicular planes?'

Material : 'Are Region A and Region B made of the same material?'

Support Relation : 'Is Region A (object A ) supported by Region B (object B )?'

Shadow : 'Are Region A and Region B in an object-shadow relationship?'

Occlusion : 'Are Region A and Region B part of the same object but, separated by occlusion?'

Depth : 'Does Region A have a greater average depth than Region B ?'

We choose these properties as they exemplify important aspects of the 3D physical scene: the Same Plane and Perpendicular Plane questions probe the 3D scene geometry; the Material question probes what the surface is made of, e.g., metal, wood, glass, or fabric, rather than its shape; the Support Relation question probes the physics of the forces in the 3D scene; the Shadow question probes the lighting of the scene; the Occlusion and Depth questions depend on the viewpoint, and probe the disentanglement of the 3D scene from its viewpoint.

Original Image

Region A

Region C

Region B

negative pair

positive pair

3.2 Datasets

To study the different properties, we adopt various off-the-shelf real image datasets with annotations for the different properties, where the dataset used depends on the property. We repurpose each dataset to support probe questions of the form: D = { ( R A , R B , y ) 1 , . . . , ( R A , R B , y ) n } , where R A , R B denote a pair of regions, and y is the binary label indicating the answer to the considered question of the probed property. For each property, we create a train/val/test split from those of the original datasets, if all three splits are available. While for dataset with only train/test splits available, we divide the original train split into our train/val splits. Table 1 summarises the datasets used and the statistics of the splits used. We discuss each property and dataset in more detail next.

Same Plane. We use the ScanNetv2 dataset [8] with annotations of planes from [24]. Regions are obtained via splitting plane masks into several regions. A pair of regions are positive if they are on the same plane, and negative if they are on different planes. The first row of Figure 2 is an example.

Perpendicular Plane. We use the ScanNetv2 dataset [8]. We use the annotations from [24] which provide segmentation masks as well as plane parameters for planes in the image, so that we can obtain the normal of planes to judge whether they are perpendicular or not. A pair of regions are positive if they are on perpendicular planes, and negative if they are not on perpendicular planes. The second row of Figure 2 is an example.

Material. We adopt the recent DMS dataset [39] to study the material property, which provides dense annotations of material category for each pixel in the images. Therefore, we can get the mask of each material via grouping pixels with the same material label together. In total, there are 46 pre-defined material categories. Regions are obtained by splitting the mask of each material into different connected components, i.e., they are simply groups with the same material labels, yet not connected. A pair of regions are positive if they are of the same material category, and negative if they are of different material categories. First row of Figure 3 is an example.

Support Relation. We use the NYUv2 dataset [36] to probe the support relation. Segmentation annotations for different regions (objects or surfaces) are provided, as well as their support relations. Support relation here means an object is physically supported by another object, i.e., the second object will undertake the force to enable the first object to stay at its position. Regions are directly obtained via the segmentation annotations. A pair of regions are positive if the first region is supported by the

negative pair

Original Image

Region A

Region C

Region B

positive pair

second region, and negative if the first region is not supported by the second region. Second row of Figure 3 is an example.

Shadow. We use the SOBA dataset [40, 41] to study the shadows which depend on the lighting of the scene. Segmentation masks for each object and shadow, as well as their associations are provided in the dataset annotations. Regions are directly obtained from the annotated object and shadow masks. In a region pair, there is one object mask and one shadow mask. A pair of regions are positive if the shadow mask is the shadow of the object, and negative if the shadow mask is the shadow of another object. Third row of Figure 3 is an example.

Occlusion. We use the Seperated COCO dataset [50] to study the occlusion (object seperation) problem. Regions are different connected components of objects (and the object mask if it is not separated), i.e., groups of connected pixels belonging to the same object. A pair of regions are positive if they are different components of the same object separated due to occlusion, and negative if they are not from the same object. First row of Figure 4 is an example.

Depth. We use the NYUv2 dataset [36], which provides mask annotations for different objects and regions, together with depth for each pixel. A pair of regions are positive if the first region has a greater average depth than the second region, and negative if the first region has a less average depth than the second region. The average depth of a region is calculated via the average of depth value of each pixel the region contains. Second row of Figure 4 is an example.

3.3 Property Probing

Take Stable Diffusion as an example, we aim to determine which features best represent different properties. To obtain features from an off-the-shelf Stable Diffusion network, we follow the approach of [38] used for DIFT, where noise is added to the input image in the latent space, and features are extracted from different layers and time steps of the model. While probing the properties, linear classifiers are used to infer the relationships between regions . The region representation is obtained by a simple average pooling of the diffusion features over the annotated region or object.

Original Image

Region A

Region C

Region B

negative pair

positive pair

+

Extracting Stable Diffusion Features. We add noise ϵ ∼ N (0 , I ) of time step t ∈ [0 , T ] to the input image x 0 's latent representation z 0 encoded by the VAE encoder:

and then extract features from the immediate layers of a pre-trained diffusion model, f θ ( · ) after feeding z t and t in f θ ( f θ is a U-Net consisting of 4 downsampling layers and 4 upsampling layers):

where f θ l is the l -th U-Net layer. In this way, we can get the representation of an image F t,l at time step t and l -th U-Net layer for the probe. We upsample the obtained representation to the size of the original image with bi-linear, then use the region mask to get a region-wise feature vector, by averaging the feature vectors of each pixel it contains, i.e., average pooling.

where v k,t,l is the feature vector of the k -th region R k . ⊙ here is a per-pixel product of the region mask and the feature. For other models, including CLIP, DINOv1, DINOv2 and VQGAN, we feed the image into the ViT/Transformer and extract features from different layers. Then use average pooling as in Equation 3 to obtain the feature for each region.

Linear Probing. After extracting features from large-scale vision models, we use a linear classifier (a linear SVM) to examine how well these features can be used to answer questions to each of the properties. Specifically, the input of the classifier is the difference or absolute difference between the feature vectors of Region A and Region B , i.e., v A -v B or | v A -v B | , and the output is a Yes/No answer to the question. Denoting the answer to the question as Q , then since the questions about Same Plane , Perpendicular Plane , Material , Shadow and Occlusion are symmetric relations, Q ( v A , v B ) = Q ( v B , v A ) . However, the questions about Support Relation and Depth are not symmetric. Thus, we use | v A -v B | (a symmetric function) as input for the first group of questions, and v A -v B (non-symmetric) for the rest of questions. We train the linear classifier on the train set via the positive/negative samples of region pairs for each property; do a grid search on the validation set to find (i) the optimal time step t (only for Stable Diffusion), (ii) the U-Net layer l for Stable Diffusion and the Transformer layer l for other models, and (iii) the SVM regularization parameter C ; and evaluate the performance on the test set. The grid search is only feasible because our proposed protocol is lightweight, and can assess the effectiveness of features for different downstream tasks with minimal resource demands.

4 Experiments

In this section, we first give details of the grid search method in Section 4.1. We then give the linear probing grid search results on features from Stable Diffusion in Section 4.2 and from other networks trained at scale in Section 4.3. Finally, we compare all models on the test set in Section 4.4.

4.1 Implementation Details and Evaluation Metric

Implementation Details. For each property, we sample the same number of positive / negative pairs, to maintain a balanced evaluation set. In terms of the linear SVM, we tune the penalty parameter C on the val split to find the best C for each property. Therefore, we are grid searching 3 parameters on the val set, namely, Timestep t ranging from 1 to 1000 (only for Stable Diffusion), U-Net Layer l covering the 4 downsampling and 4 upsampling layers for Stable Diffusion and Transformer Layer l for other networks, and the SVM penalty parameter C ranging over 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 . The timestep is searched with a stride of 20 steps, since the difference in performance around the optimal value varies by less than 0.01 within 20 steps. In practice the C parameter is always between 0 . 1 and 1 , so we carry out a finer search over values between 0 . 1 and 1 . 0 in steps of 0 . 1 . The linear SVM is solved using the libsvm library [6] with the SMO algorithm, to get the unique global optimal solution. Please refer to the appendix for more implementation details.

Evaluation Metric. All protocols are binary classification, therefore, we use ROC Area Under the Curve (AUC Score) to evaluate the performance of the linear classifier, as it is not sensitive to different decision thresholds.

4.2 Results for Stable Diffusion

𝐸

!

𝐸

"

𝐸

#

𝐸

$

𝐷

$

𝐷

#

𝐷

"

𝐷

!

(a)

(b)

(c)

The results for grid search are shown in Table 2. For Stable Diffusion U-Net Layer, D l means the l -th layer of the U-Net decoder, i.e., upsampling layer, from outside to inside (right to left), and we provide an illustration of the layers in Figure 5(a).

We can make observations: First, the best time step for different properties varies, but the optimal time step is usually before 400. This is expected as a large time step adds too much noise, so not

much information is contained about the image. Second, in terms of the layer, the best U-Net layer is always D 3 in the decoder rather than the encoder. The optimal layer is in the middle, as D 1 is too close to the noise space and D 4 has just started decoding. Further explorations using Stable Diffusion features for downstream tasks could thus start from the U-Net decoder layers, especially D 3 . Third, in terms of the performance on the test set, we find that Stable Diffusion can understand very well about scene geometry, support relations, shadows, and depth, but it is less performant at predicting material and occlusion. Examples of its failure are shown in Figure 5 (b) (c). As noted in [50] and [22], grouping all separated parts of an object due to occlusion remains challenging even for state-of-the-art detection and segmentation models. The appendix gives grid search results at all time steps and layers.

4.3 Results for CLIP/DINO/VQGAN Features

In this section we show grid search results for OpenCLIP [18, 30] pre-trained on LAION dataset [33], DINOv1 [5] pre-trained on ImageNet dataset [9], DINOv2 [27] pre-trained on LVD142M dataset [27], and VQGAN [11] pre-trained on ImageNet dataset [9]. We use the best pre-trained checkpoints available on their official GitHub - ViT-B for DINOv1, ViT-G for OpenCLIP and DINOv2, and the 48-layer transformer checkpoint for VQGAN. Similar to Stable Diffusion, for each of these models, we conduct a grid search on the validation set in terms of the ViT/Transformer layer and C for SVM, and use the best combination of parameters for evaluation on the test set.

Grid search results are reported in Table 3. It can be observed that different layers of different models are good at different properties.

4.4 Comparison of Different Features Trained at Scale

We compare the state-of-the-art pre-trained large-scale vison models' representations on various downstream tasks in Table 4. It can be observed that the Stable Diffusion and DINOv2 representations outperform OpenCLIP, DINOv1 and VQGAN for all properties, indicating the potential of utilizing Stable Diffusion and DINOv2 representations for different downstream tasks with the optimal time steps and layers found in Section 4.2 and Section 4.3.

5 Discussion and Future Work

In this paper, we have developed a general and lightweight protocol to efficiently examine whether models pre-trained on large scale image datasets, like CLIP, DINO, VQGAN and Stable Diffusion, have explicit feature representations for different properties of the 3D physical scene.

It is interesting to find that different time steps of Stable Diffusion and different layers of DINOv2 representations can handle several different physical properties at a state-of-the-art performance, indicating the potential of utilising the Stable Diffusion and DINOv2 models for different downstream tasks.

However, for some properties such as material and occlusion, the networks are not distilling the information in a manner that can be used by a linear probe. This could indicate that these properties are not modelled well by the network or that more than a linear probe is required to tease them out. We show examples of the prediction failures for these properties in Figure 5. In the appendix, we show that such prediction failures also occur in generated (i.e. synthetic) images. It is worth noting that occlusion is a challenge even for the powerful Segment Anything Model (SAM) [22], where the model 'hallucinates small disconnected components at times'.

In the appendix, we provide preliminary results of using the probed Stable Diffusion feature for downstream tasks (Section E). We also provide examples of another use case of spotting Stable Diffusion generated images based on the properties that the model is not good at (Section B).

This paper has given some insight into answering the question: 'To what extent do large vision models understand the 3D scene' for real images. Of course, there are more properties that could be investigated in the manner proposed here. For example, contact relations [14] and object orientation [45], as well as the more nuanced non-symmetric formulations of the questions. Please refer to arxiv version of the paper [52] for future updates.

Broader Impact. This paper studies the learned representations of large vision models, and can help people better understand what has been learnt by these vision foundation models. We do not think there is any negative societal impact from this investigation.

Acknowledgements. This research is supported by EPSRC Programme Grant VisualAI EP / T028572 / 1, a Royal Society Research Professorship RP \ R1 \ 191132, an AWS credit funding, a China Oxford Scholarship and ERC-CoG UNION 101001212. We thank Yash Bhalgat, Minghao Chen, Subhabrata Choudhury, Kai Han, Tengda Han, Jaesung Huh, Vladimir Iashin, Tomas Jakab, Gyungin Shin, Ashish Thandavan, Vadim Tschernezki, Jianyuan Wang and Yan Xia from the Visual Geometry Group for their help and support for the project.

References

Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In International Conference on Learning Representations (ICLR) , 2022.

David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B. Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In Proceedings of the International Conference on Learning Representations (ICLR) , 2019.

Anand Bhattad, Daniel McKee, Derek Hoiem, and DA Forsyth. Stylegan knows normal, depth, albedo, and more. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021.

Chih-Chung Chang and Chih-Jen Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) , 2(3):1-27, 2011.

Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2009.

Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal , 1341(3):1, 2009.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the International Conference on Computer Vision (ICCV) , 2019.

Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2017.

David F Fouhey, Abhinav Gupta, and Andrew Zisserman. 3d shape attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Walter Goodwin, Sagar Vaze, Ioannis Havoutis, and Ingmar Posner. Zero-shot category-level object pose estimation. In European Conference on Computer Vision (ECCV) , 2022.

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition? In The Eleventh International Conference on Learning Representations (ICLR) , 2023.

Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.

Ali Jahanian, Xavier Puig, Yonglong Tian, and Phillip Isola. Generative models as a data source for multiview representation learning. In International Conference on Learning Representations (ICLR) , 2022.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) , 2024.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. Proceedings of the International Conference on Computer Vision (ICCV) , 2023.

Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Openvocabulary object segmentation with diffusion models. In Proceedings of the International Conference on Computer Vision (ICCV) , 2023.

Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. Planercnn: 3d plane detection and reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019.

Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2015.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023.

Suraj Patni, Aradhye Agarwal, and Chetan Arora. Ecodepth: Effective conditioning of diffusion models for monocular depth estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR) , 2024.

Luigi Piccinelli, Christos Sakaridis, and Fisher Yu. idisc: Internal discretization for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML) , 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2022.

Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik, David A Forsyth, and Anand Bhattad. Shadows don't lie and lines can't bend! generative models don't know projective geometry... for now. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Neural Information Processing Systems Datasets and Benchmarks Track (NeurIPS) , 2022.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the International Conference on Computer Vision (ICCV) , 2017.

Zifan Shi, Yujun Shen, Jiapeng Zhu, Dit-Yan Yeung, and Qifeng Chen. 3d-aware indoor scene synthesis with depth priors. In European Conference on Computer Vision (ECCV) , 2022.

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision (ECCV) , 2012.

Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR) , 2014.

Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Paul Upchurch and Ransen Niu. A dense material segmentation dataset for indoor and outdoor scene parsing. In European Conference on Computer Vision (ECCV) , 2022.

Tianyu Wang, Xiaowei Hu, Chi-Wing Fu, and Pheng-Ann Heng. Single-stage instance shadow detection with bidirectional relation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng-Ann Heng, and Chi-Wing Fu. Instance shadow detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. Proceedings of the International Conference on Computer Vision (ICCV) , 2023.

Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. In Robotics: Science and Systems (RSS) , 2018.

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.09414 , 2024.

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In 13th European Conference on Computer Vision (ECCV) , 2014.

Guanqi Zhan, Weidi Xie, and Andrew Zisserman. A tri-layer plugin to improve occluded detection. British Machine Vision Conference (BMVC) , 2022.

Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. Amodal ground truth and completion in the wild. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Guanqi Zhan, Chuanxia Zheng, Weidi Xie, and Andrew Zisserman. A general protocol to probe large vision models for 3d physical understanding. arXiv preprint arXiv:2310.06836 , 2023.

Fan Zhang, Shaodi You, Yu Li, and Ying Fu. Atlantis: Enabling underwater depth estimation with stable diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zeroshot semantic correspondence. Advances in Neural Information Processing Systems (NeurIPS) , 2023.

Kaifeng Zhang, Yang Fu, Shubhankar Borse, Hong Cai, Fatih Porikli, and Xiaolong Wang. Self-supervised geometric correspondence for category-level 6d object pose estimation in the wild. Proceedings of the International Conference on Learning Representations (ICLR) , 2023.

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the International Conference on Computer Vision (ICCV) , 2023.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Appendix

A More Implementation Details

Extracting Stable Diffusion Features. Following DIFT [38], when we extract Stable Diffusion features, we add a different random noise 8 times and then take the average of the generated features. The process can be completed in one forward pass as we are using a batch of 8. We use an empty prompt '' as the text prompt.

Train/Val Partition. For the partition of train/val split, we select the train & val images from different scenes for the NYUv2 [36] and ScanNetv2 [8] dataset.

Sampling of Images. For the train/val/test splits, if the number of images used is less than the original number of images in the datasets, we randomly sample our train/val/test images from the original datasets.

Sampling of Positive/Negative Pairs. For each image, if the number of possible negative pairs is larger than the number of possible positive pairs, we randomly sample from the negative pairs to obtain an equal number of negative and positive pairs, and vice versa. In this way, we keep a balanced sampling of positive and negative pairs for the binary linear classifier. As can be observed in Table 1, the number of train/val pairs for different properties are different, although we keep the same number of train/val images for different properties. This is because for different properties the availability of positive/negative pairs are different. For depth , we select a pair only if the average depth of one region is 1.2 times greater than the other because it is even challenging for humans to judge the depth order of two regions below this threshold. For perpendicular plane , taking the potential annotation errors into account, we select a pair as perpendicular if the angle between their normal vectors is greater than 85°and smaller than 95°, and select a pair as not perpendicular if the angle between their normal vectors is smaller than 60°or greater than 120°.

Region Filtering. When selecting the regions, we filter out the small regions, e.g., regions smaller than 100 pixels, because regions that are too small are challenging even for humans to annotate.

Image Filtering. As there are some noisy annotations in the [24] dataset, we manually filter the images whose annotations are inaccurate.

Linear SVM. The feature vectors are L2-normalised before inputting into the linear SVM. The binary decision of the SVM is given by sign ( w T v + b ) , where v is the input vector to SVM:

for the Same Plane , Perpendicular Plane , Material , Shadow and Occlusion questions, and

for the Support Relation and Depth questions.

Extension of Separated COCO. To study the occlusion problem, we utilise the Separated COCO dataset [50]. The original dataset only collects separated objects due to occlusion in the COCO 2017 val split, we further extend it to the COCO 2017 train split for more data using the same method as in [50]. As the original COCO val split is too small (only 5k images v.s. 118k images in the COCO train split), we get our partition of train/val/test splits by dividing the generated dataset ourselves.

Computing Resources. We run experiments on CPU with 20G memory for SVM training/testing.

B More Stable Diffusion Generated Images

Here we give more examples of Stable Diffusion generated images as mentioned in the caption of Figure 1. We show examples for: Scene Geometry in Figure 6; Material , Support Relations , and Shadows in Figure 7; and Occlusion and Depth in Figure 8.

The observations match our findings on studying the Stable Diffusion features - Stable Diffusion 'knows' about a number of physical properties including scene geometry, material, support relations, shadows, occlusion and depth, but may fail in some cases in terms of material and occlusion.

As mentioned in Section 5, we can spot Stable Diffusion generated images via the properties it is not good at. Take the second row of Figure 7 and the first row of Figure 8 as examples - we can spot that the images are generated by Stable Diffusion because of the failure to generate a clear boundary between two different materials, and the failure to connect separated parts due to occlusion.

Original Image

Masked Image

Inpainted Image

Inpainting Mask

二

Original Image

Masked Image

Inpainted Image

Inpainting Mask

Original Image

Masked Image

Inpainted Image

Inpainting Mask

70

C Train/Val AUC Results of Stable Diffusion Features

Table 5 shows the train/val AUC of the SVM grid search results for Stable Diffusion features at the best combination of time step, layer and C as in Table 4.2.

D AUC Curves for Stable Diffusion Features Grid Search

As mentioned in Section 4.2, we provide curves for AUC at different layers and time steps of probing Stale Diffusion features in Figure 9 - Figure 15.

Same Plane

0.975

D4

0.950

D2

D1

0.925

E1

E2

0.900

E3

E4

0.875

AU

0.850

0.825

0.800

0.775

0

100

200

300

400

500

Timestep

Perpendicular Plane

0.90

D4

D3

D2

0.85

D1

E1

E2

0.80

E3

E4

0.75

AUC

0.70

0.65

0.60

0

100

200

300

400

500

Timestep

Material

D4

0.800

D2

D1

0.775

E1

E2

E3

0.750

E4

0.725

0.700

0.675

0.650

0

100

200

300

400

500

Timestep

Support Relation

D4

D3

0.92

D2

D1

E1

0.90

E2

E3

E4

0.88

AUC

0.86

0.84

0.82

0

100

200

300

400

500

Timestep

Shadow

D4

0.95

D3

D2

0.90

D1

E1

E2

0.85

E3

0.80

E4

AUC

0.75

0.70

0.65

0.60

0.55-

0

100

200

300

400

500

Timestep

Occlusion

D4

D3

0.825

D2

D1

0.800

E1

E2

0.775

E3

E4

0.750

AUC

0.725

0.700

0.675

0.650

0

100

200

300

400

500

Timestep

Depth

D4

D3

0.98

D2

D1

E1

0.96

E2

E3

0.94

E4

AUC

0.92

0.90

0.88

0.86

0.84

0

100

200

300

400

500

Timestep

E Preliminary Results of Applying Probed Feature for Downstream Tasks

As mentioned in Section 5, we provide preliminary results of using the features selected by our probe for downstream tasks. We consider two tasks: surface normal estimation and depth estimation.

Surface Normal Estimation. For this, we use the specific time step and layer of the Stable Diffusion features selected by the 'perpendicular plane' probe. We directly inject the SD extracted features into the model iDisc [29] at the layer where the network encodes the image feature. When injecting Stable Diffusion feature and training for only 1000 iterations (whereas, [29] train 45000 iterations in total), the model performs better on most of the metrics compared with [29] reported results (see Table 6), which indicates that our feature probing strategy is relevant for corresponding 3D tasks.

Depth Estimation. We use the Stable Diffusion feature from the specific time step and layer (Table 2 Row 7) selected by the linear probe for depth, as the input to a simple convolutional network to predict the depth. The network is trained on the NYUv2 Depth training dataset with the SD features frozen. For comparison we train the same convolutional network using image features from a frozen ResNet-50 pre-trained on ImageNet. In Table 7 we can observe that the SD features have a substantially higher performance. This again illustrates the potential of the features selected by the probe for downstream applications.

In summary, we have shown two examples where features from Stable Diffusion, selected by our linear probe, support downstream 3D tasks. Features could also be selected for other downstream tasks, including Instance Shadow Detection [41], Material Segmentation [39], and Support Relation Inference [36].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have discussed the limitation and future work in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We described the details of the dataset and experiments in Section 3.2 and Section 4.1, as well as in Section A in the appendix. We have also included the code and data in the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have included our code and data in the supplementary.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have specify these in Section 3.2 and Section 4.1 of the main paper, as well as in Section A of the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information in Section A in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research confirms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have a paragraph to discuss broader impact before the references.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have properly credited the creators or original owners of assets. The name of the license are CC-BY 4.0 for all assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have the related information in the supplementary zip.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to evaluate how well large-scale vision models encode 3D properties of the scenes depicted in images. The paper proposes a general and lightweight protocol that involves training discriminative classifiers on the features of pre-trained models to assess their encoding of several physical properties of 3D scenes. These properties include scene geometry, materials, support relations, lighting, shadow, occlusion and depth.\n\nThis protocol is applied to several large vision models, such as CLIP, DINOv1, DINOv2, VQGAN, and Stable Diffusion. The findings indicate that features from Stable Diffusion and DINOv2 are particularly adept at discriminative learning for properties like scene geometry, support relations, shadows, and depth. However, they perform less well for occlusion and material properties. The results also show that different layers and time steps of these models are good at different properties, which could have potential applications for 3D physical understanding.\n\nSTRENGTHS:\n1. This paper proposes a protocol to assess the 3D awareness for current large-scale vision models. The protocol considers various physical properties and is lightweight.\n\n2. The overall structure and writing are easy to follow.\n\n3. The problem investigated in this paper is very interesting. It provides a new perspective to interpret the features learned by large-scale models, especially their 3D awareness. This is very valuable for future studies.\n\nWEAKNESSES:\n1. It seems unclear to me whether the proposed probes can really reflect the 3D understanding of large vision models, due to properties to be probed and the way to probe them. For example, the material, support relation and shadow properties, from my point of view, can be well identified with 2D clues (e.g. appearance and 2D spatial location). For occlusion and depth, using linear SVM to answer binary questions may not be enough to assess these properties.\n\n2. There is a lack of baselines for the linear probing. It is unclear how a 3D-aware model trained explicitly with 3D data will respond to probe questions (upper bound). Also, it is better to show how the models with little 3D awareness will react to the probes (lower bound). Otherwise, it is less informative just comparing between large vision models, especially when their scores are close and pretty high as shown in table 4.\n\n3. The paper “Probing the 3d awareness of visual foundation models” [1] shares a very similar goal with this paper. It’s suggested to add comparison and discussion with that paper. \n\n4. It would be more comprehensive to evaluate more large vision models such as SAM [2] and MAE [3].\n\n[1] Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. CVPR, 2024.\n\n[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. ICCV, 2023.\n\n[3] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022\n\nQUESTIONS:\n1. Have the large-scale models been trained on the dataset chosen in this paper? Does it have influence on the probe results?\n\n2. In Table 4, instead of using a random classifier as the baseline, it is fairer to train a feature extractor from scratch with your probing training set.\n\n3. Following the second point in Weakness, could you provide the upper bound and lower bound of the probe? That’s essential to justify that the probing tasks can actually reflect 3D awareness.\n\n4. Currently, the input to the linear SVM is designed as the difference between the averaged pooled features from two patches. Will pooling and subtraction operations lead to information loss? Is there any better way to formulate the linear probing?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates several mainstream large vision models for their 3D physical understanding. The authors curated a binary classification benchmark covering a set of important 3D properties based on publicly available 2D image datasets and linearly probed different layers and different time steps of the vision models. Consequently, the authors find that DINOv2 and Stable Diffusion perform the best among the probed models, while some properties such as material and occlusion challenge them all. Consequently, this work provides a general linear probing protocol and dataset to evaluate a vision model's capability in 3D physical understanding.\n\nSTRENGTHS:\n1. Significance. The reviewer thinks this work studies a significant question. With the emergence of large vision models trained on 2D images, how and how well they could be applied to 3D-related tasks is important and of growing interest.\n2. Originality. This work defines a set of properties and corresponding tasks for evaluating 3D physical understanding. This is a novel benchmark for evaluating vision models.\n3. Clarity. The writing is clear and easy to follow. Section 3 presents clear demonstrations of the datasets and task definitions.\n4. Quality. Through and organized experiments are conducted. Grid search is applied to timesteps and layers.\n\nWEAKNESSES:\n1. Lack of in-depth analysis and understanding. While the paper spends a big part describing the methods and experiments, an analysis of the performance differences seems to be glossed over. As many of the popular vision models are benchmarked, what are the authors' speculations and thoughts on the cause of the varied performance? Does the difference in data and training objectives play a part? The reviewer believes providing further analysis and understanding of the models will make this paper appreciated by more audiences.\n\nQUESTIONS:\n- For curating the dataset, how are the regions selected? For those obtained from annotation masks, are all the regions kept in the final dataset? Is there any human intervention and selection?\n- Does the size of regions affect the task difficulty?\n- Does patch size affect the performance? What are the patch sizes of the evaluated models?\n\nLIMITATIONS:\nThe authors have discussed some limitations and broader impacts in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
In order to efficiently examine whether large vision models have explicit feature representations for different properties of the 3D physical scene, the paper proposes to linearly probe the features of different layers (and different time steps) from LVMs on specially designed binary classification problems. Extensive experiments demonstrate the effectiveness of the proposed probing scheme.

STRENGTHS:
1. As far as I am concerned, the paper is the first to investigate the 3D knowledge learned by LVMs in a lightweight manner with linear probing.
2. For quality, the authors conduct extensive experiments on several datasets to validate the effectiveness of their strategy.
3. For clarity, the paper is well written and easy to follow.
4. For significance, it is crucial to investigate to what extent the pretrained LVMs understand the 3D physical world and the applications.

WEAKNESSES:
Although the paper unveils the 3D perception abilities of pretrained LVMs using linear probing, it is still very simple and far from real application in downstream tasks. It would be better if the author could give more downstream applications in addition to the one in the appendix.

QUESTIONS:
Please see the weakness section for details.

LIMITATIONS:
The limitations are well discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11