A General Protocol to Probe Large Vision Models for 3D Physical Understanding

Authors: guanqi  zhan
chuanxia  zheng
weidi  xie
andrew  zisserman
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:20:39

Abstract: Our objective in this paper is to probe large vision models to determine to what extent they ‘understand’ different physical properties of the 3D scene depicted in an image. To this end, we make the following contributions: (i) We introduce a general and lightweight protocol to evaluate whether features of an off-the-shelf large vision model encode a number of physical ‘properties’ of the 3D scene, by training discriminative classifiers on the features for these properties. The probes are applied on datasets of real images with annotations for the property. (ii) We apply this protocol to properties covering scene geometry, scene material, support relations, lighting, and view-dependent measures, and large vision models including CLIP, DINOv1, DINOv2, VQGAN, Stable Diffusion. (iii) We find that features from Stable Diffusion and DINOv2 are good for discriminative learning of a number of properties, including scene geometry, support relations, shadows and depth, but less performant for occlusion and material, while outperforming DINOv1, CLIP and VQGAN for all properties. (iv) It is observed that different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, are good at different properties, unlocking potential applications of 3D physical understanding.

Main paper: 
 introduction: 
 The large-scale pre-trained vision foundation models have achieved great success in computer vision tasks, including classification (CLIP [18, 30] ), segmentation (DINO [5, 27] ), and image generation (VQGAN [11] , Stable Diffusion [31] ) with strong generalisation capabilities. However, they are mainly trained with 2D images, which are the projection of the 3D physical world. This naturally raises the question of to what extent these large-scale vision models have learned about the 3D scene depicted with only the 2D images. Our objective in this paper is to investigate this question, and we do this precisely by determining whether features from these large-scale pre-trained vision models can be used to estimate the true geometric and physical properties of the 3D scene. If they can, then that is evidence that the models encode the 3D properties. For example, as an indication that Stable Diffusion is 3D and physics aware, Figure 1 shows the result of the off-the-shelf Stable Diffusion model [31] inpainting masked regions in real images -it correctly predicts shadows and supporting structures.Figure 1 : Motivation: What do large vision models know about the 3D scene? We take Stable Diffusion as an example because Stable Diffusion is generative, and so its output is an image that can be judged directly for verisimilitude. The Stable Diffusion inpainting model is here tasked with inpainting the masked region of the real images. It correctly predicts a shadow consistent with the lighting direction (top), and a supporting structure consistent with the scene geometry (bottom). This indicates that the Stable Diffusion model generation is consistent with the geometry (of the light source direction) and physical (support) properties. These examples are only for illustration and we probe a general Stable Diffusion network to determine whether there are explicit features for such 3D scene properties. The appendix provides more examples of Stable Diffusion's capability to predict different physical properties of the scene.To answer this question, we propose a general and lightweight evaluation protocol to systematically and efficiently 'probe' a pre-trained network on its ability to represent a number of 'properties' of the 3D scene and viewpoint. The protocol could be used for any network and any property of interest. We have probed properties including: 3D structure and material of the scene, such as surface layout; lighting, such as object-shadow relationships; and viewpoint dependent relations such as occlusion and depth.The protocol involves three steps: First, a suitable real image evaluation dataset is selected that contains ground truth annotations for the property of interest, for example the SOBA dataset [41] is used to probe the understanding of lighting, as it has annotations for object-shadow associations. This provides a train/val/test set for that property; Second, a grid search is carried out over the layers and time steps of the Stable Diffusion model, and transformer layers for other models, to select the optimal feature for determining that property. The selection involves learning the weights of a simple linear classifier for that property (e.g. 'are these two regions in an object-shadow relationship or not'); Third, the selected feature (layer, time step) and trained classifier are evaluated on a test set, and its performance answers the question of how well the model 'understands' that property.In short, we probe scene geometry, material, support relation, shadow, occlusion and depth, to answer the question "To what extent do large vision models encode 3D properties of the scene?". We apply this protocol to a wide range of networks that are trained at large scale, including OpenCLIP [18, 30] , DINOv1 [5] , DINOv2 [27] , and VQGAN [11] . This covers networks trained generatively (Stable Diffusion), with self-supervision (DINOv1 & DINOv2), with weak supervision (OpenCLIP), and by auto-regression (VQGAN).From our investigation, we make three observations: First, the Stable Diffusion and DINOv2 networks have a good 'understanding' of the scene geometry, support relations, the lighting, and the depth of a scene, with Stable Diffusion and DINOv2 having a similar and high prediction performance for these properties. However, their prediction of material and occlusion is poorer. Second, Stable Diffusion and DINOv2 generally demonstrate better performance for 3D properties than other networks trained at large scale: OpenCLIP, DINOv1, and VQGAN. Third, different time steps of Stable Diffusion features, as well as different transformer layers of DINO/CLIP/VQGAN, perform best for different 3D physical properties.Why is an understanding of the networks' ability to predict 3D properties useful? There are four reasons: (1) It begins to answer the scientific question of the extent to which these networks implicitly model the 3D scene; (2) The features we determined that are able to predict 3D physical properties can be used for this task, e.g. to predict shadow-object associations or support relations. This could either be carried out directly by incorporating them in a prediction network, in the manner of [51] ; or they can be used indirectly as a means to train a feed forward network to predict the properties [42, 43] ; (3) By knowing what properties Stable Diffusion is not good at, we have a way to spot images generated by Stable Diffusion, as has been done by [32] ; (4) It also reveals which properties the network could be trained further on to improve its 3D modelling, e.g., via extra supervision for that property. 
exploration of pre-trained models: 
 Building on the success of large-scale vision networks, there has been significant interest from the community to understand what has been learned by these complex models. On discriminative models, for example, [26, 49] propose inverse reconstruction to directly visualize the acquired semantic information in various layers of a trained classification network; [12, 13, 57] demonstrate that scene classification networks have remarkable localization ability despite being trained on only image-level labels; and [10, 34, 37] use the gradients of any target concept, flowing into the final convolutional layer to produce a saliency map highlighting important regions in the image for predicting the concept. In the more recent literature, [7] explores what has been learned in the powerful transformer model by visualizing the attention map.On generative models, researchers have mainly investigated what has been learned in GANs, for example, GAN dissection [3] presents an analytic framework to visualize and understand GANs at the unit-, object-, and scene-level; [44] analyse the latent style space of StyleGANs [20] . The most recent work [32] studies the 3D geometric relations in generated images, such as vanishing points and shadows, and notes that the errors made can be used to discriminate real from generated images.There is concurrent work [1] exploring the capability of predicting depth, surface normal and geometric correspondence for visual foundation models. To probe the depth and surface normal capability, a prediction network is trained using frozen multi-layer features from the foundation models. The capability for computing geometric correspondence between images is investigated using the dense spatial feature maps directly. In contrast to their work, we have studied a wider range of properties, covering both 3D geometric properties and 3D physical properties. Additionally, we have proposed a simple, general, yet efficient protocol for any property and any model, and have investigated the performance of different time steps and layers of models for different properties.Apart from understanding the representation in pre-trained models, there has been a recent trend for exploiting models trained at large scale, to tackle a series of downstream tasks. For example, leveraging generative models for data augmentation in recognition tasks [16, 19] , utilising large vision models for semantic segmentation [2, 46] , open-vocabulary segmentation [23] , depth map estimation [21, 28, 35, 47, 48, 53, 56] , correspondence estimation [17, 25, 27, 38, 54] and pose estimation [15, 55] . More recently, [4] searched for intrinsic offsets in a pre-trained StyleGAN for a range of downstream tasks, predicting normal maps, depth maps, segmentations, albedo maps, and shading.There have been works studying different 3D physical properties for scene understanding, including shadows [40, 41] , material [39] , occlusion [50] , scene geometry [24] , support relations [36] and depth [36] . However, these works focus on one or two physical properties, and most of them require training a model for the property in a supervised manner. In contrast, we use a single model to predict multiple properties, and do not train the features. 3 Method -Properties, Datasets, and ClassifiersOur goal is to examine the ability of large-scale vision models to understand different physical and geometrical properties of the 3D scene, including: scene geometry, material, support relations, shadows, occlusion and depth. Specifically, we conduct linear probing of the features from different layers and time steps of the Stable Diffusion model, and different transformer layers for other models including OpenCLIP, DINOv1, DINOv2 and VQGAN. First, we set up the questions for each property (Section 3.1); and then select real image datasets with ground truth annotations for each property (Section 3.2). We describe how a classifier is trained to answer the questions, and the grid search for the optimal time step and layer to extract a feature for predicting the property in Section 3.3. 
properties and questions: 
 Here, we study the large vision model's ability to predict different properties of the 3D scene; the properties cover the 3D structure and material, the lighting, and the viewpoint. For each property, we propose questions that classify the relationship between a pair of Regions, A and B, in the same image, based on the features extracted from the large vision model. The properties and questions are:• Same Plane: 'Are Region A and Region B on the same plane?'• Perpendicular Plane: 'Are Region A and Region B on perpendicular planes?'• Material: 'Are Region A and Region B made of the same material?'• Support Relation: 'Is Region A (object A) supported by Region B (object B)?'• Shadow: 'Are Region A and Region B in an object-shadow relationship?'• Occlusion: 'Are Region A and Region B part of the same object but, separated by occlusion?'• Depth: 'Does Region A have a greater average depth than Region B?'We choose these properties as they exemplify important aspects of the 3D physical scene: the Same Plane and Perpendicular Plane questions probe the 3D scene geometry; the Material question probes what the surface is made of, e.g., metal, wood, glass, or fabric, rather than its shape; the Support Relation question probes the physics of the forces in the 3D scene; the Shadow question probes the lighting of the scene; the Occlusion and Depth questions depend on the viewpoint, and probe the disentanglement of the 3D scene from its viewpoint. (A, B ) are a positive pair, while (A, C) are negative. The images are from the ScanNetv2 dataset [8] with annotations for planes from [24] . In the first row, the first piece of floor (A) is on the same plane as the second piece of floor (B), but is not on the same plane as the surface of the drawers (C). In the second row, the table top (A) is perpendicular to the wall (B), but is not perpendicular to the stool top (C).To study the different properties, we adopt various off-the-shelf real image datasets with annotations for the different properties, where the dataset used depends on the property. We repurpose each dataset to support probe questions of the form: D = {(R A , R B , y) 1 , . . . , (R A , R B , y) n }, where R A , R B denote a pair of regions, and y is the binary label indicating the answer to the considered question of the probed property. For each property, we create a train/val/test split from those of the original datasets, if all three splits are available. While for dataset with only train/test splits available, we divide the original train split into our train/val splits. Table 1 summarises the datasets used and the statistics of the splits used. We discuss each property and dataset in more detail next. Same Plane. We use the ScanNetv2 dataset [8] with annotations of planes from [24] . Regions are obtained via splitting plane masks into several regions. A pair of regions are positive if they are on the same plane, and negative if they are on different planes. The first row of Figure 2 is an example. Perpendicular Plane. We use the ScanNetv2 dataset [8] . We use the annotations from [24] which provide segmentation masks as well as plane parameters for planes in the image, so that we can obtain the normal of planes to judge whether they are perpendicular or not. A pair of regions are positive if they are on perpendicular planes, and negative if they are not on perpendicular planes. The second row of Figure 2 is an example.Material. We adopt the recent DMS dataset [39] to study the material property, which provides dense annotations of material category for each pixel in the images. Therefore, we can get the mask of each material via grouping pixels with the same material label together. In total, there are 46 pre-defined material categories. Regions are obtained by splitting the mask of each material into different connected components, i.e., they are simply groups with the same material labels, yet not connected. A pair of regions are positive if they are of the same material category, and negative if they are of different material categories. First row of Figure 3 is an example. Support Relation. We use the NYUv2 dataset [36] to probe the support relation. Segmentation annotations for different regions (objects or surfaces) are provided, as well as their support relations. Support relation here means an object is physically supported by another object, i.e., the second object will undertake the force to enable the first object to stay at its position. Regions are directly obtained via the segmentation annotations. A pair of regions are positive if the first region is supported by the [36] . The paper (A) is supported by the table (B), but it is not supported by the chair (C). Third row: the shadow images are from the SOBA dataset [41] . The person (A) has the shadow (B), not the shadow (C). second region, and negative if the first region is not supported by the second region. Second row of Figure 3 is an example.Shadow. We use the SOBA dataset [40, 41] to study the shadows which depend on the lighting of the scene. Segmentation masks for each object and shadow, as well as their associations are provided in the dataset annotations. Regions are directly obtained from the annotated object and shadow masks. In a region pair, there is one object mask and one shadow mask. A pair of regions are positive if the shadow mask is the shadow of the object, and negative if the shadow mask is the shadow of another object. Third row of Figure 3 is an example.Occlusion. We use the Seperated COCO dataset [50] to study the occlusion (object seperation) problem. Regions are different connected components of objects (and the object mask if it is not separated), i.e., groups of connected pixels belonging to the same object. A pair of regions are positive if they are different components of the same object separated due to occlusion, and negative if they are not from the same object. First row of Figure 4 is an example.Depth. We use the NYUv2 dataset [36] , which provides mask annotations for different objects and regions, together with depth for each pixel. A pair of regions are positive if the first region has a greater average depth than the second region, and negative if the first region has a less average depth than the second region. The average depth of a region is calculated via the average of depth value of each pixel the region contains. Second row of Figure 4 is an example.Take Stable Diffusion as an example, we aim to determine which features best represent different properties. To obtain features from an off-the-shelf Stable Diffusion network, we follow the approach of [38] used for DIFT, where noise is added to the input image in the latent space, and features are extracted from different layers and time steps of the model. While probing the properties, linear classifiers are used to infer the relationships between regions. The region representation is obtained by a simple average pooling of the diffusion features over the annotated region or object. Extracting Stable Diffusion Features. We add noise ϵ ∼ N (0, I) of time step t ∈ [0, T ] to the input image x 0 's latent representation z 0 encoded by the VAE encoder:EQUATIONand then extract features from the immediate layers of a pre-trained diffusion model, f θ (•) after feeding z t and t in f θ (f θ is a U-Net consisting of 4 downsampling layers and 4 upsampling layers):EQUATIONwhere f θ l is the l-th U-Net layer. In this way, we can get the representation of an image F t,l at time step t and l-th U-Net layer for the probe. We upsample the obtained representation to the size of the original image with bi-linear, then use the region mask to get a region-wise feature vector, by averaging the feature vectors of each pixel it contains, i.e., average pooling.EQUATIONwhere v k,t,l is the feature vector of the k-th region R k . ⊙ here is a per-pixel product of the region mask and the feature. For other models, including CLIP, DINOv1, DINOv2 and VQGAN, we feed the image into the ViT/Transformer and extract features from different layers. Then use average pooling as in Equation 3 to obtain the feature for each region.Linear Probing. After extracting features from large-scale vision models, we use a linear classifier (a linear SVM) to examine how well these features can be used to answer questions to each of the properties. Specifically, the input of the classifier is the difference or absolute difference between the feature vectors of Region A and Region B, i.e., v A -v B or |v A -v B |, and the output is a Yes/No answer to the question. Denoting the answer to the question as Q, then since the questions about Same Plane, Perpendicular Plane, Material, Shadow and Occlusion are symmetric relations,Q(v A , v B ) = Q(v B , v A ).However, the questions about Support Relation and Depth are not symmetric. Thus, we use |v A -v B | (a symmetric function) as input for the first group of questions, and v A -v B (non-symmetric) for the rest of questions. We train the linear classifier on the train set via the positive/negative samples of region pairs for each property; do a grid search on the validation set to find (i) the optimal time step t (only for Stable Diffusion), (ii) the U-Net layer l for Stable Diffusion and the Transformer layer l for other models, and (iii) the SVM regularization parameter C; and evaluate the performance on the test set. The grid search is only feasible because our proposed protocol is lightweight, and can assess the effectiveness of features for different downstream tasks with minimal resource demands. 
experiments: 
 In this section, we first give details of the grid search method in Section 4.1. We then give the linear probing grid search results on features from Stable Diffusion in Section 4.2 and from other networks trained at scale in Section 4.3. Finally, we compare all models on the test set in Section 4.4. The timestep is searched with a stride of 20 steps, since the difference in performance around the optimal value varies by less than 0.01 within 20 steps. In practice the C parameter is always between 0.1 and 1, so we carry out a finer search over values between 0.1 and 1.0 in steps of 0.1. The linear SVM is solved using the libsvm library [6] with the SMO algorithm, to get the unique global optimal solution. Please refer to the appendix for more implementation details.Evaluation Metric. All protocols are binary classification, therefore, we use ROC Area Under the Curve (AUC Score) to evaluate the performance of the linear classifier, as it is not sensitive to different decision thresholds. The results for grid search are shown in Table 2 . For Stable Diffusion U-Net Layer, D l means the l-th layer of the U-Net decoder, i.e., upsampling layer, from outside to inside (right to left), and we provide an illustration of the layers in Figure 5 (a).𝐸 ! 𝐸 " 𝐸 # 𝐸 $ 𝐷 $ 𝐷 # 𝐷 " 𝐷 ! (a) (b) (c)We can make observations: First, the best time step for different properties varies, but the optimal time step is usually before 400. This is expected as a large time step adds too much noise, so not much information is contained about the image. Second, in terms of the layer, the best U-Net layer is always D 3 in the decoder rather than the encoder. The optimal layer is in the middle, as D 1 is too close to the noise space and D 4 has just started decoding. Further explorations using Stable Diffusion features for downstream tasks could thus start from the U-Net decoder layers, especially D 3 . Third, in terms of the performance on the test set, we find that Stable Diffusion can understand very well about scene geometry, support relations, shadows, and depth, but it is less performant at predicting material and occlusion. Examples of its failure are shown in Figure 5 (b) (c). As noted in [50] and [22] , grouping all separated parts of an object due to occlusion remains challenging even for state-of-the-art detection and segmentation models. The appendix gives grid search results at all time steps and layers. In this section we show grid search results for OpenCLIP [18, 30] pre-trained on LAION dataset [33] , DINOv1 [5] pre-trained on ImageNet dataset [9] , DINOv2 [27] pre-trained on LVD-142M dataset [27] , and VQGAN [11] pre-trained on ImageNet dataset [9] . We use the best pre-trained checkpoints available on their official GitHub -ViT-B for DINOv1, ViT-G for OpenCLIP and DI-NOv2, and the 48-layer transformer checkpoint for VQGAN. Similar to Stable Diffusion, for each of these models, we conduct a grid search on the validation set in terms of the ViT/Transformer layer and C for SVM, and use the best combination of parameters for evaluation on the test set.Grid search results are reported in Table 3 . It can be observed that different layers of different models are good at different properties. However, for some properties such as material and occlusion, the networks are not distilling the information in a manner that can be used by a linear probe. This could indicate that these properties are not modelled well by the network or that more than a linear probe is required to tease them out.We show examples of the prediction failures for these properties in Figure 5 . In the appendix, we show that such prediction failures also occur in generated (i.e. synthetic) images. It is worth noting that occlusion is a challenge even for the powerful Segment Anything Model (SAM) [22] , where the model 'hallucinates small disconnected components at times'.In the appendix, we provide preliminary results of using the probed Stable Diffusion feature for downstream tasks (Section E). We also provide examples of another use case of spotting Stable Diffusion generated images based on the properties that the model is not good at (Section B).This paper has given some insight into answering the question: 'To what extent do large vision models understand the 3D scene' for real images. Of course, there are more properties that could be investigated in the manner proposed here. For example, contact relations [14] and object orientation [45] , as well as the more nuanced non-symmetric formulations of the questions. Please refer to arxiv version of the paper [52] for future updates.Broader Impact. This paper studies the learned representations of large vision models, and can help people better understand what has been learnt by these vision foundation models. We do not think there is any negative societal impact from this investigation.Here we give more examples of Stable Diffusion generated images as mentioned in the caption of Figure 1 . We show examples for: Scene Geometry in Figure 6 ; Material, Support Relations, and Shadows in Figure 7 ; and Occlusion and Depth in Figure 8 .The observations match our findings on studying the Stable Diffusion features -Stable Diffusion 'knows' about a number of physical properties including scene geometry, material, support relations, shadows, occlusion and depth, but may fail in some cases in terms of material and occlusion.As mentioned in Section 5, we can spot Stable Diffusion generated images via the properties it is not good at. Take the second row of Figure 7 and the first row of Figure 8 as examples -we can spot that the images are generated by Stable Diffusion because of the failure to generate a clear boundary between two different materials, and the failure to connect separated parts due to occlusion.Figure 6 : Stable Diffusion generated images testing scene geometry prediction. Here and for the following figures, the model is tasked with inpainting the masked region of the real images. Stable Diffusion 'knows' about same plane and perpendicular plane relations in the generation. When the intersection of two sofa planes (first row), two walls (second and sixth row), two cabinet planes (third row), two pillar planes (fourth row) or two fridge planes (fifth row) is masked out, Stable Diffusion is able to generate the two perpendicular planes at the corner based on the unmasked parts of the planes.Inpainted Image Inpainting Mask Stable Diffusion 'knows' about support relations and shadows in the generation, but may fail sometimes for material. Rows 1-2: Material; Rows 3-4: Support Relation; Rows 5-6: Shadow. In the first row, the model distinguishes the two different materials clearly and there is clear boundary between the generated pancake and plate; while in the second row, the model fails to distinguish the two different materials clearly, generating a mixed boundary. In the third row and fourth rows, the model does inpaint the supporting object for the stuff on the table and the machine. In the fifth and sixth rows, the model manages to inpaint the shadow correctly. Better to zoom in for more details. As mentioned in Section 4.2, we provide curves for AUC at different layers and time steps of probing Stale Diffusion features in Figure 9 -Figure 15 . As mentioned in Section 5, we provide preliminary results of using the features selected by our probe for downstream tasks. We consider two tasks: surface normal estimation and depth estimation.Surface Normal Estimation. For this, we use the specific time step and layer of the Stable Diffusion features selected by the 'perpendicular plane' probe. We directly inject the SD extracted features into the model iDisc [29] at the layer where the network encodes the image feature. When injecting Stable Diffusion feature and training for only 1000 iterations (whereas, [29] train 45000 iterations in total), the model performs better on most of the metrics compared with [29] reported results (see Table 6 ), which indicates that our feature probing strategy is relevant for corresponding 3D tasks.Table 6 : Preliminary results of using the probed feature for the downstream task of normal estimation. Here we show the results of injecting the selected Stable Diffusion feature into iDisc [29] .Please see text for more details. Depth Estimation. We use the Stable Diffusion feature from the specific time step and layer (Table 2 Row 7) selected by the linear probe for depth, as the input to a simple convolutional network to predict the depth. The network is trained on the NYUv2 Depth training dataset with the SD features frozen. For comparison we train the same convolutional network using image features from a frozen ResNet-50 pre-trained on ImageNet. In Table 7 we can observe that the SD features have a substantially higher performance. This again illustrates the potential of the features selected by the probe for downstream applications. In summary, we have shown two examples where features from Stable Diffusion, selected by our linear probe, support downstream 3D tasks. Features could also be selected for other downstream tasks, including Instance Shadow Detection [41] , Material Segmentation [39] , and Support Relation Inference [36] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have included our code and data in the supplementary.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have specify these in Section 3.2 and Section 4.1 of the main paper, as well as in Section A of the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We have a paragraph to discuss broader impact before the references. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly credited the creators or original owners of assets. The name of the license are CC-BY 4.0 for all assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix:  


Decision: This paper proposes to explore large vision models to determine to what extent they "understand" different physical properties of the 3D scene depicted in an image. The authors conduct extensive experiments with a variety of models and reach some encouraging conclusions. After a rebuttal discussion period, the paper received all positive reviews. The reviewers agreed that the proposed protocol is essential, the results are encouraging, and the paper is well written. AC recommends that the authors incorporate all reviewers' feedback into the final camera-ready version and congratulates the paper on its acceptance.

Reviews: review: paper_summary: paper_summary: In order to efficiently examine whether large vision models have explicit feature representations for different properties of the 3D physical scene, the paper proposes to linearly probe the features of different layers (and different time steps) from LVMs on specially designed binary classification problems. Extensive experiments demonstrate the effectiveness of the proposed probing scheme.
strength_weakness: strength_weakness: Although the paper unveils the 3D perception abilities of pretrained LVMs using linear probing, it is still very simple and far from real application in downstream tasks. It would be better if the author could give more downstream applications in addition to the one in the appendix.
questions: questions: Please see the weakness section for details.
limitations: limitations: The limitations are well discussed in the paper.
paper_summary: paper_summary: This paper investigates several mainstream large vision models for their 3D physical understanding. The authors curated a binary classification benchmark covering a set of important 3D properties based on publicly available 2D image datasets and linearly probed different layers and different time steps of the vision models. Consequently, the authors find that DINOv2 and Stable Diffusion perform the best among the probed models, while some properties such as material and occlusion challenge them all. Consequently, this work provides a general linear probing protocol and dataset to evaluate a vision model's capability in 3D physical understanding.
strength_weakness: strength_weakness: 1. Lack of in-depth analysis and understanding. While the paper spends a big part describing the methods and experiments, an analysis of the performance differences seems to be glossed over. As many of the popular vision models are benchmarked, what are the authors' speculations and thoughts on the cause of the varied performance? Does the difference in data and training objectives play a part? The reviewer believes providing further analysis and understanding of the models will make this paper appreciated by more audiences.
questions: questions: - For curating the dataset, how are the regions selected? For those obtained from annotation masks, are all the regions kept in the final dataset? Is there any human intervention and selection?
- Does the size of regions affect the task difficulty?
- Does patch size affect the performance? What are the patch sizes of the evaluated models?
limitations: limitations: The authors have discussed some limitations and broader impacts in Section 5.
paper_summary: paper_summary: This paper aims to evaluate how well large-scale vision models encode 3D properties of the scenes depicted in images. The paper proposes a general and lightweight protocol that involves training discriminative classifiers on the features of pre-trained models to assess their encoding of several physical properties of 3D scenes. These properties include scene geometry, materials, support relations, lighting, shadow, occlusion and depth.

This protocol is applied to several large vision models, such as CLIP, DINOv1, DINOv2, VQGAN, and Stable Diffusion. The findings indicate that features from Stable Diffusion and DINOv2 are particularly adept at discriminative learning for properties like scene geometry, support relations, shadows, and depth. However, they perform less well for occlusion and material properties. The results also show that different layers and time steps of these models are good at different properties, which could have potential applications for 3D physical understanding.
strength_weakness: strength_weakness: 1. It seems unclear to me whether the proposed probes can really reflect the 3D understanding of large vision models, due to properties to be probed and the way to probe them. For example, the material, support relation and shadow properties, from my point of view, can be well identified with 2D clues (e.g. appearance and 2D spatial location). For occlusion and depth, using linear SVM to answer binary questions may not be enough to assess these properties.

2. There is a lack of baselines for the linear probing. It is unclear how a 3D-aware model trained explicitly with 3D data will respond to probe questions (upper bound). Also, it is better to show how the models with little 3D awareness will react to the probes (lower bound). Otherwise, it is less informative just comparing between large vision models, especially when their scores are close and pretty high as shown in table 4.

3. The paper “Probing the 3d awareness of visual foundation models” [1] shares a very similar goal with this paper. It’s suggested to add comparison and discussion with that paper. 

4. It would be more comprehensive to evaluate more large vision models such as SAM [2] and MAE [3].

[1] Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. CVPR, 2024.

[2] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. ICCV, 2023.

[3] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022
questions: questions: 1. Have the large-scale models been trained on the dataset chosen in this paper? Does it have influence on the probe results?

2. In Table 4, instead of using a random classifier as the baseline, it is fairer to train a feature extractor from scratch with your probing training set.

3. Following the second point in Weakness, could you provide the upper bound and lower bound of the probe? That’s essential to justify that the probing tasks can actually reflect 3D awareness.

4. Currently, the input to the linear SVM is designed as the difference between the averaged pooled features from two patches. Will pooling and subtraction operations lead to information loss? Is there any better way to formulate the linear probing?
limitations: limitations: Yes.

Comments: comment: Thanks for the rebuttal and I would maintain my positive rating.
Thanks for the rebuttal and thanks for addressing my questions. I am happy to keep my rating.