The 3D-PC: a benchmark for visual perspective taking in humans and machines

## Abstract

Abstract Visual perspective taking (VPT) is the ability to perceive and reason about the perspectives of others. It is an essential feature of human intelligence, which develops over the first decade of life and requires an ability to process the 3D structure of visual scenes. A growing number of reports have indicated that deep neural networks (DNNs) become capable of analyzing 3D scenes after training on large image datasets. We investigated if this emergent ability for 3D analysis in DNNs is sufficient for VPT with the 3D perception challenge ( 3D-PC ): a novel benchmark for 3D perception in humans and DNNs. The 3D-PC is comprised of three 3D-analysis tasks posed within natural scene images: 1. a simple test of object depth order , 2. a basic VPT task ( VPT-basic ), and 3. another version of VPT ( VPT-Strategy ) designed to limit the effectiveness of “shortcut” visual strategies. We tested human participants (N=33) and linearly probed or text-prompted over 300 DNNs on the challenge and found that nearly all of the DNNs approached or exceeded human accuracy in analyzing object depth order . Surprisingly, DNN accuracy on this task correlated with their object recognition performance. In contrast, there was an extraordinary gap between DNNs and humans on VPT-basic . Humans were nearly perfect, whereas most DNNs were near chance. Fine-tuning DNNs on VPT-basic brought them close to human performance, but they, unlike humans, dropped back to chance when tested on VPT-Strategy . Our challenge demonstrates that the training routines and architectures of today’s DNNs are well-suited for learning basic 3D properties of scenes and objects but are ill-suited for reasoning about these properties as humans do. We release our 3D-PC datasets and code to help bridge this gap in 3D perception between humans and machines.

## 1 Introduction

In his theory of cognitive development, Piaget posited that human children gain the ability to predict which objects are visible from another viewpoint before the age of 10 [, ]. This “Visual Perspective Taking” (VPT) ability is a foundational feature of human intelligence and a behavioral marker for the theory of mind []. VPT is also critical for safely navigating through the world and socializing with others (Fig. 1A). While VPT has been a focus of developmental psychology research since its initial description [, , ] (Fig. 1B), it has not yet been studied in machines.

Figure: Figure 1: Visual Perspective Taking (VPT) is the ability to analyze scenes from different viewpoints. (A) Humans rely on VPT to anticipate the behavior of others. We expect that this ability will be essential for creating the next generation of AI assistants that can accurately anticipate human behavior (images are CC BY-NC). (B) VPT has been studied in developmental psychology since the mid-20^th century using cartoon or highly synthetic stimuli. For example, Piaget’s “Three Mountains Task” asks observers to describe the scene from the perspective of a bear (image from []). (C) Here, we use Gaussian Splatting [] to develop a 3D scene generation pipeline for the *3D perception challenge* (3D-PC), to systematically compare 3D perception capabilities of human and machine vision systems. (D) The 3D-PC tests 1. Object depth perception, and 2. VPT.
Refer to caption: extracted/5646958/figures/teaser.png

One of the more surprising results in deep learning has been the number of concomitant similarities to human perception exhibited by deep neural networks (DNNs), trained on large-scale static image datasets [, ]. For example, DNNs now rival or surpass human recognition performance on object recognition and segmentation tasks [, , ], and are the state-of-the-art approach for predicting human neural and behavioral responses to images []. There is also a growing number of reports indicating that DNNs trained with self-supervision or for object classification learn to encode 3D properties of objects and scenes that humans are also sensitive to, such as the depth and structure of surfaces [, , , , , , , , , ]. Are the emergent capabilities of DNNs for 3D vision sufficient for solving VPT tasks?

Here, we introduce the *3D perception challenge* (3D-PC) to address this question and systematically compare 3D perceptual capabilities of humans and DNNs. The 3D-PC evaluates observers on (Fig 1): 1. identifying the order of two objects in depth (depth order), 2. predicting if one of two objects can “see” the other (VPT-basic), and 3. another version of VPT that limits the effectiveness of “shortcut” solutions [] (VPT-Strategy). The 3D-PC is distinct from existing psychological paradigms for evaluating VPT [, , ] and computer vision challenges for 3D perception [, ] in two ways. First, unlike small-scale psychology studies of VPT, the 3D-PC uses a novel “3D Gaussian Splatting” [] approach which permits the generation of endless real-world stimuli. Second, unlike existing computer vision challenges, our approach for data generation means that the 3D-PC tests and counterbalances labels for multiple 3D tasks on the exact same images, which controls for potential confounds in analysis and interpretation. We expect that DNNs which rival humans on the 3D-PC will become ideal models for a variety of real-world applications where machines must anticipate human behavior in real-time, as well as for enriching our understanding of how brains work (Fig. 1A).

##### Contributions.

We built the 3D-PC and used it to evaluate 3D perception for human participants and 327 different DNNs. The DNNs we tested represented each of today’s leading approaches, from Visual Transformers (ViT) [] trained on ImageNet-21k [] to ChatGPT4 [] and Stable Diffusion 2.0 [].

- •
We found that DNNs were very accurate at determining the depth order of objects after linear probing or text-prompting. DNNs that are state-of-the-art on object classification matched or exceeded human accuracy on this task.
- •
However, DNNs dropped close to chance accuracy on VPT-basic, whereas humans were nearly flawless at this task.
- •
Fine-tuning the zoo of DNNs on VPT-basic boosted their performance to near human level. However, the performance of the DNNs — but not humans — dropped back to chance on VPT-Strategy.
- •
Our findings demonstrate that the visual strategies necessary for solving VPT do not emerge in DNNs from large-scale static image training or after directly fine-tuning on the task. We release the 3D-PC data, code, and human psychophysics at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT) to support the development of models that can perceive and reason about the 3D world like humans.

## 2 Related work

##### 3D perception in humans.

The visual perception of 3D properties is a fundamentally ill-posed problem [, ], which forces biological visual systems to rely on a variety of assumptions to decode the structure of objects and scenes. For example, variations in the lighting, texture gradients, retinal image disparity, and motion of an object all contribute to the perception of its 3D shape. 3D perception is further modulated by top-down beliefs about the structure of the world, which are either innate or shaped by prior sensory experiences, especially visual and haptic ones. In other words, humans learn about the 3D structure of the world in an embodied manner that is fundamentally different than how DNNs learn. In light of this difference, it would be remarkable if DNNs could accurately model how humans perceive their 3D world.

##### Visual perspective taking in humans.

VPT was devised to understand how capabilities for reasoning about objects in the world develop throughout the course of one’s life. At least two versions of VPT have been introduced over the years [, ]. The version of VPT that we study here — known in the developmental literature as “VPT-1” — is the more basic form, which is thought to rely on automatic feedforward processing in the visual system []. In light of the well-documented similarities between feedforward processing in humans and DNNs [, ], we reasoned that this version of VPT would maximize the chances of success for today’s DNNs.

##### 3D perception in DNNs trained on static images.

As deep neural networks (DNNs) have increased in scale and training dataset size over the past decade, their performance on essentially all visual challenges has improved. Surprisingly, this “scale-up” has also led to the emergence of 3D perceptual capabilities. For example, DNNs trained with a variety of self-supervised learning techniques on static image datasets learn to represent the depth, surface normals, and 3D correspondence of features in scenes [, , , , , , , , ]. While similarities between DNNs and human 3D perception have yet to be evaluated systematically, it has been shown that there are differences in how the two reason about the 3D shape of objects []. The 3D-PC complements prior work by systematically evaluating which aspects of human 3D perception today’s DNNs can and cannot accurately represent.

##### Limitations of DNNs as models of human visual perception.

Over recent years, DNNs have grown progressively more accurate as models of human vision for object recognition tasks [, ]. At the same time, these models which succeed as models of human object recognition struggle to capture other aspects of visual perception [] including contextual illusions [], perceptual grouping [, ], and categorical prototypes []. There are also multiple reports showing that DNNs are growing less aligned with the visual strategies of humans and non-human primates as they improve on computer vision benchmarks [, , ]. The 3D-PC provides another axis upon which the field can evaluate DNNs as models of human vision.

## 3 Methods

##### The 3D-PC .

To enable a fair comparison between human observers’ and DNNs’ 3D perceptual capabilities, we designed the 3D-PC framework with two goals: 1. posing different 3D tasks on the same set of stimuli, and 2. generating a large number of stimuli to properly train DNNs on these tasks. We achieved these goals by combining 3D Gaussian Splatting [], videos from the Common Objects in 3D (Co3D) [] dataset, and Unity [, ] into a flexible data-generating framework.

Figure: Figure 2: 3D-PC examples. We tested 3D perception in images generated by Gaussian Splatting. Each image depicts a green camera and a red ball. These objects are placed in the scene in a way that counterbalances labels for depth order task and VPT-basic tasks.
Refer to caption: extracted/5646958/figures/examples.png

Our procedure for building the 3D-PC involved the following three steps. 1. We trained Gaussian Splatting models on videos in Co3D (Fig. 1C). 2. We imported these trained models into Unity, where we added green camera and red ball objects into each 3D scene, which were used to pose visual tasks (Fig. 1D). 3. We then generated random viewpoint trajectories within each 3D scene, rendered images at each position along the trajectory, and derived ground-truth answers for depth order and VPT tasks for the green camera at every position from Unity.

Our approach makes it possible to generate an unlimited number of visual stimuli that test an observer’s ability to solve complementary 3D perception tasks (depth order and VPT) while keeping visual statistics constant and ground truth labels counterbalanced across tasks. For the version of 3D-PC used in our evaluation and released publicly at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT), the depth order and VPT-basic tasks are posed on the same set of 7,480 training images of 20 objects and scenes, and a set of 94 test images of 10 separate objects and scenes (Fig. 2). We held out a randomly selected 10% of the training images for validation and model checkpoint selection.

To build the VPT-Strategy task, we rendered images where we fixed the scene camera while we moved the green camera and red ball objects to precisely change the line-of-sight between them from unobstructed to obstructed and back. We reasoned that this experiment would reveal if an observer adopts the visual strategy of taking the perspective of the green camera, which is thought to be used by humans [], from other strategies that relied on less robust feature-based shortcuts. This dataset consisted of a test set of 100 images for 10 objects and scenes that were not included in depth order or VPT-basic.

Figure: Figure 3: Human accuracy for object depth order and VPT-basic tasks. Bars near 50% are label-permuted noise floors; lines are group means. The difference is significant, *** $=p<0.001.$
Refer to caption: extracted/5646958/figures/humans.png

##### Psychophysics experiment.

We tested 10 participants on depth order, 20 on VPT-basic, and 3 on VPT-Strategy. 33 participants were recruited online from Prolific. All provided informed consent before completing the experiment and received $15.00/hr compensation for their time (this amounted to $5.00 for the 15–20 minutes the experiment lasted). These data were de-identified.

Participants were shown instructions for one of the 3D-PC tasks, then provided 20 examples to ensure that they properly understood it (Appendix Fig A.1). These examples were drawn from the DNN training set. Each experimental trial consisted of the following sequence of events overlaid onto a white background: 1. a fixation cross displayed for $1000ms$; 2. an image displayed for $3000ms$, during which time the participants were asked to render a decision. Participants pressed one of the left or right arrow keys on their keyboards to provide decisions.

Images were displayed at 256$\times$256 pixel resolution, which is equivalent to a stimulus between $5^{\mathrm{o}}-11^{\mathrm{o}}$ of visual angle across the range of display and seating setups we expected our online participants used for the experiment.

##### Model zoo.

We evaluated a wide range of DNNs on the 3D-PC, which represented the leading approaches for object classification, self-supervised pretraining, image generation, depth prediction, and vision language modeling (VLM). Our zoo includes 317 DNNs from PyTorch Image Models (TIMM) [], ranging from classic models like AlexNet [] to state-of-the-art models like EVA-02 [] (see Appendix 1 for the complete list). We added foundational vision models like MAE [], DINO v2 [], iBOT [], SAM [], and Midas [] (obtained from the GitHub repo of []). We also included Depth Anything [], a foundational model 3D scene analysis and depth prediction [], as well as the Stable Diffusion 2.0 [] image generation model. Finally, we added state-of-the-art large vision language models (VLMs) ChatGPT4 [], Gemini [], and Claude 3 []. We evaluated a total of 327 models on the 3D-PC.

Figure: Figure 4: DNN performance on the depth order and VPT-basic tasks in the 3D-PC after linear probing or prompting. (A, B) DNNs are significantly more accurate at depth order than VPT-basic. Human confidence intervals are S.E.M. and ***: $p<0.001$. (C, D) DNN accuracy for depth order and VPT-basic strongly correlates with object classification accuracy on ImageNet. Dashed lines are the mean of label-permuted human noise floors.
Refer to caption: extracted/5646958/figures/results.png

##### Model evaluation.

We evaluated all models except for the VLMs on the depth order and VPT-basic tasks in this challenge by training linear probes on image embeddings from their penultimate layers. Linear probes were trained using PyTorch [] for 50 epochs, a $5e$-4 learning rate, and early stopping (see Appendix A.5 for details). Training took approximately 20 minutes per model using NVIDIA-RTX 3090s. We tested the Stable Diffusion 2.0 model by adopting the evaluation method used in [] (see Appendix A.7 for details). We evaluated the VLMs by providing them the same instructions and training images (along with ground truth labels) given to humans, then recording their responses to images from each task via model APIs.

To test the learnability of the 3D-PC, we also fine-tuned each of the TIMM models in our zoo to solve the tasks. To do this, we trained each of these models for 30 epochs, a $5e$-5 learning rate, and early stopping (see Appendix A.5 for details). Fine-tuning took between 3 hours and 24 hours per model using NVIDIA-RTX 3090s.

## 4 Results

Figure: Figure 5: DNN performance on the depth order and VPT-basic tasks in the 3D-PC after fine-tuning. (A) Fine-tuning makes DNNs far better than humans at the depth order task and improves the performance of several DNNs to be at or beyond human accuracy on VPT-basic. (B) Even after fine-tuning, there is still a significant difference in model performance on depth order and VPT-basic tasks, $p<0.001$. (C, D) DNN accuracy on both tasks after fine-tuning correlates with ImageNet object classification accuracy. Human confidence intervals are S.E.M. and ***: p < 0.001. Dashed lines are the mean of label-permuted human noise floors.
Refer to caption: extracted/5646958/figures/ft_results.png

##### Humans find VPT easier than determining the depth ordering of objects.

Human participants were on average 74.73% accurate at determining the depth order of objects, and 86.82% accurate at solving the VPT-basic task (Fig. 3; $p<0.001$ for both; statistical testing done through randomization tests []). Humans were also significantly more accurate at solving VPT-basic than they were at the depth order task.

##### DNNs learn depth but not VPT from static image training.

DNNs showed the opposite pattern of results on depth order and VPT-basic tasks as humans after linear probing or prompting (Fig. 4): 15 of the DNNs we tested fell within the human accuracy confidence interval on the depth order task, and three even outperformed humans (Fig. 4A). In contrast, while humans were on average 86.82% accurate at VPT-basic, the DNN which performed the best on this task, the ImageNet 21K-trained beit [], was 53.82% accurate. Even commercial VLMs struggled on VPT-basic and were around chance accuracy (ChatGPT4: 52%, Gemini: 52%, and Claude 3: 50%). The depth order task was significantly easier for DNNs than VPT-basic ($p<0.001$), which is the opposite of humans (Fig. 4B).

##### ImageNet accuracy correlates with the 3D capabilities of DNNs.

What drives the development of 3D perception in DNNs trained on static images? We hypothesized that as DNNs scale up, they learn ancillary strategies for processing natural images, including the ability to analyze the 3D structure of scenes. To investigate this possibility, we focused on the TIMM models in our DNN zoo. These models have previously been evaluated for object classification accuracy on ImageNet, which we used as a stand-in for DNN scale [, , ]. Consistent with our hypothesis, we found a strong and significant correlation between DNN performance on ImageNet and depth order task accuracy ($\rho=0.66,p<0.001$, Fig. 4C). Despite the very low accuracy of DNNs on VPT-basic, there was also a weaker but still significant correlation between performance on this task and ImageNet ($\rho=0.34,p<0.001$, the difference in correlations between the tasks is $\rho=0.32,p<0.001$; Fig. 4D). These results suggest that monocular depth cues develop in DNNs alongside their capabilities for object classification (^1^11More work is needed to identify a causal relationship between the development of monocular depth cues and object recognition accuracy.). However, the depth cues that DNNs learn are poorly suited for VPT.

Figure: Figure 6: Even DNNs fine-tuned on VPT-basic fail on VPT-Strategy. (A) To better characterize the strategy used by humans and DNNs to solve VPT, we devised a new test, VPT-Strategy, in which the green camera and red ball are moved through a scene while holding the scene camera and a centrally-positioned object still. This task is easily solvable if an observer estimates the line-of-sight of the green camera; other strategies, such as those that rely on specific image features (feature based), may be less effective. (B) Examples of VPT-Strategy stimuli along with the ground-truth label (top-row) and predictions by a ViT large after linearly probing or fine-tuning for VPT-basic (bottom-row). Decision attribution maps from each version of the ViT large, derived from “smooth gradients” [], are overlaid onto bottom-row images (purple/blue=linearly probed, yellow/green=fine-tuned). The fine-tuned ViT locates the green camera and red ball but renders incorrect decisions. (C) DNNs fine-tuned on VPT-basic fail to solve VPT-Strategy; they rely on a brittle feature-based strategy. Humans, on the other hand, are 87% accurate; they likely estimate line-of-sight.
Refer to caption: extracted/5646958/figures/raytracing_recovered.png

##### DNNs can solve VPT-basic after fine-tuning.

One possible explanation for the failure of today’s DNNs on VPT-basic is that the task requires additional cues for 3D vision that cannot be easily learned from static images. To explore this possibility, we fine-tuned each of the TIMM models in our DNN zoo to solve depth order and VPT-basic (Fig. 5A). There was still a significant difference between DNN performance on the two tasks (Fig. 5B, $p<0.001$), but fine-tuning caused 97% of the DNNs to exceed human accuracy on depth order, and four of the DNNs to reach human accuracy on VPT-basic. DNN performance on the tasks more strongly correlated with ImageNet accuracy after fine-tuning than linear probing (compare Fig. 5C/D and Fig. 4C/D). We also compared the errors these DNNs made on both tasks to humans. We found nearly all of the fine-tuned DNNs were aligned with humans on depth order, and a handful were aligned with humans on VPT-basic (Fig. A.3).

##### DNNs learn different strategies than humans to solve VPT.

The ability of DNNs to reach human-level performance on visual tasks by adopting strategies that are different from humans has been well-documented [, , ]. Thus, we devised a new experiment to understand if DNNs learn to solve VPT in the same way as humans do after fine-tuning. In developmental psychology, it has been proposed that humans estimate the line-of-sight of objects for VPT because they respond in predictable ways after the positions of objects in a scene are slightly adjusted [, ]. Inspired by this psychological work, we created the VPT-Strategy task to evaluate the types of visual strategies used by DNNs and humans to solve VPT (Fig. 6A).

VPT-Strategy has observers solve the VPT task on a series of images rendered from a fixed camera viewpoint as the green camera and red ball are moved incrementally from one side of the screen to the other, passing by an occluding object in the process. This means that we can precisely map out the moments at which the green camera has a clear view of the red ball, when that view is occluded, and when the view becomes unoccluded once more. DNNs behave differently than humans on this task: humans were 87% accurate, but the highest performing DNN, the Swin Transformer [] trained on ImageNet-21k, was only 66% accurate (Fig. 6B, C). In other words, while DNNs can be fine-tuned to approach human accuracy on VPT-basic, the strategy they learn is brittle, generalizes poorly, and is likely ill-suited for reasoning about the 3D world.

## 5 Discussion

Deep neural networks (DNNs) have rapidly advanced over recent years to the point where they match or surpass human-level performance on numerous visual tasks. However, our 3D-PC reveals there is still a significant gap between the abilities of humans and DNNs to reason about 3D scenes. While DNNs match or exceed human accuracy on the basic object depth order task after linear probing or prompting, they struggle remarkably on even the basic form of VPT that we test in the 3D-PC. Fine-tuning DNNs on VPT-basic allows them to approach human-level performance, but unlike humans, their strategies do not generalize to the VPT-Strategy task.

A striking finding from our study is the strong correlation between DNNs’ object classification accuracy on ImageNet and their performance on depth order and VPT-basic. This correlation suggests that monocular depth cues emerge in DNNs as a byproduct of learning to recognize objects, potentially because these cues are useful for segmenting objects from their backgrounds. The difference in DNN effectiveness for depth order versus VPT-basic, however, indicates that these cues are not sufficient for reasoning about the 3D structure of scenes in the way that VPT demands.

Thus, today’s approaches for developing DNNs, which primarily focus on static image datasets, may be poorly suited for enabling robust 3D perception and reasoning abilities akin to those of humans. Incorporating insights from human cognition and neuroscience into DNNs, particularly in ways biological visual systems develop 3D perception, could help evolve more faithful models of human intelligence.

A key limitation of our study is that our version of VPT represents the most basic form studied in the developmental psychology literature. While solving this task is evidently an extraordinary challenge for DNNs, it is only one small step towards human-level capabilities for reasoning about 3D worlds in general. Far more research is needed to identify additional challenges, architectures, and training routines that can help DNNs perceive and reason about the world like humans do. We release our 3D-PC data and code at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT) to support this goal.

## 6 Acknowledgements

Funding for this project was provided by the Office of Naval Research (N00014-19- 1-2029) and ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A0004). Additional support was provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant S10OD025181.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes] In the discussion.
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] Appendix section A.3.
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[N/A]
(b)
Did you include complete proofs of all theoretical results?
[N/A]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See methods.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Appendix section A.5
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report error bars over human performance in all figures. We also report model/error bars in performance and correlation with humans (Appendix fig. A.3).
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix section A.5.
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes] We used existing models and libraries and cited each.
(b)
Did you mention the license of the assets?
[Yes] See Appendix section A.1.
(c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See methods.
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] See methods.
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes] See appendix section A.8
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A]
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] See methods.

## Appendix A Appendix

### A.1 Author Statement

As authors of this dataset, we bear all responsibility for the information collected and in case of violation of rights and other ethical standards. We affirm that our dataset is shared under a Creative Commons CC-BY license.

### A.2 Data Access

We release benchmarking code and data download instructions at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT).

### A.3 Potential negative societal impacts of this work

The most obvious potential negative impact of our work is that advancing visual perspective taking (VPT) capabilities in artificial agents could potentially enable militaristic applications or surveillance overreach. However, we hope that our benchmark will aid in the development of AI-based assistants that can better anticipate and react to human needs and social cues for safer navigation and interaction. We also believe that our benchmark will guide the development of better computational models of human 3D perception as well as the neural underpinnings of these abilities.

### A.4 Data Generation

To generate data for the 3D-PC, we first trained 3D Gaussian Splatting [] models on videos from the Common Objects in 3D (Co3D) [], which yielded 3D representations of each scene. We then imported trained models into Unity [] using Unity Gaussian Splatting [] and added 3D models of the green camera and red ball to each. Finally, we rendered 50 images along a smooth viewpoint camera trajectory sampled near the original trajectory used for training the Gaussian Splatting model. For each 3D scene, we created 5 positive and 5 negative settings for VPT.

To generate VPT-basic, the generation process was repeated for 30 Co3D videos from 10 different categories. We removed any images where the green camera and red ball were not visible. We then split the images into a training set of 7480 images from 20 scenes and a testing set of 94 images from 10 other scenes. For the depth order task, we used the same data splits but removed any ambiguous samples where the objects were similarly close to the camera. The resulting dataset for the depth order task contains 4787 training images and 94 testing images. The same set of testing images is used for both model and human benchmarks.

For VPT-Strategy, we used the same process to generate data from 10 additional Co3D scenes not included in VPT-basic and additionally controlled the positions of the green camera and the red ball. The angle between these two objects was held constant while we moved them so that their line of sight was unobstructed, obstructed, and then unobstructed once again. For each Co3D scene, we rendered 10 settings from a fixed viewpoint camera position, resulting in 100 images in total for VPT-Strategy.

### A.5 Model Zoo

We linearly probed 317 DNNs from Pytorch Image Models (TIMM) [] (Table 1) along with foundational vision models following the procedures in  []. All DNNs were trained and evaluated with NVIDIA-RTX 3090 GPUs from the Brown University Center for Computation & Visualization. All linear probes were trained for 50 epochs, with a $5e-4$ learning rate, a $1e-4$ weight decay, a $0.3$ dropout rate, and a batch size of 128. We fine-tuned each of the TIMM models for 30 epochs, a $5e-5$ learning rate, $1e-4$ weight decay, $0.7$ dropout rate, and a batch size of 16. Linear probing took approximately 20 minutes per model, and fine-tuning varied from 3 to 24 hours on a NVIDIA-RTX 3090 GPU.

**Table 1: The 327 DNN models used in our study.**
| Architecture | Model | Versions |
| --- | --- | --- |
| CNN | ConvMixer [] | 3 |
| ConvNeXT [] | 10 |  |
| DenseNet [] | 4 |  |
| DLA [] | 5 |  |
| DPN [] | 6 |  |
| EfficientNet [] | 4 |  |
| GhostNet [] | 1 |  |
| HRNet [] | 8 |  |
| LCNet [] | 3 |  |
| MixNet [] | 4 |  |
| MnasNet [] | 3 |  |
| MobileNet [] | 14 |  |
| RegNet [] | 6 |  |
| Res2Net [] | 5 |  |
| ResNet [] | 26 |  |
| ResNeSt [] | 3 |  |
| RexNet [] | 5 |  |
| ResNext [] | 2 |  |
| SPNASNet [] | 1 |  |
| TinyNet [] | 2 |  |
| VGG [] | 14 |  |
| Transformer | BEiT [] | 9 |
| CAFormer [] | 6 |  |
| CaiT [] | 3 |  |
| ConViT [] | 3 |  |
| CrossViT [] | 2 |  |
| DaViT [] | 3 |  |
| DeiT [] | 12 |  |
| EfficientFormer [] | 7 |  |
| EVA [] | 9 |  |
| FocalNet [] | 6 |  |
| LeViT [] | 5 |  |
| MaxViT [] | 6 |  |
| MobileViT [] | 3 |  |
| MViT [] | 3 |  |
| PiT [] | 8 |  |
| PVT [] | 7 |  |
| Swin [] | 16 |  |
| Twins-SVT [] | 5 |  |
| ViT [] | 36 |  |
| Volo [] | 7 |  |
| XCiT [] | 6 |  |
| PoolFormer [] | 8 |  |
| Hybrid | CoaT [] | 7 |
| CoAtNet [] | 8 |  |
| EdgeNeXt [] | 1 |  |
| Visformer [] | 2 |  |
| Foundation | Depth Anything [] | 1 |
| DINOv2 [] | 1 |  |
| iBoT [] | 1 |  |
| MAE [] | 1 |  |
| MiDas [] | 1 |  |
| SAM [] | 1 |  |
| VLM | ChatGPT4 [] | 1 |
| Gemini [] | 1 |  |
| Claude 3 [] | 1 |  |
| Diffusion | Stable Diffusion 2.0 [] | 1 |

### A.6 VLM Evaluation

We evaluated the following proprietary VLMs on the VPT-basic and depth order tasks: GPT-4 (gpt-4-turbo), Claude (claude-3-opus-20240229), and Gemini (gemini-pro-vision). To evaluate these VLMs, we used their APIs to send queries containing 20 training images, with ground truth answers as context, as well as a test image. The prepended 20 training images meant that for every example in the challenge, VLMs were given the opportunity to learn, “in-context”, how to solve the given task.

The prompt we used for the depth task was “In this image, is the red ball closer to the observer or is the green arrow closer to the observer? Answer only BALL if the red ball is closer, or ARROW if the green arrow is closer, nothing else.” and the prompt for the VPT-basic task was “In this image, if viewed from the perspective of the green 3D arrow in the direction the arrow is pointing, can a human see the red ball? Answer only YES or NO, nothing else”. We evaluated each model’s generated responses across multiple temperatures, ranging from $0.0$ to $0.7$ in increments of $0.1$, and we report the average of the best 3 runs. Note that while this evaluation approach gives the VLMs more opportunities to perform well on our benchmark than other models, they still struggled immensely (see main text).

### A.7 Stable Diffusion Evaluation

We followed the method of Li et al. [] to evaluate Stable Diffusion 2.0 on the 3D-PC. This involved trying multiple prompts to optimize the zero-shot classification performance of the Stable Diffusion 2.0 model, on VPT-basic and depth order tasks. For VPT-basic we found that the prompt "A photo with red ball is visible from the green arrow’s perspective" for positive class and "A photo with red ball not visible from the green arrow’s perspective" for the negative class led to the best performance. For the depth order task, the prompt with the highest performance was "A photo with green arrow closer to the camera as compared to red ball" and "A photo with red ball closer to the camera as compared to green arrow" for positive and negative classes respectively.

### A.8 Human Benchmark

We recruited 30 participants through Prolific, compensating each with $5 upon successful completion of all test trials. Participants confirmed their completion by pasting a unique system-generated code into their Prolific accounts. The compensation was prorated based on the minimum wage. We also incurred a 30% overhead fee per participant paid to Prolific. In total, we spent $195 on these benchmark experiments.

#### A.8.1 Experiment design

At the outset of the experiment, we acquired participant consent through a form approved by the Brown University’s Institutional Review Board (IRB). The experiment was performed on a computer using the Chrome browser. Following consent, we presented a demonstration with instructions and an example video. Participants had the option to revisit the instructions at any time during the experiment by clicking a link in the top right corner of the navigation bar.

Figure: Figure A.1: An experiment trial.
Refer to caption: extracted/5646958/figures/persp_expwindow.png

Figure: Figure A.2: The consent screen.
Refer to caption: extracted/5646958/figures/consent_persp.png

In the depth order task, the participants were asked to classify the image as “positive” (the green arrow in closer to the viewer) or “negative” (the red ball is closer) using the right and left arrow keys respectively. The choice for keys and their corresponding instances were mentioned below the image on every screen (See Appendix Fig. A1. Participants were given feedback on their response (correct/incorrect) during every practice trial, but not during the test trials. In the VPT tasks, the choices were “the green arrow/camera see the red ball” or “the green arrow/camera can not see the red ball”.

The experiment was not time-bound, allowing participants to complete it at their own pace. Participants typically took around 20 minutes. After each trial, participants were redirected to a screen confirming the successful submission of their responses. They could start the next trial by clicking the “Continue” button or pressing the spacebar. If they did not take any action, they were automatically redirected to the next trial after 1000 milliseconds. Additionally, participants were shown a “rest screen” with a progress bar after every 40 trials, where they could take additional and longer breaks if needed. The timer was turned off during the rest screen.

### A.9 Human vs. DNN decision making on VPT-basic

We compared the decision strategies of humans and DNNs on VPT-basic by measuring the correlations between their error patterns with Cohen’s $\kappa$ []. Model $\kappa$ scores were mostly correlated with accuracy on VPT-basic after linear probes and fine-tuning (Fig. A.3). However, while nearly all DNNs were highly correlated with human error patterns after fine-tuning, the correlation between $\kappa$ scores and task accuracy disappeared (Fig. A.3B, purple dots).

Figure: Figure A.3: Error pattern correlations (Cohen’s $\kappa$) between humans and DNNs on VPT-basic
Refer to caption: extracted/5646958/figures/ft_kappa.png

### A.10 Datasheet for datasets

Motivation

For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description.

The dataset was designed to test 3D perception in humans and DNNs, with an emphasis on the capabilities of each for visual perspective taking (VPT). Humans rely on VPT everyday for navigating and socializing, but despite its importance, there has yet to be a systematic evaluation of this ability in DNNs.

Who created this dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?

This dataset was created by this paper authors, who are affiliated with the Carney Institute for Brain Science at Brown University and the Cognitive Sciences Department at UC Irvine.

Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.

Funding for this project was provided by the Office of Naval Research (N00014-19- 1-2029) and ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A0004). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV). We acknowledge the Cloud TPU hardware resources that Google made available via the TensorFlow Research Cloud (TFRC) program as well as computing hardware supported by NIH Office of the Director grant S10OD025181.

Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

The instances contain images of real-world objects and scenes along with shapes generated with computer graphics.

How many instances are there in total (of each type, if appropriate)?

There are 7574 images in the training and testing sets.

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).

We release all data.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description.

Each instance consists of an image rendered from 3d Gaussian Splatting [] models trained on Co3D [] scenes.

Is there a label or target associated with each instance? If so, please provide a description.

The images are labeled for VPT and depth order tasks. In the VPT task, an image is labeled as positive when the red ball is visible from the green camera’s perspective. In the depth task, an image is labeled as positive when the red ball is further away than the green arrow from the viewer. For both tasks, we label positives as 1 and negatives as 0.

Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.

N/A

Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.

N/A

Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.

We provide training, validation and testing splits in the released dataset. The training set contains images rendered from 20 unique scenes from 10 categories. The testing set images are rendered from 10 additional scenes from the same categories. We randomly selected 10% of the training set as the validation set.

Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.

N/A

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.

The dataset uses videos from the Co3D dataset [], which is publicly available under CC BY-NC 4.0 license.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description.

N/A

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.

N/A

Does the dataset relate to people? If not, you may skip the remaining questions in this section.

Yes

Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.

No

Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.

No, all results are anonymous.

Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.

N/A

Any other comments?

Collection Process

How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.

All images were rendered from 3D gaussian splatting [] models trained on videos from Co3D []. We imported the model into Unity [, ] to render images.

What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?

We used Unity [] and Unity Gaussian Splatting [] to edit the scenes and label them in 3D view.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

N/A

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

The paper’s authors were involved in the data collection process.

Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.

N/A

Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.

Does the dataset relate to people? If not, you may skip the remaining questions in this section.

Yes

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

As described in the Methods, we collected data from online participants through Prolific, and we also collected data in-person for several subjects.

Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.

Yes. See Section A.8 for details.

Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.

Yes. See Fig A.2 for the consent screen with the exact language used.

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).

Yes. The participants were provided with our contact information and were encouraged to reach out in such cases.

Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.

Our experiment was approved by the IRB board at Brown University.

Any other comments?

Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.

We used Unity to label images for VPT and depth tasks. We removed images where the objects of interest (red ball and green camera) were not visible.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.

N/A

Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.

N/A

Any other comments?

Uses

Has the dataset been used for any tasks already? If so, please provide a description.

We evaluated vision DNNs on the dataset. Please refer to the main paper for details.

Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.

The code and data are publicly available at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT)

What (other) tasks could the dataset be used for?

We mainly expect the dataset to be used for evaluating 3D perception capabilities of new vision or vision-language DNNs.

Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms?

N/A

Are there tasks for which the dataset should not be used? If so, please provide a description.

N/A

Any other comments?

Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.

Yes, we will release the dataset to the public at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT)

How will the dataset be distributed (e.g., tarball on website, API, GitHub) Does the dataset have a digital object identifier (DOI)?

We provide download instructions at [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT)

When will the dataset be distributed?

The dataset is available from June 5th, 2024.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.

We release our data under a Creative Commons CC-BY license.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.

N/A

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.

N/A

Any other comments?

Maintenance

Who will be supporting/hosting/maintaining the dataset?

The authors will be hosting and maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Contact the corresponding author through email.

Is there an erratum? If so, please provide a link or other access point.

N/A

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)?

We are actively working on expanding the dataset with new instances and tasks. We will update our GitHub repository accordingly for any dataset update.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.

Human participant data was de-identified, and there are no time limits on its retention.

Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users.

Yes, we will maintain old versions of the dataset on our website.

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.

We are open to any suggestions and contributions through our GitHub repository. [https://github.com/serre-lab/VPT](https://github.com/serre-lab/VPT)

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: The paper mentions in its appendix that when testing Visual Language Models (VLMs), they provided in-context examples to help guide the models. However, it's unclear whether they also included step-by-step reasoning examples like chain-of-thought prompting, which can probably help the models better understand how to approach these perspective-taking tasks.
soundness: 3
strengths: - The paper used 3D Gaussian Splatting to create large sets of realistic images. 
- This method gave them precise control over the test conditions, helping them mitigate confoudning factors.
- Their analysis revealed interesting insights: while DNNs can learn to understand depth from static images, they don't develop the ability to take others' perspectives. They also found that a network's accuracy on ImageNet correlates with the 3D capabilities of DNNs they tested.
summary: Visual Perspective Taking (VPT) refers to our ability to understand how others see the world from their viewpoint. This paper has developed a new benchmark to test whether DNNs can perform this skill. Their test includes three components: determining the order of objects by depth, VPT-basic, and VPT-strategy that eliminates potential shortcuts solutions.

While DNNs matched or even exceeded human ability in determining object depth order, they struggled significantly with VPT-basic, performing at nearly random levels. In contrast, humans excelled at these same tasks with almost perfect accuracy. Even when DNNs are fine-tuned for VPT-basic, they still failed at VPT-Strategy.

These findings suggest that DNNs don't naturally develop the ability to take others' perspectives, even after extensive training on static images or fine-tuning on VPT-basic tasks.
weaknesses: Using only green cameras and red target points in their test images raises a few questions: Would the main results still hold if they used other colors or camera shapes? Is the paper assuming the DNNs can easily spot the camera and determine its viewing angle in the images? If so, how much do the camera's physical properties (like its size, line thickness, or color) affect the results?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the Weaknesses. Specifically, please answer:

- How do you evaluate the correctness of the ground truth labels generated by the proposed method?
- Can you do some experiments to show whether or not the DNNs learn to perform the depth order task based on relative sizes?
- How will the DNNs perform if they are finetuned on VPT-Strategy data? Do you think the results may change your perspective on the VPT-Strategy’s role?
soundness: 1
strengths: - The proposed benchmark is clearly motivated and well situated in the literature.
- The methodology is clearly explained with nice figures.
- Extensive experiments are conducted to show how the challenge can boost the 3D perception research of deep learning.
summary: This paper proposes 3D-PC, a new benchmark for visual perspective taking (VPT), an ability to predict what other perspectives see. The benchmark consists of three tasks: 1) telling which object is closer (depth order), 2) predicting if a camera can see a ball (VPT-basic), 3) another version of 2) where the occlusion is flipped. The authors then tested 33 human participants and over 300 DNNs on the benchmark, showing that despite a super-human performance on depth order, DNNs are near chance on VPT tasks.
weaknesses: - I think for a benchmark paper, one of the most important things is the correctness of the ”ground truth labels”. Although the method proposed may sound reasonable, it is hard to tell how unlikely the “GTs” are inaccurate. After all, the adopted 3D Gaussian Splatting method is not perfect. Therefore, I expect some experiments or at least some discussions on the correctness of GTs.
- I think the descriptions for VPT-Strategy are insufficient. I could only understand what this task was about when I read Figure 6.
- I don’t think we can say that DNNs learn depth from the depth order experiments. It seems to be very likely that the linear-probed DNNs only learned to solve the task based on the relative sizes of the red ball and green camera. Therefore, I don’t see the meaning of the depth order task.
- It seems that the role of the VPT-Strategy task is to test if the observer estimates the line-of-sight of the camera. However, I think it cannot take such a role. I guess if you finetune the DNNs on VPT-Strategy data, they will perform much better. If so, we can’t say they are estimating the line-of-sight because they are likely still using features to perform the task. Therefore, I don’t think the VPT-Strategy experiments are very meaningful.

Minor
- It is claimed that AlexNet is included in the model zoo (L250), but I didn’t find it in Appendix Table 1?
- Figure 2: VPT-easy → VPT-basic.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness for more information about potential questions and improvements.
soundness: 3
strengths: The paper has good insight about cognitive understanding of 3D perception, exploring the gap between human and DNN capabilities in perspective-taking. By demonstrating that DNNs trained on static images struggle with tasks requiring true 3D reasoning (notably in VPT-basic and VPT-perturb as shown in Figures 4 and 6), the authors highlight a significant limitation of current model approaches. The paper also nicely suggests incorporating cognitive science insights as a potential path forward.
summary: This paper introduces the 3D-PC benchmark, a new evaluation framework for visual perspective-taking (VPT) in humans and DNNs, focusing on spatial understanding and social interactions. The benchmark consists of three tasks: Object Depth Ordering, Basic VPT, and VPT-Perturb, where the latter is designed to prevent DNNs from using shortcut strategies. Through evaluation of 33 human participants and over 300 DNNs, the results show that while DNNs achieve human-comparable performance in depth ordering, they show clear limitations in VPT tasks, particularly VPT-Perturb. While fine-tuning helps DNNs match human performance on Basic VPT, they perform at chance level on VPT-Perturb, indicating a gap in spatial reasoning generalization. The findings demonstrate limitations in DNNs trained on static image datasets for 3D reasoning, suggesting current architectures lack the visual strategies needed for human-like perspective-taking.
weaknesses: 1. While the paper presents interesting tasks for studying 3D understanding, it could benefit from broader connections to the current 3D computer vision landscape.

2. Adding some discussion about how their findings relate to work in depth estimation and 3D segmentation would help put their cognitive insights into a wider context.

3. Figure 6 provides an overview of DNN performance but deeper exploration about why models fail on VPT tasks would be better. Including detailed failure cases or decision attribution maps could shed light on where models diverge from human performance, particularly in VPT-perturb where models rely on “shortcut” strategies.

Meta-review: This paper has developed a new benchmark to test whether DNNs can perform Visual Perspective Taking (VPT). Rigorous experiments reveal new insights that DNNs fail at VPT tasks. 

The paper is well-written. The methodology for studying the problem and the benchmark are novel and interesting. The insights obtained from this study are useful for the future development of AI models in 3d depth perception.

Main concerns on depth order and the explanations why models struggle at VPT are addressed during the rebuttal.

All the reviewers reached an agreement to accept the paper after the rebuttal.

Decision: Accept (Poster)