The 3D-PC: a benchmark for visual perspective taking in humans and machines

Authors: Drew Linsley, Peisen Zhou, Alekh Karkada Ashok, Akash Nagaraj, Gaurav Gaonkar, Francis E Lewis, Zygmunt Pizlo, Thomas Serre
Primary area: applications to neuroscience & cognitive science
Keywords: 3D vision, visual cognition, developmental psychology, visual reasonsing

Abstract:
Visual perspective taking (VPT) is the ability to perceive and reason about the perspectives of others. It is an essential feature of human intelligence, which develops over the first decade of life and requires an ability to process the 3D structure of visual scenes. A growing number of reports have indicated that deep neural networks (DNNs) become capable of analyzing 3D scenes after training on large image datasets. We investigated if this emergent ability for 3D analysis in DNNs is sufficient for VPT with the 3D perception challenge (3D-PC): a novel benchmark for 3D perception in humans and DNNs. The 3D-PC is comprised of three 3D-analysis tasks posed within natural scene images: (i.) a simple test of object depth order, (ii.) a basic VPT task (VPT-basic), and (iii.) a more challenging version of VPT (VPT-perturb) designed to limit the effectiveness of "shortcut" visual strategies. We tested human participants (N=33) and linearly probed or text-prompted over 300 DNNs on the challenge and found that nearly all of the DNNs approached or exceeded human accuracy in analyzing object depth order. Surprisingly, DNN accuracy on this task correlated with their object recognition performance. In contrast, there was an extraordinary gap between DNNs and humans on VPT-basic. Humans were nearly perfect, whereas most DNNs were near chance. Fine-tuning DNNs on VPT-basic brought them close to human performance, but they, unlike humans, dropped back to chance when tested on VPT-perturb. Our challenge demonstrates that the training routines and architectures of today's DNNs are well-suited for learning basic 3D properties of scenes and objects but are ill-suited for reasoning about these properties like humans do. We release our 3D-PC datasets and code to help bridge this gap in 3D perception between humans and machines.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730623361193

thread_type:
official_review

<thread_object>
creator: reviewer_gktt

<content>
summary:
This paper introduces the 3D-PC benchmark, a new evaluation framework for visual perspective-taking (VPT) in humans and DNNs, focusing on spatial understanding and social interactions. The benchmark consists of three tasks: Object Depth Ordering, Basic VPT, and VPT-Perturb, where the latter is designed to prevent DNNs from using shortcut strategies. Through evaluation of 33 human participants and over 300 DNNs, the results show that while DNNs achieve human-comparable performance in depth ordering, they show clear limitations in VPT tasks, particularly VPT-Perturb. While fine-tuning helps DNNs match human performance on Basic VPT, they perform at chance level on VPT-Perturb, indicating a gap in spatial reasoning generalization. The findings demonstrate limitations in DNNs trained on static image datasets for 3D reasoning, suggesting current architectures lack the visual strategies needed for human-like perspective-taking.

strengths:
The paper has good insight about cognitive understanding of 3D perception, exploring the gap between human and DNN capabilities in perspective-taking. By demonstrating that DNNs trained on static images struggle with tasks requiring true 3D reasoning (notably in VPT-basic and VPT-perturb as shown in Figures 4 and 6), the authors highlight a significant limitation of current model approaches. The paper also nicely suggests incorporating cognitive science insights as a potential path forward.

weaknesses:
1. While the paper presents interesting tasks for studying 3D understanding, it could benefit from broader connections to the current 3D computer vision landscape.

2. Adding some discussion about how their findings relate to work in depth estimation and 3D segmentation would help put their cognitive insights into a wider context.

3. Figure 6 provides an overview of DNN performance but deeper exploration about why models fail on VPT tasks would be better. Including detailed failure cases or decision attribution maps could shed light on where models diverge from human performance, particularly in VPT-perturb where models rely on “shortcut” strategies.

questions:
See weakness for more information about potential questions and improvements.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683729890

thread_type:
official_review

<thread_object>
creator: reviewer_az4r

<content>
summary:
This paper proposes 3D-PC, a new benchmark for visual perspective taking (VPT), an ability to predict what other perspectives see. The benchmark consists of three tasks: 1) telling which object is closer (depth order), 2) predicting if a camera can see a ball (VPT-basic), 3) another version of 2) where the occlusion is flipped. The authors then tested 33 human participants and over 300 DNNs on the benchmark, showing that despite a super-human performance on depth order, DNNs are near chance on VPT tasks.

strengths:
- The proposed benchmark is clearly motivated and well situated in the literature.
- The methodology is clearly explained with nice figures.
- Extensive experiments are conducted to show how the challenge can boost the 3D perception research of deep learning.

weaknesses:
- I think for a benchmark paper, one of the most important things is the correctness of the ”ground truth labels”. Although the method proposed may sound reasonable, it is hard to tell how unlikely the “GTs” are inaccurate. After all, the adopted 3D Gaussian Splatting method is not perfect. Therefore, I expect some experiments or at least some discussions on the correctness of GTs.
- I think the descriptions for VPT-Strategy are insufficient. I could only understand what this task was about when I read Figure 6.
- I don’t think we can say that DNNs learn depth from the depth order experiments. It seems to be very likely that the linear-probed DNNs only learned to solve the task based on the relative sizes of the red ball and green camera. Therefore, I don’t see the meaning of the depth order task.
- It seems that the role of the VPT-Strategy task is to test if the observer estimates the line-of-sight of the camera. However, I think it cannot take such a role. I guess if you finetune the DNNs on VPT-Strategy data, they will perform much better. If so, we can’t say they are estimating the line-of-sight because they are likely still using features to perform the task. Therefore, I don’t think the VPT-Strategy experiments are very meaningful.

Minor
- It is claimed that AlexNet is included in the model zoo (L250), but I didn’t find it in Appendix Table 1?
- Figure 2: VPT-easy → VPT-basic.

questions:
Please see the Weaknesses. Specifically, please answer:

- How do you evaluate the correctness of the ground truth labels generated by the proposed method?
- Can you do some experiments to show whether or not the DNNs learn to perform the depth order task based on relative sizes?
- How will the DNNs perform if they are finetuned on VPT-Strategy data? Do you think the results may change your perspective on the VPT-Strategy’s role?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730861935436

thread_type:
official_review

<thread_object>
creator: reviewer_nwpn

<content>
summary:
Visual Perspective Taking (VPT) refers to our ability to understand how others see the world from their viewpoint. This paper has developed a new benchmark to test whether DNNs can perform this skill. Their test includes three components: determining the order of objects by depth, VPT-basic, and VPT-strategy that eliminates potential shortcuts solutions.

While DNNs matched or even exceeded human ability in determining object depth order, they struggled significantly with VPT-basic, performing at nearly random levels. In contrast, humans excelled at these same tasks with almost perfect accuracy. Even when DNNs are fine-tuned for VPT-basic, they still failed at VPT-Strategy.

These findings suggest that DNNs don't naturally develop the ability to take others' perspectives, even after extensive training on static images or fine-tuning on VPT-basic tasks.

strengths:
- The paper used 3D Gaussian Splatting to create large sets of realistic images. 
- This method gave them precise control over the test conditions, helping them mitigate confoudning factors.
- Their analysis revealed interesting insights: while DNNs can learn to understand depth from static images, they don't develop the ability to take others' perspectives. They also found that a network's accuracy on ImageNet correlates with the 3D capabilities of DNNs they tested.

weaknesses:
Using only green cameras and red target points in their test images raises a few questions: Would the main results still hold if they used other colors or camera shapes? Is the paper assuming the DNNs can easily spot the camera and determine its viewing angle in the images? If so, how much do the camera's physical properties (like its size, line thickness, or color) affect the results?

questions:
The paper mentions in its appendix that when testing Visual Language Models (VLMs), they provided in-context examples to help guide the models. However, it's unclear whether they also included step-by-step reasoning examples like chain-of-thought prompting, which can probably help the models better understand how to approach these perspective-taking tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732323794961

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We thank the reviewers for their extensive feedback. We are confident that we have addressed the main critiques, which we summarize below along with our responses:

**Questions about the depth-order task (zZ4R, gKTT):** Humans rely on a number of monocular depth cues, such as linear perspective, texture gradients, occlusions, and relative size. As pointed out by the reviewers, the relative size of the green/red target objects is a strong cue for our depth-order task. DNN attribution maps indicate that they focus on both objects when solving the task, and that this focus increases as a function of performance on the depth-order task (Appendix Fig. 9b). Moreover, in our original submission we reported that models which were more accurate on the depth-order task were also more aligned with humans in their error patterns on the task (Appendix Fig. 8, purple dots, linear probe: $\rho=0.23, p < 0.001$, finetune: $\rho=0.24, p < 0.001$). In summary, the models in our zoo focused on object features — and potentially their relative sizes — to solve the depth-order task. However, relative size is also a cue that humans rely on for depth perception, and models and humans are strongly correlated in their decision making on the task.

More to the point raised by some of the reviewers, the purpose of including a depth-order task in the 3D-PC was to enable a broad comparison of the 3D perception capabilities of models and humans. We counterbalanced labels of depth-order and VPT-basic, and showed a strong dissociation between human 3D perception and 3D perceptual capabilities of nearly every kind of image-based model in use today — from ViTs to diffusion models and from CNNs to vision language models.

We believe that presenting the VPT results alone would be an incomplete snapshot of DNN 3D capabilities. Indeed, one of the most surprising results in our study is that DNNs, trained on static images, are growing progressively better at depth-order as they improve in object classification. In contrast, while there is a similar trend on VPT, the slope of this improvement is significantly smaller than on depth-order, which implies that a different modeling paradigm is needed for human-level VPT but not for other 3D vision tasks like depth-order. We discussed these findings and their connection to biological vision in our original submission (Discussion and Related work), and based on reviewer feedback, we have expanded our related work to connect to more current 3D computer vision work.

**Why do DNNs struggle at VPT (NwpN, zZ4R, gKTT)?** We demonstrated that DNNs combined with linear probes could not solve VPT-basic. Even after fine-tuning on VPT-basic, models could not generalize to new views where a human-like “line-of-sight” strategy would work (VPT-Strategy). In our original submission, we included an experiment to understand what strategy DNNs end up learning from fine-tuning (Appendix 9). Specifically, we computed attribution maps on every image for each model, and then looked at the overlap of these maps with the target objects (red ball/green camera). Plotting VPT-basic accuracy against object localization shows a significant correlation (Fig. 9c, $\rho=0.49, p < 0.001$): models learn to solve the task by becoming more exquisitely tuned to the object locations, but without learning to estimate line-of-sight between them; a strategy which we referred to as “feature-based” (Fig. 6A). Thus, our findings show that new architectures and/or data diets and/or training routines are needed for DNNs to learn a human-like strategy and achieve human-level performance on VPT.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper has developed a new benchmark to test whether DNNs can perform Visual Perspective Taking (VPT). Rigorous experiments reveal new insights that DNNs fail at VPT tasks. 

The paper is well-written. The methodology for studying the problem and the benchmark are novel and interesting. The insights obtained from this study are useful for the future development of AI models in 3d depth perception.

Main concerns on depth order and the explanations why models struggle at VPT are addressed during the rebuttal.

All the reviewers reached an agreement to accept the paper after the rebuttal.