SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Qualitative comparison of inference algorithms. Cells are colored based on the posterior distribution over goals if the agent is observed in that cell. Cells marked  $\times$  had all samples rejected. Gray shaded cells were excluded from analysis. $^{2}$  We show results for 10 samples and 1,000 (1k) samples, comparing rejection sampling as described by Lopez-Brau et al. [2020], our method, and human subjects. We produce near-convergent human-like inferences with only 10 samples.
Task	Rejection
(10)	Rejection
(1k)	Ours (10)	Ours (1k)	Humans
Gridworld
(two doors,
as in prior
work)	Previously
shown to be
matched by
model; see
[Lopez-Brau
et al., 2022]
Gridworld
(start
anywhere)
Keys
(holding
no key)
Keys
(holding
pink key)
Keys
(holding
green key)
p(goal = green gem | x)
p(goal = blue gem | x)	0.0
1 - p(goal = blue gem | x)	0.6
0.8
1.0
p(goal = red gem | x)
3.3 Comparison to human judgements
We recruited  $N = 200$  participants and collected judgements for a variety of "snapshots" in each of our domains (see experimental design in Appendix A). We found that our model predicts human intuitions well (see last columns of Tables 1 and 2), with strong correlations across domains (see Figure 3). Thus, our work not only replicates the findings of previous work [Lopez-Brau et al., 2020, 2022], but also shows that those findings continue to hold in domains that previous inference algorithms could not scale to.
2 Shaded cells in Row 3 are excluded because they are inaccessible without holding a key. Shaded cells in Row 4 are excluded because if the agent already picked up the pink key earlier, there is no reason to then move
Table 2: Qualitative comparison of inference algorithms. Blocks are colored according to inferred probability of that block having been touched by the person stacking the blocks (red is high probability, blue is low). We show results for 10 samples and 1,000 (1k) samples, comparing rejection sampling, our method, and human subjects. We produce near-convergent inferences with only 10 samples. In comparison, rejection sampling is unable to make any inference with 10 samples, and sometimes even fails with 1,000 samples. When it succeeds, its predictions are high-variance and overconfident.
Rejection (10)	Rejection (1k)	Ours (10)	Ours (1k)	Humans
(all samples rejected)	G	G	G	G
E	E	E	E
D	I	R	D	I	R	D	I	R	D	I	R
(all samples rejected)	G	I	G	I	G	I	G	I
E	G	E	G	E	G	E	G
R	D	D	R	D	R	D	R	D
(all samples rejected)	(all samples rejected)	G	G	G
E	E	E	E
D	R	I	D	R	I	D	R	I	D	R
(all samples rejected)	I	G	I	G	I	G	I	G
G	D	R	G	D	R	G	D	R	G	D
(all samples rejected)	I	G	I	G	I	G	I	G
G	D	R	G	D	R	G	D	R	G	D
(all samples rejected)	E	E	E	E
I	D	G	G	R	I	D	G	G	I	D
(all samples rejected)	(all samples rejected)	E	E	E
R	D	I	G	G	R	D	I	G	G
Cognitive science Human social cognition and “theory of mind” are well-modeled by Bayesian inverse planning [Baker et al., 2009, Jara-Ettinger, 2019, Baker et al., 2017], which infers an agent's goals from its observed actions. Inverse planning predicts human inferences about multiple agents [Kleiman-Weiner et al., 2016, Wu et al., 2021], social scenes [Ullman et al., 2009, Netyahu et al., 2021], emotion [Ong et al., 2015], and agents who make mistakes [Zhi-Xuan et al., 2020].
A growing body of work studies how people make inferences about the past from a "snapshot" of present physical evidence [Smith and Vul, 2014, Gerstenberg et al., 2021, Lopez-Brau and Jara-Ettinger, 2020, Paulun et al., 2015, Yildirim et al., 2017, 2019]. Recently, Lopez-Brau et al. [2020,
far away from the doors. Similarly, in Row 5, there is no reason to move that far left after picking up the green key. We found that querying repeatedly at those extreme locations confuses human participants about the task setup. The single cell above the green key in Row 4 is an exception we wished to test: one might think from that position that the agent is trying to collect both keys, leading to uncertainty about the goal. But people (and the model) agree that the agent still wants the red gem.
Table 3: Quantitative comparison of inference algorithms (see Section 3.2). We show the average total variation distance (TV) of a 10-sample posterior estimate against a converged posterior, averaged over 100 trials. Lower is better. Our method does significantly better than rejection sampling [Lopez-Brau et al., 2022] for each task, whether the converged "ground truth" is computed with 1,000 samples using our method, or 10,000 samples using rejection sampling.
Benchmark (TV against Ours @ 1k)	Rejection @ 10	Ours @ 10
Grid (two doors)	0.063	0.0257
Grid (starting anywhere)	0.159	0.0538
Keys (observed holding no key)	0.409	0.108
Keys (observed holding pink key)	0.388	0.157
Keys (observed holding green key)	0.381	0.119
Blocks	0.985	0.358
Benchmark (TV against Rejection @ 10k)	Rejection @ 10	Ours @ 10
Grid (two doors)	0.0705	0.0336
Grid (starting anywhere)	0.153	0.0836
Keys (observed holding no key)	0.672	0.410
Keys (observed holding pink key)	0.456	0.382
Keys (observed holding green key)	0.463	0.312
Blocks	Rejection @ 10k did not converge
2022] asked how people make inferences about the past and future of agents from static physical evidence they leave behind, which is something even children can do [Pelz et al., 2020, Jacobs et al., 2021]. We build on this line of work by dramatically accelerating inference, and extending it to more sophisticated domains where previous methods could not scale.
Artificial intelligence Bayesian approaches have been successful in approaches to plan recognition, the problem of inferring an agent's plan from observed actions [Ramirez and Geffner, 2009, 2010, Sohrabi et al., 2016, Charniak and Goldman, 1993]. Our work provides a method for plan recognition from a single state snapshot, with no need to observe actions.
The reinforcement learning community has long sought to learn an agent's reward function by observing the actions it takes via inverse reinforcement learning or IRL [Ng et al., 2000, Arora and Doshi, 2021, Ziebart et al., 2008]. Recently, Shah et al. [2019] proposed "IRL from a single state." Their method, "Reinforcement Learning by Simulating the Past" (RLSP), learns reward functions based on a single observation of a human in the environment, assuming that the observation is taken at the end of a finite-horizon MDP of fixed horizon  $T$ . We build on RLSP in three ways: (1) RLSP is highly sensitive to the time horizon hyperparameter  $T$ . We dispense with the fixed-horizon assumption altogether, integrating over past trajectories of all possible lengths. (2) Unlike Shah et al., we do not
Figure 3: Our inferences correlate well with human responses. In the "grid" plot, colors represent goal gem colors. In the "keys" plot, colors represent which keys (if any) the agent is seen holding, and we show  $p(\text{goal is blue gem} \mid x)$ . In the "blocks" plot, we show  $p(\text{touched} \mid x)$  for each block.
assume the snapshot was taken at the end of the agent's journey—the agent can be observed partway. (3) Our sampling-based method scales to significantly larger domains, because we do not have to integrate exhaustively over all possible trajectories.
The Deep RLSP algorithm of Lindner et al. [2020] also addresses point (3) above by sampling past trajectories. However, while Deep RLSP's maximum-likelihood formulation is designed to recover robot policies, our Bayesian approach is designed specifically to model human intuitions about agents' goals. Thus, unlike Deep RLSP, our method incorporates priors over start states in a principled way, and produces human-like uncertainty quantification.
5 Limitations and future work
Sampling over goals In this paper, we showed how to scale inference to a large set of possible initial states. However, we compute the posterior by enumeration over all possible goals, which takes linear time in the size of the space of goals. To scale to larger goal spaces as well, it may be possible to use Markov Chain Monte Carlo to sample goals conditioned on the observed state, similar to what Veach and Guibas [1997] propose for path tracing in computer graphics. This may seem challenging because we only stochastically estimate the likelihood  $p(x \mid g)$ . However, because our estimator is unbiased, it is still possible to use it to compute valid Metropolis-Hastings transitions [Andrieu and Roberts, 2009].
Cognitive plausibility Our method's sample efficiency suggests that it may resemble how humans actually do this task, similar to how few-shot algorithms have shown promise in modeling human behavior in other domains [Vul et al., 2014]. Following previous work using eye-tracking studies to investigate cognitive processes underlying simulation and planning [Gerstenberg et al., 2017], we hope to use eye-tracking to compare human strategies to our algorithm. In the language of Marr [1982], this would allow us to go beyond the computational account of Lopez-Brau et al. [2020] and take a first step towards an algorithmic account of inverse planning with snapshot states.
Theoretical analysis The time complexity of taking a single sample using our algorithm is straightforward: it scales linearly with the Russian roulette depth and the number of possible goals (although sub-linear runtime in number of goals is possible through sampling as described above). However, the more pertinent question is that of sample complexity. While our algorithm is unbiased, analyzing the rate of convergence as a function of the size of the sample space and complexity of the domain are topics for future work.
Investigating potential applications While our goal in this paper was to model how humans perform this inference task, extensions of our method could be applied to a variety of practical problems: some examples include understanding dynamic action in photographs of sports, interpreting step-by-step instruction manuals, and forensic reconstruction based on available static evidence.
From analysis to synthesis We began this paper with Hemingway's short story—and indeed, more generally, artists have long depicted dynamic action through static scenes [McCloud, 1993]. In future work we hope to consider the inverse problem of designing interesting, evocative, or ambiguous scenes by optimizing over inference [Chandra et al., 2023a,b, 2022], and designing expressive and human-interpretable robotic "gestures" similar to legible planning [Dragan et al., 2013].
In this paper, we offered a cognitively-plausible algorithm for making inferences about the past and the future based on the observed present. Building on prior work from both the cognitive science and AI communities, and drawing inspiration from the Monte Carlo rendering literature in computer graphics, we presented a highly sample-efficient method for performing such inferences. Finally, we showed that our method matched human intuitions on a variety of challenging inference tasks to which previous methods could not scale.
We thank Joe Kwon, Tan Zhi-Xuan, Alicia Chen, Max Siegel, Lionel Wong, and John Cherian for thoughtful conversations as we developed these ideas. This research was funded by NSF grants #CCF-1231216, #CCF-1723445 and #2238839, and ONR grant #00010803. Additionally, KC was supported by the Hertz Foundation, the Paul and Daisy Soros Fellowship, and an NSF Graduate Research Fellowship under grant #1745302, and TC was supported by an NDSEG Fellowship.
Marc K Albert and Donald D Hoffman. The generic-viewpoint assumption and illusory contours. Perception, 29(3):303-312, 2000. URL https://journals.sagepub.com/doi/pdf/10.1068/p3016.
Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. Aether: An embedded domain specific sampling language for monte carlo rendering. ACM Transactions on Graphics (TOG), 36 (4):1-16, 2017. URL https://dl.acm.org/doi/pdf/10.1145/3072959.3073704.
Christophe Andrieu and Gareth O Roberts. The pseudo-marginal approach for efficient monte carlo computations. 2009. URL https://www.jstor.org/stable/30243645.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021. URL https://www.sciencedirect.com/science/article/am/pii/S0004370221000515.
James Arvo and David Kirk. Particle transport and image synthesis. In Proceedings of the 17th annual conference on Computer graphics and interactive techniques, pages 63-66, 1990. URL https://dl.acm.org/doi/10.1145/97880.97886.
Chris L Baker, Noah D Goodman, and Joshua B Tenenbaum. Theory-based social goal inference. In Proceedings of the thirtieth annual conference of the cognitive science society, pages 1447-1452. Cognitive Science Society Austin, TX, 2008. URL https://cocolab.stanford.edu/papers/BakerEtAl2008-Cogsci.pdf.
Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329-349, 2009. URL https://www.sciencedirect.com/science/article/pii/S0010027709001607.
Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4): 0064, 2017. URL https://www.nature.com/articles/s41562-017-0064.
Leland Lavele Carter and Edmond Darrell Cashwell. Particle-transport simulation with the monte carlo method. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 1975.
Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Designing perceptual puzzles by differentiating probabilistic programs. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-9, 2022.
Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Acting as inverse inverse planning. In *Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings* (SIGGRAPH '23 Conference Proceedings), aug 2023a. doi: 10.1145/3588432.3591510.
Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Storytelling as inverse inverse planning. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023b.
Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly rational planning agents. Advances in neural information processing systems, 33:19238-19250, 2020. URL https://proceedings.neurips.cc/paper/2020/file/df3aabc649f9e3b674eeb790a4da224e-Paper.pdf.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
A Experimental design
For each of our experiments reported in Section 3.3, we recruited  $N = 200$  participants from Prolific [Palan and Schitter, 2018]. Participants were paid $15 per hour ($1.25 total for blocks and grid domains, and $2.00 for the keys domain), and our experiments were conducted with IRB approval.
Participants were first familiarized with the environment, through both text instructions and a sample video of an agent performing the task in the domain. Then, they were told that their objective was to infer the agent's goal from a single snapshot. They answered several questions to check their comprehension of both the domain and the task they were asked to perform, and were not allowed to continue unless they answered the comprehension questions correctly. The full experimental design is available in HTML format in the supplementary materials. No data was excluded from our analyses.
B Numerical test of correctness
Programming sophisticated importance sampling routines is a challenging and bug-prone engineering effort [Cusumano-Towner et al., 2019, Anderson et al., 2017, Pharr et al., 2016]. To test that our algorithm is unbiased, i.e. that it produces correct likelihoods in expectation, we compared likelihoods computed by rejection sampling and our sampler using converged estimates (25,000 samples each). For this experiment we used a uniform  $4 \times 4$  grid-world, with the prior on start states being uniform along the first row ( $x = 0$ ) and the goal being the far corner (3, 3). The results of this experiment are shown in Figure 4. Our estimator has a dramatically different implementation than rejection sampling (compare Algorithms 1 and 2). However, the computed likelihoods are indistinguishable at every cell in the grid, even in "corner-case" cells such as the goal cell itself. This provides a strong check that our algorithm and its implementation are both indeed correct.
C Additional domains
We used our algorithm to perform inferences in three additional domains. The purpose of these domains is to show the remarkable flexibility of our method: how it can make interesting inferences in a wide variety of settings. Though we did not collect human subject data for these domains, we show results for cases where the inference task is relatively straightforward.
Our sampler is unbiased and precisely matches rejection sampling
Figure 4: Our sampler's likelihoods precisely match rejection sampling, with and without bidirectional path tracing, giving a strong numerical check of our method's correctness (Appendix B).
Figure 5: The student is observed heading south around the wall. A rational inference is that the student started at home, and went around the wall to check what the far food truck was. Seeing that it was Lebanese and not Mexican (their favorite), the student disappointedly turns around to make peace with the nearby Korean food. As shown on the heatmap to the right, our model captures this joint belief-desire inference, predicting that the student now knows what is at both trucks, and reconstructing the student's likely preference ordering over the three cuisines. Note: the belief label “ $K \setminus ?$ ” means that the student thinks the south-west parking spot has a Korean food truck parked, but is unsure about the north-east parking spot. See Appendix C.1.
C.1 Food trucks (joint belief/desire inference)
The food trucks domain, taken from the cognitive science literature [Baker et al., 2017], is a Partially Observable Markov Decision Process (POMDP). It consists of a  $5 \times 10$  gridworld with an opaque wall in the middle. A hungry graduate student wakes up at home (one side of the wall) and wishes to eat at a food truck. There are two parking spots where food trucks usually park, and three kinds of food trucks that could be parked at each of those spots: Korean, Lebanese, and Mexican (K, L, and M). The graduate student might have preferences among the cuisines, but might also be uncertain about which trucks are parked at each spot today. Thus, they might engage in information-seeking behavior by looking behind the opaque wall, and then choosing a food truck to walk to based on their preferences. The inference task is to determine (a) the student's preferences over food trucks, and (b) the student's (current) belief state about which truck is at each parking spot.
Using this domain, Baker et al.'s inverse planning model was able to jointly infer the student's beliefs and desires from an observed trajectory; those inferences closely matched responses from human subjects. Here, we perform the same type of inference, but from a single observed snapshot.
For example, in the example in Figure 5, the student is observed moving south next to the wall. A Korean food truck is parked in the southwest parking spot, and a Lebanese food truck is parked in the northeast spot. Seeing this scene, a reasonable inference is that the student went looking around the wall to see if the Mexican food truck (their favorite) was parked on the other side. Seeing that it was Lebanese food instead, the student turns around and makes peace with the nearby Korean food. Indeed, our model captures this inference: in the joint posterior distribution over both beliefs and desires, our model is confident that the student now knows that the northeast truck has Lebanese food, and furthermore that the student's favorite food is Mexican.
A more sophisticated inference emerges if the student is observed moving north instead of south (Figure 6). Now, a reasonable inference is that the student dislikes Korean food, and is going around the wall to check what is at the other truck. The model captures this: it favors the hypothesis that the student is unsure what is at the northeast truck, and also places high weight on Korean being the least favorite food option.
However, as is visible on the right half of the heatmap, the model also places some weight on the possibility that the student knows that there is Lebanese food and prefers it, or that the student (mistakenly) believes there is Mexican food and prefers that.
Figure 6: Here, the student is observed going north instead of south. A more sophisticated inference emerges, showing that the student is likely uncertain about which truck is parked behind the wall. See Appendix C.1
Figure 7: Two agents are observed by a security camera in an art museum. Who is the guard, who is the thief, and where is the thief trying to escape to? Our model predicts that the guard is the blue agent, the thief is the pink agent, and that the exit is in the bottom right. See Appendix C.2.
C.2 Heist (multi-agent domain)
In this multi-agent domain inspired by classic stimuli in cognitive science [Baker et al., 2008, Southgate and Csibra, 2009, Heider and Simmel, 1944], two agents—blue and pink—occupy a  $7 \times 7$  gridworld representing an art museum. One of the agents is a “thief,” whose objective is to escape the museum by reaching the exit, and the other is a “guard,” whose objective is to catch the thief. There are four doors in the room, only one of which is an exit, and the rest of which are dead ends. Both agents know which door is the exit, but this information is not visible to the observer (all doors are rendered identically). The inference tasks are to look at a snapshot of the two agents and jointly infer (a) which agent is the thief and which is the guard, and (b) which door is the exit.
In the example in Figure 7, it is clear from the snapshot that the blue agent is the guard and is chasing the pink agent, the thief, to the bottom-right corner. The model reproduces this inference, though also acknowledges the possibility that the thief might actually be heading onward past the bottom-right, to the bottom-left corner instead.
Figure 8: In these examples, it is unclear who the guard and thief are—however, it is clear where the exit is. The model reproduces this uncertainty as desired. See Appendix C.2.
The next two examples (Figure 8) are ambiguous cases: the two agents are in symmetric positions, so it is unclear who is who. Here, the model can determine with high confidence where the exit is, but remains uncertain about who is the thief and who is the guard.
Finally, in the last example (Figure 9), it is unclear whether a blue guard is blocking a pink thief from heading to the top-right corner, or whether a pink guard is blocking the blue thief from heading the the bottom-right corner. Indeed, the model reproduces this ambiguity.
C.3 Cart-pole (continuous state space with physical dynamics)
The cart-pole domain is a classic problem in reinforcement learning and optimal control. The goal is to balance a pole in an upright position, by moving the cart left or right. The state space of this
Figure 9: In this example, it is unclear whether a blue guard is blocking a pink thief from heading to the top-right corner, or whether a pink guard is blocking the blue thief from heading the bottom-right corner. The model reproduces this joint uncertainty as desired. See Appendix C.2.
domain consists of four continuous numbers: the horizontal position of the cart and its velocity, and the angle of the pole along with its angular velocity. The inference tasks are to look at a snapshot image—which only shows the cart position and the pole angle—and determine the velocity of the cart and the angular velocity of the pole. Note that rejection sampling cannot solve this task because the probability of a randomly-sampled trace passing through the observed state is zero.