Title: Inferring the Future by Imagining the Past
Venue: NeurIPS 2023
Accepted: yes
Keywords: cognitive science

cogsci

inverse planning

Bayesian inference

theory of mind

Monte Carlo

inverse reinforcement learning
Abstract: A single panel of a comic book can say a lot: it can depict not only where the characters currently are, but also their motions, their motivations, their emotions, and what they might do next. More generally, humans routinely infer complex sequences of past and future events from a *static snapshot* of a *dynamic scene*, even in situations they have never seen before.

In this paper, we model how humans make such rapid and flexible inferences. Building on a long line of work in cognitive science, we offer a Monte Carlo algorithm whose inferences correlate well with human intuitions in a wide variety of domains, while only using a small, cognitively-plausible number of samples. Our key technical insight is a surprising connection between our inference problem and Monte Carlo path tracing, which allows us to apply decades of ideas from the computer graphics community to this seemingly-unrelated theory of mind task.

Inferring the Future by Imagining the Past

Abstract

A single panel of a comic book can say a lot: it can depict not only where the characters currently are, but also their motions, their motivations, their emotions, and what they might do next. More generally, humans routinely infer complex sequences of past and future events from a static snapshot of a dynamic scene , even in situations they have never seen before.

In this paper, we model how humans make such rapid and flexible inferences. Building on a long line of work in cognitive science, we offer a Monte Carlo algorithm whose inferences correlate well with human intuitions in a wide variety of domains, while only using a small, cognitively-plausible number of samples. Our key technical insight is a surprising connection between our inference problem and Monte Carlo path tracing, which allows us to apply decades of ideas from the computer graphics community to this seemingly-unrelated theory of mind task.

1 Introduction

Hemingway's shortest short story simply reads 'For sale: baby shoes, never worn.' There is no action in this sentence-it is just a description of a static scene-but as readers, we nonetheless infer a complex and tragic backstory from the 'snapshot' that Hemingway describes. This remarkable ability comes naturally to humans: we routinely reconstruct motives from evidence (e.g. at a crime scene), recognize intentions from unfinished tasks (e.g. grading incomplete homework), and enjoy artistic depictions of dynamic action in static media (e.g. a Renaissance 'tableau' or sculpture).

How do we do it, so quickly and so intuitively? A long line of empirical work in cognitive science has shown that people, even infants and young children [Southgate and Csibra, 2009, Gergely and Csibra, 2003, Jara-Ettinger et al., 2015, Hamlin et al., 2007], make rapid, flexible, and robust theory-of-mind judgments 'out of the box': in novel domains they have never seen before, and without extensive pretraining on data. This remarkable ability has motivated decades of work in AI (see Section 4), which has successfully addressed the problem of inferring an agent's goal from a trajectory of observed actions in a few- or even one-shot manner [Baker et al., 2017, 2009]. These methods cast the problem of action understanding as Bayesian inference: P ( goal | actions ) ∝ P ( actions | goal ) P ( goal ) , where P ( actions | goal ) is modeled by comparing the observed actions to the optimal actions a rational agent would take towards that goal.

However, if we only observe a single state snapshot, these methods break down-there are simply no actions to condition on. Instead, we must jointly infer not only where the agent might be going, but also where it came from. Recently, Lopez-Brau et al. [2020, 2022] performed this inference by rejection-sampling possible paths taken by the agent. Their model's predictions are remarkably close to human judgements. However, rejection sampling is extremely inefficient-it is slow even on simple problems, and simply does not scale to more sophisticated problems, suggesting that there is more to how humans perform such inference.

!

s

(1)

s

(2)

s

(3)

π

s:g

(3)

π

s:g

(1)

π

s:g

(2)

g

x

🙂

🏁

🏁

🏁

Lopez-Brau et al. sample s ~ p(s) and then paths from s to g . However, most paths do not pass through x…

π

x:g

(1)

g

x

🏁

🏁

🏁

π

x:g

(3)

π

x:g

(2)

🙂

π

:x

(1)

π

:x

(2)

π

:x

(3)

We in stead sample two kinds of paths: forward from x to g , and backwards from x through the space of possible s 's.

g

x

🙂

🏁

🏁

🏁

Suppose you observe an agent at x . If they started from some s ~ p(s) , what is the likelihood they are going to goal g ?

area shaded to

represent

p(s)

In this paper, we propose a dramatically more efficient algorithm that recovers human-like judgements even on significantly more sophisticated problems. We take inspiration from the field of computer graphics, which has developed a wealth of Monte Carlo algorithms for 'path tracing.' Our key insight is an analogy between tracing paths of light through 3D scenes and paths of agents through planning domains. This insight allows us to import decades of algorithmic advances from computer graphics to the seemingly-unrelated social intelligence problem of snapshot inference.

In Section 2 we formalize this problem and present a new Monte Carlo algorithm for sampling approximate solutions. Then, in Section 3, we show that our algorithm is up to 30 , 000 × more efficient than prior work. Finally, in Section 3.3 we demonstrate via three behavioral studies that our model's predictions match human judgements on new, scaled-up tasks inaccessible to prior work.

2 Our proposed algorithm

We describe our method in the setting of Markov Decision Processes (MDPs). An MDP is a tuple ⟨S , A , T , R , γ ⟩ , where S is the set of possible states, A is the space of possible actions, T : S × A × S → [0 , 1] gives the transition probability, R : S × A → R is the reward function and γ ∈ [0 , 1] is a discount factor. We will specifically consider MDPs that have a single 'goal state' g with positive reward, such as a grid world with a target square the agent needs to reach. We will assume that g is known to the agent, but not to us (the observers).

Consider an agent that begins in some initial state s ∼ p ( s ) . While the agent is on its trajectory from s to g , we observe a 'snapshot' of the agent in some interim state x . Given only x (and not s !), our aim is to infer p ( g | x ) .

Applying Bayes' rule, we have p ( g | x ) ∝ p ( x | g ) p ( g ) , where p ( g ) is our prior belief about what g might be. The likelihood p ( x | g ) can be computed by marginalizing over all possible start states s and all possible paths π s : g from s to g , where a path π s : g is defined as a sequence of states π s : g = ( s 1 , . . . s n ) such that s 1 = s and s n = g . Performing this marginalization, and noting that p ( s | g ) = p ( s ) because s and g are independent, we have:

To evaluate the likelihood of a snapshot p ( x | π s : g , s, g ) , we apply the size principle [Tenenbaum, 1998, 1999, Griffiths and Tenenbaum, 2006], analogous to the generic viewpoint assumption in computer vision [Freeman, 1994, Albert and Hoffman, 2000]. In this case, the principle states that the snapshot was equally likely to have been taken anywhere along the path, and therefore the likelihood of a snapshot conditioned on a path is inversely proportional to the length of the path. If δ ( x ∈ π ) indicates whether path π passes through x , and | π | indicates the length of π , then p ( x | π s : g , s, g ) is given by δ ( x ∈ π s : g ) | π s : g | -1 . As a consequence of this definition, all paths that do not pass through x contribute zero probability, and thus can be dropped from the integral.

To evaluate the likelihood of a path p ( π s : g | s, g ) , we apply the principle of rational action: agents are likelier to take actions that maximize their utility [Dennett, 1989, Jara-Ettinger et al., 2016]. There are many ways to formalize this intuition. A common option, which we adopt, is to say that at each step, the agent chooses an action with probability proportional to the softmax over its Q -values at its current state, with some temperature β . That is, p ( x → x ′ | g ) ∝ ∑ a exp ( βQ g ( x, a )) T ( x → x ′ | a ) , where T ( x → x ′ | a ) is the transition probability from x to x ′ if action a is taken, and p ( π | g ) ∝ Π t p ( x t → x t +1 | g ) .

We now have all the ingredients we need to evaluate p ( x | g ) . However, to compute it exactly we would need to integrate over all possible initial states s , and the set of paths π s : g , which could be infinite (agents might wander for arbitrarily long, albeit with vanishingly low probability). To approximate the likelihood in finite time, Lopez-Brau et al. turn to Monte Carlo sampling (Algorithm 1). They rejection-sample paths π s : g by sampling a candidate start state s ( i ) ∼ p ( s ) , simulating a rollout of the agent to sample a path π ( i ) s : g ∼ p ( π ( i ) s : g | s ( i ) , g ) , and then averaging the integrand over these samples. With N samples, their unbiased likelihood estimator is given by ˆ p ( x | g ) = 1 N ∑ N i =1 δ ( x ∈ π ( i ) s : g ) | π ( i ) s : g | -1 .

Unfortunately, in practice this scheme is extremely slow: even in a 7 × 7 gridworld with fewer than 49 states (only 2 of which were possible initial states), Lopez-Brau et al. report taking over 300,000 trajectory samples per goal to perform inference. In the rest of this section, we will describe a series of algorithmic enhancements that allow for comparable inference quality with just 10 samples per goal (i.e. 30,000 × fewer). We will develop our algorithm (Algorithm 2) through three insights.

2.1 First insight: only sample paths through the observed state

Our first insight is that δ ( x ∈ π ) is extremely sparse-most paths likely do not pass through x , and so most naïve path samples contribute zero to the estimator. We would like to only sample paths that pass through x . Any such path can be partitioned at x into two portions, π s : x and π x : g . Let us integrate separately over those portions.

This suggests a more efficient Monte Carlo sampling scheme. Rather than rejection-sampling paths π ( i ) s : g directly from s to g , we can independently sample two path segments: a 'past' path π ( i ) s : x from s to x , and a 'future' path π ( i ) x : g from x to g . Any such path is guaranteed to pass through x , so no samples are wasted.

However, we now have two new problems. First, it is not clear how to sample paths π ( i ) s : x from s to x , because rollouts of a simulated agent are unlikely to pass through x on their way to g . We could imagine using a second planner just to chart paths from s to x , but this would require a lot of additional planning work. Second, we still have to sample s . If the space of initial states is small (e.g. a room only has one or two doors), then this is no issue. However, in practice this space might be very large or even infinite. For example, if you observe someone driving to work in the morning, their home could be anywhere in the city. Furthermore, most of these states might be inaccessible or otherwise implausible, and it would be a waste of computational resources to consider them. In the next section, we show how to solve both of these problems by tracing paths backwards in time .

2.2 Second insight: sample π : x backwards in time

Our second insight is that we can collapse the first two integrals by jointly integrating over the domain of all paths π : x that terminate at x , no matter where they started from. Say a path π : x begins at π : x [0] . Then, we can rewrite our likelihood as below.

This suggests that we should sample π ( i ) : x backwards through time, starting from x . No matter how we extend this path, we obtain a valid path from π ( i ) : x [0] to x . We illustrate this in Figure 1.

An analogy to path tracing in computer graphics is helpful here. When rendering a 3D scene, a renderer must integrate over all paths of light that begin at a light source in the scene and end at a pixel on the camera's film-a problem formalized by the rendering equation [Kajiya, 1986]. Of course, these paths may be reflected and refracted stochastically by several surface interactions along the way. Rather than starting at one of the millions of light sources in the scene and tracing a ray hoping to eventually reach the camera film, renderers instead start at the camera and trace rays backward into the scene until they inevitably reach a light source.

Similarly, here we trace paths backwards from x into the pasts corresponds to a light source, each action taken by the agent corresponds to a stochastic surface interaction, and x corresponds to a pixel on the camera. Indeed, our integral is analogous to the rendering equation, bringing to our disposal the entire Monte Carlo light transport toolbox-a toolbox the rendering community has spent decades developing. (These techniques were pioneered by Veach [1998], though see Pharr et al. [2016] for an accessible review.) The particular ideas we borrow are the following:

Importance sampling The first idea is to be deliberate about how paths are extended backwards in time. We could sample predecessor states uniformly at random-however, that would lead to 'unlikely' or irrational paths. Instead, we preferentially select a predecessor state x prev based on the likelihood of transitioning to the current state from x prev. Then, we re-weight our path sample appropriately so that our estimator remains unbiased (see Algorithm 2, line 14).

Russian roulette termination The next concern is when to stop extending a path into the past. In principle, paths could be infinitely long in some domains. However, at some point paths become so unlikely that extending them is not worth the computational effort of tracing them. An unbiased finite-time solution to this problem is given by the Russian roulette method [Carter and Cashwell, 1975, Arvo and Kirk, 1990]: at each step, we toss a weighted coin, and only continue extending the path if it comes up 'heads.' Then, we weight subsequent samples appropriately to keep the estimator unbiased (see Algorithm 2, line 11).

Bidirectional path tracing The last concern is that the path may not ever 'find' the region of state space where the agent could have started. Consider a setting where the space of possible initial states is large but sparse-for example, if we know the agent started from a red house somewhere in the city. Importance sampling path predecessors with Russian roulette termination is not guaranteed to stop at a red house, whereas forward-sampling paths from a random red house is not guaranteed to pass through the observed state. In computer graphics, this situation is analogous to rendering a scene with many lights that are occluded from the camera. The classic solution is bi-directional path tracing [Lafortune and Willems, 1993, Veach and Guibas, 1995]: first, we 'cache' some paths forward-simulated from randomly sampled lights (see Algorithm 3). Then, when backwards-tracing rays, we find opportunities to connect them to a continuation in the cache (see Algorithm 2, line 8).

As a correctness check, we compare the numerical results from our method and the rejection sampling approach of Lopez-Brau et al. [2020] in Appendix B, showing that likelihoods computed from the two methods match closely as expected.

8

a

a.

2.3 Third insight: this algorithm allows for on-line planning

Our method is agnostic to the algorithm used to compute the agent's policy, which allows us to choose from any combination of reinforcement-learning or heuristic search approaches. Model-based methods like value iteration require an expensive pre-computation of the value function for all possible goals and states, even though some of those states may not ever be visited by our algorithm. It seems implausible that humans do this, because we make judgements quickly even in new domains. Moreover, adding obviously-irrelevant states should not make the problem harder to solve-we should simply not bother planning from those states until they somehow become relevant.

Motivated by this, in domains that can be expressed as classical planning problems (i.e. domains with a single 'goal state' with positive reward), we use an online A* planner instead of precomputing policies by value iteration. We define p ( x → x ′ | g ) to be a softmax over the difference in path costs between x and x ′ to g as computed by the planner: p ( x → x ′ | g ) ∝ ∑ a exp( β ( C ( x → g ) -C ( x ′ → g )) . That is, the agent is likelier to move to states that will bring it closer to the goal. To avoid re-planning from scratch for every evaluation of p ( x t -1 → x t | g ) , we run A-star backwards from the goal to the current state. This lets us re-use the bulk of its intermediate computations (known distances, evaluations of the heuristic, etc.) between queries.

Algorithm 3 Grow the bidirectional path tracer's cache (to be called repeatedly)

3 Experiments

To evaluate our sampling algorithm, we chose a suite of benchmark domains reflecting the variety of inferences humans make:

Simple gridworld We re-implement the 7 × 7 gridworld domain from Lopez-Brau et al. (Figure 2). The agent seeks one of three gems and at every timestep can move north, south, east or west. The viewer's inference task is to determine which gem the agent seeks. While Lopez-Brau et al. fix two possible starting-points ('entryways') for the agent, our method can optionally relax this constraint and instead have a uniform prior over the start state.

Doors, keys, and gems (multi-stage planning) This is a more advanced 8 × 8 gridworld, inspired by Zhi-Xuan et al. [2020]. The agent is blocked from its gem by doors , which can only be opened if the agent is has the correct keys (the agent always starts out empty-handed). The inference task is to look at a snapshot image and determine which gem the agent seeks. For example, if we observe the snapshot in Figure 2, we might infer that the agent plans to get the green key to obtain the blue gem.

Word blocks (non-spatial) In this domain, the agent spells a word out of the six letter blocks by picking and placing them in stacks on a table. However, they are interrupted (e.g. by a fire alarm) and have to leave the room before finishing. The inference tasks are to look at the blocks left behind and determine (a) which word the agent was trying to spell, and (b) which blocks the agent has touched.

Additional domains Appendix C shows results from additional domains from the cognitive science literature that further reflect the flexibility of our method. We demonstrate domains with partial observability (C.1), multiple agents (C.2), and continuous state spaces with physics (C.3). The breadth of our experiments reflects the typical scope of related work in cognitive science.

3.1 Qualitative analysis

Tables 1 and 2 show some example inferences made by our algorithm. Each cell is colored according to the posterior distribution over goals. If the inference algorithm produced no valid samples for a cell, that cell is marked with an × symbol. With just 10 samples, our method's posterior inferences are near-convergent and align well with human responses. In comparison, with 10 samples rejection sampling typically produces extremely noisy predictions, and often simply fails to produce any non-rejected samples at all-indeed, in the blocks domain even 1,000 samples are not always enough to produce a single non-rejected sample.

3.2 Quantitative analysis

We report the total variation TV ( x ) = 1 2 ∑ g i | ˆ p ( g i | x ) -p ( g i | x ) | between the true posterior and inferences made using 10 samples of both our method and rejection sampling, averaged for 100 trials and across all of the inference tasks in the benchmark. We take the true posterior to be our method's estimate with 1,000 samples, though we also show results if the true posterior is taken to be rejection sampling with 10,000 samples. Our results are shown in Table 3. Across all domains, our algorithm substantially outperforms rejection sampling.

Task

Rejection

(10)

Rejection

(1k)

Ours (10)

Ours (1k)

Humans

Gridworld

(two doors,

as in prior

work)

Previously

shown to

be

matched

by

model;

see

[Lopez-Brau

et al., 2022]

Gridworld

(start

anywhere)

(holding

no key)

(holding

pink key)

(holding

green key)

?

×

6

p(goal =green gem|x)

0.0

0.2

0.4

0.6

0.8

1.0

1 - p(goal = blue gem | x)

p(goal = red gem |x)

p(goal = blue gem | x)

p(goal = red gem |x)

3.3 Comparison to human judgements

We recruited N = 200 participants and collected judgements for a variety of 'snapshots' in each of our domains (see experimental design in Appendix A). We found that our model predicts human intuitions well (see last columns of Tables 1 and 2), with strong correlations across domains (see Figure 3). Thus, our work not only replicates the findings of previous work [Lopez-Brau et al., 2020, 2022], but also shows that those findings continue to hold in domains that previous inference algorithms could not scale to.

4

Rejection (10)

Rejection (1k)

Ours (10)

Ours (1k)

Humans

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

(all samples

rejected)

G

G

G

G

G

G

E

E

E

E

D

R

D

R

D

R

R

GER

G

G

G

E

G

G

E

G

D

R

D

R

D

R

D

G

G

G

G

G

G

E

E

R

D

R

D

R

G

E

E

G

E

G

E

G

D

R

D

R

G

D

R

D

R

D

R

P

0.0

0.2

0.4

0.6

0.8

1.0

p(touched |x)

4 Related work

Cognitive science Human social cognition and 'theory of mind' are well-modeled by Bayesian inverse planning [Baker et al., 2009, Jara-Ettinger, 2019, Baker et al., 2017], which infers an agent's goals from its observed actions. Inverse planning predicts human inferences about multiple agents [Kleiman-Weiner et al., 2016, Wu et al., 2021], social scenes [Ullman et al., 2009, Netanyahu et al., 2021], emotion [Ong et al., 2015], and agents who make mistakes [Zhi-Xuan et al., 2020].

A growing body of work studies how people make inferences about the past from a 'snapshot' of present physical evidence [Smith and Vul, 2014, Gerstenberg et al., 2021, Lopez-Brau and JaraEttinger, 2020, Paulun et al., 2015, Yildirim et al., 2017, 2019]. Recently, Lopez-Brau et al. [2020, far away from the doors. Similarly, in Row 5, there is no reason to move that far left after picking up the green key. We found that querying repeatedly at those extreme locations confuses human participants about the task setup. The single cell above the green key in Row 4 is an exception we wished to test: one might think from that position that the agent is trying to collect both keys, leading to uncertainty about the goal. But people (and the model) agree that the agent still wants the red gem.

2022] asked how people make inferences about the past and future of agents from static physical evidence they leave behind, which is something even children can do [Pelz et al., 2020, Jacobs et al., 2021]. We build on this line of work by dramatically accelerating inference, and extending it to more sophisticated domains where previous methods could not scale.

Artificial intelligence Bayesian approaches have been successful in approaches to plan recognition , the problem of inferring an agent's plan from observed actions [Ramırez and Geffner, 2009, 2010, Sohrabi et al., 2016, Charniak and Goldman, 1993]. Our work provides a method for plan recognition from a single state snapshot, with no need to observe actions.

The reinforcement learning community has long sought to learn an agent's reward function by observing the actions it takes via inverse reinforcement learning or IRL [Ng et al., 2000, Arora and Doshi, 2021, Ziebart et al., 2008]. Recently, Shah et al. [2019] proposed 'IRL from a single state.' Their method, 'Reinforcement Learning by Simulating the Past' (RLSP), learns reward functions based on a single observation of a human in the environment, assuming that the observation is taken at the end of a finite-horizon MDP of fixed horizon T . We build on RLSP in three ways: (1) RLSP is highly sensitive to the time horizon hyperparameter T . Wedispense with the fixed-horizon assumption altogether, integrating over past trajectories of all possible lengths. (2) Unlike Shah et al., we do not

Grid, r=0.94

Blocks, r=0.76

1.0

1.0

1.0

0.8

0.8

0.8

S

=

0.6

S

0.6

0.6

2

0.4

0.4

2

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

0.00

0.25

0.50

0.75

1.00

model

Model

model

assume the snapshot was taken at the end of the agent's journey-the agent can be observed partway. (3) Our sampling-based method scales to significantly larger domains, because we do not have to integrate exhaustively over all possible trajectories.

The Deep RLSP algorithm of Lindner et al. [2020] also addresses point (3) above by sampling past trajectories. However, while Deep RLSP's maximum-likelihood formulation is designed to recover robot policies, our Bayesian approach is designed specifically to model human intuitions about agents' goals. Thus, unlike Deep RLSP, our method incorporates priors over start states in a principled way, and produces human-like uncertainty quantification.

5 Limitations and future work

Sampling over goals In this paper, we showed how to scale inference to a large set of possible initial states. However, we compute the posterior by enumeration over all possible goals, which takes linear time in the size of the space of goals. To scale to larger goal spaces as well, it may be possible to use Markov Chain Monte Carlo to sample goals conditioned on the observed state, similar to what Veach and Guibas [1997] propose for path tracing in computer graphics. This may seem challenging because we only stochastically estimate the likelihood p ( x | g ) . However, because our estimator is unbiased, it is still possible to use it to compute valid Metropolis-Hastings transitions [Andrieu and Roberts, 2009].

Cognitive plausibility Our method's sample efficiency suggests that it may resemble how humans actually do this task, similar to how few-shot algorithms have shown promise in modeling human behavior in other domains [Vul et al., 2014]. Following previous work using eye-tracking studies to investigate cognitive processes underlying simulation and planning [Gerstenberg et al., 2017], we hope to use eye-tracking to compare human strategies to our algorithm. In the language of Marr [1982], this would allow us to go beyond the computational account of Lopez-Brau et al. [2020] and take a first step towards an algorithmic account of inverse planning with snapshot states.

Theoretical analysis The time complexity of taking a single sample using our algorithm is straightforward: it scales linearly with the Russian roulette depth and the number of possible goals (although sub-linear runtime in number of goals is possible through sampling as described above). However, the more pertinent question is that of sample complexity . While our algorithm is unbiased, analyzing the rate of convergence as a function of the size of the sample space and complexity of the domain are topics for future work.

Investigating potential applications While our goal in this paper was to model how humans perform this inference task, extensions of our method could be applied to a variety of practical problems: some examples include understanding dynamic action in photographs of sports, interpreting step-by-step instruction manuals, and forensic reconstruction based on available static evidence.

From analysis to synthesis We began this paper with Hemingway's short story-and indeed, more generally, artists have long depicted dynamic action through static scenes [McCloud, 1993]. In future work we hope to consider the inverse problem of designing interesting, evocative, or ambiguous scenes by optimizing over inference [Chandra et al., 2023a,b, 2022], and designing expressive and human-interpretable robotic 'gestures' similar to legible planning [Dragan et al., 2013].

6 Conclusion

In this paper, we offered a cognitively-plausible algorithm for making inferences about the past and the future based on the observed present. Building on prior work from both the cognitive science and AI communities, and drawing inspiration from the Monte Carlo rendering literature in computer graphics, we presented a highly sample-efficient method for performing such inferences. Finally, we showed that our method matched human intuitions on a variety of challenging inference tasks to which previous methods could not scale.

Acknowledgments and Disclosure of Funding

We thank Joe Kwon, Tan Zhi-Xuan, Alicia Chen, Max Siegel, Lionel Wong, and John Cherian for thoughtful conversations as we developed these ideas. This research was funded by NSF grants #CCF-1231216, #CCF-1723445 and #2238839, and ONR grant #00010803. Additionally, KC was supported by the Hertz Foundation, the Paul and Daisy Soros Fellowship, and an NSF Graduate Research Fellowship under grant #1745302, and TC was supported by an NDSEG Fellowship.

References

Marc K Albert and Donald D Hoffman. The generic-viewpoint assumption and illusory contours. Perception , 29(3):303-312, 2000. URL https://journals.sagepub.com/doi/pdf/10.1068/p3016.

Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. Aether: An embedded domain specific sampling language for monte carlo rendering. ACM Transactions on Graphics (TOG) , 36 (4):1-16, 2017. URL https://dl.acm.org/doi/pdf/10.1145/3072959.3073704.

Christophe Andrieu and Gareth O Roberts. The pseudo-marginal approach for efficient monte carlo computations. 2009. URL https://www.jstor.org/stable/30243645.

Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence , 297:103500, 2021. URL https://www.sciencedirect.com/ science/article/am/pii/S0004370221000515.

James Arvo and David Kirk. Particle transport and image synthesis. In Proceedings of the 17th annual conference on Computer graphics and interactive techniques , pages 63-66, 1990. URL https://dl.acm.org/doi/10.1145/97880.97886.

Chris L Baker, Noah D Goodman, and Joshua B Tenenbaum. Theory-based social goal inference. In Proceedings of the thirtieth annual conference of the cognitive science society , pages 14471452. Cognitive Science Society Austin, TX, 2008. URL https://cocolab.stanford.edu/papers/ BakerEtAl2008-Cogsci.pdf.

Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition , 113(3):329-349, 2009. URL https://www.sciencedirect.com/science/article/pii/ S0010027709001607.

Chris L Baker, Julian Jara-Ettinger, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour , 1(4): 0064, 2017. URL https://www.nature.com/articles/s41562-017-0064.

Leland Lavele Carter and Edmond Darrell Cashwell. Particle-transport simulation with the monte carlo method. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM (United States), 1975.

Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Designing perceptual puzzles by differentiating probabilistic programs. In ACM SIGGRAPH 2022 Conference Proceedings , pages 1-9, 2022.

Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Acting as inverse inverse planning. In Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings (SIGGRAPH '23 Conference Proceedings) , aug 2023a. doi: 10.1145/ 3588432.3591510.

Kartik Chandra, Tzu-Mao Li, Joshua Tenenbaum, and Jonathan Ragan-Kelley. Storytelling as inverse inverse planning. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 45, 2023b.

Eugene Charniak and Robert P Goldman. A bayesian model of plan recognition. Artificial Intelligence , 64(1):53-79, 1993. URL https://www.researchgate.net/profile/Robert-Goldman-5/publication/ 222330914_A_Bayesian_model_of_plan_recognition/links/5c7d77d0299bf1268d390c71/ A-Bayesian-model-of-plan-recognition.pdf.

Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: a general-purpose probabilistic programming system with programmable inference. In Proceedings of the 40th acm sigplan conference on programming language design and implementation , pages 221-236, 2019.

Daniel C Dennett. The intentional stance . MIT press, 1989.

Anca D Dragan, Kenton CT Lee, and Siddhartha S Srinivasa. Legibility and predictability of robot motion. In 2013 8th ACM/IEEE International Conference on Human-Robot Interaction (HRI) , pages 301-308. IEEE, 2013. URL https://ieeexplore.ieee.org/iel7/6476064/6483487/06483603. pdf.

William T Freeman. The generic viewpoint assumption in a framework for visual perception. Nature , 368(6471):542-545, 1994. URL https://www.nature.com/articles/368542a0.pdf.

György Gergely and Gergely Csibra. Teleological reasoning in infancy: The naıve theory of rational action. Trends in cognitive sciences , 7(7):287-292, 2003.

Tobias Gerstenberg, Matthew F Peterson, Noah D Goodman, David A Lagnado, and Joshua B Tenenbaum. Eye-tracking causality. Psychological science , 28(12):1731-1744, 2017.

Tobias Gerstenberg, Max Siegel, and Joshua Tenenbaum. What happened? reconstructing the past through vision and sound. 2021. URL https://psyarxiv.com/tfjdk/.

Thomas L Griffiths and Joshua B Tenenbaum. Optimal predictions in everyday cognition. Psychological science , 17(9):767-773, 2006. URL https://journals.sagepub.com/doi/pdf/10.1111/j. 1467-9280.2006.01780.x.

J Kiley Hamlin, Karen Wynn, and Paul Bloom. Social evaluation by preverbal infants. Nature , 450 (7169):557-559, 2007.

Fritz Heider and Marianne Simmel. An experimental study of apparent behavior. The American journal of psychology , 57(2):243-259, 1944.

Colin Jacobs, Michael Lopez-Brau, and Julian Jara-Ettinger. What happened here? children integrate physical reasoning to infer actions from indirect evidence. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society , volume 43, 2021.

Julian Jara-Ettinger. Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences , 29:105-110, 2019.

Julian Jara-Ettinger, Hyowon Gweon, Joshua B Tenenbaum, and Laura E Schulz. Children's understanding of the costs and rewards underlying rational action. Cognition , 140:14-23, 2015.

Julian Jara-Ettinger, Hyowon Gweon, Laura E Schulz, and Joshua B Tenenbaum. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences , 20(8):589-604, 2016. URL https://www.sciencedirect.com/science/article/pii/ S1364661316300535.

James T Kajiya. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques , pages 143-150, 1986. URL https://dl.acm.org/doi/pdf/10. 1145/15922.15902.

Max Kleiman-Weiner, Mark K Ho, Joseph L Austerweil, Michael L Littman, and Joshua B Tenenbaum. Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In CogSci , 2016.

Eric P Lafortune and Yves Willems. Bi-directional path tracing. In Compugraphics' 93 , pages 145-153, 1993. URL https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi= c480a06ab16aa18673cc872f811e5177a74fb790.

David Lindner, Rohin Shah, Pieter Abbeel, and Anca Dragan. Learning what to do by simulating the past. In International Conference on Learning Representations , 2020.

Michael Lopez-Brau and Julian Jara-Ettinger. Physical pragmatics: Inferring the social meaning of objects. 2020. URL https://psyarxiv.com/mnf4y/.

Michael Lopez-Brau, Joseph Kwon, and Julian Jara-Ettinger. Mental state inference from indirect evidence through bayesian event reconstruction. In CogSci , 2020. URL https://cognitivesciencesociety. org/cogsci20/papers/0085/0085.pdf.

Michael Lopez-Brau, Joseph Kwon, and Julian Jara-Ettinger. Social inferences from physical evidence via bayesian event reconstruction. Journal of Experimental Psychology: General , 2022. URL https://psyarxiv.com/4zu9n.

David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT press, 1982.

Scott McCloud. Understanding comics: The invisible art. Northampton, Mass , 7:4, 1993.

Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B Tenenbaum. Phase: Physically-grounded abstract social events for machine social perception. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 845-853, 2021. URL https://ojs.aaai.org/index.php/AAAI/article/view/16167/15974.

Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml , volume 1, page 2, 2000. URL https://www.eecs.harvard.edu/cs286r/courses/spring06/papers/ngruss_irl00. pdf.

Desmond C Ong, Jamil Zaki, and Noah D Goodman. Affective cognition: Exploring lay theories of emotion. Cognition , 143:141-162, 2015.

Stefan Palan and Christian Schitter. Prolific. ac-a subject pool for online experiments. Journal of Behavioral and Experimental Finance , 17:22-27, 2018.

Vivian C Paulun, Takahiro Kawabe, Shin'ya Nishida, and Roland W Fleming. Seeing liquids from static snapshots. Vision research , 115:163-174, 2015.

Madeline Pelz, Laura Schulz, and Julian Jara-Ettinger. The signature of all things: Children infer knowledge states from static images. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society , 2020. URL https://psyarxiv.com/f692k/.

Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically based rendering: From theory to implementation . Morgan Kaufmann, 2016. URL https://pbr-book.org/3ed-2018/Monte_Carlo_ Integration.

Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.

Miquel Ramırez and Hector Geffner. Plan recognition as planning. In Proceedings of the 21st international joint conference on Artifical intelligence. Morgan Kaufmann Publishers Inc , pages 1778-1783. Citeseer, 2009. URL https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf& doi=0de92a30c8a92a693705b53eaf602bbf258b4f39.

Miquel Ramırez and Hector Geffner. Probabilistic plan recognition using off-the-shelf classical planners. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI 2010) , pages 1121-1126. Citeseer, 2010. URL https://cdn.aaai.org/ojs/7745/ 7745-13-11274-1-2-20201228.pdf.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. Preferences implicit in the state of the world. International Conference on Learning Representations (ICLR) , 2019. URL https://arxiv.org/pdf/1902.04198.

Kevin Smith and Edward Vul. Looking forwards and backwards: Similarities and differences in prediction and retrodiction. In Proceedings of the Annual Meeting of the Cognitive Science Society , volume 36, 2014.

Shirin Sohrabi, Anton V Riabov, and Octavian Udrea. Plan recognition as planning revisited. In IJCAI , pages 3258-3264. New York, NY, 2016. URL http://www.cs.toronto.edu/~shirin/ Sohrabi-IJCAI-16.pdf.

Victoria Southgate and Gergely Csibra. Inferring the outcome of an ongoing novel action at 13 months. Developmental psychology , 45(6):1794, 2009.

Joshua Tenenbaum. Bayesian modeling of human concept learning. Advances in neural information processing systems , 11, 1998. URL https://proceedings.neurips.cc/paper/1998/file/ d010396ca8abf6ead8cacc2c2f2f26c7-Paper.pdf.

Joshua Tenenbaum. Rules and similarity in concept learning. Advances in neural information processing systems , 12, 1999. URL https://proceedings.neurips.cc/paper/1999/file/ 86d7c8a08b4aaa1bc7c599473f5dddda-Paper.pdf.

Christopher D Twigg and Doug L James. Backward steps in rigid body simulation. In ACM SIGGRAPH 2008 papers , pages 1-10. 2008. URL https://dl.acm.org/doi/pdf/10.1145/1399504. 1360624.

Tomer Ullman, Chris Baker, Owen Macindoe, Owain Evans, Noah Goodman, and Joshua Tenenbaum. Help or hinder: Bayesian models of social goal inference. Advances in neural information processing systems , 22, 2009.

Eric Veach. Robust Monte Carlo methods for light transport simulation . PhD thesis. Stanford University, 1998. URL https://www.proquest.com/docview/304456010?pq-origsite=gscholar& fromopenview=true.

Eric Veach and Leonidas Guibas. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques , pages 145-167. Springer, 1995. URL https://citeseerx.ist.psu.edu/document? repid=rep1&type=pdf&doi=3c9921cf30576edffd4d611e8f5d05e94421b57d.

Eric Veach and Leonidas J Guibas. Metropolis light transport. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques , pages 65-76, 1997.

Edward Vul, Noah Goodman, Thomas L Griffiths, and Joshua B Tenenbaum. One and done? optimal decisions from very few samples. Cognitive science , 38(4):599-637, 2014. URL https: //onlinelibrary.wiley.com/doi/pdfdirect/10.1111/cogs.12101.

Sarah A Wu, Rose E Wang, James A Evans, Joshua B Tenenbaum, David C Parkes, and Max Kleiman-Weiner. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science , 13(2):414-432, 2021.

Ilker Yildirim, Tobias Gerstenberg, Basil Saeed, Marc Toussaint, and Josh Tenenbaum. Physical problem solving: Joint planning with symbolic, geometric, and dynamic constraints. arXiv preprint arXiv:1707.08212 , 2017.

Ilker Yildirim, Basil Saeed, Grace Bennett-Pierre, Tobias Gerstenberg, Joshua Tenenbaum, and Hyowon Gweon. Explaining intuitive difficulty judgments by modeling physical effort and risk. arXiv preprint arXiv:1905.04445 , 2019.

Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly rational planning agents. Advances in neural information processing systems , 33:19238-19250, 2020. URL https://proceedings.neurips.cc/paper/2020/file/ df3aebc649f9e3b674eeb790a4da224e-Paper.pdf.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

A Experimental design

For each of our experiments reported in Section 3.3, we recruited N = 200 participants from Prolific [Palan and Schitter, 2018]. Participants were paid $15 per hour ($1.25 total for blocks and grid domains, and $2.00 for the keys domain), and our experiments were conducted with IRB approval.

Participants were first familiarized with the environment, through both text instructions and a sample video of an agent performing the task in the domain. Then, they were told that their objective was to infer the agent's goal from a single snapshot. They answered several questions to check their comprehension of both the domain and the task they were asked to perform, and were not allowed to continue unless they answered the comprehension questions correctly. The full experimental design is available in HTML format in the supplementary materials. No data was excluded from our analyses.

B Numerical test of correctness

Programming sophisticated importance sampling routines is a challenging and bug-prone engineering effort [Cusumano-Towner et al., 2019, Anderson et al., 2017, Pharr et al., 2016]. To test that our algorithm is unbiased, i.e. that it produces correct likelihoods in expectation, we compared likelihoods computed by rejection sampling and our sampler using converged estimates (25,000 samples each). For this experiment we used a uniform 4 × 4 grid-world, with the prior on start states being uniform along the first row ( x = 0 ) and the goal being the far corner (3 , 3) . The results of this experiment are shown in Figure 4. Our estimator has a dramatically different implementation than rejection sampling (compare Algorithms 1 and 2). However, the computed likelihoods are indistinguishable at every cell in the grid, even in 'corner-case' cells such as the goal cell itself. This provides a strong check that our algorithm and its implementation are both indeed correct.

C Additional domains

We used our algorithm to perform inferences in three additional domains. The purpose of these domains is to show the remarkable flexibility of our method: how it can make interesting inferences in a wide variety of settings. Though we did not collect human subject data for these domains, we show results for cases where the inference task is relatively straightforward.

Our sampler is unbiased and precisely matches rejection sampling

(0, 0)

(0, 1)

(0, 2)

(0, 3)

(1, 0)

(1, 1)

(1, 2)

(1, 3)

(2, 0)

(2, 1)

(2, 2)

(2, 3)

(3, 0)

(3, 1)

(3, 2)

(3, 3)

Observed location x

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Likelihood p(x | g=(3, 3))

Ours

Ours + BDPT

Rejection

0.00

0.05

0.10

0.15

0.20

Likelihood (rejection)

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

Likelihood (ours)

Ours

Ours + BDPT

K=L=M

K<L=M

L<K=M

M<K=L

L=M<K

K=M<L

Desire

K=L<M

K<L<M

K<M<L

L<K<M

L<M<K

M<K<L

M<L<K

Currentbelief

C.1 Food trucks (joint belief/desire inference)

The food trucks domain, taken from the cognitive science literature [Baker et al., 2017], is a Partially Observable Markov Decision Process (POMDP). It consists of a 5 × 10 gridworld with an opaque wall in the middle. A hungry graduate student wakes up at home (one side of the wall) and wishes to eat at a food truck. There are two parking spots where food trucks usually park, and three kinds of food trucks that could be parked at each of those spots: Korean, Lebanese, and Mexican (K, L, and M). The graduate student might have preferences among the cuisines, but might also be uncertain about which trucks are parked at each spot today. Thus, they might engage in information-seeking behavior by looking behind the opaque wall, and then choosing a food truck to walk to based on their preferences. The inference task is to determine (a) the student's preferences over food trucks, and (b) the student's (current) belief state about which truck is at each parking spot.

Using this domain, Baker et al.'s inverse planning model was able to jointly infer the student's beliefs and desires from an observed trajectory; those inferences closely matched responses from human subjects. Here, we perform the same type of inference, but from a single observed snapshot.

For example, in the example in Figure 5, the student is observed moving south next to the wall. A Korean food truck is parked in the southwest parking spot, and a Lebanese food truck is parked in the northeast spot. Seeing this scene, a reasonable inference is that the student went looking around the wall to see if the Mexican food truck (their favorite) was parked on the other side. Seeing that it was Lebanese food instead, the student turns around and makes peace with the nearby Korean food. Indeed, our model captures this inference: in the joint posterior distribution over both beliefs and desires, our model is confident that the student now knows that the northeast truck has Lebanese food, and furthermore that the student's favorite food is Mexican.

A more sophisticated inference emerges if the student is observed moving north instead of south (Figure 6). Now, a reasonable inference is that the student dislikes Korean food, and is going around the wall to check what is at the other truck. The model captures this: it favors the hypothesis that the student is unsure what is at the northeast truck, and also places high weight on Korean being the least favorite food option.

However, as is visible on the right half of the heatmap, the model also places some weight on the possibility that the student knows that there is Lebanese food and prefers it, or that the student (mistakenly) believes there is Mexican food and prefers that.

K=L=M

K<L=M

L<K=M

M<K=L

L=M<K

K=M<L

Desire

K=L<M

K<L<M

K<M<L

L<K<M

L<M<K

M<K<L

M<L<K

Currentbelief

Top

left

Bottom

left

Top

right

Bottom

right

Goal

0.0

0.2

0.4

0.6

0.8

1.0

Model prediction

Blue agent is thief

(Pink agent is guard)

Pink agent is thief

(Blue agent is guard)

C.2 Heist (multi-agent domain)

In this multi-agent domain inspired by classic stimuli in cognitive science [Baker et al., 2008, Southgate and Csibra, 2009, Heider and Simmel, 1944], two agents-blue and pink-occupy a 7 × 7 gridworld representing an art museum. One of the agents is a 'thief,' whose objective is to escape the museum by reaching the exit, and the other is a 'guard,' whose objective is to catch the thief. There are four doors in the room, only one of which is an exit, and the rest of which are dead ends. Both agents know which door is the exit, but this information is not visible to the observer (all doors are rendered identically). The inference tasks are to look at a snapshot of the two agents and jointly infer (a) which agent is the thief and which is the guard, and (b) which door is the exit.

In the example in Figure 7, it is clear from the snapshot that the blue agent is the guard and is chasing the pink agent, the thief, to the bottom-right corner. The model reproduces this inference, though also acknowledges the possibility that the thief might actually be heading onward past the bottom-right, to the bottom-left corner instead.

Top

left

Bottom

left

Top

right

Bottom

right

Goal

0.0

0.2

0.4

0.6

0.8

1.0

Model prediction

Blue agent is thief

(Pink agent is guard)

Pink agent is thief

(Blue agent is guard)

Top

left

Bottom

left

Top

right

Bottom

right

Goal

0.0

0.2

0.4

0.6

0.8

1.0

Model prediction

Blue agent is thief

(Pink agent is guard)

Pink agent is thief

(Blue agent is guard)

国

The next two examples (Figure 8) are ambiguous cases: the two agents are in symmetric positions, so it is unclear who is who. Here, the model can determine with high confidence where the exit is, but remains uncertain about who is the thief and who is the guard.

Finally, in the last example (Figure 9), it is unclear whether a blue guard is blocking a pink thief from heading to the top-right corner, or whether a pink guard is blocking the blue thief from heading the the bottom-right corner. Indeed, the model reproduces this ambiguity.

C.3 Cart-pole (continuous state space with physical dynamics)

The cart-pole domain is a classic problem in reinforcement learning and optimal control. The goal is to balance a pole in an upright position, by moving the cart left or right. The state space of this

Top

left

Bottom

left

Top

right

Bottom

right

Goal

0.0

0.2

0.4

0.6

0.8

1.0

Model prediction

Blue agent is thief

(Pink agent is guard)

Pink agent is thief

(Blue agent is guard)

domain consists of four continuous numbers: the horizontal position of the cart and its velocity, and the angle of the pole along with its angular velocity. The inference tasks are to look at a snapshot image-which only shows the cart position and the pole angle-and determine the velocity of the cart and the angular velocity of the pole. Note that rejection sampling cannot solve this task because the probability of a randomly-sampled trace passing through the observed state is zero.

We use an off-the-shelf pre-trained Proximal Policy Optimization (PPO) controller [Schulman et al., 2017] from stable-baselines3 [Raffin et al., 2019] to compute a probability distribution over actions. Inference in this domain is complicated by the fact that computing backward dynamics in physical simulation is challenging and often ill-posed. While previous work has proposed analytic approaches [Twigg and James, 2008], we instead train a neural network to approximate the reverse physical dynamics. We place a unit Gaussian prior over the velocities, and use a von Mises distribution as a prior over the initial pole angle. We infer the velocities of the system by sampling candidate pairs of cart and pole velocities (stratified in an 11 × 11 grid) and computing likelihoods using our algorithm.

The inferred posteriors are intuitive and track the relative stability of the position in each snapshot (Figure C.3). For example, in part (a), the pole has almost completely fallen over, and so our method infers that the pole has a large negative angular velocity, and is falling fast towards the ground. At the same time, it infers that the cart is moving fast to the left, because the controller is likely attempting to re-balance. In comparison, for part (f), the pole is nearly upright, so the model predicts that the pole is not rotating, and that the cart might be moving left or right to keep the pole balanced.

(a)

Angular velocity

5.0

(d)

Angular velocity

5.0

0.0

0.0

-5.0

-5.0

-2.0

0.0

2.0

-2.0

0.0

2.0

Cart velocity

Cart velocity

Angular velocity

5.0

Angular velocity

5.0

(b)

(e)

0.0

0.0

-5.0

-5.0

-2.0

0.0

2.0

-2.0

0.0

2.0

Cart velocity

Cart velocity

Angular velocity

5.0

Angular velocity

5.0

(c)

（f)

0.0

0.0

-5.0

-5.0

-2.0

0.0

2.0

-2.0

0.0

2.0

Cart velocity

Cart velocity

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an efficient Monte-carlo algorithm to infer goals from single snapshots. The problem the authors consider is the same as in work by Lopez-Brau (2020, 2022) but previous solutions are slow as they apply rejection sampling. In this work, the authors use the insight that one can sample a valid path by sampling the part of the path \"before\" the current snapshot and \"after\" the snapshot independently, decresing the computational complexity by a large amount. Further, A-star search is used to find likley paths. The authors validate their algorithm on different grid worlds and provide results of a human study indicating that inferences of their algorithm coincide with human judgements.\n\nSTRENGTHS:\nThe problem of inferring goals from a single snapshot is interesting, as a major limitation of most IRL methods is that they only work in a certain context (state/actions of the full trajectory).\n\nThe paper is clearly written and easy to understand.\n\nThe presented method is much more efficient than previous methods in terms of scalability.\n\nThe plausability of the results found by the algorithm is supported by human experiments.\n\nWEAKNESSES:\nQuantities are not clearly defined. Are states/actions/goals continuous or discrete? In the formulas, one integrates over states and sums over actions. It might be very helpful for the reader to formally define the MDP and its sets. Further, what are \"paths\" formally, what does the indexing operator \"[0]\" mean for paths?\n\nWhile the main idea of the algorithm to sample the path \"before\" and \"after\" is very simple, the overall algorithm seems quite hacky to me: The basic idea is combined with other tweaks such as roulette termination and A-star search to avoid unlikely paths. With these, the considered tasks can be efficiently solved but there is no particularly nice linking theory.\n\nIn my opinion, the contribution is not strong enough for a NeurIPS submission as the main (conceptual) difference to previously published algorithms is not that large. The essential difference is that parts of the path before and after are sampled, together with other tweaks.\n\nI would find it helpful if the algorithm clearly indicated which quanitity it computes (p(x | g)), this is not direcly visible.\n\nThe plots in Table 1 do not have a colorbar, therefore the precise meaning and interval of the color's values is not clear to me.\n\nQUESTIONS:\nThe regarded grid worlds all have a discrete state/action space. Is the algorithm applicable to the continuous domain?\n\nWhen introducing the algorithm, you write that the situation of the agent you consider is an MDP but you would relax this assumption later in the paper. Where do you do that?\n\nLIMITATIONS:\nLimitations are well discussed. To my understanding, only discrete state/action/goal spaces are considered. This point could be added.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers proposes a new approach to the inference problem of inferring an agent's goal state from information about a single state. This approach is based on the bidirectional Monte Carlo sampling of trajectory sequences, both from the agent's given state $x$ to the goal state $g$, and from an initial state $s$ to $x$. The authors show near convergent inference of the approach with 10 samples on two gridworld based environments, and a word game kind of environment when evaluated against human judgement.\n\nSTRENGTHS:\n- The evaluations show a strong \"near\" convergence result of only 10 samples, even for multi-stage planning. This indicates their sampling strategy is practically sample-efficient, and a huge improvement over prior approaches requiring more than 300,000 samples (reportedly).\n- The connection drawn from path tracing in computer graphics is a fresh insight, which the authors show applies well to the goal/previous states' inference problem, and can potentially have broader scope applications such a for spatial navigation tasks etc.\n- The use of human judgement (using 200 participants) for evaluation is also a very reasonable choice instead of hand-engineered reward functions designed by a single person/a few people.\n- The authors also show efficient sampling in a variety of other domains (in the appendix) which indicates it is more broadly applicable.\n\nWEAKNESSES:\n- _Experiments_: Something I might myself be unclear about, so let me know if that is the case-- there are no evaluations comparing the proposed model in this paper with the model shown by [Lopez-Brau et al 2022](https://psyarxiv.com/4zu9n), is this only because of sample-efficiency? It would be interesting to see how \"nearly\" convergent their proposed approach is in the cases where it can be applied.\n\n- _Theoretical analysis_: What is the complexity of the sampling with the size of the state space, the number of accessible states, the number of starting/goal states etc-- is it possible to perform some kind of theoretical analysis of this? \n\nI'll be very happy to increase my score on the addressal of these concerns/doubts.\n\nQUESTIONS:\n- Have you checked how the sampling scales with an increasing number of possible starting states, and an increasing number of possible goal states? It could potentially be interesting to see.\n- Why is incremental A-star search used for planning on-line?\n- I am wondering if its possible to emphasize more on the scope of the applications of the proposed sampling approach-- what other domains can it be used in, apart from tasks that look like spatial navigation? The word blocks is one good example, also the cartpole experiment. There are intuitive physics based experiments, or reasoning based tasks which involve inferring over a sequence of images say, can this approach be applied there?\n\nLIMITATIONS:\nThe authors adequately address the limitations of their approach, especially when it comes to sampling over goals, or scaling up to a larger goal space. There is just one minor point here-- the approach is based on a combination of well-founded insights and ideas, which when put together, work well, but there seems to be a lot of hand-engineering/domain-specific engineering in assembling these together, which I think should be acknowledged in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm for efficiently inferring the goal of an RL agent from just observing its current (single) state $x$. The method  improves substantially upon rejection sampling based prior work by 1) only sampling paths through $x$ by separating the path into past and future 2) sampling the past path backwards in time starting from $x$ and 3) using incremental A-star search for planning.\nThe efficacy of the approach is demonstrated in two gridworld domains and a word-blocks game (as well as three more in the supplementary). The posterior estimates over the goal of the agent qualitatively match human judgements, and converges several orders of magnitude faster than vanilla rejection sampling.\n\nSTRENGTHS:\n* The studied problem is interesting and potentially impactful for multi-agent RL or human-agent interactions.\n* The algorithmic modifications are well motivated and the efficiency improvement above simple rejection sampling is impressive. \n* The paper validated the results against human judgements and found a high correlation\n* The presentation is unusually clear, and a fun read.\n\nWEAKNESSES:\nThe quantitative evaluation in Table 3 assumes the ground-truth to be a converged estimate of the presented method. This leaves open the possibility that the TV is better than the rejection sampling baseline because the presented method is biased in some way. The concern is partially addressed with the numerical validation check in the supplementary section B, which in my opinion should at least be mentioned in the main paper. Another way to strengthen the results in Tab 3 would be to also compare to a \"ground-truth\" estimate from (very?) many vanilla rejection samples. If the presented method is still better at approximating this ground-truth with few samples than rejection sampling with few samples, that would be strong evidence.\n\nQUESTIONS:\n* I am unclear about the role of A* in the proposed algorithm (sec 2.3). I assume that $p(x\\rightarrow x') \\propto \\sum_a \\exp (\\beta(C(x \\rightarrow g) - C(x' \\rightarrow g)) $ takes the role of the A* heuristic. But in A* this heuristic is used to decide which node to further expand  next, and I cannot find the equivalent of this search process in Algorithm 2. Could you please clarify the correspondence of A* with the proposed algorithm.\n* I am unclear about the shaded cells in in Table 1. The description states that \"Shaded cells were excluded from the analysis because it would be irrational for the agent to be there for any goal\".  I assume \"shaded cells\" refers to the gray cells in the rows corresponding to the keys world. But the shading of cells doesn't make sense to me. In row 3 it seems to correspond to inaccessible areas (which make sense to exclude). In row 4 the bottom third is shaded except for a single spot next to the green key. Why would that spot be less irrational than the surrounding ones? In row 5 the leftmost column is shaded which also seems odd. Is it impossible for the agent to start there? Also why are no cells shaded in row 1? Please clarify.\n\nLIMITATIONS:\nThe authors discuss an important limitation of their work regarding the sampling over goals. \nAnother limitation that may be worth pointing out is that the separation of paths into independent paths for past and future depends on the environment to be Markovian (this *is* mentioned in the paper). More specifically that the current states $x$ contains all information connecting the past and the future. For more realistic environments this would for example have to include quantities like velocity or any relevant inner state of the agent in addition to the current position. Relaxing this assumption would require accounting for latent states and thus be equivalent to observing a distributions over current states $x$ instead of a concrete state.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the problem of inferring the goal state $g$ of an agent given only a single state $x$ in a trajectory, i.e., inferring $p(g|x)$.\nFor this, the authors claim that we need to integrate all possible initial states, and thus, sampling past trajectories is necessary for Monte Carlo estimation.\nThey propose several techniques to improve the sampling efficiency, and the main technique is to sample the past trajectory in reverse, starting from the given state $x$.\n\nSTRENGTHS:\nThe key intuition and motivation are clearly described.\n\nWEAKNESSES:\n### The key assumptions are not clearly stated\n\nI think the main text does not explain the key assumptions, such as\n- What is given? A fully-trained agent? A fully transparent environment with perfectly known dynamics?\n- How can we perform the past and future sampling?\n\n### Soundness of the main argument\n\nMost importantly, I am not convinced that explicitly sampling the past trajectory is necessary to infer $p(g|x)$.\nSince $x$ contains all the information that we have, sampling the past trajectory seems like an unnecessary complication.\nI think we can train a predictor that produces $p(g|x)$ directly from a decent amount of offline trajectories.\n\nMoreover, I believe there is a serious mistake in Eq.(2).\nThe left hand side of Eq.(2) is $p(x|g)$, but the right hand side assumes that every path passes through $x$ and does not consider the cases where $x$ is not included in the path.\nAs the authors noted in L81-82, most paths would not pass through $x$, so there should be a term that account for this fact.\n\n### Limited scope of the experiments\n\nThe experiments are limited to simple toy problems like grid world.\n\nQUESTIONS:\n- Please state the key assumptions\n- Can you prove that sampling the past trajectory is absolutely necessary to predict $p(g|x)$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}