What do you know? Bayesian knowledge inference for navigating agents
Authors: Matthias Schultheis
Jana-Sophie Schönfeld
Constantin A. Rothkopf
Heinz Koeppl
Venue: NeurIPS 2025 poster ()
Keywords: Bayesian reinforcement learning
computational cognitive science
human behavior
inverse reinforcement learning
theory of mind
Primary area: neuroscience_and_cognitive_science

Abstract: Human behavior is characterized by continuous learning to reduce uncertainties about the world in pursuit of goals. When trying to understand such behavior from observations, it is essential to account for this adaptive nature and reason about the uncertainties that may have led to seemingly suboptimal decisions. Nevertheless, most inverse approaches to sequential decision-making focus on inferring cost functions underlying stationary behavior or are limited to low-dimensional tasks. In this paper, we address this gap by considering the problem of inferring an agent's knowledge or awareness about the environment based on a given trajectory. We assume that the agent aims to reach a goal in an environment they only partially know, and integrates new information into their plan as they act. We propose a Bayesian approach to infer their latent knowledge state, leveraging an approximate navigation model that optimistically incorporates partial information while accounting for uncertainty. By combining sample-based Bayesian inference with dynamic graph algorithms, we achieve an efficient method for computing posterior beliefs about the agent's knowledge. Empirical validation using simulated behavioral data and human data from an online experiment demonstrates that our model effectively captures human navigation under uncertainty and reveals interpretable insights into their environmental knowledge.

What do you know?
Bayesian knowledge inference for navigating agents
Matthias Schultheis
Jana-Sophie Schönfeld
Center for Cognitive Science
Center for Cognitive Science Technische Universität Darmstadt
Technische Universität Darmstadt
Darmstadt, Germany
Darmstadt, Germany
matthias.. seriiiis..chullhhheo Schulthees@tuu dardstadd.d.ad.....m.mm
aaaasophie.cchonfelllo.. tu-darmstadt.co
Constantin A. Rothkopf
Heinz Koeppl
Center for Cognitive Science
Center for Cognitive Science
Technische Universität Darmstadt
Technische Universität Darmstadt
Darmstadt, Germany
Darmstadt, Germany
constantin, rothhortttohhhooooooeooooo citeuuddrmmtaate
heinz wwwpkkkootptdddrastttttaaee
Abstract
Human behavior is characterized by continuous learning to reduce uncertainties about the world in pursuit of goals. When trying to understand such behavior from observations, is essential to account for this adaptive nature and reason about the uncertainties that may have led to seemingly suboptimal decisions. Nevertheless, most inverse approaches to sequential decision-making focus on inferring cost functions underlying stationary behavior or are limited to oww-dimessional tasks. this paper, we address this gap by considering the problem inferring an agent's s knowledge awareness about the environment based given trajectory. We assume that the agent aims to reach a goal an environment they only partially know, and integrates Bayesian approach to infer their latent knowledge state, leveraging an approximate navigation model that optimistically incorporates partial information while account- ing for uncertainty. By combining sample-based Bayesian inference with dynamic graph algorithms,
information into their plan as they act. We propose
achieve efficient method for computing posterior beliefs about the agent's 's knowledge. Empirical validation using simulated behavioral data and human data from an online experiment demonstrates that our model effectively captures human navigation under uncertainty and reveals interpretable insights into their environmental knowledge.
Introduction
As humans, navigate world full of uncertainty. Whether walking through city, browsing the internet, or finding the way to the airport, we constantly plan under partial knowledge, balance risks with opportunities, and revise our strategies between action and learning is a hallmark of adaptive, intelligent behavior.
gather new information. This interplay
Consider the following scenario: you driving to the airport using your usual route, which you know typically the fastest. Midway through, you encounter unexpected construction blocking the road. You turn around and take longer detour. To an external observer unaware of your prior knowledge, your behavior might appear irrational or poaalinncogruentt- of heading directly to the airport? However, by reasoning about what you likely knew when the decision was made, a more accurate and interpretable explanation emerges: you simply did not
detour into a neighborhood instead
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

know about the construction when you started, and your actions reflect effective planning under that partial knowledge. This example highlights fundamental challenge in understanding and modeling human behavior: observed decisions are shaped not only by goals and preferences, but also by what they know-and don't 't know. Nevertheless, most existing inverse sequential decision-making approaches focus on recovering cost functions representing the agent's 's objectives 2 or assume stationary policies with complete knowledge of the environment structure [3] 4 More general formulations exist 15 6). but their computational cost limits them to very low-dimensional domains. These assumptions complicate the applicability to learning agents in general and human behavior in natural tasks in particular, which are characterized by uncertainty, learning, and adaptation.
In this work, we address the problem of inferring an agent's 's latent knowledge state in a partially known environment based on observed decisions. We focus on navigation tasks where agents act toward a goal, while simultaneously integrating new information about the environment's structure into their plans. From an observed trajectory, our goal is to infer which parts of the world the agent knew beforehand and which were newly incorporated while acting. We assume that, as researchers, we have complete knowledge of the task and environment, but must reason about the subject's latent knowledge state. Although we frame our approach in terms of knowledge of the environment, the same formulation applies to reasoning about agent's 's memory or awareness during planning, enabling us, for example, to infer what information subjects transferred between tasks, which parts of the environment were represented in memory, and how awareness influenced planning.
The key challenge of this problem lies in the complexity of reasoning over a large space of possible knowledge states and their corresponding optimal policies. To address this, we combine sample-hased Bayesian inference with a tractable navigation model hased on dynamic graph algorithms to efficiently compute posterior beliefs over the agent's 's knowledge state. We evaluate our approach using two simulated tasks and behavioral data from an online human experiment. Our results demonstrate that the model successfully captures human navigation behavior under incomplete information and provides reasonable beliefs of the subjects' prior knowledge in this task, improving in many cases over both an agnostic and heuristic baseline.
1.1 Related work
The problem of inferring goals, beliefs, and knowledge when observing trajectories of agents was approached with Bayesian inverse planning [5] 6 in environments with up to 6 belief states. While this method could in principle be applied to our setting of inferring the knowledge of navigating agents, the exact computation of both the posterior belief and the underlying policy is individually intractable in the high-dimensionn
belief space we consider in our tasks (up to 2 169 belief states). Other work on Bayesian inverse planning used online path planning 7 and adaptive shortest-path algorithms for varying goal states (8 9.. Rabinowitz et al. [10] introduced with ToMnet a general approach to learn representations of an agent's mind that can used to predict future behavior,
lacking an explicit representation of the agent's knowledge.
In the field of reinforcement learning, inverse approaches for sequential decision-making mostly focus on learning an agent's goals in the form of a reward function and fall under the term inverse reinforcement learning (IRL) [ 2 [I] Some of these works also consider the setting of spatial navigation [12] There is also work to infer internal dynamics models [3]115 and construals [I6] of observed agents. Some work on IRL specifically addresses the scenario where agents have uncertainties by regarding POMDPs [7 4 and can additionally infer variabilities and subjective beliefs [8] 19]. Further, inferring reward functions of learning agents was considered in 200 21 and inferring their learning rules in 22 There is also work on inferring an agent's ' belief about an unknown discount parameter [23 and work on belief inference under incomplete model knowledge for observed fully informed agents (24). Ashwood et al. (25] considered the problem of estimating reward functions that evolve randomly over time, which shares similarities to our work, as it also deals with nigh-dimensional inference with non-stationary policies. They deal with the problem by using an expectation mwwiioommmmimiiiiiiioin scheme and separate the stationary state- dete-de reward component from the time-dependen goal map process. In our case, however, state and time not separable: the knowledge at a certain time depends on the history of visited states.
For acting under uncertainty, the tradeoff between maximizing reward and gathering information is known as the exploration-explootatii tradeoff (26). The exploration behavior of humans has been mostly studied in bandit tasks, where actions do not influence the state of the environment (27-30)
2

---

In this regard, there is work considering restless bandit tasks B1,, spatially correlated bandits (29]. and contextual bandits [2]. It was also shown that humans can adapt their exploration behavior Research on how humans form representations for planning and acting when learning the dynamics of the environment was regarded for non-spatial [35] and spatial navigation tasks [3).. Humans' ability to adapt their planning strategy depending on the structure of the problem was investigated in [37]. The variability of humans' planning has
to the structure of the environment 33
34
been modeled using shortest path algorithms in a softmax policy [38 8 39] Our planning model is conceptually related to the Explicit Explore or Exploit (E²) algorithm 44), which similarly separates
known and unknown states. However, while E² treats unknown states maximally rewarding to encourage exploration, our model assigns a traversal probability for uncertainty, leading instead to uncertainty aversion. Inferring the goals and plans of artificial agents has been regarded in the field of plan recognition [41,43]. Furthermore, our method can be interpreted as learning properties about the given graph (i.e., the knowledge variables) and is therefore related to graph learning [44), which aims to reconstruct a graph from data. Standard graph learning approaches, however, typically do not capture subjective, temporally updated knowledge states.
2 Background
2.1 Markov decision processes (MDPs) and shortest paths
For modeling decision-making we consider deterministic finite Markov decision processes (MDPs) (26) of the form (S.A, where S = {s....., ]N} and A = {ao..... aad } are the finite state and action space, respectively. T S x A + denotes the state transition function and C: S x A + R is the cost function. If all components of the MDP are known, standard offline reinforcement learning (RI.) algorithms for MDP planning can be used to learn the optimal value function to find policy IT: S x A + [0 minimizing the long-term cost.
Deterministic finite MDPs can be equivalently modeled as graphs G = -V,E), where the nodes
V = S represent states and the edges 3 = {s,,s) | 3a E A T(s,a))) = 5'} C V x V denote possible transition between them 45 46 An equivalent cost function C E + R specifies the cost for each transition. The optimal policy for navigating between two states can be obtained by solving the shortest path problem. This problem consists of finding the shortest path between the two nodes such that the sum of the visited edge costs is minimized [47]. By applying the dynamic programming principle, the single-source shortest path problem for non-negative edge costs can be efficiently solved using Dijkstra's 's algorithm 48). The * algorithm uses a heuristic to guide the search to increase efficiency. D* lite [49] is dynamic version of *+, which can efficiently update the shortest path solution if costs of the graph change.
2.2 Bayes-adaptive MDPs
In layes-adaptive MDPs (BAMDPs), the goal is to solve an MDP while being uncertain about its transition kernel T (0 During interaction, the belief ò & about the unknown transition function parameters E θ is updated using the observed information I about the transition via Bayes rule, PII 0) P().. A BAMDP
be formally defined via state augmentation as an MDP (S', "",,A,',C") where S = S x & is the set of hyperstates, capturing jointly the state as well as the belief about 0. The augmented transition function T' S' A + S' specifies how the state and belief jointly evolve. The quantities A and C" coincide with the underlying MDP. In principle, the augmented MDP specifies the optimal policy 11, but its computation is intractable except for a few special cases [51 52 Approximate methods have been proposed [53], such as BEETLE [54]. which approximates the value function of the augmented MDP but remains tractable only for now-dimensional problems (in the original work, MDPs with up to nine states). Commonly, scalable approximations either modify the cost function to visit states that are unknown (exploration bonus) (55 57 or act optimistically in the face of uncertainty, e.g., by applying Thompson sampling For shortest path problems in the robotic domain, was
P(O I)) x
58 59, or combinations of both (6) 61
suggested to act purely optimistically
49
3

---

3 Method and model
We propose an approximate probabilistic inference method to estimate an agent's latent knowledge about transitions in the world based on single observed navigation trajectory. Specifically, we model the agent's ss knowledge in a binary form using a vector k = (k1.... kx). where each k; E {0, 1 indicates whether state in the MDP is known (A; 1) or unknown (k. = 0) to the agent. If a state 8 is known, we assume the agent has access to its transitions, and denote the set of all these known k V k;} S E. For unknown states, the agent instead considers a C VVxX V. with potential costs CPot g -> I. which are believed
transitions by 8 = {(+,; 8,) set of potential transitions got to exist with probability q. The agent's knowledge evolves over time: it begins with the initial knowledge vector k, and states become known as soon as they have been visited. This knowledge formulation can be easily extended, for example, model agents that acquire knowledge of nearby states within a certain raaduus, or existence probabilities depending on the structure of the environment. In the following, we formalize the knowledge inference problem, i.c., to reason about a latent k, and describe how compute sample-hased solution given likelihood and prior model with a single evaluation per sample. Then, we show how to incrementally compute the likelihood and the prior by leveraging approximate solution the agent's planning problem. An implementation of our algorithm is publicly available under the MIT Licence] For the planning algorithm, we adapted an implementation of D* lite, also available under the MIT License.
3.1 Bayesian knowledge inference
For reasoning about the agent's 's latent knowledge k = (kI,..., kN) about the world, we model our belief about &; using a corresponding Bernoulli random variable K; Bernoull(pp.). and define K = (K1,..,, Kv). Given an observed trajectory D = (s²,...., ,,"). our goal is to infer the posterior distribution P(K) D). Bayes' theorem gives P(K D) P(D K) P(K). showing that with a likelihood and prior model, the posterior can be determined. Exact inference is intractable, as
computing the normalization factor requires summing over all 2 possible values of K, which scales exponentially in the number of states N.
To approximate the posterior, we employ Gibbs sampling [62] At each iteration, we sample single variable K; from P(K; K_; D), where K-; denotes the vector of all knowledge variables except K,, and K-; takes values of the previous sample k For Bernoulli K,, each new sample = (k....., kN) either remains equal to the previous sample weeev or differs by single bit flip at index i. The probability for bit flip is determined by the full conditional distribution
K K., D) = (1 P(D K, = k. K., P(K, = - K
P(D K, = L.eey K. k
= k prev
-
P(K; = -h P.ov
see Section A. for the derivation. This formulation is computationally efficient: To produce sample, we only need to evaluate the prior ratio and single likelihood P(D K; = -keeey K K))), since the likelihood of the past sample, P(D K, = R.pee K_;), was already computed in the previous iteration. However, even with this formulation, the inference procedure remains challenging: Each likelihood evaluation involves solving a planning problem under partial knowledge, corresponding to a BAMDP, which is commonly intractable. In Section 3.2, we present tractable approximation of the likelihood, which can be efficiently computed for the derived Gibbs update in an incremental way. In Section 3.3 we describe a prior model suitable to navigation tasks. The complete algorithm for the knowledge inference procedure is given in Algorithm]
3.2 Planning model and likelihood
The likelihood of trajectory given the agent's initial knowledge is given by P(D (8 (8?....,57) K) = -'= g(tt), where के : S S + [0, 1] denotes the agent's policy representing the probability to transition from state g to 3++1 at time step f. For fully informed agents in deterministic finite MDPs, the optimal policy is to follow deterministically the shortest path to the goal 45). To model stochastic decision-making of planning humans, prior work has modeled
-ow.-oeoooel---i-eeeeeeeee inference

---

Algorithm/ 1 Bayesian knowledge inference
Output: Samples [k'...., k"} from the distribution P D)
Input: Number of samples L, Partial world model W, Trajectory D
1: Initialize k" with N random boolean elements
2: Compute likelihood of k" using Algorithm
3: for i in (1..... ....,,,, [.] do
4: Select from (1......N N) randomly
5: Initialize w-wtchh + k'
6:
Switch the value kewwich
7:
Compute likelihood for wiich by updating the past shortest path result (Algorithm 2
8:
Compute prior fraction for kwwitch using Eq. (3)
9:
Compute acceptance probability using Eq. T
10: With probability p. set k' <- wwiichh otherwise k <- k!
11:
end for
agents using softmax policies of moving from state to s' if there is a corresponding edge as
39]. In partieular, Chandra et al. [39] model the probability
(s, x exp (s)), with Vy(s') C(s C(ss.g))
where C(s.g) g) denotes the cost of the shortest path from to goal which is computed using the A# algorithm [63] and (s') describes the reduction of in shortest path length towards g. To extend this model to our problem setting, we face two key challenges: First, the agent has only partial knowledge of the environment, turning the problem into a BAMDP. Exact planning this setting is in most cases infeasible, as the number of belief states grows exponentially with the number of unknown states. Second, the Gibbs sampling procedure from Section 3.1 requires evaluating a likelihood for each sample draw. Since the agent's ' knowledge evolves while acting, the policy must be recomputed at every time step with the updated knowledge, requiring a large number of policy computations.
We address the first problem by extending the stochastic policy model from Eq. (8) to approximate planning under partial knowledge: Instead of computing the shortest path under full knowledge, we model the agent to plan based on expected costs under probabilistic transitions reflecting their uncertainty. Specifically, we assume that uncertain transitions succeed with probability q, leading to an expected cost of (s, for a transition from to s', while known transitions retain their true cost C(s, s'). This approximation can be interpreted as planning optimistically with a penalization for uncertainty that is proportional to the risk of failure. Importantly. it assumes that planning is based on the current belief only, ignoring effects of knowledge acquisition along the path. We use these expected costs for the softmax policy in Eq. ( and assume that the agent replans at each time step based on its updated knowledge.
To mitigate the second challenge, the excessive recomputation of policies, we observe that the knowledge vector k is updated only one position both between sampling iterations and trajectory steps. This leads to a modification of only a relatively small number of graph edges in the graph. We take advantage of this by employing a dynamic shortest path algorithm. specifically the D* lite algorithm [49] instead of A+, This enables the reuse of past intermediate results and reduces the computational cost for policy computations significantly. Section A.2 provides more detailed derivation of the likelihood and policy model, and presents the full algorithm for incrementally computing the likelihood (Algorithm)
3.3 Spatial prior
For the prior model, we make the plausible assumption that a subject is more likely to possess information about a state if nearby states are known, for example because the subject has visited this region. To capture this spatial correlation of knowledge values, we use the Ising model [64]. The conditional probabilities for the Ising model are given by
P(K; = kv k-))
II
exp(1)s(ka kg)), with (ks ky) = =23 Σ 1)) (-1)8,
(s.)
R2
5

---

World
World
World
World
Log Loxe intrate
60
20
40-
10-
20-
-10-
-10
20
OPT
World
World
World
World
Log Liahhood inteese
40
10.
42
20
-10-
-20
OPT-UC
68
OPT4UC
OPT-UC
Figure I: Evaluation of the planning models on human data. The graphs show the difference of log likelihood of all transitions between the random actor model and each respective planning model for individual trajectory (depicted as dots). Positive values indicate an improvement compared to the random actor model. The black triangles represent medians. For most trajectories, the optimistic models demonstrate improvements over both the random actor and uncertainty bonus model. For an extended figure of World Fig. 11 in the appendix.
where we defined k-; ki(k;) As (-ii- ks) = - (his hio), we obtain for the prior fraction Eq.
P(K, k.rrey P(K;
K
= exp(-22p)) (kss (aa),,
prov
K
which enables us to sample from the posterior distribution as outlined in Algorithm ! This prior allows the incorporation of the structure of the agent's 's knowledge without significantly increasing computational cost, as its computation cost is negligible compared to planning during inference.
Evaluation
We evaluate method problem. In the evaluation, we compare our proposed approximate model for planning under partial knowledge against several other planning models: The optimal policy for fully informed agent (FD, a random actor model (R), a model based on exploration bonus (EB), a purely optimistic model (OPT), and our approximate model (OUR) from Section 3.2] which can be interpreted an optimistic model with penalization for uncertainty. Detailed information about these planning models is provided in Section A.7 and hyperparameters are listed Section A.8| An additional complexity comparison between our method and exact inference is provided in Section A.3
domains: series of grid-world navigation tasks and parking
4.1 Application to grid world navigation tasks
For the grid world navigation tasks, we designed grid worlds, where only a subset of fields was sampled as initially known. There, the goal is to control an avatar from a start field to a target field using a minimum number of steps. To validate our method on human behavioral data, we developed a browser game that allowed human subjects to navigate through these partially occluded grid worlds. We conducted an online experiment with 52 participants recruited via the Prolific platform. The experiment was approved by the local institutional review board (IRB). Detailed information about the grid worlds and the experiment design provided in Section A.4
Planning model To evaluate how accurately our proposed approximate planning model captures human behavior in comparison to other methods, we computed the likelihoods of the trajectories collected in the human experiment, Figure the log-likelihood differences between the random actor model and each the other planning models. The results indicate that, for most trajectories, our proposed approximation and the optimistic planning model achieve higher log- likelihoodds than the

---

(FI) Optimal with full knowledge
(R) Random policy
(EB) Uncertalitty bonus
(OPT) Purely optimistic
(OUR) Optimistic with uncertainty penalization
Figure 2: Models for planning under uncertainty. Subjects need navigate through the partially visible maze from a start field (smiley) to a target (flag) using minimum number of steps. Hatched fields indicate initially visible fields. (FI) The rational agent with complete world knowledge follows the shortest path on the left side. (R) The random agent shows undirected behavior and only reaches the goal after a large number of steps. (EB) The model with exploration bonus explores locally until it discovers the path to the goal. (OPT) A purely optimistic agent plans as if all unknown fields were navigable and heads towards the goal if uncertain. (OUR) Our proposed approximation, which is optimistic with penalization for uncertainty, initially navigates towards the goal due to its uncertainty but prefers the known path thereafter. This trajectory was most frequently observed in our online experiment.
other models. A Wilcoxon signed-rank test confirms that these differences are statistically significant for nearly all comparisons, with the exception of the exploration-bonus model in World 0 and the purely optimistic model World 7 (see Section A. for details). Although this analysis provides evidence for differences between models, it does not explain why penalizing uncertainty is important for modeling human behavior in these grid worlds.
To qualitatively differentiate the planning strategies. we designed one of the eight grid worlds (World 0) be smaller x 5) and structured such that each planning model predicts distinct behaviors. Fig. Disows the grid world along with the predicted trajectory when the most likely action is taken at each decision. Matching these unique trajectories to the online experiment data gave the following results: 21 % of the subjects chose path FI, 19% chose path EB, 21 % chose path OPT, and most participants, 35%, chose path OUR, representing our proposed approximate planning model. 4% of the trajectories did not fit any of the trajectories, Note that subjects who followed path FI could not have done so based on complete knowledge; their choice may have been influenced by variability or the belief that the "most obvious" path would be blocked, based on their previous trial experience.
Inference model To evaluate our proposed inference approach, we empirically compared it against two baselines. The first one is simple agnostic model that assumes all fields are known with equal probability of 0.5, except for the start and goal positions. The second baseline additionally assigns a probability of 0. to field where the agent turns around, based on the reasonable assumption that at these points the agent received new information. While our approach generates samples showing correlations between the fields, for the evaluation, we only consider the empirically estimated marginal probabilities. Consequently, our evaluation likely underestimates the information contained in the samples about the true posterior. We evaluate our method on two datasets: (1) trajectories from simulated agents using our proposed approximate planning model, and (2) trajectories collected in our online experiment. Since we controlled which fields were initially visible to participants, we can compare the inferred beliefs against the ground-truth knowledge.
We applied the inference approaches to each trajectory to estimate the probabilities of the fields being known. Figure 3 presents the results for two grid worlds (0 and 4). Although there are occasional false belief's due to the high variability of the subjects' behavior, the inferred explanations produced by our method appear reasonable. Additional inference results for the other worlds and more human trajectories can be found in Fig. 15 in Section A.6 For quantitative evaluation, we assessed the likelihood of the true knowledge variables under the inferred marginal knowledge beliefs, shown in Fig. 4 On simulated data, our method shows clear improvements over both baselines, and Wilcoxon signed-rank test confirms that these differences are significant (see Section A. for details). On experimental data, the quantitative results are more mixed: while we observe overall improvements in most environments, there are exceptions due to the high variability in participants' trajectories. The Wilcoxon signed-rank test indicates significant differences in five of the eight worlds. Inference for
7

---

(A0)
(B+CO)
(D0)
(E0)
(FO)
(B5)
(C5)
(D5)
(E5)
Figure 3: Inferred knowledge beliefs the considered inference methods. (A) Subject's initial view of World (B-E) Inferred beliefs (probability that field is known) for grid worlds 0 (top) and 5 (bottom). Panels (B, C, D) display trajectories when following the most probable actions of the optimistic policy with uncertainty cost. Panel (B) shows the agnostic baseline, (C) the baseline based on turnarounds, and (D) our proposed method. In (DO), it infers that the fields on the left side must have been partly unknown the subject, as well as the fields beneath the goal field, otherwise the agent would have probably taken the path on the left. Panel (E) shows randomly sampled trajectories from the probabilistic policy, where (E0), the inference method is certain that the subject could have known the field diagonally to the goal. (FO) shows the trajectory of subject "hgryw", where the probability of the left side and the fields beneath the goal incorrectly receive high probability. In (E5) the agent makes an additional detour due to the randomness in the policy, revealing that it did know the top path the goal and the dead end.
World 7 was particularly challenging, the path to the goal straightforward, that knowledge had very limited effect on behavior and behavioral variability leads to false beliefs.
To evaluate the efficiency gains resulting from reusing past results using the dynamic graph algorithm, we ran inference on the trajectories 50 simulated agents in world using the A algorithn instead of D lite. Generating 1,000 samples on world 0 took an average of 7.48 hours (SD) 1.53 hours) with A*, compared to 42 minutes (SD minutes) with D* lite, representing an average time reduction factor of 10.7 times. Details about the used hardware used for these experiments are provided in Section A.8
4.2 Application to the parking problem
While exact planning in BAMDPs is usually intractable, the parking problem (65 is an exception where the task structure allows for an exact solution. We use this task evaluate our inference method with approximate planning against the true optimal policy. In this task, driver proceeds along sequence of parking spots, each potentially free occupied, with parking costs decreasing along the route. However, if the driver passes the last parking spot without having parked, they need to park in the garage high cost. Importantly, the availability of each parking spot is unknown until the driver reaches it, making this problem BAMDP. A formal description of the task and the exact solution are provided Section A.5
We first evaluate how well the considered planning models approximate the exact policy. For randomly sampled parking occupancies, we compute the policies for all models at each parking spot (Fig.5% and report their mean KI. divergence from the exact optimal policy (Fig. [BB. Our proposed model closely approximates the true optimal policy by reaching the lowest KL, with the exploration bonus model as the next-best approximation. Next, we assess how the choice of the planning model
https://pypi.. wwtpproooctt/staaa...o

---

Simulated data World 0
World
World
World
World
World
World 6
World
Lag Liaahoood Incrusia
10
Buscline
Ouis
Bosclinc
Ours
Base inc
Ours
Baseline
Ours
Boschicc
Ours
Boso inc
Ours
Basoiie
Ours
Bascfine
Ours
Experiment World 0
World 1
World 2
World
World
World
World 6
World 7
Logg Liulhood tnenuaaa
Besdine
Ours
Besdline
Cur
Baselino
Ours
Buseline
Gurs
Bass Pm
Olrs
Baschno
Outa
Baselino
Our
Baseine
Our
Figure 4: Evaluation of the knowledge inference approach. The graphs show the difference in log likelihood of the true knowledge information between an agnostic model and the inference methods for individual trajectories (depicted dots) based on the marginal belief for each field. Positive values indicate an improvement compared to the agmostic model. The black triangles represent medians. For simulated data, method clearly shows improvements over both the agnostic model and the heuristic. Despite the high variability in the experimental data causing numerous false beliefs, our method demonstrates overall improvements over the baseline. World was particularly challenging for the inference as the optimal path was straightforward and therefore uninformative.
affects inference performance. Since all policies in this task can be computed efficiently, we run the knowledge inference procedure using each planning model to perform inference trajectories generated using the exact optimal policy. Inference results using our proposed approximate planning model are shown in Fig. 5C. Similar to the grid world setting, we also compare against agnostic beliefs (AGN) and a manually designed heuristic baseline (HEUR). This heuristic assigns free parking spots ahead of chosen parking 0.1 probability of being known, while other spots are known with belief 0.5. Figure 5D shows the average KL divergences of the inferred knowledge beliefs between the exact policy and other planning models. We find that both our proposed planning model and the exploration bonus lead to inference accuracies very close to the exact planning model, since the exploration bonus trades off garage cost and potential rewards of late parking similarly in this task.
5 Conclusion and future work
In this paper, we have addressed the problem of inferring the knowledge of an observed agent that acts under partial knowledge about the structure of the environment. We formulated the problem as an approximate Bayesian inference task and proposed an approximate planning model that allows for efficient evaluation within the inference procedure. We evaluated the method on both artificially generated data and data obtained from an online experiment using two task domains. By exploiting the incremental structure of the agents' knowledge using D* lite, we achieved a significant speedup compared to naively using the standard A* planning algorithm. While our approximate planning model proved effective for modeling the exact policy, the benefit of uncertainty penalization in modeling real human behavioral data still requires further investigation. The evaluation results suggest that our inference approach can provide a reasonable belief about the agent's 's world knowledge, particularly where it strongly influences behavior. As next step, method could be applied to analyze, for example, how subjects' awareness depends on topological features when solving mazes under time pressure. It could also be readily used to study spatial memory by examining which parts of a larger maze participants recall based on its local structure.
Our approach comes with several limitations. Even if the approach is tailored to medium sized state spaces, the approach is unlikely to scale to very high dimensions in its current form. Further scaling to larger state and belief spaces, and beyond binary knowledge representations, for example by employing amortized inference or advanced sampling methods, would be an interesting direction for future research. Currently, all unknown transitions modeled with single probability of

---

NL D. www.cffeeeeet be lett ecdd pall
10"
30
107
10
1AD
oF
OURDWA
10
HAND CPT OUR
68
PAND CMP
D0R
HBUR
ADN
Figure 5: Evaluation of the parking problem. (A) Choice probabilities for the considered planning models. Hatched fields indicate initially known parking spots. (B) Mean KL divergences of planning models from the exact optimal policy for generated problems. The black triangles indicate medians. (C) Inferred knowledge beliefs based on simulated parking decisions using the exact planning model. (D) Mean KL divergences between inferred beliefs using the exact and other planning models for generated problems. Both inference with our proposed planning model and exploration bonus lead to similar beliefs as inference under the exact model.
Crosses indicate the chosen spots.
traversability, which simplifies inference but could be extended in future work to allow different probabilities for different structures, such as corridors and junctions. Moreover, although the inference approach already captures the correlated structure of beliefs, this structure remains mostly latent in the data. Making this correlation structure accessible and enabling efficient recomputation when given additional information (conditioning) would beneficial. Also, investigating how additional information sources, such as eye-tracking data [66], could be used to increase the reliability of the approach would be a valuable direction for future research. Furthermore, future work could aim to consider planning in non-spatial domains, for example by using the PDDL formalism (7].
common challenge in behavioral experiments that participants' decisions often exhibit dependen- cies across trials and depend on prior experience. Furthermore, human subjects have been shown learn about the structure of the experiment design [33 34]. These factors could have introduced additional noise and biases to the experimental data. Improving experimental design to reduce these influences or explicitly modeling such influences hierarchically would increase the interpretability of the results. Future work could also extend our approach to more naturalistic settings, including stochastic environments, and explore incremental approaches for other types of planning [67]. Ex- ploring different forms of the experiment, such augmented reality, might also prove helpful in this regard.
Acknowledgments and Disclosure of Funding
We thank Dominik Straub, Fabian Tatai, and Philipp Fröhlich for insightful discussions. This work was supported by the Hessian Ministry of Higher Education, Research, Science and the Arts and its LOEWE research priority program 'WhiteBox' under grant L0EWE/2211/5119003// 001 (0010))777 Computations for the evaluation were performed on the geghpperformnce computer Lichtenberg the NHR Centers NHR4CES at Technische Universität Darmstadt.
References
[1] Andrew Y Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICML), volume 2000.
[2] Brian Zichart, Andrew I. Maas, Andrew Bagnell, and Anind Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial intelligence (AAAI), volume 23, pages 1433- 1438, 2008.
Zhengwei Wu, Paul Schrater, and Xaq Pitkow. Inverse POMDP: Inferring what you think from what you do. arXiv preprint 1rXiv:1805.09864 2018.
[4] Minhae Kwon, Saurabh Daptardar, Paul R Schrater, and Xaq Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 7898-7777 7909, 2020.
10

---

[5] Chris L Baker. Rebecca Saxe, and Joshua B Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):32--34,, 2009.
[6] Chris L Baker, Julian Jara- Eitingrr, Rebecca Saxe, and Joshua B Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 144):0064, 2017.
[7] Tan Suan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka, Online Bayesian goal inference for boundedly rational planning agents. In Advances in Neural Information Processing Systems (NeurlPS), volume 33, pages 19238-19250, 2020.
[8] Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka, and Joshua B Tenenbaum. Inferring the goals of communicating agents from actions and instructions. Proceedings of the AAAI Symposium Series, volume 2, pages 26-33, 2023.
[9] Tan Zhi-Xuan, Lance Ying. Vikash Mansinghka, and Joshua Tenenbaum. Pragmatic instruction following and goal assistance via cooperative sunuage-guidesd inverse planning. In International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 2094-2103, 2024.
[10] Neil Rabinowitz, Frank Perbet, Francis Song. Chiyuan Zhang. SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International Conference on Machine Learning (ICML), pages 4218-4227, 2018.
[11] Julian Jara-Ettinger. Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences, 29: 100-110, 2019.
[12] Constantin A Rothkopf and Dana H Ballard. Modular inverse reinforcement learning for visuomotor behavior. Biologicul cybernetics, 107:477-490, 2013.
[13] Sid Reddy, Anca Dragan, and Sergey Levine Where do you think you're 'r going?: Inferring beliefs about dynamics from behavior. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[14] Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In International Conference on Artificial Intelligence and Statistics (AISTATS), pages 102-110, 2016.
[15] Matthew Golub, Steven Chase, and Byron Yu. Learning internal dynamics model from control demonstration. In International Conference on Machine Learning (ICML), pages 606-614, 2013.
[16] Sunayana Rane, Mark Ho, Ili Sucholutsky, and Thomas L Griffiths. Concept alignment as prerequisite for value alignment. arXiv preprint arXiv:2310.20059, 2023.
[17] Jac- Deug Choi and Kee-Eung Kim. Inverse reinforeement learning in partially observable environments. Journal of Machine Learning Research (JMLR), 12:691 2011.
[18] Matthias Schultheis, Dominik Straub, and Constantin A Rothkopf. Inverse optimal control adapted to the noise characteristics of the human sensorimotor system. In Advances in Neural Information Processing Systems (NewrlPS). volume 34, pages 9429-9442, 2021.
[19] Dominik Straub, Matthias Schultheis, Heinz Koeppl, and Constantin A Rothkopf. Probabilistic inverse op- timal control for non-lineas linei partially observable systems disentangles perceptual uncertainty and behavioral costs. Advances in Neural Information Processing Systems (NeurIPS). volume 36, pages 7065-7092. 2023.
[20] Giorgia Ramponi, Gianluca Drappo, and Marcello Restelli, Inverse reinforcement learning from gradient- based learner. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 2458-2468, 2020.
[21] Alexis Jacq, Matthicu Geist, Ana Paiva, and Olivier Pictquin. Learning from learner In International Conference on Machine Learning (ICML), pages 2990-2999, 2019.
[22] 7oe Ashwood, Nicholas A. Roy, Ji Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making, In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 3442-3453, 2020.
[23] Matthias Schultheis, Constantin Rothkopf, and Heinz Koeppl. Reinforcement learning with non- exponential discounting. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 6649-3663 2022.

---

[24] Lance Ying. Tan Zhi-Xuan. Lionel Wong, Vikash Mansinghka, and Josh Tenenbaum. Grounding language about belief in a Bayesian theory-of-mind. In Annual Meeting of the Cognitive Science Society (CogSci). volume 46, 2024.
[25] Zoe Ashwood, Aditi Jha, Jonathan W Pillow. Dynamic inverse reinforcement learning for characterizing animal behavior. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 29663-29677,, 2022.
[26] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[27] Nathaniel D Daw, John P O' O'doherty, Peter Dayan, Ben Seymour, and Raymond J Dolan, Cortical substrates for exploratory decisions in humans. Nature, 87777795):77--87, 2006.
[28] Samuel J Gershman. Deconstructing the human algorithms for exploration. Cognition, 173:34 42, 2018.
[29] Charley M Wu, Eric Schulz. Maarten Speekenbrink, Jonathan D Nelson, and Bjöm Meder: Generalization guides human exploration in vast decision spaces. Nature human behaviour, 2(12):915-922 2018.
[30] Charley M Wu, Eric Schulz, Timothy J Pleskac, and Maarten Speekenbrink. Time pressure changes how people explore and respond to uncertainty. Scientific reports, 12(1):4122, 2022.
[31] Maarten Speekenbrink and Emmanouil Konstantinidis. Uncertainty and exploration a restless bandit problem. Topics in cognitive science, 7(2):351 117551137,,20 2015.
[32] Alihan Hüytik, Daniel Jarrett, and Mihacla van der Schaar. Inverse contextual bandits: Learning how behavior evolves over time. In International Conference on Machine Learning (ICML), pages 9506-9524, 2022.
[33] Daniel Acuna and Paul R Schrater. Structure learning in human sequential decision-making. In Advances in Neural Information Processing Systems (NeurIPS), volume 21, 2008.
[34] CS Green, C Benson, D Kersten, and P Schrater. Alterations in choice behavior by manipulations of world model. Proceedings of the National Academy of Sciences, 107 16601-1666011,, 2010.
[35] Zeb Kurth-Nelson Marcos Economides, Raymond Dolan, and Peter Dayan, Fast sequences of non-spatial state representations in humans. Neuron, 91(1):194- 2041199-224,22 2016.
[36] Mark K Ho, David Abel, Carlos G Correa, Michael L. Littman, Jonathan D Cohen, and Thomas L Griffiths. People construct simplified mental representations to plan. Nature, 606(7912):129-11) 2022.
[37] Frederick Callaway, Bas van Opheusden, Sayan Gul, Priyam Das, Paul M Krueger, Thomas I. Griffiths, and Falk Lieder. Rational use of cognitive resources in human planning. Nature Human Behaviour, 6(8): 1112-1125, 2022.
[38] Mathew Monfort, Brenden M Lake, Brian Ziebart, Patrick Lucey, and Josh Tenenbaum. Softstar: Heuristic- guided probabilistic inference. Advances in Neural Information Processing Systems (NeurlPS), vol- ume 28, 2015.
[39] Kartik Chandra, Tony Chen, Tzu-Mao Li, Jonathan Ragan-Kelley, and Josh Tenenbaum. Inferring the future by imagining the past. In Advances in Neural Information Processing Systems (NeurIPS), volume 36. pages 21 196-21216, 2023.
[40] Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49:209 22299 2002.
[41] Fugene Chamnik and Rohert P Goldman, A Bayesian model of plan recognition. Artificial Intelligence, 64 (1):53-79, 1993.
[42] Miguel Ramírez and Hector Geffner. Probabilistic plan recognition using off-the-shelf classical planners. In AAA/ Conference on Artificial intelligence (AAAT), volume 24, pages 1121-1126, 2010.
[43] Shirin Sohrabi, Anton V Riabov, and Octavian Udrea. Plan recognition as planning revisited. In Interna- tional Joint Conference on Artificial Intelligence (IJCAI), pages 3264, 2016.
[44] William L Hamilton. Graph representation learning. Morgan & Claypool Publishers, 2020.
[45] Dimitri Bertsekas. Dynamic programming and optimal control: Volume I, volume 4. Athena scientific, 2012.
12

---

[46] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynumic Progrumming. John Wiley & Sons, 1994.
[47] Shimon Even. Graph algorithms. Cambridge University Press, 2011.
[48] Edsger W Dijkstra. A note on two problems in connexion with graphs. Numerische Mathematik, 1: 269-271, 1959.
[49] Sven Koenig and Maxim Likhachev. Improved fast replanning for rohot navigation in unknown terrain. In IEFE International Conference on Robotics and Automation (ICRA), volume I, pages 968-975, 2002.
[50] Michael O' O'Grrdon Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
[51] Jan Sternby. A simple dual control problem with an analytical solution. IEEE Transactions Automatic Control, 21(6):840 844, 1976.
[52] John C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society Series B: Statistical Methodology, 41(2)188-164, 1979.
[53] Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian reinforcement learning:
A survey. Foundations and Trends in Machine Learning, 8(5-6)3359 2015.
[54] Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution discrete Bayesian reinforcement learning. In International Conference on Machine Learning (ICML), pages 697-704, 2006.
[55] Nuttapong Chentanez. Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 17, 2004.
[56] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrel. Curiosity-drive exploration by self supervised prediction. In International Conference on Machine Learning (ICML), pages 2778 2787, 2017.
[57] Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abheel. VIME: Variational information maximizing exploration. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016
[58] Malcolm Strens. Bayesian framework reinforcement learning. In International Conference on Machine Learning (ICML), volume 2000, pages 943-950, 2000.
[59] lan Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. In Advances in Neural Information Processing Systems (NeurIPS), volume 26, 2013.
[60] Zico Kolter and Andrew Y Ng. Near-Bayesian exploration in polynomial time. In International Conference on Machine Learning (ICML), pages 513-520, 2009.
[61] Arthur Guez, David Silver, and Peter Dayan, Efficient Bayes-adaptive reinforcement learning using sample-based search. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 2012.
[62]
Christopher M Bishop. Pattern recognition and machine learning. Springer, 2006.
[63] Peter E Hart, Nils J Nilsson, and Bertram Raphacl. A formal basis for the heuristic determination of minimum cost paths. IEEE Transactions on Systems Science and Cybernetics, 4(2):100- 107, 1968.
[64] Kevin P Murphy, Machine learning: A probabilistic perspective. MIT press, 2012.
[65]
Dimitri Bertsekas. A course in reinforcement learning. Athena Scientific, 2024.
[66] Florian Kadner, Hannah Willkomm, Inga Tbs, and Constantin Rothkopf. Finding your way out: Planning strategies human maze-solving behavior. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023.
[67] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International Joint Conference on Artificial Intelligence (IJCAI), volume pages 2586-259) 2007.
[68] Samuel Sanford Shapiro and Martin Wilk. An analysis of variance test for normality (complete samples). Biometrika, 52(3-4):591-61 1965.
[69] Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics: Methodology and distribution, pages 196-202. Springer, 1992.
[70] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1): 289-300. 1995.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The mentioned approach is introduced in the paper, evaluated, and the results are discussed in the evaluation section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

There are no theoretical results. A derivation of Eq. ① is provided in A..
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The evaluation section contains information about the procedures of the evaluation. In the appendix we give additional details about how the experiments were conducted and the evaluation performed. and hyperparameters are provided. We additionally provide the code and the data set used for evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
15

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Both the code and dataset publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides https///oooeeabaaaiiooonoooooooa
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips cc/ wtc/publicc/ /useeguudddee/eeeubbiiiii
Codeeobmiiiie iastuubbiisssiii ssisssiicccyyyii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
They are provided in Section A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results for each individual trajectory, besides the median, are scattered in the plots. For the runtime analysis, we provided the standard deviation along with the mean.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information is provided in Section A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All participants were paid more than the minimum wage of both US and UK, 1.44 (UK) and the study was approved by the local IRB. With the IRB conditions, all data-related concerns were handled (e.g., minimal unidentifiable information, consent, Societal impact and potential harmful consequences are discussed in the conclusion section.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Negative societal impacts are discussed in the conclusion section. Positive impacts are described in the introduction but not in direct relation to socicty.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The code and dataset be published do not pose any risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provided information about the D* and A implementations use in work.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The provided code and experimental data is documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Instructions with screenshots are provided in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: There were no potential risks for the study participants and the study was approved by the local IRB.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy pettps://neurre http///arrisccc/ooofffrrree/e//p cc/Coonfereecce//2225LLL.. for what should or should be described.
20

---

A Appendix
A.1 Derivation of the switch acceptance probability
P(K; - K D)
P(K; -k!
K-;,D D)
(4)
p( Dioy
K D) + P(k
K .Ki.D) D)
P(K; k:p
K D)
(5)
P(K; = -.i K D)
P(D K; prev K
P(K; hr K
(6)
P(D K₁
ki " K_i) P(K,
- K
A.2 Further information on likelihood and planning model
The likelihood of the visited states D
{3) ,....,5""" 5² given the knowledge values decomposes as
T-1
P(D = (3t 5T) K = k) = II T. ( 1
where ne Te : [0, denotes the stochastic policy at time t, taking the form
T(s,8') x exp (s(C,(s,)) (((())))))(()
(8)
Here, C. S x S R is the function measuring the expected shortest path cost between two states 8 and 8y at time 1, formally defined as
C(st, 83) Crot (st, 8;)/4
if (A) v K3) A (st, 8)) EE, if (kt kf) A (s1, ENn otherwise.
C,s,, 85)
0
where Ky is the knowledge value for state 8; updated for time t, which is k; = 1 (the field was so the state has been visited before.
initially known) or if {s'...., 5t-1),
naive computation of the policy for each sample and time step is computationally expensive. We observe that within one trajectory between two time steps, the agent gains only knowledge of the currently visited field 8; If the field assumed to initially known, i.e., k; not change and we can directly reuse the planning results from the previous time step. Furthermore, within the Gibbs sampling process, the knowledge value is switched only for one field 8y. If the field was part of the trajectory up the current time step t. i.e., 8j E {3' ..... 3*), the agent knows the field at time in both the previous Gibbs sample and the current one. In this case, we can directly reuse the planning result from the past iteration. Otherwise, in comparison to the past sample, only the edges potentially connected to 8j may change: For all 8m E S with (8j, 8m) E & C(sj, 8m) potentially increases there is no edge between 8j and 8m. i.e., (8),8m) E E, and may decrease otherwise. By using a dynamic shortest path algorithm such as D* lite, we can update the planning results from the past sample, updating the few affected edges. For the first sample, where we do not have previous planning results, we can reuse the planning results from the respective past time step (for iterations 1) instead. With this, we only need solve the full shortest path problem once, for the first sample for f = 1. The complete algorithm for computing the likelihood is provided in Algorithm
the policy does
A.3 Complexity analysis
In the following, we compare the complexity of our proposed algorithm to exact inference as used in Bayesian inverse planning. For exact inference, we must consider all 2 combinations of values of K. where N is the number of states. For each configuration, computing the agent's 's policy and likelihood results in a complexity of O(2 Cp Planning in the belief space using value iteration has complexity O(SS]2.. [Al/e) = O(22N 4/e), with the precision of value iteration, since the belief MDP has N states. Additionally, planning has to be performed up to N times for each configuration,
21

---

Algorithm 2 Likelihood computation
Output:
Likelihood P(D k), Planning results for all I...., T-
Input:
Partial world model W, Knowledge values k, Trajectory D
{8l
optional: Planning results of previous iteration peree, Knowledge switching value k,
1- 1: for t in ( do
2:
Get state index from trajectory state 5
3:
if 1 and k[x] then
4:
P[[] < Pt
5:
else i prov and j E (8t
8 then
6:
4 -
7:
else if Prrev prev then
8:
Update planning results Pprev [t] with switch at j
9:
else if > 1 then
10:
Update planning results P[d] with switch at
11:
else
12:
Compute planning results P(t]
13:
end if
14:
Compute policy n, using Eq. ( with P(t)
15:
Compute likelihood of D(t + using Eq.
16:
end for
as the agent is assumed to replan along the trajectory (and then all fields have been visited, including the goal). The results in total complexity for inference of O(N 23N /e).
In contrast, our Gibbs-sampling-Be based approach scales linearly with the number of variables: we sample each K; conditioned on all others, leading a complexity of (L-N-- NCCplamnig), where L is the number of samples. Our planning approximation avoids the complete belief space and uses the shortest path algorithm D* lite, which has worst-case complexity similar to Dijkstra, O(E)| | log [V]) = (44 N + N log N) = O(N log N). Similar to Bayesian inverse planning, planning needs to be done N times per sample. Therefore, the total cost is only OL N N N Mog N) = O(L N log N).
A.4 Navigation experiment design and procedure
Using the Prolific platform, we recruited 52 participants from the US and UK who were fluent in English. The experiment took between 6 and 10 minutes and each participant was paid f1.70 plus a performance bonus of up to f1. The bonus was based on the reward earned during a randomly chosen trial. The reward was set to 100 Naups pence. Participants' earnings were greater than the minimum wage of E1 1 .,4/ (UK) and total of E165.08 was spent on participant compensation. To participate in our study, subjects had to read and accept our information and declaration data protection policy, allowing us to process and publish the behavioral data in anonymized form.
After presenting the introductory screens (Fig. (6) to the participants, the actual experiment began. In each trial of the experiment, subjects were shown a grid world, where only a subset of the fields was visible. The remaining fields were hidden and therefore unknown the participants. The starting field and goal position were always initially visible (Fig. 7 The avatar could be controlled with the keyboard in up to four directions (left, right, up, down) and could only move if there no wall in the chosen direction. There was a minimum interval between two successive movements of 300 ms. As soon as the avatar entered unknown field, it became visible along with the surrounding walls and remained visible for the rest of the trial (Fig.B) 8).
Participants were instructed to reach the goal within minimum number of steps to maximize their reward and, consequently, their bonus payment. Before the actual experiment, participants completed 7 introductory mazes to familiarize themselves with the task at varying levels of complexity (Fig. ( These maps were not considered for bonus payments or for evaluation. The actual experiment consisted of 8 mazes, which were presented in random order. At the end of the experiment, participants were asked to indicate their sex and age in a short survey and could leave a comment (Fig. 10P.
Grid worlds 1-7 were manually designed by placing the start and goal positions at arbitrary locations, drawing path between them (of varying complexity, for example, the path of grid world I is very
22

---

tragghe yournd Hieds make saan remeebde
Pera diged memale wnyoor troggh. Croc you sna arcon your fimnd
cxphane
where have cenn
The mormays quee wit soucsu
you your mankegg Lucily youltnd
Evey you drange the room the food anrout decrusis
Wall
black
(a)
(b) Part 2
can rengara though tample by your
You muliple renns, each map. percee as bonua papment to
The reward (ood) mrs ad on the abd of scree. poesubiiy
you
relyyet
(d) Part
Figure 6: Introduction displayed to the participants. The introduction gives cover story to explain the setup and goals. Participants were instructed how the avatar was controlled and details about the bonus payment given. Participants could go back and forth in the introduction for the they wanted to reread some passages.
reward: 100
Figure 7: World 6 before movement. Only some fields of the world were visible right from the start, always including the start and goal field. Visible, walkable fields were grey, walls were red. The goal was marked by the black flag, the player's position by the yellow smiley. In this example, the player could walk to the right (known field), to the left, down (unknown fields). Unknown fields were not visible upfront but became visible as soon as the player visited them. The reward, displayed on the right, was always 100 the beginning of level and reduced by with each move.
complicated while for world 4 it is straightforward), and extending the path by dead-ends. We used all the worlds we designed in the experiment, providing transparency about the performance of our method by showing cases where beliefs hard identify (e.g., grid world where the path had low complexity). Moreover, the varying complexity prevents humans from learning about the experiment's structure, e.g., learning prefer indirect paths (which probably still occurred some extent in the conducted experiment).
A.5 Parking problem
The parking problem can be formally defined as a graph BAMDP (V,,E,C)
hyperstates V" V x & consisting of states (nodes) V and knowledge states
states V (1,.... M) {1,, G), where M the number of parking spots, the terminal state and goal, and G the garage state,
23

---

reward: 100
reward: 76
(a) View at the beginning
(b) View after navigating through parts of the world
Figure 8: Example visibility and reward changes upon movement. At the beginning of the level, before moving, only some fields of the world were visible and the reward was set to 100. The start and goal fields were always visible right from the beginning. When moving through the world, unknown fields that have been explored remained visible throughout the whole trial. The reward is reduced by with each move. The reward of 76 in (b) corresponds to 24 moves having been made.
reward:
98
reward:
99
Figure 9: Different world complexities. Worlds could implement straight passages the goal (left) or passages that consisted of multiple turns (right). The two shown worlds both had their starting points at the bottom left corner, the block above visible from the beginning. They were part of the test worlds with the purpose of familiarizing the participants with the concept of the experiment.
Please answer all questions carefully before clicking the button the bottom finish.
What your ago?
select option
What your sex?
select an option
Do have anything would like note?
survey
Figure 10: Survey at the end of the experiment. The survey consisted of questions about the participants and gave the possibility to leave additional comments. Selectable options for age were <25, 25-34, 35-44, 45-54 and 55+. Selectable options for gender were female, male, and other. Both questions also had the option prefer not answer.

---

20-
World 0
28 OP1
Figure 11: Evaluation of the planning models on human data for World (extended plot)
knowledge states + P([1....... M)) M)), indicating the "known" parking spots, where the occupancy is known. Here, P denotes the power set operator.
transitions & Econtinue park U Egarage with
G)} the actions to continue to Econtimue = {(i, 1), for 1, ..... M 1) ((M,(M the next parking spots,
Epark {i, L), for i = I,... M - 1, if parking i is free} the actions to park,
ggarage = ((G, the action to park at the garage,
if (st, 83) E Econtinues
8, +
(st, $j) E Eparks
the cost function 85)
M
+1
(8t, 85) E Egarager
8O
otherwise.
For the parking problem, all "continue" transitions are always known, while the "parking" transitions only known if the respective parking spot 8; is known, i.e., k; 1. In contrast to the formulation of Bertsekas [65], we assume that the agent has initial knowledge about some the parking spots, which we aim infer.
A.6
Additional results
In the following we provide additional evaluation results.
A.6.1
Grid world tasks
For the grid world tasks, we present the following additional evaluation results:
Additional plots showing the trajectories the considered policies analogous to Fig. 2 for worlds 1-7 shown in Fig. 14
An extended plot for the quantitative evaluation of the planning models (from Fig. shown in Fig.
is
Additional results of our proposed inference approach for additional worlds and subjects provided in Fig. 15 Examples were picked to give an overview of the trajectory variabilities encountered in the experiment.
To evaluate the influence of the prior, we conducted an ablation and performed inference using a uniform prior. The results were qualitatively very similar (see Fig. 12, but the relative improvements between the baseline and our method were on average half as large (average factor 1.991). This suggests that while the prior has some influence, it is not essential for the effectiveness of our method.
In addition, we performed statistical tests to investigate whether the differences observed in the evaluation significant:
First, we considered the evaluation of the inference method. To choose suitable test, we first checked the assumptions of normality for the datapoints of each world and method combination using the
25

---

World
World
World
World 3
World 4
World
World
World
Liathood D forencs
0.0
10
7.5-
50
5.0
5.0
2.5
2,5
5.0
2.5
0.0
0.0
-2.-
255
-75
-7.5
Ours
Banabre
Dunt
Same Oura
Ourt
Baaobre
Durs
Stane
Beeetre
Duna
Figure 12: Ablation study of inference on human navigation data with a uniform prior. Ours* denotes our inference method with uniform prior. In comparison to the results in Fig. 4 our method performs qualitatively very similarly, but the relative improvements between the baseline and our method were average half as large (average factor 1.991).
Shapiro- Wilk test [68 with = 0.05. We found that only subset of the data (4/16) followed a normal distribution. Given that the t-test requires normality assumptions to be met, applied the Wilcoxon signed-rank test [69]. This non non-parametric test does not assume normality and can be used assess the differences between the central values of the baseline and method for each world. We applied a False Discovery Rate (FDR) correction using the Benjamini- control for multiple comparisons. Given that all worlds have individual belief spaces and represent different tasks, we consider them separately. The corrected p-values (q-values) of the Wilcoxon signed-rant -ank test are shown in Table I For simulated data, for each world, significant differences between our method and the baseline were found. For the human experiment data, differences were significant for five of the eight worlds.
method (70) to
world
simulated data
real human data
q-value
significant
q-value
significant
0
0.0000007368
Yes
0.09551332
No
0.0000000013
Yes
0.00000003
Yes
2
0.0000000172
Yes
2.05887162
No
0.0000000248
Yes
0.0401 1417
Yes
4
0.0000000016
Yes
0,00000008
Yes
0.0000000013
Yes
0,002 14453
Yes
6
0.0000000013
Yes
0,62287947
No
0.0000110184
Yes
0.00325433
Yes
Table 1: Adjusted p-values and significance results inference log likelihood differences between our approach and the baseline using the Wilcoxon signed-rank test.
For the planning models, we applied the same procedure. Again, the Shapiro-Wiis Wilk test only indicated normality for 19% of the planning model-world combinations. We therefore proceeded with the Wilcoxon signed-rank test to check for significant differences between our proposed model and other planning models. The resulting adjusted p-values (q-values) are shown in Table The differences of all combinations were significant, except for the exploration bonus model world O, and the purely optimistic model in world 7.
A.6.2 Parking problem
For the parking problem, we provide a figure comparing the log-likelihoods the true knowledge values under the inferred beliefs relative to agnostic beliefs in Fig. [3] One observe that inference using exact inference leads to similar likelihoods as when using the exploration bonus approximate method. We conclude that for the parking problem, these planning methods approximate the exact solution accurately.
A.7 Planning model: Exploration bonus
For the exploration bonus model, we designed the following MDP: Transitions were only considered from known fields; from unknown fields all transition probabilities were set to zero. Entering non-visible field led to exploration reward of 1, reaching the goal to reward of 10. The policy
26

---

world
fully informed
exploration bonus
optimistic
q-value
significant
q-value
significant
q-value
significant
0.0000127346
Yes
0.5892034940
No
0.0006017581
Yes
0.0000000290
Yes
0.0431046667
Yes
0.0006887471
Yes
2
0.0000000017
Yes
0.0003253850
Yes
0.5847222927
No
0.0000000017
Yes
0.0000000469
Yes
0.0000000028
Yes
0.0000000648
Yes
0.0000295632
Yes
0.0007460520
Yes
0.0000000023
Yes
0.0000076167
Yes
0.0000000529
Yes
0.0000000017
Yes
0.0092637150
Yes
0.0097015615
Yes
0.0185740168
Yes
0.2507408261
No
0.0018645117
Yes
Table 2: Adjusted p-values and significance results of planning log likelihood differences using the
Wilcoxon signed-rank test.
2
Log Likellhooo increase
RAND
OPT
EB
OUR
EXACT
HEUR
Figure 13: Additional results for the parking problem. Knowledge likelihoods under inferred beliefs when using different planning models for the inference method, relative to agnostic beliefs.
was formulated softmax over the Q function 26
} (+e((s.))))
(8, exp
The Q function was computed using tabular value iteration [26]. For the value iteration, a maximum horizon of 100 steps considered and it was previously stopped if the infimum norm of the Q function difference between two consecutive iterations was smaller than 0.01. The softmax temperature set to 0.1.
For the parking problem, however, for each unoccupied state, the goal is directly accessible. With the previously described exploration bonus formulation, the agent would greedily follow transitions to the goal. Therefore, for this problem, we chose a different approximation: We plan similarly to our approximation, with the difference that instead of scaling the cost for uncertain decisions, agents incur a negative cost (reward) they transition to an unknown field. This formulation makes agents "explore" unknown parking spots for while, but park their car soon as the high garage cost exceeds potential rewards through exploring new parking spots. This makes the exploration bonus strategy very similar
approximation and the optimal policy for the parking problem, explaining the similar performance observed in the evaluation.
A.8 Hyperparameters and hardware
Throughout the experiments, we used the following hyperparameters:
Total number of samples for the inference: 1000
Number of samples rejected for burnin phase: 100
Parameter for the Ising model (spatial prior): / 0.2
Cost for transitions from/to known fields:
27

---

Belief of traversability for unknown fields: 0.5
Softmax temperature for our proposed planning model: 1
Softmax temperature for optimistic planning model: 0.8
Computing resources used: Intel Xcon@ Platinum 9242 Processor core per trajectory)
For the parking problem, we used the following hyperparameters:
Number of parkings: 10
Probability of parking being free: 0.3
Probability of knowing parking spot: 0.3
Belief of traversability for unknown fields: q 0.3
The hyperparameters sample sizes were selected such that the results did not vary anymore. Costs and temperatures were set to standard values, e.g., the cost for known transitions was set to 1 as in the work by Chandra et al. [39]. and the cost for unknown transitions to twice as much, and the resulting trajectories were checked for plausibility. A low cost makes the agent more optimistic when acting under uncertainty (preferring unknown fields, potentially yielding shorter paths), while increasing the cost makes the agent more risk-averse, preferring known paths that might be longer. The hyperparameter of the prior was selected manually to produce moderately connected knowledge regions (neither fully fragmented nor overly clustered). The same hyperparameters. were used both for generating the ground-truth knowledge values in the experiment (the fields initially displayed to the participants) and for inference. The temperature of OPT since the uncertainty cost leads to higher total costs, which would decrease stochasticity otherwise. For the grid world experiments, the parameter of the spatial prior was selected manually to produce moderately connected knowledge regions (neither fully fragmented nor overly clustered). The same hyperparameters were used both for generating the ground-truth knowledge values in the experiment (the fields initially displayed to the participants) and for inference.
slightly smaller as for OUR,
28

---

Rere ben
Woldd JOPPAU Byareto
Weild poruerieee
Music PNPT
ROYEL COAMOIEI
None IETHI DERIENN EONEEEEEEEN
Pedt CITE FUN onterne
Must JOPO PANN WORK
Figure 14: Models for planning under uncertainty for Worlds 1
29

---

TEL
F
F
Smndesd NAT Mote
Aet bot Mert
Soleen lant Wodd!
Solent Shat Wole
Salust aedl Whene
Suigrt Seree Civee
Figure 15: Additional results our inference approach showing the variability of subjects' behavior
30


Reviews:
[{"review_id": "LSDERSofnc", "reviewer": "Reviewer_pZq4", "summary": "This paper address the problem of inferring a latent decision making model from observed behaviours of another agent. The problem differs from standard inverse RL or inverse optimal control, in that the latent model may contain uncertainty about the true world model, i.e., the dynamics of some states may not be known to the observed agent. The problem is addressed by modelling each state of the underlying model as known or unknown in the latent model, and then performing Gibbs sampling to compute a posterior over whether states are known or unknown, with the likelihood being given with respect to the observed policy. The application setting is to recover navigation models from observation.", "questions": "How does this technique compare to conventional IRL on these experiments?\nWhy is the posterior so high variance on the experiment data for figure 4?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The problem in general is interesting, the written text is clear, and I enjoyed reading the paper.\n\nThe major limitation of this paper is that it is not really clear what the purpose of the result is. I understand the overall problem of recovering a latent model to model behaviour, and I like the problem setting that recognises that the acting agent may in fact not have a complete model of the world. However, the restriction to a discrete MDP, and the assumption that the *only* thing the acting agent does not know is the transition dynamics both strike me as relatively artificial, and poorly aligned with inferring real navigation models. (Inferring latent navigation models is actually a very active area of research in spatial psychology. Reading about how traffic engineers recover navigation models is super interesting!) The provided experimental results show the technique works, but are not especially compelling motivation. The underlying techniques (inference by sampling with likelihood computed from the observed trajectory) seems relatively straightforward, so without a clearer connection to a real problem, it's hard to know what the contribution is.\nI was also surprised not to see a comparison to conventional IRL --- this work becomes even more interesting if there is a setting where IRL learns the wrong thing, which leads to poor performance, but the BAMDP setting learns the correct concept.\n\nThe authors list of limitations is fairly good, but these limitations are not trivial, and make me wonder even more if the proposed approach is actual useful. I had similar concerns about scalability. The authors propose using additional information sources such as eye tracking, which I completely agree with as an important step. I think that human navigation models are much more based on perceptual features than on a latent MDP where the human is not sure about the transition dynamics of some of the states.\n\nThe experimental results are reasonable, but the figures are extremely difficult to parse, partly because there are some key features that are not described --- it took quite a while to work out for figure 3 that the more blue a square is, the more likely it is that the human did not know it. I am also surprised at how high variance the posterior is for the experiment data in figure 4 -- the baseline seemed to outperform the BAMDP technique for some worlds.\n\nI am generally inclined to support this paper, but the absence of a connection to a real application or a deep algorithm advance, the contribution of this paper is modest at best.", "was_revised": true, "final_justification": "I read the authors response and found it helpful. I continue to be positively disposed towards this paper.", "rebuttal": "We thank the reviewer for their thoughtful and encouraging review. Below, we address the main concerns in detail.\n\n## Purpose and applicability of the results\nWe appreciate this opportunity to clarify the motivation and intended use of our method. Our work is situated in computational cognitive science, where the central goal is to explain observed behavior through computational models, and not just predict it. In many controlled navigation tasks, the goal is known, but the structure of the environment is either unknown, must be memorized, or is too complex for precise planning. As researchers interested in human planning, we would like to decode the information of the environment that is relevant for the specifically followed trajectory of a subject. With this, we can gain insights about transferability (which information did subjects transfer between tasks), spatial memory (which parts of the environment were present in the subject's memory), or awareness of certain parts of the environment when planning. Our method serves as a tool to infer precisely this information. As a concrete example, we are actively using this inference framework in ongoing work to analyze how subjects' awareness depends on topological features such as corridors or junctions when solving a maze under time pressure. Further, the method could be concretely used for investigating spatial memory by investigating which parts of a larger maze a subject remembered based on its local structure.\n\n## Generality and scalability\nWe acknowledge that our approach is scoped to discrete MDPs with uncertainty over transitions. However, to the best of our knowledge, this is the first approach that enables inference over high-dimensional knowledge states from behavioral data. In our experiments, we applied the method to a task with 169 binary latent variables, which corresponds to a belief space of size $2^{169}$. This is significantly beyond what existing knowledge inference techniques, such as Bayesian inverse planning, can handle. As mentioned above, this scalability allows practical application in controlled behavioral experiments. Smaller continuous domains could also be addressed via discretization. While scaling to more complex continuous environments remains an open challenge, we believe our work lays important groundwork for doing so.\n\n## Comparison to IRL\nWe did not include a direct comparison to IRL because our method and IRL infer fundamentally different quantities: IRL estimates preferences (e.g., a cost function), while our approach infers latent knowledge (i.e., beliefs about world structure). These are not directly comparable, especially since beliefs and rewards live in different spaces. That said, in terms of trajectory likelihoods (\"performance\"), we expect that IRL fits the observed trajectories equally well by assigning high rewards to visited states (e.g., when the agent turns). However, most would agree that this explanation would be less appropriate, as similarly illustrated by the airport example in our introduction. If the reviewer feels it would be helpful, we are happy to include a learned cost function plot via IRL in the appendix of the final version to further illustrate this point.\n\n## Clarification of Figure 3\nThank you for pointing out the missing label for the colorbar in Figure 3. We will add it in the revised version to improve clarity.\n\n## Posterior Variance in Figure 4\nWe already briefly discussed the high variance of likelihoods in the evaluation and future work section. The variance arises from the behavioral variability in the human experiment, which could be caused by inattentiveness, exploratory behavior, or higher-level learning. In Figure 13 (last three columns) in the appendix, we show representative examples of the collected human data. Many participants especially in worlds 6 and 7 do not follow the most \"obvious\" or direct path. This could be explained by the experience that these paths were oftentiimes blocked in previous mazes, so they avoided them. To minimize such effects, we included a diverse set of gridworlds (e.g., world 6, where the direct path is indeed the shortest) and introductory trials. Nevertheless, we were surprised by the high variability of behavior and consider this an interesting direction for future investigation."}, {"review_id": "XCPLWIB6cy", "reviewer": "Reviewer_93XY", "summary": "+ Summary & Contributions\n\t- This paper tackles the problem of inferring agent knowledge about the environment based on an input trajectory of agent behavior.\n\t- Opting to directly model whether individual states are either \"known\" or \"unknown\" explicitly, the authors employ Gibbs sampling for performing approximate Bayesian inference to determine which transitions of the environment are known to the agent given a trajectory of behavior.\n\t- Empirical results against a variety of competing models of human participants' planning based on partial knowledge of the transition function confirm that the proposed model is competitive at providing a plausible explanation of human planning behavior and accurate inferences about the environment.", "questions": "My questions for the authors should be clear from the comments above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "+ Quality\n\t- Strengths\n\t\t- N/A\n\t- Weaknesses\n\t\t* Major\n\t\t\t- I suspect I may be misunderstanding something fundamental about the problem the authors are trying to solve. A state $s \\in \\mathcal{S}$ of a finite MDP is considered known (with $\\tilde{k} = 1$) if, for each possible action $a \\in \\mathcal{A}$, the agent knows the exact successor state $s' = \\mathcal{T}(s,a)$. It seems like the exposition of Section 3.1 highlights the fact that only maintaining direct beliefs about $\\mathbf{K}$ discards too much information for tractable, exact Bayesian inference thereby necessitating the use of a subsequent Gibbs sampling procedure. Alternatively, since the MDP is finite and deterministic, why not proceed in the standard way by maintaining beliefs about the full MDP transition function as a collection of $|\\mathcal{S}||\\mathcal{A}|$ independent Dirichlet distributions (one for each possible next-state transition distribution)? Since the transition dynamics are deterministic, a reasonable prior would, for instance, be uniform with an initial parameter $\\alpha_0 \\ll 1$ (or even $\\alpha_0 \\ll 0.1$) such that any single observation of a next-state transition for any state-action pair results in a (nearly) Dirac delta distribution with all mass concentrated on the observed next state. Then, with any prior over the MDP transition function (my understanding is that the authors already assumed access to the prior $p(\\mathbf{K})$), one can obtain the exact Bayesian posterior via Dirichlet-categorical conjugacy. From this Dirichlet posterior distribution, it seems like one should be capable of inducing posterior beliefs over $\\mathbf{K}$ by simply iterating through all state-action pairs and assigning $\\tilde{k}_i$ appropriately for each state if all successor states for each action are known or not. If my reasoning up to this point is correct, it would seemingly obviate Section 3.1 as well as any subsequent references to challenges incurred due to the use of the Gibbs sampling procedure. Meanwhile, it seems like the resulting solution would bear some resemblance to the shortest path computation described in L177-186 while accounting for the per-transition probabilities under the Dirichlet prior/posterior.\n\t\t\t- The authors assume that there is some parameter $q$ used to model uncertain transitions. Presumably this parameter is in the unit interval $q \\in [0,1]$, but it seems quite bizarre that it is just a single value used for all transitions. Again, maybe this is just a convenient yet crude modeling choice based on representing epistemic uncertainty in $\\mathbf{K}$ directly, rather than over the MDP transition function itself. \n\t\t* Minor\n\t\t\t- The presentation of RL and Bayesian RL in Section 2 is somewhat lacking relative to what a machine learning paper might offer. Of course, space is always at a premium and (related to my subsequent comments) this is perhaps a deliberate choice for a paper not so focused on a machine learning audience.\n\n+ Clarity\n\t- Strengths\n\t\t- The paper is clearly written and reasonably structured.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- The construction blockage on the way to the airport example used in the introduction doesn't really make much sense in a fully-observable environment. An external observer receiving the sequence of states encountered by the driver/agent should presumably get to observe the unexpected construction blockage just as the agent did, giving an immediate understanding of why the detour in the airport route occcurs.\n\t\t\t- I'm guessing there's a typo in the notation for $\\mathcal{E}^k$ (L129) since $\\tilde{k}_j$ isn't defined; maybe $s'$ is supposed to be $s'_j$, although I don't think this captures the idea that the successor state for each possible action executed from $s_i$ is known.\n\t\t\n\n+ Originality\n\t- Strengths\n\t\t- Overall, the approach taken by this paper seems novel although I am not familiar enough with the cognitive science literature and existing models of human exploration to evaluate the novelty of the work with certainty.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- The exposition of L124-138 is reminiscent of the classic E^3 algorithm (Kearns & Singh, 1998) but with a Bayesian twist. It would be nice for the authors to draw that connection, which would resonate quite strongly with RL researchers who read this paper.\n\n+ Significance\n\t- Strengths\n\t\t- While these results may be insightful for a cognitive science audience, there seems to be little or no insight communicated for a machine learning and, in particular, reinforcement learning (RL) audience. I list this as a strength because, based on my reading of the paper, this is the authors' intention and is meant to be seen as a feature, not a bug.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- A caveat of the strength comment directly above is that, while this paper may offer excellent cognitive science contributions, it seems like a rather poor fit for a machine learning venue like NeurIPS. Presumably, this paper at NeurIPS would intend to draw in RL researchers, in which case, there is a glaring gap in the exposition for what computational RL researchers should take away from these results. \n\t\t* Minor\n\t\t\t- N/A\n\t\t\t\n\t\t\n+ Final Remarks\n\t- While this paper scores highly on clarity and while I have no major critiques with regards to originality, I have (potentially) identified some major issues with this paper on the axis on technical quality. Putting those issues aside, the more glaring obstacle is an issue of significance; this paper reads more as a cognitive science paper than a machine learning paper, with very little contribution towards the latter. I would highly encourage the authors to resubmit this work (perhaps without any modification) to a cognitive science venue where it will have a much higher chance of being encountered by a cognitive science audience who would appreciate these contributions and findings in human experiments.", "was_revised": true, "final_justification": "After the discussion period, I know have clarity into the exact problem the authors endeavored to solve, where an initial prior or knowledge state must be inferred based on trajectory observations where the underlying strategy employed by an agent to map initial beliefs onto the observed behavior is unknown. To address this problem via Bayesian inference, the authors posit a particular planning model and corresponding likelihood function (Section 3.2) from which they may perform posterior inference over the agents initial/prior state of knowledge about the world.\n\nEmpirical results affirm that this proposed model for inference is more reliable than a diverse collection of baseline computational models. Additional experiments also study the robustness of the model to inaccuracies in the authors' choice of underlying planning model.\n\nWhile I am not in the best position to evaluate this work on the axis of originality, I suspect is that this work would score highly on this axis provided that the baseline methods are reasonably chosen. Their choices seem reasonably to my eyes and my impression from the other reviewers' comments is that this was not a point of contention. As for significance, I'll defer to other reviewers with more experience on assessing computational models for cognitive science, which is outside my area of expertise.", "rebuttal": "We thank the reviewer for the detailed comments and the opportunity to clarify our contributions and modeling choices. We believe most concerns stem from a misunderstanding, and we are happy to clarify them below.\n\n## Dirichlet beliefs for exact inference about $K$\nWe appreciate the reviewer's thoughtful suggestion to use a Dirichlet prior over transitions, which is indeed the classical Bayesian RL approach for modeling planning under transition uncertainty. Our main goal, however, is not to model how the agent learns about the environment, but rather to infer what the agent already knew at the start of a given trajectory, which makes it an inverse problem.\n\nIn the reviewer's suggested formulation, our goal would be to infer the prior (which we term \"knowledge\") about state transitions based on a given trajectory, which the reviewer proposed to set to $\\alpha_0 \\ll 1$. This would require inference over the space of prior distributions, making the posterior a distribution over Dirichlet parameters. This inference problem cannot be addressed by conjugacy as the likelihood non-trivially depends on the initial knowledge via the agent's policy. With this formulation, it is not even clear how the agent's policy under the Dirichlet belief can be computed, which is required for evaluating the likelihood.\n\nTo avoid this complexity, we consider a simplified model: Instead of Dirichlet beliefs for each transition, we assume binary knowledge for each state: each state is either fully known or unknown (with respect to its transitions).\nIn addition to the lower-dimensionality of the representation, this model is structurally advantageous for computation: First, since our belief about the agent's knowledge is modeled as Bernoulli random variables, we can evaluate the conditional distributions of one belief variable, given all the others, enabling the use of Gibbs sampling. Second, we can efficiently solve the planning problem by modeling it as a shortest path problem. Third, the combination of Gibbs sampling and shortest path formulation with dynamic shortest path algorithms ($D^*$ lite) can be leveraged to reuse intermediate results in the sampling process.\n\nIn summary:\n- The prior $p(K)$ is over the agent’s initial knowledge $K$.\n- The likelihood depends on the agent’s policy given that knowledge, which involves solving a planning problem.\n- This planning problem depends non-trivially on the structure of $K$, making exact Bayesian inference intractable.\n- We use Gibbs sampling not because information has been discarded, but because evaluating the posterior over knowledge states requires marginalizing over a large space.\n\n## Use of a single parameter $q$\nThe choice of a single parameter $q$ to model the probability of traversability of an unknown transition results from the simplified modeling choice that each state is either known (and its transitions are deterministic) or unknown (and transitions succeed with probability $q$). While this is a simplification, it is sufficient for many cognitive modeling tasks where we are interested in whether a subject takes into account certain information or not. Extending the model to allow an individual value for each transition would lead to the Dirichlet model again, which is not tractable. Nevertheless, in some cases, we can lift the restriction of a single $q$ to a certain degree, for example, when using different values for different structures in the maze, such as corridors and junctions.\n\n## Airport Example\nWe appreciate the feedback on the airport example in the introduction. The purpose of the example is not to argue that the environment is partially observable, but rather to illustrate why inferring the agent's prior knowledge is crucial for interpreting behavior. The external observer has access to the complete environment, including the construction blockage. However, the agent's detour into the blocked road only makes sense in this environment if we assume that the agent did not know about the construction in advance. This illustrates the type of inference our method performs.\n\n## Typo in line 129\nThank you for pointing this out. Indeed, $s'$ should be $s_j$ to correctly reference the knowledge of that state. It captures the idea that either the predecessor or successor must be known to know a transition.\n\n## Connection to $E^3$ algorithm\nWe appreciate the reviewer's insight regarding the connection to the $E^3$ algorithm. Our planning model indeed shares the idea of separating known and unknown states, but there are also important differences: In $E^3$, unknown states lead to an absorbing state with maximal reward, which incentivizes exploration. In our model, in contrast, unknown states are traversable with probability $q$, while with probability $1 - q$, the agent stays in the same state. This leads to uncertainty aversion, rather than exploration. We will include this comparison in the revised version to help readers better contextualize our approach.\n\n## Suitability for NeurIPS\nWe respectfully disagree with the claim that our work is a poor fit for NeurIPS. NeurIPS has long welcomed interdisciplinary contributions that combine machine learning with cognitive science, neuroscience, and related fields. As stated in the call for papers, topics include \"Neuroscience and cognitive science (e.g., neural coding, brain-computer interfaces)\" alongside \"Reinforcement learning (e.g., decision and control, planning, hierarchical RL, robotics)\". Our work lies at the intersection of reinforcement learning, Bayesian inference, and cognitive modeling. Several recent NeurIPS papers (see references [1-4] below) tackle similar questions on models for navigation and control. We therefore believe it contributes meaningfully to the NeurIPS community.\n\n## References\n[1] Zhi-Xuan, T., Mann, J., Silver, T., Tenenbaum, J., & Mansinghka, V. (2020). Online Bayesian goal inference for boundedly rational planning agents. Advances in neural information processing systems, 33, 19238-19250.\n\n[2] Binz, M., & Schulz, E. (2022). Modeling human exploration through resource-rational reinforcement learning. Advances in neural information processing systems, 35, 31755-31768.\n\n[3] Chandra, K., Chen, T., Li, T. M., Ragan-Kelley, J., & Tenenbaum, J. (2023). Inferring the future by imagining the past. Advances in Neural Information Processing Systems, 36, 21196-21216.\n\n[4] Damiani, F., Anzai, A., Drugowitsch, J., DeAngelis, G., & Moreno Bote, R. (2024). Stochastic optimal control and estimation with multiplicative and internal noise. Advances in Neural Information Processing Systems, 37, 123291-123327."}, {"review_id": "uP2xLH2u3G", "reviewer": "Reviewer_Sttp", "summary": "This paper presents a Bayesian inference approach to estimate an agent’s latent knowledge about the environment based on observed navigation trajectories. The approach integrates partial environmental knowledge (known state transitions) into an approximate navigation model, leveraging Bayesian inference and dynamic graph algorithms (specifically D* lite) for computational efficiency. The authors validate their approach through both simulated data and human-generated data.", "questions": "1. Can the authors clarify situations or conditions under which the proposed Bayesian inference method provides significant practical usability improvements over simpler models, such as the purely optimistic approach?\n2. Can the authors clarify the precise meaning of \"low-dimensionality\" as used throughout the manuscript? Providing a threshold or clearer contextual understanding would improve clarity and traceability.\n3. Given the similarities to graph learning, can the authors explicitly differentiate why their Bayesian inference approach is preferable or more beneficial compared to standard graph learning techniques?\n4. In section 2.2, BAMDPs are described with uncertainty limited to transitions. Could the authors reword this section to reflect that BAMDPs more generally encapsulate uncertainty over model parameters, explicitly stating the focus here on transition uncertainty?\n5. The assumption that states become \"known\" upon visitation resembles incremental state revelation in MDPs. Could the authors clarify if \"knowledge\" specifically refers to edges/transitions and explain the rationale behind this choice?\n6. Why was KL divergence specifically chosen as the metric for comparing policy similarity in the parking evaluation?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper introduces a novel Bayesian inference framework for inferring agents' latent knowledge in navigation tasks, providing a clear contribution to existing methodologies. The integration of Bayesian inference with dynamic graph algorithms (D* lite) notably improves computational efficiency, making the approach feasible for more scenarios. The empirical validation using both simulated and real human experimental data adds significant value, demonstrating the method's potential for practical applicability and for understanding human decision-making under uncertainty.\n\nWeaknesses:\nWhile the paper is clearly structured, several areas require further clarification and elaboration. \n- The definition of \"low-dimensionality\" and the clarity around the specific types of uncertainty considered in BAMDPs need refinement.\n- The problem framing could be clarified regarding its similarity to graph learning problems. \n- The complexity claims lack explicit computational complexity analyses, and there is limited clarity regarding scalability limitations relative to alternative methods. \n- Statistical analyses and detailed methodological justifications, particularly regarding evaluation measures (e.g., KL divergence) and visual representations, also need improvement. \n- Another potential limitation is the practical usability of the method compared to simpler methods (e.g., purely optimistic approaches). Although the proposed method demonstrates improved log-likelihood compared to a random baseline, simpler methods (OPT) also show improvements. Clarifying when we might expect practical usability benefits of this Bayesian inference approach over simpler alternatives is warranted.\n\nOther comments:\nThe paper would greatly benefit from ensuring figures are introduced and referenced in a logical and timely manner within the text, as early introductions without immediate explanations currently disrupt readability. Additionally, the rationale behind visual representations, such as connecting data points in Figures 1 and 4, should be explicitly stated or reconsidered in favor of clearer visualizations like box or violin plots. Referencing statistical analyses from the appendix within the main text, particularly when discussing results that rely heavily on distributional comparisons, would substantially enhance clarity and interpretability. The labeling in Figure 3 is somewhat unclear, making it difficult to immediately interpret. Additionally, the caption currently includes extensive detail that could be more effectively integrated into the main body of the text to improve clarity and readability.", "was_revised": true, "final_justification": "The authors have addressed most of my initial concerns, and their clarifications on scope, dimensionality, and computational complexity were helpful. While I still view the contribution as modest and see it as a borderline accept, I have no strong objections to acceptance and think the strengths slightly outweigh the weaknesses.", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, as well as for the positive evaluation of our contribution. Below we address each of the reviewer's points and hope these clarify the reviewer's questions.\n\n## Clarification of \"low-dimensionality\"\n\nIn our context, \"low-dimensionality\" refers to settings where the number of latent knowledge configurations (i.e., belief states) is small enough to allow for exact Bayesian inference. Although we cannot define a universal threshold, in practice this typically means fewer than ~5 states in the underlying MDP. For example, in Bayesian inverse planning (ref. [5,6] in the paper), the food truck example involves 3 trucks at 3 locations, resulting in only 6 combinations = belief states. We also use this terminology to describe BEETLE. This method is applied to MDPs with up to 9 states, but it only solves the (forward) planning problem for a BAMDP, which is only a part of the knowledge inference problem. In contrast, we applied our method to problems with up to 169 states, corresponding to a latent belief space of size $2^{169}$. We will clarify these distinctions in the final version of the paper to make our use of the word \"low-dimensional\" more precise.\n\n## Relation to graph learning\nAs discussed in the paper, deterministic planning problems are equivalent to shortest path problems. Our approach can therefore be interpreted as learning properties about the given graph (i.e., the knowledge variables). However, our approach differs from standard graph learning, which typically reconstructs the actual environment graph from data. Instead, our goal is to reason over an agent's subjective knowledge of the graph's edges. This requires modeling both partial knowledge and adaptive planning under uncertainty. Standard graph learning approaches typically do not capture such subjective, temporally-updated knowledge states. We will use the additional page of the camera-ready version to include this relation in the final version.\n\n## Computational complexity\nWe agree that a more detailed discussion of complexity provides additional value. For exact inference, as in Bayesian inverse planning, we must consider all $2^N$ combinations of values of $K$, where $N$ is the number of states. For each configuration, computing the agent's policy and likelihood results in a complexity of $O(2^n \\cdot C_\\text{planning})$. Planning in the belief space using value iteration has complexity $O(|S|^2 \\cdot |A| / \\epsilon) = O(2^{2N} \\cdot 4 / \\epsilon)$, with $\\epsilon$ the precision of value iteration, since the belief MDP has $2^N$ states. Additionally, planning has to be performed up to $N$ times for each configuration, as the agent is assumed to replan along the trajectory (and then all fields have been visited, including the goal). The results in a total complexity for inference of $O(N \\cdot 2^{3N} / \\epsilon)$.\n\nIn contrast, our Gibbs-sampling-based approach scales linearly with the number of variables: we sample each $K_i$ conditioned on all others, leading to a complexity of $O(L \\cdot N \\cdot C_\\text{planning})$, where $L$ is the number of samples. Our planning approximation avoids the complete belief space and uses the shortest path algorithm $D^*$ lite, which has worst-case complexity similar to Dijkstra, $O(|E| + |V| \\log |V|) = O(4 \\cdot N + N \\log N) = O(N \\log N)$. Similar to Bayesian inverse planning, planning needs to be done $N$ times per sample. Therefore, the total cost is $O(L \\cdot N \\cdot N \\cdot N \\log N) = O(L \\cdot N^3 \\cdot \\log N)$. In the final version of the paper, we will complement our runtime comparison with this analysis.\n\n## Evaluation measures and visualizations\nWe used the KL divergence in the parking evaluation to quantify how closely different planning approximations match the optimal Bayesian planner and to measure their influence on the inference results. We could use this metric only for this problem, as it is one of the few cases where the exact Bayesian-optimal policy under epistemic uncertainty is computable. For the gridworld experiment, we instead used the log-likelihood of human trajectories under each model, as ground truth policies are not available and we are given experimental data.\n\nRegarding visualizations: the grey lines connect data points from the same trajectory. We agree that the black line connecting medians may be misleading and will remove it. We will use the additional space in the camera-ready version to expand the description on this.\n\n## Practical usability compared to simpler methods\nThis may reflect a misunderstanding. The only existing simpler method is to do exact planning and/or inference (i.e., Bayesian inverse planning), which is only feasible for very small environments (such as the parking task). Alternatively, one could craft rules by hand, as we did for the baselines in the gridworld tasks, where we assumed no knowledge in case of turn-arounds. In our experiments, we use the optimistic planning model as an ablation to justify a potential cost for uncertainty, but it does not make the inference problem easier: We still need to solve the high-dimensional inference problem and find in each iteration the shortest path for solving the planning problem approximately. \n\n## Figure placement and referencing statistical analyses\nWe agree that figure placement is not optimal for reading. Due to space constraints, we had to distribute figures across pages. In the final version, due to the extra page, figures will land one page later, which should solve this issue. Additionally, as suggested, we will reference and briefly discuss the statistical analysis results from the appendix to improve clarity.\n\n## Figure 3 labeling and caption\nAs also noted by Reviewer 93XY, we will add a colorbar label to improve clarity. Our intention in using detailed figure captions was to make the paper easier to interpret through visual inspection. If clarity would be improved by integrating these details into the main text, we are happy to make that revision.\n\n## Definition of BAMDPs in Section 2.2\nWe based our definition of BAMDPs on the original work by Duff [1] and the survey by Ghavamzadeh et al. [2]. Both define BAMDPs in terms of uncertainty over transition dynamics. We agree that BAMDPs can be generalized to include uncertainty over other model parameters (e.g., rewards).\n\n## Relations to incremental state revelation\nWe agree with the reviewer that our formulation, in which states become \"known\" upon visitation, is indeed similar to incremental state revelation mechanisms used in exploration-focused RL algorithms such as $E^3$. We clarified this connection also in the comment to reviewer 93XY. The most important differences are that we consider the inverse problem and that, in our case, model uncertainty leads to aversion. We will make this connection clearer in the final version.\n\n## Clarifying \"knowledge\" as edges/transitions\nYes, \"knowledge\" in our model refers to information about the transitions (edges) in the environment. In many cognitive tasks, the goal (e.g., a target location) is explicit and known. The challenge lies in discovering or reasoning about how to reach them. Modeling knowledge over transitions allows us to infer what structural information agents incorporated into their plans. This enables us, for example, to investigate how topology influences awareness or memory in human planning. With that, our approach complements approaches like inverse RL, which aims at inferring goals. We will clarify this in the revised text.\n\n## References\n[1] M. Duff (2002). Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst\n\n[2] M. Ghavamzadeh, S. Mannor, J. Pineau, & A. Tamar (2015). Bayesian reinforcement learning: A survey. Foundations and Trends in Machine Learning, 8(5-6), 359-483"}, {"review_id": "ISOuzyU3Tm", "reviewer": "Reviewer_cJ9W", "summary": "This paper proposes a Bayesian approach for inferring the partial knowledge of agents in navigation scenarios. The approach posits a model where the transitions for a state are either known or unknown. Inference combines Gibbs sampling and D\\* lite. Experiments in grid worlds and a parking scenario compare the proposed approach to a number of baselines with both simulated and real human data.", "questions": "1. Can you clarify how $q$  was set in experiments?  \n2. Can you further justify the spatial prior?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**Writing.** The paper is very clear and overall very well written. The organization is good and I did not find major typos or grammatical issues. It was also an enjoyable read.\n\n**Systematic experiments**. The experiments are careful and systematic. There were several points while reviewing that I had a question that was then answered. Although the environments are relatively simple, they are similar to those used in related work.\n\n**Novelty.** The paper does seem sufficiently novel, although the scope is a bit narrow (see below).\n\n**Nice approach.** I like the idea of using D\\* lite and I appreciate the empirical justification showing that it is much faster than using A\\*.\n\n\n## Weaknesses\n\n**Scope.** While the paper does go beyond prior work by allowing for partial knowledge, it is also more narrowly scoped than prior work because of the focus on navigation and spatial tasks. For example, the approach of Zhi-Xuan et al. (2020), which I believe is the most similar prior work, also uses a combination of Bayesian inference and heuristic search, but they use a more general PDDL formalism that does not make any assumptions about navigation or space. The spatial assumption here seems fundamental to the model, which says that  the transition distribution for a state is either fully known or unknown---this makes sense for navigation, but maybe not so much for other environments with partial knowledge.\n\n**Spatial prior.** I do not fully understand the need for the spatial prior as opposed to a uniform prior. Are empirical results worse without the spatial prior? Since the uniform prior is simpler than a spatial one, it would be good to run an ablation. It is also not ideal that the spatial prior introduces another hyperparameter, and it’s not clear how this hyperparameter was selected (I referred to the Appendix A.7 as well).\n\n**Hierarchical Bayesian model.** The paper mentions the possibility that the human subjects may learn about the shared structure of the environments throughout the course of the experiment, but even within a single environment, it is conceivable that subjects are performing a kind of hierarchical inference where they are reasoning, for example, about the global probability of obstacles existing or parking slots being open. For example, looking at Figure 2, that kind of hierarchical inference could lead to the rightmost trajectory, as an alternative to the “optimistic with uncertainty penalization” model that is hypothesized in the paper.\n\n**Minor: lines on unordered plots**. I don’t think the mean lines on the plots in Figures 4 and 5 are informative because there is no ordering relation between the approaches on the x axis. I can understand the motivation for including lines to connect individual participant data in Figure 4, but the mean lines don’t seem to make sense.", "was_revised": true, "final_justification": "My only remaining concern is that the paper's impact may be limited by its narrow scope. But I think the paper's strengths outweigh its weaknesses.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We are pleased that the clarity, novelty, and systematic experimentation of the paper were appreciated. Below, we respond to the reviewer’s specific comments and suggestions.\n\n## Scope and spatial assumptions\nWe agree that our formulation makes use of the spatial structure, which makes it particularly suited for navigation tasks. This structure is useful for two reasons: First, it provides a natural way to define optimistic planning over unknown transitions, enabling efficient inference. Second, it allows the use of heuristic search methods such as D* lite or A*, where spatial distance provides an effective heuristic.\nWhile the referenced work by Zhi-Xuan et al. (2020) uses a more general formalism based on PDDL, their approach also makes use of a spatial heuristic by using A* for planning and to limit search depth in their bounded rational planning model. Thus, both approaches implicitly exploit structured environments, though to different extents.\n\nWhile the spatial domain already offers a wide range of applications for our approach, we agree that extending our method to non-spatial domains could be an interesting and important future direction. We will note this in the revised future work section.\n\n## Spatial prior\nThe spatial prior captures the intuition that knowledge (or awareness/memory) of neighboring fields is likely to be correlated. For example, if a human knows one field in a corridor, they are likely to know neighboring fields as well. This prior allows the incorporation of the structure of the agent's knowledge without significantly increasing computational cost, as its computation cost is negligible compared to planning during inference.\n\nImportantly, our approach does not depend on the spatial prior. We have now conducted an ablation using a uniform prior and found that results were qualitatively very similar, but the relative improvements between the baseline and our method were on average half as large (average factor 1.991). This suggests that while the prior has some influence, it is not essential for the effectiveness of our method. We will include this ablation in the appendix of the final version. Nevertheless, in Bayesian inference, providing the flexibility to specify prior knowledge is conceptually important to ensure meaningful posterior beliefs.\n\nThe hyperparameter of the prior was selected manually to produce moderately connected knowledge regions (neither fully fragmented nor overly clustered). The same hyperparameters were used both for generating the ground-truth knowledge values in the experiment (the fields initially displayed to the participants) and for inference. We will clarify this procedure in the appendix.\n\n## Hierarchical Bayesian Inference\nWe fully agree with the reviewer that human participants may perform higher-level reasoning about the structure of the tasks across and within environments, such as estimating the likelihoods of dead ends. As mentioned in our future work section, such hierarchical learning may account for the variability in our experimental data, including trajectories that appear irrational at first glance.\n\nOur work provides a foundational step to understand observed learning behavior by making inference about latent knowledge tractable for a relatively simple learning model. Hierarchical learning would considerably increase complexity, but could be a promising direction for future work. We will add this point to the future work section. This suggestion also aligns well with our broader research goals, which include investigating how subjects generalize knowledge across environments and whether parameters like $q$ (the belief over transition probability) change across maze topologies.\n\n## Lines in Figure 4 and 5\nWe appreciate the reviewer's comment that the black line connecting medians may be misleading, which was also noted by reviewer Sttp. We will remove it in the final version.\n\n## Choice of hyperparameter $q$\nFor the gridworld experiments, we set $q=0.5$ without tuning, to model an equal chance for success and failure for unknown transitions. In the parking experiment, we used $q=0.3$, which corresponds to the true probability of a parking spot being available (as used for generating the environments). This value was manually chosen to produce meaningful behavior when using the optimal policy: the agent should neither always park at the first nor last opportunity. We will clarify this setting in the final version."}]

Author rebuttal:

Meta-review: This paper proposes a method for inferring an agent's knowledge of environment states—distinguishing between known and "unknown" areas—from observed navigation paths. Unlike standard inverse reinforcement learning approaches, it explicitly accounts for agent uncertainty. The method employs Gibbs sampling to perform Bayesian inference over state knowledge. The paper addresses an important problem and is clearly written, with experiments that are thoughtfully designed and rigorously executed. However, reviewers initially expressed concerns regarding the perceived novelty and scope of the contribution, suggesting that the work might appear somewhat narrow or incremental relative to the existing literature. Initial questions about the model motivation and spatial prior justification were effectively addressed in the rebuttal, leading to a more favorable overall evaluation of the paper.
Decision: Accept (poster)