Title: What do you know? Bayesian knowledge inference for navigating agents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian reinforcement learning

computational cognitive science

human behavior

inverse reinforcement learning

theory of mind
Abstract: Human behavior is characterized by continuous learning to reduce uncertainties about the world in pursuit of goals. When trying to understand such behavior from observations, it is essential to account for this adaptive nature and reason about the uncertainties that may have led to seemingly suboptimal decisions. Nevertheless, most inverse approaches to sequential decision-making focus on inferring cost functions underlying stationary behavior or are limited to low-dimensional tasks. In this paper, we address this gap by considering the problem of inferring an agent's knowledge or awareness about the environment based on a given trajectory. We assume that the agent aims to reach a goal in an environment they only partially know, and integrates new information into their plan as they act. We propose a Bayesian approach to infer their latent knowledge state, leveraging an approximate navigation model that optimistically incorporates partial information while accounting for uncertainty. By combining sample-based Bayesian inference with dynamic graph algorithms, we achieve an efficient method for computing posterior beliefs about the agent's knowledge. Empirical validation using simulated behavioral data and human data from an online experiment demonstrates that our model effectively captures human navigation under uncertainty and reveals interpretable insights into their environmental knowledge.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a Bayesian approach for inferring the partial knowledge of agents in navigation scenarios. The approach posits a model where the transitions for a state are either known or unknown. Inference combines Gibbs sampling and D\\* lite. Experiments in grid worlds and a parking scenario compare the proposed approach to a number of baselines with both simulated and real human data.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**Writing.** The paper is very clear and overall very well written. The organization is good and I did not find major typos or grammatical issues. It was also an enjoyable read.\n\n**Systematic experiments**. The experiments are careful and systematic. There were several points while reviewing that I had a question that was then answered. Although the environments are relatively simple, they are similar to those used in related work.\n\n**Novelty.** The paper does seem sufficiently novel, although the scope is a bit narrow (see below).\n\n**Nice approach.** I like the idea of using D\\* lite and I appreciate the empirical justification showing that it is much faster than using A\\*.\n\n\n## Weaknesses\n\n**Scope.** While the paper does go beyond prior work by allowing for partial knowledge, it is also more narrowly scoped than prior work because of the focus on navigation and spatial tasks. For example, the approach of Zhi-Xuan et al. (2020), which I believe is the most similar prior work, also uses a combination of Bayesian inference and heuristic search, but they use a more general PDDL formalism that does not make any assumptions about navigation or space. The spatial assumption here seems fundamental to the model, which says that  the transition distribution for a state is either fully known or unknown---this makes sense for navigation, but maybe not so much for other environments with partial knowledge.\n\n**Spatial prior.** I do not fully understand the need for the spatial prior as opposed to a uniform prior. Are empirical results worse without the spatial prior? Since the uniform prior is simpler than a spatial one, it would be good to run an ablation. It is also not ideal that the spatial prior introduces another hyperparameter, and it’s not clear how this hyperparameter was selected (I referred to the Appendix A.7 as well).\n\n**Hierarchical Bayesian model.** The paper mentions the possibility that the human subjects may learn about the shared structure of the environments throughout the course of the experiment, but even within a single environment, it is conceivable that subjects are performing a kind of hierarchical inference where they are reasoning, for example, about the global probability of obstacles existing or parking slots being open. For example, looking at Figure 2, that kind of hierarchical inference could lead to the rightmost trajectory, as an alternative to the “optimistic with uncertainty penalization” model that is hypothesized in the paper.\n\n**Minor: lines on unordered plots**. I don’t think the mean lines on the plots in Figures 4 and 5 are informative because there is no ordering relation between the approaches on the x axis. I can understand the motivation for including lines to connect individual participant data in Figure 4, but the mean lines don’t seem to make sense.\n\nQUESTIONS:\n1. Can you clarify how $q$  was set in experiments?  \n2. Can you further justify the spatial prior?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Bayesian inference approach to estimate an agent’s latent knowledge about the environment based on observed navigation trajectories. The approach integrates partial environmental knowledge (known state transitions) into an approximate navigation model, leveraging Bayesian inference and dynamic graph algorithms (specifically D* lite) for computational efficiency. The authors validate their approach through both simulated data and human-generated data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper introduces a novel Bayesian inference framework for inferring agents' latent knowledge in navigation tasks, providing a clear contribution to existing methodologies. The integration of Bayesian inference with dynamic graph algorithms (D* lite) notably improves computational efficiency, making the approach feasible for more scenarios. The empirical validation using both simulated and real human experimental data adds significant value, demonstrating the method's potential for practical applicability and for understanding human decision-making under uncertainty.\n\nWeaknesses:\nWhile the paper is clearly structured, several areas require further clarification and elaboration. \n- The definition of \"low-dimensionality\" and the clarity around the specific types of uncertainty considered in BAMDPs need refinement.\n- The problem framing could be clarified regarding its similarity to graph learning problems. \n- The complexity claims lack explicit computational complexity analyses, and there is limited clarity regarding scalability limitations relative to alternative methods. \n- Statistical analyses and detailed methodological justifications, particularly regarding evaluation measures (e.g., KL divergence) and visual representations, also need improvement. \n- Another potential limitation is the practical usability of the method compared to simpler methods (e.g., purely optimistic approaches). Although the proposed method demonstrates improved log-likelihood compared to a random baseline, simpler methods (OPT) also show improvements. Clarifying when we might expect practical usability benefits of this Bayesian inference approach over simpler alternatives is warranted.\n\nOther comments:\nThe paper would greatly benefit from ensuring figures are introduced and referenced in a logical and timely manner within the text, as early introductions without immediate explanations currently disrupt readability. Additionally, the rationale behind visual representations, such as connecting data points in Figures 1 and 4, should be explicitly stated or reconsidered in favor of clearer visualizations like box or violin plots. Referencing statistical analyses from the appendix within the main text, particularly when discussing results that rely heavily on distributional comparisons, would substantially enhance clarity and interpretability. The labeling in Figure 3 is somewhat unclear, making it difficult to immediately interpret. Additionally, the caption currently includes extensive detail that could be more effectively integrated into the main body of the text to improve clarity and readability.\n\nQUESTIONS:\n1. Can the authors clarify situations or conditions under which the proposed Bayesian inference method provides significant practical usability improvements over simpler models, such as the purely optimistic approach?\n2. Can the authors clarify the precise meaning of \"low-dimensionality\" as used throughout the manuscript? Providing a threshold or clearer contextual understanding would improve clarity and traceability.\n3. Given the similarities to graph learning, can the authors explicitly differentiate why their Bayesian inference approach is preferable or more beneficial compared to standard graph learning techniques?\n4. In section 2.2, BAMDPs are described with uncertainty limited to transitions. Could the authors reword this section to reflect that BAMDPs more generally encapsulate uncertainty over model parameters, explicitly stating the focus here on transition uncertainty?\n5. The assumption that states become \"known\" upon visitation resembles incremental state revelation in MDPs. Could the authors clarify if \"knowledge\" specifically refers to edges/transitions and explain the rationale behind this choice?\n6. Why was KL divergence specifically chosen as the metric for comparing policy similarity in the parking evaluation?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n+ Summary & Contributions\n\t- This paper tackles the problem of inferring agent knowledge about the environment based on an input trajectory of agent behavior.\n\t- Opting to directly model whether individual states are either \"known\" or \"unknown\" explicitly, the authors employ Gibbs sampling for performing approximate Bayesian inference to determine which transitions of the environment are known to the agent given a trajectory of behavior.\n\t- Empirical results against a variety of competing models of human participants' planning based on partial knowledge of the transition function confirm that the proposed model is competitive at providing a plausible explanation of human planning behavior and accurate inferences about the environment.\n\nSTRENGTHS AND WEAKNESSES:\n+ Quality\n\t- Strengths\n\t\t- N/A\n\t- Weaknesses\n\t\t* Major\n\t\t\t- I suspect I may be misunderstanding something fundamental about the problem the authors are trying to solve. A state $s \\in \\mathcal{S}$ of a finite MDP is considered known (with $\\tilde{k} = 1$) if, for each possible action $a \\in \\mathcal{A}$, the agent knows the exact successor state $s' = \\mathcal{T}(s,a)$. It seems like the exposition of Section 3.1 highlights the fact that only maintaining direct beliefs about $\\mathbf{K}$ discards too much information for tractable, exact Bayesian inference thereby necessitating the use of a subsequent Gibbs sampling procedure. Alternatively, since the MDP is finite and deterministic, why not proceed in the standard way by maintaining beliefs about the full MDP transition function as a collection of $|\\mathcal{S}||\\mathcal{A}|$ independent Dirichlet distributions (one for each possible next-state transition distribution)? Since the transition dynamics are deterministic, a reasonable prior would, for instance, be uniform with an initial parameter $\\alpha_0 \\ll 1$ (or even $\\alpha_0 \\ll 0.1$) such that any single observation of a next-state transition for any state-action pair results in a (nearly) Dirac delta distribution with all mass concentrated on the observed next state. Then, with any prior over the MDP transition function (my understanding is that the authors already assumed access to the prior $p(\\mathbf{K})$), one can obtain the exact Bayesian posterior via Dirichlet-categorical conjugacy. From this Dirichlet posterior distribution, it seems like one should be capable of inducing posterior beliefs over $\\mathbf{K}$ by simply iterating through all state-action pairs and assigning $\\tilde{k}_i$ appropriately for each state if all successor states for each action are known or not. If my reasoning up to this point is correct, it would seemingly obviate Section 3.1 as well as any subsequent references to challenges incurred due to the use of the Gibbs sampling procedure. Meanwhile, it seems like the resulting solution would bear some resemblance to the shortest path computation described in L177-186 while accounting for the per-transition probabilities under the Dirichlet prior/posterior.\n\t\t\t- The authors assume that there is some parameter $q$ used to model uncertain transitions. Presumably this parameter is in the unit interval $q \\in [0,1]$, but it seems quite bizarre that it is just a single value used for all transitions. Again, maybe this is just a convenient yet crude modeling choice based on representing epistemic uncertainty in $\\mathbf{K}$ directly, rather than over the MDP transition function itself. \n\t\t* Minor\n\t\t\t- The presentation of RL and Bayesian RL in Section 2 is somewhat lacking relative to what a machine learning paper might offer. Of course, space is always at a premium and (related to my subsequent comments) this is perhaps a deliberate choice for a paper not so focused on a machine learning audience.\n\n+ Clarity\n\t- Strengths\n\t\t- The paper is clearly written and reasonably structured.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- The construction blockage on the way to the airport example used in the introduction doesn't really make much sense in a fully-observable environment. An external observer receiving the sequence of states encountered by the driver/agent should presumably get to observe the unexpected construction blockage just as the agent did, giving an immediate understanding of why the detour in the airport route occcurs.\n\t\t\t- I'm guessing there's a typo in the notation for $\\mathcal{E}^k$ (L129) since $\\tilde{k}_j$ isn't defined; maybe $s'$ is supposed to be $s'_j$, although I don't think this captures the idea that the successor state for each possible action executed from $s_i$ is known.\n\t\t\n\n+ Originality\n\t- Strengths\n\t\t- Overall, the approach taken by this paper seems novel although I am not familiar enough with the cognitive science literature and existing models of human exploration to evaluate the novelty of the work with certainty.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- N/A\n\t\t* Minor\n\t\t\t- The exposition of L124-138 is reminiscent of the classic E^3 algorithm (Kearns & Singh, 1998) but with a Bayesian twist. It would be nice for the authors to draw that connection, which would resonate quite strongly with RL researchers who read this paper.\n\n+ Significance\n\t- Strengths\n\t\t- While these results may be insightful for a cognitive science audience, there seems to be little or no insight communicated for a machine learning and, in particular, reinforcement learning (RL) audience. I list this as a strength because, based on my reading of the paper, this is the authors' intention and is meant to be seen as a feature, not a bug.\n\t- Weaknesses\n\t\t* Major\n\t\t\t- A caveat of the strength comment directly above is that, while this paper may offer excellent cognitive science contributions, it seems like a rather poor fit for a machine learning venue like NeurIPS. Presumably, this paper at NeurIPS would intend to draw in RL researchers, in which case, there is a glaring gap in the exposition for what computational RL researchers should take away from these results. \n\t\t* Minor\n\t\t\t- N/A\n\t\t\t\n\t\t\n+ Final Remarks\n\t- While this paper scores highly on clarity and while I have no major critiques with regards to originality, I have (potentially) identified some major issues with this paper on the axis on technical quality. Putting those issues aside, the more glaring obstacle is an issue of significance; this paper reads more as a cognitive science paper than a machine learning paper, with very little contribution towards the latter. I would highly encourage the authors to resubmit this work (perhaps without any modification) to a cognitive science venue where it will have a much higher chance of being encountered by a cognitive science audience who would appreciate these contributions and findings in human experiments.\n\nQUESTIONS:\nMy questions for the authors should be clear from the comments above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address the problem of inferring a latent decision making model from observed behaviours of another agent. The problem differs from standard inverse RL or inverse optimal control, in that the latent model may contain uncertainty about the true world model, i.e., the dynamics of some states may not be known to the observed agent. The problem is addressed by modelling each state of the underlying model as known or unknown in the latent model, and then performing Gibbs sampling to compute a posterior over whether states are known or unknown, with the likelihood being given with respect to the observed policy. The application setting is to recover navigation models from observation.\n\nSTRENGTHS AND WEAKNESSES:\nThe problem in general is interesting, the written text is clear, and I enjoyed reading the paper.\n\nThe major limitation of this paper is that it is not really clear what the purpose of the result is. I understand the overall problem of recovering a latent model to model behaviour, and I like the problem setting that recognises that the acting agent may in fact not have a complete model of the world. However, the restriction to a discrete MDP, and the assumption that the *only* thing the acting agent does not know is the transition dynamics both strike me as relatively artificial, and poorly aligned with inferring real navigation models. (Inferring latent navigation models is actually a very active area of research in spatial psychology. Reading about how traffic engineers recover navigation models is super interesting!) The provided experimental results show the technique works, but are not especially compelling motivation. The underlying techniques (inference by sampling with likelihood computed from the observed trajectory) seems relatively straightforward, so without a clearer connection to a real problem, it's hard to know what the contribution is.\nI was also surprised not to see a comparison to conventional IRL --- this work becomes even more interesting if there is a setting where IRL learns the wrong thing, which leads to poor performance, but the BAMDP setting learns the correct concept.\n\nThe authors list of limitations is fairly good, but these limitations are not trivial, and make me wonder even more if the proposed approach is actual useful. I had similar concerns about scalability. The authors propose using additional information sources such as eye tracking, which I completely agree with as an important step. I think that human navigation models are much more based on perceptual features than on a latent MDP where the human is not sure about the transition dynamics of some of the states.\n\nThe experimental results are reasonable, but the figures are extremely difficult to parse, partly because there are some key features that are not described --- it took quite a while to work out for figure 3 that the more blue a square is, the more likely it is that the human did not know it. I am also surprised at how high variance the posterior is for the experiment data in figure 4 -- the baseline seemed to outperform the BAMDP technique for some worlds.\n\nI am generally inclined to support this paper, but the absence of a connection to a real application or a deep algorithm advance, the contribution of this paper is modest at best.\n\nQUESTIONS:\nHow does this technique compare to conventional IRL on these experiments?\nWhy is the posterior so high variance on the experiment data for figure 4?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}