Modeling Boundedly Rational Agents with Latent Inference Budgets
Authors: Athul Paul Jacob
Abhishek Gupta
Jacob Andreas
Venue: ICLR 2024 poster (2024)
Keywords: neurosymbolic
planning
rationality
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: We study the problem of modeling a population of agents pursuing unknown goals subject to unknown computational constraints. In standard models of bounded rationality, sub-optimal decision-making is simulated by adding homoscedastic noise to optimal decisions rather than actually simulating constrained inference. In this work, we introduce a latent inference budget model (L-IBM) that models these constraints explicitly, via a latent variable (inferred jointly with a model of agents’ goals) that controls the runtime of an iterative inference algorithm. L-IBMs make it possible to learn agent models using data from diverse populations of suboptimal actors. In three modeling tasks—inferring navigation goals from routes, inferring communicative intents from human utterances, and predicting next moves in human chess games—we show that L-IBMs match or outperforms Boltzmann models of decision-making under uncertainty. Moreover, the inferred inference budgets are themselves meaningful, efficient to compute, and correlated with measures of player skill, partner skill and task difficulty.

Published as a conference paper at ICLR 2024
MODELING BOUNDEDLY RATIONAL AGENTS WITH LATENT INFERENCE BUDGETS
Athul Paul Jacob
Abhishek Gupta University of Washington abhgupta@cs. www.:iisssssiingtoo...ooo edu
Jacob Andreas MIT jdaêmit edu
MIT
apjacobemit..
ABSTRACT
We study the problem of modeling population of agents pursuing unknown goals subject unknown computational constraints. In standard models of bounded rationality, sub-optimal decision-making making is simulated by adding homoscedastic noise to optimal decisions rather than explicitly simulating constrained inference. In this work, we introduce latent inference budger model (L-IBM) that models agents' computational constraints explicitly, via latent variable (inferred jointly with model of agents' goals) that controls the runtime of an iterative inference algorithm. L-IBMMs make it possible to learn agent models using data from diverse populations of suboptimal actors. In three modeling tasks tasks-inferrin goals from routes, inferring communicative intents from human utterances, and predicting next moves in human chess games- we show that L-IBMs match outperform Boltzmann models of decision-making under uncertainty. Inferred inference budgets are themselves meaningful, efficient to compute, and correlated with measures of player skill, partner skill and task difficulty.
navigation
INTRODUCTION
Building effective models for multi-agent decision-making- -whtthe cooperative or adversarial- requires understanding other agents' goals and plans. To help a friend navigate new environ- ment, we must first understand where they want to go; to beat an opponent at chess, we must be able to predict their likely next moves. But decision-making, in humans and machines, is subject to computational constraints. Decision-makers often act suboptimally, relying heuristics and ap- proximations to choose their actions. Techniques that do not account for this suboptimality carefully may attribute behavior differing intentions rather than different inference procedures.
How should we interact with agents seeking to accomplish unknown goals subject to unknown computational constraints? In standard models of bounded rationality (Luce, 1959), sub-optimal decision-making is simulated by adding noise to optimal decisions rather than explicitly simulating constrained inference. This results in models that treat agents as uniformly suboptimal in a way that fails to account for sub-optimal inference algorithms or for non-homogenous suboptimality.
In this paper, we describe simple approach for building models of agents given traces of their behavior. Our approach explicitly models agents' "inference budgets", via a latent variable that con- trols the runtime of each agent's 's inference procedure. We show that for agents performing inference using anytime algorithms (algorithms that can be terminated at any point and return approximately correct solutions) inference budgets can be efficiently inferred from example behaviors. A diverse set of multi-agent decision-making rational models of human language production, and Monte Carlo tree search admit imputation of inference budgets in this framework.
-nncluuns-incluuinn graph-based planning algorithms, recursive-
In three diverse agent modeling tasks-inferring navigation goals from routes, inferring commu- nicative intents from human utterances, and predicting subsequent moves in human-human chess matches- we show that our approach matches or outperforms Boltzmann models of decision- making under uncertainty. Moreover, inferred inference budgets themselves meaningful, cor- relating with measures of player skill, partner skill, and task difficulty. Our results show that sub- optimal human decision-making be efficiently modeled with computationally constrained ver-

---

Figure 1: Inferring rewards from boundedly-rational trajectories. The agent will move to the blue star (a), but prefers move toward the orange available (b). When locating the orange star requires solving a harder search problem. however. the agent seeks the blue star instead, indicating that its search abilitics are limited (c). Our proposed approach tomatically infers the budget that the agent uses when planning (d). Knowing this budget, we could perhaps Published as a conference paper at ICLR 2024
sions of standard search algorithms. By doing so, obtain both accurate models of humans' mecisinn-making and informative measures of their inferential capacity.
BACKGROUND AND PROBLEM FORMULATION
We study the problem of modeling one or more agents given given traces of their behavior. In particular, we assume that we observe a collection of trajectories (state-action sequences) produced by agents n² + a acting in a Markov decision process to maximize some reward function (T). Even when (r) is known to agents, inferring optimal actions is often intractable, so agents in the real world will in general approximate optimal behavior subject to some (unknown) computational constraints (which may differ from agent to agent). From this data, we seek to infer agent models и defined terms of (1) estimates R of reward function R (known to agents but not modelers), and (2) descriptions of the computational limitations that govern agents' choice of actions. In other words, we seek to model both what agents wish to do and what agents will actually do in any given state. Fig. I shows a conceptual example: assuming the agent receives different reward for reaching each of the two goals, the three trajectories depicted there cannot be generated by the optimal policy for any reward function, but can be explained by model that only look ahead to a limited number of positions in the maze.
Throughout this paper, we will model agent ac- tions as arising from an approximate infer- ence procedure a(a s; R, that takes as input a reward function and computational budget B. We model agents by inferring values of R and B given the executed trajectories Ti.
(8)
(b)
(c)
The ability to infer goals from suboptimal (and even completely unsuccessful) plans is key human skill, present in children as young as 18 months (Meltzoff, 1995). Computational mod- els of bounded rationality thus have a long his- tory in artificial intelligence, cognitive science, and behavioral economics. But what does this suboptimality look like in practice, and how should we model and infer the inference bud- get 3 simply from observations of behavior?
(d)
Powge
when both
One of the most widely used models of bound- edly rational decision-making is the so-called Boltzmann model (Luce, 1959), in which agents take actions according to
assist this agent by providing a targeted hint (move right) at the beginning of its trajectory.
(a s; R, 3) x exp( R(s,a)} (1)
This equation has number of appealing interpretations, e.g. the policy that achieves target reward while maximizing entropy. It has been used to model not just the selection of actions. but also trajectories, preferences, corrections, and more-see Jeon et al. (2020) for recent survey. More elaborate approaches in this family also predict / conditioned on the current state or action history, making it possible model state-dependent skill (Beliaev al., 2022).
However, Boltzmann models have a significant limitation: the probability of generating an action in Eq. (1) depends only on the true value of that action, and not on the cost of acquiring a high- quality value estimate in the first place. To see why this might be a problem, consider again the trajectories depicted the conceptual example Fig. ((b-c), which differ only in the difficulty of the search problem, and not in the cost of the optimal trajectory at all. A model of boundedly rational decision-makinn makinn with the form of Eq. (1) cannot account for this difference.
There is large body of other approaches on modeling human planning under resource constraints in psychology, economics and in classical AI (Callaway et al., 2022; Russell & Wefald, 1991; Huys et al., 2015; 2012; Camerer et al.. 2004; Griffiths et al., 2019; Boddy & Dean, 1989; van Opheusden et al., 2023, inter alia). Notably, Zhi-Xuan et al. (2020) build an explicit hierarchical Bayesian model of symbolic planning procedure while inferring per-timestep inference budgets.
2

---

Published as a conference paper at ICLR 2024
In general, these approaches make strong assumptions about how planning is performed. Here, seek to develop general framework that avoids strong assumptions about either the functional form of the reward model the algorithmic form of the planning procedure. Related approaches were proposed by Evans & Goodman (2015) and Shah et al. (2019), though with simpler environments inference procedures. The framework we develop be applied to model real-world behavior in tasks as diverse as language generation and chess gameplay.
INFERRING REWARDS AND INFERENCE BUDGETS FROM BEHAVIOR
As motivated in Section 2, our goal is model agents acting to optimize an unknown value function subject to an unknown computational constraint. In practice, we often want to model populations comprising multiple agents or agent sub- cobb-opulations (T m m2..... x with shared reward function R (e.g. winning at chess) but differing computational constraints.
To do so, we assume we have access to collection of trajectories {r}: T. ....... Ti with each collection of trajectories {r], generated by different agent or sub-population We model these trajectories as drawn from the following generative process:
1. at each timestep, agent draws budget 3 from agent-specific prior Prudger (3 17))
2. म. chooses actions according to budget-constrained inference procedure गं (a 8; R², 3)
Because budgets may vary between trajectories, learning model of these agents ultimately consists of learning reward parameters 0 and agent-specific budget-generating parameters y) while marginal- izing over latent budgets themselves. We do so via maximum posteriori inference, optimizing:
arg max 0,7
Σ log [(a| s; arg max re(T) (s,a)ET
Σ log Prodger (3 m)) TTa s; Rg. 3)
(2)
0,7
re(r) (s,A)ET
If T(a | 8; s, 18", 3) is an arbitrary inference algorithm, Eq. (2) might present a challenge: this infer- ence procedure must be run for all possible values of B, which will in general be intractable. Under what circumstances can we optimize this equation efficiently? The key observation in this paper is that if x an anytime inference algorithm (Dean Boddy, 1988), we can evaluate n values of /3 as quickly as we can evaluate one, making this optimization tractable.
Definition 1. An anytime algorithm is one that runs for i timesteps and produces sequence of inference states f2.. fi), where every fo can be computed from fi-1 in (1)) time, and fs can be used to select an action according to some xaa s; R. fi).
As we will see shortly, many canonical inference algorithms used in single- and multi-agent decision-making scenarios have this from. In these cases, rather than letting the budget parame- ter 3 determine noise or suboptimality, we may use it to parameterize the runtime of the agent's s inference procedure itself, writing:
log [a s; mh) = log Σ (Sruntiie nx) TTa s; Rg, f
(3)
where we have denoted the budget Brntiim to indicate that it parameterizes the runtime of the any- time inference algorithm. Crucially-by by connnttion-mmmptinng this sum up some maximum requires no more time than computing its final term.
The remainder of this paper looks at instantiations of this basic modeling framework in three differ- ent domains. In Section 4, we study the problem of inferring navigation goals from maze domain using truncated graph search algorithm. In Section 5, we study rational speech (RSA) for inferring communicative intents from human utterances. Finally, in Section 6, we model human action prediction in chess using Monte-Carlo tree search (MCTS).
SOLVING MAZES WITH TRUNCATED DEPTH-FIRST SEARCH
We begin with a pedagogical example of latent inference budget model applied to a simple, single- agent decision- making task: maze navigation. Agents are placed random position in maze with
3

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
(f)
Figure 2: Examples of the maze task. (a) Example of the value function heuristic applied to each state in the Red indicates low value states and blue indicates high value states. (b)-(f) depicts the example trajectories agents with depth budgets of 1, 2, 5, 20.
five exits. Each exit is a state e, associated with a reward RG. Agents attempt to navigate toward the highest scoring exit by taking navigation actions (north, east, south, west). Here our goal to recover the rewards RG that single agent associates with each exit, along with agent budget parameters 7), given observations of the agent's 's behavior.
4.1 AGENT MODEL
We assume that agents select navigation actions using a heuristic with a known functional form, in which the value of state is approximated
V(8) = Σ, -ls-ell- Ee--ll--iil--
(4)
where ||s s'Il measures the Manhattan distance between pair of states (i.e. maze positions). Intuitively, we model agents as "attending" to each exit in proportion to both its distance and asso- ciated reward. We assume that agents use this heuristic to perform truncated breadth-first search. In state s, agents first estimate the value of each action a by computing the value of the best state reachable in Sruntime actions, starting with a. Formally:
Quantime
max
((TBauine)
(5)
T: - Suniie
where To and Trnmme respectively denote the first and last actions in the trajectory т. Finally, agents select actions in proportion these Q-values (Haarnoja et al., 2017):
T 8; Prantime; R) x (aaas)
(6)
With this agent parameterization, Eq. (2) can be computed efficiently:
Proposition Truncated breadth-first Search (TBFS) an anytime inference algorithm. (Repre- sent each inference state fa as the set of frontier states and values reachable from each starting action. To compute fs+1. add the unexplored children of these states to the set.)
4.2 DATA
In this pedagogical example, we treat the agent model in Section 4.1 as the true data data-generating process. We fix a set of parameters R; and 3runtimes generate a collection of synthetic trajectories using Eq. (6), then attempt to recover these parameters using Eq. (2). (This allows us to validate the feasibility of our approach under ideal conditions later sections will apply it to real datasets of human- generated actions). In particluar, generate 5 agents with runtime budgets of 1, 2, 5, 10, and 20 respectively. Example trajectories from each these agents are depicted in Fig. 2.
4.3 EVALUATION
We compare L-IBMs with Boltzmann model in which agents select actions according to:
Quemp (a
Beemp max R(T)
7:50=a
where R(T) denotes the final reward obtained along the complete trajectory (i.e. upon reaching some exit R,). We also compare to simple baselines in which the agent performs truncated search up constant (not inferred) depth. We evaluate these models in two ways:
4

---

Published as a conference paper at ICLR 2024
Inferred Stantime
Inferred Sump
(L-IBM)
Predictions
Predictions
20
Inferred Brantme
Inferred Biore
10
0.3
0.6
Approach Accuracy
0.4
Bruntime -0
0.2
2-
Sruntime
-0
1-
Inferred Sump 20
2
10
20
2
5 10
20
0.0
L-IBM
Brnnim
Ground Bruntime
(a)
(b)
Table 1: Agent action diction accuracies in maze navigation. L-IBM signif-
Figure Inferred parameters yn (distributions over (3) for the maze navigation task. (a) L-IBM almost perfectly recovers these parameters, while (b) the icantly outperforms base- lines.
Boltzmann model shows no significant differences across inferred Sromp-
Predicting actions. In held-out states, we evaluate models' exact-match accuracy in predicting an agent's next action. Results are shown in Table Models that assume constant depth perform worst. While Boltzmann models are better able to predict agents' next actions than these fixed- budget models, they are significantly outperformed by L-IBM.
Predicting rewards. We also evaluate whether inferred prior distributions over recover the true values used to generate the data. Results for L-IBM and the Boltzmann model shown in Fig. 3a. can be seen that L-IBM almost perfectly recovers these parameters (suggesting that prediction errors in Table 1 result entirely from errors in the inferred reward parameters R,). Meanwhile, the Boltzmann model shows no significant differences in inferred Semmp across depth budgets, empha- sizing the discrepancy between the mdoels of suboptimality.
Together, these results show that L-IBM is computationally tractable and capable of making accurate predictions and inferring meaningful parameters in simple search problems. In the remainder of the paper, we apply it to modeling real human behavior in more complex decision-making tasks.
5 PRAGMATIC LANGUAGE UNDERSTANDING WITH RATIONAL SPEECH ACTS
The next task we consider focuses on pragmatic language under- standing-inferring speakers' communicative intents from their ut- terances. Humans readily produce and understand language in ways that deviates from its "literal" meaning. In Table 2, for example, color that would be described on its own by most speakers purple instead labeled blue in some contexts. A large body of work cognitive science models this kind of context-based language un- derstanding as the result of an iterative inference process (Frank Goodman, 2012; Franke, 2013): for example, in Row 2 of Table 2, a speaker might choose describe the highlighted color as blue by reasoning that naïve listener might resolve purple to the second color in the row. A more sophisticated listener, in turn, can predict this speaker behavior, and successfully infer the intended meaning, But this kind of recursive reasoning about other agents can be com- putationally demanding, and requires sophisticated internal models of other language users. Experimental evidence suggests that is de- ployed selectively, and different degrees by different language users (Franke & Degen, 2016). We use L-IBMs to determine when, and to what extent, this kind of recursive reasoning is used during language production.
Context
Utterance
1.
purple
2.
blue
3.
blue
Example of the ref- erence color (within the black box) and the two distractor col- ors, along with the utterance pro- duced by speaker from the col- ors context (Monroe et al., 2017). Notice how the context affects the utterance, even as the reference color remains fixed.
Table 2:
Our experiments focus on refference game of the kind depicted in Table 2 (Monroc et al., 2017). Reference games are staple of research on pragmatic language use. reference game, both listener and speaker observe set of candidate referents (e.g. colors). The speaker is privately given one of the colors as a target; they must then produce natural language utterance for the listener. Finally, the listener selects a color patch, and both players win if they agreed on the target.
5

---

Published as a conference paper at ICLR 2024
By fitting an L-IBM to utterances and choices in human reference games, investigate (1) whether we can infer whether humans are engaged in pragmatic reasoning from behavior alone, (2) whether there are differences between players in their ability to reason about their interlocutors, and (3) whether these differences actually predict communicative success (i.e. whether players with greater inference budgets are better at making themselves understood).
5.1 AGENT MODEL.
We build on the Rational Speech Acts (RSA) model of Frank & Goodman (2012). This model frames communication as one in which Bayesian listeners and speakers reason recursively about each others' beliefs in order to select utterances and actions. The starting point of RSA is literal listener TL that maps utterances u to actions according to their non-contextual meanings. (In Ta- ble 2, literal listener hearing the word purple might choose randomly between the first two colors in the second row, as both would be reasonably described as purple out of context.) The literal listener may be implemented by any model (e.g. lookup table neural network; Andreas & Klein, 2016) with parameters 9. Next, given a reference target t, pragmatic speaker Ty chooses an utterance in proportion the probability that will cause literal listener to take the right action:
(u t) x p( selects & upon hearing u) = = |u)
(RSA speakers standardly parameterized with additional Boltzmann rationality parameter, which we will discuss momentarily.) Finally, pragmatic listeners observe speaker utterances u, and reason about which reference targets were most likely to have produced those utterances:
x (t u) = p(TT intends to signal t u.) x x |t) p(t)
Crucially, this process may be repeated, with speakers I's reasoning about weer-mor-ssohisticatee speakers vi- etc. But how many rounds of iteration actually explain human behavior? the latent inference budget model framework, may model this by embedding RSA inside L-IBM,, with the budget 3 parameterizing the number RSA iterations performed by each agent:
ss t;
Σ Brantime (3 s) 0,
(10)
su( t; 2 (u|l t)
(11)
(and analogously for nT.))
Proposition 2. Rational Speech Acts (RSA) is an anytime inference algorithm. (Each inference state 8
fa g or T.. Each of these can be derived from the other constant time via Eqs.8-9.
5.2 DATA
For this task, we use the data collected by Monroe et al. (2017). Each game consists of roughly 50 rounds played between a human speaker and a human listener. each round, the speaker observes target color along with two distractors. The speaker produces an utterance and the listener has to click on one of the colors. The dataset consists of 46,994 rounds across 948 games. We create a 80/10/10 split across train, valid and test sets. Monroe al. stratify the dataset into three difficulties (easy, medium and difficult) based on perceptual similarity between colors and distractors. Because each game is annotated with unique identifier for both the speaker and the listener, we may further stratify the dataset according to player skill: we compute the fraction of games won by each (speaker, listener) pair, then group these pairs into six buckets according their win rate percentile relative to other players. This allows us to examine the relationship between inference budget and both task difficulty and communicative success.
5.3 MODELS
Following Monroe et al. (2017), we implement the literal listener > using transformer model that receives all three colors (represented as HSL vectors) and a natural language utterance as input, and predicts the index of the target color output. We embed this listener model within the speaker- listener recursion defined by Eq. (9), then train it end-to-end (with budget parameters n)) on the Colors in Context data using Eq. (2).

---

Published as a conference paper at ICLR 2024
Inferred Sropp
Inferred Brumime (L-IBM)
Inferred foom
0.8
1.0
9.6
Pore
0.8
Inferped Poeeee
0.8
0.6
0.5
0.4
Inferred
0.2
6.2
2
4
5
4
0.0
M Diffculty
Skill level
02
Difficutty
Skill Level
(a)
(c)
Figure 5: Inferred distributions over 3 in RSA. X-axis indicates the difficultyy level (Easy, Medium, Hard) or the player skill level (between and 6, 6 being the most skilled players). The inferred Suep across difficulty in a) and player skill in not meaningful it for Sumiie c) and c) When separating games by difficulty, L-IBM infers that the non-litera speaker employed only for the hardest condition. d) When separating games by player skill, we infer that the weakest players can be modeled exclusively as literal speakers, while stronger players can be modeled as mix of literal and pragmatic speakers.
The constant of proportionality in Eq. (8) involves a sum over all natural language strings, which is cannot be computed efficiently. Here, also following (Monroe et al., 2017), we perform a sampling- based approximation: we train literal speaker model to generate plausible utterances, then sum over finite number of such samples obtain distribution over strings. See McDowell & Goodman (2019) for more details. The literal speaker is parameterized identically to the literal listener, but outputs strings rather than color indices.
In experiments investigating the relationship between task difficulty and inference budget, we fit one т per condition (easy, medium, hard). In experiments investigating the relationship between communicative success and inference budget, we fit one Y per skill level (between and 6).
5.4 EVALUATION
Standard implementations of RSA modifies Eq. (8) include Boltzmann parameter for speakers:
Ts(" t;; (3 (xp log - u))
(12)
Like our Bruntimes this parameter is intended to model possibly sub-optimal behavior on the part of speakers and listeners. We compare L-IBM to model of this form. In particular, we fix the number RSA iterations to one, use the same data as above to estimate literal listener parameters jointly with a prior distribution over Bremp:
T& (u t; 0;
Σ B
Piemp (3 () 1)
(13)
where r3 is defined as in Eq. (12).
Table 3 shows different models' ability to predict the target referent given human speaker utterances. Consistent with the findings of (Monroe et al., 2017), because even literal models have access to all three referents, all model variants can achieve good task performance. When we look at inferred val- ues for Stuntime and Premp- however, begin to significant differences between models. When stratifying games by difficulty, we infer that the non-literal speaker is employed only for the hardest conditions. When stratifying games player skill, infer that the weakest players eled exclusively as literal speakers, while stronger players can be modeled as a mix of literal and pragmatic speakers. To the best of our knowledge, this is the first example of an RSA-type model being used to infer individual differences in pragmatic language use within a speaker population; we expect that these tools may be of independent interest the cognitive science community. Ad- ditional experiments, predicting the object that the listener picked instead of the one the speaker is presented can be found in Appendix C.
be mod-
PLAYING CHESS WITH MONTE-CARLO TREE SEARCH
Finally, we turn from cooperative to adversarial decision decision-making tasks. We focus on chess, a popular two-player sequential game widely used benchmark for Al systems. Here, we interested in

---

Published as a conference paper at ICLR 2024
Model
Type
Accuracy
Model
Type
Accuracy
Sumiime 0
83.3
II
42.06
Srutins
83.0
Sramine
100
43.64
Interred Phemp
player skill
83.9
Inferred puct
Active Elo
43.77
Inferred Prostime (I--IBM)
player skill
84.0
Inferred Pauntine (L-IBM)
Active Elo
44.17
Inferred Phamp
difficulty
83.5
Inferred Ppcct
Opponent Elo
43.84
Inferred
Sumiime (L-IBM)
Opponent Elo
44.17
Inferred Prumimm (L-IBM)
difficulty
82.7
Inferred
Spuc
Time Control
43.61
Inferred Sanin.(--IMM
Time Control
44.15
Table 3: Performance of different RSA models in predicting the speaker target. All models (including literal models and fixed-depth RSA models) achieve similar predictive performance-becaute eral models have access to all three referents, all model variants achieve good task performance. Branzime 0 represents the base literal listener.
Table 4: Accuracy of predicting agent's next ac- tion in chess. Models with MCTS outperform the depth-D (imitation learning) baseline. Learning sub- population-specitae B enhances performance, with L- L-IBM-based learning of Suuntme consistently out- performing Speer by slight margin.
even lit-
modeling human chess play-specifically, observing data from a population of sub-optimal agents with a common reward function (winning the game) and attempting to infer those agents' com- putational constraints. human human play, there can be numerous sources of such constraints: player paired against strong opponent will likely to plan for longer than against a weaker opponent; some variants (like blitz chess) deliberately limit players' Lume-pe--ove (and, we might expect, the quality of their plans). Given dataset of human games played under different time constraints and player strengths, can we use I.-IBM to model variability in players' decisions across game states?
6.1 AGENT MODEL
In this work, we model chess players as selecting actions using Monte Carlo tree search (MCTS). Recent work (Jacob et al., 2022) has shown that MCTS is a good model of strong human players. Here, following (Silver et al., 2018; 8; 2016; Jacob et al., 2022; Grill et al., 2020), we implement one of the most common modern forms of MCTS, which uses a value function V predicting the expected total future reward and a policy prior 7. to guide exploration. At high level, MCTS operates by incrementally growing a game tree starting at the root node, repeatedly picking some path to explore down the tree, performing value function evaluation and then walking back up the tree updating all the value estimates based on that result. At each node, MCTS treats action selection as a multi- armed bandit problem. We use standard exploration policy (Kocsis & Szepesvári, 2006): during inference at each node of the search tree, we choose actions according to:
arg max Qe(a 8) + Ppuert 8) (VE,, ((s,b))((Ng a) +1)
(14)
where (r(s, a) is the estimated expected future reward for from playing action in state at iteration 4, the visit count N N(s, a) the number of times a has been explored from 8, T 7" 8) an "anchor" policy, and Spuce is tunable parameter trading off exploration versus exploitation. After expanding Sruntime nodes of this tree, an agent's 's final action is sampled from a distribution:
(a s; Peantime) Spuct v Prantime
x. (als)
(15)
(s, 1100+1---0000 Brainn (a
where 7 is chosen such that x forms a proper probability distribution.
Proposition 3. MCTS is anytime inference algorithm. (Let each inference state fa be the tree of nodes and visitation counts after 3 evaluations. This tree is refined by evaluating Eq. (15) once.)
With (a 8; Brantime) as defined above, we may instantiate an L.IBBM for MCTS:
runtime (t/u;
Σ
Prodget (Pruntime 17.) π <(; 8, Sruntime)
(16)
Seaatine
We train the base initial policy To and value model up as two different output heads of deep neural network using imitation learning. Our architecture is a 4-block residual network similar to those used in prior work (MclIroy-Young et al., 2020). Unlike previous sections, we do not learn the value functions jointly with Prodget- Instead, we first learn a single value function, then fit Prodger (Spuct | 7)) and Prudger (Brutime 7)). We stratify players into sub-populations according to player Elo (a proxy for player skill), and opponent Elo and time control (both proxies for task difficulty). As in Section 5, we estimate separate 1₁ for each group within each stratified dataset.
8

---

Published as a conference paper at ICLR 2024
Inferred Spuct
0.8
0.8
0.6
0.5
0.4
0.5-
02
02
022
800
900
1400 1500 2000 2300
800
900
1400
1500 2000 2100
DZ Time Control
Player El Rating
Opporent Elo Rating
Inferred Smiiime (L-IBM)
0.6
0.6
1000
0.4
150
Inferred
50-
50-
0.3
0.3
03
800
1400 1500 2000 22.00
0.2
800
900 1400
1500 2000
21.00
Huyer Rating
Opponient fatiin
Figure 6: Inferred distributions over / in Chess using MCTS. The horizontal axis indicates the player Elo rating, opponent rating buckets and time control: Ultra Bullet (U), Bullet (B), Blitz (BZ). Rapid (R) and Classical (C). The top row depicts the distributions for Spoct and the bottom row depicts the distributions for Sruniie. When the player's strength, opponent's strength, or time increases, Brumime infers greater runtime.
6.2 DATA
We use similar data previous models of human chess play Miclrroy-Yourg al., 2020): First, a dataset Daarg containing roughly 6 million moves, used to train the base value function; second, a dataset small containing roughly 75,000 moves, used build population- -speciffe models. Damall includes metadata describing players' Elo ratings (a measure of strength) and game formats (the amount of time players had select moves). See Appendix B for details.
6.3 EVALUATION
Unlike the two domains studied above, there is already an established literature on modeling sub-optimal behavior via MCTS outside the Boltzmann framework. The most successful current approach models individual differences in play (Jacob al., 2022) by fitting Spuct. We thus compare to baseline in which 1), parameterizes distribution over values of Spuct rather than tree expansions.
Accuracy (in terms of top-one predictions and negative log- ligeeiikihood)) is reported in Table 4. As in past work, find that models that with explicit search outperform imitation-leamnn baseline. Learning sub-population specific/ /3 improves the performance even further, with L-IBM-basee -aaeed learn- ing of Bruntime consistently outperforming puct by small margin.
Inferred budget parameters are shown Fig. 6. Here, we observe that as the player strength or the opponent strength increases measured by the Elo ratings, Suntime infers higher runtime. We also observe the same as the time control increases: Sruntime infers higher runtime as the duration of each move of the game increases. Spuct shows a weaker, stronger, or as the time control increases, Spuct infers lower values of Spuce. indicating that players are deviating from the prior and are relying more on the search Q-values.
similar trend: as the agents or opponents get
CONCLUSION
We have described latent inference budget models, family of approaches for modeling agents acting to achieve unknown goals subject to unknown constraints on their inferential capabilities. Instead of assuming either global optimality of decision-making proach explicitly infers the runtime that agents devote to approximate inference. This paradigm is applicable to all anytime inference algorithms. In three domains maze navigation, pragmatic lan- guage understanding, and playing chess we demonstrated that it can outperform classical models of bounded rationality while imputing meaningful measures of human skill and task difficulty.
uniform suboptimality, our ap-

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by the National Science Foundation under grants IIS-2238240, IIS- 2212310, and a seed grant from the MIT Schwartzman College Computing "Artificial Intelligence for Augmentation and Productivity" program. Thanks to Jennifer Hu for helpful discussions about modeling individual differences Rational Speech Act models.
REFERENCES
Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp- 1173-1182, 2016.
Mark Beliaev, Andy Shih, Stefano Ermon, Dorsa Sadigh, and Ramtin Pedarsani. Imitation learning by estimating expertise of demonstrators. In Proceedings of the International Conference Machine Learning, pp. 1732-1748. PMLR, 2022.
Mark Boddy and Thomas Dean. Solving time-dependent planning problems. Brown University, Department of Computer Science, 1989.
Frederick Callaway, Bas van Opheusden, Sayan Gul, Priyam Das, Paul M Krueger, Thomas I. Grif- fiths, and Falk Lieder. Rational use of cognitive resources in human planning. Nature Human Behaviour, 6(8):1112-1125, 2022.
Colin F Camerer, Teck-Hua Ho, and Juin-Kuan Chong. A cognitive hierarchy model of games. The Quarterly Journal of Economics, 119(3):861-899 2004.
Thomas Dean and Mark S Boddy. An analysis of dime-dependent planning. Proceedings of the Annual Meeting the Association for the Advancement of Artificial Intelligence, volume 88, pp. 49-54, 1988.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT; Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171 1171-4186, 2019.
Owain Evans and Noah D Goodman. Learning the preferences of bounded agents. In NIPS Work- shop on Bounded Optimality, 2015.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Sci- ence, 336(6084):998 99889999999999999999888888888889 2012.
Michael Franke. Game theoretic pragmatics. Philosophy Compass, 8(3):269-284, 2013.
Michael Franke and Judith Degen. Reasoning in reference games: Individual-vs. population-level probabilistic modeling. PloS one, 11(5)x0154854, 2016.
Thomas Griffiths, Frederick Callaway, Michael Chang, Erin Grant, Paul Krueger, and Falk Lieder. Doing more with less: Meta-rcasoning and meta-learning in humans and machines. Cur- rent Opinion in Behavioral Sciences, 29:24-30, 2019.
Cean-Bastien Grill, Florent Altché, Yunhao Tang, Thomas Hubert, Michal Valko, Ioannis Antonoglou, and Rémi Munos. Monte-Carlo tree search as regularized policy optimization. In In Proceedings of the International Conference on Machine Learning, pp. 3769-3778 PMLR, 2020.
Tuomas Haaroja, Haoran Tang, Pieter Abbeel. and Sergey Levine. Reinforcement learning with deep energy-based based policies. In Proceedings of the International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
10

---

Published as a conference paper at ICLR 2024
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathamiel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández. del Río, Mark Wiebe, Pearu Peterson, Pierre Gérand-Marchant, Kevin Shep- pard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825);357-362, September 2020. URL https: //doi.org/10. 1038/s s4 586-1866688 064-2649-
Quentin JM Huys, Neir Eshel, Elizabeth O'Nions, Luke Sheridan, Peter Dayan, and Jonathan Roiser. Bonsai trees in your head: How the Pavlovian system sculpts goal-directed choices by pruning decision trees. PLoS Computational Biology, (03::10002100, 2012.
Quentin JM Huys, Niall Lally. Paul Faulkner, Neir Eshel, Erich Seifritz, Samuel J Gershman, Peter Dayan, and Jonathan Roiser. Interplay of approximate planning strategies. Proceedings of the National Academy of Sciences, 112(10)33988-1003 2015.
Athul Paul Jacob, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human-like gameplay with KL-Regularized search. In Proceedings of the International Conference Machine Learning, pp. 9695-9728. PMLR, 2022.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances Neural Information Processing Systems, pp. 4415- 4426, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Levente Koesis and Csaba Szepesvári. Bandit based Monte-Carle planning. In Proceedings of European Conference on Machine Learning, pp. 282-293. Springer, 2006.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Joseph Gonzalez, Ken Gold- berg, and Ion Stoica. Ray rllib: A composable and scalable reinforcement learning library. arXiv preprint arXiv:1712.09388 85, 2017.
R Duncan Luce. Individual choice behavior. 1959.
Bill McDowell and Noah Goodman. Learning from omission. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pp. 619-628, 2019.
Reid Mcllroy- Mellroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhu- man with human behavior: Chess as a model system. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 1677-1687, 2020.
Reid Mcllroy- Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learn- ing models of individual behavior in chess. In Proceedings of the ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1253-1263 1263, 2022.
Andrew N Meltzoff. Understanding the intentions of others: Re-enactment of intended acts by 18-month-old children. Developmental Psychology, 31(5):838, 1995.
Will Monroe, Robert XD Hawkins, Noah D Goodman, and Christopher Potts. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325-338, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, hegh-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2185555855, 2020.

---

Published as a conference paper at ICLR 2024
Stuart Russell and Eric Wefald. Principles of metareasoning. Artificial Intelligence, 49(1-3)3361- 395, 1991.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference on Machine Learning, pp. 5670-5679. PMLR, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Mare Lanctot, al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587)44844455 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play Science, 362 1144, 2018.
Bas van Opheusden, Ionatan Kuperwajs, Gianni Galbiati, Zahy Bnaya, Yunqi Li, and Wei Ji Ma. Expertise increases planning depth human gameplay. Nature, pp. 1-6, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-aar natural language processing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 38-45, 2020.
Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online Bayesian goal inference for boundedly rational planning agents. Advances in neural informa- tion processing systems, 2020.
12

---

Published as a conference paper at ICLR 2024
A
TRAINING HYPERPARAMETER
We will detail the training hyperparameter details in this section.
A.1
MAZE
All models in Section 4 were trained using the Adam optimizer (Kingma & Ba. 2015). where the learning rates were sweeped across the following values 0.5., 1e - 0.05, le 2, 5e -3, 3, le 5e 1e--4, 1e-1e--5 4, 5e 5] for 50 epochs. The values presented in Table 1 were picked from the model with the best validation accuracy across the learning rates. We implemented truncated BFS using Pytorch (Paszke al., 2019) and Numpy (Harris al., 2020) and also used the mazelib library to generate the data.
A.2 COLORS IN CONTEXT
The models trained in Section 5 based on the transformer architecture and trained from scratch. The speaker model was trained based on the TS model (Raffel et al., 2020) with the following hyperparameters described Table 5. The speaker was trained with batch size of 64 using the Adam optimizer with learning rate le for 25 epochs.
Parameter
Value
Number of Layers
Parameter
Value
Number of Heads
4
Hidden Size
64
Model Dimension
32
Number of Hidden Layers
Key- Value Dimension
16
Number of Attention Heads
Feedforward Dimension
32
Internediate Hidden Size
256
Table 5: Hyperparameter configuration of the speaker model based on T5 (Raffel al., 2020).
Table Hyperparameter configuration of the listener model based on BERT.
All the listener models were based on the BERT (Devlin et al., 2019) model with the configuration described in Table 6. The listener models were trained using Adam and the learning rates were sweeped across the following values [le - 3, 5e - 4, 1e - 4, 5e - 5] for upto 50 epochs. The values presented in Table 8 and Table were picked from the model with the best validation accuracy across the learning rates. We trained the models using Pytorch (Paszke et al., 2019) and Huggingface (Wolf al., 2020) libraries. We specifically implemented RSA using Pytorch.
A.3 CHESS
The value and policy network used in Section 6 are based on an architecture that is a 4-block resid- ual network similar those used in prior work Mcllroy-Young et al. (2020); Jacob al. (2022); Mcllroy- Young et al. (2022). The policy and value network was trained using Adam with learning rate of 0.001, a batch size of 4096 and for upto 30 epochs. The epoch used in the rest of the section was picked based on the validation accuracy.
In the second set of fine-tuning experiments, for every set of conditioning type, a simple feedforward network was trained using Adam with a batch size of 512. The models in Section 6 were picked by selecting the learning rates between le - 3, 5e 4, le 4, 5e 5 with the best validation accuracy.
For chess, the base policy and value functions were trained using Ray library (Liang et 2017) and Pytorch. MCTS was specifically implemented using Numpy. We also used the pettingzoo library for simulating moves.
B CHESS DATA
D Dianee consists of 5,974,872 moves the training split, 60,968 in the validation split and 60,969 moves in the test set. These data points were randomly sampled from the January, 2019 database
13

---

Published as a conference paper at ICLR 2024
release of chess website (lichess). small consists of 50,000 moves in the training split, 12,041 moves in the validation split and 12,040 moves the test split. These data points were randomly sampled from the February, 2019 lichess database release but filtering such that only those players with Elo ratings in the following buckets were considered: [800-1000), [1400-1600] and [2000- 2200].
The dataset contains different types of time control. increasing duration, they are Ultra Bullet, Bullet, Blitz, Rapid and Classical (see Table 7).
Time control
Estimated Duration (seconds)
UltraBullet
< 29
Bullet
179
Blitz
< 479
Rapid
1499
Classical
2 1500
Table 7: Estimated game durations across different time controls.
The time controls used in our work have estimated durations that are defined in Table 7:
C ADDITIONAL EXPERIMENTS: COLORS IN CONTEXT
In this section, we include additional experiments for the pragmatics domain where we train the models to predict the object that the listener picks. We present the results of a similar set of experi- ments as Section Table 8 and Fig. 8. We specifically note that the inference based approaches outperform the baselines in this setting.
Model
Type
Accuracy
Srootime (Literal listener)
80.4
Puntimme
81.8
Inferred Premp
player skill
82.3
Inferred Bruntime (L-IBM)
player skill
83.1
Inferred Premp
difficulty
82.7
Inferred Prontime (L-IBM)
difficulty 82.1
Table Performance of different RSA models in predicting the speaker target. The based models outperform the baseline models: literal models and fixed-depth RSA models.
1.0
10
0.8
1.0
Inferred Proms
0.6
10
0.8
0.8
Funire
0.4
Infereed
0.6
0.4
0.4
M
0.0
0.2
Difficutty
SkdlL
Difficulty
0.0
(a)
(b)
(c)
(d)
Figure 8: Inferred distributions (Easy, Medium, Hard)
in RSA. with the listener target. X-axis indicates the difficulty level being the most skilled players). The inferred Bree
player skill level (1 across difficulty in a) and player skill in b) is not as meaningful as is for Suumime in d). When separating games by player skill, we infer that the weakest players can be modelled with smaller mix towards pragmatic speakers compared to stronger players
14

---

Published as a conference paper at ICLR 2024
D ADDITIONAL DISCUSSION
D.1 RELATIONSHIP BETWEEN Ppuct AND ELO RATING IN CHESS
Suect (as used in popular strength-modeling approaches like Silver al. (2016)) doesn't simply control exploration vs. exploitation. Instead, it biases exploration towards an initial policy prior TO. As Spuct tends infinity, identical to playing n.. When Spuct is set to 0, it is equivalent to greedily picking the search Q-values. In Fig. 6 and it relates to ELO rating, stronger players start deviating more from their base policy x² to instead depend more on their MCTS search Q-values. Therefore indicating that stronger players rely more search compared to weaker players.
notice that
15


Reviews:
[{"review_id": "AxA7tNOra7", "reviewer": "Reviewer_PxXG", "summary": "The paper proposes a method for modeling bounded rational agents through the inference of latent budget. The method seeks to model an agent's computationally constrained inference by explicitly inferring the latent variable associated with the computational budget jointly with the agent's goals. This is used to both infer agent intent as well the underlying budget, which is correlated with agent competency. The proposed method is experimentally evaluated over three tasks: maze navigation, language understanding, and chess.", "questions": "1) What exactly does beta_temp intuitively represent in the RSA task? This is not entirely clear to me, nor why the figures in Table 3a/b seem to show such uninformative results. Do you have any insight as to why the inferred beta_depth seems to be more informative than beta_temp (in the sense that it seems more correlated with player skill)?\n2) What is the relationship between beta_puct and ELO rating in Sec. 6? For time control it seems reasonable that longer time budgets would enable more exploration, but it seems there would be a trade-off between exploration and exploitation with respect to ELO rating. So it's not clear to me what this relationship is expected to be.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and motivated. The variety of experimental settings is helpful for gauging model performance, and in particular the inclusion of an experiment with human-generated data (RSA) is valuable in showing performance when the exact underlying agent model is unknown.\n* The inference of agent budget and the correlation to skill is an interesting direction, and may have value in both agent-agent and human-agent interactions.", "weaknesses": "* The accuracy differences in the RSA and chess tasks are fairly marginal, and seem to indicate that it is difficult to jointly infer both latent budgets and intent in more complicated tasks.\n* The modeling of the latent budget requires fairly strong assumptions about the underlying reasoning mechanism of the agent. In addition to potential misalignment of assumptions, this also leads to situations like Sec. 5.3 where a constant of proprotionality is approximated over the set of all natural strings which are potential additional sources of error.\n* The results graphs are difficult to parse with the frequent reference to variable names that have unintuitive meanings. I realize these are referring to parameters in the agent models and corresponding sub-populations, but it is difficult for the reader to draw conclusions when the results are represented in terms of abstract terms such as beta_temp, beta_depth, beta_runtime, beta_puct, etc. This is compounded by the fact that terms like \"depth\" are used frequently and seem to have context-specific meanings that are not always fully defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "noHjkEekEh", "reviewer": "Reviewer_egSD", "summary": "In this work, the authors address the problem of modeling agents with bounded rationality and possibly unknown preferences. They begin by pointing out that the standard Boltzmann model depends only on the return of a particular action and doesn’t account for the structure of a problem in determining the probability of a suboptimal outcome given a particular inference budget for an agent. \n\nIn (2), the agents present L-IBM, a MAP inference algorithm that allows us to infer the reward model and the parameters of the budget distribution for a particular agent from data assuming the agent follows an anytime inference algorithm. Then in the following sections they give instantiations of their setup in maze solving, language generation, and chess. \n\nIn maze solving, they assume the search algorithm is a truncated BFS algorithm with a known heuristic. They show that this is an anytime inference algorithm and then demonstrate on simulated inference data that the L-IBM method is able to entirely recover the correct data-generating parameters while the Boltzman model performs horribly. \n\nIn language, they use a reference game–where one participant must try to communicate with another effectively and must model their understanding well enough to communicate. Here, they borrow a model of Bayesian listeners and speakers from the cognitive science literature. The primary feature of interest in this model is the number of “layers deep” to go in modeling that your partner can model that you know things and you can model that your partner can model that you know things and so on. They again show that this is an anytime inference model and demonstrate using a transformer and an existing dataset of human utterances and choices. Here, the bounded rationality model is able to determine that more skilled players behave in ways that would be considered “deeper” than those that are less skilled. \n\nHere, the anytime inference algorithm is MCTS as used in AlphaGo and the recent Diplomacy works. In this setting, there are two budget parameters \\beta_UCT, and \\beta_runtime. The method is used to estimate each of these parameters. Using a dataset of human games of players with varying Elo ratings and time controls, the quality of moves is inferred to these two beta parameters and they are shown to correlate with a longer time control, stronger opponent, and stronger player (as one would expect a priori).", "questions": "How hard is it in general to predict the player’s next move in chess?\nIs there an extension of anytime inference that can handle “amortized computations” like dynamic programming for value functions or other pre-planning methods?\nWhat did you practically use to implement the inference procedures in this paper? I think the methods would be useful to comment on a bit.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results. You can incorporate Markdown and Latex into your review. See https://openreview.net/faq.\n\nI think this paper tackles an interesting question and gives a thorough and principled answer. As we introduce other cognitive agents into the world we ought to have a framework for evaluating their and our behavior on the same terms. This feels like a step towards a more general understanding of this. I think the boundedly rational agents as described in this paper are a substantial generalization of the Boltzmann model. \n\nThe evaluation on 3 domains is thorough and I appreciated the care taken to make them easy to understand and well presented. I found the figures and data presentation to be compelling and the ideas presented have had me update my model of how to think about this kind of thing going forward.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.\n\nAt some point in the beginning of the paper the problem statement seemed too general to grab on to and I was mentally scraping around for how to add structure in order to come up with ideas for solutions. I think it would be a nicer reading experience to include a comment gesturing in the direction of anytime inference algorithms in the front part of the paper so as to anticipate this. \n\n\nI’m not sure how to interpret the numerical figures for something like chess in Figure 5. Naively, I tend to think that the effects on predicting next actions are not very large. \n\nIt is not clear to me how much this can be used in the broad main thrusts of ML research today and this work might be of limited use to practitioners. However, I think it is interesting and probably valuable for subjects like reward-free Offline RL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIlW5XHFWc", "reviewer": "Reviewer_KCR8", "summary": "The paper presents budget constrained bounded rationality models. In contrast with classic Boltzman rationality, deviations from rationality are formalized as (unknown) limitations on computation instead of simply noise in selection. A simple formulation of budgets constructed as a model over possible inference budgets, and it is observed that inference can be computed efficiently for anytime models. The approach is demonstrated in three domains: inferring goals from actions in navigation, inferring communicative intent, and predicting chess moves. The budgets model individual performance and have meaningful interpretations.", "questions": "I would really love to see some additional comparisons. Other than that, I find the paper to be a nice contribution. \n\nI would rate this a 7, but I don't seem to have that option...", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents an interesting and intuitive formalization of bounded rationality based on constrained computation. \n- The empirical comparisons are nice. \n- The results are interesting.", "weaknesses": "- There are a few ad hoc decisions buried in the middle, which make the story less clear. \n- Comparisons with other proposals are a bit lacking. This is not the first paper to propose limitations on Boltzman rationality. \n- A more detailed set of results would be nice. \n\nDetailed comments: \n- \"consider again the trajectories depicted in Fig. 1(b–c), which differ only in the difficulty of the search problem, and not in the cost of the optimal trajectory at all.\" There are some pretty big assumptions hidden in here. \n- \"learning a model of these agents ultimately learning reward parameters θ and agent-specific budget-generating parameters\" missing word?\n- The sampling algorithm for the speech task doesn't seem to be motivated by the idea of a budget. \n- I am not a fan of the visualization in figure 6. It is quite hard to parse.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WZv7jKCf7O", "reviewer": "Reviewer_q6an", "summary": "The paper tackles modeling bounded rational agents based on their trajectories. It focuses on determining $pi (a | s, R, \\beta)$, where $R$ represents the reward function and $\\beta$ denotes the computational budget. Contrasting the classical Boltzmann model that overlooks varying agent rationality, this work innovatively models $\\beta$ as a variable derived from an agent-centric distribution $p_{\\beta | \\eta}$, with $\\eta$ being inferred using MAP.", "questions": "- **Literature Review**: Please provide a thorough literature review.\n\n- **Future Directions**: For example, are there plans to address the adaptability of $\\beta$ based on past trajectories? Or would you develop a rigorous theoretical guarantee for the model?\n\n- **Practical Implications**: How does the authors envision the practical implications of this work? In which domains or applications do they see this method having the most significant impact? For example, one of the hottest research recently with real-world impact is on large language models. How may the classical bounded rational agents study make contributions to this direction?\n\n- **Computational Complexity**: Can the authors comment on the computational complexity of their method, especially concerning the inference of $\\eta$ (also please provide more details on the inference methodologies being used)? How scalable is it for large-scale applications?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- **Originality**: One of the primary strengths of the paper is its original approach to model $\\beta$ using a learnable parameter. This innovation distinguishes it from prior works, offering a fresh perspective in the domain of modeling bounded rational agents. The perspective is novel and fresh.\n\n- **Significance**: The introduction of a learnable parameter for $\\beta$ can potentially change the way we perceive agent behaviors. This approach, if refined and built upon, could pave the way for more advanced and adaptive models in the future. It also hints at the possibility of its application across different domains, making the work potentially impactful.", "weaknesses": "- **Clarity and Presentation**: The paper often comes across as convoluted, making it difficult for readers unfamiliar with the domain to grasp its core concepts. Clearer explanations, along with better-organized sections, would significantly improve comprehension.\n\n- **Lack of Thorough Literature Review**: The paper does not delve deep into existing works, leaving readers unaware of the full landscape of related research. A dedicated literature review, even if placed in the appendix, would help contextualize the presented work better.\n\n- **Quality of Experiments**: The experimental section lacks rigorous validation on diverse datasets or scenarios, or providing substantial comparisons with more state-of-the-art existing models or methodologies, potentially limiting the generalizability of the proposed method. \n\n- **The Modeling of $\\beta$**: The paper models $\\beta$ in a manner that appears static across the trajectory. However, in realistic scenarios, it would be logical to assume that the distribution of $\\beta$ is adaptive (either in a Bayesian or frequentist manner) and may change at every time step, conditional on histories. This richer representation can offer a more meaningful interpretation of agent behaviors.\n\n- **Ambiguities in Parameter Inference**: While the idea of modeling $\\beta$ with a learnable parameter is commendable, the paper does not sufficiently justify or explain the underlying mechanisms behind the chosen methodology for inferring $\\eta$.\n\n- **Robustness Concerns**: Given the intrinsic variability in agent behaviors, how robust is the proposed method to outliers or erratic trajectories? Were there specific tests or validations done in this regard? I suggest the authors to provide some ablation study in future revisions.\n\nI am happy to raise my score if the authors address some of the concerns above.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)