Language and Experience: A Computational Model of Social Learning in Complex Tasks
Authors: Cédric Colas
Tracey Mills
Ben Prystawski
Michael Henry Tessler
Noah Goodman
Jacob Andreas
Joshua B. Tenenbaum
Venue: ICLR 2026 Poster (2026)
Keywords: cognitive science; social learning; cultural learning; causal learning; bayesian models of cognition
Primary area: applications to neuroscience & cognitive science
TL;DR: Modeling human social and cultural learning as the joint inference of world models from language and experience, enabling cross-embodiment knowledge transfer

Abstract: The ability to combine linguistic guidance from others with direct experience is central to human development, enabling safe and rapid learning in new environments. How do people integrate these two sources of knowledge, and how might AI systems? We present a computational framework that models human social learning as joint probabilistic inference over structured, executable world models given sensorimotor and linguistic data. We make this possible by turning a pretrained language model into a probabilistic model of how humans share advice conditioned on their beliefs, allowing our agents both to generate advice for others and to interpret linguistic input as evidence during Bayesian inference. Using behavioral experiments and simulations across 10 video games, we show how linguistic guidance can shape exploration and accelerate learning by reducing risky interactions and speeding up key discoveries in both humans and models. We further explore how knowledge can accumulate across generations through iterated learning experiments and demonstrate successful knowledge transfer between humans and models—revealing how structured, language-compatible representations might facilitate human-machine collaborative learning.

Published as a conference paper at ICLR 2026
LANGUAGE AND EXPERIENCE: A COMPUTATIONAL MODEL OF SOCIAL LEARNING IN COMPLEX TASKS
Cédric Colas MIT
Tracey Mills
Ben Prystawski
Michael Henry Tessler Google DeepMind
MIT
Stanford University
Noah Goodman Stanford University
Jacob Andreas
Joshua Tenenbaum MIT
MIT
ABSTRACT
The ability to combine linguistic guidance from others with direct experience is central to human development, enabling safe and rapid learning in new environ- ments. How do people integrate these two sources of knowledge, and how might Al systems? We present computational framework that models human social learning as joint probabilistic inference over structured, executable world models given sensorimotor and linguistic data. We make this possible by turning pre- trained language model into a probabilistic model of how humans share advice conditioned on their beliefs, allowing our agents both to generate advice for others and to interpret linguistic input evidence during Bayesian inference. Using behavioral experiments and simulations across 10 video games, we show how linguistic guidance can shape exploration and accelerate learning by reducing risky interactions and speeding up key discoveries in both humans and models. We further explore how knowledge learning experiments and demonstrate successful knowledge transfer between hu- mans and models moodels-reveallee revealing how structured, language-compatible representations might facilitate human-machine collaborative learning.
accumulate across generations through iterated
Code: gitthbb.oo//ccolll///gggggg.ggep
Demo: catttcoolss.commmmmmmm////aaggga
INTRODUCTION
Imagine learning to forage mushrooms in autumn woods. Each outing provides direct experience of promising slopes and soil conditions. But an experienced forager's advice can transform your exploration in two crucial ways: "Never touch the red ones with white spots; they 're deadly" helps you avoid fatal mistakes, while "Look for chanterelles near oak trees after warm rains" turns random wandering into focused search. This ability integrate linguistic guidance with direct experience fundamental to human intelligence, enabling not only safer and more efficient learning, but also the accumulation of knowledge across generations (Tomasello, 2009; Boyd et al., 2011). Yet, still lack a general computational account of how humans combine these two modes of knowledge acquisition inform exploration and decision-male making in complex tasks.
Current computational models capture only fragments of this dual learning capability. Reinforcement learning (RL) agents, for example, can master complex tasks but require extensive trial-and-error millions of interaction steps before achieving proficiency (Sutton and Barto, 2018; Mnih et al., 2015). Theory-based RL. mitigates this limitation by combíning planning with Bayesian inference over structured world models, achieving human-like sample efficiency, yet remains incapable of social learning (Tsividis et al., 2021: Griffiths et al., 2010; Lake et al., 2017). Attempts to bridge RL integrate linguistic input into the learning process but rely
this gap with linguage-conditionn on massive amounts of paired experience and language, making real-world application impractical (Zhong et al., 2020; Luketina al., 2019). While large language models (LLMs) excel at processing linguistic guidance flexibly (Brown et al., 2020), they struggle with interactive planning and embodied learning (Valmeekam al., 2023; Paglieri et al., 2024). Bayesian models of social cognition provide

---

Published as a conference paper at ICLR 2026
Eeperimental paradigm
then plays
advice
abvice
Player 1 (uumon or mode))
ayout pettey squares they soll your ophr green way
Cuuman model)
nhoough bariers
b) Example tnjectury a model for Player 2, leaming from esperience and linguinticquidene
Ep Sold Sky
Ep (aV S.ppp
Fp tit Step
Ap Fut Seep!
Bay velowe Jiters doentt mosee
Rey (cnn kill green. and ii maaes
Exy new day frrnngs
pupple legtgrees)
sumrpl thooy
Interpetation ysltoe kclla ro
lighigress alsa cnsage
Apribaset
Speitaies
Spritedet
theeideece: letroshll
dacicius
Norearatar
pellee Razccnree
intacassionderr
and blaslle
2ed Intoraile
Interable
Tecaisepitnoon Bet
green
poss
qusen
darkhise
Insetactionee:
Choier pranaage
Agen mirmans 1620 treotes
2003
Pesminat aoskees
green dackbice ccliapiice revartal
sedariblue
Leb0ve darkblln wis raluo
Figure 1: Overview of the model and experiments a) Experimental design: players (participants models) are given N=10 lives to learn to play a new video game, either from experience only (Player 1) from experience and advice written by previous player (Player 2). b) Example learning trajectory: The model maintains beliefs about possible game rules and objectives (programs at bottom), and constantly refines them based on an initial linguistic guidance and new incoming experience.
promising step towards integrating observed behaviors and language infer goals (Shafto et al., 2014; Jara-Ettinger et al., 2016), yet they typically operate in simple, non-interactive tasks with predefined hypothesis spaces (Ying et al., 2023; Zhi-Xuan al., 2024).
Our primary aim is to contribute to cognitive science by offering a computational account of how humans integrate language and experience. We propose Bayesian framework that treats linguistic guidance and direct experience as complementary evidence sources learners can leverage to infer executable, program-like world models. To make this possible, we introduce three key contributions:
LMs as speaker models:
We leverage LMS to approximate the probability that a human with specific world beliefs would produce particular linguistic advice, enabling our model to interpret and generate human-interpretable guidance.
Inference from linguistic evidence: This speaker model is used to evaluate the plausibility of received advice under different candidate world models, allowing linguistic input to shape Bayesian inference alongside experiential learning.
LM-accelerated inference: We use LMs to transform advice into targeted proposal distributions, guiding Bayesian updates towards the most promising regions of the hypothesis space.
These mechanisms allow our computational model learn efficiently from naturalistic linguistic input, update beliefs in ceal-time, and share discoveries with others Tahers-forring forming the basis for more structured and communicative learning systems.
We validate our framework through human experiments and computational simulations, showing that linguistic guidance accelerates learning, shapes exploration strategies. and supports knowledge transfer across generations as well as between humans and models. These results reveal how language drives cultural transmission and illustrate how structured, language-compptible representations can facilitate Auman-AI collaborative learning.
2
VIDEO GAMES AS LEARNING ENVIRONMENTS
Video games provide an ideal experimental paradigm for studying social and individual learning (Allen et al., 2024): they offer rich causal environments for systematic exploration, create natural pressure to learn efficiently through costs like lost lives. and enable the study of how complex mechanics can be communicated through language. Drawing inspiration from earlier work modeling human causal learning (Tsividis et al., 2021; Tomov et al., 2023), and cultural knowledge transmission (Tessler et al., 2021), we reuse a suite of 10 games defined in the Video Game Description Language (VGDL) (Schaul, 2013).
See detailed related work Appendix Section A.
2

---

Published as a conference paper at ICLR 2026
VGDL lets define games by specifying object types, collision effects, rewards, and win/loss conditions (e.g. yellow objects are deadly, vic-
Spritadet
darkblue Kocingkvatar
orange lancrable
yellow aandcatec
pasple Tenovable
tory requires eliminating green objects; see Fig-
Lightgreen > Chasez prorange
rod :3sorable
ure 2 and DSL. description in Appendix Sec-
Interectionsen
killSprite zevardedl
tion C). This formal representation enables pre-
dazkblue orange <illSprite reward=-]
lightgreenn purple -ceppack
cise experimental control and provides struc-
ozange Light ki3llppite
pasple daziblus Ailllpzite
tured language for our model to represent and update hypotheses about game dynamics. Play- ers, however, encounter the games grids of colored squares, having to discover game dy- namics and objectives through exploration. Our games were designed to test diverse learning capabilities: some require spatial reasoning to navigate hazards (pushBoulders) or use telepor- tation mechanies (portals); others demand quick tactical decisions like shooting threats (aliens) defending resources (plaqueAttack). Most challenging are games requiring systematic experimenta- tion discover novel objects through object combinations (relational). Play them here.
killlgrite
cominnatoonee
iosero : darkblue Nin-Falsc loseco i-green Nin-2zue
Figure 2:
Game example (beesAndBirds), Players must discover rules and objectives on their own, sometimes helped with advice from others, to solve 4 game levels (right: levels and 2/4).
COMPUTATIONAL MODEL OF INTEGRATED LEARNING
We frame game learning problem of sequential decision-making under uncertainty (Kaelbling et al., 1998). At each time step I, agents select actions az to maximize expected cumulative rewards:
OC
ag = arg max Exsy (2.a))
where s; and r; are the agent's state and reward time and y is discount factor. The main challenge lies in the agent's uncertainty about the transition function P(s;+1 (;, a), which is governed by unknown game dynamics.
Our approach extends the theory-based RL framework by proposing to infer causal world models jointly from experience and linguistic guidance to support goal-directed planning and strategic exploration (Tsividis et al., 2021). The model alternates between three phases: 1) it constructs and infers a posterior over probabilistic world models given both experience and linguistic guidance, 2) plans action sequences by identifying and pursuing high-value interactions that balance exploration and exploitation, and (3) executes these actions in the game. The following sections detail each component of this learning loop.
3.1 INFERENCE OF STRUCTURED, CAUSAL WORLD MODELS
Our agent models its environment with distribution over structured world models, each represented as a probabilistic program specifying game rules and objectives. These beliefs are continually updated as the agent gathers new evidence from gameplay experience E and linguistic guidance L. We formalize these beliefs update as a Bayesian inference over possible world theories T::
P(T E, L) P(E P(LIT) P(T),
where P(T) encodes prior beliefs over plausible world models, while P(E T) and P(L (T) measure the consistency of theory T with experiential and linguistic evidence, respectively. As illustrated in Figure 1b, the agent continuously refines its beliefs, incrementally integrating new data from experience and reinterpreting language to narrow down its hypothesis space. In the following subsections, detail: 1) the search space of possible world models, 2) the likelihood functions that quantify the fit of experience and linguistic guidance, and 3) the inference algorithm that efficiently approximates the posterior distribution over possible worlds.
space of possible worlds. World models are programs specifying the transition function and reward structure, while inference consists of updating posterior distribution these executable programs. Each candidate world model, or theory T. represented as a VGDL program that specifies object types for each object color (e.g. missile, shooting avatar), interaction effects between objects
3

---

Published as a conference paper at ICLR 2026
(e.g. collision with yellow kills avatar), reward functions (e.g.+1 +1 when avatar kills green), and win/loss conditions (e.g. kill all green). see example in Figure 2 and complete list of VGDL primitives in Appendix Section C. We define simplicity- biased prior P(T) over the search space, favoring theories with fewer rules: 1) object types are uniformly distributed, 2) any object pair has a p = 0.25 chance of interacting, with interaction types sampled uniformly, and 3) each object's death has 0.1 chance of contributing to win or loss conditions. Sampling a theory from this prior involves generating object types, object pair interactions, and win/loss conditions accordingly. These theories are executable: they can be compiled into playable games that lets the agent simulate trajectories interally.. Play them yourself on our demo website.
Likelihood from experience. To estimate the likelihood P [T), we first decompose the agent's s experience F a sequence of symbolic state transitions- - into a sequence of discrete events e, (object movements, appearance or disappearance, rewards, and win/loss events). We assume that these local events are conditionally independent given the theory T, which lets us factorize the likelihood as P(E|T) IIm_, P(e; (T). Because candidate world models are executable, we can estimate P(e, ()) through simulations. Specifically, we replay the agent's 's actions under T by initializing the game engine to the agent's previous state and executing its chosen action. We then track the occurrence of each observed event e, across 20 independent simulations, using their frequencies as empirical estimates of P(ex (T).
Likelihood from language. Linguistic advice received from other agents serves as evidence for evaluating candidate theories T, modeled through Bayesian Theory-of-Mind (Baker al., 201 1). We formalize P (T) as the probability that a speaker, believing T to be true, would produce the observed message L. To approximate this, we use a language model (LLaMA- 3. 1-70B) as probabilistic speaker model. Given a description T, the LM is prompted to generate advice for a future player, see prompt in Appendix The likelihood is then estimated as the LM's probability of producing the exact message L: P[L|T) = (L prompt(T)). This approximation measures how well 7 explains the speaker's linguistic behavior: egg. if T. contains the advice "avoid yellow at all cost!", then PLL| T)) will be higher if T contains the rule "yellow kills avatar," than if it does not.
Although (L (T)) not an accurate model of human speakers, our inference procedure relies only on relative likelihoods across theories. What matters for the posterior is the pattern of likelihood differences between candidate theories, not the absolute scale. This use of approximate generative models is common in Bayesian cognitive modeling, where the goal is to capture how linguistic evidence shifts beliefs across hypotheses rather than to recover exact speaker probabilities.
Inference algorithm. The space of possible theories is vast-exceeding 10a0 configurations for games with just five objects -aciss--mn making exact Bayesian inference intractable. To approximate the particle filter with Metropolis rejuvenations (Metropolis et al., 1953; Chopin, 2002). We maintain a population of M = 20 candidate theories and iteratively refine them by: (I) resampling theories proportional to their posterior probability, and (2) proposing local modifications guided by observed events and linguistic guidance (see details in Appendix Section E). These modify exactly one rule at a time: an object's 's type, the interaction between a pair of objects, a win condition. a loss condition, or a reward function. They are accepted in proportion to the ratio of posterior probabilities between modified and original theories: Puccept = min(1, (1, P(T' E. P.T E, L)) (Metropolis et al., 1953). This process efficiently approximates the posterior distribution P(T E, L) over candidate theories, with each particle Ti assigned a weight w; proportional to its posterior probability given experience and linguistic guidance. The resulting distribution represents the agent's belief over possible game dynamics and objectives. The agent executes one inference step (1 resampling step 5 rejuvenation steps) every 20 environment steps, and 20 inference steps every time a new kind of object appears in the scene, or when the agent dies; see full pseudo-code of the inference algorithm in Appendix Section D.
posterior distribution P[T E, L).
Language-guided proposals. We use the same LM to bias the proposal of game rules - e.g. after receiving the message "'yellow kills you," " the LM may propose rules capturing this lethal interaction:
P(rnew Told: E, L) x ((Po(rew TE + (T
L, (ompt(L,TT)))
where Fnow is a candidate rule (e.g. "yellow kills avatar"), Vold is the current rule, Py is the base proposal distribution, and PLM is the probability the language model assigns to that rule given the message. This is implemented by prompting the LM with the received message L and instructing it to answer multiple-choice questions about specifie VGDL. rules: e.g. does the yellow object: 1)
4

---

Published as a conference paper at ICLR 2026
kills the avatar, 2) steps back against the avatar, etc.; detailed prompt in Appendix Section J. This process biases inference towards theories containing rules most compatible with received advice, resulting in faster convergence.
3.2 GOAL-DIRECTED PLANNING AND STRATEGIC EXPLORATION
To maximize its expected long-term utility, the agent must balance exploration- - gathering informa- tion refine its world model and exploitation leveraging its current understanding achieve game objectives. Planning is guided by the maximum a posteriori (MAP) theory TMAP, inferred during Bayesian updates: the agent's best estimate of game rules and objectives. Based on TMaP simulations, the agent selects high-level goals, and plans action sequences to achieve them.
Goal sampling. Based TMAP, the agent defines a space of high-level goals object-object interactions that it can cause in the environment: collisions between the agent, something it can push or shoot, and any other object. Each goal assigned two values: 1) an exploitation value reflecting its contribution to game objectives (c.g. higher if it is thought to trigger reward or win), and 2) an exploration value representing its potential to reduce model uncertainty, measured as the disagreement about what would happen across the M = 20 candidate world models. Subgoals are sampled in proportion to their combined value, balancing both learning and game progress.
Action planning. To achieve these goals, the agent optimizes 10-step action sequences using TMAP for simulation. Initial action plans are refined through simple genetic algorithm to maximize both game rewards and progress toward goals. To prevent catastrophic errors, the agent performs ten 3-step lookaheads to detect possible deaths or major deviations from the expected reward, triggering replanning when necessary. More details about planning can be found in Appendix Section F.
3.3 GENERATING LINGUISTIC GUIDANCE FOR OTHERS
The agent generates linguistic advice for future players by sampling from the same speaker model used to evaluate language likelihood during inference effectively translating its MAP theory TMAp into natural language Lgenerased ~ PLM((L (L, prompt(TMAe)), This provides optimal speaker modeling when message emitters computational agents and approximate modeling when they human. By using the LM both to interpret linguistic guidance and to generate it, the agent captures key aspects of Bayesian Theory of Mind modeling how humans communicate their beliefs and how they interpret the beliefs of others through language.
3.4 BASELINE MODELS
We compare our approach to three baselines: 1) Oracle: a model that plans solve the game using ground-truth game rules, 2) Deep RL: a Double Deep Q-Network agent implementing pure trial-and-erre learning without structured representations (Mnih al., 2015; Van Hasselt al., 2016), and 3) pure LM: an LM agent (LLaMA-311-7BB) that leverages state-of-the-ar techniques to scaffold long-term decision making: ReAct approach (Yao et al., 2022), use of scratch pad (Nye et al., 2021) and chain-of-thouge reasoning towards beliefs updating and plan formation (Wei et al., 2022); see detailed prompt Appendix Section J). These baselines test the importance structured representations and goal-directed planning. Note that we do not compare language-connditoone RL baselines (Luketina et al., 2019; Colas et al., 2022). These methods rely on thousands of episodes of paired (state, language) supervision learn how map linguistic inputs to action policies, and they do not generalize to new, idiosyncratic messages seen only once. our design, each player receives a single novel message per game mirroring human one-shot social learning language- conditioned RI. would receive no opportunity to learn from language and would behave identically to pure deep RL, which we already include bascline.
4 EXPERIMENTAL PARADIGM
To investigate how humans and computational models integrate experiential and linguistic evidence during learning, we conducted a series of IRB-approved experiments comparing three learning conditions (see Figure la):
5

---

Published as a conference paper at ICLR 2026
1. Experience: Players learn solely through direct interaction with the game.
2. Experience + human message: Players receive additional advice from previous human players.
3. Experience + model message: Players receive additional advice from previous model players.
This design allows us to examine both the effectiveness of linguistic guidance and potential asymme- tries in muman- knowledge transfer. In each condition, players had 15 lives solve four levels of each game, advancing only after completing the current level.
Participants. We recruited 122 participants through Prolific play randomly-assigned games. To ensure task engagement while maintaining a representative sample, we excluded participants who failed to complete at least one level 2 3 games (final N=120). Participants were randomly assigned to one of the conditions (N=40 each). In social conditions, each participant received advice from a randomly-scledd experience-only condition (I-to-1 mapping).
previous player (cither human or model) who had completed the game in the
Procedure. All participants first completed a brief tutorial game to familiarize themselves with the interface and basic game mechanics. In social conditions, participants read advice from previous players before starting each new game, and during gameplay. After completing each game (either by winning depleting lives), participants wrote advice "to help fulure players who have not yel played the game." This prompt encouraged participants to distill their learned knowledge into linguistic guidance, see full instructions in Appendix Section
Analysis approach. To evaluate learning efficiency, we tracked both the number of lives required complete each level and the total proportion of levels completed. We normalized area- under-curve (nAUC) metric (E [0, 1l) to integrate both proficiency and learning efficiency. To analyze message effectiveness, we manually coded advice content along four dimensions: the fraction of useful information about game dynamics, about loss conditions, about win conditions, and the presence incorrect information. This coding scheme lets us examine how specific types of linguistic guidance shape exploration and learning outcomes. We will report differences between conditions as AnAAUC).
Computational simulations. We conducted 20 simulation runs per condition, matching the human sample size to enable direct comparison. The model received the same information as human participants: in social conditions, it processed the same messages (human- model-generated that humans received, while in the experience-only condition, learned purely through interaction. We also iterated learning experiments where sequence of 10 agents, each given two lives, played the game and passed a message to the next agent. This design tests whether partial knowledge can accumulate incrementally across generations, mirroring human cultural learning (Tessler al., 2021).
5 RESULTS
We first examine how humans and models learn novel games from pure experience, before analyzing how linguistic guidance shapes this process. We then look at hoden-model transfer, before exploring how our model can accumulate partial knowledge across generations. Our results reveal both striking similarities and systematic differences in learning strategies across humans and models.
5.1 LEARNING FROM EXPERIENCE
How efficiently can humans and models learn novel game dynamics through pure experience? Both demonstrated remarkable sample efficiency, with median participants solving of 10 games and our model solving all 10 games within 10-life budget (Figure 3). However, systematic differences emerged in games requiring specific cognitive capabilities (see per-game plots in Appendix Figure 7). In relational, which demands systematic exploration of object combinations, humans showed bimodal pattern: 25% achieved model-like efficiency by systematically test- ing interactions, while 40% failed to solve even out of
levels salvea
3-
madel
human
2-
deep RL
pure LLM
oracle model
10
15
life
Figure 3: Learning from experience.
Median across games (N=10, IQ range).

---

Published as a conference paper at ICLR 2026
four levels. This split suggests that while humans can perform systematic experimentation, not every- one defaults to it unlike our model which explicitly reasons about information gain. Conversely, in avoidGeorge, which requires rapid planning protect allies, models consistently outperformed humans (median levels: 4 vs 0), likely due to their capacity for accurate short-term planning.
The importance of structured reasoning becomes clear when comparing against baselines. Pure deep RL. (double DQN) failed to solve any level within 10 lives, while pure LM agents never solved more than one level per game, often solving none (7/10 games). The stark difference between these baselines and both human and model performance underscores the value of structured theories of game dynamics supporting efficient exploration and recision-making Knowing the model of the world (oracle model) lets agents solve most games within their first life (see Figure 3 and Appendix Figure 7).
5.2 LEARNING FROM EXPERIENCE AND HUMAN LANGUAGE
Having established baseline learning capabilities, we next examine how linguistic guidance from pre- vious human players shapes exploration and learning outcomes. Our results demonstrate substantial benefits from social learning while revealing key patterns in effective knowledge transmission.
Benefits of linguistic guidance. How does linguistie guidance shape learning? Both humans and our model showed significantly faster learning when provided with human-written advice, reducing median attempts needed by 1.75 for humans (4-2.255 and 1.25 for models (2.5- (see Figure 4). To quantify these benefits in terms of learning speed, computed a normalized area under the learning curve (nAUC). A fixed-effect model controlling for game difficulty revealed significant improvements from linguistic guidance for both humans (A(AAUC)
0.12, p 2.2 10 and models (A (nAUC) = 0.04, = 3.3 x 10-2). These benefits were most pronounced in games with opaque mechanics like relational, multiple hazards like portals and jaws, where guidance could directly communicate critical interactions. Benefits were smallest in games requiring primarily motor skills like aliens, missile or plaqueArack.
What makes effective guidance? Analysis of message content revealed systematic patterns in communication and message effectiveness. Most messages (88% contained information about game dynamics ("[avoidGeorge) light blue can transform green into them"), with many also including information about win (64%, e.g. "[jaws] The objective is to stay alive") and loss conditions (74%, e.g. "[avoidGeorge) you lose if all squares get turned purple"). A small but notable fraction (1 %) contained errors (e.g. "[relational] Push all the blue into the orange," when blue should contact yellow). Longer messages and those containing detailed win conditions proved particularly beneficial to both models and humans(p = 6.3 x 10 and p 0.031 respectively). Importantly, messages that helped humans also helped models, shown by significant correlations in performance gains across nAUC (r 0.17), lives to first level (r = 0.21), and lives to second level (r = 0.16, all p < 0.02) suggesting shared mechanisms for integrating linguistic and experiential knowledge.
How language shapes exploration. Linguistic guidance systematically altered how players explored game mechanies. Messages warning about dangers significantly reduced costly mistakes: in average, humans and models receiving such warnings experienced between 37% and 67% fewer deaths in avoidGeorge, relational, jaws, and aliens. Messages about key mechanics accelerated their discovery: in relational, players informed about tool creation discovered essential combinations 43% to 83%
avoldGnarge
bnesdndRintt
JNM
miseleCComand
widel in sporkedd
cl
preconaitions
prchboulders
016
186
16
180
Figure 4: Learning from experience and human advice. In most games, both humans (blue) and our model (orange) learn significantly faster with human advice (plain lines) than without (dashed lines). see median in Figure 5a. The pure LLM baseline does learn significantly better with advice. solving the first level in only 3/10 games (N=20, median + interquartile range).
7

---

Published as a conference paper at ICLR 2026
faster, while in plaqueAttack they learned to revive allies 43% to 62% faster when told about this possibilityy. However, incorrect advice could also mislead: in avoidGeorge, the human player and the model who were wrongly warned about the danger of green blocks (in fact harmless) completely avoided them for the first three episodes, while others players interacted with them in average 5 times in the same period period-demonsteen demonstrating how unhelpful linguistic advice can also shape direct experience.
5.3 FROM COGNITIVE MODELS TO LEARNING PARTNERS?
Our model demonstrated the ability to efficiently learn from human-generated generated advice. But can it also help humans return? Can it generate guidance that is useful to other models and, more importanly. to humans? Leveraging its LM-based advice generation capability (Section 3.3), our model produced detailed, pedagogical advice that rivaled human teaching. For example, in preconditions:
"Control the darkblue square with arrow keys. Your goal is to kill all gold objects by touching them, earning points along the way. Watch out for green objects- touching them will kill you unless you have white resources to protect yourself. Collect white resources to safeguard against green, and use them to kill green objects if needed, but be aware that each kill will cost resource." "
Glodel-generater guidance significantly improved learning for both humans (AmAUUC)= 0.15, < 10-5) and models (A(nAUC)= 0.088, p 10-5) compared to experience alone. Interestingly, models learned better from model- than human-generated advice (A(AUUC)= 0.052, p < 10-1)), while these gave humans only modest advantage (A(nAUC)= 0.035, p 0.26, n.s.).
These asymmetries reveal important differences in human and model communication. When we had our model generate advice based on human players' trajectories. models still learned better from this model-generated guidance than from human-written messages (AAaACC)= 0.21, p < 10 10-*), This suggests the asymmetry stems not from differences in knowledge, but from communication style. Human messages often included metacognitive strategies ("take time to look for safe patterns"), analogies ("Orange is like lerminator"), or emotional content ("[1) was left very confused")) aspects that human learners readily use but our model finds harder to interpret. These differences highlight the potential and challenges for language-medee more examples of human- and model-generated messages in Appendix Section I and
human-machine collaborative learning. We show website.
Ablating language-guiie guided proposals in our model leads to a significant performance drop compared to the full version (A(nAUC) -0.058, p = 7.2 x 10-4). This ablated version still leverages language likelihood, which lets it outperform learning from experience alone (A(nAUC) = -0.030, p = 5.7 10 10-3), see Method Section 3.1 and Appendix Figure 8.
Variability across games: Our results reveal interesting patterns in the success and failures of linguistic guidance (Figure 5b). Models learning from other models show consistent benefits across all games because models generate more comprehensive messages and process advice more reliably than humans on average. In contrast, human learners show minimal improvements in games requiring rapid reactions and precise motor control (portals, plaqueAttack, aliens, missileCommand) where linguistic advice cannot substitute for motor practice. Models struggle learn from humans in
human (median over games) model (median over games)
buman (hunan mag) bumtn (modal mag)
model (human meg) model imodel meg)
lovele salved
AWAMC
experience
exp + human mag model
30
plagee
pracond
bouldars
life
(a)
(b)
Figure 5: Bidirectional human-model learning. a) Median performance across games for humans and models learning from experience alone (blue), experience combined with human- generated (orange) or moner-gene (green) linguistic guidance (N=10, shaded: interquartitle range). b) Performance improvements over individual learning baselines for each game (error bar: sem).
8

---

Published as a conference paper at ICLR 2026
relational, where humans often provide imprecise descriptions of complex rule interactions, and plaqueAttack, where humans frequently omit important mechanics they discovered. In contrast, all learners benefit from both human and model advice in games like avoidGeorge, where the critical strategy is non-obvious, and beexAndBirds,, jaws, and pushBoulders, where key dangers are memorable and straightforward to describe. These patterns suggest that social learning strategies like selecting pedagogical teachers or aggregating multiple sources of advice could mitigate some failure cases.
5.4 GENERATIONAL LEARNING
While the previous experiments allowed social learners to benefit from fully explored game mechanics, real-world learning often relies on partial and imperfect knowledge transmission. To investigate whether our model could replicate this gradual accumulation of knowledge, we designed iterated learning experiment inspired by Tessler et al. (2021). In this setting, each agent interacts with the game environment for only two lives before generating advice to the next agent. This cycle continues across 10 generations for each of the 10 games, with performance tracked generation by generation.
Our results show that performance reliably increases across generations in all games where models do not already achieve mastery from generation (preconditions and aliens) (Figure 6). Some games reached near-complete mastery by Generation 2, while others showed more gradual improvements. A fixed-effect model controlling for game variability confirmed this trend, showing significant improvements over the first generation for all others (A(nAUC) E [0.44, 0.57), all < 10 10-19).
However, in plaqueAttack and relational, we observe occasional regressions where later genera- tions underperform earlier ones, highlighting the brittleness of single-teacher transmission. These regressions stem from structural properties of the games. Relational requires coordinating many inter- dependent transformation rules, and even when players know the correct rules, a single mis-push can irreversibly block progress within the two-life limit, making performance volatile across generations. PlaqueAttac involves fast-paced action with two different viable survival strategies (eliminating attackers or reconquering damaged bases). Messages can describe only one of these strategies, which can inadvertently steer later generations away from exploring the alternative, resulting in intermittent drops in performance. These task-specific constraints explain why cumulative improvement less stable in these games than in others with more linguistically compressible mechanics. This phe- nomenon could be mitigated through the integration of multiple teachers or teacher selection (Kendal et al., 2018; 8; Schultner et al., 2024). Together, these findings indicate that agents can build upon fragmented experiences to gradually refine their world models over generations, mirroring learning dynamics observed in human populations (Tessler et al., 2021).
DISCUSSION
Our approach aligns with longstanding research program in which human mental representations
modeled as structured, program-like generative theories, and learning is understood as probabilistic inference over these structures (Griffiths al., 2010; Lake et al., 2017; Rule et al., 2020). This perspective has been highly successful in explaining human causal reasoning (Tenenbaum et al., 2006; Griffiths and Tenenbaum, 2009), intuitive physics (Battaglia et al., 2012: Smith et al., 2023), or social reasoning (Baker et al., 2011; Ying et al., 2024): people form hypotheses about latent mechanisms,
portalc
precondttion
relatiunal
Figure 6: Generational learning via linguistic advice. Median performance of N 20 agents (black) across generations (2 lives each). Our models affords generational learning: performance increases across generations, beyond the performance of individual agents (generation 1).

---

Published as a conference paper at ICLR 2026
simulate their consequences, and revise them in light of new evidence. Executable, program like world models are a natural next step in this line of work (Tsividis et al., 2021), and our contribution can be seen extending this framework to the domain of social learning by integrating linguistic guidance and direct experience within a unified inferential model.
Our experiments showed that computational model reproduces key features of human social learning: advice reduces the attempts needed for success by shaping exploration, supports genera- tional knowledge accumulation, and even allows model-generated guidance help human learn- ers ers-ddeonstratinn bidirectional knowledge exchange.
While VGDL is only a coarse approximation of human game representations, our results show it captures enough structure to study social learning and afford bidirectional human- model knowledge transfer. Future work could leverage to model the emergence of shared representations through linguistic interaction, potentially driving representational convergence rather than requiring pre-aligned representations.
methods (Ellis et al., 2020; Wong et al., 2021)
Our results show that linguistic guidance often speeds and safeguards exploration, but when advice wrong it can restrict search- mirroring the "double-edged sword of pedagogy" observed in developmental psychology (Bonawitz et al., 2011). Humans counter this issue by evaluating testimony against prior causal theories before integrating it (Harris et al., 2018; Sobel and Kushnir, 2013). To match this sophistication, computational models will need meta-cognitive mechanisms to judge the reliability of advice and adjust their exploration accordingly.
This paper opens several avenues for future work. gennan-generated messages often include game abstractions, high-level strategies, and planning heuristics that our model currently cannot leverage. Extending the framework to interpret and learn from these richer forms of guidance c.g. through inference of auxiliary reward functions, planning abstractions and strategies - could unlock enhanced social learning capabilities (Silver al., 2024, e.g., ). It would also be valuable to examine in more detail which linguistic abstractions - beyond rule and win/loss information - facilitate robust generational transfer, building on methodologies from prior human veratedd-earnnng studies in VGDL environments (Tessler et al., 2021). Beyond passive learning, our model could be further extended to make decisions about who to learn from based on perceived expertise or success a capacity known cesstigebbaseddde based social learning (Kendal et al., 2018; Schultner et al., 2024). Future work could also investigate how different LLM families behave within our framework their dual roles as advice generators (speakers) and approximations of human speakers (speaker models). Because our method uses LLMs both to produce pedagogical messages and to evaluate the likelihood of human-wrir written advice, comparing families with different inductive biases could reveal how model-specific language priors shape both message interpretation and learning outcomes.
as prestige-
From an artificial intelligence perspective, an important direction for future work is extending this framework to real-world, continuous, and more complex environments. Doing so will require advances in program synthesis, scalable probabilistic inference, and hardware capable of performing inference over rich, unstructured programming languages such as Python (Tang et al., 2024; Lehrach al., 2025). This is active and rapidly growing area of research, 2 with significant investment and recent progress in LLM d-iivenn code generation and executable world modeling (Cusumano-Towee et al.. 2019; Lew et al., 2023; Loula al., 2025). Although this may seem challenging, humans routinely construct rich executable models in code physical engines, video game environments, simulation of complex systems which allow them to reason about highly complex processes, explore counterfactual scenarios, and deepen their understanding of the world. These practices illustrate the feasibility and potential benefits of scaling program-like world models to richer domains.
Finally, our results point to exciting possibilities for human-machine collaborative learning. Our model not only benefits from human-generated guidance but also contributes back through effec- tive, pedagogical advice - closing the collaborative loop. This demonstrates first step toward bidirectional learning systems capable of supporting human learners. The future directions outlined above handling richer linguistic guidance, adaptive trust in information sources, and scaling to open-ended domains - would represent major steps toward AI systems that not only learn efficiently but also teach, collaborate, and adapt within complex social learning networks, augmenting collective intelligence in hybrid Auman-Al communities (Colas et al., 2022; Brinkmann et al., 2023; Collins al., 2024).
"c.g., see recent library scaling probahilistic programming with GPUs via Jax.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Section details the full model specification and inference procedure. Section 4 describes the human and model experimental design. The appendices provide additional details including: the description of all VGDI. primitives, the inference pseudo-code, details about guided proposals and planning, instructions used for human data collection and full prompts used by the LM components of our model. The codebase will be released at sathuu..com/colas//ngggge saapinnggugeaaa__n____eeae laaiige__nd_e_eeeee and_exxerience, Together these resources allow full replication the experiments.
ACKNOWLEDGEMENTS
Cédric Colas is partly funded by the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Cun grant agreement No 101065949. This project was supported by Intel and the National Science Foundation under grants CCF-221 7064 and IIS- 2212310. Research was additionally sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2- 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
AUTHOR CONTRIBUTIONS
CC led the development of the computational model, conducted simulation experiments, and wrote the initial draft of the paper. and BP led the design and implementation of the human experiments, including participant recruitment and data collection. CC, MHT, NG, JA, and JT contributed to the formulation research questions, the high-level design of the computational framework, and editing the manuscript. All authors provided feedback and helped shape the final version of the paper.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
Kelsey Allen, Franziska Brändle, Matthew Botvinick, Judith E Fan, Samuel Gershman, Alison Gopnik, Thomas L Griffiths, Joshua K Hartshorne, Tobias U Hauser, Mark K Ho, et al. Using games to understand the mind. Nature Human Behaviour, pages 1-9, 2024.
Chris Baker, Rebecca Saxe, and Joshua Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the annual meeting of the cognitive science sociely, volume 33, 2011.
Chris L Baker, Julian Jara-Eitinger, Rebecca Saxe, and Joshua Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 1(4):
0064, 2017.
2017.
Peter Battaglia, Tomer Ullman, Joshua Tenenbaum, Adam Sanborn, Kenneth Forbus, Tobias Ger- stenberg, and David Lagnado. Computational models of intuitive physics. In Proceedings of the annual meeting of the cognitive science society, volume 34, 2012.
Elizabeth Bonawitz, Patrick Shafto, Hyowon Gweon, Noah D Goodman, Elizabeth Spelke, and Laura Schulz. The double-edged sword of pedagogy: Instruction limits spontaneous exploration and discovery. Cognition, 120(3):322-330, 2011.
Robert Boyd, Peter J Richerson, and Joseph Henrich. The cultural niche: Why social learning is essen- tial for human adaptation. Proceedings of the National Academy of Sciences, 108(supplement 2): 10918-10925, 2011.
11

---

Published as a conference paper at ICLR 2026
Levin Brinkmann, Fabian Baumann, Jean- Cean-François Bonnefon, Maxime Derex, Thomas F Müller, Anne-Marie Nussberger, Agnieszka Czaplicka, Alberto Acerbi, Thomas Griffiths, Joseph Hen- rich, et al. Machine culture. Nature Human Behaviour, 7011):185--186, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask Your Humans: Using Human Instructions to Improve Generalization Reinforcement Learning. Proc. of ICLR, 2021.
Nicolas Chopin. A sequential particle filter method for static models. Biometrika, 89(3):539-552,
2002.
Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Dominey, and Pierre-Yves Oudeyer. Language cognitive tool to imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 33:3761-3774, 2020.
Cédric Colas, Tristan Karch, Clément Moulin-Frier, and Pierre-Yve Oudeyer. Language and culture internalization for human like autotelic ai. Nature Machine Intelligence, 4(12):1068-100 2022.
Katherine M Collins, Ila Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao E Zhang, Tan Zhi-Xuan, Mark Ho, Vikash Mansinghka, al. Building machines that 2024.
learn and think with people. Nature Human Behaviour, 8 8(10):
Marco F Cusumano- Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: seneral-purpose probabilistic programming system with programmable inference. Proceedings of the 40th acm sigplan conference on programming language design and implementation, pages 221-236 2019.
Pierre Del Moral, Arnau Doucet, and Ajay Jasra. Sequential monte carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3):411-44) 2006.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Selam-Lezama, and Joshua B. Tenenbaum. DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning. arXiv:2006.0838) [cs], June 2020. URL http::///// / www.w.iww.aaaaaaaee..ea.o org/abs/ 2006. C8381. arXiv: 2006.08381.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998-999) 2012.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(1 1):818888881 829, 2016.
Prasoon Goyal, Scott Niekum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv: 2019.
Thomas I. Griffiths and Joshua B Tenenbaum. Theory-based causal induction. Psychological review, 116(4):661, 2009.
Thomas L Griffiths, Nick Chater, Charles Kemp, Amy Perfors, and Joshua B Tenenbaum. Probabilistic models of cognition: Exploring representations and inductive biases. Trends in cognitive sciences, 14(8):377-64, 2010.
Paul Harris, Melissa Koenig, Kathleen H Corriveau, and Vikram Jaswal. Cognitive foundations of learning from testimony. Annual Review of Psychology, 69(1):251 (:21))2552223 2018.
Felix Hill, Andrew K. Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L.. McClelland, and Adam Santoro. Emergent Systematic Generalization in a Situated Agent. Proc. of ICLR, 2020.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical Decision Making by Generating and Following Natural Language Instructions. Proc. of NeurlPS, 2019.
12

---

Published as a conference paper at ICLR 2026
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. Proc. of ICML, 2022.
Julian Jara-Ettingel Hyowon Gweon, Laura E Schulz, and Joshua B Tenenbaum. The naïve utility calculus: Computational principles underlying commonsense psychology. Trends in cognitive sciences, 20(8):589-666 2016.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language abstraction for hierarchical deep reinforcement learning. Proc. of NeurIPS, 2019.
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134, 1998.
Rachel L Kendal, Neeltje J Boogert, Luke Rendell, Kevin N Laland, Mike Webster, and Patricia L Jones. Social learning strategies: Bridge-building between fields. Trends in cognitive sciences, 22 (7):651-665, 2018.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
Wolfgang Lehrach, Daniel Hennes, Miguel Li, Antoine Dedieu, Jordi Grau-Moya, Mare Lanctot, Atil Iscen, et al. Code world models for general game playing. arXiv preprint arXiv: 10.04542, 2025.
Xinghua Lou, Carter Wendelken, Zun
Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint 230.002306.03081, 2023.
João Loula, Benjamin LeBrun, Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, et al. Syntactic and semantic control of large language models via sequential monte carlo. arXiv preprint arXiv:2504.1313,, 2025.
Jelena Luketina. Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. arXiv preprint arXiv: 1906.03926 2019.
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. Equation of state calculations by fast computing machines. The journal of chemical physics 2166):087-1002, 1953.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 5 518(7540:559-5533 2015.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding Language for Transfer in
Deep Reinforcement Learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi. Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling. In International Conference on Machine Learning, pages 26311 PMLR, 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint 1r001:211..00114 2021.
Davide Paglieri, Bartlomiej Cupiat, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, et al. Balrog: Benchmarking agentic IIm and vlm reasoning on games. arXiv preprint 1rXiv:24/1.13544 2024.
Joshua Rule, Joshua Tenenbaum, and Steven Piantadosi. The child as hacker. Trends in cognitive sciences, 24(1 1)9100900--15 2020.
13

---

Published as a conference paper at ICLR 2026
Tom Schaul. A video game description language for model-based or interactive learning. In 2013 IEEE Conference on Computational Inteligence in Games (CIG), pages 1-8. IEEE, 2013.
David Schultner, Lucas Molleman, and Bjöm Lindström. Feature-based reward learning shapes human social learning strategies. 2024.
Patrick Shafto, Noah D Goodman, and Thomas Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55-89, 2014.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas, Skill Induction and Planning with Latent Language. Proc. of ACL, 2021.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Cote, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. Proc. of ICLR, 2021.
Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B Tenenbaum, Leslie Kaelbling, and Michael Katz. Generalized planning in pddl domains with pretrained large language models. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 20256-20264, 2024.
Kevin Smith, Peter Battaglia, and Joshua Tenenbaum. Integrating heuristic and simulation-based reasoning in intuitive physics. 2023.
David M Sobel and Tamar Kushnir. Knowledge matters: how children evaluate the reliability of testimony as process of rational inference. Psychological Review, 120(4):779, 2013.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Hao Tang, Darren Key, and Kevin Ellis. Worldcoder, model-based Ilm agent: Building world models by writing code and interacting with the environment. arXiv preprint 2xXv:2240..12275 2024.
Joshua Tenenbaum, Thomas I. Griffiths, and Charles Kemp. Theory-based bayesian models of inductive learning and reasoning. Trends in cognitive sciences, 10(7):309-318, 2006.
Michael Henry Tessler, Jason Madeano, Pedro A Tsividis, Brin Harper, Noah D Goodman, and Joshua Tenenbaum. Learning to solve complex tasks by growing knowledge culturally across generations. arXiv preprint arXiv:2107.13377 2021.
Michael Tomasello. The cultural origins of human cognition. Harvard university press, 2009.
Momchil Tomov, Pedro Tsividis, Thomas Pouncy, Joshua B Tenenbaum, and Samuel Gershman. The neural architecture of theory-based reinforcement learning. Neuron, 111(8): (118):1331-1333 2023.
Pedro A Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel Gershman, and Joshua B Tenenbaum. Human-level reinforcement learning through theory- based modeling, exploration, and planning. arXiv preprini arXiv:2107. 12544, 2021.
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their plans? arXiv preprint arXiv:2310.08//8. 2023.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
Natalia Vélez and Hyowon Gweon. Learning from other minds: An optimistic critique of reinforce- ment learning models of social learning. Current opinion in behavioral sciences, 38:1 10-115, 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
Catherine Wong, Kevin Ellis, Joshua B. Tenenbaum, and Jacob Andreas. Leveraging Language to Learn Program Abstractions and Search Heuristies. Proc. of ICML. 2021.
14

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:2210.06629, 2022.
Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka, and Joshua Tenenbaum. Inferring the goals of communicating agents from actions and instructions. In Proceedings of the AAAI Symposium Series, volume pages 26-33, 2023.
Lance Ying, Tan Zhi-Xust Lionel Wong, Vikash Mansinghka, and Joshua Tenenbaum. Under- standing epistemic language with a bayesian theory of mind. arXiv preprint 14X0:22488.1002, 2024.
Tan Zhi-Xuan, Paul Stefan Lunis, Nathalie Fernandez Echeverri, Vikash Mansinghka, and Josh Tenenbaum. Language models informative goal priors in bayesian theory of mind. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023.
Tan Zha, Lance Ying, Vikash Mansinghka, and Joshua B Tenenbaum. Pragmatic instruction following and goal assistance via cooperative language-guide inverse planning. arXiv preprint 1rXiv:2402.179300 2024.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to new environment dynamics via reading. ICLR, pages 1-17. ICLR, 2020.
15

---

Published as a conference paper at ICLR 2026
Appendices
APPENDIX CONTENTS
Extended related work
16
Learning from experience alone
17
C VGDL: Game primitives, state and action spaces
17
D Inference pseudo-code
19
E Guided proposals
19
F Details about planning
20
G Human data collection: instructions and participant recruitment
21
Computational resources
22
I Example messages
22
Prompts
25
K Baseline comparisons
35
A EXTENDED RELATED WORK
Language and RL. Language has emerged as a powerful tool to enhance reinforcement learning (RL) by conveying state abstractions (Narasimhan et al., 2018), world dynamies (Zhong et al., 2020), auxiliary reward functions (Goyal et al., 2019), and task decompositions (Shridhar et al., 2021; Sharma et al., 2021). Its inherent compositionality and abstraction capabilities allow agents to pursue more abstract goals (Jiang al., 2019). generalize effectively across diverse environments (Zhong et al., 2020) and goals (Hill et al., 2020; Colas et al., 2020), and structure long-term decision makiing more effectively (Hu al., 2019; Chen al., 2021). Despite these advances, most language- based RL. approaches rely on substantial amounts paired data to ground linguistic information in agent experience, limiting their scalability and resemblance models (LMs) promise more flexible combination of language and decision-making (Ahn et al., 2022; Huang et al., 2022), yet they struggle to learn complex embodied skills that require low- level perception and temporally extended actions (Valmeekam et al., 2023; Paglieri et al., 2024). Nottingham al. (2023) use LLMs generate complete world model hypotheses that are then verified through experience. Our approach introduces Bayesian framework that treats experience and language as two complementary sources of evidence the inference of world models. This joint inference enables rapid adaptation to new linguistic inputs and tasks from the very first interaction, bypassing the need for extensive paired data.
human-like social learning. Language
Bayesian models of social cognition. Bayesian models of social cognition have proven to be powerful tools for modeling theory of mind (ToM) the ability to infer the hidden goals, beliefs, and intentions of others from observable behavior linguistic cues (Baker et al., 2011; 2017; Frank and Goodman, 2012; Goodman and Frank, 2016). These models formalize social inference as an inverse planning problem, where observers assume that agents act approximately rationally towards their
16

---

Published as a conference paper at ICLR 2026
goals and use this assumption to infer likely mental states (Baker et al., 2011: 2017). Extensions of these frameworks to language understanding have resulted in the Rational Speech Acts model, which interprets communication recursive social reasoning: listeners reason about what speaker intends to convey based on the assumption that speakers choose utterances optimally, given their beliefs (Frank and Goodman, 2012; Goodman and Frank, 2016). More recent work integrates Bayesian reasoning with modern machine learning to enable richer social inferences. For instance, (Zhi-Xuan al., 2023) leverage large language models (LLMs) priors in Bayesian goal inference system, allowing for the efficient suggestion and evaluation of likely goals in complex environments. Similarly, (Ying et al., 2024) introduce
ToM model that translates natural language statements about beliefs into formal epistemic representations, enhancing agents' ability to reason about others' knowledge and intentions. By combining linguistic input with structured probabilistic models, these approaches extend traditional ToM beyond purely behavioral cues, opening new avenues for interactive and socially aware Al agents (Vélez and Gweon, 2021). However, these models operate in settings where the transition dynamiis are fully known and deterministic, and contrast, our
inference is restricted to identifying a latent goal or belief state from demonstrations. setting requires agents to infer the entire causal structure each new environment object types, interaction rules, and win/loss conditions-directi language and action substantially more challenging. Our work builds on these ideas by leveraging Bayesian inference to jointly interpret linguistic and experiential data, allowing agents to efficiently acquire world models from sparse social interactions.
from experience, making joint inference from
B LEARNING FROM EXPERIENCE ALONE
mondiCCemmnt
slenunditcck
Figure 7: Learning from experience. Both humans and our model achieve high sample efficiency across most games. Systematic differences emerge in relational, which requires systematic testing of object combinations, and avoidGeorge, which demands rapid and precise planning. Deep RI. and pure LM baselines fail to solve games efficiently, highlighting the value of structured representations (N=20, median + interquartile range).
C VGDL: GAME PRIMITIVES, STATE AND ACTION SPACES
We work with subset of the VGDL domain. Here the possible types of avatars:
MovingAvatar: controllable player that can move in the four directions with a certain speed based keyboard presses
ShootAvatar: MovingAvatar that can also shoot objects stype when the player presses the space bar
FlakAvatar: ShootAvatar that can only move sideways and always shoot upwards.
Here are the possible object types and their parameters:
Immovable: object that cannot move
Flicker: object that disappears after total steps
SpawnPoint: object that spawn objects stype with probability p
ResourcePack: object that can be collected (see interaction addResource and removeResource)
Passive: object that be pushed (see interaction bounceForward)
Missile: object that moves in one direction with a certain speed and an original orientation.
They can change direction (see interactions turnAround and reverse/Drection
17

---

Published as a conference paper at ICLR 2026
Bomber:
the combination of missile and spawner (with their combined parameters)
Chaser: object that moves in the direction the nearest target object st yte, with certain speed
RandomMPC: object that moves in a random direction with certain speed
Portal: object that can teleport another object contacting it to another exit object (see interaction teleportTo).
All moving objects move every coo 1 down environment steps.
Interactions describe what happens when two objects contact. Here are the possible interaction types and their parameters:
nointeraction: nothing happens
killSprite: the second object kills the first object
transformo: the second object transforms the first object into a third object st stype
resove:esource: the second object decreases the count of resource from the first object
addResource: the second object increases the count of resource of the first object
killfHassess: the second object kills the first if has less than I resource stype
stepBack: objects step back (second steps back first, if not possible the second does)
bounceForward: the second object pushes the first if possible (e.g.. unless is blocked)
urnAround: the first object (a missile) does one block down and switches direction when encountering the second object
reverseDirectton: the first object (a missile) reverses direction when encountering the second object
killSprite and transformTo interactions can further lead to a positive (+1), negative (-1) or null (0) reward.
Lose conditions can be:
Timeout:
the player loses if it runs out of time before solving the task
CoumtsZZerooobjs): the player loses at least one of the objects objs has no remaining instances in the game.
Win conditions can be:
Survive: the player wins if it survives long enough
(ount)sZerooobjs):: the player wins if the count of all objects objs goes to zero (e.g., they were all killed disappeared)
Size of the search space and game difficulty.
The size of the search space is direct function of the number of objects. The avatar must have avatar type among 3, each other object should have an object type among 10; each pair of objects must have an interaction type among 10; win and lose conditions can apply any list of objects from 0 to all. Without accounting for type parameters (e.g., chasing object have a target, transform interaction transform objects into specific type, etc), game spaces already scale 1090 for the smallest games involving 5 objects Command, preconditions). Other games might have up to 10 objects (plaqueAttack, portals). The difficulty of a game is not necessarily correlated to the size the search space. For instance, some object types can be inferred quickly from movement observations, while others require direct interactions. The layout and game rules might also make exploration, or planning more less complex for different games.
Perception and action. Players have five possible actions: four directional moves (arrow keys) and shoot action (space bar) when the game includes ShootAvatar or FlakAvatar. Computational agents also have access to a NOOP action, while human players can simply choose not to act. After taking non-NOOP action, computational agents must wait for environment steps before being able to take a new one, a way of capturing human reactive time (5 fps). The game runs at 20 frames per second. Humans perceive the game visually, with the current reward displayed at the top (see
18

---

To p(?) Published as a conference paper at ICLR 2026
Figure 2), whereas computational agents perceive symbolic states consisting of object properties (object id, color, position), rewards, and win/loss events.
D INFERENCE PSEUDO-CODE
Algorithm shows the pseudo-code of the inference algorithm- a particle filter with Metropolis rejuvenations - used to update the agent's s beliefs about possible games it might be playing, supported by experiential and linguistic evidence (Del Moral al., 2006; Metropolis et al., 1953). The Perturb operator replaces one rule of the VGDL game description at time, according to guided proposals (see Appendix Section below). We picked the number of particles be N 20.shis involves trade-off:
Exploration: too few theories lead to under-reprresentation of uncertainty, reducing the model's ability to identify informative subgoals and to explore sufficient alternative variations in the space of possible thcories.
Computation: each Metropolis update requires updating all particles and computing language likelihoods via the LLM, which is the computational bottlenek of the approach.
Empirically, 10 particles was faster but sometimes failed to discover tigh-posterior theories quickly enough for the agent survive, while 20 ensured a more robust and fast exploration of the theory space. More particles might slightly improve the performance of the inference, but will make the system slower to run and experiment with.
Algorithm Particle filter with Metropolis rejuvenation
Params: . number of particles/theoriess, P(T) prior over theories, S, R number of steps and rejuvenation
steps.
Inputs: inference step s, new experiential data D., particles O, (for 0).
bold indicates vectors size N
if s = then
Sample initial particles from prior
for E [1, S] do
D inference step
T. T.1
for [1, R. do
MCMC rejuvenation step Propose move
T + Perturb(T.)
a < p(D..T)
D Metropolis acceptance
p(D..T))
for i E [1, N] if U(0,1) < a' do
T: T"
Accept move
w + Norm(p (emn(p(D..1 D2, I:))
Update weights SMC resampling step
D
T. + Resample(T.... w)
Return: mrg TH Tus p( D., T14s) Ti4s)
D Return MAP theory
Tus
E GUIDED PROPOSALS
We use biased proposals to initialize the set of 20 candidate theories and to generate rejuvenation moves (Perturb operator in Algorithm 1). These proposals guided by both experience and linguistic evidence, accelerating convergence towards theories that better explain the agent's observations.
Experience-driven proposals bias the sampling process in the following ways:
objects observed to have moved cannot be assigned an object type incompatible with movements (e.g., Immovable, or Flicker),
objects moving in one direction are more likely to be assigned type object types that move linearly (Missile and Bomber than object types that allow movements in all directions (RandomNPC., Chaser),
objects pairs involved in collisions preceding observed rewards are more likely to be assigned loward-generating interactions.
19

---

Published as a conference paper at ICLR 2026
model (median over games)
3
tevels solved
(experience) model (exp + model msg)
(exp msg, proposals)
0-
10
life
Figure 8: Ablation of language-guided proposals. a) Median performance across games for models learning from experience alone (blue), or experience and language, using language likelihood and language-guided proposals (orange), or ablating language-guidee proposals (green). (N=20, error bar: interquartile range).
F DETAILS ABOUT PLANNING
Our model strategically balances curiosity-driven exploration with goal-directed exploitation, mirror- ing human soollmm-oovviggg strategies (Tsividis et al., 2021). From the current best theory TMPP, we generate candidate subgoals: specific collisions between pairs of objects that the agent can cause to occur by either touching another object itself, or by pushing spawning object onto it. Candidate subgoals are assigned values based on their exploration and exploitation potentials:
(alue = Exploration (alue(g) + Exploitation (aluiaa))ooValu((g)))))) Value(g).
The exploration value measures disagreement between theories in the current population:
Exploration Value()) max; count(i (g)
Σ, count(i (gg)
where count(i g) is the count of theories assigning interaction type i to subgoal The exploitation value rewards key game mechanics:
10 if g contributes to win condition
8 g protects essential resources
VaplottationValue())) =
6 if g creates useful tools
2 if g collects resources or rewards
0 else
To achieve these goals, the model evolves short 10-step action sequences (e.g. move left, shoot, move up) through stochastic search, using TMAP simulate outcomes. Action plans are iteratively mutated and refined over three generations using a simple genetic algorithm, where mutations crop and regrow action sequences from uniformly sampled mid-point. Each sequence is evaluated according
V(a) = Rgame (a) + Rgoal (a) + Rwintoss (a),
where Ryame is the game's 's reward function under TMAP, Rgoal rewards progress toward the selected subgoal (between and 1), and Rwintess provides +100 for winning and- -100 for losing.
Additionally, the model performs 10 3-step lookaheads to avoid catastrophic errors, triggering replanning when the originally predicted value deviates significantly from the distribution of values for newly simulated trajectories: V (Coriginaa)
max (ajookaabadd) ).
#E1...10
This safety mechanism prevents the agent from executing plans that appeared promising under limited simulation but fail under more extensive testing, or when unexpected changes in the environment render the original plan ineffective.
20

---

Published as a conference paper at ICLR 2026
G HUMAN DATA COLLECTION: INSTRUCTIONS AND PARTICIPANT RECRUITMENT
We recruited 122 adult English- participants through Prolific to play 5 randomly-assigned games. To ensure task engagement while maintaining representative sample, we excluded partici- pants who failed complete least one level 2 3 games (final N=120). Participants' median completion time was 49.30 minutes, and the median hourly pay rate was $10.41/hr.
Before playing the games, participants read the following instructions:
Instructions for human data collection
this experiment, you will play di fifrentt games before describing each game to another participant who has not yet played.
You will play 5 different games. Each game will be played using the arrow keys and spacebar your keyboard.
You will start each game with lives. Each gane has levels. Your goal is to win all levels of the game in as few lives as possible.
You lose life when you lose level. There are different ways of winning and losing. Possible ways win level include removing all objects of particular colors, reaching a particular object, or surviving for long enough.
Possible ways to lose level include allowing all objects of particular colors die di disapeer, or by solving the level fast enough.
You move en to the next level of game only after winning the current level. You will finish game when you win all 4 levels, when you have lost lives.
After finishing game, you will be asked to describe the game to another participant who has not yet played. Your goal is help them solve the game in few lives as possible. You will be required to spend at least seconds describing each gane.
\textbf (Benus opportunity: Selve each game in as few lives as possible!!
On each game, you can win up to $1$0.205 based on your performance. Your rank en each game will be computed relative to other participants, based on the number of levels you complete, and how many lives you take to complete them. For each game you will earn the max bonus multiplied by the proportion of other participants you rank!
\textbf [Benus opportunity: Write useful descriptions!]
On each game, you win up to an additional S1$0.205 based your descriptien, Another group of participants will read descriptions before playing each game, and receive bonuses up to $150.20$ based on their relative rank. On each game, you will additionally receive the bonus earned by the player who reads your description, so help them win!
// l0pticnally, for participants in condition and 3 (social learning)
when starting each game, you will be shown message from another player who has already played the game. You should use this message to help you play the game, but keep in mind that it's possible that it contains mistakes.
When starting each game, participants in Condition received no game specific advice, as shown in Figure 9a, while participants in Conditions 2 and 3 saw message from previous player, as shown
21

---

Published as a conference paper at ICLR 2026
in Figure 9b. Players in Conditions 2 and 3 still read the message as they play. After completing each game, participants in all conditions were asked to describe the game to a future player, as shown in Figure 9c.
Play game!
Write message help player
you've the game
describing:
how
how you can lose the game other key interactions
Please use the color legend to the right when referring to game elements.
LIG-TGWW
(a) Game play, Condition 1.
DARKBLUE
OREEN
LIGHTBLUE
Play game!
tu
ORANGE
(c) After playing the game.
(b) Game play, Conditions and 3.
Figure 9; Screenshots of the game screens for the different experimental conditions. (a) Condition I: individual gameplay instructions. (b) Conditions 2 and social gameplay instructions. (c) Message
interface shown to participants for writing advice to future players. Note that players in the social conditions can see the message during gameplay.
H COMPUTATIONAL RESOURCES
The model requires GPU to the LM, using the vLLMM library to speed up the generation of proposals. Simulation runs vary in function of game complexity (slower with more objects) and the learning speed of the agent (runs end early when all levels are solved), taking anywhere between an hour and a day, depending on games and conditions.
NVIDIA A100 (80Gb). We use prompt caching
EXAMPLE MESSAGES
Below example messages generated by humans and model players in the social learning experi- ments.
Example of human messages in social learning experiment
[beesAndBir rds] "Move the darkblue square by pressing the arrows keys. You win the game the toucning the green square, can also touch red but not needed. Avoid touching the moving yellow square because they can kill you, The green squares can destroy the yellow and orange squares when you free it from the purple squares. You can destroy purple squares by touching it."
shoot and hit the pink moving squares. You are also needed to dodge the red squares that are being shot by the pink squares. You will dodge by using the left, right, up
[aliens] "You
space
22

---

Published as a conference paper at ICLR 2026
and down keys. You must destroy all the pink squares before they reach the bottom."
[jaws] think if you nove the block over the puzple and reds you will win the game. You will lose the game if you go over other color blocks."
[preconditiens] "eat the white tiles to enable you knock down the green tiles toget to the yellow tiles. your objective is to get to the yellow tiles"
the main goal is protect the GREEN tiles by making sure the RED tiles and the YELIOW tiles don's t touch them and you can win even if you only have ONE GREEN tile left. As soon as you click START, they will start moving and you need to act fast by trying to interact with the RED and YELLOW tiles make them di disappear using the SPACEBAR while next to them. move faster than the red so try to prioritize them if possible but there will be times that you would need to sacrifice one GREEN tile to make one YELLOW tile disappear because at that same time, you can possibly eliminate 2-3 RED tiles because they slower. You win the least one GREEN tile intact and you lose when you losc all five of them."
[missile crissile_ccmmante ccmmand]]
YELLOW tiles
level by keeping
BeishBoulders "Move darkblue square by using the arrow keys. Avoid touching the orange squares because they kill you, push the green into orange to make orange disappear. Avoid pink square because it can kill, move pink by pushing green into it. You can touch lightblue square to make it disappear. Also you can touch yellow square to make it disappear and win the game."
[avoi [avoidGeorge] "Game 4: Move your dark blue square quickly to avoid the light blue square. Use the brown spacebar to turn the green squares back to purple. Shoot the squares next to you with the brown square eliminate them.
[plaqueAttack] "Use your arrows to shoot from your square to protect the red and yellow squares. If the other colors get to red and yellow and turn them. before you eliminate them, you lose."
[relational onal] "Use the arrow keys to move the orange cubes to merge the darkblue and white cubes by pressing the space bar key. Avoid colliding with the rest of the cubes in the game"
[portals] "Yeu need to move your character over the gold and light blue squares. To win the game you need to get to the light green square. In the levels coming up you needs to move to dark blue to get to different areas. Do not let the other colors touch you because you will lose life."
from light blue
Example of model messages in social learning experiment
[beesAndBirdee rds] "Use arrow keys to move your darkblue. To win, eliminate all green by toucning them, earning points for each kill. orange, but you can
Be careful: you' die it you touch yellow lightgreen to your advantage it can kill both yellow and orange. Touch red and purple to kill them, but they' re not your main objective."
your darkblue square and spacebar to shoot blue squares. To win, eliminate all orange, pink, and brown shooting blue at pink will kill them and earn.
[aliens] "Use left and right keys
23

---

Published as a conference paper at ICLR 2026
points, but watch out as pink can kill you on contact. Protect yourself f from pink and red, which can also kill you, and avoid touching them. Blue objects help by killing pink, but be that blue and white objects kill each other, and pink can generate more red objects."
[jaws] "Centrol your darkblue square with arrow keys. Your goal is to careful, as orange, brown,
survive fer as long as possible. pink objects will kill you on contact orange objects will ever chase you. Stay away from them and try to navigate around the safe darkgray, purple, and green objects to stay alive."
[precenditiens] "Use arrew keys to move your darkblue square. Your goal is to eliminate all gold objects by touching them to earn points. Be careful, as green objects will kill you on contact unless you have white resource to protect yourself collect white resources stay safe. they'll take a resource from you in the process."
can also kill green objects by touching them, but
[missile [missile_comaand cemmand] "Use arrow keys to move the darkblue square and spacebar to shoot pink. To win, eliminate all red and gold shoot pink at them to kill and carn points. Protect green objects, as you' 11 lose if they all dic, and try to avoid letting red and gold kill them as this will also cost you points. Red and gold objects will chase green, so use this to your advantage and take them out with pink squares."
[pushBoulders] "Use arrow keys to nove your darkblue square. To win, eliminate all gold objects by touching them or pushing green onto them you'
points for each gold destroyed. Be careful:
touching crange or pink will kill you. Green it can alsc kill gold and earn you points, so use it to your advantage by pushing it onto gold. Protect yourself and keep pushing green to clear the way to victory, as dying means losing the game."
valuable ally, as
[avoidGecrge] "Use arrow keys to control your darkblue square and spacebar to shoot brown. To win, survive long enough while keeping green objects alive they can be created by shooting brown at purple or letting crange touch purple. Watch out: lightblue objects will kill you en contact and can also protect your green! You'1l earn points when brown or orange transform purple inte green, but lose points if lightbl ue kills you."
green back into purple, so
keys nove your darkblue and spacebar to shoot lightred. Your goal to eliminate all purple, orange, brown, and lightblue while keeping yellow and red alive. To help you, lightred objects will kill orange and brown on contact, so position them to clash. You can also shoot lightred at orange and brown to kill them directly. Touching green objects transforms them into yellow, earning you points, and touching blue objects transforms them into red, also earning points. Be careful, as orange and brown can transform yellow into green, lesing you points, and blue objects can"
[plaqueAttack] "Use
[relational] "Use arrow keys to move your darkblue square. To win, eliminate all blue objects, which can be done by pushing them onto yellow objects when they kill each other, you gain points. He careful not to touch white objects, as they' kill you. You transform objects
also help your strategy: touch red to turn them into yellow, touch pink to turn then into orange, and push orange onto purple to turn them into yellow."
[portals] "Use arrow keys eliminate all lightgreen objects by touching them earn points and win the game. Be careful darkgreen, pink, and lightred objects
your darkbiue square. Your goal is
24

---

Published as a conference paper at ICLR 2026
will kill you on contact, so avoid them. You can teleport using 1ightblue to gold and blue to lightorange to help navigate. Brown objects move randomly, but won't harm you, and other objects staticnary, so use them to your advantage.
J PROMPTS
Prompt for language likelihood estimation and language generation
Game Message Generation Task
You writing standardized messages describe 2D video game pased on the game's s mechanics.
IHF Instructions
Each message should explicitly state:
How to centrol the game
How to win
How you could lose/die
How to get/lose points
Key game mechanics that aren't obvious while playing
### Context
The player controls the darkblue object
In some games, it can shoot other objects of particular color Rach object color is associated conditionins how they move
particular object type
Effects can apply when two colors collide, eg one can kill the other, one can transform the other into a third color, one can push the other,
Possible wining conditions include: surviving long enough, or killing all objects of one or several target colors
Possible lesing conditions include: reaching a timeout before wining, or seeing all objects in a list of colors dying/dissappaaring
### Writing Rules
Mention all important interactions in the game
State each interaction in a single clear sentence
List all killing and transformation interactions Always specify win and loss conditions
I# Example
Who you are and how you
You control the darkblue square with arrow keys.
How you win and lose:
You get points when pink objects transform blue objects into purple objects.
You lose peints when darkred objects kill purple objects.
You win the game when all blue and orange objects are dead.
You lose if all purple and lightred objects die. What you can do:
You shoot pink squares by pressing space
You can kill orange objects by shooting pink objects at them. You can transform blue objects into purple objects by shooting pink objects at them.
what can kill you:
Orange objects will kill you if you touch them.
Other possible interactions:
You can push lightred objects.
25

---

Published as a conference paper at ICLR 2026
Darkred objects can kill purple objects by touchi them.
Darkred objects kill lightred objects by touching them.
Other objects:
Pink objects move along one direction.
Darkred objects chase the nearest purple object.
Orange objects move along one direction.
Purple and blue objects do not nove.
Message:
You contrel darkblue square with arrow keys. You can shoot pink objects by pressing space bar. Orange objects kill you when they touch you, Pink objects kill orange objects when they touch them. Pink objects transform blue objects into purple objects when they touch them. You can push ligatred objects. Darkred objects kill purple and lightred objects when they touch them. Pink objects move in one direction. Darkred objects chase the nearest purple object. Orange objects move one direction. Purple and blue objects don't move. You get points when pink objects transform blue objects into purple objects. You lose points when darkred objects kill purple objects. You win when all blue and orange objects are dead. You lose if all purple and lightred objects die.
FFF Task
(THEORY DESCRIPTION)
Message:
(MESSAGE)
# we compute the likelihood of these tokens
Prompt for making rules proposals
Game Rules Analysis Task
You are analyzing messages written by video gane players to help you play game for the first time. These games feature colored objects 2D space, where players control darkblue object. The messages hazards, core mechanics, and give you tips to help you win the game.
interacting describe controls, scoring systems, win/loss conditions,
#+F Task
You will be asked to answer four types of questions:
what is the type of objects of [color1]?
What happens to objects of [colerl] when they collide with objects [color2]?
Does the player need to kill all objects of [colorl] to win?
Does the player lose when all objects of [colorl] die?
Answer questions by looking for explicit statements in the message about:
How objects move and how they behave, eg can you use them as a portal, can you push them, do they chase other objects, etc.
Objects killing other objects
Oojects transforming into other objects
Min conditions involving killing objects
Loss conditions involving object death
||| Rules for Answering
Only information explicitly stated in the message
You are the darkblue object.
Do not make assumptions about interactions not mentioned
26

---

Published as a conference paper at ICLR 2026
Pick the "I don't know" answer if the corresponding information is ambiguous, centradicting, absent
Pay attention to interaction direction it kills B, it doesn't mean kills A
Each question comes with fixed set of possible answer. You must pick one of the answer. If the correct answer is not in the list, pick the "I don't know" answer.
If the way to win is kill some object, then having objects disappear/die does NOT cause you to lose
these
If you need to "protect [colorl] objects" or "kill [color2] objects before they kill transform [colorl] objects", you must answer:
"You lose if all [color1] objects die disappear", note that transformations kill the transformed objects.
If you need to reach touch kill by touching [colorl] objects to win, this means you can kill colorl objects. You MUST answer: "[color]] objects die when they collide with darkb' ue objects". if the message says you can "collect [colorii":
either [color]] is a resource, and you must answer that "you can collect [color]] resources"
OR [colorl] is net resource, and you must answer that "you can kill [color]] objects"
If the message says you can destroykil/eiiiiate [colorl] objects if you have enough [color2] resources, the two following statements are truc:
"[color]] objects die when they collide with darkblue objects but also take [color2] resource from them!
AND "darkblue objects die if they don't have enough resources when they collide with [colorl] objects"
lack resources also die when they touch you, and take one resource from you.
Opjects that kill you when
"[[color]] objects die when they collide with darkblue objects but also take [color2] resource from them"
when [celerl] objects take away some resource / your health when you touch them, they also die:
"[color]] objects die when they collide with darkblue objects but also take [color2] resource from them'
Use the after "Reasoning:" to reason about the game before gencrating your answer.
#*# Example Game
Message from the player:
"Orange objects kill you they touch you. You kill purple objects when you touch them. Lightgreen objects chase orange objects and can kill them when they touch them. You win when you kill all purple objects. You lose if orange kills you."
What happens to darkblue objects when they collide with orange objects?
1) nothing happens darkblue objects when they collide with orange oojects
2) darkblue objects die when they collide with orange objects
3) darkblue objects get transformed when they collide with orange objects
4) darkblue objects steal rescurce from orange objects when both
collide
5) darkblue objects die if they don't have enough resources when they
collide with orange objects
6) I don't know something clse
Reasoning:
27

---

Published as a conference paper at ICLR 2026
Answer (pick one from the list):
2) darkblue objects die when they collide with orange objects
What happens to orange objects when they collide with darkblue objects?
1) nothing happens to orange objects when they collide with darkblue objects
2) orange objects die when they collide with darkblue objects
3) orange objects get transformed when they collide with darkblue oojects
4) orange objects steal resource from darkblue objects when both collide
5) orange objects die if they don't have enough resources when they
collide with darkblue objects
6) I don't know something else
Reasoning:
Answer (pick one from the list): 6) I don't t know something clse
orange kill darkblue (the player), but don't t know what happens orange when they touch (maybe nothing)
Note:
How orange objects behave?
1) orange objects cannot move
2) orange objects do not move and disappear after a certain time
3) orange objects regularly ggawn//enerate other objects
4) orange objects can be pushed
5) orange objects
along axis
6) orange objects move along one axis and regularly spawn/generate objects
7) orange objects chase or flee another object
8) orange objects
randomly
9) orange objects portals
10) don't t know
Reasoning:
Answer (pick one from the list): 10) don't t know
How do lightgreen objects behave?
1) lightgreen objects cannot move
2) lightgreen objects do not move and disappear after certain time
3) lightgreen objects regularly spawn/generate other objects
4) lightgreen objects can pushed
5) lightgreen objects move along one axis
6) lightgreen objects move along one axis and regularly spawn/ generate objects
7) lightgreen objects chase or flee another object
8) lightgreen objects move randemly
9) lightgreen objects are portals
10) don't know
Reasoning:
28

---

Published as a conference paper at ICLR 2026
Answer (pick one from the list):
7) lightgreen objects chase or flee another object
Do you need to kill purple objects to win?
1) To win you need to reach/touch purple objects
2) you need kill all purple objects
3) win you don't need to kill all purple objects
4) To win I don't know if you need to kill or reach/touch all purple
objects
Reasoning:
Answer (pick one from the list): 2) To win you need to kill all purple objects
Do you need to kill orange objects to win?
1) win you need reach/touch orange objects
2) you need kill all orange objects
3) To win you don't t need to kill all orange objects
4) To win I don't t know if you need to kill or reach/touch all orange
objects
Reasoning:
Answer (pick one from the list): 4) To win I don't knox if you need to kill reach/touch all orange objects
you lcse if orange objects die?
1) You lcse all orange objects die or disappear
2) You don't lose if all orange objects die or disappear
3) I don't t know if you would lose if all orange objects die disappear
Leasoning:
Answer (pick one from the list): 2) You don't lose if all orange objects die disappear
HI Example Game 2
Message from the player:
"You can shoet brown objects. Brown objects kill gold objects when they touch them. Gold objects transform purple objects into lightred objects when they touch them. You transform lightred objects into purple objects when you touch then. You lose if all purple objects are transformed into lightred objects. You win if you survive long enough."
What happens to gold objects when they collide with brown objects?
1) nothing happens to gold objects when they collide with brown.
objects
2) gold objects die when they collide with brown objects
3) gold objects get transformed when they collide with brown objects
4) gold objects steal resource from brown objects when both collide
29

---

Published as a conference paper at ICLR 2026
5) gold objects die if they don't have enough resources when they dollide with brown objects 6) don't know something else
Reasoning:
Answer (pick one from the list):
2) gold objects die when they collide with brown objects
What happens to purple objects when they collide with gold objects?
1) nothing happens purple objects when they collide with gold objects
2) purple objects die when they collide with gold objects
3) purple objects get transformed when they collide with gold objects
4) purple objects steal resource from gold objects when both collic dd
5) purple objects die if they don't t have enough resources when they
collide with gold objects
6) I don't know something else
Reasoning:
Answer (pick one from the list): 3) purple objects get transformed when they collide with gold objects
What happens to lightred objects when they collide with gold objects? nothing happens lightred objects when they collide with gold objects
2) lightred objects die when they collide with gold objects
3) lightred objects get transformed when they collide with gold objects
lightred objects steal rescurce from gold objects when both collide
5) lightred objects dic if they don't have enough resources when they
collide with gold objects
6) don't know something else
Reasoning:
Answer (pick one from the list): 6) don't know something else
How do lightred objects behave?
1) lightred objects cannot move
2) lightred objects do not move and disappear after a certain time
3) lightred objects regularly spawn/generate other objects
4) lightred objects can be pushed
5) lightred objects move along
axis
6) lightred objects move along one axis and regularly spawn/generate objects
7) lightred objects chase or flee another object
8) lightred objects move randomly
9) lightred objects are portals
10) don't know
Reasoning:
Answer (pick one from the list):
10) don't t know
30

---

Published as a conference paper at ICLR 2026
Do you need to kill brown objects win?
1) To win you need to reach/touch brown objects
2) To win you need to kill all brown objects
3) To win you don't need to kill all brown objects
4)
I don't know you need to kill or reach/touch all brown
objects
Reasoning:
Answer (pick one from the :
3) To win you don't t need to kill all brown objects
Do you lose if purple objects die?
1) You lcse if all purple objects die or disappear
2) You don't lose if all purple objects die or disappear
3) don't know if you would lose if all purple objects die di disappear
Reasoning:
Answer (pick one from list): 1) You lose all purple objects die or disappea
Do you lcse lightred objects die?
1) You lose if all rig objects die or disappear
2) You don't lose if all lightred objects die or disappear
3) I don't know if you would lose if all lightred objects die or disappear
Reasoning:
Answer (pick from the list):
2) You don't lose if all lightred objects die sappear
### Task
Message from the player: [MESSAGE)
[QJESTION] [CHOICES]
Reasoning:
Answer (pick from the list): ICANDIDATE ANSWER] we compute the likelihood of all possible candi date anawer tokens
Prompt for the pure-LLM baseline
(besicttyleeettffmllyy.....
You trying solve video game you have player before.
### Information about the game:
31

---

Published as a conference paper at ICLR 2026
The game contains colored square objects of 8i it is defined by:
object type fcr each object color (eg missile, randon NPC, an inmovable object, pushable object, portal, etc.)
interactions for each possible pair of object color (eg no interaction, one dies, one is transformed into the other, one pushes steps back, etc.) These define what happens when
the other, objects of these two colors collide.
wining conditions: you may win by surviving long enough or when you kill all objects from specific list of colors (eg kill all green and white to win)
losing conditions: you may lose if you don't win fast enough, or if all objects of specific color die (eg you lose either all red die or all blue die)
Each object color correspends to different type of object, but al 1 objects of one color have the same type and behave the same way Each pair of object colors can have most type interaction Some objects (colorl) can transform certain objects (color2) into yet other objects (color3).
Possible ways to act in the game are to touch objects, push objects shoot. Yourself, the objects you push, shoot interact with other objects. Sometimes they will push then, kill them, transform them. Sometimes nothing happens.
#+# State description:
You will shown your recent actions and the corresponding history of game states, as well as object movements between these states A game state is composed of lists of object positions grouped by object color
is between x=0 (LEFT) and x-max_x (RIGHT), and y is between y=0 (TOP) and y-max_y (BOTTOM).
Fositiens are indicated
(x,
when x increases, objects move to the right, when x decreases, objects move to the left. When increases, objects move downwards, when decreases, objects move upwards. Moving UP decreases movement (0, -1).
Objects have size 1. They collide when delta_x or delta_y is below 1.
Look for wall positions, you cannot move through walls and shouldn' try to in the direction of wall.
### Analysis:
Observe the recent history of states and act ons in the envi ronment Opserve your scratchpad where you store your analysis of the game and strategy to solve it
If your previous action was unsuccessful, try to understand why. If you wen or lost in the previcus episode, try to understand why. Be careful about picking the right action to go where you want to go.
Try to predict the movements of important noving objects so you either catch or avoid them with your next action. Analyze your scratchpad, should you rephrase it to be more compact and up to date?
Analyze both update your understanding of the game, your strategy, and prepare for action.
Don't forget to plan for your next action. You might want to explore the world and touch, push or shoot objects, or you might want to plan tcwards the goal.
#+# Scratchpad update:
Decide whether to append information to the scratchpad or to replace the text in the scratchpad by new text. The scratchpad should contain everything you know about the environment: possible interactions, what kills you, what you think the goal of the game is, what makes you lose, etc.
32

---

Published as a conference paper at ICLR 2026
The scratchpad should contain your current plan to explore or solve
the game.
The scratchpad is the only menory that will be transmitted to the next decision making step, write down everything you need.
To append text, use <append> text to append goes here </apperd> To replace text, use <replace> new text goes here </replace> When replacing the scratchpad, all replaced text will be forgotten, be careful.
No need to write down observations in the scratchpad, you will be provided with the last observations every tine you are asked to generate new action.
Write your goal in the scratchpad so you can keep it in mind the next step.
The scratchpad has a maximum size of 400 words, you should sometimes use the <replace> connand to summarize its content and keep it concise. You start each new episode with episode. It will contain useful information about the game you' re playing (eg how you can die, win, or important game dynamics), but may also centain outdated information (eg object positions, previous strategy from the last episode).
scratchpad from the last
When reaching the end of the episode, rewrite the scratchpad to transmit information about what you learn so you can do better in the next episcde.
#+# Actiens:
Possible actions are: UP, DOWN, LEFT, RIGHT, SPACR_RAS BAR and NOOP.
UP decreases (0, -1), DOWN increases y (0, +1), LEFT decreases x (-1, 0), RIGHT increases x (+1, 0), SPACE BAR and NOOP do not move you.
SPACE_BAR let's you shoot spawn objects in some games (not all). Try SPACE_BAR sometimes to see if anything happens, and write to the scratchpad what happens.
Make sure your actions get you closer to your goal. It's easy to pick the wreng action and get further away, pay attention!
Every time you pick an action that is not NOOP, you will have to wait 4 cnvirenment steps before picking a new action. If you select NOOP you can select another actien at the next cavironment step.
### Answer fermat:
Analyze what you know about the game and the history of recent game states
Update the scratchpad
Select the action to execute next
Your response should be formatted in this way:
first the analysis <analysis>
write here your reflection and analysis of available informatior, what you understand about the game, your plan to update the scratchpad and your strategy for picking the next action. </analysis>
then update the scratchpad with either <append>
text to append to the scratchpad goes here
</append>
or <replace>
text to replace the scratchpad with here </replace>
finally pick an action among Ue DOWN, LEFI, RIGHT, SPACE_BA BAR, NOOP
33

---

Published as a conference paper at ICLR 2026
<act> action goes here </act>
#ff Example
[HF Recent history game history appears here
### Old scratchpad
Orange objects can kill you you should watch out. Shooting red objects seem to kill them. I'm not sure what the goal is yet. push blue objects but not purple ones. strategy: I should try to touch the yellow object to see what happens, maybe need to reach it?
H+f Example answer
<analysis>
It seems orange just killed the blue object? : should add this to the scratchpad. I'm still relatively far from the yellow object should keep moving in that direction.
My next move should be UP.
</analysis>
<append> Orange kill blue when they touch.
</append>
<ast>
</act>
34

---

Published as a conference paper at ICLR 2026
K BASELINE COMPARISONS
Figure 10 shows the performance of the deep RL baseline when run for up to 2,000 episodes.
should be performed for the LLM agent. This is not feasible for several reasons. First, the LLM agent extremely expensive execute: it produces a chain-of-thoughh plan every time step, and concatenating these traces quickly saturates the context window, preventing the model from carrying information across many episodes. Second, longer training is unlikely to change outcomes. Inspection of the model's reasoning traces shows persistent difficulties in (1) inferring rules from observations, (2) forming coherent multi-step plans, and (3) executing those plans reliably. These limitations mirror recent findings on LLM performance in long-horizon video-game -ame benchmarks (e.g.. Balrog Al (Paglieri et al., 2024)), where LLMs perform poorly even when given full rule descriptions. VGDL games are even more challenging, each new game introduces novel rules that must be inferred from scratch. For these reasons, Long-horizon rollouts are not expected to materially improve the LLM baseline.
For completeness, considered whether similar
lens
avoidGearge
beesAnddirds
Jaws
2-
30
1,00
2000
1600
2000
1009
2000
miccile command
bbquucAttack
portals
preconditions
2
1000
2000
1800
2000
3000
2000
1000
2000
RushBoudddess
rnlationa!
1000
2000
3:000
2070
Figure 10: Deep RL performance with extended training horizons (2,000 episodes per game, double DQN). Aliens is solved within the first lives, as it brings a dense straightforward signal (+1 for each alien shot). beesAndBirds, jaws and missileCommand can also be solved within the first 2,000 episodes, as they also demonstrate relatively straightforward reward signals. In all other games, DDQN fails to explore sufficiently, e.g., solving none of the levels in relational, preconditions, and avoidGeorge due to sparse rewards. These results illustrate the difficulty of learning to solve unknown VGDL games without explicit world modeling.
35


Reviews:
[{"review_id": "dE8jvviJfh", "reviewer": "Reviewer_GnG7", "summary": "This paper proposes a novel computational framework for social learning in AI, which rigorously integrates linguistic guidance and direct sensorimotor experience through joint probabilistic inference. The core innovation is leveraging a pre-trained Large Language Model (LLM) as a probabilistic model of human advice ($P(\\text{Language} | \\text{Model})$). By treating language as evidence to infer underlying structured, executable world models, the agent achieves superior sample efficiency and rapid, safe knowledge acquisition in complex tasks, significantly outperforming conventional learning methods.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty in Theoretical Framework: The paper introduces an innovative computational model that treats language not merely as an input feature, but as probabilistic evidence for Bayesian inference over potential world models. This is a theoretically rigorous approach to language grounding.\n\n2. High Sample Efficiency: By leveraging linguistic guidance as a strong prior, the method significantly reduces the hypothesis space, leading to substantially faster learning and higher sample efficiency compared to purely experiential methods.\n\n3. Emphasis on Structured Knowledge: The model focuses on inferring structured, executable world models (rules and mechanisms), which inherently promotes generalization across tasks and provides better interpretability than learning simple policies or value functions.\n\n4. Robust Integration: The probabilistic nature of the framework allows for the principled integration of two heterogeneous data sources (noisy language and imperfect experience), making the system robust to ambiguity and uncertainty in human advice.\n\n5. Applicability to Theory of Mind: By converting the LLM into $P(\\text{Language} | \\text{Model})$, the agent performs a form of reverse inference, effectively modeling the human advisor's underlying knowledge or \"mind,\" which is a powerful concept in social learning.", "weaknesses": "1. Scalability Challenge in Model Space: While the framework operates over \"structured world models,\" the computational feasibility of performing inference (or even representation) over the vast space of possible models ($\\text{Model}$) in real-world, high-dimensional, or continuous state environments remains a major concern. The paper needs to better address how efficient search or approximation is maintained.\n2. Experimental Complexity Limitation: Although the title claims to address \"Complex Tasks,\" the typical experimental setups (often discrete environments like Block-World or Grid-World, based on the context of similar research) may not fully demonstrate the method's effectiveness and scalability in truly unstructured, high-fidelity, or continuous environments, limiting the generalizability claim.\n3. Insufficient Comparison to Language Baselines: The comparison with existing methods may be incomplete. Specifically, the paper often omits a direct comparison with state-of-the-art deep reinforcement learning (DRL) techniques that use language as an auxiliary input feature or recent LLM-based planning methods (e.g., Chain-of-Thought planning), which could be a strong alternative baseline for incorporating guidance.\n4. Technical Validity of the LLM Probability Approximation: The paper's core mechanism, the Language Likelihood $P(\\text{Language} | \\text{Model})$, is approximated using the uncalibrated generation probability of a large proprietary model (LLAMA-3.1-70B) conditioned on a constructed prompt: $P(L|T) \\approx P_{LM}(L|\\text{prompt}(T))$. This approach raises concerns about its technical validity. LLM generation probabilities are generally not guaranteed to be accurately calibrated for strict Bayesian inference, which is central to the framework. The reliance on prompt engineering for a fundamental probabilistic term introduces a fragility that needs more rigorous theoretical justification or empirical validation of the approximation accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r19prOHsbh", "reviewer": "Reviewer_kQQx", "summary": "The paper presents a framework that models human social learning as joint probabilistic inference over structured world models, integrating both linguistic and sensorimotor information. The core method used is to transform a pretrained language model into a probabilistic model of human advice, enabling agents to generate and interpret guidance. Experiments across ten video games show that linguistic input enhances exploration efficiency, safety, and learning speed. Iterated learning experiments further demonstrate knowledge accumulation and transfer between humans and models, supporting more effective human–AI collaboration.", "questions": "The difficulty of different games can vary significantly depending on the complexity of their underlying theories and gameplay mechanics. It would be helpful to clarify whether the size of the candidate theory set varies across games and how much this variation affects performance. I am also interested in more details about the selection of M (candidate theory), particularly the reasoning behind choosing 20.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well-written, with clear motivation and an accessible presentation. The formulation of joint inference over experiential and linguistic guidance through Bayesian updating is both insightful and valuable.\n\n(2) The human evaluation experiments on building better learning partners are interesting and effectively demonstrate the role of linguistic guidance in improving video game performance. Moreover, the study’s extension to bidirectional communication between humans and models provides an elegant and symmetric perspective on social learning.", "weaknesses": "(1) Limited generalization: The current modeling approach is evaluated across different video games. However, in real-world agentic environments, the belief space can be far more complex, posing additional challenges for both modeling and linguistic guidance. As a result, the proposed method may have limited applicability beyond specific domains.\n\n(2) Model selection discussion: It would be valuable to include an analysis of different LLM choices. Stronger models may offer more effective guidance for both LLMs and human experts—or potentially the opposite. An additional study could examine whether the same phenomenon shown in Figure 5 persists when the guidance-providing model and the student model belong to different model families.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5LOLNpBQaJ", "reviewer": "Reviewer_46yR", "summary": "This paper explores how language and experience jointly support social learning in humans and computational agents. The work takes the view of human social learning as probabilistic inference over world models. Empirically, they study human participants and Q-learning agents performing ten games, examining how linguistic guidance shapes exploration and accelerates learning. A pretrained large language model (LLM) is used both to generate advice for other agents and to interpret linguistic input as evidence.\nThe results show that linguistic advice improves learning efficiency for both humans and models. Human participants learn equally well from model- and human-generated advice, whereas models benefit more from model-generated advice. The authors attribute this to differences in communication styles — human messages often include metacognitive reflections and analogies that LLMs find difficult to parse.\nThe paper also investigates iterated (generational) learning, where agents interact with the environment for limited “lifetimes” and pass advice to subsequent agents. Performance generally increases across generations, though with some regressions depending on the game.", "questions": "1. What exactly constitutes the “computational model” here — is it the Q-learning setup augmented with linguistic input, or the LLM itself viewed as a Bayesian speaker–listener model? \n2. Is there any quantitative or behavioral evidence that the LLM performs probabilistic inference rather than pattern-based generation? \n3. Why do some forms of advice promote generational improvement while others do not? Are there identifiable linguistic or structural features  that predict transfer success?\n4. Given that human messages contain analogies and metacognitive strategies that LLMs fail to parse, could fine-tuning on human teaching data improve model interpretability and alignment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is clear and engaging.\n* The combination of behavioral experiments with humans and artificial agents is very creative.\n* The results on cross-generational knowledge transfer are intriguing and connect well to theories of cumulative culture and iterated learning.\n* The work highlights a bidirectional exchange between humans and models — both learn from each other, which is a timely topic to consider.", "weaknesses": "* _Framing_ The title and framing promise a computational model of social learning, but the core of the paper is some empirical observations of social learning among and between agents and humans. The authors should consider revising the title or clarifying that the contribution is primarily empirical and conceptual, not a new formal model. \n* It is unclear what are the structured world models under the claimed “joint probabilistic inference over structured world models.”\n* The claim that the LLM performs Bayesian inference is not supported by direct evidence. The LLM generates or interprets advice, but it is not shown to approximate a Bayesian posterior or perform probabilistic updates.\n* The results for iterated (generational) learning vary substantially across games (e.g., improvement for Bess and Birds, but not Plaque Attack or Missile Command). It remains unclear why certain tasks show cumulative improvement while others regress, and what properties of the games modulate this effect.\n* Figure 6 (the Aliens game) may not be a good test of generational learning, as performance saturates quickly, leaving no room for measurable improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3s6UvHtpLr", "reviewer": "Reviewer_LmtC", "summary": "This paper develops a method to integrate natural language advice into planning as well as transform experience into natural language advice. The authors demonstrate that both human-generated and model-generated advice improve sample efficiency in learning a suite of tasks. \n\nOverall, I recommend acceptance because the method facilitates effective knowledge transfer between humans and models, providing a valuable avenue for future work on human-AI co-learning. However, because the method is restricted to the VGDL game, it could be difficult to scale to real-world tasks.", "questions": "**Questions for clarification:**\n\nYou mention \"prompting the LM with the received message L and instructing it to answer multiple-choice questions about specific VGDL rules.\" Wouldn't this give the model an advantage over the human or pure LM baseline because the multiple choice answers already narrow what the feasible rules are? Is it possible to evaluate the method without this step by sampling candidate rules directly from the LM?\n\n---\n\n**Minor comments:**\n\n- It would be helpful to add effect sizes to the reported results.\n- The authors state that \"We compare our approach to three baselines: 1) Oracle: a model that plans to solve the game using ground-truth game rules, (Line 467) \" but the oracle baseline is not shown in the main results.\n- In the paragraph \"Variability across games,\" the authors conclude that \"These patterns suggest that social learning strategies like selecting high-performing teachers\" (line 400), but I don't think this conclusion follows from the results. The results don't imply that higher-performing participants gave better advice.\n- The authors state that \"We show more examples of human- and model-generated messages in Appendix Section I and on our website (Line 385).\" The website only has some game demos. I couldn't find the examples of human and model-generated messages.\n\n---\n\n**Suggestions for improvement (not factored into score):**\n\nThe authors state \"Human-generated messages often include game abstractions, high-level strategies, and planning heuristics that our model currently cannot leverage (Line 467).\" While the authors do provide examples of human-generated and model-generated messages in the appendix, it would be helpful to more systematically categorize the differences between human and model-generated messages. This would be beneficial for further work on human-AI collaboration.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results convincingly show that advice, whether human-generated or model-generated, improves learning performance. The evaluation is conducted across multiple tasks, demonstrating the robustness of the approach. The method is well-motivated and the results are intuitive (advice improves learning, humans can learn better from human advice than model advice). The generational learning experiments provide additional evidence that the model-generated advice is effective, strengthening the paper's claims.", "weaknesses": "As the authors acknowledge, the method is currently limited to the VGDL game space. Scalability to more complex, diverse, or realistic environments could be difficult.\n\nThe prompts for integrating natural language advice could potentially have information leak (see questions below), which may bias the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I5P74kktU1", "reviewer": "Reviewer_YLjG", "summary": "The authors propose a Bayesian model by which agents use both linguistic guidance and direct experience to develop a world model of their environment. Their proposed agent updates their distribution of world models based on compatibility with game observations and linguistic guidance from a model, weighted by probability assigned by an LLM-based speaker model. Experiments on the VGDL suite of games shows that their agents learn significantly more quickly than Deep RL and LLM agent-based baselines, and that linguistic guidance greatly accelerates learning when added to direct experience. They also find that model-written guidance can also improve learning for human participants, although both models and humans learn better from their own guidance.", "questions": "- It’s not clear to me from the citation given in 3.4 why language-conditioned RL methods require thousands of episodes to learn language-policy mappings, especially since the citation is from 2019 and more recent methods use LLMs to become more sample efficient (e.g. \"Language instructed reinforcement learning for human-ai coordination”, Hengyuan Hu et al, ICML 2023). Since this seems to be more relevant to the proposed method than the Deep RL or LLM Agent baselines, it would be useful to spend more time justifying the decision to not benchmark against them.\n- I’d recommend including median hourly pay for the crowdworkers, either in Appendix G or Section 4.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Thorough theoretical framing of the important problem of learning jointly from language and experience, from which the proposed model clearly follows.\n- Results across a variety of game settings show that the model learns much more efficiently than Deep RL and LLM agent-based approaches, suggesting that the model does, in fact, approximate human learning mechanisms better than existing methods.\n- Rigorous study of the impact of human vs model guidance, as well as differences in communication styles, motivates some pretty interesting future directions related to collaboration in human+AI communities.", "weaknesses": "- The baselines used are both highly limited within the time horizon given, which limits interpretation of results. It would be useful to run the pure LLM and DQN baselines over a longer horizon until some learning happens, to get a better sense of how much more efficient the proposed model is. This could also help us better understand the relative gains from incorporating linguistic guidance in the model vs pure LLM settings, which would strengthen the argument that the new agent better models human learning improvements from linguistic guidance.\n- As the authors note, they evaluate agents over a very limited set of game environments. Fully open-ended environments would be out of scope of the paper, but it’s not clear to me that the games used are significantly more complex than the environments used in past work (e.g. the 2024 Zhi-Xuan work cited) that are dismissed as overly simplistic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)