LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Authors: Anian Ruoss
Fabio Pardo
Harris Chan
Bonnie Li
Volodymyr Mnih
Tim Genewein
Venue: ICML 2025 poster (2025)
Keywords: in-context learning
imitation learning
multimodal
long context
reasoning
interactive decision-making
Primary area: general_machine_learning->evaluation

Abstract: In this paper, we present a benchmark to pressure-test today’s frontier models’ multimodal decision-making capabilities in the very long-context regime (up to one million tokens) and investigate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1 as policies across a battery of simple interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a simulated cheetah. We study increasing amounts of expert demonstrations in the context — from no demonstrations to 512 full episodes. Across our tasks, models rarely manage to fully reach expert performance, and often, presenting more demonstrations has little effect. Some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observations as text or images and the impact of chain-of-thought prompting. To help quantify the impact of other approaches and future innovations, we open source our benchmark that covers the zero-, few-, and many-shot regimes in a unified evaluation.

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Anian Ruoss'
Fabio Pardo Harris Chan'
Bonnie Li" Volodymyr Mnih Tim Genewein
Abstract
the ability to learn new patterns and imitate algorithms in context (Reid et al., 2024; Mirchandani et al., 2023). Both types of capabilities demonstrate that LMs can manipulate learned knowledge and respond to new information in flexi- ble and non-trivial ways. These capabilities, both necessary for reasoning and decision-maie making, have led the recent surge of using L.Ms as agents by sampling an action from the model (Mirchandani et al., 2023; Palo & Johns, 2024).
In this paper, we present a benchmark to pressure- test today's 's frontier models' multimodal decision- making capabilities in the very long-context regime (up to one million tokens) and investi- gate whether these models can learn from large numbers of expert demonstrations in their context. We evaluate the performance of Claude 3.5 Son- net, Gemini 1.5 Flash, Gemini 1.5 Pro, Gem- ini 2.0 Flash Experimental, GPT-4o, ol ol-mini, ol- preview, and ol as policies across a battery of simple interactive decision-making tasks: play- ing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and controlling a sim- ulated cheetah. We study increasing amounts of expert demonstrations in the context- - from no demonstrations to 512 full episodes. Across our tasks, models rarely manage to fully reach expert performance, and often, presenting more demon- strations has little effect. Some models steadily improve with more demonstrations on a few tasks. We investigate the effect of encoding observa- tions as text or images and the impact of chain-of- thought prompting. To help quantify the impact of other approaches and future innovations, we open source our benchmark that covers the zero-, few-, and many-shot regimes in unified evaluation.
While LMs have been shown to be able to perform non-trivial reasoning and decision-making in some do- mains (Huang et al., 2022; Yao et al., 2023; Romera-Paredes et al., 2024), there are also many negative results where LMs fail to perform pocision-making tasks that are very simple for humans, even when LMs arguably possess great factual knowledge of the task. For example, LMs struggle to play le- gal moves, let alone beat amateur humans, in chess (Carlini, 2023). At the same time, state-of-the-an LMs have detailed expert knowledge of chess when queried in natural language. But this declarative knowledge fails to translate into effec- tive deciisionmmaking i.c., "know-how" (Ryle, 1949). This "knowing-doing gap" is also observed in the recently re- leased BALROG benchmark (Paglieri et al., 2025), which evaluates zero-shot capabilities (i.e., without expert demon- strations) of state-of-the-arr LMs on interactive decision- making tasks in the long time horizon setting: 5 game envi- ronments from Baby AI (Chevalier-Boisee Nethack (Küttler et al., 2020). Overall, Paglieri al. (2025) find that "models struggle significantly with more challeng- ing tasks" and highlight few- and many-shot evaluation as an important potential solution. Our benchmark (developed concurrentyy) addresses exactly this gap by covering the full range from zero-shot to many-shot evaluation.
et al., 2019) to
1. Introduction
The simple recipe of minimizing next-token prediction er- ror at scale has led to large multimodal foundation models (LMs) with remarkably general capabilities (OpenAI, 2023; Anil et al., 2023; Anthropic, 2024a). Importantly, these capabilities come two flavors: (i) the ability to produce outputs of high quality from a short and often underspecified prompt (e.g., writing essay about novel topic), and (ii)
The general question arising from these results is whether LMs in principle have the capabilities to solve interac- tive decision-making tasks but misunderstand the "out-of- distribution" specification given by the zero-shot prompt few examples, or whether the problem is more deeply rooted. Accordingly. our paper focuses on whether conditioning on a large number of expert demonstrations (state-action tra- jectories) helps unlock the decision- makinn capability of pretrained LMs. In doing so. we test the multimodal in- context learning capabilities of modern LMs at their limits,
'Google DeepMind. Correspondence to: Anian Ruoss <ani- @nr@ woogllooooooooooooo com²>, Tim Genewein <timgen @ @ww.gggoooooooooooooooooo com>>
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
LMAct Benchmark
X
Environment
Atari- Phoerix
Chess Stockfiih 16
Crosswords
DM Control Cheetan Run
Gridwerld navigation
Tie Tac Toe
Expert
Expert poley
Uradl salver
Experf policy
Shortest poth
Miniaxx search
Coordinatos
RGB
representations
ASCI
ASCII Coordinates
ASCII
FEN
PGN
LM Agent
Observation
Preamble
Demo opisodes to
Seperator
Test episode
LM Call
Environment
A
Action
govental
D_ <ac_0_>>
Ulseervcc lone
<AC___22
denonetrrat.ons1
Xbservation:
Actionn
Figure 1. LMAct overview. Our multimodal benchmark consists of six decision-making tasks that come with an expert policy and polentially multiple state representations. For evaluation, LM performance measured on lest episodes with unseen initial states. LMs are conditioned on a generic decision makinn preamble (fixed across all tasks), followed by 0 to N demonstration episodes, and separator that indicates the start the current episode (N can up to 512, with up to 100 steps per episode, both depending on the task. The maximum context length is LM tokens). In each step of the test episode an action is generated by the LM's predicted continuation of the context. The resulting environment interaction produces the next observation that is added to the growing context of state action pairs.
with contexts that are up to a million tokens long (with thousands of output tokens). Our main research question is:
mostly independent of the number of demonstrations. some cases we observe strong in-context learning.
We run control experiment where LLM agents need to replay the single demonstration episode in the context, where all models except for ol ol-mini perform well.
Can state-of-the-ane LMs learn to act in dynamic environments by generalizing from large numbers of multimodal in-context expert demonstrations?
We open-source our in-context imitation learning benchmark that covers the zero-, few-, and many- shot regime in unified manner, including all ex- pert demonstrations and evaluation code at https: / //github. com/ google gteepmind/1mmacee
Main Contributions See Figure/ for tasks and methodology. We make the following key contributions:
We conduct a comprehensive empirical evaluation of the multimodal in-context imitation learning capabili- ties of state-of-the--rt LMs (Claude 3.5 Sonnet, Gem- ini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Exper- imental, GPT-4o, ol ol-mini, ol-preview, and o1) on battery of interactive, potentially long-horizon, tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and DM Control task.
2. Methods
We now briefly describe the models we evaluate (Sec- tion 2.1), our benchmark environments (Section 2.2), how we construct the prompt (Section 2.3), and our evaluation protocol (Section 2.4). Full details are given in Appendix
We show that - even when optimizing the prompt (number of demonstrations, chain-of-thought prompt- ing, etc.) for each model and task - frontier LMs fail to reach expert performance on Atari, chess, and DM Control. Some models approach expert performance crosswords, grid world, and tic-tac-toe. All models beat the random action baselines except on Atari.
2.1. Models
We evaluate the current (closed-weights) frontier LMs: Claude 3.5 Somnet (the 2024-10-22 version) (Anthropic, 2024b;c), Gemini 1.5 Flash and Gemini 1.5 Pro (the 002 versions) (Reid et al., 2024), Gemini 2.0 Flash Experimen- tal (Google DeepMind, 2024b), GPT-40 (the 2024-08-06 version) (OpenAI, 2024a; OpenAI et al., 2024a), ol ol-min and ol ol-preview (OpenAI, 2024c), and ol (OpenAl et al., 2024b). Except for ol-m and ol or-preveww, all models process multimodal prompts, though their exact specifica-
We vary the number of expert demonstration episodes in the context from up to 512 (the limit depends on the model and the task) and find that performance is
2

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
tions differ (see Appendix B.1). We use temperature 0 for all models (except for ol-mini, o1 ol-preview, and o1, which have fixed temperature of 1 (OpenAI, 2024d)). We set the maximum (ouiput) sample length to 2048 tokens for all models (except for o1-mini, ol-preview, and ol), which is more than sufficient to achieve strong performance (see our ablation in Fig. A7). In contrast, the performance of ol- mini, ol-preview, and ol crucially depends on the number of "reasoning tokens" (see Fig. A7), so we use maximum (output) sample length of 8192 tokens as good compro- mise between cost and performance (2048 tokens would lead to severe performance degradation on our tasks- see our ablation in Appendix C.I). We post-process the model outputs by removing all the leading/trailing white spaces and only consider the text after the keyword "Action:", i.c., we discard all Schain-oftthouglee reasoning traces.
historical information as a string, (iii) the Portable Game Notation (PGN), a plain text format for recording chess games via the move history given in algebraic chess no- tation, and (iv) an RGB image of the board. We always represent actions via the algehraic chess notation. To ensure variability in the demonstration and evaluation episodes, we use the openings from the Encyclopedia of Chess Open- ings (Matanović, 1978). which we randomly sample without replacement. We play all games for at most 100 steps (termi- nating early in case of win/draw/loss; the average number of steps per game is 38) and assign reward of to win, O to a draw, -1 to a loss, and 0 to all other states.
Crossword We create a large collection of 7 x 7 cross- words using the genxword crossword generator (Whitlock, 201 and a list of 55 189 clues with the lowest difficully rating collected by Matthew Ginsberg (individual clues may appear in multiple crosswords with low probability). Each episode is a distinct crossword represented as an ASCII crossword grid followed by two lists of clues, one for the "Across" words and one for the "Down" words (see Fig. A4), A valid action consists of either "A" (for across) or "D" (for down), followed by the word's index and the word itself. We assign a reward of to a correct word, 0 to an incorrect word with correct length or to a correct word that has been already been placed, and we terminate the episode with a reward of -1 for incorrect word. We evaluate 25 steps (which is sufficient to place the approx. 10 words per cross- word on average) and terminate early if all words are filled. We generate the expert demonstrations by simply outputting the solution for each clue one by one in random order.
2.2. Environments
We consider a battery of well-known interactive decision- making environments: the Phoenix game from Atari 2600 (Bellemare et al., 2013), chess, crosswords, the chee- tah run task from the DM Control suite (Tassa et al., 2018) grid world navigation, and tic-tac-toe. We briefly describe each environment below (full details in Appendix B.2). Since sampling is deterministic for most models (i.e., the temperature is O, see Section 2.1), we introduce variability in the demonstration and evaluation episodes by varying the initial conditions (e.g., via the environment seed for DM Control or by using different openings for chess; see below).
Atari Phoenix We chose Phoenix as a representative Atari task that has somewhat dense rewards, which is im- portant since we can only evaluate 400 frames (with action repeat 4, i.e., 100 steps), or roughly 6 seconds of play due to context size limitations. Phoenix also forms part of the set of 5 games that highly predictive of the performance on the full Atari suite (Aitchison et al., 2023). We use the Arcade Learning Environment (Bellemare et al., 2013) ver- sion, and for expert demonstrations we use the Gato training data (Reed et al., 2022). We use the original (i.c., not down- sampled or grayscale) images as observations (see Fig. A1).
DM Control - Cheetah Run We use the Cheetah Run task from the DM Control Suite (Tassa et al., 2018). We represent observations as string in the style of a Python dictionary of position and velocity vectors with individual values between 1 and (see Fig. A5). Each episode begins in a new, randomly initialized state. We only evaluate the first 100 steps of an episode (due to context length limits). As for Atari, we use the Gato training data (Reed et al., 2022) to create expert demonstrations (details in Appendix B.2).
Grid World We consider a 12 x 12 grid world with walls, effectively yielding 10 x 10 grid, with a single player and a single target (no obstacles). Actions are up, down, left, and right, and since the grid is fully observable, memorization is not required. We evaluate three different state representa- tions (see Fig. A6): (i) an ASCII encoding of the 2D grid, (ii) the player and target coordinate tuples provided as plain text, and (iii) the RGB image of the grid. The reward is 1 if the player reaches the target and 0 otherwise. Episodes run for a maximum of 25 steps (reaching any target from any initial position in 10 x 10 grid requires at most 18 steps). We randomly initialize the player/target locations for every
Chess We evaluate models against the weakest-possible version of Stockfish 16 (Romstad et al., 2008), i.e., level O (which corresponds to an Elo of approx. 1320). We further weaken this version by evaluating only 1 node. We generate expert demonstrations with the strongest (default) version of Stockfish, i.e., level 20 with time limit of 50ms per move as the agent (note that the opponent, i.e., the "environment", remains fixed as the weakest version of Stockfish), We eval- uate four different state representations (see Fig. A3): (i) a 2D ASCII encoding of the board, (ii) the Edwards Notation (FEN), which encodes the board and very limited
3

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
episode. The expert demonstrations correspond one of the shortest paths between the player and the target.
context lengths (and different text/image tokenizers), the maximum number of demonstration episodes that fit into the context depends on the model, the task, and the state rep- resentation format. For example, for a given model, we may only by able to use 16 demonstration episodes with RGB observations but up to 256 ASCII demonstration episodes.
Tic-Tac- We play tic-tac-toe against a random policy that picks an empty spot uniformly at random. For demon- stration episodes the expert uses minimax search (i.e., it plays an optimal action), while the environment remains fixed as a random policy. We evaluate two state representa- tions (see Fig. A2): (i) an ASCII encoding of the 2D grid, and (ii) the RGB image of the grid. To ensure variability, each game starts from an opening state (similar to chess openings), which we draw uniformly across all initial game states (as a result, even the optimal minimax strategy can only win 85% of the games and draws the rest). We assign a reward of to a win, 0 to a draw, -1 to a loss, and 0 else.
We always evaluate 100 episodes with different initial condi- tions (each episode is evaluated individually) and report the average score. The maximum episode duration is 100 steps and the score per episode is the cumulative reward over all steps (we never show reward information to the mod- els). For each evaluation episode we uniformly subsample (without replacement) the demonstration episodes (for the frozen part of the prompt) from a precomputed pool of up to 1000 distinct demonstrations, We ensure that all evaluation episodes have initial states that differ from the demonstra- tion episodes (except for the replay control experiments in Appendix C.2). We only perform very minimal postprocess- ing (see Section 2.1) of the LM generations to obain the action and (deliberately) reject semantically correct actions that are wrongly formatted, as matching the action format an important aspect of our imitation learning benchmark.
2.3. Prompt
Our prompt consists of two main parts: (i) the expert demon- stration episodes (Listing 1), and (ii the trajectory of the evaluation episode (including the episode's 's previous actions and environment states; see Listing 2). We use generic decision-maken making zero-shot preamble at the beginning of (i), and a short separator prompt at the beginning of (ii) (see Fig. and Listings 1 and 2). The expert demonstrations are fixed across an evaluation episode, but resampled across episodes, while the current trajectory starts with a single ini- tial state and grows as more state-action pairs are observed (up to maximum of 100 steps). We do not include an environment task description (e.g., the rules of tic-tac-toe) in the preamble. We may, however, show the available le- gal actions, which depend on the environment, in each step at the end of (ii) (see Listing 2), depending on whether it is beneficial per model and task (see our ablations in Ap- pendix C.4). Similarly, we may use chain-of-thought (Wei et al., 2022) style prompt at the very end of (ii) asking the action
3. Results
We now present our comprehensive empirical evaluation of the (closed-source) frontier models on our benchmark for interactive decision-maken with long multimodal context. We investigate how performance changes when presenting more demonstration episodes in the context (see Section 2.2 for task overview and Appendix B.2 for details and illus- trations). For each model and task, we first ablate whether to use chain-of chain-of-thought prompting and whether to show the legal actions in the prompt (results in Appendix C.4). We use one demonstration episode (i.e., still many individual demonstration steps) for these ablations, as a compromise between lower computational demands and being represen- tative of the in-context learning setting. Due to the very low monthly rate limits of the Anthropic API (Anthropic, 2024d), we do not sweep over the number of demonstrations for Claude 3.5 Sonnet and only run the ablation. We also do not evaluate it on Atari since it can only process 100 images at once (a single demonstration episode hits that limit).
model to provide a reasoning before proposing (see Listing 2), again, depending on whether it increases performance in the ablations (Appendix C.4). We make both of these decisions per model and task, i.c., the same model may use shaii--ff-hhuuhhe for one task but not another.
2.4. Evaluation Protocol
In every evaluation step, we condition the model on the current context, upon which it generates an action that we feed to the environment. We then concatenate the result- ing next environment state (prepended with the action that produced it) to the growing context. We ablated whether to show the model's previous actions in the evaluation trajec- tory (see Appendix 1.4.1) and found that it mostly improves performance, so we always include them. If a model fails to generate a legal action, we uniformly sample one of the le- gal actions (we visualize the percentage of illegal actions in Appendix C.3). Since the models have different maximum
3.1. Best Scores Per Model/Task
Fig. 2 shows the highest overall score per task and model across all settings (number of demonstrations, observation format, showing legal actions, -f-thughht prompting). Different models thus may use different settings on the same task, and the same model may use different settings across tasks. In Section 3.2, we will keep the observation format and number of demonstrations constant across all models per data point. As stated above, results for Claude 3.5 Son-
4

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Atarl Phoenix
Chess Stockfish (Level node)
Crosswords (7x7)
400
Score
Score
Score
200
GPT-40
Genini 2.0 Flash Exp.
Gemiri 1.5 Flash
Random
Gato Data
Random
o1-mini
Cloude 3.5 Sonnet
Gemini 2.0 Flash Exp. al c1-previex
01 GPT-00 Stockftsh
Random
Gemini 1.5 Hlss
Gemini 2.0 Flash Exp.
o1-mini
Gomini 1.5 Pro
Cloude 3.5 Sonnet
GPT40
o1-greview
o1
Drade
Gemini 15
Gemini 1.
Gemini 1.5
Contral: Cheetah
World (12x12)
Tic-Tac-Toe vs. Random
1.0
0.75
30
Score
20
Scooe
Score
0.50
0.25
0.0
0.00
Random
ol-mml
p1-preulew
Claune 3.5 Sonnet
Genini 1.5 flash
GPT-40
Gemiri 2.0 Flash Exp. Gemini 1.5 Pro
Esto Data
Random
Gemini 1.5 Pro
ol-preview
Gemiri 2.0 Flash Exp. Exp.
oi-mini
Claude 3.5 Sonnet
GPT-40
Random
Gemini 3.5 Flash
Gemini 1.5 Pro
Gemiri 2.0 Flash Exp. Exp. Claude 3.5 Sannet
GPT-4a
oi-mini
ol-preview
Minimax
Shortes
Genini 1.5
Figure 2. Best scores per model and task across all observation formats, numbers of demonstration episodes, and ablations (chain-of- thought, showing legal actions). Accordingly, different bars in a panel may he based on different settings. The expert policy (which produced the demonstrations) is an upper baseline. The lower baseline randomly selects legal action each step. Claude 3.5 Sonnet, ol-mini, and ol -preview cannot be evaluated on Atari Phoenix because they cannot process (enough, for Claude 3.5 Sonnet) images.
net are from the ablation (i.e., 1 demonstration episode). Fig. shows that models often struggle to match expert scores - even in their best setting. Exceptions are grid worlds, which most models largely solve, tic-tac-toe, where the ol models achieve near-optimal score (against ran- dom opponent), and crosswords, which ol-preview and ol almost solve. On the other hand, all models always outper- form the random action baseline except for Atari - Phoenix. LMs tend to repeat actions (see Fig. A14 for a detailed anal- ysis), and Phoenix holding down the firing button without releasing results single shot (no auto-fire). In contrast, random actions produce fair amount of (successful) shots.
Atari Phoenix (RGB)
Average Score
Gemini 1.5 Flash
300
Gemini 3.5 Pro
Exp. GPT-4
01
Random Baseline
100
Expert (Gato Data)
of Expert Demonstration Episodes
Figure 3. In-context imitation learning on Atari Phoenix (RGB observations). Almost all models benefit (mildly) from one demon- stration episode, not from more (GPT-4o and cannot multiple demonstration episodes in the context). While no model outperforms the random baseline, Gemini 1.5 Flash performs best.
3.2. In-Context Imitation Learning
We now investigate the in-context imitation learning capa- bilities of today's frontier models, by varying the number of demonstration episodes from zero-, to few-, and many-shot evaluations. The largest number of demonstrations depends on the models' context sizes, the task, and the tasks' obser- vation formats (images require more tokens than text). We omit Claude 3.5 Sonnet due to its low monthly token limit.
tions (only possible for the Gemini models) do not improve the performance further. All models struggle to outperform the random action baseline because of their tendency to repeat actions (and thus fire very little, since Phoenix has no nuto-firee.. Table A1 contains the ablation results. Models rarely output illegal actions (see Fig. A15), i.e., they reliably (re)produce the correct action format. Atari - Phoenix is arguably the hardest task in our benchmark, since it only comes with image observations (no text representation), and has very large demands w.r.t. context size (high frame rate,
Atari Phoenix Fig. 3 shows that on Atari Phoenix all models except ol improve slightly by having one demonstra- tion episode, as opposed none. However, more demonstra-
5

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
somewhat high resolution images, etc.). Despite LMs' mas- sive scale, playing Atari games well (at least by naively feeding raw images to the context) seems currently beyond reach, both in terms of capabilities, but also w.r.t. context size and compute demands (cf. Waytowich et al. (2024)).
Crosswords (7x7) (ASCI))
10
Gerrini
Gerrini 1.5 Pro
Gerrini 2.0 Flash Exp.
Average
GPT-40
6l-mini
ol ol-preview
0l
Chess Stackrish [Levll 0: 1 node) (ASCII)
Random Baseline
1.0
Expert (Oracle)
Average Score
0.5
Gemini
Gemini Flash
2
Export Demonstration Episodes
2
2
24
24
2
2a
Gemini 2,0 Flash Exp.
0.0
GPT-40
Figure 5. In-context imitation leaming (using clues with the simplest rating) with ASCII observations. The performance of most models is largely unaffected by the number of expert demonstration episodes. ol -repreveww and ol solve most crosswords, while other models struggle varying degrees.
7 crossword puzzles
-0.5
e1
Random Baseline
Expert (Stackfiah)
-1.0
25 24
27
2a
24
Expert Demonstration Episades
(a) ASCII observations
shows LMs' performance cross- words with simple clues (as rated by Matthew Ginsberg), While the models achieve very different overall scores, their individual performance is largely independent of the number of demonstrations (except for ol-mini and Gem- ini 2.0 Flash Experimental, which degrade with more demonstrations). Overall, ol-preview and ol perform best, almost completely solving all the (simple) crosswords. Fig. A17 shows that the number of illegal actions (i.e., where models either suggest a word of incorrect length or fail to respect the "Across" vs. "Down" format) is quite high for all models and roughly inversely proportional to their puzzle- solving competence. We use chain-of chain-of-thought prompting for some but not all models (see our ablation results in Ta- ble A5), but we never show the legal actions (which would be unrcasonably long list of words with correct length).
Crosswords Fig. 5
Chess vs. Stackfish (Level D: node) (FEN)
1.0
Gerini 1.5 Flash
Aversge Score
0.5
Gemini
Gemini 2,0 Flash Exp.
0.0
GPT44
61-miri
01-preview
-0.5
61
Random Baseline
Expert (Stockfich)
-1.0
24 25
24
2
21
20
Expert Demonstration Episades
(b) FEN observations
Chess vs. Stockfish (Level node) (PGN)
1.0
Gemini 1.5 Flas
Average Score
0.5
Gemini Flash Exp.
0.0
GPT-6p
61-mini
O1-previsw
-0.5
61
Random Baseliee
Chess Fig. 4 shows LMs' chess-playing performance against the weakest version of Stockfish 16 (Romstad et al., 2008) (i.e., level 0, = 1300 Elo), further restricted to only evaluating a single node. We investigate four observation formats: ASCII, FEN, PGN, and RGB (ol-mini and ol- preview cannot process images). Showing more demonstra- tions has little effect on performance, and models rarely manage to beat (score 1) or draw (score 0) against this very weak opponent. The results show that playing chess (with- out any scaffolding or fine-tuning) is still out of reach for current LMs. Fig. A16 reveals that the models often out- put illegal actions (which also does not improve with more demonstrations) - even though the legal actions are pro- vided in the prompt (cf. the ablations in Tables A2 to A4).
Expert (Stockfich)
-1.0
2
24
2
Expert Demonstration Episades
(c) PGN observations
Chess vs. Stockfish (Level 1 node) (RGB)
1.0
Average Score
0.5
Gemini 1.5 Flash
Gemini
Gemini Flash Exo.
0.0
CPT-4o
ol
-0.5
Expert (Stockfich)
-100
2'
2'
2
2
Expert Demonstration Episades
(d) RGB ohservations
Tic-Tac-Toe Fig. 6 shows the results for playing tic-tac- toe against a random-action opponent. Since the demon- stration and evaluation episodes start from different initial states, episodes begin with partially filled boards, which cannot always be won (i.c., optimal score < 1). The ol models reach expert performance with ASCII observations
Figure 4. In-context imitation learning on chess against the weak- est variant of Stockfish (level 0, 1300 Flo), further restricted to one node. The models almost always lose (i.c., score 1) and do not benefit from more demonstrations. The PGN observations enable the best results, in particular for GPT-4o, which performs best but still loses majority of games against this weak opponent.
6

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Tic-Tac- Toe vs. Random (ASCI))
Grid World (12x12) (ASCII)
0.8
Average Score
0.6
Gemini 1.5 Flash
Average Score
0.8
Gemini 1.5 Flash
Gemini
Gemini 2.0 Flash Exp.
Gemini 2.0 Flash Exp.
0.4
CPT-4o
GPT-4o
01-mini
01-mini
0.2
02- provlow
0.4
01-preview
0.0
Random Baseline
0.2
Random Baseline
Expert (Minimax)
Expert [Shortest Path)
0
21
27
2
27
27
20
2
2
2
27
4P of Expert Demonstration Episodes
# of Expert Demanstration Episades
(a) ASCII observations
(a) ASCII observations
Tic-Tac-oe vs. Random (RGB)
Grid World (12x12) (Coordinates)
0.8
Average Scare
0.6
Gemini 1.5 Flash
Awerage Score
0.8
Gemini 1.5 flash
Gemini 1.5
Gemini 1.5 Pro Gemini 2.0 Flash Exp. CPT-4o
Gemini 2.0 flash Exp.
0.4
GPT-4o
a1-trini
0.2
Random Baseline
0.4
01: O1-pprvieww
Expert (Minimax)
Random Baseline
0.0
Expert (Shortest Fath)
21 22
2
27
20
2
2'
2¹
2
2'
2'
2
# of Expert Demonstration Episodes
# of Expert Demonstration Episodes
(b) RGB observations
(b) Coordinate observations
Grid World (12x12) (RGB)
Figure 6. In-context imitation learning on tic-tac-toe against ran- dom action adversary. Apart from the ol models on ASCII obser- vations, all models struggle to play better than random baseline. ol ol-mini and o1-preview improve with more demonstrations and both reach expert performance at 256 demonstration episodes.
Aversge Score
0.8
Gemini 1.5 Flash
Gemini 1.5 Pro
0.6
Exp. GPT-4o
Random Baseline
Expert [Shortest Patto
0.2
and show signs of in-context learning. The other models struggle to outperform the random action baseline (as does ol with RGB observations). Fig. A20 shows that the models (apart from Gemini 1.5 Pro on RGB) generate few illegal actions, implying that the models' weak performance is caused by outputting suboptimal rather than illegal actions.
20
22
20
# of Expert Demanstration Episades
(c) RGB observations
Figure 7. In-context imitation learning for navigating to a target in 12 grid world (see Fig. A6) using the commands: up. down, left, right. On ASCII, most models deteriorate with more demon- strations. For coordinate observations (player and target tuple) and RGB images, Gemini improves with more demonstrations (except Gemini 2.0 Flash Experimental on RGB), indicating in-context learning across very long context. GPT-4o (and ol on RGB) shows no such trend but already achieves high zero- and few-shot performance (al is near-perfect ASCIT and coordinates).
Grid World Fig. 7 shows how LMs perform on the task of navigating a simple grid world. The Gemini models steadily improve with more demonstrations for coordinate and RGB image (1.5 models only) observations, demon- strating strong in-context learning with very long contexts. While GPT-4o, ol-mini and ol -preview do not benefit from more demonstrations, ol performs very well in almost all settings. On ASCII observations, most models (except ol and Gemini 1.5 Flash) deteriorate with more demonstrations. Fig. A19 shows the illegal actions for each model, revealing that ol-preview's performance drops on ASCII and coor- dinate observations strongly correlate with the number of illegal actions. All other models rarely generate illegal ac- tions after one demonstration episode. Overall, grid world is the easiest task in our benchmark where all models perform quite well (some almost optimally) with the right combina- tion of state representation and number of demonstrations.
DM Control - Cheetah Run Fig. 8 shows LMs' ability to control a simulated (half) cheetah from DM Control (Tassa et al., 2018). We encode observations and actions as strings (see Fig. A5). For all models, except ol-mini, a single demonstration episode helpful but more than two demon- strations degrade performance. The ol models struggle to significantly outperform the random action baseline. Gem- ini 1.5 Pro and Gemini 2.0 Flash Experimental achieve the highest score, reaching roughly half the expert score. Ex- cept for ol ol-mini, all models mostly generate legal actions (see Fig. A18) - even without any demonstration episodes

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
DM Control: Cheelah- - Run (Coordinates)
pretrained LMs have fairly general ability to recognize and imitate algorithmic patterns in their context (Mirchandani et al., 2023), which is in line with the memory-based meta- learning view on (universal) in-context prediction (Ortega et al., 2019; Grau-Moya et al., 2024). If universal enough, models would, with enough observations, recognize and correctly continue environment-agent interaction patterns, See Appendix A for a discussion of additional related work.
20
Flash
Average Score
Gemini 1.5 Pro
Gemini 2.0 Flash Exp.
20
GPT-4p
o2-minl
o1-preview
10
Random Baselline
Expert (Gato Data!
2a 24
22
2
2
Expert Demonstration Episades
While find some cases of steady in-context learning, in the majority of cases LMs' performance is largely indepen- dent of the number of demonstrations. It is unclear whether in-context imitation learning is not well suited to commu- nicate desired agentic behavior (i.e., models do not "under- stand" the task purely from demonstrations), or whether models have difficulty to effectively use a dense long con- text. To provide additional insight we perform "replay" control experiment, where a single episode is shown in con- text, and the same exact episode is evaluated, such that models only need to "copy" actions from the demonstration episode. We find that models (except ol-mini) perform well in this control experiment on all tasks (see Appendix C.2).
Figure& 8. In-context imitation learning on the cheetah run task from the DM Control suite using position/vclocity vector observations encoded text. Gemini 1.5 Pro and Gemini 2.0 Flash Experi- mental perform best with a single demonstration episode, GPT-40 and Gemini 1.5 Flash with two demonstration episodes, with more demonstrations degrading performance. Gemini 1.5 Pro achieves the highest performance, roughly half the expert score.
(the action format can be inferred from the past actions in the evaluation trajectory since we randomly sample legal action if the model generates an illegal one). Therefore, the poor performance in the zero-shot regime (i.e.. 0 demonstra- tion episodes) cannot be explained by the models' potential lack of knowledge of the action format. Instead, models (with the exception of ol-mini) manage to learn non-trivial policy from one two demonstration episodes (but fail to leverage more episodes to further improve performance).
Limitations We perform an evaluation via closed-source APIs and thus have little control over how the data is pro- cessed and fed to the underlying models. Since models behind the APIs can be updated at any time, it is possible that our results may not be quantitatively reproducible soon after publishing this manuscript. Despite our best efforts in evaluating different prompt formats, we cannot rule out that even small changes to the prompt could lead to better results. Accordingly, our current results are a lower bound on the models' performance. We can also not guarantee that our observation formats do not cause tokenization issues across all APIS (e.g., a loss of structure for a 2D grid in ASCII). Finally, while our environments are simpler than complex real-world scenarios like robotics, they require the same fundamental agentic capabilities (long-context mul- timodal understanding, in-context imitation learning) that are often tested in robotics research and likely necessary for world success. Accordingly, our LMAct benchmark serves as a controlled testbed to evaluate these core skills, di- agnose current model limitations, and guide research toward more generally capable agents. Although direct transfer is beyond this paper's scope, findings on our benchmark can inform future work addressing that challenge.
4. Discussion & Related Work
Many large-scale benchmarks have been developed to test the general capabilities of frontier LMs, including the Chatbot Arena (Chiang et al., 2024) or LiveBench (White et al., 2025). Other benchmarks, such Frontier- Math (Glazer et al., 2024), TaskBench (Shen et al., 2024), GameBench (Costarelli et al., 2024). and Atari-GPT (Way- towich et al., 2024), specifically investigate the reason- ing or interactive decision decision-making capabilities of (L)LMs. Most closely related to our work, and published in parallel, the BALROG (Paglieri et al., 2025) benchmark evaluates frontier LMs' zero-shot reasoning on multimodal decision- making tasks in five game environments from Baby AI to NetHack. Our work complements these benchmarks by focusing on in-context imitation learning with long multi- modal context, filling a gap in the current literature. Our benchmark is currently in its "easiest" form (e.g., grid world without obstacles) and can casily be made more challenging.
Future Work For Cheetah Run (Fig. 8) and grid world navigation with ASCII demonstrations (Fig. 7) we observe that the models' performance deteriorates with increasing numbers of expert demonstrations, a phenomenon we refer to as "in-context interference". For both tasks our analysis of the percentage of illegal actions (Figs. A18 and A19) suggests the problem is not due to an increased number of illegal actions, but, instead, due to increasingly subopti- mal actions. We have conducted a set of initial investiga-
There are few reasons to be optimistic about in in-context imitation learning with long-context models. Many demon- strations should, in principle, improve performance over few-shot learning (as shown by Agarwal et al. (2024); Jiang et al. (2024) on non- ion-interactive tasks). and modern LMs have long enough contexts to test this at scale. Additionally,
8

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
tions into these failure modes (see Appendices C.2 C.4 and Fig. A14), but a definitive answer what causes in- context interference would require a thorough investigation, and, therefore, presents an interesting direction for future work. Another promising avenue for future work is to focus on models capable of general in-context reinforcement learn- ing, which bit different than our in-context imitation setting (in principle, all our tasks could easily be extended by providing additional reward observations). It also seems plausible that pretraining finetuning with data from inter- active decision-making tasks, and, in particular, in-context imitation of an expert policy, would be quite effective. Fi- nally, we also think that evaluating LMs' performance on partially observable tasks presents an interesting direction for future research (we primarily investigate fully observ- able tasks). Partially observable tasks require consistent integration of information across several, potentially non- adjacent, time steps, which is great test of a model's ability to densely attend to the information in the context. However, under partial observability, there is a theoretical problem of self-delusion in imitation of an expert that has hidden information (e.g., the hidden belief state of the agent; see Ortega et al. (2021)). We want to keep our benchmark free from these complicating issues and think that the right time to move to such harder tasks is when frontier models eas- ily solve simple, fully observable tasks at expert level (we are currently at beginner level). Nevertheless, we believe there is great value in benchmarks on interactive decision- making tasks under partial observability (perhaps better suited for in-context reinforcement learning, which avoids the self-delusion problem), and refer to, e.g., the BALROG benchmark (Paglieri al., 2025).
Impact Statement
The (partial) automation of intellectual labor will have sig- nificant societal and socioeconomic impact, both positive and negative. One current hurdle is the automation of general interactive decision-making tasks, as our bench- mark demonstrates. If solved, this would lead to more au- tonomous AI systems and agents that could facilitate or even partly take over intellectual labor across broad range of ap- plications and domains. We believe this vision warrants both enthusiasm and caution, and the scientific community and developers of Al technology bear responsibility in raising awareness of potential negative socioeconomic outcomes, and in helping develop potential mitigations. Having said that, our work is benchmark that reports the current state of affairs (showing the limitations of current frontier LMs) and serves as one yard stick to measure future innovations and progress against, but does not propose innovations to improve capabilities of AI systems. While our benchmark may contribute indirectly by helping develop more capable AI systems faster, we believe this is far outweighed by the benefit of having clearer picture of current capabilities and progress provided by our benchmark.
A Acknowledgments
We thank Anna Mitenkova, Dipanjan Das, Jordi Grau-Moya, Joel Veness, Kate Olszewska, Li Kevin Wenliang. Laurent Orseau, Marcus Hutter, Matthew Aitchison, Matthew Gins- berg, Minmin Chen, Orhan Firat, Satinder Baveja, Shaobo Hou, Zhengdong Wang, and the anonymous reviewers for their helpful feedback and insightful discussions.
Diipajaaa
References
5. Conclusion
Agarwal, R., Singh, A., Zhang, L.,, Bohnet, B., Rosias, L., Chan, S. C. Y., Zhang, B., Anand, A., Abbas, Z.,, Nova, A., Co-Reyes, J. D.,, Chu, E., Behbahani, F. M. P., Faust, A., and Larochelle, H. Many-shot in-context learning. In NeurIPS, 2024.
We evaluated the multimodal in-context imitation learning capabilities of some of the world's most advanced AI mod- els on interactive prcision-making tasks - tasks that are simple for humans but challenging for state-of-the-art LMs. Our results show that, even with hundreds of demonstra- tion episodes, context lengths of up to one million tokens, and thousands of output tokens, models often struggle to reach expert performance, thereby failing to translate their (factual) knowledge about the tasks' solution strategies into effective decision-making. Solving this problem will be crucial for the next generational leap in LM capabilitiiss towards general agents. Despite our focus on in-context im- itation learning, we believe it will be interesting to compare against other methods, including fine-tuning, retrieval- -based methods, rewarddwwwddeeoodd conditionn.....innnn.oooo. etc. Our open-soure bench- mark (https: : / /i thub. com/ com/google google-deepmil nd/ 1mm_ mm) serves as yard stick measure progress toward that goal, and we are excited see which innovations will be needed to solve all our simple tasks.
Aitchison, M., Sweetser, P., and Hutter, M. Atari-5: Distill- ing the arcade learning environment down to five games, In ICML, Proceedings of Machine Learning Research, 2023.
Anil, R., Borgeaud, S., Wu, Y., Alayrac, J., Yu, J., Sori- cut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Petrov, S., Johnson, M., Antonoglou, I., Schritwieser, J., Glaese, A., Chen, J., Pitler, E., Lilli- crap, T. P., Lazaridou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Doherty, R., Collins, E., Meyer, C., Ruther- ford, E., Moreira, E., Ayoub, K., Goel, M., Tucker, G., Piqueras, E.,, Krikun, M., Barr, L., Savinov, N., Danihelka,
9

---

LMAct: A Benchmark for In-Contest Imitation Learning with Long Multimodal Demonstrations
I., Roelofs, B., White, A., Andreassen, A., von Glehn, T., Yagati, L.,, Kazemi, M., Gonzalez, L.,, Khalman, M., Sygnowski, J., and et al. Gemini: A family of highly capable multimodal models. arXiv:2312.11805 2023.
Bulatov, A., Kuratov, Y., and Burtsev, M. Recurrent memory transformer. In NeurlPS, 2022.
Carlini, N. Playing chess with large language models, 2023. URL https::/niccho/ carlini c writing//022/chesss
Anthropie. The claude model family: Opus, sonnet, haiku. Technical report, Anthropie, 2024a.
html.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A.. Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. In NeurlPS, pp. 15084-15097 2021.
Anthropic. Claude 3.5 sonnet, 2024b.
URL
https: /tt:://wwwww www.aapicc....... com/ news/ claude-3- 3-5 3--iee--e----et.....- Accessed: 2024-11-26.
Anthropie.
Introducing computer use, a new claude 355 sonnet, and claude 3.5 haiku, 2024c.
Cherepanov, E., Staroverov, A., Yudin, D., Kovalev, A. K., and Panov, A. I. Recurrent action transformer with mem- ory. arXiv:2306.09444 2024.
URL https:/ www.wwwww...w.....//co com/news/ 3-5-model ---an-cooputer--sse 2024-11-26.
Accessed:
Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. Babyai: A platform to study the sample efficiency of grounded language learning. In ICLR, 2019.
Anthropic. Using the api rate limits, 2024d. URL https:/ /ocs. anthropic wwt.coo///eaeeeee wwt.c//////////.eeeeee /n/api// rate-limits. Accessed: 2024-11-26.
Anthropic. Build with claude vision, 2024e. URL wnt.ropc.....icccoooooo.. con/e///ccc///////eee /ocs/
Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating Ilms by human preference. In ICML, 2024.
build-win welld--itt-clladde /vis vison. Accessed: 2024- 11-26.
Barth-Maron, G., Hoffman, M. W., Budden, D., Dabney, W., Horgan, D., TB, D., Muldal, A., Heess, N., and Lilli- crap. T. P. Distributed distributional deterministic policy gradients. In ICLR, 2018
Costarelli, A., Allen, M., Hauksson, R., Sodunke, G., Hari- haran, S., Cheng, C., Li, W., and Yadav, A. Gamebench: Evaluating strategic reasoning abilities of LLM agents. arXiv:2406.06613, 2024.
Bauer, J., Baumli, K., Behbahani, F. M. P., Bhoopchand, A., Braddey-SShhieg N., Chang, M., Clay, N., Collis- ter, A., Dasagi, V., Gonzalez, L., Gregor, K., Hughes, E., Kashem, S., Loks Thompson, M., Openshaw, H., Palker-Holder, J., Pathak, S., Nieves, N. P., Rakicevic, N., Rocktäschel, T., Schroecker, Y., Singh, S., Sygnowski, J., Tuyls, K., York, S., Zacherl, A., and Zhang, L. M. Human-timescuul timescale adaptation in open-ended task space. In ICML, 2023.
Duan, Y., Andrychowicz, M., Stadie, B. C., Ho, J., Schnei- der, J., Sutskever, I., Abbeel, P., and Zaremba, W. One- shot imitation learning. In NIPS, 2017.
Fang, K., Liu, F., Abbeel, P., and Levine, MOKA: open- world robotic manipulation through mark-based visual prompting. In Robotics: Science and Systems, 2024.
Fiekas, N. python-chess, 2012.
URL https:
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-279, 2013.
/ h/python-cheree chess. www.thedocs..oo/ latest/index html.
Fu, L., Huang. H., Datta, G., Chen, L. Y., Panitch, W. C., Liu, F., Li, H., and Goldberg, K. In-context imitation learning via next-token prediction. arXiv:2408.15980 2024.
Brooks, E. A., Walls, L., Lewis, L., and Singh, S. Large language models can implement policy iteration. In NeurlPS, 2023.
Brown, T. B., Mann. B., Ryder, N., Subbiah, M., Kaplan,
Gallouédec, Q., Beeching, E., Romac, C., and Dellandréa, E. Jack of all trades, master of some, a multi-purpose transformer agent. 4rX1v:2402.09844, 2024.
J. Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss. A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS, 2020.
Genewein, T., Delétang, G., Ruoss, A., Wenliang, L. K., Catt, E., Dutordoir, V., Grau-Moya, J., Orseau, L., Hutter, M., and Veness, J. Memory- based meta-learnig on non- stationary distributions. In JCMI., 2023.
10

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Genewein, T., Li, K. W., Grau-Moya, J., Ruoss, A., Orseau, L., and Hutter, M. Understanding prompt tuning and in-context learning via meta-learning arXiv:2505.17010. 2025.
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Sel- vatici, M., Grefenstette, E., and Rocktäschel, T. The nethack learning environment. In NeurIPS, 2020.
Lee, J., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning. In NeurIPS, 2023.
Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., San- dler, R., Sevilla, J., Ren, Q.. Pratt, E., Levine, L.., Barkley, G.. Stewart, N., Grechuk, B., Grechuk, T.,, and Enugandla, S. V. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. vrXiv:2411.048/1 2024.
Li, S., Puig, X., Paxton, C.,, Du, Y., Wang, C., Fan, L., Chen, T., Huang, D., Akyürek, E., Anandkumar, A., An- dreas, J., Mordatch, I., Torralba, A., and Zhu, Y. Pre- trained language models for interactive decision decision-making NeurlPS, 2022.
Google. Understand and count tokens, 2024. URL
https://af wow.goooolll....ooo.ooooooooooo. dev..eeeeeeeeeeeee gemi ni-apiiiiipp// docs/tokens. Accessed: 2024-11-26.
Ma, Y. J., Hejna, J., Fu, C.,, Shah, D., Liang, J., Xu, Z., Kirmani, S., Xu, P., Driess, D., Xiao, T., Bastani, O., Jayaraman, D., Yu, W., Zhang, T., Sadigh, D., and Xia, F. Vision language models in-context value learners. In ICLR, 2025.
Google DeepMind. Gemini 2.0 flash experimen- google/ Technologgs ht/gg/mmnn/ffffee Accessed: 2025- 01-21.
tal, 2024a. URL
Matanović, A. Encyclopaedia of Chess Openings. Batsford Limited, 1978.
Google DeepMind. Introducing gemini 2.0: our new ai model for the agantic 2024b. URL httst
sooggl/e/ecccoooorrr///ogeeeeeep googlee--eethh---------de-----e Accessed: 2025-01-21.
Mikulik, Delétang, G., McGrath, T., Genewein, T., Mar- tic, M., Legg, S., and Ortega, P. A. Meta-trained agents implement bayes-optimal agents. In NearIPS, 2020.
Grau-Moya, J., Genewein, T., Hutter, M., Orseau, L., Delétang, G., Catt, E., Ruoss, A., Wenliang, I.. K., Mat- tern, C., Aitchison, M., and Veness, J. Learning universal predictors. In ICML, 2024.
Mirchandani, S., Xia, F., Florence, Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, D., and Zeng, A. Large language models general pattern machines. In CoRL, 2023.
Hafner, D. Benchmarking the spectrum of agent capabilities. In ICLR, 2022.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. arXiv:1312.5602,201 2013.
Hessel, M., Danihelka, I., Viola, F., Guez, A., Schmitt, S., Sifre, L., Weber, T.. Silver, D., and van Hasselt, H. Muesli: Combining improvements in policy optimization. In ICMI., 2021.
OpenAl. GPT-4 technical report. arXiv:2303.08774.200
OpenAL Hello gpt-4o, 2024a. URL htthtss:://ppeaii htt/innex/heell-ae 26.
Huang, W., Abbeel, P., Pathak, D.,, and Mordatch, I. Lan- guage models zero-shot planners: Extracting action- able knowledge for embodied agents. In ICML, 2022.
Accessed: 2024-11-
Jiang, Y., Irvin, J., Wang, J. H., Chaudhry, M. A., Chen, H., and Ng, A. Y. Many-shot in-context learning in multimodal foundation models. arXiv:2405.09798, 2024.
OpenAL. Models, 2024b. URL https://platform openai.com/ wwt.c/cccccccccccccttcc..o....co ddocs/models Accessed: 2024-11- 26.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. In NeurIPS, 2022.
OpenAI. Introducing openai ol op-vvee, 2024c. URL imrroduciingg--pnaaai-----p-pppe 2024-11-26.
https:/ /oppaa///pp/i....// com/ /nnex/
Accessed:
Kuratov, Y., Bulatov. A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A. Y., and Burtsev, M. BABILong: Test- ing the limits of Ims with long context reasoning-in-a- haystack. In NewrlPS, 2024.
OpenAL. https://pplaffommcoennt www.cp.cnnaoocooeoooooooomm docs/
Reasoning models, 2024d.
URL
guides/reasoningg Accessed: 2024-11-3

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
OpenAl. Vision, 2024e. URL https://platfomm
suka, L.,, Weng, L.., McCallum, L.,, Held, L.,, Ouyang, L., Feuvrier, I.., Zhang, L.., Kondraciuk, I.., Kaiser, L.,, Hewitt, L., Metz, L.,, Doshi, L., Aflak, M., Simens, M., Boyd, M., Thompson, M.. Dukhan, M., Chen, M., Gray, M., Hudnall, M., Zhang, M., Aljubeh, M., Litwin M., Zeng, M., Johnson, M., Shetty, M., Gupta, M., Shah, M., Yatbaz, M., Yang. M. J., Zhong, M., Glaese, M., Chen, M., Janner, M., Lampe, M., Petrov, M., Wu, M., Wang, M., Fradin, M., Pokrass, M., Castro, M., de Castro, M. O. T. Pavlov, M. Brundage, M., Wang, M., Khan, M., Murati, M., Bavarian, M., Lin, M., Yesildal, M., Soto, N., Gimelshein, N., Cone. N., Staudacher, N., Summers, N., LaFontaine, N., Chowdhury, N., Ryder, N., Stathas, N., Turley, N., Tezak, N., Felix, N., Kudige, N., Keskar, N., Deutsch, N., Bundick, N., Puckett, N., Nachum, O., Okelola, O., Boiko, O., Murk, O., Jaffe, O., Watkins, O., Godement, O., Campbell-Moore, O., Chao, P., McMil- lan, P., Belov, P., Su, P., Bak, P., Bakkum, P., Deng. P., Dolan, P., Hoeschele, P., Welinder, P., Tillet, P., Pronin, P., Tillet, P., Dhariwal, P., Yuan, Q. Dias, R., Lim, R., Arora, R., Troll, R., Lin, R., Lopes, R. G., Puri, R., Mi- yara, R., Leike, R., Gaubert, R., Zamani, R., Wang, R., Donnelly, R., Honsby, R., Smith, R., Sahai, R., Ramchan- dani, R., Huet, R., Carmichael, R., Zellers. R., Chen, R., Chen, R., Nigmatullin, R., Cheu, R., Jain, S., Altman, S., Schoenholz, S., Toizer, S., Miserendino, S., Agarwal, S., Culver, S., Ethersmith, S., Gray, S., Grove, S., Metzger, S., Hermani, S., Jain, S., Zhao, S., Wu, S., Jomoto, S., Wu, S., Shuaiqi, Xia, Phene, S., Papay. S., Narayanan, S., Coffey, S., Lee, S., Hall, S., Balaji. S., Broda, T., Stramer, T., Xu, T., Gogineni, T., Christianson, T., Sanders, T., Patwardhan, T., Cunninghman, T., Degry, T., Dimson, T., Raoux, T., Shadwell, T., Zheng, T., Underwood, T., Markov, T., Sherbakov, T., Rubin, T., Stasi, T., Kaftan, T., Heywood, T., Peterson, T., Walters, T., Eloundou, T., Qi, V., Moeller, V., Monaco, V., Kuo, V., Fomenko, V., Chang, W., Zheng, W., Zhou, W., Manassra, W., Sheu, W., Zaremba, W., Patil, Y., Qian, Y., Kim, Y., Cheng, Y., Zhang, Y., He, Y., Zhang, Y., Jin, Y., Dai, Y., and Malkov, Y. Gpt-4o system card. arXiv:2410.212777 2024a
openai homm/oca///// / httgg/gddd//iiiieeeee Accessed: 2024-11-26.
OpenAI, Hurst, A., Lerer, A., Goucher, A. P., Perelman,
A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., Madry, A., Baker- Whitcomb, A., Beutel, A., Borzunov, A., Carney, A., Chow, A., Kirillov, A., Nichol, A., Paino, A., Renzin, A., Passos, A. T., Kir- illov, A., Christakis, A., Conneau, A., Kamali, A., Jabri, A., Moyer, A., Tam, A., Crookes, A., Tootoochian, A., Tootoonchian, A., Kumar, A., Vallone, A., Karpathy, A., Braunstein, A., Cann, A., Codispoti, A., Galu, A., Kon- drich, A., Tulloch, A., Mishchenko, A., Baek, A., Jiang, A., Pelisse. A., Woodford, A., Gosalia, A., Dhar, A.. Pan- tuliano, A., Nayak, A., Oliver, A., Zoph, B., Ghorbani, B., Leimberger. B., Rossen, B., Sokolowsky, B., Wang, B., Zweig, B., Hoover, B., Samic, B., McGrew, B., Spero. B., Giertler, B., Cheng, B., Lightcap, B., Walkin, B., Quinn, B., Guarraci, B., Hsu, B., Kellogg, B., Eastman, B., Lu- garesi, C., Wainwright, C., Bassin, C., Hudson, C., Chu, C., Nelson, C., Li, C., Shern, C. J., Conger, C., Barette, C., Voss, C., Ding, C., Lu, C., Zhang, C., Beaumont, C., Hallacy, C., Koch, C., Gibson, C., Kim, C., Choi, C., McLeavey, C., Hesse, C., Fischer, C., Winter, C.,, Czar- necki, C., Jarvis, C., Wei, C., Koumouzelis, C., Sherburn, D., Kappler, D., Levin, D., Levy, D., Carr, D.,, Farhi, D., Mely, D., Robinson, D., Sasaki, D., Jin, D., Valladares, D., Tsipras, D., Li, D., Nguyen, D. P., Findlay, D., Oiwoh, E., Wong. E., Asdar, E., Proehl, E., Yang, E., Antonow, E., Kramer, E., Peterson, E., Sigler, E., Wallace, E., Brevdo, E., Mays, E., Khorasani, F., Such, P., Raso, F., Zhang, F. von Lohmann, F., Sulit, F., Goh, G., Oden, G., Salmon, G., Starace, G., Brockman, G., Salman, H., Bao, H., Hu, H., Wong, H., Wang. H., Schmidt, H., Whitney, H., Jun, H., Kirchner, H., de Oliveira Pinto, H. P., Ren, H., Chang, H., Chung, H. W., Kivlichan, I., O' Connell, I,, O'Connell, I., Osband, I., Silber, L., Sohl, I., Okuyucu, I. Lan, I., Kostrikov, I., Sutskever, I., Kanitscheider, I., Guirajani, I., Coxon, J., Menick, J., Pachocki, J., Aung, J., Betker, J., Crooks, J., Lemnon, J., Kiros, J., Leike, J., Park, J., Kwon, J., Phang, J., Teplitz, J., Wei, J.. Wolfe, J. Chen, J.. Harris, J., Varavva, J., Lee, J. G., Shieh, J. Lin, J., Yu, J., Weng, J.. Tang, J., Yu, J., Jang. J., Candela, J. Q., Beutler, J., Landers, J., Parish, J., Hei- decke, J. Schulman, J., Lachman, J., McKay, J.. Uesato, J.. Ward, J., Kim, J. W., Huizinga, J.. Sitkin, J., Kraai- jeveld, J., Gross, J.. Kaplan, J.. Snyder, J., Achiam, J., Jiao, J., Lee, J., Zhuang, J., Harriman, J., Fricke, K., Hayashi, K., Singhal, K., Shi, K., Karthik, K., Wood, K., Rimbach, K., Hsu, K., Nguyen, K., Gu-Lemberg, K., Button, K., Liu, K., Howe, K., Muthukumar, K., Luther, K., Ahmad, L., Kai, L., Itow, L., Workman, L., Pathak, L., Chen, L.,, Jing L.,, Guy, I.., Fedus, L.,, Zhou, L.,, Mamit-
OpenAL, Jaech, A., Kalai, A., Lerer, A., Richardson, A.. El- El-Kishky, A., Low, A., Helyar, A., Madry, A., Beu- tel, A., Carney, A., Iftimie. A., Karpenko, A., Passos, A. T., Neitz, A., Prokofiev, A., Wei, A., Tam, A., Bennett, A., Kumar, A., Saraiva, A., Vallone, A., Duberstein, A., Kondrich, A., Mishchenko, A., Applebaum, A.. Jiang, A., Nair, A., Zoph, B., Ghorbani, B., Rossen, B., Sokolowsky, B., Barak, B., McGrew, B., Minaiev, B., Hao, B., Baker, B., Houghton, B., McKinzie, B., Eastman, B., Lugaresi,
C., Bassin, C., Hudson, C., Li, C. M., de Bourcy, C., Voss,
C., Shen, C., Zhang, C., Koch, C., Orsinger, C., Hesse, C., Fischer, C., Chan, C., Roberts, D., Kappler. D., Levy,
D., Selsam, D., Dohan, D., Farhi, D., Mely, D., Robinson,
12

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
D., Tsipras, D., Li, D., Oprica, D., Freeman, E., Zhang, E., Wong, E.,, Proehl, E., Cheung, E.,, Mitchell, E., Wal- lace, E., Ritter, E., Mays, E., Wang, F., Such, F. P., Raso, F., Leoni, F., Tsimpourlas, F.. Song, F., von Lohmann, F., Sulit, F. Salmon, G., Parascandolo, G., Chabot, G., Zhao, G., Brockman, G., Leclere, G., Salman, H., Bao, H. Sheng, H., Andrin, H., Bagherinezhad, H., Ren, H., Lightman, H., Chung. H. W., Kivlichan, I., O'Connell, I. Osband, I., Gilaberte, I. C.,, Akkaya, I., Kostrikov, I,, Sutskever, I., Kofmman I., Pachocki, J., Lennon, J., Wei, J., Harb, J., Twore, J., Feng. J., Yu, J.. Weng, J., Tang, J., Yu, Candela, J. Q. Palermo, J., Parish, J., Heidecke, J., Hallman, J., Rizzo, J., Gordon, J., Uesato, J., Ward, J., Huizinga, J., Wang, J., Chen, K., Xiao, K., Singhal, K., Nguyen, K., Cobbe, K., Shi, K., Wood, K., Rimbach, K., Gu-Lemberg, K., Liu, K., Lu, K., Stone, K., Yu, K., Ahmad, L., Yang, L., Liu, L., Maksin, L., Ho, L.,, Fedus, L., Weng, I.,, Li, I.., McCallum, I.., Held, I.., Kuhn, I.., Kondraciuk, L., Kaiser, L., Metz, L., Boyd, M. Trebacz, M., Joglekar, M., Chen, M., Tintor, M., Meyer, M., Jones, M., Kaufer, M., Schwarzer, M., Shah, M., Yatbaz, M., Guan, M. Y., Xu, M., Yan, M., Glaese, M., Chen, M., Lampe, M., Malek, M., Wang, M., Fradin, M., McClay, M., Pavlov, M., Wang, M., Wang, M., Murati, M., Bavar- ian, M., Rohaninejad, M., McAleese, N., Chowdhury, N., Chowdhury, N., Ryder, N., Tezak, N., Brown, N., Nachum, O., Boiko, O., Murk, O., Watkins, O., Chao, P., Ashbourne, P., Izmailov, P., Zhokhov, P., Dias, R., Arora, R., Lin, R., Lopes, R. G., Gaon, R., Miyara, R., Leike, R., Hwang, R., Garg, R., Brown, R., James, R., Shu, R., Cheu, R., Greene, R., Jain, S., Altman, S., Toizer, S., Toyer, S., Miscrendino, S., Agarwal, S., Hernandez, S., Baker, S., McKinney, S., Yan, S., Zhao, S., Hu, S.. San- turkar, S., Chaudhuri, S. R., Zhang, S., Fu, S., Papay, S., Lin, S., Balaji, S., Sanjeev, S., Sidor, S., Broda, T., Clark, A., Wang. T., Gordon, T., Sanders, T., Patwardhan, T., Sottiaux. T., Degry. T., Dimson, T., Zheng, T., Garipov, T. Stasi, T., Bansal, T., Creech, T., Peterson, T., Eloundou, T., Qi, V., Kosaraju, V., Monaco, V., Pong, V., Fomenko, V., Zheng, W., Zhou, W., McCabe, W., Zaremba, W., Dubois, Y., Lu, Y., Chen, Y., Cha, Y., Bai, Y., He, Y., Zhang, Y., Wang, Y., Shao, Z., and Li, Z. Openai ol system card. 24X2:2412.1162000 2024b.
B., Pérolat, J., Everitr, T., Tallec, C., Parisotto, E., Erez, T., Chen, Y., Reed, S. E., Hutter, M., de Freitas, N., and Legg, S. Shaking the foundations: delusions in sequence models for interaction and control. arXiv:2110.10819, 2021.
Paglieri, D., Cupiat, B., Coward, S., Piterbarg, U., Wolczyk, M. Khan, A., Pignatelli, E., Eukasz Kuciński, Pinto, I.,, Fergus, R., Foerster, J. N., Parker-Holder, J., and Rocktäschel, T. Balrog: Benchmarking agentic 1lm and vim reasoning on games. In ICLR, 2025.
Palo, N. D. and Johns, E. Keypoint action tokens enable in-context imitation learning in robotics. In Roboties: Science and Systems, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, Language models are unsupervised multitask learners. 2019.
Raparthy, S. C.,, Hambro, E., Kirk, R., Henaff, M., and Raileanu, R. Generalization to new sequential decision making tasks with in-context learning. In ICML, 2024.
Reed, S. E., Zolna, K., Parisotto, E., Colmenarejo, G., Novikov, A., Barth-Maron G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg. J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, R., Vinyals, O., Bordbar, M., and de Freitas, N. A generalist agent. Trans. Mach. Learn. Res., 2022.
Reid, M., Savinov, N., Teplyashin, D., Lepikhin, D., Lilli- crap, T. P., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., Antonoglou, I., Anil, R., Borgeaud, S., Dai, A. M., Millican, K., Dyer, E., Glaese, M., Sotti- aux, T., Lee, B., Viola, F., Reynolds, M., Xu, Y., Molloy, J., Chen, J., Isard, M., Barham, P., Hennigan, T., Mcllroy, R., Johnson, M., Schalkwyk. J., Collins, E., Rutherford. E., Moreira, E., Ayoub, K., Goel, M., Meyer, C., Thom- ton. G., Yang. Z., Michalewski, H., Abbas, Z., Schucher, N., Anand, A., Ives, R., Keelingg J., Lenc, K., Haykal, S., Shakeri, S., Shyam, P., Chowdhery, A., Ring, R., Spencer, S., Sezener, E., and et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXhv:2403.055300 2024.
Ortega, P. A., Wang, J. X., Rowland, M., Genewein, T., Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., Jayakumar, S. M., Mc- Grath, T. Miller, K. J., Azar, M. G., Osband, I., Ra- binowitz, N. C., György, A., Chiappa, S., Osindero, S., Teh, Y. W., van Hasselt, H., de Freitas, N., Botvinick, M. M., and Legg, S. Meta-learning of sequential strate- gies. arXiv:1905.0300 1905.0330, 2019.
Romera-Paredes, B., Barckatain. M., Novikov, A.. Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., Kohli, P., and Fawzi, A. Math- ematical discoveries from program search with large lan- guage models. Nat., 2024.
Romstad, T., Costalba, M., Kiiski, J., Linscott, G., Nasu, Y., Isozaki, M., Noda, H., and et al. Stockfish, 2008. URL https: https://stock stokkktkkktkkk f shchesshhss.ooo org.
Ortega, P. A., Kunesch, M., Delétang, G., Genewein, T., Grau-Moya, J., Veness, J., Buchli, J., Degrave, J., Piot,
13

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Ruoss, A., Delétang, G., Medapati, S., Grau-Moya, J., Wen- liang, L. K., Catt, E.,, Reid, J., Lewis, C. A., Veness, J., and Genewein, T. Amortized planning with large-scale transformers: A case study on chess. In NeurlPS, 2024.
Wang, Y., Zhang, B., Chen, J., and Sreenath, K. Prompt robot to walk with large language models. In CDC, 2024.
Waytowich, N. R., White, D., Sunbeam, M., and Goecks, V. G. Atari-GPT: Investigating the capabilities of mul- timodal large language models low-level policies for atari games. arXiv:2408. 59500, 2024.
Ryle, G. The concept of mind. University Of Chicago Press, 1949.
Schultz, J., Adamek, J., Jusup, M., Lanctot, M., Kaisers, A., M., Perrin, S., Hennes, D., Shar, J., Lewis, C., Ruoss,
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, E. Chi, E. H., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Zahavy, T., Veličković, P., Prince, L., Singh. S., Malmi, E., and Tomašev, N. Mastering board games by external and internal planning with language models. In ICML, 2025.
Wenliang, L. K., Ruoss, A., Grau-Moya, J., Hutter, M., and Genewein, T. Why is prompting hard? un- derstanding prompts on binary sequence predictors. arXiv:2502.10760, 2025.
Shen, Y., Song, K., Tan, X., Zhang, W., Ren, K., Yuan, S., Lu, W., Li, D., and Zhuang, TaskBench: Bench- marking large language models for task automation. In NeurlPS, 2024.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Dey, S., Shubh- Agraba,, Sandha, S. S., Naidu. S. V., Hegde, C., LeCun, Y., Goldstein, T., Neiswanger, W. and Goldblum. M. LiveBench: A challenging, contamination-liman LLM benchmark. In ICLR, 2025.
SIMA Team, Raad, M. A., Ahuja, A., Barros, C.,, Besse, F.,
Bolt, A., Bolton, A., Brownfield, B.. Buttimore, G., Cant,
M., Chakera, S., Chan, S. C. Y., Clune, J., Collister, A.,
Copeman, V., Cullum, A., Dasgupta, I., de Cesare, D.,
Trapani, J. D., Donchev, Y., Dunleavy, E.,, Engelcke, M., Faulkner, R., Garcia, F,, Gbadamosi, C., Gong, Z., Gonza- lez, L., Gupta, K., Gregor, K., Hallingstad, A. O., Harley, T., Haves, S., Hill, F., Hirst, F.,, Hudson, D. A., Hudson, J., Hughes-Fitt, S., Rezende, D. J., Jasarevic, M., Kampis, L., Ke, N. R., Keck, T., Kim, J., Knagg, O.,, Kopparapu, K., Lampinen, A. K., Legg, S., Lerchner, A., Limont, M., Liu, Y., Loos-Soommonnn, M., Marino, J.. Cussons, M., Matthey, L.. Mcloughlin, S., Mendolicchio, P., Merzic, H., Mitenkova, A., Moufarek, A., Oliveira, V., Oliveira, Y. G., Openshaw, H., Pan, R., Pappu, A., Platonov, A., Purkiss, O., Reichert, D. P., Reid, J., Richemond, P. H., Roberts, T., Ruscoe, G., Elias, J. S., Sandars, T., Sawyer, D. P., Scholtes, T., Simmons, G., Slater, D., Soyer, H., Strathmann, H., Stys, P., Tam, A. C., Teplyashin, D., Terzi, T., Vercelli. D., Vujatovic, B., Wainwright, M., Wang, J. X., Wang. 7., Wierstra, D., Williams, D., Wong, N., York, S., and Young, N. Scaling instructable agents across many simulated worlds. arXiv:2404.10177 2024.
Whitlock, D. genxword, 2011. URL https://githh:. com/ wwwerrwn/geeewweeeee...
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang. M., Wang. J., Jin, S., Zhou, E., Zheng, Fan, X., Wang, X., Xiong, L.,, Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., Yin, Z., Dou, S., Weng. R., Qin, W., Zheng, Y., Qiu, X., Huang, X., 7hang, Q., and Gui, T. The rise and potential of large language model based agents: survey. Sci. China Inf. Sci., 2025.
Xu, M., Shen, Y., Zhang, S., Lu, Y., Zhao, D., Tenenbaum, J. B., and Gan, C. Prompting decision transformer for few-shot policy generalization. In ICML, 2022.
Yang, C., Wang, X., Lu, Y., Liu, H., Le, Q. V., Zhou, D., and Chen, X. Large language models optimizers. In ICLR, 2024.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In ICLR, 2023.
Sridhar, K., Dutta, S., Jayaraman, D., and Lee, I. REGENT: A retricval- auggmeted generalist agent that can act in- context in new environments. In NeurlPS 2024 Workshop on Open- World Agents, 2024.
Yin, Y., Wang. Z., Sharma, Y., Niu, D., Darrell, T., and Herzig, R. In-context learning enables robot action pre- diction in llms. 1rXiv:2410.127822 2024.
Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq. A., Lillicrap, T. P., and Riedmiller, M. A. Deepmind control suite. 1001.0000000090 2018.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IROS, 2012.
14

---

LMAct: A Benchmark for In- Imitation Learning with Long Multimodal Demonstrations
A. Related Work
The emergence of strong in-context learning capabilities with increasing model and training data scale LLMs was first observed in the GPT-2 paper (Radford et al., 2019) and even gave the title to the GPT-3 paper (Brown et al., 2020). Soon after, the recipe of next-token prediction at scale was applied to build agents from large sequential predictors, resulting, e.g., in the decision (pretrained) transformer (Chen et al., 2021; Lee et al., 2023), the generalist agent GATO (Reed et al., 2022), and its more recent open source variant JAT (Gallouédec et al., 2024). While the empirical results were certainly surprising at the time, at least in theory, in-context learning must necessarily arise as a core feature of a sequential predictor trained to minimize next token log loss over an implicit meta distribution of data (Ortega et al., 2019; Mikulik et al., 2020; Genewein et al., 2023), and could, in principle, even lead to universal in-context predictors (Grau-Moya et al.. 2024). An explicit application of this memory-basee meta-learning principle at scale is the "adaptive agent" from Bauer et al. (2023), which shows that an embodied agent in 3D environment can adapt to novel task instances on human timescales (i.e., single- or low double digit numbers of interaction episodes) purely in context and across a vast set of tasks. The SIMA Team et al. (2024) conducted another impressive large-scale application of training and fine-tuning complex vision- agent across a large set of environments, using instruction conditioning together with in-context learning.
Instead of pretraining separate models for general decision-making many researchers have also attempted to directly use the knowledge and reasoning capabilities of LLMs and VLMs for building agents that interact with an environment (see Xi et al. (2025) for 2023 survey of LLM-based agents) - either by fine-tuning (Li et al., 2022) or purely via in-context demonstrations (Palo& & Johns, 2024). In both cases, two open question are: (i) how best represent environment observations as tokens, and (ii) how to best elicit the rocision-making capabilities of pretrained LMs. Mirchandani et al. (2023) find that pretrained LLMs are "general pattern machines" that can learn to complete complex token sequences via in-context learning, including agent-environne alphabet does not have significant impact, suggesting that LLMS may be able to deal with many different ways of translating observations into tokens. Perhaps more important is the question whether observations should be state-action sequences (as in imitation learning and our work) or whether they should include rewards (for reward conditioning or in-context RL as in Mirchandani et al. (2023) and Raparthy al. (2024)). While this is still unclear for pretrained LMs, Ruoss al. (2024) find that, when training large transformer play chess, performance is roughly the same for imitation learning compared to learning to predict state or action values (as long as the amount of data for all three variants is equal). Schultz et al. (2025) extend these results on chess to LLMs by distilling the search proceedure into the model by linearizing the search trees. Ma et al. (2025) find that, when prompted appropriately. a pretrained VLM (Gemini 1.5 Pro) can produce good value estimates for real-world robotic tasks. The second open problem, how to best prompt LMS for decision-making is a very active research area (Wenliang et al., 2025; Genewein et al., 2025), with a lot of focus on designing or learning zero-shot prompts, such as the famous "Let's think about this step by step." (Kojima et al., 2022) and "Take a deep breath and work on this problem step-by-step." " (Yang et al., 2024), which has led to many chain-of- of-thought prompting schemes (Wei et al., 2022). Besides better zero-shot prompts, advanced sampling and prompt optimization schemes have been explored, such as iterative prompt refinement where an agent starts with some demonstrations in the context, then interacts with the environment, and potentially replaces lower performing episode with a higher performing one (Mirchandani et al., 2023; Brooks et al., 2023).
interactions. They even find that in many cases, randomly swapping the
With the recent availability of long context models, a third possibility compared to improving zero-shot or few-shot prompts has emerged: many-shot in-context learning, i.e., prompting with many demonstrations, on the order of having a full small dataset in the context. Both Agarwal et al. (2024) and Jiang et al. (2024) show that many-shot prompts (hundreds or thousands of examples in the prompt) improve pretrained LM performance over few-shot prompts in non-interactive tasks. Our paper explores the same direction, with potentially hundreds of full episodes in the context, for interactive decision At the time of writing, querying long context models with many tokens comes with high computational cost, but it would be interesting to investigate how specialized models that were specifically developed for long-context tasks (Bulatov et al., 2022; Cherepanov et al., 2024) would fare on our benchmark. An alternative could be to virtually extend the context via retrieval based methods, e.g., REGENT (Sridhar et al., 2024), which trains a retrieval based agent. The retrieval problem is currently not fully solved (as also pointed out in Paglieri et al. (2025)), and today's 's largest state-of-the-eee LMs do not offer retrieval via their APIs and/or have not been trained perform retrieval. Accordingly, placing all the data in the context, as in our work, allows estimating LM performance independent of whether retrieval works well. Our results thus serve as an upper baseline to calibrate retrieval based methods against.
tasks.
Another line of work has investigated in-context learning for control (Duan et al., 2017: Xu et al., 2022; Fu et al., 2024; Fang et al., 2024; Yin et al., 2024; Wang et al., 2024). While these methods focus on developing novel agents with strong in-context learning capabilities, our paper's primary goal is to benchmark the current state of existing, general-purpose
15

---

LMAct: A Benchmark for In-Contest Imitation Learning with Long Multimodal Demonstrations
frontier LMs on such tasks. Understanding how to best leverage insights from specialized agent research enhance these large, pre-trained models remains an open question, which underscores the importance of establishing clear benchmarks like LMAct.
Comparing capabilities of LMs that span many tasks and domains requires large-scale benchmarks, such as the Chatbot Arena (a.k.a. LMSys, Chiang et al. (2024)) or LiveBench (White et al., 2025). Other benchmarks, such as FrontierMath (Glazer et al., 2024), Taskbench (Shen et al., 2024), Gamebench (Costarelli et al., 2024), and BABILong (Kuratov et al., 2024), specifically investigate the reasoning or interactive decision makinng capabilitiess of LLMs. Closely related these and our work, and published in parallel to our work, the BALROG (Paglieri et al.. 2025) benchmark evaluates state-of-the-an LMs (the same in our work with the exception of ol-mini and ol ol-preview and the addition of the Llama 3 models, which we do not evaluate) on multimodal reasoning and pacisionnmmkingg tasks by using set of 5 increasingly harder game environments from Baby AI to Nethack. Like our work, the authors find that state-of-the game environments. The authors also make the observation that models arguably possess a lot of knowledge about their tasks when queried appropriately, but that ofate-of-the-ar LMs have a "knowing- paper performs zero-shot evaluation and lists few-shot and many-shot evaluations as an important open research question (the released codebase supports few-shot evaluations, but these evaluations have not been performed the time of writing), Similar to BALROG, Waytowich et al. (2024) evaluate the zero-shot same-playing capabilities of frontier LLMs (GPT-4V Turbo, GPT-4o, Gemini 1.5 Flash, and Claude 3 Haiku) on 8 different Atari games (but not the Phoenix game that we consider). Compared to these previous benchmarks, ours is the only one that puts the emphasis on imitation learning with long context in multimodal interactive environments - a regime that pushes against the limits of modern LMs' in-context learning and reasoning capabilities, both from an engineering and a capabilities perspective. Furthermore, our benchmark covers the zero-shot, few-shot, and many-shot setting in a unified and thus easily comparable evaluation.
LMs struggle significantly in challenging
gap. Unlike our work, the BALROG
B. Experimental Details
In this section, we provide additional details on our experimental setup.
B.1. Models
Claude 3.5 Sonnet Claude 3.5 Sonnet has maximum context of 200k tokens, but can only process 100 images a time. In our setting, single episode can already consist of up to 100 images. As a result, we can, e.g., not evaluate in-context learning for Claude 3.5 Sonnet on Atari. Claude 3.5 Sonnet uses approximately (width px height px0/750 tokens per image (Anthropic, 2024e). Finally, note that the monthly token limits for Claude 3.5 Sonnet are very low compared to the other models ($5000 spend limit per month (Anthropic, 2024d)), which is why we can only conduct a somewhat limited evaluation of this model.
Gemini 1.5 Flash, Gemini 1.5 Pro, and Gemini 2.0 Flash Experimental The Gemini 1.5 models have a maximum context window of 10 million tokens. In practice, we restrict ourselves to million tokens due the prohibitive cost of evaluating even longer prompts. The Gemini API considers images to be of fixed size, meaning that they consume fixed number of tokens (currently 258), regardless of their file size (Google, 2024). Gemini 2.0 Flash Experimental supports up to 1M input tokens and 8k output tokens (Google DeepMind, 2024a).
GPT-4o The GPT-40 model has context length of 128k tokens and can process up to 250 images per prompt (OpenAI, 2024b). We use the "auto" resolution setting to process the images, which automatically determines whether to use a "low" or "high" resolution based on the image size (OpenAI, 2024c). In the "low" resolution mode, the model represents an image with 85 tokens. In the "high" resolution mode, the model first consumes the low-resolution image (using 85 tokens) and then creates additional detailed crops of the image using 170 tokens per 512px 512px tile to cover the whole image.
o1-mini, ol- ol-preview, and o1 ol-mini and ol --rreview have a context window of 128k tokens with up to 65k output tokens but cannot process images (OpenAI, 2024d) so we only evaluate them in the text- environments (c.g., not on Atari). The ol model has a context window of 200k tokens and up to 100k output tokens.
16

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
8
X
X
O
X
(a) ASCII
(b) RGB
Figure A2. Sample observations for each state representation format from our tic-tac- environment.
Figure AI. RGB observation for
Phoenix game for Atari 2600.
b
b
n
[Event "World Championship "]
P P
P
P
P
P P
[Site "London"]
[Date "2024.10.29"]
[Round "1"]
[White "Stockfish")
[Black "Gemini"]
[Result "*"]
P
A
A
A
BNBOKBNG
1. b3 d5 2. e3 e5 3. c4 *
(a) ASCII
(b) PGN
(c) RGB
rnbgkbnr babgkbar/ppppppop pppppppp StAtDppp/////0 //88////////////////////////1//8 PPWB////PPPPPPPPPP /RNBOKBNR KQkq
(d) FEN
Figure A3. Sample observations for each state representation format from our chess environment, all of which we generate with the python-chess library (Fickas, 2012). Note that, unlike the ASCII, FEN, and RGB, which show the opening board state, the PGN corresponds to more advanced position to visualize the move list (which would be empty for the opening board state).
B.2. Environments
Visualizing State Representations representations for environments which representations exactly depends the environment. For example, for chess, evaluate four different formats: (i) ASCIL (ii) FEN, (iii) PGN, and (iv) RGB images. We visualize all formats for each environment in Figs. A1 to A6.
To evaluate multimodal in-context imitation learning, consider multiple state
Atari Phoenix Unlike chess where superhuman expert policy (i.e., Stockfish) is publicly available, or tic-tac-toe where the optimal policy can be described with closed-form algorithm, the best performance on Atari 2600 games is generally obtained by strong reinforcement learning (RL) agents (Mnih et al., 2013; Hessel et al., 2021). However, rather than training an RL policy from scratch (which can be finicky), we make use of the training data corpus of the Gato project (Reed et al., 2022). Concretely, Gato trained a Muesli (Hessel et al., 2021) agent 200M steps and randomly recorded roughly 20k episodes generated by the agent during training. As result, the dataset also contains trajectories from the beginning of training where the agent does not yet perform well, so we only consider the last 2048 trajectories (i.e., the final stages of training). We further subsample these trajectories by only keeping the 256 highest-scoring as expert demonstrations. Since we only evaluate the first 100 steps (i.e., 400 frames with an action repeat of 4), we subsample w.r.t. the cumulative reward in the first 400 frames and not w.r.t. the entire episode. Overall, we obtain a collection of demonstration episodes with a high average return of 459.9 in the first 400 frames. In our experiments we match the Gato setting (Reed et al., 2022), i.c., sticky actions and no uncontrolled random initial no-ops. To ensure variability in the evaluation episodes, we manually perform different numbers of no-ops (based on the random seed) the beginning of every
17

---

LMAct: A Benchmark for In- In-Context Imitation Learning with Long Multimodal Demonstrations
F8
02
03
4
G
T
E
*#
A
05
K
4#
M
0
R
06
07
88
A
*#
##
N
44
44
08
46
8
E
10
*#
#
+#
Across:
Across:
2: So or Go Fish
2: So oE Go Fish
hold 'em atake
hold stake
5: Sierra Club co-founder John
5: Sierra club co-founder John
Cueto stat
Dueto stat
8: Eye sore
Eye sore
10: Shorn female
10: Shorn female
11: Store
ges station
11: Store gas station
Eown:
Bown:
Libatiin 2: It's set in e setting
Sushi
libation 2: It's set in a setting
3: Sincere, as an opology
Sincere, as an apology
7: Seating sections
Seating sections
9: Afteznoon event in Chelsea
9: Afternoon event in Chelses
(a) Initial crossword
(b) Solved crossword
Figure A4. Sample observations from our crossword environment. Fig. A4a shows the imitial crossword and Fig. A4b shows solved crossword after all the words have been placed in their corresponding slots. We create the crosswords of size 7 7 using the genxword crossword generator (Whitlock, 2011) and list 55 189 clues collected by Matthew Ginsberg (we only use clues with lowest difficulty rating; the full list contains 236615 615 clues).
episode before starting the evaluation, which changes the initial state from which the agent has control (since enemies in Phoenix keep moving during this time).
DM Control - Cheetah Run There is generally no closed-form or publicly available expert policy for the tasks from the DM Control suite. Thus, we also leverage the Gato training corpus to generate our expert demonstration episodes for cheetah run. For this task the Gato project trained a D4PG (Barth-Maron et al., 2018) agent, and, like for Atari, we only consider the last 10k episodes (since those correspond to the later stages of training and thus better performance). Ideally we would want to subsample the gighest-scoring trajectories as expert demonstrations, but, unfortunately, the underlying MuJoCo (Todorov et al., 2012) physies have changed since the time the Gato data collection, which means that the rewards in the dataset no longer match the observation-actions pairs our DM Control environment (i.e., replaying the actions from the same initial state does not yield the same return). Thus, first replay the actions for every trajectory in our collection (setting the initial state based on the first observation) and use the returns to subsample the 1000 highest-scoring trajectories (again only considering the cumulative reward in the first 100 steps). Overall, we obtain a collection of demonstration episodes with a high average return of 37.4 in the first 100 steps.
B.3. Prompts
The frozen part and the dynamic part of the evaluation prompt are illustrated in Listing 1 and Listing 2, respectively.
C. Additional Results
In this section, we present additional results and ablations from our experimental evaluation.
C.1. Ablating the Maximum Sample Length
The ol family of models tends to generate (long) internal "reasoning traces" before returning an output. Thus, the maximum sample length not large enough, these models may not have enough "reasoning tokens" and therefore do not
18

---

LMAct: A Benchmark for In-Contess Imitation Learning with Long Multimodal Demonstrations
Listing frozen part of the evaluation prompt, which contains the expert demonstration episodes stays constant throughout an evaluation episode. In this example, we have 8 demonstration episodes with 10 steps each and RGB observations. Before feeding the prompt to the model, we replace the observation and action placcholders with the actual observations (i.c., images this case) and action strings.
demonstration_prre proonpprpppttt
You are a powerful reinforcement learning agent, You can effectively identify policy exposed by demenstrations and repreduce it in a new situation.
Here are a number of demonstraticns:
Observation: <IMG_0_0> 0> Action: <AC_0_0>
Observation: <IMG_ 0 1>
Action: <AC_0_1>
10 Observation: <IMG 99 Action: _AC_0__ 9>
11 Observation: <IMG_1_0> Action: <AC_1_0>
13
Observation: <IMG_1_1> Action: <AC_1_1>
14
15
Observation: <IMG_1_99 9 Action: <AC_1_99
18
19 Observation: <IMG_7_0s Action: <AC_7_0>
20
Observation: <IMG_7_1> Action: <AC_7_1>
Observation: <IMG_7_9> Action: <AC_7_9>
23
Listing 2. The dynamic part of the evaluation prompt containing the evaluation trajectory. While stepping through an environment, we append this prompt the Listing I in every evaluation step (e.g., for the 3rd step here), again replacing the observation and action placeholders with the actual observations and actions. This example also shows the legal moves (lines to 10) and uses chain-of-thought prompting (lines 12 to Wei (2022)), both of which may be omitted depending on ablations in Appendix C.4 (which vary for each model-task combination).
evaluation prompt .. This is the current situation:
Observation: <IMG_8_00 Action: <AC_8_0>
Observation: <IMG_8_1x Action: <Ac_8_1> Observation: ZIMG_8_22
In this situation, this is the list of all the moves that are legal:
10 no action, jump left, left
11
12 Given the demonstrations and the current situation, you should infer the next logical action. Check that the chosen actien is in the set of legal moves. Think step by step
14 and very briefly explain your reascning for choosing this action. You must anawer with
15 the reasoning followed by the acticn the following format:
16 Reasoning:
Action:
19

---

LMAct: A Benchmark for In-Contes, Imitation Learning with Long Multimodal Demonstrations
ppos' tfont: 0000099422239793316999 .... -9.032292537204244775588
'velocity": 10.06424573024514888, ... 6.196622999133810011
(a) Observation
0. 80407661199221311 9.9588138635922255, -1.0, 9.79326653229446111 1.C, 1.0]
(b) Action
Figure A5. A sample observation and action from the cheetah run task from the DM Control suite (Tassa et al., 2018). Note that the observation is actually presented to the model on a single line (we use the multi-line representation here for ease of visualization). Moreover, we only show the truncated position and velocity vectors (represented by the ellipsis). The full position vector contains 8 elements, and the full velocity vector contains 9 clements. All elements are float 64 converted to string.
produce an output (in which case the API returns an empty sample). Since we want to report each model's best performance on our benchmark, we therefore ablate the maximum number of sample tokens and choose the configuration that trades off good performance and low cost (since, after a certain point, more tokens generally do not improve performance but only increase the cost). The maximum sample length also has an impact on the other models, particularly when using chain-of- off---fftthhhhhh reasoning, but a relatively low sample length typically suffices for our tasks (unlike the ol models which require large maximal sample length).
To that end, Fig. A7 shows our ablation of the maximum sample length for Claude 3.5 Sonnet, GPT-4o, ol-mini, -preevie Gemini 1.5 Flash, and Gemini 1.5 Pro on all three observation types (ASCII, coordinates and RGB) for the grid world navigation task with 1 demonstration episode. Fig. A7 shows the average score over 100 evaluation episodes for maximum sample lengths from 32 to 32768 tokens (Claude 3.5 Sonnet only supports 8192 output tokens and GPT-4o only supports 16384 output tokens). Unsurprisingly, the performance of Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Flash, and Gemini 1.5 Pro is largely unaffected by the number of sample tokens, i.e., even 32 tokens are sufficient to achieve their best performance (with the exception of Claude 3.5 Sonnet and GPT-4o on RGB observations, where they benefit from having 64 tokens). This is in stark contrast with ol-mini and ol op-rrvve,, which require between 4096 and 8192 output tokens achieve their optimal performance (with very steep degradation below 4096). To verify whether this sharp rise in performance is actually due to the model not generating valid action, Fig, A7 also visualizes the average percentage of illegal actions per episode over the maximum sample length, which (inversely) correlates very strongly with performance for the ol models. We therefore set the maximum sample length to 8192 for the ol models as a tradeoff between cost and performance (the maximum for ol -rrevview is 32768, the maximum for -m-i is 65536 (OpenAI, 2024d)). This should enable the ol models to achieve their best performance on our benchmark - even though they do so at much higher (computational) cost than the other models.
C.2. Replaying a Demonstration Episode
In-context imitation learning requires several different skills, one of which being able to locate and retrieve the relevant demonstration(s) from the context. We therefore conduct "sanity check" experiment where we provide a single demon- stration episode and "replay" the exact same episode for evaluation (akin to multimodal sequence copying task). Thus, for every step, the model only has to find the correct location in the demonstration episode in its context and reproduce its corresponding action. Accordingly, we slightly change the evaluation setup since the next observation (both action and next state) in the evaluation trajectory is always determined by the demonstration episode and not by the action generated by the model (i.e., we perform teacher forcing rather than a dynamic evaluation). As a result, we are not interested in a model's return but in how often it manages to match the action from the demonstration episode. Like in our other experiments, we evaluate 100 different episodes. For the sake of simplicity, we do not use chain-of-thour th-thhught prompting, do not show the legal actions in the prompt, and only consider single demonstration episode. Other than that, we leave the prompt the same as in our default experimental setup (i.e., including the same preamble and separator).
Figs. A8 to A13 show the average action replay accuracy per evaluation step for all frontier models on all six environments (for all state representation formats separately). We observe that models are generally capable of replaying the demonstration episodes, with slightly lower performance deeper into the episode. The significant exception ol -mini which struggles with the replay task in most settings. Similarly, Gemini 1.5 Flash fails to replay the actions for tic-tac-toe with RGB
20

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
xall
well
Hall
Hall
well
Mall
Hall
wall 1
well
vall
Nall
well
Noll 1 tile
tilc
tile
t:le
tile
1 tile
1
t11e
tilc
M0l1
wall
1 tile
t116
tile
t:1e
:ile
1 tile
player
1 :ile
cile
tile
Hell
wall
1 t11e
tile
tile
1:1
tile
1 tile
tile
1
:11e
[
tile
Well
xall
1 t110
til6
tile
t:le
tile
1 tile
tilc
1 111e
tile
tile
Mol1
1 t110
t110
ti1a
t:16
t11
1 tilc
ti1c
1 1110
ti1e
W0l1
wall
1 t11e
tazget
tile
1:10
:11e
1 tile
ti1e
1 :110
c13e
tile
wall
<ell
1
111
Li1e
Ui1e
1
1
Lite
1114
1
111
Li3e
(ile
[
well
xal1
1 t110
t126
ti10
t:1e
c1le
t420
t:1c
1 1110
c126
Hal1
t110
t118
1:10
t110
t110
t:10
1 1110
t110
Well
mall
111
1136
Uie
111
Uile
Lite
L114
1
111
013e
Li1e
well
xell
1
we11
Mell
well
well
well
1
9611
Mel1
wall
well
(a) ASCII
('player': 12 7. 'target': [6, 2]
(b) Coordinates
(c) RGB
Figure A6. Sample observations from our grid world environment for all three state representation formats. For the RGB observations
(Fig. A6c), we use the sprites from Crafter (Hafner, 2022) for the walls, the player, and the target (the floor is a black squarc).
observations. Note that, in theory, the observation type is irrelevant for this task since the models could just count the number of observations in the evaluation trajectory and select the corresponding action in the demonstration trajectory.
C.3. Illegal Actions
As described in Section 2.4, we do not terminate the evaluation episode early if a model does not produce valid action (except on the crossword task) but instead randomly sample one of the actions that are legal in the current state and continue the evaluation with the observation produced by that action. To differentiate illegal actions from acting randomly (over legal actions), we compute how often models actually propose an illegal action and visualize the percentage of illegal actions per episode over the number of expert demonstrations Figs. A15 to A20 (analogous to Figs. 3 to 8). Note that, since we perform an ablation over showing the legal moves the prompt (see Appendix C.4), in theory, all models could have the necessary information to sample legal action all times (whether we actually show this or not in our main experiments depends whether it improves model performance in the ablations). Nevertheless, models do produce illegal actions actions across most environments. For example, on the grid world navigation task with coordinate observations ol -preview produces almost 100% illegal actions with 2 or more expert demonstrations in the context. Interestingly, the ability to produce legal actions also depends on the observation type: For tic-tac-toe, all models mostly produce legal actions with ASCI observations, but the Gemini 1.5 models generate up to ~ 60% illegal actions for RGB observations from the same environment.
observe that
21

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Grid World (12x12)- ASCII
Grid World (12x12) ASCII
1.0
100
Average % of Illegal Actions
0.8
80
Claude 3.5 Sonnet
Average Score
GPT-4o
0.6
01-mini
ol-preview
0.4
Gemini
0.2
20
Gemini Exp.
0.0
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
Maximum Sample Length
Maximum Sample Length
(a) ASCII observations
Grid World (12x12)- Coordinates
Grid World (12x12)- Coordinates
1.0
100
Average of iliegal Actions
Claude 3.5 Sonnet
Average Score
GPT-4o
0.6
60
o1
al Ol-mini
0.4
40
al-preview
Gemini 1.5 Pro
Gemini 1.5 Flash
0.2
Gemini 2.0 Rlash Exp.
27
2
2
2
2
2
2
2
25
Maximum Sample Length
Maximum
Sample Length
(b) Coordinate observations
Grid World (12x12) RGB
Grid World (12x12) RGB
Average % of Illegal Actions
100
0.8
80
Average Score
Claude 3.5 Sonnet
0.6
60
GPT-4o
o1
Gemini 1.5 Pro Gemini 1.5 Flash
Gemini 2.0 Rash Exp.
0.2
20
0.0
23
2
27
2
2
21
21
222
2
21
2
23
26
2"
2a
2a
210
2
2
2
2
235
Maximum Sample Length
Maximum Sample Length
(c) RGB observations
Figure A7. Ablating the maximum sample length for all three observation types (ASCII, coordinates, and RGB images) of the grid world navigation task with 1 demonstration episode (the left panels show the average score, the right panels show the percentage of illegal actions). As expected, 32 output tokens are sufficient for Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Flash, and Gemini 1.5 Pro to achieve their best performance on the task. In contrast, ol ol-mini and ol -reevvew require between 1096 and 8192 output tokens achieve their best performance. With less than 8192 output tokens, the ol models do not have enough internal "reasoning tokens" at their disposal to produce an output, which ol -preview are text-only models and therefore cannot process RGB image observations.
seen by the sharp increase the percentage of illegal actions in right panels. Note that -ini and
22

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Atari Phoenix - RGB
GPT-69 Gemin Flash Gemri 1lash
10
16 20
36
36
40
42
44
45
45 50
52
56
64
86
66
70
72
74
76
76
80
82
84
86
88
90
92
94
96
98
Stop
0.8
1.0
Action Replay Accuracy
Figure A8. Replaying the demonstration episode for the Phoenix game (with RGB observations) from Atari 2600. The color visualizes the models' average accuracy when attempting to replay the action for given step. The Gemini 1.5 models generally perform well at replaying the demonstrations, regardless of the step, though they show slight degradation in performance towards the 80-step mark, from which they immediately recover again. The performance of GPT-4o shows a slightly higher degradation towards step 80, but also immediately recovers thereafter. Note that Claude 3.5 Sonnet cannot process more than 100 images time, cannot evaluate it on this task. Similarly, ol-mini and ol ol-preview are text-only and, therefore, cannot process RGB observations.
Table Ablating the use chain-of- thtougghtt prompting and whether or not to show legal actions in the prompt for Phoenix game from Atari 2600. For this lask, all models perform best without legal actions and without chain-of-thought
Model
Observation
Legal Actions
Chuin-of Thoun-uf-Thought
Averuge Scure
False
42.83
False
True
6.16
Gemini 1.5 Flash
RGB
True
True
6.46
False
21.86
False
True
8.68
Gemini 1.5 Pro
RGB
False
6.33
Thue
6.21
False
31.80
False
True
11.40
Gemini 2.0 Flash Exp.
RGB
False
15.20
Trie
True
10.20
False
Truc
5,40
GPT-4o
RGB
False
6.40
True
True
5,40
False
29,00
False
True
19.20
ol
RGB
23.00
True
Truc
16.40
C.4. Hyperparameter Ablations
As mentioned in Section 3, we ablate the use of chain-of- of-though prompting (Wei et al., 2022) and whether or not to include the list of actions that legal given the current observation in the prompt for each model-task combination since we want to report the best- best-possible performance the models results in Tables Al to A10. We highlight the best score for each model-obserrvtion pair in bold and use the corresponding hyperparameter combination for our sweeps over the number of expert demonstration episodes in Section 3.2.
attain on benchmark. For completeness, we present all the ablation
C.4.1. INCLUDING PAST ACTIONS
Over the course our experimental investigation of frontier models' performance in dynamic environments, we noticed that these models have tendency of getting stuck into repeating their previous action. We therefore conduct an ablation where we do not show the past actions of the evaluation trajectory the prompt (the past observations remain in the prompt, as do the observations and actions of the demonstration episode). Tables All to A13, A15 and A16 show the performance for all frontier models the grid world navigation task with expert demonstration episode with and without the past actions in the prompt. In general, all models benefit from having the full history in the prompt (i.c.. both the previous observations and actions), even if that allows them to repeat their previous action more easily (in most environments they can still infer the previous action from the two past observations). The only exception is Claude 3.5 Sonnet (see Table All), which does not show clear trend terms of including the past actions. Given these results, we decide to always include the past actions in the prompt for all models and environments in all our other experiments.
23

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Chess Steckfish (aavel 6: mede)-AScI ASCii
diude 3.5 Sonnut
GPT-46
Gemini Rlah Exp
Stop
0.8
Action Replay Accuracy
ASCII observations
Chess ws. Stockfish (Level 0: 1 mode) FEN
Sonnat
GF1-46
53-mal
olproveww
Gem 1.5
Gemhii
20
35
35
Stop
3.0
0.2
0.4
5.6
0.5
100
Avg. Action Rep bay Accuracy
(b) FEN observations
Cbude
6l-mil
61 proview
Sem ml 0.5 Pro
Gemini 2.5 Hash
Gemini Meah
Aug. Rep lay Accuracy
(c) PGN observations
Stockfish RGB
Cbude 3.5 Sonnat
CP7-4o
Gemisi Rosh Exp
Stop
3.0
0.6
0.8
Avg. Action Rep by Accuracy
(d) RGB ohservations
Figure A9. Replaying the demonstration episode for the different observation types from chess environments. The color visualizes the models' average accuracy when attempting to replay the action for a given step. All models generally perform well across all observation types, except ol -ini, which shows strong performance degradation towards end of the episode across all three observation text observation types (recall that ol mini and ol -preview are text-only models, and, therefore, cannot process RGB images). With the exception Claude Sonnet, models struggle (to varying degrees) to replay the few actions PGN observations.
24

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Crewwerd (17) (x7)) ASC
67T46
63
Gamii 15 Tash
10
21
12
0.4 Action Roplay Accurwoy
0.6
14
Figure A10. Replaying the demonstration episode for our crossword environment with ASCII observations. The color visualizes the models' average accuracy when attempting replay the action for given step, All models generally perform well, except for ol ol-mini, which completely fails at this task. Moreover, Claude 3.5 Sonnet, Gemini 1.5 Flash, and Gemini 1.5 Pro struggle to replay the first step.
Control:
Run Coordinates
GPT
provice
Ge Tiash Gentri flash
10
14
16
18 20 22 24
28
28
30
56
38
40
42
44
45
48 50 Step
56
64
86
68
70
72
76
78
80
82
84
86
90
92
94
96 98
0.4
0.6
0.8
1.0
Aug. Action Replay Accuracy
Figure AIL. Replaying the demonstration episode for the cheetah run task from the DM Control suite (with coordinate observations). The color visualizes the models' average accuracy when attempting replay the action for given step. GPT-4o, Gemini 1.5 Flash, and Gemini 1.5 Pro generally perform well. In contrast, ol-mini and ol ol-preview significantly struggle with this task. Claude 3.5 Sonnet shows patches of slightly poorer performance around steps 32, 64, and 96, but performs well otherwise.
Table A2. Ablating the use of chain-of-thought prompting and whether or not to show legal actions in the prompt for Claude 3.5 Sonnet on chess. For this task, Claude 3.5 Sonnet always profits from having the legal actions the prompt. thain-of-thouglle prompting improves performance on FEN and RGB observations, hut not on ASCII and PGN observations.
Model
Observation
Legal Actions
Chain-of-
Average Score
False
-0.98
False
True
-0.99
ASCI
False
40.88
True
True
False
+096
False
True
0.99
FEN
False
.0.9
True
Claude 3.5 Sonnet
True
40.55
(2024- 10-22)
False
-0.92
False
True
-0.97
PGN
False
+0.82
True
Thue
.0.66
False
-0.96
False
True
-0.99
RGB
False
-0.4
True
True
41.1
25

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Clubk Sar
11
Actian Heplay Auconic
Adian boalle
(a) ASCII observations
(b) Coordinate observations
Clace
37T-40
Greis Raan
(c) RGB observations
Figure A12. Replaying the demonstration episnde for the all observation types from grid world navigation task. The color visualizes the models' average accuracy when attempting to replay the action for a given step. All models generally perform well across all observation types, except for ol-mini ASCII observations. Moreover, the Gemini 1.5 models struggle step 12 with ASCHI observations, and ol-mini struggles with the last step for coordinate observations. Note that ol-mini and ol-preview are text-only models and, therefore, cannot process RGB observations.
26

---

LMAct: A Benchmark for In-Contess Imitation Learning with Long Multimodal Demonstrations
Table A3. Ablating the use thain-of-thought prompting and whether or not to show legal actions the prompt for Gemini 1.5 Flash, Gemini 1.5 Pro, and Gemini 2.0 Flash Experimental on chess. For this lask, the models almost always profit from having the legal actions in the prompt. Chain-of-thought prompting sometimes improves performance, but not across the board.
Model
Observation
Legal Actions
Chain-of- Thought
Average Scure
False
-1.00
False
True
ASCII
False
+0,98
True
-0.99
False
-0.99
False
True
-0.99
False
-0.99
Thie
Gemini 1.5 Flash
False
-0.88
False
PGN
Thue
-0.99
False
-0,96
True
True
-0.97
False
-1.00
False
-1.00
RGB
False
-0.97
True
True
-1.00
False
-1.00
False
True
-0.99
ASCII
False
-0.96
True
True
-0.97
False
Truc
-1.00
FEN
False
.0,95
True
True
-0.97
Gemini 1.5 Pro
False
-0.93
False
True
.0.4
PGN
-0,90
True
Thie
-0.97
False
-1.00
False
RGB
Falsc
-0.99
True
True
-0.97
False
ASCII
True
0.99
False
.0.93
True
False
-0.95
False
True
-0.99
FEN
True
True
-0.97
Gemini 2.0 Flash Exp.
False
-0.87
False
PGN
False
-0.91
True
.0.96
False
True
0.99
False
-0.95
True
27

---

LMAct: A Benchmark for In-Contest Imitation Learning with Long Multimodal Demonstrations
Table A4. Ablating the use of chain-of- thought prompting and whether or not to show legal actions in the prompt for GPT-4o, ol-mini, ol ol-preview, ol on chess. this task, the models always profit from having legal actions in the prompt. Chain-of thauuht prompting sometimes improves performance, but not across the board.
Model
Observation
Legal Actions
Chain-of-Thouuge
Average Score
False
-1.00
False
True
-1.00
ASCII
False
-0.44
True
-0.92
False
-0.94
False
True
-0.98
FEN
False
-0.98
True
True
-0.90
GPT-4o
False
-063
False
Truc
-0.97
PGN
False
-0.88
True
True
-0.93
-0.90
False
True
-0.99
RGB
False
-0.84
True
True
-0.91
False
-0.99
False
True
-0.99
ASCII
False
-0.96
True
True
-0.96
False
-1.00
False
True
-1.00
ol-mini
False
-0.89
True
Truc
-0.92
False
-0.99
Fulse
Thue
-0.97
PGN
False
-0.84
True
True
-0.83
-0.99
Fulse
True
-0.99
ASCII
-0,93
True
True
-0.94
False
-0.99
Fulse
True
-0.99
ol --previeww
FEN
False
-0.95
True
True
-0,89
False
-0.98
False
True
-0.98
False
-0.95
True
-0.90
False
-0.98
Fulse
True
-0.98
ASCII
False
-0.85
True
True
-0.87
-0.95
False
True
-0.89
FEN
-0.82
True
True
-0.87
ol
False
-0.80
False
True
-0.84
PGN
False
-0.85
True
True
-0.79
False
.0.95
False
True
-0,97
RGB
False
-0.82
Truc
True
-0.8
28

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
(P74)
Closke Sannd
Genrii 15
Gonani
Garln Di..
Push Exp.
Step
Stap
2.A
0.0
6. 5.
1.5
Ang. Action Replay Accurncy
Aug. Action Rop by Accurscy
(a) ASCII observations
RGB observations
Figure A13. Replaying the demonstration episode for the all observation types from our tic-tac-toc enviromment. The color visualizes the models' average accuracy when attempting to replay the action for given step. All models generally perform well across all observation types except for Gemini 1.5 Flash on RGB images, where it fails completely. Claude 3.5 Sonnet and Gemini 1.5 Pro also struggle (to varying degrees) with the step (step 0) for RGB observations. Note -ini ol -preview are text-only models and, therefore, cannot process RGB image observations.
Table A.5. Ablating the use of chain-of-thougle prompting for the task of solving crosswords. Chain-of-thought prompting sometimes improves performance, but not across all models. For this task, showing legal actions is infeasible (as it would include all possible (i.c., thousands of) words of the correct length for every slot), do not ablate it.
Observation
Legal Actions
Chaln-of- Thought
Average Score
Claude 3.5 Sonnet
False
6.73
(2024-10-22)
ASCII
False
Thue
6.81
Fulse
1.99
Gemini 1.5 Flash
ASCII
False
True
Gemini
False
Talse
3.15
True
5.50
False
3.70
Gemini 2.0 Flash Exp.
ASCII
False
True
3.98
False
5.76
GPT-4o
ASCII
False
True
6.67
al-mini
ASCIT
False
Truc
4.82
False
9.57
al prevveww
ASCII
False
False
9.55
ASCII
False
True
9,56
29

---

LMAct: A Benchmark for In-Contess Imitation Learning with Long Multimodal Demonstrations
GPT-4o Atari Phoenix
Acari Phoenix
Flac. Elements Episode
Frec Elements per Episode
0.4
0.25
0.3
0.25
0.20
0.1
0.00
0.0
20
40
60
80
100
20
40
60
80
Repeated Action Subsequence Length
Repeated Action Subsequence Length
(a) GPT-40
ol
Gemini Flash
Atari Phoenix
Gemini 1.5 Atari Phoenix
Gemini Exp.
Prac. Elements Episede
0.6
Frac. Epicode
0.5
Frac Elements per Episode
0.8
0.5
0.4
0.4
0.6
0.4
0.2
0.2
0.1
0.1
0.2
Repeated Action Subsequence Length
Repeated Action Subsequence Length
Repeated Action Subsequence Length
(c) Gemini 1.5 Flash
(d) Gemini 1.5 Pro
(c) Gemini 2.0 Flash Experimental
Figure A14. Fraction of elements per repeated action subsequence length in the Phoenix game from Atari 2600 (without demonstration episodes). To compute these results, we partition each evaluation episode into segments consisting of the same action generated consecutively. The segment length across all time steps in all evaluation episodes that fall into each segment length. All three models have many episodes (e.g.. more than 60% for Gemini 1.5 Flash) where they repeat same action throughout almost the entire episode. Accordingly, they fire very rarely thus achieve low score (cf. Fig. 3), since, in the Phoenix game, in order to fire repeatedly, the firing button needs to be pressed and released repeatedly - constantly holding it down (i.e., repeating the previous action) only results in single shot.
the x-axis, and the height of each bar the fraction of actions (out of the total number of actions)
Atari Phoorix (RCB)
80
Gemini 1.5 Flash
Average of
Gemini 2.0 Flash Exp.
40
GPT-40
22
22
24
Exp ert Demonstration Episodes
Figure A15, Average percentage illegal actions per episode over number expert demonstrations for Phoenix game (RGB observations) from Atari 2600. All models mostly generate legal actions. Note that o -imi and ol op-rvvv are text-only and, thus, cannot process RGB observations.
30

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Chess vs. Stackfish (Level D: node) (ASCII)
Chess Stockfish (Level D: node) (FEN)
Average % iilllggg Actions
100
Aver % illlgall Actions
100
80
Gemiri 1.5 Flash
80
Gemini 1.5 Rlssh
60
Gemiri 2.0 Flash Exp.
60
Gemini 2.0 Flash Exp.
GPT-4g
GP7-4o
40
llprreviev
o1-preview
20
20
20
2
22
2
2
2
2
2
24
2
2
2
2
2
2
4 of Expert Demonstration Episodes
Expert Demonstration Episodes
(a) ASCII observations
(b) FEN observations
Chess vs. Stockfish (Level 0: node) (PGN)
Chess vs. Stockfish (Level 0: node) (RGB)
Average of llgggal Actions
100
Aveeaage of Illepll Actions
100
Gemiri 1.5 Flash
Gemini 1.5 flash
Gemini 2.0 Flash Exp.
GPT-4
Gemini 20 Flash Exp.
GPT-4o
01-preview
2 22
23
2 21 22
2
2
# of Expert Demonstration Episodes
# Expert Demonstration Eplsodes
(c) PGN observations
(d) RGB observations
Figure A16. Average percentage of illegal actions per episode over number expert demonstrations all state representation formats from our chess environment. GPT-4o, ol-mini, and ol-preview rarely generate illegal actions (except for GPT-4o with PGN observations). In contrast, the Gemini 1.5 models consistently struggle to generate legal actions across all state representation formats and with higher rates of illegal actions with increasing numbers of expert demonstrations in the context. Note that ol ol-mini and ol o1-preview text-only models and therefore cannot process RGB image observations.
Crosswords (7x7) (ASCII)
DM Control: Cheetsh Run (Coordinates)
Average of llleqel Actions
100
Actions
80
Genini Flah
80
Gemini 2.0 Exp.
Gemini 1.5 Pro
60
Genini 1.5 Pro
Genini Flash Exp.
CPT40
40
a1-mirl
preview
Average
al-greview
a1
29
22
20
24
28
27
2
Expert Demonstration Episodes
Expert Demonstration
Figure A17. Average percentage of illegal actions per episode the number of expert demonstrations for crossword vironment with ASCII observations. All models produce high percentage of illegal actions (hetween 40% and 90% of all the actions in an episade).
Figure A18. Average percentage of illegal actions episode over the number of expert demonstrations for the cheetah run task from the DM Control suite with coordinate observations. models, except ol-mini, consistently produce mostly legal actions.
31

---

LMAct: A Benchmark for In-Contess Imitation Learning with Long Multimodal Demonstrations
(12x12) (ASCI))
(12x12) (Coordinatess
Average of Illegaa Actione
100
Average of Illegaa Actions
Gemiri 1.5 Flash
Gemini 1.5 flash
Gemini 1.5 Pro
Gemini 1.5 Pro
Flash GPT-4o
Flash
GP7-4o
61-mini
61-mini
01-preview
ol-preview
01
# of Expert Demonstration Episodes
Expert Demonstration Episodes
(a) ASCII observations
(b) Coordinate observations
(RGB)
Averoge % of illegaa Actions
100
Flash
Geminii 1.5 Pro
2.0 Flash
GPT-4o
2-
27
Expert Deronsttation Episodes
(c) RGB observations
Figure A19. Average percentage of illegal actions per episode over the number of expert demonstrations for all state representation formats form our grid world navigation task. The ol models increasingly struggle to produce legal actions with increasing numbers of expert demonstrations in the context, with ol- ol-preview having very high percentage of illegal actions. In contrast, GPT-4o, Gemini 1.5 Flash, and Gemini 1.5 Pro consistently produce legal actions. Note that ol-mini and ol ol-preview are text-only models and therefore cannot process RGB image observations.
vs. (ASCII)
Random (RGB)
Average 36 of illegal Actions
100
Average of Illegaa Actions
Flash
Gemini 1.5 Pro
Gemini 1.5 Pro
GPT-4o
Gemini Flash
01-mini
GPT-4o
o1
2a
25
27
25 Expert Demorstration Episodes
Expert Demonstration Episodes
(a) ASCII observations
(b) RGB observations.
Figure A20. Average percentage of illegal actions per episode over the number of expert demonstrations for all state representation formats from our tic-tac-toc environment. With ASCII ohservations, all models mostly produce legal actions. In contrast, for RGB image observations, the Gemini 1.5 models struggle to produce legal actions (Gemini 1.5 Pro independently of the number of expert demonstrations and Gemini 1.5 Flash increasingly with the number of expert demonstrations). Note that ol ol-mini and ol orepreveew are text-only models and therefore cannot process RGB image observations.
32

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A6. Ablating the of chain-of-thought prompting and whether simulating cheetah from the DM Control Suite. For this task, most models do not benefit from chain-of-thought prompting or showing the legal actions in the prompt (note there are infinitely many legal actions which we represent with the following string:
show legal actions the prompt for the task of
A comma-separated 1ist (enclosed by square brackets) of values between -1 and 1. ').
Model
Observation
Legal Actions
Chain-of- Chuungof-Thought Averuge Scure
False
4,70
Claude Sonnet
Coordinates
Falsc
True
3.53
(2024-100 10-22)
False
10.20
Truc
Thie
1.33
Talse
3,93
False
Gemini 1.5 Flash
Coordinates
Truc
False
0.39
True
2.31
False
True
1.26
Gemini
Coordinates
False
7.69
Truc
Talse
15,96
True
1.91
Gemini 2.0 Flash
Coordinates
13.53
True
True
1.85
False
11.03
False
GPT-4o
Coordinates
False
1.00
True
True
0.92
False
Talse
0,80
Thue
0,46
ol-mini
Coordinates
False
0.78
True
True
False
3.16
False
Thie
1.04
ol-preview
Coordinates
False
4.08
True
Truc
False
2.08
ol
Coordinates
Trie
2.14
True
Truc
1.84
Table Ablating the use chain-of-thought prompting and whether or not to show legal actions prompt for Claude 3.5 Sonnet the grid world navigation task. For ASCII and coordinate observations, the performance is largely independent of the ablated settings. RGB images, showing the legal actions using chain-of-thought prompting performs best.
Model
Observation
Legal Actions
Chain-of-Thoughe
Average Score
False
1.00
False
True
1.00
ASCII
False
True
Tiue
False
0.96
False
Claude 3.5 Sonner (2024-10-22)
Coordinates
Thie
1,00
False
0.99
True
True
False
True
0.40
RGB
False
0.21
True
True
33

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A8. Ablating the of chain-of th-uuuhhttthht prompting and whether show legal actions the prompt for Gemini 1.5 Flash. Gemini 1.5 Pro. and Gemini 2.0 Flash Experimental on the grid world navigation task. The best- best-performing setting is model and observation dependent.
Modell
Observation
Legal Actions
Chuin-of- Thouuht
Averuge Scure
False
True
0.08
False
0.11
True
Truc
0.15
False
0.21
False
True
0.10
Gemini Flash
Coordinates
True
True
0.08
False
0.16
False
RGB
True
False
0.14
True
0.12
False
Thue
0.25
False
0,51
True
False
0,27
False
True
0.13
Gemini
Coordinates
True
True
0.12
False
False
0.17
RGB
False
0.24
True
0.24
Fulse
True
0.41
ASCII
False
0.25
True
True
False
False
Thie
0.22
Gemini Exp. Coordinates
False
0,30
True
Thue
0.16
False
0.18
Falsc
True
0.37
RGB
False
0.23
Thie
0.43
34

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A9. Ablating the chain-of- th-thhtgtt prompting and whether or not show legal actions the prompt for GPT-4o, ol ol-mini, ol ol-preview, and ol the grid world navigation task. The best-perforning setting is model and observation dependent. Note that ol-mini and ol-preview are text-only and, therefore, cannot process RGB observations.
Model
Observation
Legal Actions
Chain-of-Thooee Thought
Average Score
False
False
True
0.57
ASCII
False
0.50
True
True
0.49
False
True
GPT-4o
Coordinates
False
0.76
True
True
0.72
False
0.52
False
True
0.70
RGB
False
0.56
True
True
0.61
False
0.20
False
True
0.22
ASCII
False
0.21
True
Truc
0.32
ol-mini
False
0.92
Fulse
Thue
0.79
Coordinates
False
0.87
True
True
0.80
0.94
Fulse
True
ASCII
True
True
ol oprrvve
False
0.99
Fulse
True
1.00
Coordinates
False
0.99
True
True
1.00
False
1.00
False
True
1,00
ASCII
False
1,00
True
True
1,00
ol
False
0.99
Fulse
True
1.00
Coordinates
False
1.00
True
Thue
1.00
False
0.81
True
0.86
RGB
True
35

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A10. Ablating the use of chain-of-thought prompting and whether show legal actions in the prompt the game of tic-tac-toe. Across almost all models and observations types, using chain-of-though prompting and showing the legal actions achieves the highest performance often by large margin. Note that ol-mini and ol-preview are text-only and, therefore, cannot process RGB
observations.
Model
Observation
Legal Actions
Shaln-of-Thoughe
Average Score
False
0.10
False
True
0.19
ASCII
False
0.14
Claude 3.5 Sonnet (2024-10-22)
Trie
False
0004
False
Thue
0.16
RGB
False
True
True
0.19
False
-0.12
False
ASCII
True
.0.5
False
-0.00
True
Thue
0.00
Gemini Flash
False
0.07
False
True
RGB
False
.0.5
Trae
True
0.10
False
Truc
-0.07
ASCII
False
.0.02
True
Gemini 1.5 Pro
False
0.08
False
True
0.06
RGB
True
Truc
0.01
False
0.06
False
ASCII
False
0.10
Gemini 2.0 Flash Exp.
True
0.18
False
True
0.07
False
0.06
True
False
0.01
False
Thue
0.15
ASCII
True
True
0.20
GPT-4o
False
0.00
False
RGB
False
0.15
Thue
0.17
False
0.25
False
True
0.26
ol-mini
ASCII
Trae
False
0.32
True
0.45
False
Truc
0.24
ol-preview
ASCII
False
0.35
True
True
0.36
False
0.58
False
True
0.58
ASCII
True
True
0.66
False
0.38
False
RGB
False
0.52
True
0.60
36

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table AIL. Ablating whether to include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) for Claude 3.5 Sonnet on our grid world navigation task. Models are prone repeating the previous action, omitting it from the prompt could alleviate this problem. For ASCII observations, Claude 3.5 Sonnet is mostly indifferent to having the actions prompt. For coordinate observations, Claude Sonnet performs better with the actions observations, the past actions deteriorate performance.
prompt, while for RGB
Model
Observation
Legal Actions
Chain-of-Thougee
Past Actions
Average Score
False
False
True
False
False
True
True
1.00
ASCII
False
1.00
False
True
True
False
1.00
True
False
0.77
False
True
0.96
False
0.84
Claude Sonnet (2024-10-22)
True
1.00
Coordinates
False
0.77
False
True
True
True
True
0.99
False
0.26
False
True
0.19
False
False
0.47
Tuue
True
0.40
RGB
False
False
True
0.21
True
False
0.68
True
True
0.53
37

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A12. Ablating whether include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) for Gemini 1.5 Flash on our grid world navigation task. Models are prone to repeating the previous action, omitting it from the prompt could alleviate this problem. this case, including the full history of the current trajectory, i.e., both the observations and the actions, almost always improves the performance of Gemini 1.5 Flash.
Model
Observation
Legal Actions
Chain-of-Thoushh
Past Actions
Average Score
False
True
0.09
False
False
0.12
True
True
0.08
ASCII
False
0.09
False
True
0.11
Trme
False
0.10
True
True
False
True
0.21
False
Falsc
0.07
True
True
0.10
Gemini 1.5
Coordinates
False
0.10
True
0.14
True
False
0.08
True
True
False
False
False
False
Tuu
True
0.12
RGB
False
0.10
False
True
0.14
True
False
0.09
True
True
0.12
38

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A13. Ablating whether include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) for Gemini 1.5 Pro our grid world navigation task. Models are prone to repeating the previous action, so omitting from the prompt could alleviate this problem. Without exception, including the past actions in the prompt always improves the performance of Gemini 1.5 Pro.
Model
Observation
Legal Actions
Chain-of-Thone Thought
Past Actions
Average Score
False
0.41
False
True
0.51
False
False
0.22
True
True
ASCII
False
False
True
0.51
Tiue
False
True
True
0.36
False
0.07
False
True
0.27
False
False
0.06
True
True
0.13
Gemini
Coordinates
False
0.02
False
True
0.16
Fulse
0.08
True
True
0.12
False
0.14
False
True
0.17
False
False
0.14
True
True
0.25
RGB
False
0.16
False
True
0.24
True
False
True
39

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A14. Ablating whether include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) for Gemini 2.0 Flash Experimental on our grid world navigation task. Models are prone to repeating the previous action, omitting from the prompt could alleviate this problem. this case, including the full history the current trajectory, i.c. both the observations and the actions, almost always improves the performance of Gemini 2.0 Flash Experimental.
Model
Observation
Legal Actions
Chain-of-Thougee
Past Actions
Average Score
False
0.19
True
0.27
False
True
True
ASCII
False
0.17
True
True
Tuue
True
0.39
False
0.07
False
True
False
True
Geminn Flash Exp.
Coordinates
True
0.22
False
0.06
False
True
True
True
0.16
False
0.25
False
False
False
0.36
True
True
0.37
RGB
Fulse
0.30
False
True
Tree
False
0.38
True
0.43
40

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A15. Ablating whether to include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) GPT-4o on our grid world navigation task. Models are prone to repeating previous action, so omitting it from the prompt could alleviate this problem. Including the past actions in the prompt always achieves the highest performance.
Model
Observation
Legal Actions
Thoughof-Thought
Past Actions
Average Score
False
0.28
False
True
0.56
False
False
0.47
Truc
True
0.57
False
0.36
False
True
0.50
True
False
0.43
True
True
0.49
False
0.28
Falsc
True
0.69
False
0.46
True
True
0.73
GPT-4o
Coordinates
False
0.21
False
True
0.76
True
False
0.48
True
True
0.72
False
False
True
0.52
False
False
0.51
True
True
0.70
RGB
False
0.51
False
True
0.56
True
False
0.61
True
True
0.61

---

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations
Table A16. Ablating whether include the past actions of the evaluation trajectory in the prompt (the actions of the demonstration episodes are always included) for ol-mini, ol ol-preview, and ol on grid world navigation task. Models are prone to repeating the previous action, omitting from the prompt could alleviate this problem. In vast majority of cases, including past actions the evaluation prompt results in the best performance for ol-mini, ol-preview, and ol. Note that ol-mini and ol orrrreveww are text-only models and, therefore, cannot process RGB images.
Model
Observation
Legal Actions
Chain-of- Thought
Past Actions
Average Score
False
0.18
False
True
False
Tiue
True
ASCII
False
0.24
True
0.23
True
False
0.29
Tiue
True
ol-mini
False
False
True
0.68
False
False
0.40
True
0.72
Coordinates
False
False
True
True
False
0.44
True
True
0.71
False
0.23
False
True
False
False
Tiue
True
ASCII
False
0.30
True
0.42
True
False
0.27
True
True
ol preview
False
False
True
0.13
False
False
0.14
True
0.13
Coordinates
False
0.11
False
True
Thee
False
True
True
0.13
False
1.00
False
True
False
False
True
True
ASCIT
False
1.00
False
True
1.00
True
False
1.00
The
True
False
True
0.99
False
False
0.98
True
1.00
ol-mini
Coordinates
False
0.98
False
True
True
False
True
True
1.00
False
0.60
False
True
False
False
True
True
0.86
RGB
False
0.65
True
0.84
True
False
0.58
True
True
42


Reviews:
[{"review_id": "ua7d9ksotX", "reviewer": "Reviewer_KyTi", "summary": "This paper benchmarks the decision-making ability of several frontier multimodal models in interactive environments through in-context imitation. It investigates whether these models can be effectively prompted with few- or many-shot demonstrations to solve interactive tasks. The overall finding is that most frontier models fail to reach expert-level performance through prompting, and their performance remains independent of the number of demonstrations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and interesting questions. We are pleased that they think that our `claims are clear and supported with convincing evidence`, that our `experiments are well designed and presented`, and that our `results are an important sanity check of the current capability of frontier models for interactive decision making`.\n\n\n**Why do more demonstrations have little and sometimes negative impact?**\n\nAt least in theory, our fully observable tasks do not require an optimal policy to attend to information more than one step in the past. Accordingly, an LM’s performance on our tasks must lie somewhere between two extremes:\n* If the LM has stored the optimal policy in its weights, then any historical information would only inform the model about the task at hand and which of its in-weights policies to select (for which even a single observation might suffice).\n* If the LM has no suitable pretrained policies in its weights and, therefore, would have to rely purely on distilling the expert behavior in the context, more demonstrations should increase performance.\n\nIn practice, the LMs’ behavior lies somewhere between these extremes and, as our results show, greatly varies per model and task. Possible “mechanical” explanations for this behavior are that LMs are pretrained to attend sparsely over their context (in which case more observations would only help marginally), have a recency bias (which would make attention beyond the current/previous episode even less likely), and are usually not explicitly trained for in-context learning from examples (although it’s a somewhat emergent capability of pre-training).\n\nWe mostly observe negative impact, i.e., “in-context interference” for Cheetah Run and gridworld navigation with ASCII observations. In both cases, our analysis of the percentage of illegal actions (Figures A18 and A19) suggests the problem is not due to an increased number of illegal actions but, instead, increasingly suboptimal actions. A definitive answer to what causes in-context interference would require a thorough investigation, which is beyond the scope of this paper (which introduces the benchmark), but we have added a brief discussion of these observations and open questions to the paper. Please also see the summarized findings of our investigations of failure modes (mostly appendix results) in our response to reviewer `46g3`, under “Could you include more analysis about failure modes and potential improvement strategies?”.\n\n\n**How should we change the training of these frontier models to be more effective with in-context prompting when solving interactive decision-making tasks?**\n\nWhile we do not have a definite answer, it seems plausible that pretraining or finetuning with data from interactive decision-making tasks, and, in particular, in-context imitation of an expert policy, would be quite effective. Another promising direction could be to focus on models capable of general in-context reinforcement learning, which is a bit different than our in-context imitation setting, but in principle, all our tasks could easily be extended by providing additional reward observations. We consider these very interesting and important directions for future research and have updated our manuscript with the above discussion.", "claims_and_evidence": "All claims are clear and supported with convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "Yes, they are related to in-context imitation of decision making with LLMs. The finding here is in line with the finding in [1] that zero-shot LMs cannot perform vision-based decision-making effectively in interactive environments. \n\n[1] BALROG: BENCHMARKING AGENTIC LLM AND VLM REASONING ON GAMES", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "While the experiments seem simple, they are well designed and presented. Even though the results are not too surprising, they are an important sanity check of the current capability of frontier models for interactive decision making.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How do you make sense of more demonstrations having little and sometimes negative impact? \n2. How should we change the training of these frontier models so that they are more effective with in-context prompting at solving interactive decision-making tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NnKSEU6V46", "reviewer": "Reviewer_YX5H", "summary": "The authors created a benchmark for empirical evaluation of the multimodal in-context imitation learning capabilities of some state-of-the-art LLMs (Claude 3.5 Sonnet, Gemini 1.5 Flash, Gemini 1.5 Pro, Gemini 2.0 Flash Experimental, GPT-4o, o1-mini, o1-preview, and o1) on several of interactive decision-making tasks: playing tic-tac-toe, chess, and Atari, navigating grid worlds, solving crosswords, and a DM Control task.\nThe authors show that even when optimizing the prompt (number of demonstrations, chain-of-thought prompting, etc.) for each model and task - frontier LLMs fail to reach expert performance on Atari, chess, and DM Control. Some models approach expert performance on crosswords, grid world, and tic-tac-toe. All models beat the random action baselines except on Atari.\nThe authors vary the number of expert demonstration episodes in the context from 0 up to 512 (the limit depends on the model and the task) and find that performance is mostly independent of the number of demonstrations. In some cases we observe strong in-context learning.\nAuthors run a control experiment where LLM agents need to replay the single demonstration episode in the context, where all models except for o1-mini perform well.\nThe authors plan to make their benchmark publicly available.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review, constructive feedback, and pointers to additional relevant related work. We are pleased that they think that `the soundness and validity of the experimental design are beyond doubt` and that `the originality of the work arises from creative combinations of existing environments to study problems in a long context`.\n\n**Why did you not evaluate the two other state-of-the-art models QWEN and Llama 3?**\n\nWe had the dilemma of choosing a set of models to evaluate and ultimately settled on *8* models but did not mean to imply that any models outside of this set are not state-of-the-art. Unfortunately, we were unable to evaluate Llama 3 due to licensing issues, which is why we decided to focus on closed-weights models (which we explicitly stated in Section 2.1). Nevertheless, we fully agree that both QWEN and Llama 3 would be interesting additions to our benchmark and have discussed them in our revised manuscript. Since the landscape of state-of-the-art models is rapidly evolving, we open-sourced our benchmark to enable the community to evaluate QWEN, Llama 3, and any other future models using our approach.\n\n\n**Why did you not evaluate transformers created for long-context tasks (such as RMT and RATE) on your benchmark?**\n\nAs the reviewer points out, RMT and RATE are specialized transformer models that were specifically developed for long-context tasks. However, as stated in Section 1, our goal is to evaluate *state-of-the-art LMs* in dynamic environments. Nevertheless, we agree that innovations around augmenting transformers with recurrency (and forms of more explicit retrieval) are very interesting approaches to potentially overcome LMs’ performance limitations in our benchmark, and we have expanded the related work section to discuss these approaches. Similar to QWEN and Llama 3, we are excited to see how the community will leverage the open-source nature of our benchmark to improve their models.\n\n\n**What is the practical usefulness of this benchmark?**\n\nWe thank the reviewer for raising the important point regarding practical usefulness and transferability. While the LMAct environments are simpler than complex real-world scenarios like robotics, they require fundamental agentic capabilities (long-context multimodal understanding, in-context imitation learning) that are often tested in robotics research and likely necessary for real-world success. LMAct serves as a controlled testbed to evaluate these core skills, diagnose current model limitations, and guide research toward more generally capable agents. Although direct transfer is beyond this paper's scope, these findings can inform future work addressing that challenge. We have added a brief discussion clarifying this focus in the revised manuscript.\n\n\n**Why did you not compare your benchmark to BabiLong?**\n\nThe BabiLong benchmark is relevant in the wider context of our work, and we have added a brief discussion in the related work section. Thanks for the pointer!", "claims_and_evidence": "The authors created a benchmark that tested some existing Large Language Models, but the existing SOTA models also include QWEN [1], LLAMA [2].\nThe benchmark is aimed at studying models on a long context, but the authors did not compare their approach with another existing benchmark that allows studying large language models on long contexts – BabiLong [3].\n\n[1] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., ... & Zhu, T. (2023). Qwen technical report. arXiv preprint arXiv:2309.16609\n[2] Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... & Vasic, P. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.\n[3] Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024). Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems, 37, 106519-106554.", "methods_and_evaluation_criteria": "The work is devoted to the creation of a large-scale benchmark, in general, the estimates that it allows to obtain are advective and can be applied to the assessment of large language models in the future. At the same time, the practical usefulness of this benchmark raises questions, because if the model successfully solves typical game problems given in the benchmark, then its portability for controlling complex real intelligent agents operating in a real environment (for example, robotic agents).", "theoretical_claims": "The work does not contain theoretical novelty, the main provisions of the work are related to obtaining empirical and practical results.", "experimental_designs_or_analyses": "The soundness and validity of the experimental design are beyond doubt. At the same time, the lack of experiments with some other SOTA models (QWEN, LLAMA, etc.) is a drawback.", "relation_to_broader_scientific_literature": "Key contributions of the paper are related to the broader scientific literature. The authors provided a fairly comprehensive analysis of existing approaches.", "essential_references_not_discussed": "The article lacks references to articles devoted to other SOTA LLMs: QWEN [1], LLAMA[2].\nThere is no reference to the benchmark for studying models on a long context Babilong [3].\nThere is no mention of specialized transformer models, specially developed for similar tasks for working on a long context: RMT [4], RATE [5].\n\n[1] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., ... & Zhu, T. (2023). Qwen technical report. arXiv preprint arXiv:2309.16609\n[2] Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... & Vasic, P. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.\n[3] Kuratov, Y., Bulatov, A., Anokhin, P., Rodkin, I., Sorokin, D., Sorokin, A., & Burtsev, M. (2024). Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems, 37, 106519-106554.\n[4] Bulatov, A., Kuratov, Y., & Burtsev, M. (2022). Recurrent memory transformer. Advances in Neural Information Processing Systems, 35, 11079-11091.\n[5] Cherepanov, E., Staroverov, A., Yudin, D., Kovalev, A. K., & Panov, A. I. (2023). Recurrent action transformer with memory. arXiv preprint arXiv:2306.09459.]", "other_strengths_and_weaknesses": "The originality of the work arises from creative combinations of existing environments for studying problems on a long context\n\nThe main weakness of the work is the insufficient completeness of comparison with existing LLMs, as well as the insufficient applicability to real-world problems.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "I would like to know from the authors whether it is possible to test transformer models created for solving LongContext tasks such as RMT[1], RATE[2] on their benchmark and what their quality indicators would be?\n\n[1] Bulatov, A., Kuratov, Y., & Burtsev, M. (2022). Recurrent memory transformer. Advances in Neural Information Processing Systems, 35, 11079-11091.\n[2] Cherepanov, E., Staroverov, A., Yudin, D., Kovalev, A. K., & Panov, A. I. (2023). Recurrent action transformer with memory. arXiv preprint arXiv:2306.09459.]", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0E1qK2UGfr", "reviewer": "Reviewer_46g3", "summary": "The paper presents a benchmark to evaluate the capabilities of today’s frontier models on multimodal decision making task in the very long context regime. The paper investigates the in-context learning abilities of these models. The authors compare a variety of the latest multimodal LM models on tasks like chess, crossword, and grid world, among others.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful assessment and constructive feedback. We are pleased that they think our `paper addresses an important problem`, our `paper covers a wide range of tasks`, and our `paper considers a wide variety of frontier models, making for a comprehensive study`.\n\n**Could you include more analysis about failure modes and potential improvements?**\n\nWe agree that one of the most important follow-ups is to analyze why models fail on our benchmarks and how to address these failures. We have already conducted a substantial set of such experiments in Appendix C, including:\n* Checking whether models can successfully “replay” (or copy) an episode, which most models can (Appendix C.2)\n* Checking to which degree bad performance can be attributed to the percentage of illegal vs. legal but suboptimal actions (Appendix C.3)\n* Investigating the fraction of repeated actions in Atari – Phoenix, which reveals that models have a high tendency to repeat the previous actions, which leads to few shots being fired, explaining why the performance is worse than the random action baseline (Figure A14)\n* Investigating whether failures can be attributed to various hyperparameter settings, e.g., checking whether showing legal actions alleviates the problem of having to infer the correct action format from the demonstrations (Appendix C.4)\n\nThe trends of these investigations are complex (often model and task-dependent), and definitive answers would require sophisticated further analysis, which we consider outside the scope of our benchmark paper. We have added a compact discussion of this analysis to Section 4 in the updated manuscript.\n\n**Do the o1 models perform better because they use a larger context length (8192 tokens) than the other models (2048 tokens)?**\n\nSection 2.1 states that the o1 models use a larger *output sample length* (i.e., the maximum number of tokens to generate, including the “thinking tokens” for o1), not a larger context length. The (input) context length is the same across all models. We ablated the maximum sample length in Figure A7 and showed that only the o1 models benefit from a larger sample length. This is because the o1 models cannot finish their reasoning traces if they do not have sufficient “thinking tokens” at their disposal. Since our goal is to evaluate the models’ best possible performance, we set the output sample length to 8192 for the o1 models (which is expensive) and 2048 for all other models (since this is cheaper and does not affect their performance).\n\n**Do you also provide the rewards from the demonstrations?**\n\nNo, we only perform in-context imitation learning, so we do not provide reward information to the models. We only use the rewards to compute the model scores for our benchmark results. We consider in-context reinforcement learning an interesting direction for future work.\n\n**Are the 100 steps per episode sufficient for most tasks? Can this number be attributed to the limited context length of current models?**\n\nYes, for chess, crossword puzzles, gridworld navigation, and tic-tac-toe, 100 steps are more than sufficient to complete an episode (e.g., for chess, the average number of steps per game is 38 — see Section 2.2). However, for Atari and DM Control, we would ideally want to evaluate more steps (e.g., we only evaluate roughly 6 seconds of Atari play — see Section 2.2), but the context size limitations of current frontier models (32 Atari demonstrations episodes with 100 steps already hit the 1M token limit) do not allow this evaluation. We chose a maximum of 100 steps to be consistent across tasks while still providing a meaningful numerical signal (e.g., the models already fail in the early stages of Atari).\n\n**Why are there a lot of illegal actions in chess and crosswords? Could adding the rules of the game to the context be helpful?**\n\nThe action space for these two tasks is more complex than for the other tasks (maybe except for Cheetah Run, which most models seem to be able to match, though). Adding game-specific explanations would probably help increase the performance, but our study is to investigate task-agnostic in-context learning behavior since such game-specific behavior may not always be available for real-world tasks (e.g., videos of human experts performing tasks in dynamic environments).\n\n**Why does the performance degrade with increasing demonstrations for Cheetah Run?**\n\nWe do not know with certainty. Figure A18 shows that it cannot be attributed to an increased number of illegal actions, i.e., the models’ actions are increasingly suboptimal rather than illegal. A definitive answer to what causes in-context interference would require a thorough investigation, which is beyond the scope of this benchmark paper, but we have added a brief discussion of these observations and open questions to the paper.\n\nWe hope our responses and revisions effectively address the reviewer’s concerns and clarify the paper's contribution.", "claims_and_evidence": "The paper investigates the performance of the latest LM models in very large context settings and studies the effect of the number of in-context demonstrations, different observation encoding schemes, and chain of thought prompting on the performance of these model. While the authors provide results for each, some insights about how to improve these models on the tasks being studied would be valuable for the community.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Going through the paper at large, the experimental design seems sound. However, it would be great if the authors could include more analysis explaining the results and potentially provide pointers for improving the current models.", "relation_to_broader_scientific_literature": "The results provided in the paper are valuable to the community for understanding the performance limitations of the latest frontier models. As mentioned above, more analysis of the results would be valuable to the community. For instance, since a lot of tasks have performances much lower than the expert performance, an analysis of common failure modes along with potential improvement strategies would be useful.", "essential_references_not_discussed": "Being a research in a closely related but not the same field, the references seem adequate to me.", "other_strengths_and_weaknesses": "Strengths\n- The authors address an important problem of evaluating the latest frontier models on a series of long-context tasks.\n- The paper covers a wide range of tasks including games (Atari, chess, tic-tac-toe), grid environments (grid world, tic-tac-toe, crosswords), and physical simulators (DMC cheetah run).\n- The paper also considers a wide variety of frontier models for the comparisons, making for a comprehensive study.\n- The paper also studies the effect of different observation encoding strategies, number of in-context demonstrations, and chain of thought prompting.\n\nWeaknesses\n- It would be great if the authors could include more analysis about failure models and potential improvement strategies to further advance the current state of these frontier models.", "other_comments_or_suggestions": "- From section 2.1, the o1 models use a larger context length (8192 tokens) than the other models (2048 tokens). Could this be the reason why they perform better than the other models on platforms like Crosswords and Tic-Tac-Toe?\n- Are rewards from the demonstrations also given as input in the context? If yes, are they given as per-step rewards or as an accumulated value (similar to value functions in reinforcement learning)?\n- Are 100 steps in an episode enough for most tasks? What about tasks that require more steps (for example in robotics where an episode could go on for a longer time)? Also, how did the authors come up with 100 steps? Can it be attributed to the limited context length of the current models?\n- Can the authors provide an intuition of why there are a lot of illegal actions in chess and crosswords? Could adding the rules of the game in context be helpful in this case?\n- It would be great if the authors could provide an insight into why the performance degrades with an increasing number of demonstrations in DMC cheetah run.", "questions_for_authors": "It would be great if the authors could address the questions from the previous section and in Weaknesses. I would be happy to raise my score after these questions are addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oTUoWsutVy", "reviewer": "Reviewer_7Vdn", "summary": "LMAct is a benchmark designed to evaluate the multimodal in-context imitation learning capabilities of state-of-the-art, closed-source large multimodal foundation models (LMs). LMAct systematically evaluates model performance over extremely long-context inputs, testing how effectively these models utilize a varying number of expert demonstrations (spanning multiple orders of magnitude, up to context saturation). The benchmark includes six interactive decision-making tasks of varying complexity: Atari (Phoenix), Chess, Crosswords, DM Control (Cheetah Run), Grid World navigation, and Tic-Tac-Toe. Results suggest that the current state of the art multimodal LMs do struggle on the benchmark and in many cases, do not benefit from the additional context/demonstrations that are being provided.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment and insightful feedback. We are pleased that they think that our `paper is particularly relevant to the current trend of reasoning models` and that `it is great that this paper identifies areas where people can evaluate their reasoning models on longer horizon tasks`.\n\n**The paper overlooks the relevant line of work on in-context learning for control.**\n\nIndeed, the papers the reviewer listed are relevant to the broader context of our work, and we have expanded the related work section to discuss all of these references. While the methods explored in that exciting line of work focus on developing novel agents with strong in-context learning capabilities, our paper's primary goal is to benchmark the current state of existing, general-purpose frontier LMs on such tasks. Understanding how to best leverage insights from specialized agent research to enhance these large, pre-trained models remains an open question, which underscores the importance of establishing clear benchmarks like LMAct.\n\n\n**The paper only has 6 environments, which could lead to quick saturation given the current speed of AI research.**\n\nAs discussed in Section 4, we designed our benchmark such that it can easily be made more difficult as the capabilities of frontier models evolve. While our benchmark is currently in its “easiest” form (e.g., grid world without obstacles, chess against Stockfish level 0 (less than 1300 Elo)), our results demonstrate that even this version reveals significant limitations in current frontier models. Therefore, we believe that our benchmark provides a strong and high-resolution signal to measure the progress of frontier models. Once our benchmark is saturated (which is far from being the case right now), we can easily construct a more challenging v2.\n\n\n**Consider including partially observable environments to expand the benchmark.**\n\nWe tried to cover a range of environments that are relatively easy for humans but where current models still struggle to reach expert performance. As a result, we chose fully observable tasks (technically, Atari requires at least two frames to determine the velocities) since they have simple, reactive policies and do not require exploration.\n\nPartially observable tasks require consistent integration of information across several potentially non-adjacent time steps, which is a great test of a model’s ability to densely attend to the information in the context. Moreover, there is the theoretical problem of self-delusions in imitation of an expert that has hidden information (e.g., the hidden belief state of the agent), which is necessary under partial observability (see Ortega et al. (2021)). We want to keep this benchmark free from these complicating issues and think that the right time to move to such harder tasks is when frontier models easily solve simple, fully observable tasks at expert level (we are currently at beginner level — see the previous response). Independent of that, we agree with the reviewer that it is great to see benchmarks on interactive decision-making tasks under partial observability (perhaps better suited for in-context RL, which avoids the self-delusion problem) and refer to, e.g., the BALROG benchmark.\n\nWe have added a brief discussion to acknowledge the value of partially observable benchmarks while clarifying our focus on fully observable tasks in the revised manuscript.\n\n*Shaking the foundations: delusions in sequence models for interaction and control.\nPedro Ortega et al.\nhttps://arxiv.org/abs/2110.10819.*\n\n\n**Does the temperature affect the in-context learning capabilities?**\n\nWe deliberately set the temperature to 0 to ensure the reproducibility of our results wherever possible (the API for the o1 models does not support this option). Whether the temperature also affects the performance is an interesting direction for future work. Note that since OpenAI’s models perform relatively well in most of our tasks, the effect of the temperature value does not seem to be very significant in our benchmark.", "claims_and_evidence": "Believe so.", "methods_and_evaluation_criteria": "This is a benchmark paper. The paper can benefit from more substantial compare and contrast with prior LLM benchmarks, especially those designed for multi-turn agents. \n\nThe paper also makes an effort to compare different multimodal representations of the state (i.e. ASCII and RGB observations). However, \n\nThe test contains 6 environments, which may not be broad enough and potentially can be saturated quickly. In addition, all environments are fully observable. It would be beneficial to 1) expand the number of environments and 2) introduce more partially observable environments.", "theoretical_claims": "N.A.", "experimental_designs_or_analyses": "Since this is a benchmark paper, see the methods and evaluation criteria section above.", "relation_to_broader_scientific_literature": "I think this is particularly relevant to both the current trend of reasoning models and agents. It is great that this paper identifies areas/tasks where people can evaluate their reasoning models and language models on longer horizon tasks and seeing many of the current LMs not being able to in-context learn them allows the community to explore further in allowing better CoT + reasoning capabilities.", "essential_references_not_discussed": "The paper seems to overlook the exploration that researchers have done in the direction of in-context learning for control. State-based ICL has a long history (which has been applied to some of the environments listed in 2.2). See the line of work following One-Shot Imitation Learning [1] and Prompting Decision Transformer [2]. Multimodal (i.e. image-action-proprio) ICL for control using next-token prediction / autoregressive models have either a trained version (ICRT [3]) or are using pre-trained LLMs (Moka [4] prompt a robot to walk [6] and RoboPrompt [5]). \n\n[1] Y. Duan et al., “One-shot imitation learning,” Advances in neural information processing systems, vol. 30, 2017.\n\n[2] M. Xu et al., “Prompting decision transformer for few-shot policy generalization,” in international conference on machine learning, PMLR, 2022, pp. 24 631–24 645. \n\n[3] F. Letian et al. \"In-context imitation learning via next-token prediction.\" arXiv preprint arXiv:2408.15980 (2024).\n\n[4] K. Fang, F. Liu, P. Abbeel, and S. Levine, “Moka: Open-world robotic manipulation through mark-based visual prompting,” Robotics: Science and Systems (RSS), 2024.\n\n[5] Yin, Yida, Zekai Wang, Yuvan Sharma, Dantong Niu, Trevor Darrell, and Roei Herzig. \"In-Context Learning Enables Robot Action Prediction in LLMs.\" arXiv preprint arXiv:2410.12782 (2024).\n\n[6] Wang, Yen-Jen, Bike Zhang, Jianyu Chen, and Koushil Sreenath. \"Prompt a robot to walk with large language models.\" In 2024 IEEE 63rd Conference on Decision and Control (CDC), pp. 1531-1538. IEEE, 2024.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "Q1. I believe ICL for control is an exciting direction. Many prior works have worked towards using autoregressive models for ICL, including but not limited to the ones in the Essential References Not Discussed section. Adding appropriate discussion to these methods can help readers find better footing in the field. \n\nQ2: The benchmark currently only has 6 environments. As mentioned earlier, the performance may saturate really quickly given the current climate of LLM research. Maybe consider expanding the benchmark or introducing harder tasks (i.e. partially observable instead of fully observable environments). \n\nQ3: (Exploratory, less relevant to my evaluation of the paper) In all experiments, it seems that the temperature is set to zero except for the openai models. Does temperature affect ICL capability? Or does it just introduce higher variances?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a benchmark for studying the decision-making abilities of multimodal models as policies in long-context tasks, e.g., game-playing or robot control. Authors find that current models perform poorly on these tasks out-of-the-box. The paper also includes experiments to identify the utility of in-context imitation learning and different observation representations for getting models to perform better, and find some mixed results. This benchmark and its analysis would be a valuable contribution to the community. I suggest the authors to incorporate the feedback provided by the reviewers, particularly when it comes to related work and failure analysis.
Decision: Accept (poster)