Vintix: Action Model via In-Context Reinforcement Learning
Authors: Andrei Polubarov
Lyubaykin Nikita
Alexander Derevyagin
Ilya Zisman
Denis Tarasov
Alexander Nikulin
Vladislav Kurenkov
Venue: ICML 2025 poster (2025)
Keywords: in-context reinforcement learning
Primary area: reinforcement_learning->batchoffline
TL;DR: A cross-domain action model pre-trained directly for In-Context Reinforcement Learning

Abstract: In-Context Reinforcement Learning (ICRL) represents a promising paradigm for developing generalist agents that learn at inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models. These findings highlight the potential of ICRL as a scalable approach for generalist decision-making systems.

Vintix: Action Model via In-Context Reinforcement Learning
Andrei Polubarov - 123 Nikita Lyubaykin 14 Alexander Derevyagin 15 llya Zisman 123
Denis Tarasov
Alexander Nikulin 16
Vladislav Kurenkov 14
Abstract
Normalized Score IQM %
Zero-Shot ICRL
Many-Shot ICRL
In-Context Reinforcement Learning (ICRL) rep- resents a promising paradigm for developing gen- eralist agents that learn inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scal- ing ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a frame- work designed facilitate ICRL,, offers pelling and competitive alternative expert distil- lation to construct versatile action models. These findings highlight the potential of ICRL. as seal- able approach for generalist decision- decision-making sys- tems. Code to be released at dunnolab/vintix.
100
-74.0
+25.0
-6.0
+20.0
80
60
40
52-shoe
O-shot
8-shot
0ishot
13-shot
0-shot
11-shot
20
0-shot
Bi-DexHands
MujoCo
Industrial
Meta-World
Benchmark
Figure I. Vintix: Self-Correction with Many-Shof ICRL. Many- shot ICRL consistently self-corrects online the training tasks (87) nearing the demonstrators performance across four domains. Optimal number of shots for many-shot ICRL shown inside the bar for each task. One shot corresponds to one online episode. See Section 3 for more details and comparisons action models based on expert distillation.
2019; Levine et al., 2020).
In contrast to online reinforcement learning, a recent boom of generative models, revitalized by Large Language Models and their emergent properties (Brown et al., 2020), sparked high amount of interest in leveraging offline data for RI. training: from specialized offline RL algorithms (Levine et al., 2020; Tarasov et al., 2024) to Action Models capable of handling many cross-domain tasks at the same time (Reed et al., 2022: Gallouédec et al., 2024; Haldar et al., 2024; Collaboration et al., 2024; Schmied et al., 2024; Sridhar et al., 2024).
1. Introduction
The pursuit of generalist control and decision makinng agents has long been a major target of the reinforcement learning (RL) research community (Sutton & Barto, 1998). These agents are envisioned to handle diverse set of tasks while and
exhibiting adaptation properties such as self-correcte self-improvement based on reward functions. Notably, tradi- tional online RL algorithms demonstrate these properties in narrow domains and can achieve exceptional performance (Berner al., 2019; Badia al., 2020; Schrittwieser al., 2020; Baker al., 2022; Team al., 2023). However, their online nature of learning and frequent reliance environment-specifee training is still considered challenge to overcome limiting the scalability (Dulac-Arnodd et al.,
Current approaches to pre-training cross-domain action models broadly fall into two categories. The first utilizes all available data, conditioning policies on return-to-ge tar- gets in a framework inspired by upside-down reinforcement learning principles (Schmidhuber, 2019; 99; Lee et al., 2022; Schmied et al., 2024). The second, and currently dominant, approach prioritizes expert demonstrations, often disregard- ing reward signals entirely, exemplified by Reed et al. (2022) and Haldar et al. (2024).
Eqqual contribution 'AIRI *Skoltech 'Research Center for Trusted Artificial Intelligence, ISP RAS "Innopolis Univer- sity 'HSE "MIPT. Correspondence to: Vladislav Kurenkov <kurenkov @ neti. Work done by dunnolab.ai.
While the field increasingly steers toward language and demonstration-guiiee paradigms over reward-centric ap- proaches, the "Reward is enough" principle (Silver et al., 2021) continues to offer foundational framework for
Proceedings of the = International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Vintix: Action Model via In-Context Reinforcement Learning
Noise Distillation
Random Demonstrator
Re4 = (fog, of, " (d o Annd
M
(-1-11))
Tasks
On
01
O2
OL-3
L--2
O
G1
a2
a.
a
Domains
71
FL
T.
(n) MasE
Causal Transformer
a
a
az
a
Dosss- Dataset
Algorithm Distillation
Figure Vintix: Approach Overview, Stage (Noise Distillation)- approximating policy improvement trajectory by injecting gradually annealed uniform noise (see Algorithm 1). Stage (Cross-Domain Dataset) combining collected multi-ppisodic sequences into shared cross-domain dataset for subsequent model training. Stage (Algorithm Distillation) runniing AD (Laskin al., 2022) with collated {s, as,a,) r) triplets collected dataset.
agents that learn maximize arbitrary rewards through trial and error. This perspective has inspired methods like Algorithm Distillation (Laskin et al., 2022), which opera- tionalizes reward-driven learning by training agents via next- token prediction over historical reinforcement learning (RL) trajectories - mechanism that directly enables In-Context Reinforcement Learning (ICRL). Though initial implemen- tations of this framework, alongside extensions such as by Zisman al. (2024a;b); Sinii et al. (2024); Nikulin al. (2024), have been demonstrated only toy and grid-based tasks within single domains, their data-driven origin posi- tions them as promising avenue for scaling reward-centric agents to broader, more complex environments.
four domains (Meta-World, MuJoCo, Bi-DexHands, Benchmarkk), along with data collection tools and instrumentation to support the development of ac- tion models eliciting ICRL behavior.
Cross-Domain Scaling of ICRL (Section 3): We pirically demonstrate that the proposed model, Vin- tix, can self-correct attain demonstrator-level per- formance on training tasks (Figure 1) and adapt to controlled parametric task variations at inference-time.
2. Approach
At the core of approach (Figure 2) is Algorithm Dis- tillation (Laskin et al., 2022), two-step Offline Meta-RL algorithm. The first step involves collecting ordered training histories from base reinforcement learning (RL) algorithms, while the second step involves a decoder-only transformer trained solely for the next-action prediction. This approach facilitates in-context learning by effectively distilling the policy improvement operator into causal sequence model. We further propose two augmentations this technique: (1) democratizing the data collection process by introducing continuous extension of the noise-distillation procedure by Zisman et al. (2024a); (2) conducting generalist agent-style cross-domant training on the acquired dataset.
In this work, we start an investigation into cross-domain action models through In-Context Reinforcement Learning (ICRL), grounded in the Algorithm Distillation framework (Laskin et al., 2022). By focusing proach to ICRL (summarized in Figure 2), we present Vin- tix, multi- task action model that exhibits initial signs of inference- time celf-correction on training tasks (Figure 1) and preliminary evidence of adaptation to parametric task variations (Section 3.3).
data-centric ap-
Our key contributions follows:
Continuous Noise Distillation (Section 2.1): We pro- pose data collection strategy extending the work of Zisman et al. (2024a) to continuous action spaces to ease the collection of training data.
2.1. Continuous Noise Distillation
Collecting learning histories can be time-consuming and computationally expensive, as it requires curated learning histories of RI. algorithms for each task individually. The
Open Tools and Datasets for ICRL (Section 2.2): We publicly release datasets for 87 tasks across
2

---

Vintix: Action Model via In-Context Reinforcement Learning
resulting trajectories may be excessively long due to poor sample efficiency and may exhibit noise due to training in- stabilities. Recently, Zisman et al. (2024a) demonstrated that learning trajectories approximated through noise dis- tillation can facilitate the emergence of in-context learning capabilities via next-action prediction. this framework, called AD policy improvement is approximated by gradu- ally reducing the proportion of random noise injected into the demonstrator policy during its execution in the environ- ment. More specifically, each time step, a random action is selected with probability while demonstrator action is chosen with probability 1 -. é is annealed throughout the trajectory, starting at e = (random policy) at the beginning and gradually decreasing to e = 0 (demonstrator policy) by the end.
Bi-DexHands
Industrial Industrial-Bnnchmark Benchmark
Normalized Return IQM (95% CI)
0.0
Meta-World
MuJoCo
The AD data collection strategy originally designed for discrete action spaces. In our work, we propose its extension to continuous action spaces, where the resulting action is defined as linear mixture of uniform random noise and demonstrator actions. This contrasts with Brown et al. (2019), who employed an epsilon-grees -greddy approach that al- ternates between fully random and fully expert actions with probability e. The pseudo-code for our noise-distillation procedure is formalized in Algorithm 1.
0.0
0.5
1.0
0.0
0.5
1,0
Trajectory Share
Figure 3. Continuous Noise Distillation trajectories. Aggregated normalized returns the collected cross-domain dataset. Returns normalized with respect to random and demonstrator scores, while trajectory lengths are reported as fraction their maximum values.
Algorithm 1 Noise distillation for continuous action spaces
2.2. Pross-Dommin Dataset
Require: Demonstrator policy TD. task environment, noise schedule E, number of time steps in the trajectory T,, tra- jectory buffer D, action space lower and upper bounds
Building upon Continuous Noise Distillation method, we then collect a cross cross- dosain dataset. Our dataset consists of environments with continuous N-dimensional vector ob- servations and continuous multi-dimensional actions. This focus was chosen to isolate the challenge of processing multi-modal inputs from the challenge of inference-time adaptation across different tasks and domains, allowing us to concentrate the latter. The collected eross cross-domain dataset consists of 87 distinct tasks spanning across four domains:
Omin: Omar
1: Sample Su from task environment
2: for E T do
3: Noise magnitude: E = Et)
4: Noise:
Uniiformm(minn amax)
5: Current action: a; (1 4) (s;) + c, *
6: Obtain (8,41,7t,4,) by executing 0; in task environ- ment
7: Append (s,,0;, (,, 84,,, to D
8:
end for
1. MuJoCo (Todorov et al., 2012) classical multi-joint dynamics control suite containing 11 various tasks with different state-action spaces and reward funcions.
The noise schedule E plays crucial role in the generation of training trajectories. We observed that linear annealing of с often results in non-smooth trajectories with abrupt transitions from random to demonstrator performance. This effect can negatively impact the model's convergence; there- fore, careful tuning of the noise schedule was necessary for certain tasks. A more detailed description of the epsilon decay functions is provided in the Appendix A.3. Figure 3 illustrates that Algorithm 1 generates trajectories with smooth reward curves, closely resembling the learning his- tories observed in Zisman et al. (2024a). For Task-Level dataset visualization please refer Appendix E.
2. Meta- World (Yu et al., 2021) benchmark for Multi- Task and Meta RL containing 45 manipulation tasks with shared state and action structure.
3. Bi- Bi-DexHHands (Chen et al., 2022) benchmark simula- tor that includes 15 diverse tasks focused on dexterous manipulation.
4. Senutriil--Behmmarkk (Hein et al., 2017) bench- mark featuring synthetic continuous control tasks that simulate the properties of real industrial problems.
3

---

Vintix: Action Model via In-Context Reinforcement Learning
Train Dataset
2.3.1. MODEL ARCHITECTURE
Sample Weight
Domain
Tasks
Episodes
Timesteps
At high level, Vintix consists of three main components: an encoder, which maps raw input sequences & into a fixed-size embedding space; a transformer backbone, which processes the encoded inputs; and a decoder, which maps the hidden states produced by the transformer back to the original action space.
Bi-DexHands
15
216k
31,7M
14,2%
Industrial-Be mnlustrial-Rnncharrk Benchmark
16
96k
24M
10.8%
Meta- Meta-World
45
670k
67M
30,1%
MuJoCo
11
665k
100M
44.9%
Overall
87
1,6M
222,7M
100 %
Table Cross-Domain Dataset summary. Aggregation is performed by summing all transitions (in the form of {st, Ot, St-1; Ti; tt across all trajectories collected for all tasks. Detailed dataset statistics can be found ad Appendix G
All input sequences 2 are split into groups based on observation and action space dimensionalities. For each group, a separate encoder and decoder MLP head are cre- ated, enabling the model to map variable observation and action spaces into a shared embedding space. It is important to note that the model is task-agnostic in the sense that it has access only to the dimensionality- based group identifier, but not to an individual task identifier. While the task identifier is unique ID assigned to each task in the dataset, the group identifier simply indicates whether a of tasks shares the same observation and action space dimensions, as well as the semantic meaning of each channel.
Table contains detailed statistics of the training dataset. More detailed information on datasets can be found in the Appendix A.
Training Demonstrators For MuJoCo and Meta-Worid we utilized demonstrator policies provided in JAT (Gal- louédec et al., 2024). However, for several tasks within Meta-World, we trained our own demonstrators due to the unsatisfactory performance of the provided experts. Bi- DexHands demonstrators were trained using official PPO (Schulman et al., 2017) implementation with an increased number of parallel environments, The demonstrator poli- cies for the Industrial Benchmark were trained using the provided scripts, which were built on top of the Stable- Baselines 3 library (Raffin al., 2021). In-depth view of demonstrators performance can be found in the Appendix H.
In contrast to Laskin et al. (2022), which processes each separate token, stack
entity in the sequence h the representations of the previous action, previous reward, and current observation into a single sequence token. This approach is consistent with token alignment methods pro- posed by Duan et al. (2016); Grigsby et al. (2024a;b). Such design choice allowed us to significantly expand the context window size by compressing its representation by factor of three.
In summary, Vintix is diction model with 24 layers, 16 heads, an embedding size of 1024, and a post-attention feed-forward hidden size of 4096. TinyLLama (Zhang et al., 2024) was chosen as the transformer backbone for the Vintix model.
next-action pre-
2.3. Training and Inference Pipeline
The input data consists of a set of multi-episodic sub- sequences sampled from the original moise-distilles trajecto-
2.3.2. TRAINING
ries, formalized as follows:
HEt+L = (fof, ab, (?),. af),a.
(m)
(f "t:i):...................
The input batches are created by collating together multiple input sequences (m)) h To standardize the data across tasks, rewards are scaled by task-specific factors (see Appendix G), and observations are normalized. This procedure sig- nificantly enhances model performance by reducing task distinguishability based on raw input values, thereby en- couraging the model to rely on contextual information to infer the task.
(0 M "142-))) Where t is the index of the sub-sequence's starting point within the full noise- distilled trajectory, L is the length of the sub-sequence, M denotes the total number of episodes within the sub- sequence, which varies due to differing episode lengths across tasks, E [o, L - 1] is a lower subscript timestamp index within the sub-sequence, j E [0, M - 1] is an up- per subscript episode index within the sub- sequenee. The global subscript n identifies a unique task, which is uni- formly sampled from multi-domain dataset, defined as: Mn. = - M where d E [1, DJ represents the do- main identifier, and na denotes the task belonging to the respective domain. The overall data pipeline closely resem- bles that of Laskin et al. (2022), with the only difference being its cross cross-domain coverage.
Afterward, sequences are processed by the corresponding en- coders to obtain fixed-size representations. The sequence of representations is then passed into transformer, whose out- puts are subsequently decoded into predicted actions apred- The training objective is to minimize the Mean Squared Er- ror loss between the predicted actions Gpred and the ground- truth trajectory actions arge- Training is conducted on 8
4

---

Vintix: Action Model via In-Context Reinforcement Learning
H100 GPUs with batch size of 64 and 2 gradient accu- mulation steps. The input sequence length L is set 8192. For more detailed hyperparameter information, refer to Ap- pendix D.
normalized performance of Vintix with that of Gallouédec et al. (2024) and Sridhar al. (2024) across training tasks in overlapping domains (MuJoCo and Meta-World). JAT and REGENT scores are taken directly from the respective papers. For Vintix, we extract the average performance over 100 episodes after reaching inference-time convergence (i.c., achieving the best k-shot performance per task).
2.3.3. INFERENCE
Model inference is performed iteratively, starting with an empty context. The model generates the first action based on the initial observation and subsequently receives the next ob- servation and reward. Each new transition tuple is appended to the context. When the context exceeds its maximum length L, the oldest token is removed, effectively imple- menting a sliding attention window (Beltagy et al., 2020). To accelerate inference, we utilized the KV-cache imple- mentation from the FlashAttention library (Dao et al., 2022). Since absolute positional encodings are incompatible with sliding attention window with K KV-caching due positional drift, we employed ALiBi encodings (Press et al., 2022), which are inherently relative in nature.
Since most of our demonstrators were sourced from JAT and REGENT also adopted random and expert scores from JAT the comparison of normalized scores is valid. How- ever, we improved expert performance for several tasks by fine-tuning hyperparameters and normalized our raw returns with respect to these newly enhanced scores. This adjust- ment lowers our normalized performance relative to JAT and REGENT, but we adopted this comparison approach as our initial focus is on evaluating the agent's 's ability to match demonstrator performance rather than comparing absolute scores.
Figure 5 illustrates that inference-ting self-correction en- ables Vintix to outperform JAT and REGENT on Meta- World and MuJoCo by significant margins. More detailed comparison can be found in Appendix I. These results may further highlight a fundamental advantage of Algorithm Dis- tillation over Expert Distillation, namely its adaptability, as initially reported in Laskin et al. (2022).
3. Results
3.1. Inference-Time Self-Correction on Training Tasks
First, we aim to verify whether the Vintix model has the ca- pability for context-based inference-time adaptation through self-correction. To achieve this, we deploy the model training tasks (ML45 split for Meta-World, ML20 split for Bi-DexHands and setpoints E [0, 75] for Industrial- Benchmark) by iteratively unrolling its actions in a cold-start manner, beginning with an empty initial context. Figure 4 illustrates that the model progressively improves its policy in each domain as the number of shots (episodes played) increases. The agent starts with suboptimal performance and gradually self-corrects by inferring task-related infor- mation from the accumulated context, ultimately reaching near-demonnstratrr-ledd performance.
3.3. Generalization Analysis
In this subsection, we take a closer look at the models per- formance in regards to both unseen parametric variations and tasks.
Generalization to Parametric Variations
Further experiments aim to assess whether the Vintix model is capable for context-based inference-time adaptation to task variations that were not encountered during training.
To evaluate this, we performed the cold-start inference procedure described in previous sections on a set of Mu- JoCo environments with unobservable variations in viscos- ity and gravity, which were also unseen during training (for more details on MuloCo parameter variations, refer to Ap- pendix 1.2.1). Additionally, we applied the same procedure to unseen environments in the Benhhharrl-Behhhaakkkk domain (setpoint E [80, 100), see Appendix A.2.4).
Notably, this improvement is both progressive and consis- tent along the shots-made axis and is observed across multi- ple domains with varying dynamics and morphology. For lesel graphs depieting inference-time time performance with an empty context, refer to Appendix F.
This behavior suggests that, despite being task-agnostic, the model can infer environment's implicit structure and utilize it to enable self-corrective adaptation across training tasks.
Figure 6 presents the experimental results. As shown, Vin- tix still able to achieve near-demonstrator performance in modified environments across both domains. However. a slight decline in convergence quality is observed- convergence speed in MuloCo and subtly diminished asymp- totic performance in Industrial-Bencet Benchmark. The slower convergence rate suggests that the model requires more iter- ations of self-correction to reach demonstrator- policy
3.2. Comparison to Related Action Models
Next, we aim to assess the performance of Vintix compared to other generalist agents trained across multiple domains and determine whether its self-corrective inference provides an advantage in matching demonstrator performance levels.
To verify this, we compare the average demonstrator-
5

---

Vintix: Action Model via In-Context Reinforcement Learning
Bi-DexHands
MujoCo
Industrial Benchmark
Meta-World
Normalized Score IQM (95%
1.20
1.10
1.02
1.05
0.99
0.99
0.92
0.90
0.99
0.95
0.90
0.80
0.60
0.96
0.85
0.40
0.70
0.94
0.80
0.60
0.75
0.20
0.50
0.92
0.70
0.00
0.40
0.90
0.65
29
21
2
2
2
2
20
2
21
22
2
2
2
2
2
2
21
2
2
2
21
2
2
2
Number of Shots
Figure 4. Dynamic self-correction with Many-Shot ICRL. The model's actions are executed iteratively in the environment without any initial context task identifier provided the agent (i.e., cold start). Although it starts with suboptimal policy, the model gradually improves through context-based self-correction. Results are aggregated over training tasks (MI.45 split for World, MI.20 split for Andustrial-enncharkk Benchmark) within each domain.
Bi- DexHands and setpoints p [0, 75] for Industrial
Normalized Score IQM (95% CI)
Vintix
REGENT
JAT
MuJoCo
Industrial Benchmark
Normal Lizaaiized.o Score IQM in %
1.10
1.05
100
0.99
0.99
0.90
0.95
80
0.90
0.70
0.85
40
0.50
parsmeter varistion
Irain
0.80
trais
parameter carbation
0.75
20
2
2
22
23
2
2
2
2
2
25
Number of Shots
MuJoCo
Meta-World
Figure 6. Cold-start many-shot inference procedure on tasks with parameter variations compared to training tasks. In MuloCo, vari- ations include changes viscosity (0.05 0.1 vs. original 0) and gravity (#: 10%). For the Industrial Benchmark, Industrial-Bncchmarkk the model on previously unseen setpoint values, p E [80, 100).
Figure Average domain-level demonstrator-nomeet normalized returns training tasks: Vintix REGENT JAT. JAT and REGENT scores are taken directly from the respective papers, while for Vintix, analyze 100 episodes after reaching inference-time convergence.
evaluate
in environments with moderate parametric variations.
scenarios, the model exhibited random-level performance on the Bin-Picking task and performed below the random baseline on the Hand- Kettle task.
Generalization to New Tasks
Finally, we evaluate Vintix's performance on entirely new tasks that were seen during training to determine whether it can perform in-context reinforcement learning by inferring task structure in this challenging setting.
This observation aligns with our intuition and existing re- search in in-context RL., where such those studied by Laskin et al. (2022); Sinii et al. (2024); Zisman et al. (2024a), authors employed signifi- cantly larger task sets in their experiments. Nevertheless, we believe this result still promising as Vintix manages to demonstrate inference-time improvement through trial and error on certain tasks, despite being trained on only 87 distinct tasks.
in toy task settings,
As in the previous experiments, we unroll actions from Vintix with no initial context on test tasks from the Meta- World ML45 split and the Bi-DexHands ML20 split (Ap- pendix A.2). Overall, we observed that Vintix not yet capable handle significantly new tasks. Figure 7 presents successful rollout and one failure case for each domain (for detailed task-level results, refer to Appendix F). On the Door-Unlock task from Meta-World Vintix achieved score, while on the Door- Poor-Open-lle O-e---ppeeaattt Inward task from Bi-DexHands, it consistently maintained of demonstrator performance. In failure
4. Related Work
47% of the mean expert-normallin
In-Context Learning. In-Context learning is often used to describe the capacity of large language models to adapt to new tasks after the training phase (Brown et al., 2020:
6

---

Vintix: Action Model via In-Context Reinforcement Learning
Successful rollouts
Failed rollouts bin-picking
RL can itself be divided into major branches: set of approaches inheriting from RL (Duan al., 2016) which encodes task-related information using the RNN's hidden state and directly leverages RL poolyy updates, including more recent transformer-based variants like AMAGO-(1,2) (Grigsby et al., 2024a;b) and RELIC (Elawady et al., 2024). Another perspective on in-context reinforcement learning formalizes the training of Meta-RL agent as an imitation learning problem. This can involve cloning optimal actions, as seen in methods like DPT (Lee et al., 2023), leveraging demonstrator's trajectories, as in ICRT (Fu et al., 2024), or utilizing the entire RL algorithm's 's learning history or its approximations derived via noise distillation (Zisman et al., 2024a). Examples of this data-centric approach include AD and its derivatives (Laskin et al., 2022; Kirsch al., 2023; Sinii et al., 2024; Nikulin et al., 2024). Our algorithm is most closely aligned with the last described category and represents a multi-domain AD trained on trajectories ob- tained through noise distillation.
door-umlock
1.00
0.125
0.80
Meta- World
0.60
0.47
0.010
Normalized Score IQM (95% CI)
0.20
0.00
-0.20
-0.100
-0.40
-0.125
2
21
2
2
2
2
21
2
2
2
dooropeninward
kettle
0.50
-0.50
-0.60
Bl Bl-Dextlands
-0.00
0.31
-0.80
0.20
0.10
-0.96
0.00
-1.10
29
2
27
23
2
2
26
20
21
2
2
21
2
26
Number of Shots
Multi-Task Learning. Multi-Task Learning (MTL) can be formalized paradigm of joint multi-task optimiza- tion, aiming to maximize positive knowledge transfer (syn- ergy) between tasks while minimizing detrimental task in- terference. Numerous studies explore Multi-Task Learning (MTL) beyond the naive minimization of the sum of in- dividual task losses, a method commonly referred to as unitary scalarization. Significant efforts have been made in the fields of massively multilingual translation, where each language pair is treated et al., 2018; 8; Radford et al., 2019; Wang et al., 2020; Li & Gong, 2021), as well as in reinforcement learning (Espeholt et al., 2018; Hessel et al., 2018; Sodhani et al., 2021: Kumar et al., 2023) and robotics (Wulfmeier et al., 2020; Wang et al., 2024). The most common solutions include various types of gradient surgeries to minimize negative interactions between different tasks (Wang et al., 2020; Yu et al., 2020), adaptive tuning of task weights (Sener & Koltun, 2019; Li & Gong, 2021), scaling of model size, and temperature tuning (Shaham et al., 2023). is also worth noting that several studies argue there no substantial evidence that special- ized multi-task optimizers consistently outperform unitary scalarization while also introducing significant complexity and computational overhead (Xin et al., 2022; Kurin et al., 2023). It is important to note that the aforementioned works on Multi-Tase RL rely on various forms of explicit task conditioning, while our approach follows a task-agnostic paradigm that implicitly infers task-related information from interaction experience.
Figure 7. Inference-time performance on new tasks. One success- ful rollout and one failure reported for both Meta-Worid and Bi- Bi-DexHands. Inference is performed without an initial con- text a agno-tgi manner.
Liu et al., 2021). In essence, in-context learning refers to an approach where the algorithm is provided with a set of demonstrations at test time, enabling to infer task-related information (Min al., 2022). In contrast, the in-weights paradigm typically relies
separate task (McCann
tinettuning the model on down- stream tasks (Finn et al., 2017; Wang et al., 2024; Ying et al., 2024). Compared to in- in-weights learning, in- in-context learning is gradient-free at deployment, which theoretically allows for a significant reduction in computational costs and facilitates the development of foundational models tasks (Sun
service, applicable to broad range of al., 2022; Dong et al., 2024). In our work, we use the many-shot in-context learning setup (Agarwal et al., 2024) with a large context window to the reinforcement learning framework.
Offline Memory-Based Meta-RL. Meta-reiiforreement learning (Meta-RL) focuses on enabling agents to adapt to new tasks, environments, or dynamics through interac- tion experience. Numerous diverse approaches exist within the field of Meta-RL (Beck et al., 2024). At a high level, Meta-RL algorithms can be broadly categorized into two main segments: those explicitly conditioned on task repre- sentation (Espeholt et al., 2018; Rakelly et al., 2019; Zhao et al., 2020; Sodhani et al., 2021) and those that infer task dynamics and reward functions from past experience often referred to as Contett" Implicit memory-bar based Meta-
Generalist Agents and Large Action Models. Although most experiments in the field of meta-reiinooreement learning (Meta- RL) are typically confined to a single domain of tasks (Duan et al., 2016; Anand et al., 2021; Laskin et al., 2022),
7

---

Vintix: Action Model via In-Context Reinforcement Learning
generalist agents aim to perform cross-domain training, of- ten integrating multiple data modalities (Gallouédec al., 2024; Reed et al., 2022). Primary advances in this area of research have been made in the field of robotic locomotion and manipulation, where researchers aim to develop gen- eralizable policies and facilitate cross-domain knowledge transfer. This approach seeks to reduce the computational complexity of training policies for robotic applications. The renaissance of generalist robot policies has been notably driven by the availability of large open-source multi-domain robotic datasets like Open X-Fmbodiment (Collaboration et al., 2024). A variety of models build upon the founda- tions established by the Open X datasets, including RT-X (Collaboration et al., 2024), RT-1 (Brohan et al., 2023), Octo (Team al., 2024). OpenVLA (Kim et al., 2024), Dy- naMo (Cui al., 2024). Another category of models relies primarily on training data derived from simulated environ- ments, with notable examples including JAT (Gallouédec et al., 2024), Gato (Reed et al., 2022) and Baku (Haldar et al., 2024). Other closely related work is by Sridhar et al. (2024), where authors propose a retrieval-based algorithm for in-context imitation in the presence of demos for new tasks. Although our work closely aligns with the field of generalist agents and works mentioned, our primary focus is on inference-tine time learning through trial and error driven by the data-centric approach.
Subsequent research has expanded this direction by en- couraging such models to maximize returns (Zhuang et al., 2024), adapting Decision Transformers to online learning settings (Zheng et al., 2022), and replacing the Transformer backbone with architectures that scale more efficiently with input length, such as Mamba (Huang et al., 2024).
Another line of work has aimed to scale this modeling paradigm to multi-domain and multi-modal environments (Reed et al., 2022: Gallouédec al., 2024), or to lever- age the in-context learning capabilities of Transformers for Meta-RL (Laskin et al., 2022; Lee et al., 2023). This lat- ter direction is most closely related to Vintix, which is a memory- memerr-based,, cross-domain Meta-RL RL method.
5. Conclusion and Future Work
The development of generalist reinforcement learning agents that adapt across domains remains critical chal- lenge, as traditional online RL methods-despite their suc- cess in narrow settings- scalability limitations due to their reliance on environment-specifte interactive training (Dulac- AAnlld et al., 2019; Levine et al., 2020). While re- cent advances in offline RL. and generative action models have expanded the potential for data-driven learning, these approaches often prioritize expert demonstrations or lan- guage conditioning over et al., 2022; Gallouédec et al., 2024; Haldar et al., 2024; Col- laboration et al., 2024; Schmied et al., 2024; Sridhar et al., 2024). In this work, we explored an alternative paradigm rooted in In-Context Reinforcement Learning (ICRL), build- ing Algorithm Distillation (Laskin et al., 2022) to create agents that learn adaptive behaviors by straightforward next-action prediction of learning histories or their proxies.
adaptation (Reed
Learned Optimizers. In contrast to traditional optimizers that follow hand-crafted update rules, learned optimizers employ parameterized update rule that is meta-trained to optimize various objective functions (Li & Malik, 2016; Andrychowicz et al., 2016; Metz et al., 2020; Almeida et al., 2021). Thus, learned optimization can be seen as an al- ternative perspective on meta-learning (learning-to-ler Io-lamn),,, with recent approaches scaling to a wide range of tasks and requiring thousands of TPU-months for training (Metz. et al., 2022). However, due to the non-stationarity and high stochasticity of the temporal difference (TD) objective, learned optimizers fail in reinforcement learning setting (Metz et al., 2022). To address these challenges, Optim4RL introduces RL-specific inductive bias (Lan et al., 2024), while OPEN enhances exploration by leveraging learnable stochasticity (Goldie et al., 2024). Both works can be consid- ered optimization-ceninn approaches to Meta-RL, whereas we adopt context-based approach.
Our proposed approach and model, Vintix, demonstrates that ICRL can extend beyond prior single-domain, grid- based benchmarks. By introducing Continuous Noise Dis- tillation, we ease the data collection process inherent to Al- gorithm Distillation and release suite of datasets and tools for 87 tasks across four domains, which we hope would be helpful in community efforts toward scalable, cross-domain action models capable of in-context reinforcement learning. Empirically, we show that Vintix exhibits self correction on training tasks and adapt to moderate controlled parametric variations without requiring gradient updates at inference time. These results, though preliminary and confined to structured settings, suggest that reward-guided ICRL. pro- vides viable pathway for agents to autonomously refine their policies in response to environmental feedback.
Sequence Modeling in RL. With the increasing use of Transformers for modeling sequential data, several concur- rent works (Chen et al., 2021; Janner et al., 2021) reformu- lated the Markov Decision Process as a causal sequence modeling problem. Chen et al. (2021) focused on reward conditioning and treated each component of the MDP as a separate token, while Janner et al. (2021) applied beam search over discretized states, actions, and rewards.
While we believe the obtained results are promising, there is a large room for improvement as challenges remain in increasing the number of domains, developing more task- agnostic architectures that would allow not only for robust
8

---

Vintix; Action Model via In-Context Reinforcement Learning
self-correction but vital generalization unseen tasks and seff-improvement beyond demonstrators' performance. We hope this work encourages further investigation into data- centric, reward-driven frameworks for cross-domain RL agentic systems.
Beck, J., Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of meta-reinnorreement learning, 2024. URL https://arxiv.org/ 2301.08028
Beltagy, L. Peters, M. E., and Cohan, A. Longformer:
The long-document transformer, 2020. URL https: /
Acknowledgements
This work was supported by the Ministry of Economic De- velopment of the RF (code 25- 919-66689--100000
Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv:1912.0660 2019.
Impact Statement
This proposed model is strictly limited to the simulated environments used for evaluation due to its architectural limitations, and therefore does not present any safety con- cemn.. However, in case one would like finetune it for real-world purposes, it does not guarantee any sort of safe behavior and cautions must be made on the part of the user.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Ope- nai gym, 2016. URL https: https///aaxvv.or////// org/ab////////aa 1 1606. 01540.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., Ibarz, J., Ichter, B., Irpan, A., Jackson, T., Jesmonth, S., Joshi, N. J., Julian, R., Kalashnikov, D., Kuang, Y., Leal, I., Lee, K.-H,, Levine, S., Lu, Y., Malla, U., Manjunath, D., Mordatch, I., Nachum, O., Parada, C., Peralta, J., Perez, E., Pertsch, K., Quiambao, J., Rao, K., Ryoo, M., Salazar, G., Sanketi, P., Sayed. K., Singh, J., Sontakke, S., Stone, A., Tan, C., Tran, H., Vanhoucke, V., Vega, S., Vuong, Q., Xia, F., Xiao, T., Xu, P., Xu, S., Yu, T., and Zitkovich, B. Rt-1: Robotics transformer for real-world control at scale, 2023. URL https:// arxiv. org/ hbsp/ab///// 2212 06817.
References
Agarwal, R., Singh, A., Zhang, L. M., Bohnet, B., Rosias, L., C.an, S., Zhang, B., Anand, A., Abbas, Z., Nova, A., Co-Reyes, J. D., Chu, E., Behbahani, F., Faust, A., and Larochelle, H. Many-shot in-context learning, 2024. URL www.http/////ttm /atpiv//axivvvv//a org/ abs 2a.bs//244...0 11018.
Almeida, D., Winter, C., Tang, J., and Zaremba, W. A generalizable approach learning optimizers, 2021. URL https: //axxi org/ abs/ 601.1.00000 00958.
Brown, D. S., Goo, W., and Niekum, Better-than- demonstrator imitation learning via automatically-ranked demonstrations, 2019. URL https:://axxi..oott abs / 000000003995
Anand, A., Walker, J., Li, Y., Vértes, E., Schrittwieser, J., Ozair, S., Weber, T., and Hamrick, J. B. Procedural gen- eralization by planning with supervied world mod- els, 2021. URL https:/ arxiv. org/ abs 2111. 01587.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu,
Andrychowiez, M., Denil, M., Gomez, S., Hoffman, W., Pfau, D., Schaul, T.. Shillingford. B., and de Freitas, N. Learning learn by gradient descent by gradient org/ abs/
J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners, 2020. URL https:// arxiv. org/ abs 22002205.20 14165.
descent, 2016. URL 1606. 04474.
Badia, A. P., Piot, B., Kapturowski, S., Sprechmann, P., Vitvitskyi, A., Guo, Z. D., and Blundell, C. Agent57: Outperforming the atari human benchmark. In Inter- national conference on machine learning, pp. 507-517. PMLR, 2020.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling, 2021. URL https://arxiv.orrt abs/ 2106. 01345.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2444 -2654,, 2022.
Chen, Y., Wu, T., Wang, S., Feng. X., Jiang, J., McAleer, S. M., Geng, Y., Dong, H., Lu, Z., Zhu, S.-C., and Yang, Y. Towards human-level bimanual dexterous manipulation

---

Vintix: Action Model via In-Context Reinforcement Learning
with reinforcement learning. 2022. URL https:// arxiv. org/ /twp:///2.... 68666.
V., Zhan, W. Zhou, W., Burgard, W., Chen, X., Chen,
X., Wang, X., Zhu, X., Geng, X., Liu, X., Liangwei, X.,
Li, X., Pang, Y., Lu, Y., Ma, Y. J., Kim, Y., Chebotar,
Y., Zhou, Y., Zhu, Y., Wu, Y., Xu, Y., Wang. Y., Bisk,
Collaboration, E., O'Neill, A., Rehman, A., Gupta, A., Mad- dukuri, A., Gupta, A., Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar, A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., Kolobov, A., Singh, A., Garg, A., Kem- bhavi, A., Xie, A., Brohan, A., Raffin, A., Sharma, A., Yavary, A., Jain, A., Balakrishna, A., Wahid, A., Burgess- Limerick, B., Kim, B.. Schölkopf, B., Wulfe, B., Ichter, B., Lu, C., Xu, C., Le, C., Finn, C., Wang, C., Xu, C.,, Chi, C., Huang, C., Chan, C., Agia, C., Pan, C., Fu, C., Devin, C., Xu, D., Morton, D., Driess, D., Chen, D., Pathak. D., Shah, D., Büchler, D... Jayaraman, D., Kalash- nikov, D., Sadigh, D., Johns, E., Foster, E., Liu, F., Ceola, F. Xia, F., Zhao, F., Frujeri, V., Stulp, F. Zhou, G., Sukhatme, G. S., Salhotra, G.. Yan, G., Feng. G., Schiavi, G., Berseth, G., Kahn, G., Yang, G., Wang. G., Su, H., Fang, H.S. Shi, H., Bao, H., Amor, H. B., Christensen, H. Furuta, H., Bharadhwaj. H., Walke, H., Fang, H., Ha, H., Mordatch, I. Radosavovic, I., Leal, I., Liang, J., Abou- Chou-Chakra, J., Kim, J., Drake, J., Peters, J.. Schneider, J., Hsu, J., Vakil, J., Bohg. J.. Bingham, J., Wu, J., Gao, J., Hu, J., Wu, J., Wu, J., Sun, J., Luo, J., Gu, J., Tan, J. Oh, J., Wu, Lu, J., Yang, J., Malik, J. Silvério, J.,
Y., Dou, Y., Cho, Y., Lee, Y., Cui, Y., Cao, Y., Wu, Y.-
H., Tang, Y., Zhu, Y., Zhang, Y., Jiang, Y., Li, Y., Li,
Y., Iwasawa, Y., Matsuo, Y., Ma, Z., Xu, Z., Cui, Z. J., Zhang. Z., Fu, Z., and Lin, Z. Open x-embodiment: Robotic learning datasets and rt-x models, 2024. URL https: /atpaaxxiivv./.i///aaa/aaaaaelea .rg. 22300.8864.
Cui, Z. J., Pan, H., lyer, A., Haldar, S., and Pinto, L. Dy- namo: In-domain dynamics pretraining for visuo-motor control, 2024. URL https: http:///axxvvv.vooe 2409. 12192.
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io- awareness, 2022. URL 2205.14133
org/ aba/
Dong, Q. Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Liu, T., Chang, B., Sun, X., Li, L., and Sui, Z. A survey on in-context learning, 2024, URL https: / /arxxa..
wwexiivvrggaaae wwg/abs//////ooooooo.e/......aas ht////bs22223 /2001. 00234.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P. RI$"2S: Fast reinforce- ment learning via slow reinforcement learning. CoRR, abs/1611.00279, 2016. URL httt:://axxiv.or// abs/1611.00 02779.
Hejna, J., Booher, J., Tompson, J., Yang, J., Salvador, J. Lim, J. J., Han, J., Wang. K., Rao, K., Pertsch, K., Hausman, K., Go, K., Gopalakrishnan, K., Goldberg, K., Byme, K. Oslund, K., Kawaharazuka, K., Black, K., Lin, K., Zhang. K., Ehsami, K., Lekkala, K., Ellis, K., Rana, K., Srinivasan, K., Fang, K., Singh, K. P., Zeng, K.-H., Hatch, K., Hsu, K., Itti, L., Chen, L. Y., Pinto, L., Fei-Fei, L., Tan, L.,, Fan, I.. J., Ott, I.., Lee, I.,, Weihs, I.,, Chen, M., Lepert, M. Memmel, M., Tomizuka, M., Itkina, M., Castro, M. G., Spero, M., Du, M., Ahn, M., Yip, M. C., Zhang, M., Ding, M., Heo, M., Srirama, M. K., Sharma, M., Kim, M. J., Kanazawa, N., Hansen, N.. Heess, N., Joshi, N. J., Suenderhauf, N., Liu, N., Palo, N. D., Shafi- ullah, N. M. M., Mees, O., Kroemer, O., Bastani, O., Sanketi, P. R., Miller, P. T, Yin, P., Wohlhart, P., Xu, P., Fagan, P. D., Mitrano, P., Sermanet, P., Abbeel, P., Sundaresan, P., Chen, Q. Vuong, Q., Rafailov, R., Tian, R., Doshi, R., Mart Mn-Mart in, R., Baijal, R., Scalise, R., Hendrix, R., Lin, R., Qian, R., Zhang, R., Mendonca, R. Shah, R., Hoque, R., Julian, R., Bustamante, S., Kir- mani. S., Levine, S., Lin, S., Moore, S., Bahl, S., Dass, S., Sonawani, S., Tulsiani, S., Song, S., Xu, S., Haldar, S., Karamcheti, S., Adebola, S., Guist, S., Nasiriany, S., Schaal, S., Welker, S., Tian, S., Ramamoorthy, S., Dasari, S., Belkhale, S., Park, S., Nair. S., Mirchandani, S., Osa, T., Gupta, T., Harada, T., Matsushima, T., Xiao, T., Kollar, T., Yu, T., Ding, T., Davchev, T., Zhao, T. Z., Armstrong, T., Darrell, T., Chung, T., Jain, V., Kumar, V., Vanhoucke,
Dulac-Amold, G., Mankowitz, D., and Hester, T. Chal- lenges of real-world reinforcement learning. arXiv preprint arXiv: 1904. 22011 2019.
Elawady, A., Chhablani, G., Ramrakhya, R., Yadav, K., Batra, D., Kira, Z.., and Szot, A. Relic: recipe for 64k steps of in-context reinforcement learning for embodied ai, 2024. URL https: /arxiv.iivv org/ orgg/bs//////eoooooo.ggoe.eeeere 2010. 02751.
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dun- ning. I., Legg, S., and Kavukcuoglu, able distributed deep-rl with importance weighted actor- learner architectures, 2018. URL https: //as:i///.i.... org/ abs /00s//800 110//1100001 01561.
Impala: Scal-
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks, 2017. URL https:/ aalaaaiiiiii www.aaiwwwiw.aaaaaaaaaa abs 1703. 03400.
Fu, L., Huang. H., Datta, G., Chen, L. Y., Panitch, W. C.-H., Liu, F. Li, H., and Goldberg, K. In-context imitation learning via next-token prediction, 2024. URL https: www.xi...gg/ab/////oo/ 1194400..119.... 15980.
10

---

Vintix: Action Model via In-Context Reinforcement Learning
Gallouédec, Q., Beeching. E., Romac, C., and Dellandréa, E. Jack of all trades, master of some, a multi-purpose trans- former agent, 2024. URL https: happs///aree http://axxvvv./ooe/oo abs/2402 09844.
Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes, 2023. httpt://aarxx
org/ abs s / 2211 15144.
Goldie, A. D., Lu, C., Jackson, M. T., Whiteson, S., and Foerster, Can learned optimization make rein- forcement learning less difficult?, 2024. URL https: //arxiv. otg. abs 2407. 07082.
Kurin, V., Palma, A. D., Kostrikov, I., Whiteson, S., and Kumar, M. P. In defense of the unitary scalarization for deep multi-task learning, 2023. URL.ttpss //axxiv.. org htgg/bb//2000 04122.
Grigsby, J., Fan, L., and Zhu, Y. Amago: Scalable in-context reinforcement learning for adaptive agents, 2024a. URL. pectps://arxxe http:/aaawwvagaa.aaaa///// wttg/bsg/////oooo.eo...oo...eea/ wwt.aassss1// 2310. 09971.
Lan, Q., Mahmood, A. R., Yan, S., and Xu, Z. Learning to optimize for reinforcement learning, 2024. URL https: //trxiv.orggg/a/ae.i.. 2302 01470.
Laskin, M., Wang. L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E.,, Gazeau, M., Sahni, H., Singh, S., and Mnih, V. In-context reinforcement learning with algorithm distilla- tion, 2022. htts:: /ttp//aaxvvvvvvvv org/ abs/ 2210. 14215.
Grigsby, J., Sasek, J., Parajuli, S., Adebi, D., Zhang, A., and Zhu, Y. Amago-2: Breaking the multi-task barrier in meta-reinforcemmnt learning with transformers, 2024b. URL httttt/s////xiiii org/ abs/24111 11188.
11188.
Haldar, S., Peng, Z., and Pinto, L. Baku: An efficient transformer for multi-task policy learning, 2024. URL https://arxti org/ abs/ 2206 07539.
Lee, J. N., Xie, A., Pacchiano, A., Chandak, Y., Finn, C., Nachum, O., and Brunskill, E. Supervised pretraining can learn in-context reinforcement learning, 2023. URL http://xii...rrraa/////a/0
Hein, D., Depeweg, S., Tokie, M., Udluft, S., Hentschel, A., Runkler, T. A., and Sterzing, V. A benchmark en- vironment motivated by industrial control problems. In 2017 IEEE Symposium Series on Computational Intel- ligence (SSC1), pp. 1-8. IEEE, November 2017. doi: 00.1109/sci.2201.882222 2287933 URL http: //dx doi. org/10. 110000..11009000 www.:/scI...1.....
14892.
Lee, K.-H., Nachum, O., Yang, M. S., Lee, L., Free- man, D., Guadarrama, S., Fischer, I., Xu, W., Jang, E., Michalewski, H., al. Multi-game decision transformers. Advances in Neural Information Processing Systems, 35: 2022.
2017. 8280935.
Hessel, M., Soyer, H., Espeholt, L.., Czameeki, W., Schmitt, S., and van Hasselt, H. Multi-task deep reinforcement learning with popart, 2018. URL https://arxix: org/abs// abs 110...19...0 04474.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems, 2020. URL htttttsss////xi..oom a abs/2005. 01 643.
Huang, S., Hu, J., Yang, Z., Yang, L.,, Luo, T., Chen, H., Sun, L., and Yang, B. Decision mamba: Reinforcement learning via hybrid selective sequence modeling, 2024. URL https::p////omm /olss///axxii /atxiviivv./////// org/ abs/2406. 00079.
Li, K. and Malik, J. Learning to optimize, 2016. URL /ttps::/arxivv.org//// wtwp:///wog////////////oe// 01885.
Li, X. and Gong, H. Robust optimization for multilingual translation with imbalanced data, 2021. URL https:
Janner, M., Li, Q., and Levine, S. Offline reinforcement
/ www.ii.org/aae..........o.. 220...111....... 07639.
learning as one big sequence modeling problem, 2021.
Liang, J., Makoviychuk, V., Handa, A., Chentanez, N., Macklin, M., and Fox, D. Gpu-accelerated robotic simu- lation for distributed reinforcement learning, 2018. URL https://aase /ttp://aaxvvvv//.v wwwpexivvvvogaaaaa wrg. abs /11./.111... 1810.05762.
URL https:://///s wawxiaxxiii..//////aaa/aaaaa//.a www.org///ao.. absp/ 02039.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. Open- vla: An spen-source sasiionlanguage-acce action model, 2024. URL www.thttps...m /ttpps://aaxiii http//axxvvvv/.. wwwpeiiioorr...eeeee abs/ 09246.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. CoRR, abs/2107. 2021. URL hattps://arxtt org/ abs///////// //t.//117...1189
Kirsch, L., Harrison, J., Freeman, C., Sohl-Dickstein, J., and Schmidhuber, J. Towards general-purpose in-context learning agents. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL https: //openreview wwt.:iiwwww..eeeeeeeeeeeeeeeeeee forum d==2De0VGGzH..
McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The natural language decathlon: Multitask learning as ques- tion answering, 2018. URL https: /ttp///avvvv////////////ee/../oa abs/1806.0 08730.

---

Vintix: Action Model via In-Context Reinforcement Learning
Metz, L., Maheswaranathan, N., Freeman, C. D., Poole, B., and Sohl-Dickstein, Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves, 2020. URL https: //arxx //arxiv.. org/ abs/ 9.09. 11243.
inference for robotics tasks. In NeurIPS 2024 Workshop Open- Weren-Worldddo Agents, 2024.
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839): 609,609, 2020.
Metz, L.,, Harrison, J.. Freeman, C. D., Merchant, A., Beyer, L., Bradbury, J., Agrawal, N., Poole, B., Mordatch, I., Roberts, A., and Sohl-Dickstein, J. Velo: Training ver- satile learned optimizers by scaling up, 2022. URL https: / aawiiiivvv. wwwpwwwwww..w...o...o.....oo wwgg/bss/....ooooo...rr...ee...e 2212 09760.
Schulman, J., Wolski, F. Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: /ttpss///axivv.oo abs/ 1707.0634.
Min. S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role demonstrations: What makes in-context learn- ing work?, 2022. URL https: /awxiiiiiiiv////ii///aaa///aaivv org/o/g//ooooo /bss/ 2202. 12837.
Sener, O. and Koltun, V. Multi-task learning as multi- objective optimization, 2019. URL https:///: //arxiv.. org/ hataabba/1111 04650.
Nikulin, A., Zisman, I., Zemtsov, A., Sinii, V., Kurenkov, V., and Kolesnikov, S. Xland-100h: A large-scale multi-task dataset for in-context reinforcement learning, 2024. URL aittps://arxte
Shaham, U., Elbayad, M., Goswami, V., Levy, O., and Bhos- ale, S. Causes and cures for interference in multilingual translation, 2023. URL https: http::///xxvvv.ri.ee.o arbaa 07530.
wtwp/bs/////aoooeeooo//eeeeeee/s 2406. 08973.
Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length ex- trapolation, 2022. URL https: /arxivivvvv/./////vv///a//////av .iiivooo org/ abs/ 2108. 12409.
Silver, D., Singh, S., Precup, D., and Sutton, S. Reward is enough. Artificial Intelligence, 299:103535, 2021.
Sinii, V., Nikulin, A., Kurenkov, V., Zisman, I., and Kolesnikov, S. In-context reinforcement learning for variable action spaces, 2024. URL https:
Radford, A.,
Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
org/ aa///a///// 22122 13327.
Sodhani, S., Zhang, A., and Pineau, J. Multi-task reinforce- ment learning with context-based representations, 2021. URL https::////// arxiv. org/ haa/abba/21122 06177.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M.. and Dormann, N. Stable-baselines: Reliable rein- forcement learning implementations.. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: //jmlr www.p//gppppppe.e..
Sridhar, K., Dutta, S., Jayaraman, D., and Lee, Re- gent: A retrieval-augmmenté generalist agent that can act in-context in new enviromments. arXiv preprint arXiv:2412.0475:: 2024.
htmml. html.
Rakelly, K., Zhou, A., Quillen, D., Finn, C., and Levine, S. Efficient off- off-policy meta-reinforcemeen learning via prob- abilistic context variables. CoRR, abs//903.000 002554, 2019. URL http: http:::/aaxvv.voooa httg/abb/19933 082554.. 08254.
Sun, T., Shao, Y., Qian, H., Huang, X., and Qiu, X. Black- box tuning for tigguage-mdeelas-a-seee 02//2201.03514 2022. URL https: http::////xxvvoo/r/// abs / 20002202.00 03514.
CoRR,
Reed, S., Zolna, K., Parisotto, E,, Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., Eccles, T., Bruce, J., Razavi, A., Edwards, A., Heess, N., Chen, Y., Hadsell, gener-
Sutton, R. S. and Barto, G. Reinforcement learning: An introduction. 1998.
R., Vinyals, O., Bordbar, M., and de Freitas, URL.httss://ariii /ttp:////avvvv..i org/ abs/ alist agent, 2022. URL httpt:t/////oc
Tarasov, D., Nikulin, A.. Akimov, D., Kurenkov, V., and Kolesnikov, S. Corl: Research-oriented deep offline rein- forcement learning library. Advances in Neural Informa- tion Processing Systems, 36, 2024.
2205. 06175.
Schmidhuber, J. Reinforcement learning upside down: Don't predict rewards-just map them to actions. arXiv preprint
Team, A. A., Bauer, J., Baumli, K., Baveja, S., Behba- hani, F., Bhoopchand, A., Bradley- N., Chang, M., Clay, N., Collister, A., et Human-timescale adaptation in an open-ended task space. arXiv preprint arXiv:2301.06608, 2023.
2019.
Schmied, T., Adler, T., Patil, V. P., Beck, M., Pöppel, K., Brandstetter, J., Klambauer, G., Pascanu, R., and Hochre- iter, A large recurrent action model: xlstm enables fast
12

---

Vintix: Action Model via In-Context Reinforcement Learning
Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Kreiman, T., Xu, C., Luo, J., Tan, Y. L., Chen, L. Y., Sanketi, P., Vuong, Q Xiao, T., Sadigh, D., Finn, C., and Levine, S. Octo: An open-source generalist robot policy, 2024. URL https: //arxiivoorg/ abs abs/ / 2405 12213.
Zhang, Zeng. G., Wang, T., and Lu, W. Tinyllama: An open-source small language model, 2024.
Zhao, T. Z., Nagabandi. A., Rakelly, K., Finn, C.. and Levine, MELD: meta-reinforrcement learning from im- ages via latent state models. CoRR, abs/2010. 2020. URL www.httpps/..mm araaxivvee httpp////wwwww../aaaab/////
139577.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems, 5026-5033, 2012. doi: 10.1 1091ROO 201.6336611999
Zheng, Q. Zhang. A., and Grover, A. Online decision trans- former, 2022. URL httpss://rxxvv.rrgg// 2202. 05607.
Zhuang, Z., Peng, D., Liu, J., Zhang, Z., and Wang, D. Reinformer: Max-return sequence modeling for offline rl, 2024. URL 08740.
Towers, M., Kwiatkowski, A., Terry, J., Balis, U., Cola, G. D., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M.. KG, A., Perez- Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Tan, H., and Younis, O. G. Gymnasium: A standard interface for reinforcement learning envi- ronments, 2024. URL https::t//// http///xxvvvv.o/o orgoorgg///....o.oooo /
org/ abs/ 2405.
Zisman, I., Kurenkov, V., Nikulin, A., Sinii, V., and Kolesnikov, S. Emergence of in-context reinforcement learning from noise distillation, 2024a. URL https: //arxiv. org/ abs 2212. .2275..
Wang, L., Chen, X., Zhao, J., and He, K. Scaling proprioceptive-vistt learning with heterogeneous pre- trained transformers, 2024. URL https://arxiv. org/abs/ 2409.
Zisman, I., Nikulin, A., Polubarov, A., Lyubaykin, N., and Kurenkov, V. N-gram induction heads for in-context rl: Improving stability and reducing data needs, 2024b. URL https:// ptass///rriv... www.vvorrgggoooo http://bs///11....e
Wang, Z., Tsvetkov, Y., Firat, and Cao, Y. Gradient vaccine: Investigating and improving multi-task opti- mization in massively multilingual models, 2020. URL https: / /ttp/aaavvv//..////a//.../...av. org/ /bsg/bb///// 2alss22110 2010.00000022 05874.
Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg. T., Neunert, M., Hertweck, T., Lampe, T., Siegel, N., Heess, N., and Riedmiller, M. Compositional transfer in hierarchical reinforcement learning, 2020. URL https: //arxiv. orgo abs / 906. 11228.
Xin, D., Ghorbani, B., Garg, A., Firat, O., and Gilmer, J. Do current multi-task optimization methods in deep learning even help?, 2022. URL https: /ttpss////xxvvvvo// org/ abs/ 2209. 11379.
Ying, C., Hao, Z., Zhou, X., Xu, X., Su, H., Zhang, X., and Zhu, J. PEAC: Unsupervised pre-training for cross- embodiment reinforcement learning. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL httph:ht//// Forum?id-LyAF fdxYYF.
httpp:://pperreiieeeee /oppreveeeiieeeeeeee net.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning, 2020. URL https:://// /ttp//aavvvv///////////e///aaav. org/ abs abs/2001..............2 06782.
Yu, T., Quillen, D., He, 7., Julian, R., Narayan, A., Shively,
H., Bellathur, A., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, 2021. URL ht https: //a //axivv. www//bs///a/ooooooooooooo.e
abs /990..191111 10897.
13

---

Vintix: Action Model via In-Context Reinforcement Learning
A. Dataset Details
A.1. General Information
MuJoCo. MuJoCo (Todorov et al., 2012) is a physics engine designed for multi-joint continuous control. For our purposes, we selected 11 classic continuous control environments from OpenAl Gym (Brockman al., 2016) and Gymnasium (Towers al., 2024).
Meta- Dora-World. Meta-World (Yu al., 2021) is an open-source simulated benchmark for meta-reinforcement learning and multitask learning consisting of 50 distinct robotic manipulation tasks. It is important note that, similar JAT (Gallouédec al., 2024), we limit the episode length to 100 transitions to reduce dataset size and increase the number of episodes that fit within the model's context. This truncation is feasible, such a horizon is often sufficient to solve the task. However, this modification complicates direct performance comparisons with models such as AMAGO-2 (Grigsby et al., 2024b), which utilize full 500-step rollouts and reports total returns over three consecutive episodes.
Bi-DexHands. HinDexHands (Chen et al., 2022) is the first suite of bi-manual manipulation environments designed for practitioners in common RL, MARL, offline RL, multi-task RL, and Meta-RL. It provides 20 tasks that feature complex, high-dimensional continuous action and observation spaces. Some task subgroups share state and action spaces, making this benchmark applicable to the Meta-RL framework.
-eaitriaal-Beehharkkk (Hein al., 2017) is suite of synthetic continuous control problems designed to model various aspects that are erucial in industrial applications, such the optimization and control of gas and wind turbines. The dynamic behavior of the environment, as well as its stochasticity, can be controlled through the setpoint parameter p. By varying this parameter from 0 to 100 in increments of 5, we generated 21 tasks that share a common state and action structure. Classic reward function was selected and subsequently down-scaled by factor of 100.
Industrial-Bencee Benchmrk.
A.2. Train vs. Test Tasks Split
To validate the inference-time optimization capability of our model, we divided the overall set of 102 tasks into two disjoint subsets. The validation subset was excluded from the training dataset. Below, we provide details of the split for each domain.
1.2.1. MuJoCo
Validation tasks for the MuloCo domain were created by modifying the physical parameters of the environments using the provided XML API. For each embodiment, the default viscosity parameter was adjusted from 0 to 0.05, and 0. Additionally, the gravity parameter was varied by #10 percent.
A.2.2. META-WORLD
The standard ML45 split was selected, with 45 tasks assigned to the training set and 5 tasks reserved for validation: bin-picking, box-close, door-lock, door-unlock, and hand-ing -inddiisert..
A.2.3. B1- BI-DEXHANDS
We adopted the ML.0 benchmark setting proposed by the original authors (Chen et al., 2022), in which 15 tasks are assigned to the training set, while 5 tasks are reserved for validation, including: door-close-ouwward, door-open-inward, door-open-outwarrl hand-kettle, and hand-over. It is important note that the hand-over task does not share the same state-action space dimensionality with any tasks in the training set, making it incompatible with the current encoder architecture. As result, we report performance of 0 (random) for this task.
A.2.4. INDUSTRIA-BEENCHMAKE BENCHMARK
For this domain, a global split based on the setpoint parameter was made, with setpoints ranging from 0 to 75 assigned to the training set, and setpoints from 80 to 100 assigned to the validation set.
14

---

Vintix: Action Model via In-Context Reinforcement Learning
A.3. Epsilon Decay Functions
The epsilon decay function defines how the noise proportion e depends on the transition number na within trajectory, The primary purpose of utilizing such function is to ensure in smooth increase in the rewards through generated trajectories. A linear decay function may not yield this behavior across all environments, some tasks exhibit rewards that increase rapidly either at the beginning or the end of trajectories when linear function is utilized. To address this variability, we employ the following generalized decay function:
V11 (na/((1- f)N,))² n2 > (1-f)N,
ns <= - (1-f)N,
((n,)=
where N, represents the maximum number of transitions in a trajectory, f the fraction of the trajectory with zero noise, and pis is parameter that controls the curvature of the decay function. By adjusting p. is possible to modulate the smoothness of the reward progression along the trajectory in order to better suit different tasks. Specifically, when rewards increase sharply at the beginning of the trajectory, setting p > 1 flattens the epsilon curve at the start and steepens it at the end. Conversely, when rewards increase in a sharp manner towards the end of the trajectory, choosing 0 < < 1 results in the epsilon curve being steepened at the start and flattened at the end.
B. Demonstrators
B.1. Training
MuJoCo. The dataset provided by JAT (Gallouédec et al., 2024) consists of demonstrations from expert RL agents on MuJoCo (Todorov et al., 2012) tasks. It was used train demonstrators through the imitation learning paradigm. The resulting models achieve performance similar to that of the original RL agents.
Meta-World. For the Meta-World (Yu et al., 2021) benchmark, Gallouédec et al. (2024) open-sourced trained agents for each task in the benchmark. These models served as demonstrators; however, detailed analysis revealed that certain agents exhibited suboptimal performance. Specifically, models trained on the disassemble, coffee-pull, coffee-push, soccer, push-back, reg-insert-side, and pick-out-of-hole tasks were either too noisy or performed unsatisfactorily. As a result, we retrained agents for these tasks.
JAT (Gallouédec et al., 2024) also provided open-source training scripts, which we utilized for hyperparameter tuning and retraining new agents, This process led improved performance for the selected tasks. However, many demonstrators across other tasks still exhibit noisy performance.
Bi-DexHands. Demonstrators were trained using the provided PPO (Schulman et al., 2017) implementation. To enhance convergence, the number of parallel environments in IsaacGym (Liang et al., 2018) was increased from 128 to 2048. In certain environments, such as the Re- Re-Orientation and Swing-Cin tasks, we were able to significantly surpass the expert performance reported in the original Bi-DexHands paper (Chen al., 2022). However, even after extensive training for over 1.5 billion timesteps, some policies continued to exhibit stochastic performance.
Industrial-Bencce Benchhaar. Demonstrators were trained using the PPO implementation from 3asble-B (Raffin et al., 2021). For all tasks, PPO was trained with advantage normalization, KL-divergence limit of 0.2, 2500 environment steps, and a batch size of 50. All agents were trained for million timesteps. Similarly to original Industrial Benchmark setup (Hein et al., 2017), discount factor was set to 0.97. To ensure better score comparability, we limited the episode length to 250 interactions. PPO agents were trained using both classic and delta rewards; however, we observed that agents trained on delta rewards achieved higher classic reward scores compared to those trained directly on scaled classic returns. Consequently, for our experiments, we utilize demonstrators trained with delta rewards.
15

---

Vintix: Action Model via In-Context Reinforcement Learning
C. Additional Experiments
C.1. Is Algorithm Distillation More Effective Than Expert Distillation?
To assess the benefits AD-style training on noise-distilled trajectories over simple Expert Distillation (ED), we conducted an experiment using model with the same architecture as Vintix Vintix-including the transformer backbone, encoders, and loss function-butt but trained exclusively expert-level demonstrations. We then evaluated both the ED and AD models using the cold-start inference procedure.
Figure 8 shows that ED underperforms relative to AD in terms of asymptotic performance across both the MuJoCo and Meta- World domains. Specifically, ED converges to an average expert snperttnommalieed score of 0.8, whereas AD reaches 0.97 on MuloCo and 0.95 on
These results highlight the critical role of policy-improvement structure in the dataset enabling self-correcting behavior and achieving superior performance. Notably, while ED's performance remains relatively stable across different shot counts in MuJoCo, shows positive trend in Meta-Worke the number of shots increases.
C.2. Does Vintix Performs In-Context Reinforcement Learning?
To investigate whether Vintix engages reinforcement learning during inference, we trained a variant of the model without access reward signals. We then compared its performance to the original Vintix model (trained with rewards) using the cold-start inference procedure training tasks from both domains.
Evaluation results (Figure 8) demonstrate that removing the reward signal significantly impairs performance. The reward- masked AD model performs worse asymptotically and exhibits slower convergence on the Meta-World domain.
These findings suggest that reward feedback is essential for effective self-improvement during inference. This supports the hypothesis that supervised training on dataset containing policy improvement mechanisms enhances the model's 's in-context reinforcement learning capabilities.
Normalized Score Mean (95% CI)
MuJoCo
Meta-World
1.1
1.1
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.5
0.6
Mope
Model.
Type
0.4
=@m AD
0.5
0.3
(otniieddddeee Bewards)
-@@
(ctaiied Bevardo
0.4
0.2
20
21
2
23
24
25
2
21
2
23
2
Number of Shots
Figure 8. Performance of AD, ED and AD without rewards on MuJoCo and Meta-World domains.
16

---

Vintix: Action Model via In-Context Reinforcement Learning
D. Hyperparameters
Hyperparameter
Value
Learning Rate
0.0003
Optimizer
Adam
Beta
0.9
Beta 2
0.99
Batch Size
64
Gradient Accumulation Steps
Transformer Layers
20
Transformer Heads
16
Context Length
8192
Transformer Hidden Dim
1024
FF Hidden Size
4096
MLP Type
GpINeoXMLP
Normalization Type
LayerNorm
Training Precision
bf16
Parameters
332100768
Table 2. Hyperparameters used in training.
17

---

Vintix: Action Model via In-Context Reinforcement Learning
E. Task-Level Dataset Visualization
assembly
basketball
button-press- topdow
button--ress-topddwwwwlll
1.0
0.5
button- putton--pesss
button- press-wes wall
coffee-button
coffee- affee--uull
1.0
0.5
0.0
coffee-push
dial-turn
disassemble
door-close
1.0
Normalized Return IQM (95% CI)
0.0
door-open
drawer-close
drawer-open
faucet-close
1.0
0.5
0.0
faucet-open
hammer
handle-precd handle-press-side piess-side
handle-press
1.0
0.5
0.0
handle-pull-side
handle-pull
lever-pull
peg-insert-side
1.0
0.5
0.0
peg-unplug-side
pick-out-of-hoo
pick-place
pick pick-place-wall
1.0
0.5
0.0
0.0
0.5
0.5
1.0
0.5
1.0
0.0
1.0
Trajectory Share
Figure 9. IQM aggregated noise-distilled trajectories for Meta-World domain (Part 1 of 2)
18

---

Vintix: Action Model via In-Context Reinforcement Learning
plate-slides sidde-back-side
plate-slide slide-back
plate-slide side
plate-slide
1.0
0.5
0.0
push-back
push
push-wall
reach
1.0
0.5
0.0
reach-wall
shelf place
soccer
stick pull
Normalized Return IQM (95% CI)
1.0
0.5
0.0
stick-push
sweep-into
sweep
window-close
1.0
0.5
0.0
window-open open
bin picking
box-close
door-lock
1.0
0.5
0.0
door-unlock
hind-insert
1.0
0.5
0.0
Trajectory Share
Figure 10. IQM aggregated noise-distileed trajectories for Meta-World domain (Part 2
19

---

Vintix: Action Model via In-Context Reinforcement Learning
blockstack
bottlecap
catchabreast
catchover2underarm
1.0
0.5
0.0
catchunderarm
doorcloseinward
graspandplace
liftunderarm
1.0
0.5
Normalized Return IQM (95% CI)
0.0
pushblock
reorientation
scissors
1.0
0.5
0.0
swingcup
switch
twocatchunderarm
doorcloseoutward
1.0
0.5
0.0
dooropeninward
dooropenoutward
kettle
over
1.0
0.5
0.0
0.5
0.5
1.0
0.0
0.5
1.0
Trajectory Share
Figure 11. IQM aggregated noise-distilled trajectories for Bi-DexHands domain
20

---

Vintix: Action Model via In-Context Reinforcement Learning
IB-000-v1
1B-005-v1
IB-010-v1
IB 03-015-v1
1.0
0.5
0.0
IB-020-v1
IB- IB-025-v1
IB IB-030-v1
IB 0B-033-v1
1.0
0.5
0.0
IB-040-V1
IB 045-v1
IB IB-050-v1
IB-055-v1
Normalized Return IQM (95% CI)
1.0
0.5
0.0
IB-060-v1
IB 13-065-v1
07-070-v
IB-075-v1
1.0
0.5
0.0
IB-080-v1
IB IB-085-v1
IB IB-090-v1
IB-095-v7 -v1
1.0
0.5
0.0
IB-100-v1
1.0
0.5
0.0
Trajectory Share
Pigure 12, 1QM aggregated noise-distileed trajectories for Industrial Benchmark domain
21

---

Vintix: Action Model via In-Context Reinforcement Learning
InvertedPendulum-vá
InverteiDoubbeeenduluuww
Humanoid v4
Walker2d- -4
1.0
0,5
Normalized Return IQM (95% CI)
0.0
Hopper-v4
Ant-v4
HumanoidStandup-v4
Swimmer-v4
1.0
0.5
0.0
Reacher-v4
Pusher-v5
HalfCheetah-v4
1.0
0.5
0.0
Trajectory Share
Figure 13. IQM aggregated noise-distiledd trajectories MuloCo domain
22

---

Vintix: Action Model via In-Context Reinforcement Learning
F. Inference Time Performance Graphs
assembly
basketball
Button-press-opdowww
button-pres-tapddowr
1,50
1.5
1.50
1,50
1.11
1.1
1.01
0.50
0.5
0.50
0.50
0.00
0.0
0.00
0.00
2-
2
2
2
2
button-press press
button-press-wee wall wall
coffee-butee button
coffee- pull
1,50
1.50
1.50
1.50
1.01
1.02
1.02
1.12
0.50
0.50
0.50
0.50
0.00
0,00
0.00
0,00
2
2
2
coffee-push
dial-turn
disassemble
door-close
1.50
1.5
1.50
1.16
1.05
1.1
1.07
0.50
0.5
0.50
0.00
0.50
0,00
0.0
0,00
Normalized Score IQM (95% CI)
23
door-open
drawer-closc
drawer- open
faucet-close
0.97
1.50
1.0
1.02
0.5
0.5
0.50
0.0
0.00
0.0
0.00
2
21
faucet-open
hammer
handle press side
handle-pres press
1.50
1.04
1.01
1.06
0.5
0.50
0.50
0.50
0.0
0,00
0.00
0.00
2
2²
2
2
2
2
handle- handle-pull-sie pull-side
handle-pull
lever-pull
peg. peg-insert-side
1.50
1,50
2,00
1.16
1.02
1.27
1,00
0.50
0.50
0.50
0.00
0.00
0.00
0,00
2
23
2
21
2
2
2
peg-unplug side
side
pick-out-off
pick place
pick pick-place placc-waclll
1.50
2.00
1.50
1.50
0.98
1.05
1.07
1.12
0.50
0.00
-1.00
-0.50
0.00
2
23
2
21
23
2'
2
Number of Shots
Figure 14, Inference performance for Meta-World domain (Part 1 of 2)
23

---

Vintix: Action Model via In-Context Reinforcement Learning
plate-slide-back-sdde
plate-slide-back
plate-slide-side
plate-slide
1.30
1.50
1.50
1.50
1.24
1.06
1.12
1.03
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
2'
23
2
2
2
push-back
push
push-wall
reach
1.50
1.50
1.5
1.50
1.16
1.15
1.04
1.0
0.50
0.50
0.5
0,50
0.00
0.00
0.0
0.00
21
2
2
21
2
2
reach-wall
shelf-place
soccer
stick-pull
1.50
1.50
1.50
1,50
Normalized Score IQM (95% CT)
1.14
1.04
0,97
0.50
0.50
0,50
0,50
0.00
0.50
0.00
0.00
21
23
21
2
21
2
2
stick push
sweep-into
sweep
window-clos
1.50
1.50
1.50
1.5
0.99
1.01
1.04
1.0
0.50
0.50
0,50
0.00
0.00
0.0
-0.50
-0.50
0.00
-0.5
21
2
21
2
21
2
2
window-open
bin-picking
box-close
door-lock
1.50
0.60
0.60
1.00
0.74
1.03
0.40
0.40
0.50
0.00
0.00
0.01
-0.03
-0.50
21
2
21
2
21
2
2
door-unlock
hand-insert
1.00
1.00
0.47
0.50
0.24
0.00
-0.50
-0.50
21
2
21
2
Number of Shots
Figure 15. Inference performance for Meta-World domain (Part 2 of 2)
24

---

Vintix: Action Model via In-Context Reinforcement Learning
blockstack
doorcloseinward
reorientation
bottlecap
1.50
2.00
0.89
1.25
0.38
1.08
0.50
0.50
0.00
0.00
0.00
-1.00
0.00
2
2
2
2
2
2
2a
2
graspandplace
scissors
catchabreast
liftunderarm
1.50
1.50
0.60
0.96
1.02
1.04
0.50
0.50
0.00
0.00
-0.50
0.00
-0.00
0.00
Normalized Score IQM (95% CT)
2a
2
2²
2
2a
2
swingcup
catchoverzunderarm
pen
switch
1.50
0.95
0.99
0.99
0.47
0.50
0.50
0.00
0.00
-0.50
0.00
22
25
2
2
2
2
catchunderarm
pushblock
twocatchunderarm
docrcloseoutward
1.50
0.60
1.00
0.78
0.50
0.99
0.00
0.50
-1.23
-0.50
0.00
0.04
-2.00
2²
2
2
2
2
2
2
2
dooropeninward
dooropenoutward
kettle
0.60
0.60
0.50
0.40
0.00
-0.50
0.15
0.00
-0.96
22
23
2a
Number of Shots
Figure 16. Inference performance for Bi-DexHands domain
25

---

Vintix: Action Model via In-Context Reinforcement Learning
IB-0-V1
IB-5-v1
IB-10-v1
IB-15-v1
1.30
1.50
1.50
1.50
1.02
1.01
1.01
0.50
0.50
0.50
0.50
0.00
25
2²
2
2
2
IB-20-v1
IB-25-v1
IB-30-v1
IB-35-v1
1.50
1.50
1.50
1.50
1.01
1.01
0.91
1.01
0.50
0.50
0.50
0.50
0.00
0.00
0.00
2
2²
2
2
2
2
IB-40-v1
[B-15-v1
IB-50-v1
IB-55-v1
1.50
1.50
Normalized Score IQM (95% CT)
1,01
0.98
0.50
0.50
0.50
0.5
0.00
0.00
22
25
2a
2
2
2
2
IB-60-vL -0-00-
IB-65-v1
IB-70-v1
IB-75-v1
0.91
0.96
0.50
0.50
0.50
0.5
0.00
0.00
2
23
2
2
2
2
IB-80-v1
IB-85-v1
IB-90-v1
IB-95-v1
0.97
0.98
0.97
0.98
0.50
0.50
0.50
0.50
0.00
0.00
0.00
27
2
2²
2
2
2
2
2
1B-100-v1
0.98
0.50
0.00
2
2
Number of Shots
Figure 17, Inference performance for Kidustrial-Benchhmee domain
26

---

Vintix: Action Model via In-Context Reinforcement Learning
InvertedPendulum-v44
IvvereddDaubleennduuuvw-
Humanoid-v4
Walker2d-v4
1.0
1.50
1.50
1.50
0.5
1.01
0.77
0.0
0.50
0.50
0,50
Normalized Score IQM (95% CT)
-0.5
0.00
0.00
0.00
2!
2²
2'
2
23
Hopper-v4
Ant-v4
HumanoidStandup-vi
Swimmer-v4
1.50
1.50
1.50
1,01
1.01
1.11
1.04
0.50
0.50
0.50
0.50
0.00
0.00
0.00
0.00
21
2'
21
2
2
2
2
2
Reacher-vd
Pusher-v5
HalfCheetah-v4
1.50
1.50
0.93
1.03
1.08
0.50
0.50
0.50
0.00
0.00
0.00
21
2
21
2
21
2
Number of Shots
Figure 18. Inference performance for MuJoCo domain
G. Dataset Size and Metadata
Trajectory
Number of
Mean Episode
State space shape
Action space
Reward
Task
Length
trajectories
Length
shape
Scaling
Hand-Block-Stand
250000
10
248.6
(428,)
(52.) )
1.0
Hand-Boulle-Cap
135000
10
123.7
(420,) )
(52,
1.0
165000
10
98.5
(422.)
(52.)
1.0
100000
10
55.6
(422,)
(52,)
1.0
Hand-Catch-Underamr
100000
10
64.9
(422,
(52,)
1.0
Hand-Door-Cloos Cwose- Inward
100000
10
23.8
(417,)
(52.) )
1.0
Hand-Graap-And-PPe
500000
8
332.6
(425,)
(52,)
1.0
500000
10
455.0
(417,)
(52,)
1.0
Hand-Pen
135000
10
120.5
(417.)
(52.)
1.0
Hand-Push-Blooo
135000
10
123.2
(428,)
(52,)
1.0
Hand-Ra Reor-eccoinnntiion
600000
7
463.9
(422,
(40.
1.0
Hand-Scissors
175000
10
149.0
(417.)
(52.)
1.0
Hand-Swing-Cup
320000
10
299.0
(417,)
(52,)
1.0
Swidd
135000
10
124.0
(417,)
(52.)
1.0
Hand- Tww-wwoolltto Cance--nneraam
100000
10
65.1
(446.)
(52.) )
1.0
Soant-Door-Clos--Ouueee
265000
10
249.0
(417,)
(52,)
1.0
Hand-Door-O Hand-Door-OpenIInwee
265000
10
249.0
(417.)
(52,)
1.0
Hand-Door-O00 Door--Open--Ouwaaar
250000
10
249.0
(417,)
(52.)
1.0
Hand- Kettle
135000
10
124.0
(417.)
(52.)
1.0
Hand-Over
100000
10
64.4
(398,) )
(40,)
1.0
Table Bi-DexHands Detailed information about collected dataset
27

---

Vintix: Action Model via In-Context Reinforcement Learning
Trajectory Length
Number of trajectories
Mean Episode
State space
Action space
Reward Scaling
Task
Length
shape
shape
assembly
100000
15
100
(39.)
(4,)
1.0
basketball
100000
15
100
(39.)
(4,)
1.0
bin-picking
100000
5
100
(39.)
(4.)
1.0
box-close
100000
5
100
(39.)
(4.)
1.0
button-press
100000
15
100
(39.)
(4.)
1.0
hutton-press-topdoown
100000
15
100
(39.)
(4.)
1.0
outton-press-toodowwr
100000
15
100
(39.)
(4.)
1.0
nutton-press-wall.
100000
15
100
(39.)
(4.)
1.0
coffce-button
100000
15
100
(39.)
(4.)
1.0
coffee-pull
100000
15
100
(39.)
(4.)
1.0
coffee-push
100000
15
100
(39,)
(4.)
1.0
dial-turn
100000
15
100
(39.)
(4.)
1.0
disassemble
100000
15
100
(39.)
(4.)
1.0
door-close
100000
15
100
(39.)
(4.)
1.0
door-lock
100000
5
100
(39.)
(4.)
1.0
door-open
100000
15
100
(39.)
(4.)
1.0
door-unlock
100000
5
100
(39.)
(4.)
1.0
drawer-close
100000
15
100
(39.)
(4.)
1.0
drawer-open
100000
15
100
(39,)
(4.)
1.0
faucet-close
100000
15
100
(39.)
(4.)
1.0
faucet-open
100000
15
100
(39.)
(4.)
1.0
hammer
100000
15
100
(39.)
(4.)
1.0
hand-insert
100000
5
100
(39.)
(4.)
1.0
handle-press
100000
15
100
(39.)
(4.)
1.0
handle-press-side
100000
15
100
(39,)
(4.)
1.0
handle-pull
100000
15
100
(39.)
(4.)
1.0
puddee-pull-sde
100000
15
100
(39.)
(4.)
1.0
lever-pull
100000
15
100
(39.)
(4.)
1.0
peg-insert-side
100000
15
100
(39.)
(4.)
1.0
peg-unplug-sin
100000
15
100
(39.)
(4.)
1.0
pick-out-of-hole
100000
15
100
(39.)
(4.)
1.0
pick-place
100000
15
100
(39.)
(4.)
1.0
pick-place-wall
100000
15
100
(39.)
(4.)
1.0
plate-slide
100000
15
100
(39.)
(4.)
1.0
plate-slide-back
100000
15
100
(39.)
(4.)
1.0
Mlteeslideeback--eee
100000
15
100
(39.)
(4.)
1.0
plate-slide-side
100000
15
100
(39.)
(4.)
1.0
push
100000
15
100
(39.)
(4.)
1.0
push-back
100000
15
100
(39.)
(4.)
1.0
push-wall
100000
15
100
(39,)
(4.)
1.0
reach
100000
15
100
(39,)
(4.)
1.0
reach-wall
100000
15
100
(39.)
(4.)
1.0
shelf-place
100000
15
100
(39.)
(4.)
1.0
soccer
100000
15
100
(39.)
(4.)
1.0
stick-pull
100000
15
100
(39.)
(4.)
1.0
stick-push
100000
15
100
(39.)
(4.)
1.0
sweep
100000
15
100
(39.)
(4.)
1.0
sweep-into
100000
15
100
(39.)
(4.)
1.0
window-close
100000
15
100
(39.)
(4.)
1.0
window-open
100000
15
100
(39.)
(4.)
1.0
Table 3. Meta- World Detailed information about collected dataset
28

---

Vintix: Action Model via In-Context Reinforcement Learning
Trajectory Length
Number of trajectories
Mean Episode
State space
Action space
Reward Scaling
Task
Length
shape
shape
Ant-v4
1000000
10
524.1
(105,)
(8, )
0.1
HaliCheetah-v4
1000000
1000
(17,)
(6. )
0.1
Hopper-v4
1000000
10
227.3
(11,)
(3. )
0.1
Humanoid-v4
10000000
15
207.7
(348,)
(17,)
0.1
HumanoidStandup-wv v4
1000000
15
1000
(348,)
(17.)
0.01
InvertedDoublleeenduuuwww
1000000
10
55.7
(9,)
(1,)
0.1
InvertedPendulum-v4
1000000
10
68.2
(4,)
(1,)
0.1
Pusher-v5
1000000
5
100
(23,)
(7.)
0.1
Reacher-v4
1000000
50
(10,)
(2,
0.1
Swimmer-v4
10000000
1000
(8,)
(2.
1.0
Walker2d-v4
1000000
10
350.2
(17,)
(6,
0.1
Table 5. MuJoCo Detailed information about collected dataset
Trajectory Length
Number of trajectories
Mean Episode
State space shape
Action space
Reward Scaling
Task
Length
shape
IB-p-0
100000
15
250
(6,
33
1.0
IB-p-5
100000
15
250
(6,
3,
1.0
IB-p-10
100000
15
250
(6,
33,
1.0
IB-p-15
100000
15
250
(6,
3,
1.0
IB-p-20
100000
15
250
(6,
3,
1.0
IB-p-25
100000
15
250
(6,
(3.
1.0
IB-p-30
100000
15
250
(6,
3,
1.0
IB-p-35
100000
15
250
(6,
1.0
IB IB-p-40
100000
15
250
(6,
(3.
1.0
IB-p-45
100000
15
250
(6,
[3
1.0
IB-p-50
100000
15
250
(6,
3,
1.0
IB IB-p-55
100000
15
250
(6,
(3
1.0
IB-p-60
100000
15
250
(6,
3.
1.0
IB-p-65
100000
15
250
(6,
3,
1.0
IB-p-70
100000
15
250
(6,
(3,
1.0
IB-p-75
100000
15
250
(6,
(3.
1.0
IB-p-80
100000
15
250
(6,
(3
1.0
IB-p-85
100000
15
250
(6,)
(3,
1.0
IB-p-90
100000
15
250
(6,
(3.
1.0
IB-p-95
100000
15
250
(6,
(3,
1.0
IB-p-100
100000
15
250
(6,
1.0
Table Cidustrril-Bennhhee Detailed information about collected dataset
29

---

Vintix: Action Model via In-Context Reinforcement Learning
H. Task-Level Performance
Random Score
Expert Score
Vintix (Normalized)
Task
assembly
45,1 3,3
297,9 24.8
1.04 0.10
basketball
3 = 1,3
558,4 = 69.3
1.02 = 0.11
bin bin-picking
2 0,6
424 = 105,7
0.01 0.01
box-close
78,7 + 11,6
520 + 122,6
-0.04 + 0.03
button-prees
31,3 4,3
642,9 + 13
0.97 0.07
Dutton-press-topdowy
31,6 + 9,8
483,7 41,7
0.94 0.14
satton-press-topdeeee
30,8 1
497,9 = 31,2
0.97 0.07
button-press-wall
10,3 4,4
675,9 15,2
1.00 0.04
butfee-button
33.1 + 7,8
731,4 = 29,1
1.00 0.06
coffee-pull
4,3 0.5
294 103,6
0.07 0.20
coffee-push
4,3 + 0.7
564,8 + 106.7
1.01 0.20
dial-turn
35,8 42,8
788,4 82,4
0.99 0.07
disassemble
40,6 + 6,8
527,3 64,6
1.00 0.12
door-close
5,5 + 1,4
541,9 25,6
1.02 = 0.05
door-lock
115,7 30,1
805,5 42,7
0.35 0.36
door-open
58,9 + 10,1
587,6 18,4
0.99 0.05
door-unlock
99.8 17,8
803,1 19,6
0.21 0.26
drawer-close
147,4 +: 291.1
868,4 + 3,3
0.96 0.01
drawer-open
126,3 23,3
492,9 2,5
1.00 0.01
faucet-close
260,9 + 26,4
755,4 20,8
0.99 0.03
faucet-open
242,7 1 24.5
746,9 12,9
0.97 0.03
hammer
93,9 + 9,7
687,6 49,9
0.97 0.12
hand-insert
2,7 + 0,8
738,2 + 36,6
0.19 0.20
handle-press
73,6 76,1
812,8 170.8
1.00 0.15
handle-press-side
59,4 + 26,8
860.3 = 32,8
0.99 0.02
handle-pull
9,5 6.5
703.3 15.2
1.00 0.02
handlc-pull-side
2,1 0,5
495,3 49,3
1.00 0.11
lever-pull
66,9 16,9
575,5 67.9
0.96 0.16
peg-insert-side
1,8 0.3
330,1 + 159,4
1.01 0.40
peg-unplug-side
4,6 2,5
528,8 + 87,1
0.75 0.34
pick-out-of-hoo
+ 49,9
401,1 105,8
0.91 0.20
pick-place
1,5 + 0,9
423,6 + 97.5
0.98 0.22
pick-place-wall
0::
515,4 + 162.5
1.04 0.16
plate-slide
76,5 12
531.1 + 152
0.99 0.34
plate-slide-bn
33,8 1 9,5
719,5 1 88
0.99 - 0.16
Hate-slideeback-shie
35,9 + 13
727.5 69.5
0.99 0.09
plate-slide-side
22,6 14
693,6 + 82,4
0.83 0.22
push
5,8
747.7 61,5
0.97 0.14
push-back
1,2 + 0,1
396,2 + 109,3
0.95 0.29
push-wall
6,5 1
749,7 11,7
0.99 0.02
reach
154 = 52,8
679,6 + 131,8
1.01 0.26
reach-wall
149 = 35.7
748,2 1 102,1
0.98 0.17
shelf-place
0 0
266.5 30
1.01 0.11
soccer
5,3 3,9
509,3 + 136,1
0.80 0.35
stick-pull
2,8 1,4
529,2 20,9
0.92 0.16
stick-push
2,9 1,1
626 = 46,9
0,90 0.28
sweep
12,4 4. 8,5
518,4 48
0.94 0.12
sweep-into
16,9 + 14,3
799 14,9
1.00 0.02
window-close
58,2 + 15
591, 38,6
0.95 0.17
window-open
42,9 1: 7.6
594,9 56.2
0.96 0.17
Table Random, expert, and Vintix scores for Mcta-Waart World domain. Note that, here, we reported scores in comparison to JAT model.
30

---

Vintix: Action Model via In-Context Reinforcement Learning
Random
Expert
Vintix (Normalized)
Task
Score
Score
Ant-v4
-459,2 + 824,1
6368,2 + 593,8
0,98 + 0,10
HalfCheetah-v4
-299,8 + 74,4
7782,8 = 109,2
0,93 - 0,03
Hopper-v4
16,2 = 8,4
3237,8 = 707,8
0,86 - 0,19
Humanoid-v4
116,5 = 31,6
7527,5 + 38,8
0,97 = 0,00
HumanoidStandup--v
37285,5 + 3178
300990,1 + 47970,1
1,02 - 0,21
InverreiDDuubieennuuluu-
56,2 + 16,1
9332,4 + 498,1
0,65 + 0,47
InvertedPendulum--4
5,6 + 2,1
1000 + 0
1,00 - 0,00
Pusher-v5
-151,9 + 8
-40,1 - 7
1,02 - 0,08
Reacher-v4
-41,7 + 3,4
-5,6 - 2,6
0,98 - 0,07
Swimmer-v4
3 - 11,2
95,5 = 3,6
0,98 = 0,06
Walker2d-v4
3,4 + 6,4
5349,7 = 254,6
1,00 = 0,02
Table 8. Random, expert, and Vintix scores for the MuloCo domain. Note that, here, reported scores in comparison to JAT model.
Random Score
Expert Score
Vintix (Normalized)
Task
shadowhandblockstcck
95,6 + 16
285 + 41,8
1.17 = 0.23
shadowhandbottlccap
110,6 + 17,6
399,9 + 57,6
0.81 = 0.25
shadowhanndatchhabbeass
1,1 + 0,6
65,6 + 14,2
0.17 = 0.32
shadowhandaatcoovezzuddeeen
4,9 + 0,7
34,2 + 6,1
0.92 = 0.24
shadowhanndaatchundeerm
1,7 0,6
25,1 6,1
0.72 = 0.39
shadowhandddoorllssiivaar
1,2 0,5
8,8 + 0,2
0.36 + 0.02
shadoowhannddoorrllsssottwwaaa
941,2 + 43,8
1377,5 = 15,7
-1.27 + 0.01
sladowwhandddooroeennvvaaar
-4,4 41,3
409,9 + 3,2
0.29 - 0.02
shadowhaanddooonpeenotvwaaa
20,4 + 40,9
617,1 + 4,4
0.13 - 0.02
shadowhaandrraspadpplecc
6,8 t 1,6
498,1 + 51,1
0.97 = 0.18
shadowhandkettle
-191,9 + 14,8
54,5 + 4,2
-0.99 + 0.00
shadowlandiffunderamm
-42,7 + 8,4
404 + 10,1
0.95 = 0.03
shadowhandover
2,7 + 0,7
34,6 + 5,8
0.95 = 0.03
shadowhandpen
4,5 + 2,4
186,3 19,5
0.52 + 0.44
shadowhandpushblock
224,7 + 66,5
457,4 + 3,6
0.98 + 0.01
shadowwhandreorientation
127,1 + 434,9
3040,1 + 1986,8
0.89 + 0.66
shadowhandscissors
-23,3 + 16,8
735,6 + 24,8
1.03 + 0.01
shadowhandswwngcup
-414,1 + 27,9
3937,9 + 601,1
0.95 + 0.13
shadowhandswitch
50,4 +/t 12,5
281,3 + 1,1
0.95 + 0.01
shadow/landwooacchudddmrmm
2,2 - 1,1
24,8 + 6
0.03 - 0.03
Table 9. Random and expert scores for Bi-DexHands domain
31

---

Vintix: Action Model via In-Context Reinforcement Learning
Random Score
Expert Score
Vintix (Normalized)
Task
industrial-benchmark--01/
-348,9 + 32,7
-180,7 + 3,1
0.94 + 0.13
industrial-eecchnrkkSS-W
-379,7 + 59
-193,8 + 2,3
1.00 + 0.02
industtial-benchharr--01111
-395,4 + 68,9
-215,1 + 2
1.01 + 0.01
industrial-eenchnaa--151111
-424,4 + 83
-229,8 + 4
1.01 + 0.02
industrial-benchmnrrk20--11
-455,8 + 88,2
-249,4 + 2,2
0.95 + 0.11
industrial-eenchmarr-25--11
-453,6 = 78,8
-272,4 + 5,5
0.95 + 0.11
industrial-benchmarrk30--1111
-480,3 + 76,6
-288,3 + 5,4
0.90 0.10
industrial-eenchhaar:33--111
-508,5 + 87
-314,1 + 6,7
1.00 + 0.03
industria--eenchmatt--0--0
-515,8 + 77,4
-337,8 + 8,8
0.99 +
industrial-ennchmark-455v/
-543,3 + 85,1
-360,9 + 7,4
0.97 + 0.04
industrialbenchhmart-50000
-574,9 + 69,8
-377,9 + 7
0.91 + 0.09
industrial-eecchmark:55551
-597,6 + 69,9
-402,1 + 6
0.99 + 0.01
industrial-eenchhattt-----1
-625,2 + 83,4
-430,3 + 4,8
0.98 + 0.02
industrial-benchmnark65--11
-649,6 + 62,2
-449,8 + 4,1
0.86 + 0.04
industrial-eenchmaatt7000//
-691,7 + 87
-471,1 1 4,2
0.95 + 0.03
industria--encchmart-55---
-717,2 + 72,3
-474,3 + 3,2
0.99 + 0.01
industria-bennchmark-80--//
-757,8 + 105,9
-485,4 + 3,1
0.96 + 0.01
industria-benchhaar-885--/
-812,8 + 154,9
-50,,6 + 2,9
0.98 + 0.01
industtial-benchhaark-999V/
-846,4 + 132,7
-522 + 3,6
0.97 + 0.01
industrialbenchmaa--955-1
-895,5 + 146,5
-54,,8 + 3,3
0.97 + 0.01
industrial-benchmar-110000
-986 + 199
-561,8 + 4,6
0.97 + 0.01
Table 10. Random and expert scores for Industrial Benchmark domain
32

---

Vintix: Action Model via In-Context Reinforcement Learning
I. Comparison with other cross-domain agents
Task
Vintix
REGENT
JAT (Full Dataset)
Ant
0.98 + 0.10
0.17
0.88 + 0.29
HalfCheetah
0.93 + 0.03
0.34
0.89 = 0.03
Hopper
0.86 + 0.19
0.12
0.76 + 0.21
Humanoid
0.97 + 0.00
0.02
0.10 + 0.02
HumanoidStandup
1.02 + 0.21
0.26
0.35 + 0.09
InvertedDDubllePenduumm
0.65 + 0.47
0.02
0.93 = 0.14
InvertedPendulum
1.00 + 0.00
0.06
0.24 + 0.05
Pusher
1.02 + 0.08
0.90
1.00 = 0.05
Reacher
0.98 + 0.07
0.90
0.99 + 0.06
Swimmer
0.98 + 0.06
0.82
1.02 + 0.04
Walker2d
1.00 + 0.02
0.05
0.95 + 0.18
Table Performance Vintix, REGENT on MuJoCo domain tasks.
33

---

Vintix: Action Model via In-Context Reinforcement Learning
Task
Vintix
REGENT
JAT (Full Dataset)
assembly
1.04 0.10
0.83
0.96 0.17
haskethall
1.02 + 0.11
0.68
-0.00 + 0.00
bin bin-picking
0.01 0.01
0.67
0.47 0.52
box-close
-0.04 = 0.03
0.93
0.89 0.39
button-press
0.97 1 0.07
0,62
0.86 1. 0.30
button-press-oopdowr
0.94 0.14
0.62
0.51 0.17
Duuton-ppees--ooddwwwwwws
0.97 0.07
0.62
0.53 0.19
Button-press-wal
1.00 = 0.04
0.94
0.94 = 0.18
coffee-button
1.00 0.06
0.84
0.38 0.41
coffce-pull
0.07 0.20
0.62
0.14 0.27
coffee-push
1.01 0.20
0.18
0.30 0.44
dial-turn
0.99 0.07
0.83
0.95 0.16
disassemble
1.00 + 0.12
2.24
0.17 3.91
door-close
1.02 0.05
1.00
0.99 0.06
door-lock
0.35 0.36
0.85
0.84 0.28
door-open
0.99 + 0.05
0.98
0.99 = 0.10
door-unlock
0.21 0.26
0.90
0.95 0.13
drawer-close
0.96 + 0.01
1.00
0.64 0.30
drawer-open
1.00 + 0.01
0.96
0.98 0.10
faucet-close
0.99 0.03
0.53
0.23 0.18
faucet-open
0.97 0.03
0.99
0.70 0.37
hammer
0.97 0.12
0.95
0.96 0.15
hand-insert
0.19 0.20
0.82
0.93 0.25
handle-pres
1.00 + 0.15
0.99
0.84 = 0.32
handle-press-side
0.99 0.02
0.99
0.01 0.09
handle-pull
1.00 0.02
0.48
0.83 0.25
handle-pull-sine
1.00 + 0.11
0,48
0,50 = 0.49
lever-pull
0.96 0.16
0.19
0.40 0.43
peg-insert-side
1.01 + 0.40
0.70
0.81 0.51
peg-unplug-side
0.75 + 0.34
0.31
0.17 0.32
pick-out-of-hole
0.91 0.20
0.01
0.00 0.00
pick-place
0.98 +: 0.22
0.99
0.32 0.48
pick-placc-wall
1.04 0.16
0.99
0.10 0.29
platc-slide
0.99 0.34
1.00
0.90 0.42
plate-slide-back
0.99 + 0.16
1.00
0.24 +: 0.00
plate-slide-back-tee
0.99 0.09
1.00
0.96 0.17
plate-slide-side
0.83 0.22
0.99
0.16 0.04
push
0.97 + 0.14
0.84
0,94 = 0.21
push-back
0.95 0.29
0.84
0.97 1.29
push-wall
0.99 + 0.02
0.81
0.21 0.30
reach
1.01 0.26
0.99
0.34 0.32
reach-wall
0.98 0.17
0.99
0.81 0.37
shelf-place
1.01 + 0.11
0.96
0.38 0.46
soccer
0.80 0.35
0.61
0.77 0.44
stick-pull
0.92 0.16
0.88
0.92 0.23
stick-push
0.90 + 0.28
0.75
0.48 + 0.48
sweep
0.94 0.12
0.91
0.01 0.04
sweep-into
1.00 = 0.02
0.91
0.99 0.06
window-close
0.95 0.17
0.90
0.67 0.39
window-open
0.96 0.17
0.97
0.98 0.12
Table 12. Performance of Vintix, REGENT and JAT on Meta- World domain tasks.
34


Reviews:
[{"review_id": "DXus73gTeF", "reviewer": "Reviewer_U2Cv", "summary": "This paper proposes a method to train a general ICRL-capable agent following a version of Algorithm distillation (aka noise distillation) across four environment suites (aka domains). Their model architecture (like JAT) makes a complete transition into one token allowing them to expand to larger contexts for ICRL. The trained agent, after it is run from a cold start (with nothing in the context) in the training environments and after it converges (in returns) after many episodes (or shots), results in a conditioned agent that can improve performance on training metaworld and mujoco tasks over JAT. Vintix also demonstrates generalization to parameter varaiations in the environments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Based on your feedback, we believe the main points of discussion can be distilled into the following:\n\n- **Does Adaptability (or inference-time learning) happen, or is it just Task Identification (context conditioning)?**\n- **How to scale to non-vector-based/proprioceptive modalities, e.g., images?**\n\nBelow, we present our responses and the experiments we conducted during the limited rebuttal phase. If there’s anything we’ve overlooked or if further clarification is needed, please let us know—we’ll respond promptly within the available timeframe.\n\n---\n\n### **Does Adaptability (or Inference-Time Learning) Happen, or Is It Just Task Identification (Context Conditioning)?**\n\nYou’re right to highlight the challenge of disentangling adaptability (i.e., inference-time learning) from task identification via conditioning. This is a recurring issue in meta-learning research, especially when attempting to distinguish in-weights learning from in-context learning at scale.\n\nTo ensure that our results are not simply a consequence of task identification — and to explore the trade-off between in-context learning and in-weights learning — we perform several key ablations of our proposed approach:\n\n- **Algorithm Distillation (AD) with Masked Rewards** — The model is trained on noise-induced improvement trajectories, but with rewards masked out during training.\n- **Expert Distillation (ED)** — The model is trained exclusively on expert-level trajectories, similar to JAT/GATO-style behavior cloning.\n\nThese ablations help shed light on the contribution of improvement trajectories and reward signals, thereby going beyond task identification and supporting the case for learning dynamics.\n\n---\n\n**Vintix vs. Expert Distillation**\n\nWe trained a model with the same architecture as Vintix (transformer backbone, encoders, loss function), but only on expert demonstrations (ED). We evaluated both ED and Vintix using the cold-start procedure (lines 238–239 in the paper).\n\nResults ([link](https://postimg.cc/N59JBH9n)) show that *Expert Distillation underperforms relative to AD* on both domains. ED reaches an average expert-normalized score of 0.8 on MuJoCo and Meta-World, while AD achieves 0.97 and 0.95, respectively.\n\nThis suggests that the *structure of policy improvement* in the dataset is valuable for enabling self-correcting behavior and high performance. Notably, ED’s performance in MuJoCo stays flat across different shot counts, but improves in Meta-World as more episodes are provided.\n\nThis implies ED partially learns task identification—especially challenging in Meta-World, where a shared encoder spans tasks. Our findings suggest ED requires ~4 episodes to infer the current task. However, *even after identifying the task, ED fails to reach AD’s asymptotic performance, despite having the same data volume*.\n\n---\n\n**Vintix vs. Algorithm Distillation with No Rewards**\n\nIn the second experiment, we aim to assess whether Vintix is utilizing both structure of policy improvement and the reward function—in other words, whether the reward signal contributes to the model's ability to self-improve during inference.\n\nTo investigate this, we re-trained the Vintix model without access to rewards and compared its performance to the original version of Vintix (trained with rewards), using the previously described cold-start inference procedure on training tasks from the MuJoCo and Meta-World domains.\n\nThe evaluation results ([link](https://postimg.cc/N59JBH9n)) suggest that the reward signal plays a role in achieving performance comparable to the demonstrator. *AD trained with masked rewards performs worse asymptotically across both domains and shows slower convergence on the Meta-World domain.*\n\nThese findings indicate that reward feedback is essential for effective self-improvement during inference, supporting the view that supervised training on a dataset containing policy improvement and rewards enhances the model’s in-context reinforcement learning capabilities.\n\n---\n\n### **How to Scale to Non-Vector-Based/Proprioceptive Modalities (e.g., Images)?**\n\nVintix currently operates on proprioceptive inputs, extending it to image-based observations is great direction for future work. To extend it to image-based observations with parametric variations, a natural approach would be to use a vision foundation model (e.g., BLIP) to encode visual input on the environment side.\n\nVariations such as object color or distractors can be applied prior to encoding and will be captured in the resulting image embeddings. These can then be passed to Vintix’s MLP encoder as n-dimensional inputs.\n\nThis preserves the existing pipeline structure: only the image encoding is delegated to the environment. As before, training requires noise-distilled policy improvement data; inference remains unchanged.", "claims_and_evidence": "Pros:\n\n* The work is very well written. It takes a simple idea (AD^{\\epsilon}) and scales it to multiple domains for the first time in ICRL.\n* The results on improved performance in both training environments and parametric variations of those is very interesting. In particular, with the advent of test-time compute, this work on ICRL appears even more timely.\n* The admission of only early signs of ICRL in new unseen tasks is a great pro!\n* The paper's claims are supported by adequate evidence.\n\nCons: \n\n* I am not certain if it is the adaptability of Vintix that improves performance over JAT in the trained environments or the context it is conditioned on before it's score is calculated. Like the inference time LLM works, is there a way to control the amount of time Vintix takes to converge to identify if taking longer results in better performance (because of a larger conditioning context)?\n* It does appear that all domains do not have image-based observations. Do the authors have a way to scale generalization to new environments with parameteric variations when the observations have images? Does this require expert demonstrations like that seen in ICIL methods (like REGENT)?", "methods_and_evaluation_criteria": "Yes. But, I have raised a couple of questions on evaluation in the cons above.", "theoretical_claims": "No proofs.", "experimental_designs_or_analyses": "Yes, I checked all experimental analyses. They are sounds. Please see \"Claims And Evidence\" for the detailed pros and cons.", "relation_to_broader_scientific_literature": "This work improves the frontier of ICRL. While some works in ICIL like REGENT have shown generalization to new environments, they do require few expert trajectories to retrieve a suitable context from. This work, building on ICRL methods like AD and AD^{\\epsilon} (and going against the trend of ICRL methods like DPT) demonstrates early signs of success in generalization to new tasks and strong signs of generalization to parametric variations of tasks/envs. The ability to modulate the amount of ICRL until convergence would allow for a sort of test-time scaling here that would be great (if the authors can do something like that).", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Addressed in \"Claims And Evidence\".", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see cons in \"Claims And Evidence\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9dwhGzzzQ", "reviewer": "Reviewer_5G9i", "summary": "This work explores In-Context Reinforcement Learning (ICRL) as a method for developing generalist agents that can learn through trial-and-error during inference. The proposed approach is built on Algorithm Distillation (AD), a prior in-context RL work. Specifically, based on AD, the authors adopt continuous noise distillation approach to construct the training datasets. In addition, instead of training on single domains, the proposed approach is trained on four different domains with a total of 87 tasks and 1.6M episodes. The results suggest that the proposed approach could learn an agent that can self-correct and improve its performance during test time across the four tested domains. The paper concludes that ICRL is a potential approach for creating scalable generalist decision-making systems.\n\n## update after rebuttal\nThe authors' response has only partially addressed my concerns. I still believe that the paper's main contribution may not be substantial. Continuous noise distillation and cross-domain datasets may not be significant enough advancements. Additionally, the lack of comparison with the vanilla AD remains a concern. While I acknowledge the difficulty of collecting all training data in a short time frame, comparing with vanilla AD using a subset of tasks could still be insightful. Therefore, I keep my score unchanged (weak reject).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and effort you devoted to reviewing our paper. We have identified the following key issues for further discussion:\n\n**Is it possible to include experiments comparing the proposed approach with vanilla AD to provide stronger evidence?**\n\nVintix diverges from standard AD through noise-distilled data methodology, replacing RL learning histories because:\n- [Zisman et al. (2024a)](https://arxiv.org/abs/2312.12275) link noise distillation to in-context learning emergence\n- It improves trajectory control by avoiding lengthy, sample-inefficient PPO histories, which demand large computational resources and exhibit unstable convergence\n- It permits low-cost dataset generation via open-source expert policies\n\nVintix functions as a standard AD framework applied to cross-domain, noise-distilled datasets. Recollecting dataset for vanilla AD requires retraining 1157 (87 tasks * 13.3 trajectories) RL agents—computationally unfeasible for rebuttal timelines\n\n**What evidence might be presented to demonstrate that standardizing reward functions with task-specific factors positively influences model performance?**\n\nWe conducted a hyperparameter-tuning experiment on the Humanoid task. Over 50 hyperparameter configurations were evaluated. [Figure 1](https://postimg.cc/PPVCyf8t) presents the results, showing that using a non-standard reward scale yielded better scores. Before training Vintix,we performed analogous sweeps on each task. We don’t claim that modifying reward scales always leads to better models,but observed performance gains under our chosen settings\n\n**What is the key novelty of the paper that makes its technical contribution sufficiently substantial?**\n\nThis work advances beyond noise integration in distillation by offering empirical principles for selecting ε-decay functions. We show noise-augmented distillation enables dynamic self-correction during inference, particularly in tasks with complex dynamic, high-dimensional actions, and partial observability. Also, we have open-sourced collected datasets for In-Context RL. To our knowledge, it was previously undertaken by the JAT, and their dataset comprises expert trajectories only.\n\nOur model training prioritized challenging domains over simplistic ones. The MetaWorld ML45 benchmark, a prominent robotic manipulation suite with 490+ citations since 2024, presents significant difficulty: state-of-the-art online MetaRL methods achieve only 0.4 success rates ([Shala et al., 2025](https://openreview.net/pdf?id=UENQuayzr1)). Industrial Benchmark introduces synthetic tasks simulating industrial optimization challenges, characterized by complex dynamics, heteroscedastic noise, delayed rewards and partial observability. Bi-DexHands addresses robotic grasping (a critical capability for human-environment interactions) through tasks testing RL and MetaRL algorithms against high-dimensional action space limitations. To our knowledge, no prior work has applied methods beyond behavioral cloning on such challenging cross-domain datasets\n\n**What is the performance of ADε in comparison with other methods?**\n\nComparison with other offline MetaRL approaches is indeed a valuable experiment. Vanilla AD is discussed in the first paragraph. AMAGO is an online off-policy MetaRL approach limited to a single domain, while Vintix is offline and trains on fixed data. AMAGO is both difficult to replicate and challenging to adapt to the offline setting. DPT is unable to learn in-context in partially observable MDPs (Appendix H of [Nikulin et al.(2024)](https://arxiv.org/abs/2406.08973)). Bi-DexHands and Industrial Benchmark are partially observable\n\nWe implemented DPT and trained it on MuJoCo. We also trained Vintix only on MuJoCo, using the same transformerparameters. [Figure 2](https://postimg.cc/3WfFzTsw) shows that DPT’s training loss exceeded Vintix’s and displayed spikes. Validation followed the same procedure with an empty initial context. As shown in [Figure 3](https://postimg.cc/m14Ws9zQ),DPT performs poorly, necessitating further tuning.No prior work, to our knowledge,has applied DPT to MuJoCo tasks, leaving it for future study\n\n**What is the difference between \"group identifier\" and \"task identifier\"?**\n\nThe task identifier is a unique ID assigned to each task in the dataset. The group identifier is an ID indicating whether a group of tasks shares the same observation and action spaces (in terms of dimensionality and the semantic meaning of each channel)\n\n**Why were some JAT experts retrained to collect the dataset?**\n\nDuring data collection, we assessed JAT demonstrations and found several low-performing experts (zero success rates). Prioritizing dataset quality and model evaluation against expert benchmarks, we retrained selected experts. However, fair comparison remained possible through score normalization against updated benchmarks, intentionally lowering Vintix’s scores relative to JAT. This imposed stricter evaluation conditions on our model", "claims_and_evidence": "The reviewer found that some of the claims made in the paper lack sufficient evidentiary support. Notably, the paper does not provide experimental results demonstrating that the proposed approach outperforms the vanilla Algorithm Distillation (AD) method. This comparison is crucial to evaluating the effectiveness of the proposed approach.\n\nFurthermore, certain claims lack concrete experimental validation. For instance, in Section 2.3.2, the authors assert that standardizing reward functions using task-specific factors significantly enhances model performance. However, this assertion is not substantiated by any empirical evidence in the presented work. The absence of such evidence makes it challenging to assess the validity of this claim.\n\nTo strengthen the paper, the authors should consider conducting additional experiments that directly compare the proposed approach with the vanilla AD method. This would provide concrete evidence regarding the relative performance of the two approaches. Additionally, the authors should provide empirical evidence to support claims regarding the impact of specific modifications, such as the standardization of reward functions, on model performance.", "methods_and_evaluation_criteria": "The reviewer has several concerns regarding the technical contributions of the paper. The two primary technical components presented are continuous noise distillation and cross-domain dataset and training. The reviewer finds neither contribution to be sufficiently substantial. Continuous noise distillation is viewed as a minor extension of the discrete noise distillation introduced in Zisman et al. (2024a), differing only in the application of uniform random noise. The use of a cross-domain dataset and training, while potentially interesting, lacks sufficient novelty or complexity to be considered a significant contribution on its own.", "theoretical_claims": "No formal theoretical claim is presented.", "experimental_designs_or_analyses": "The reviewer finds that the evaluation of the proposed approach is lacking, as there are no baseline comparisons included in the main experiments (Figure 4). The inclusion of baselines such as algorithm distillation [a], AMMAGO [b], and Decision-Pretrained Transformer [c] would enable a more thorough evaluation of the effectiveness of the proposed approach by providing a point of reference and comparison.\n\n[a] In-context Reinforcement Learning with Algorithm Distillation, Laskin, 2022\n\n[b] AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents, Grigsby, 2024\n\n[c] Supervised Pretraining Can Learn In-Context Reinforcement Learning, Lee, 2023", "relation_to_broader_scientific_literature": "This paper is strongly related to algorithm distillation (AD) [a]. It applied AD to a cross-domain setting.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Please see the above discussion.", "other_comments_or_suggestions": "Line 174 - 175 (left) reads “...has access only to the dimensionality-based group identifier, but not to an individual task identifier.” Please clarify what \"group identifier\" and \"task identifier\" mean, and provide examples of each to illustrate the distinction.", "questions_for_authors": "In the comparison with JAT, the authors mentioned that data collection for the proposed approach uses improved expert performance, which is not used in the baselines. Please explain why the same expert cannot be used for both the baseline and the proposed approach to ensure a fair comparison.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYFaW35LKm", "reviewer": "Reviewer_bYMy", "summary": "This paper explores the potential of In-Context Reinforcement Learning (ICRL) for developing generalist agents capable of learning and adapting through trial-and-error interactions at inference time. The authors present Vintix, a fixed, cross-domain action model that leverages the Algorithm Distillation (AD) framework to learn behaviors across various tasks. Vintix uses Continuous Noise Distillation to collect training data from multiple domains. The model demonstrates significant self-correction capabilities, achieving near demonstrator-level performance on multiple training tasks and adapting to parametric variations at inference time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. Based on your feedback, we believe the central point of discussion is the practicality — in a broad, real-world sense — of the proposed approach. In particular, we address the following raised topics:\n\n- **Do the selected domains represent a broad spectrum of challenging, real-world-relevant tasks?**\n- **Does the proposed data collection method offer advantages over existing approaches (e.g., learning histories from PPO or optimal action labels as in DPT)?**\n\nWhile our work focuses on simulated environments and does not yet make claims about real-world transfer, we argue below that both our choice of domains and our data collection methods are well-justified. Moreover, we believe they provide a strong foundation for the continued development of action models within the framework of in-context reinforcement learning.\n\n---\n\n### **Do the selected domains encompass a broad spectrum of difficult and real-world tasks?**\n\n> The chosen tasks for experimental validation are overly standardized and fail to encompass a broad spectrum of extreme or unconventional scenarios. This narrow focus could severely limit the applicability of the method in complex real-world environments.\n\nWhen benchmarking models of this kind, it is important to balance standardized environments—which ensure reproducibility and fair comparison—with task suites that reflect real-world complexity. A central goal of our work is to contribute datasets and results to the broader community, laying a foundation for further scaling and development.\n\nThis objective imposes constraints on domain selection: environments must be open-source and widely adopted by the research community. While MuJoCo is included as a field standard, the remaining domains were chosen for their practical relevance and the complexity of the challenges they present:\n\n- **Meta-World ML45**: A highly challenging benchmark where state-of-the-art online Meta-RL algorithms achieve a success rate of just 0.4 [(Shala et al., 2025)](https://openreview.net/forum?id=UENQuayzr1). It is widely used and practically motivated, with over 490 citations since 2024.\n\n- **Industrial Benchmark**: A synthetic suite that models industrial optimization problems with complex dynamics, heteroscedastic noise, delayed multi-objective rewards, and partial observability. **Notably, the benchmark was explicitly designed to test RL algorithms under conditions resembling real-world industrial control problems.**\n\n- **Bi-DexHands**: Grasping is a core challenge in robotics, critical for tasks in human-centric environments [(Billard et al., 2019)](https://www.science.org/doi/10.1126/science.aat8414). Despite extensive research, it remains difficult due to the high-dimensional action space. Bi-DexHands offers a diverse suite of grasping tasks, and is recognized as it was accepted to last year’s NeurIPS Benchmarks and Datasets track last year.\n\n---\n\n### **Does the proposed collection method offer advantages over the existing ones (e.g, learning histories from PPO or optimal action labels as in DPT)?**\n\n> The paper fails to clearly elucidate the practical advantages of Continuous Noise Distillation in real-world scenarios. Besides, the paper lacks detailed discussion regarding the potential impact and value of this method in tangible applications.\n\nContinuous Noise Distillation significantly simplifies dataset collection for Algorithm Distillation. In vanilla AD, a new RL agent must be trained for each learning history, often requiring millions or billions of steps. This process is time-consuming, unpredictable, and limits scalability.\n\nIn contrast, Continuous Noise Distillation enables efficient, controllable data collection using only a demonstrator policy. Users can set the length of each learning history, reducing computational overhead and enhancing practicality in time- and cost-sensitive scenarios.\n\n> The dataset construction and utilization process is complex and potentially cumbersome, posing substantial challenges for practical implementation. This intricate dependency on specific steps and environments restricts the generalizability and reproducibility of the method.\n\nWhile dataset construction does require some effort, it remains relatively simple compared to other In-Context RL methods. It only needs demonstrator policies and environment access. By contrast, [AD](https://arxiv.org/abs/2210.14215) requires full RL learning histories, and [DPT](https://arxiv.org/abs/2306.14892) needs expert-provided target actions. Generalist agents like [JAT](https://arxiv.org/abs/2402.09844), [GATO](https://arxiv.org/abs/2205.06175), and [Baku](https://arxiv.org/abs/2406.07539) also rely on expert demonstrations. Although simplifying data collection for In-Context RL is an important direction, it is beyond the scope of this work.", "claims_and_evidence": "Clear.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "The chosen tasks for experimental validation are overly standardized and fail to encompass a broad spectrum of extreme or unconventional scenarios. This narrow focus could severely limit the applicability of the method in complex real-world environments.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n\t1.\tThis paper proposes a novel data collection strategy, which incrementally reduces the uniform noise injected into demonstrator policies. This extended mechanism enhances data collection efficiency and makes algorithm distillation more feasible in reward-oriented reinforcement learning.\n\t2.\tUsing an improved transformer architecture for model training, detailed optimizations ensure efficient learning and adaptability. This standardized training process enhances the model's generalization and applicability.\n\t3.\tExperimental results demonstrate that the Vintix model can self-correct using contextual information at inference time, progressively reaching near-demonstrator levels. It shows strong cross-domain generalization capability.\n\t4.\tThe paper constructs a large cross-domain dataset covering 87 tasks across 4 domains.\n\nWeaknesses：\n\t1.\tThe chosen tasks for experimental validation are overly standardized and fail to encompass a broad spectrum of extreme or unconventional scenarios. This narrow focus could severely limit the applicability of the method in complex real-world environments.\n\t2.\tThe paper fails to clearly elucidate the practical advantages of Continuous Noise Distillation in real-world scenarios. Besides, the paper lacks detailed discussion regarding the potential impact and value of this method in tangible applications.\n\t3.\tThe dataset construction and utilization process is complex and potentially cumbersome, posing substantial challenges for practical implementation. This intricate dependency on specific steps and environments restricts the generalizability and reproducibility of the method.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T12minueoU", "reviewer": "Reviewer_LDFi", "summary": "The paper introduces Vintix, a cross-domain action model capable of in-context reinforcement learning (ICRL). \n\nThe key contributions include: \n\n(1) Continuous Noise Distillation, extending existing work to continuous action spaces; \n\n(2) a cross-domain dataset spanning 87 tasks across 4 environments (Meta-World, MuJoCo, Bi-DexHands, Industrial-Benchmark); and \n\n(3) empirical evidence showing the model's ability to self-correct during inference through many-shot ICRL. The authors demonstrate that their model outperforms previous approaches like JAT on Meta-World (+32%) and MuJoCo (+13.5%), while also showing some ability to adapt to parametric variations in environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. Based on your feedback, we have identified these key points:\n\n1. **How does Vintix compare to the latest action models?**\n2. **Does Vintix learn to adapt or does it mimic expert through task identification?**\n3. **How does the quality of dataset impact performance?**\n4. **Providing more detailed supplementary materials.**\n\nWe include responses and experiments conducted within the rebuttal period. If anything is missing, feel free to reach out — we will respond as promptly as possible\n\n---\n\n### **How does Vintix compare to the latest action models?**\n\nTo broaden the comparison beyond JAT, we include expert-normalized scores for REGENT on MuJoCo and MetaWorld — the two domains shared across all works ([Table 1](https://postimg.cc/1n863r8d), [Table 2](https://postimg.cc/FfhhDCrt))\n\nVintix outperforms both JAT and REGENT in normalized scores but lags behind REGENT on entirely unseen tasks. Notably, unlike REGENT, Vintix received no demonstrations for these tasks and operated in a fully cold-start setting.\n\n---\n\n### **Does Vintix learn to adapt or does it mimic expert through task identification?**\n\n**Exp 1**\n\nWe evaluated whether AD-style training on noise-distilled trajectories outperforms behavioral cloning with context, using a model identical to Vintix but trained solely on expert data. Both models were evaluated using the procedure described in lines 238–239 of the paper.\n\n[Figure 1](https://postimg.cc/XG777HmK) shows that ED converges to ~0.8, while AD reaches 0.97 (MuJoCo) and 0.95 (MetaWorld). This highlights the importance of policy-improvement data for self-correcting behavior. ED shows stable performance in MuJoCo and surprisingly improves with more shots in MetaWorld, likely due to task inference still occuring under a shared encoder (requiring 4 episodes in context) but still underperforms AD despite being trained on the same amount of data.\n\n**Exp 2**\n\nSecondly, we examine whether Vintix relies on the reward signal for self-improvement during inference. To test this, we re-trained Vintix without access to rewards and compared it to the original model using the cold-start inference on tasks from the MuJoCo and MetaWorld domains.\n\n[Figure 1](https://postimg.cc/XG777HmK) shows that reward feedback is crucial: the masked-reward variant performs worse asymptotically in both domains and converges more slowly on MetaWorld. These results suggest that training on data reflecting policy improvement is essential for enabling in-context reinforcement learning\n\n---\n\n### **How does the quality of dataset impact performance?**\n\nTo evaluate the impact of dataset quality, we collected a MuJoCo dataset using an untuned noise decay schedule, which led to non-smooth policy improvement on some tasks. We re-trained the AD model on both datasets and compared them using the cold-start evaluation\n\n[Figure 2](https://postimg.cc/HVQGR0PT) shows that certain tasks were strongly affected by poor decay scheduling, while others remained stable. [Figure 3](https://postimg.cc/xXtZV6GV) indicates that models trained on lower-quality data exhibit weaker asymptotic performance\n\nDespite reaching an expert-normalized score of 0.81, these findings highlight the importance of using high-quality data with smooth, progressive improvement to maximize performance\n\n---\n\n### **Providing more detailed supplementary materials**\n\nWe provide a link to an [anonymous repo](https://anonymous.4open.science/r/vintix-rebuttal-icml-2025-7F33 ) with Vintix code and [training dataset]( https://tinyurl.com/426ckafn). Extra supplementary material is available in paper’s appendix\n\n---\n\n**Discussion of sequence based RL approaches** With the rise of Transformers for modeling sequential data, several works ([1](https://arxiv.org/abs/2106.01345), [2](https://arxiv.org/abs/2106.02039)) formulated MDP as a causal sequence modeling problem. ([1](https://arxiv.org/abs/2106.01345)) focused on reward conditioning treating each MDP element as a separate token, while ([2](https://arxiv.org/abs/2106.02039)) applied beam search over discretized SAR tuples.\n\nSubsequent research has expanded this area by making models that maximize returns ([3](https://arxiv.org/abs/2405.08740)), adapting DT to online learning ([4](https://arxiv.org/abs/2202.05607)), and replacing the Transformer with SSM backbones like Mamba ([5](https://arxiv.org/abs/2406.00079)).\n\n**On suboptimal demonstrators** We did not filter out failed expert trajectories to avoid biasing the dataset, particularly in cases where failures may be correlated. Instead, we addressed noisy behavior by further training the demonstrators.\n\n**On generalization to unseen tasks**\n\n1. *Scaling the dataset* AD benefits from a large number of tasks to generalize effectively. We plan to expand the dataset with new domains.\n2. *Domain-invariant architecture* Using VLA-like models to map modalities into a shared embedding space may improve cross-domain transfer and reduce reliance on task identification.", "claims_and_evidence": "The core claim about ICRL, self-correction capabilities are supported by convincing experiments, and showing good performance with increasing context. \n\nAnd the model structure", "methods_and_evaluation_criteria": "The methods are appropriate for investigating in-context adaptation. The Continuous Noise Distillation technique is a sensible extension to continuous domains, and the benchmark environments are standard in the field. However, the evaluation could be strengthened by comparing against more contemporary baselines beyond JAT, and by more rigorously analyzing the quality of the collected datasets and their impact on performance.", "theoretical_claims": "No formal theoretical claims or proofs are presented in the paper.", "experimental_designs_or_analyses": "The experimental design generally sound. The authoers use appropriate metrics and provide confidence intervals. The staged approach to evaluation (self-correction on training tasks, comparison to baselines, generalization to parametric variations and new tasks) makes sense.\n\nHowever, this work lack ablation studies or discussions about dataset quality, cumulative learning and task diversity. Authors should discuss the these three key notes. I think it could be pretty important for future works and can certificant current findings in RL and LLM.", "relation_to_broader_scientific_literature": "This work builds upon Algorithm Distillation (Laskin et al., 2022) and extends it to multiple domains and continuous action spaces. It relates to the broader trend of in-context learning in LLMs, applying similar principles to RL. The paper properly situates itself relative to related work in Meta-RL (both memory-based approaches like Duan et al. and offline methods), generalist agents like Gato, and learned optimizers.", "essential_references_not_discussed": "This paper lacks discussion of sequence modeling approaches to RL like decision transformer, trajectory transformer and other works.", "other_strengths_and_weaknesses": "Strengths:\n\nThe cross-domain approach is ambitious and moves beyond single-domain studies common in previous work.\nThe open-sourced datasets could be valuable to the community.\n\nWeaknesses:\n\nThe paper lacks sufficient analysis of dataset quality and its impact on performance.\nLimited generalization to truly new tasks raises questions about whether the model is actually learning adaptation strategies or merely memorizing task-specific behaviors.", "other_comments_or_suggestions": "The paper would benefit from a clearer discussion of the limitations of the approach, particularly regarding generalization. A more rigorous analysis of what constitutes \"learning\" in this context would strengthen the theoretical foundation. I think this work more like gato.", "questions_for_authors": "How would you distinguish the learning happening in your approach from standard supervised learning on expert trajectories? What evidence suggests that your model is actually performing reinforcement learning rather than just mimicking behavior?\nCan you provide more detailed analysis of the quality of the datasets used for training and how this affects performance? Was there any filtering of suboptimal demonstrations? \nThe limited generalization to new tasks seems problematic for a method focused on adaptation. What modifications might improve zero-shot transfer to entirely new tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethics_expertise_needed": ["Other expertise"]}}]

Author rebuttal:

Meta-review: This paper introduces an extension of a prior work on noise distillation for In-Context Reinforcement Learning (ICRL). The majority of the reviewers appreciated comprehensive evaluation across multiple continuous control domains, achieving strong performance on trained tasks and the potentially valuable contribution of its large-scale dataset generated via continuous noise distillation. There was a suggestion about the lack of comparison to vanilla AD. Although I understand AD is computationally expensive, I would encourage the authors to include comparison to AD at least on one task to show the relative strength. Assuming that this is reflected in the camera-ready version, I recommend to accept this paper.
Decision: Accept (poster)