ICML 2025

Vintix: Action Model via In-Context Reinforcement Learning
Andrei Polubarov * 1 2 3 Nikita Lyubaykin * 1 4 Alexander Derevyagin * 1 5 Ilya Zisman 1 2 3 Denis Tarasov 1
Alexander Nikulin 1 6 Vladislav Kurenkov 1 4
Abstract
In-Context Reinforcement Learning (ICRL) rep-
resents a promising paradigm for developing gen-
eralist agents that learn at inference time through
trial-and-error interactions, analogous to how
large language models adapt contextually, but
with a focus on reward maximization. However,
the scalability of ICRL beyond toy tasks and
single-domain settings remains an open challenge.
In this work, we present the first steps toward scal-
ing ICRL by introducing a fixed, cross-domain
model capable of learning behaviors through
in-context reinforcement learning. Our results
demonstrate that Algorithm Distillation, a frame-
work designed to facilitate ICRL, offers a com-
pelling and competitive alternative to expert distil-
lation to construct versatile action models. These
findings highlight the potential of ICRL as a scal-
able approach for generalist decision-making sys-
tems. Code to be released at dunnolab/vintix.
1. Introduction
The pursuit of generalist control and decision-making agents
has long been a major target of the reinforcement learning
(RL) research community (Sutton & Barto, 1998). These
agents are envisioned to handle a diverse set of tasks while
exhibiting adaptation properties such as self-correction and
self-improvement based on reward functions. Notably, tradi-
tional online RL algorithms demonstrate these properties in
narrow domains and can achieve exceptional performance
(Berner et al., 2019; Badia et al., 2020; Schrittwieser et al.,
2020; Baker et al., 2022; Team et al., 2023). However,
their online nature of learning and frequent reliance on
environment-specific training is still considered a challenge
to overcome limiting the scalability (Dulac-Arnold et al.,
*Equal contribution
1AIRI 2Skoltech 3Research Center for
Trusted Artifi...