Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies
Authors: Felix Chalumeau
Daniel Rajaonarivonivelomanantsoa
Ruan John de Kock
Juan Claude Formanek
Sasha Abramowitz
Omayma Mahjoub
Wiem Khlifi
Simon Verster Du Toit
Louay Ben Nessir
Refiloe Shabe
Arnol Manuel Fokam
Siddarth Singh
Ulrich Armel Mbou Sob
Arnu Pretorius
Venue: NeurIPS 2025 oral ()
Keywords: reinforcement learning
inference strategies
complex decision-making
Primary area: reinforcement_learning
TL;DR: Using search strategies at inference-time can provide massive performance boost on numerous complex reinforcement learning tasks, within only a couple seconds of execution time.

Abstract: Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many digital or simulation-based applications allow for an inference phase that utilises a specific time and compute budget to explore multiple attempts before outputting a final solution. In this work, we show that such an inference phase employed at execution time, and the choice of a corresponding inference strategy, are key to breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain up to a 126% and, on average, a 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during execution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. We make all of our experimental data and code available.

Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies
Felix Chalumeau"
Daniel Rajoonarvaniveeoaanntts""""" Ruan de Kock I
Claude Formanek
Sasha Abramowitz
Omayma Mahjoub'
Wiem Khlifi
Simon Du Toit
Louay Ben Nessir
Refiloe Shabe
Noah De Nicola
Arnol Fokam
Siddarth Singh UIrich Mbou Sob
Arnu Pretorius 1.2
InstaDeep
SSellenbosch University
Abstract
Reinforcement learning (RL) systems have countless applications, from energy- grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the art RL systems, trained until convergence, hit performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many dig- ital simulation-based applications allow for inference phase that utilises a specific time and compute budget to explore multiple attempts before out- putting final solution. this work, we show that such an inference phase employed at execution time, and the choice of corresponding inference strat- egy, are key breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain to a 126 % and, on average, 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during exe- cution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. Our experimental data and code are available https: //sttes///ete.eeeeeeeeseeeeeeses httsssttsssgggll....//e wtt.view.oo.ee/e/eeeeeeeee http:/infeeennccce inwfrrenceesttratggeess... strategies-rl.
126.4%
85.5% 86.8%
53.3%
47.9%
42.3%
32.3%. 35.0%
36.5%
21.8% 22.3%
10.4% 14.0%
Tasks
Figure 1: Improvement from using inference-time search zero-shot state-of-the-art. Across 17 complex reinforcement learning tasks, we obtain consistent and significant performance gains using only 30 second search budget during execution.
"qqual contribution. Corresponding author: cha.lunccueinstaddppp.om
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Learning to solve sequential decision-matee tasks is a central challenge in artificial intelligence (AI), with far-reaching applications ranging from energy-grid optimisation (Ahmad et al., 2021) and autonomous logistics (Laterre et al., 2018) to molecular discovery (Olivecrona et al., 2017) and drug design (Popova et al., 2018). Complex sequential real-world problems that cannot be solved by traditional optimisation techniques are inherently complex and require navigating high-diment sigh-dimensional solution spaces. Reinforcement Learning (RL) presents a promising avenue to improve our capacity to find efficient solutions to these problems, but despite stunning progress over the past decade, such as human-level performance in Atari games (Mnih et al., 2015). defeating the world champion in the game of Go (Silver et al., 2016) or aligning Al systems with human preferences (Stiennon et al., 2020), current approaches are facing challenges that prevent their common deployment in most real-world systems (Dulac-Arool) et al., 2020).
A major source of this difficulty lies in the combinatorial nature of many decision-making tasks. As the problem size increases, the space of possible solutions grows exponentially (Karp, 1975). In multi-agent systems, the challenge compounds: agents must coordinate in environments where only partial information is available, the joint action space is combinatorial, and optimal behaviour depends on precise interaction with other agents (Bernstein al., 2000; Canese et al., 2021). These properties make it fundamentally difficult to rely on the zero-shot performance of a trained policy, even if that policy was optimised convergence on representative training distribution. This causes the gap between zero-shot performance and optimality to grow substantially with increasing complexity (see Fig. 1).
However, numerous practical applications are not restricted to producing a single zero-shot solution. Instead, inference is often permitted to take place over a few seconds, minutes or hours, with a given computational resource. Furthermore, models and simulators are often accessible and very efficient (e.g., energy grid management, train scheduling, package delivery, routing, printed circuit board design) and provide either an exact score or a very accurate approximation. In other applications where the gap reality may be larger (e.g.. protein design, roboties), improving the solution under the simulated score can still arguably provide significant improvement towards the real objective (Hayes et al., 2025; Dona al., 2024; Hundt et al., 2019; Rao et al., 2020).
This opens up an opportunity: rather than relying on a single attempt of the trained policy, the time budget and compute capacity can be leveraged to actively search for better solutions using multiple attempts, following an inference-time strategy. For instance, progressively building a tree of possible solutions, adapting the policy using outcomes of past attempts. Even straightforward strategies can provide significant performance improvement with low time cost. For instance, generating a large batch of diversified solutions in parallel, using stochastic sampling, rather than a single greedy solution: given a modern GPU, this enables to produce hundreds of solutions, for the same wall-clock time, enabling massive exploration no time cost.
These strategies are rarely emphasized in existing benchmarks (Papoudakis et al., 2021; Mahjoub et al., 2025), and many practitioners invest months of research trying improve the zero-shot performance of their models on scenarios where only marginal improvement may still be achieved. Whereas they could unlock performance gains from inference-tine search, at negligible wall-clock time cost with moderate compute capacity (Fig. 1).
Research in RL for Combinatorial Optimisation (CO) has produced efficient inference strate- gies (Bello* et al., 2017; Hottung et al., 2022; Choo et al., 2022; Chalumeau et al., 2023b), often referred to as active search, or online adaptation methods. However, their empirical study is still limited to a few problems, a narrow range of budget settings (Chalumeau et al., 2025), and barely no insight on their scaling properties. In the multi-agent case, there is no study on inference strategies for collaborative teams: most work on team adaptation focus on Ad Hoc Teamwork (Mirsky et al., 2022; Wang et al., 2024a; Ruhdorfer et al., 2025), which is adjacent to our objective. Interestingly, most recent studies about the impact of inference strategies come from the Large Language Model (LLM) literature (Snell et al., 2025; Muennighoff et al., 2025; Wu et al., 2025), where the adequate combination of efficient models with inference strategies is currently state-of- oftte-of-theartt (SOTA).
In this work, we formalise and investigate the role of inference strategies in complex decision-making tasks. To capture the full complexity of tasks described above, we formulate our problem setting as a decentralised partially observable Markov decision process (Dec-POMDP) (Kaelbling et al.. 1998).

---

This is instead of the typical single-agent MDP used in many RL studies. We make this choice for several reasons: (1) it more accurately maps onto many complex real-world problems of interest, (2) Dec-POMDPs subsume MDPs by being strictly more complex (Bernstein et al., 2000), and (3) because of this, we expect our findings to translate to all simpler problem formulations. Within this setting, we provide unifying view of popular inference-strategy paradigms, including policy sampling, tree search (Choo et al., 2022), online fine-tuning (Bello* et al., 2017; Hottung et al., 2022), and diversity-based search (Chalumeau et al., 2023b). Strikingly, we show that across wide range of specifically selected difficult RL problems, inference strategies boost performance on average by 45%, over zero-shot SOTA. Furthermore, in the best of cases, this boost can be large as 126%. All of this, using only a couple of seconds of additional execution time.
Our results call for shift in how RL systems are evaluated and deployed: inference strategies are not a minor post-processing step, but a key performance driver in realistic conditions. This work sets the foundation for a more nuanced view of inference in sequential decision-making and provides the tools to build systems that can scale with compute. All our code and experimental data can be accessed at: https:/ //site////ttss// google www/view/ inf wwweeeesttategieeeee.-
2 Related work
Inference Strategies from RL for CO Beyond naive stochastic sampling, several paradigms have been explored to generate the best possible solution using trained policy checkpoint during inference. Online fine-tuning (Bello* et al., 2017) retrains all policy parameters with RL using past attempts. Hottung et al. (2022) re-trains only subset of the policy's 's parameters to reduce memory and compute overheads, enabling more attempts for a given inference budget, and adds an imitation learning term to the RL term, to force exploration close to the best solution found so far. Macfarlane et al. (2024) also investigates inference-time policy improvement via sequential updates. Choo al. (2022) uses tree search, with simulation guided node estimates under budget constraints, which outperforms Beam Search and Monte Carlo Tree Search (Coulom, 2006). Diversity- methods: inspired by previous diversity-seeking
approaches, like unsupervised skill discovery (Eysenbach et al., 2019; Sharma et al., 2019; Kumar et al., 2020) and quality-diversin (Chalumeau et al., 2023a; Cully and Demiris, 2017). Grinsztajn et al. (2023) introduces an RL objective that trains population of diverse and specialized policies, efficient for few-shot performance. Chalumeau et al. (2023b) uses this objective and encodes the diversity in a continuous latent space that can be searched at inference-time. introducing the SOTA method COMPASS; meanwhile Hottung et al. (2024) uses a similar approach but with discrete encoding space.
These works have introduced most of the inference strategies we consider in this paper, but they fall short on three important aspects that we aim to improve: (i) they evaluate inference strategies on benchmarks where over 95% zero-shot optimality is already achieved, leaving little room for meaningful gains, (ii) these benchmarks rely domain-specific tricks such starting points or instance augmentations; and (iii) methods are compared under a unique budget setting, overlooking the fact that relative performance depends on the available compute and time budget. In addition, their ability to scale with compute remains unexplored, despite being a critical property.
Policy adaptation in Meta-RL and Offline-to-Online RL Meta-RL and offline-to-online RL both design mechanisms for policy adaptation, thereby sharing close conceptual links with several popular inference strategies. COMPASS (Chalumeau et al., 2023b) and VariBAD (Zintgraf et al., 2021) both condition a policy on a latent space and search it to adapt. MEMENTO (Chalumeau et al., 2025) and RL2 (Duan al., 2016) similarly rely memory and learned rule to search the policy space. DIMES (Qiu et al., 2022) explicitly performs meta-RL following MAML's methodology (Finn et al., 2017). While we are not aware of inference strategies directly inspired by the cetiine-to-oning RL literature, Nakamoto et al. (2023) and Mark et al. (2025) introduce mechanisms which could be beneficial at inference time.
is worth highlighting fundamental distinctions between these fields. Inference strategies focus on (i) the solutions found rather than the learned policy, (ii) maximum rather than average performance, and (iii) single instances at test time, whereas most meta-RI. and offinnetooonline RI. methods adapt from distribution another distribution to improve generalisation.
3

---

1. TRAINING PHASE
2. INFERENCE PHASE
Compute & Time budget
Training distribution
New instance
Inference strategies
Stechastic sampling
Tree search
Decision-makinn making policy
Trained policy
Ondine fime- tuniing
COMPAGS search
Figure 2: Numerous applications of RL involve two distinct phases: (1) training phase, typically unconstrained in time and compute, during which policy is optimized over representative distribu- tion of problem instances; and (2) an inference phase, where a limited time and compute budget are allocated to solving new instance. The inference phase is often overlooked, despite its crucial role in complex tasks where partial observability and the combinatorial growth observation and action spaces make good solutions unattainable through zero-shot exccution alone.
Search and adaptation in Multi-Agent RL There is only limited work on inference strategies for MARL. Most work about search and adaptation within MARL focus the challenge of Ad Hoc Teamwork (Yourdshahi et al., 2018; 8: Hu et al., 2020; Mirsky et al., 2022; Wang et al., 2024a; Ruhdorfer et al., 2025; Hammond et al., 2025), often in the form of zero-shot coordination, where agents must generalize to new partners at execution time. While these lines work share some methodological similarities, for instance using diversity-secking training (Long et al., 2024; Lupu et al., 2021) or adapting through tree search (Yourdshahi et al., 2018), they pursue fundamentally different goals and remain orthogonal. In our work, our focus is primarily solving difficult and complex industrial optimisation tasks.
Inference-time compute for LLMs Recent advances in LLMs are closely intertwined with the of inference strategies (Snell et al., 2025; Wei al., 2022: Wang et al., 2024b), and a growing effort has gone into studying their scaling properties (Muennighoff et al., 2025; Wu et al., 2025). However, the typical inference-time setting is usually different from ours. LLMs have very costly forward passes, and cannot access the exact score of their answers, but can approximate them using reward model (Ouyang al., 2022). Most popular strategies for LLMs are designed for few shots, namely sampling and ensembling (e.g., majority voting).
Overall, numerous efficient inference strategies have been proposed in the literature, yet their efficiency under various settings remains unexamined. Multi-agent RL, despite its inherent complexity, rarely considers inference-time search beyond Ad Hoc Teamwork. Moreover, the broader field of decision-making has not systematically studied how inference strategies scale with compute. Our work aims at filling these gaps by extending the evaluation of inference strategies in RI, demonstrating major performance gains over wide range of budget settings with impressive scaling properties.
Finding the best solution for a given time and compute budget
3.1 Preliminaries
We focus RL approaches, and use neural network (policy) that can construct a solution by taking a sequence of actions. This policy is optimised during a training phase and then used during an new
inference phase, along with an inference strategy, to construct the best possible solution problem instance under a given time and compute budget. These two phases, illustrated in Fig. 2, have different assumptions, objectives and constraints, detailed the following paragraphs.
Problem instances We assume that each problem instance can be formulated as a Dec-POMDP
(Kaclbling et al., 1998), defined by the tuple M (N, S, O, Ω A. R, P, 7, H). Here N is the

---

number of agents, S the environment state space, O - - O' the joint agent observation space, S S O the observation function, A = - A' the joint action space, Rt : 8 x A + R the shared reward function, P: S x A + AS the environment transition function, the scalar y E [0. 1] the discount factor and H the finite episode horizon. At each timestep, the environment occupies a state S, which is mapped to a joint partial observation On via S2 The joint action is then sampled following the joint policy, a ~ π @+) and is executed; after which the environment transitions to the next state 8,+1, following P( 8t, az), and the team receives a reward = R(s,, at).
Training Phase We assume a distribution of problem instances D, that can be sampled from during training. The joint policy Te. parameterised by 0, is used to construct a solution sequentially by taking joint actions conditioned on the joint observation at each timestep. We use RL train this policy to maximise the expected return obtained when building solutions to instances drawn from the distribution D over a horizon H: (TT) = Ep Σ 7 R(5;, ae)
This training objective corresponds to single attempt (zero-shot) Ideally, this objective should an- ticipate the multiple attempts allowed at inference, but this is hard to scale. Recent works incorporate such few-shot objectives (Grinsztajn et al., 2023; Chalumeau et al., 2023b, 2025), but none can yet scale beyond 200 attempts.
The training phase is usually loosely constrained in terms of time and compute capacity, as typically industrial stakeholders are willing to invest days, weeks or even months of training to obtain high-perfee gigghperforming policy that can generate accurate solutions when deployed in production. Hence, in our experiments, we train all policies until convergence.
Inference Phase At inference time, a new problem instance P is drawn from a distribution D (possibly different than D). Here, there are typically hard constraints to outputting a final solution: fixed time limit Taax constrains wall-clock execution, and a compute capacity Bmax constrains the number of operations that can be done in parallel. The trained policy πe can be used within these constraints to generate solutions to the problem, and the best solution is ultimately used. The reward function R can still be used to score attempted solutions and inform subsequent attempts. Inference strategies can be defined as a function I : (p, To. B m + (ai .... aHz) that uses the base policy To and any additional inference-ting search, adaptation, storage, or optimisation methods under the budget Tiux and Byux to produce the best possible solution to the problem instance defined by the sequence of actions (a; )iiig. The objective can hence be written as:
I(I) R(st, ai) s.t. C(Z) < B max, T(I)= S T
where C(I) and T(I) represent the compute and time cost of the inference strategy. This formulation highlights that, unlike traditional RL, where zero-shot performance is the primary measure, we focus on strategies which enable further improvement under given constraints. We provide three real-world examples in Appendix H that correspond to this problem setting, illustrating practical scenarios where inference-time search is both natural and feasible.
Inference strategies differ in how they explore the solution space and how they incorporate the outcomes of previous attempts to influence future sampling. Their effectiveness depends on several contextual factors, including the parameter count of the pre-trained policy, the problem's 's underlying structure, the episode horizon, the nature of the reward function, but most critically, on the available time and compute budget.
3.2
One budget, many possibilities: inference- time search and adaptation
In this section, we detail four types of inference strategies and how we adapt them to work in the multi-agent setting. We implement and release all of these methods in JAX (Bradbury et al., 2018).
Stochastic policy sampling The first natural lever to improve solution quality is to re-sample from a stochastic policy. In other words. beyond the creation of a unique greedy solution (i.e., using = arg max,' (a(ao))) over a trajectory of observations), one can sample stochastically (as in a ~ TT(-(o)) in order create diverse solutions. Multi-agent policy sampling generalises easily to the multi-agent case by sampling from the joint action distribution, (o))
5

---

Tree search These methods store information about partial solutions using past attempts to pref- erentially search promising regions of the solution space without updating the pre-trained policy. Simulation guided beam search (SGBS) (Choo et al., 2022) provides the best time to performance balance the literature, outperforming Monte Carlo Tree Search (Coulom, 2006). Like most tree searches, SGBS has three steps: expansion, simulation, pruning. Expansion uses the policy to decide on the most promising next actions (i.e., a = top- K (~e(-1o) ) from the current node (partial solu- tion). A simulated rollout of episode is produced greedily using πhe and the return is collected for each node. Pruning keeps only the best nodes found so far based on the return. Solely the expansion step needs to be adapted for Multi-agent SGBS. This is trivial when the explicit joint actions are accessible (de Witt et al., 2020; Yu et al., 2022), since we can still select the top ones (i.e., a= top- ( (-) (-)o))). For methods using auto-regressive action selection (Mahjoub et al., 2025), having access to the top joint actions is intractable, hence we sample K times stochastically from the same node (i.e., a a.i,,-., a [[K] (-))))
Online fine-tuning These methods keep updating policy parameters at inference time. Given a base policy Ta, online fine-tuning optimises 8 using inference-time rollouts and policy gradient updates: 8' = 0 + (TT)) where J(Te) represents an adaptation objective. In line with (Bello* et al., 2017), we keep maximising expected returns (Bello* et al., 2017) over past attempts the fixed instance (instead of over a training distribution). Multi-agent online fine-tuning re-trains TO on the new instance using the MARL algorithm that used during pre-training (Bello* * al., 2017; Mahjoub et al., 2025; de Witt et al., 2020; Yu et al., 2022).
Diversity-based approaches These methods pre-train collection of diverse specialised policies which can be used to search for the most appropriate solution at inference-time. COMPASS (Chalumeau al., 2023b) encodes specialised policies in continuous latent space L by augmenting pre- trained policy to condition on both the observation and a latent vector sampled from & (i.e., a ~ To(- | o, 2), z L): effectively creating a continuous collection of policies. COMPASS achieves SOTA in single-agent RL for CO. To avoid having the latent space of Multi-agent COMPASS growing exponentially with the number of agents, we keep one latent space for all agents (i.e., ~ Te( o, z),
L). This allows for tractable training, and for efficient inference search with the covarinnce matrix adaptation evolution strategy CMA-ES (Hansen and Ostermeier, 2001). Aside from being multi-agen agent, we keep the training and inference phases close to the original method described in Chalumeau al. (2023b), and provide further details and explanations in Appendix F.2.
Unlike other inference strategies, COMPASS includes an additional training phase, which remains accessible since the training phase is unconstrained (all policies are trained until convergence). Creating a COMPASS checkpoint from a pre-trained base policy involves adding parameters to process the latent vectors (Appendíx F.2), resulting in modest increase in model size. This increase never exceeds 2% of the total policy size, and has negligible impact on the overall computational or memory footprint. We report all parameter counts in Appendix I.
4 Experiments
Set of 17 complex RL tasks
In our experimental study, we combine popular MARL algo- rithms and inference strategies and benchmark them on a of complex RL tasks from the literature. Each task was specif- ically selected for its difficulty. We evaluate all base policies with and without inference-time search across a wide range of budget settings. Our experiments constitute the largest-ever study of inference strategies for decision-making
Connector
StarCraft (SMAC)
RWARE
Wide range of settings
tine
compute
complesty
Baselines We use three MARL approaches obtain our base policies: Independent PPO (de Witt et al., 2020) (IPPO) and Multi-Agent PPO (Yu et al., 2022) (MAPPO), which are widely used and well- MARL methods, and the recent SOTA sequence modelling approach SABLE (Mahjoub et al., 2025). Fach of these, referred to as base policies, is evaluated with all four inference strategies introduced in Section 3.2, namely stochastic sampling, SGBS, online fine-tuning and COMPASS.
Evaluation of inference strategles
Figure 3: Overview of our evalua-
tion tasks and experimental study.

---

Tasks Mahjoub et al. (2025) established SOTA over the most comprehensive MARL benchmark published in the field to date. Interestingly. their results demonstrate that there remain certain tasks for which no existing method (including SABLE) is able to achieve good performance. Specifi- cally, these are tiny-2ag-hard, tiny-4ag-hard, small-4ag, small-4ag-hard medium-4am aedium-4ag-harde medium-6ag, large-4ag. large-4ag- -ardd large- large-8ag- Bardhard, xlarge-4ag and xlarge-4ag-har hard from Multi-Robot Warehouse (Papoudakis et al., 2021) (RWARE), smacv2_ _ep___0_uiitttt units and smacv2_20 _occt_20_uuits units from the StarCraft Multi-Agent chal- lenge (Samvelyan et al., 2019; Ellis et al., 2023) (SMAC), and con- 1om=10xxxx10 and con-15x15xd0 from Connector (Bonnet et al., 2023).
Each environment (illustrated on Fig. 3) in- troduces distinct challenges that contribute to its complexity. RWARE requires agents to coordinate in order to pick up and de-
x 20 steps
Converged
X COMPASS
Normalised performance
1.0
liver packages without collision and has
0.
very sparse reward signal. In SMACv2 tasks, a team cooperates in real-time com- bat against enemies across diverse scenar-
0.8
ios with randomised generation. Connec- tor models the routing of printed circuit board where agents must connect to desig- nated targets without crossing paths. All three environments feature combinatorial and tigh-dimensional action spaces, partial observability and the need for tightly coor- dinated behaviours, making these 17 tasks a compelling test-bed for complex RL with desirable properties modelling aspects of real-world tasks. We use A-Xdd imple- mentations of Connector and RWARE from Jumanji (Bonnet et al., 2023) and for SMAC from JaxMARL (Rutherford et al., 2023).
02
large-4ap 4ag con con-10x2x11
can 15x33a
0.0
targe-4 targe-laghon Aaghard
large- large-Sag-tar Sag-hardd hadd
large-Sag Bagg-e-ggg
mealum- Aar-hard
medium 4ag
medium- Bag
smary2 10 uniss
smark2 20 units
small- 4ag-tard
small-4ag -ag
tiny- Zag-bard
riny frad
starge-Aa 4ag starge starge-4ag-hard
Figure Training the SOTA algorithm Sable to con- vergence is not enough to achieve optimal zero-shot performance. We report the mean normalised perfor- mance per task and report 95% bootstrap confidence intervals in Appendix L.
Training base policies To obtain clear performance ceilings for each algorithm and best isolate the effects of inference strategies, we train all base policies until convergence. For the sake of continuity with previous work with truncated training budgets, typically of 20M steps, we report the zero-shot results for each converged checkpoint compared to its previous corresponding reported performance on Fig. 4. We observe that in most tasks (14 out of 17), the converged policy stays below 70% normalised performance, demonstrating that the benchmark still far from saturated. COMPASS requires an additional training phase, which reincarnates the existing base policies to create the latent space specialisation. For each base policy and task, we also train the COMPASS checkpoint until convergence. This leads to 102 trained policy checkpoints.
Evaluating performance during inference Evaluating inference strategies in a way that unbiased and aligned with real-world settings is challenging. Most papers report results where the budget is based on a number of attempts, hence not directly incorporating the time cost of the inference strategy. The time costs are reported, but it is tough to analyse due to the plurality of hardware used to obtain them. Having re-implemented all of the baselines in the same code base and setting the budget in terms of time (in seconds), we can avoid this bias in our study. We use the same fixed hardware for GPU. For statistical robustness, we always
all our experiments, namely a NVIDIA-AN SEIIIIAAA10---- -100-SXM4-800000 run 128 independent seeds. In all cases, we control for Bmax by varying the permitted number of batched parallel attempts instead of altering hardware between experiments. For aggregation across multiple tasks we follow the recommendations made by Agarwal et al. (2021) and use the rliable library to compute and report the inter-quartile mean (IQM) and 95% stratified bootstrap confidence intervals.
To train the base policies, we re-use the hyperparameters reported in Mahjoub et al. (2025), which have been optimised for our tasks. For the inference strategies, we follow recom-
Hyperparameters
7

---

mendations from the literature (Choo et al., 2022; Chalumeau et al., 2023b). All hyperparameters choices are reported in Appendix
4.1 A couple of seconds all you need
In this section, we demonstrate that inference-ting search can help reach close to maximum task performance, using base policies for which zero-shot performance stagnates around 60%.
Experiments To demonstrate that inference-time strategies are accessible, we use a small budget: 30 seconds, and compute capacity enabling generate 64 solutions in parallel. Each base policy evaluated greedily for a single attempt, and then evaluated with the search budget, using each inference strategy. We report the performance distribution over the 17 tasks Fig. 5, and the performance gains offered by the best inference-ting search over the best zero-shot on Fig.
1.0
Normalised perfor performance score (10M)
066
0.2
Discussion We can draw four main con- clusions. First, inference-time search does provide a massive performance boost over zero-shot, which stands for every base pol- icy. For the SOTA zero-shot method SABLE, this translates to pushing the best-ever achieved aggregated performance by more than 45% and creating system (SABLE +COMPASS) that achieves close to 100% win-rate in all tasks where this metric is available. Second, the improvement enabled over zero-shot performance increases significantly (almost exponentially) with respect to the complexity of the task (see Fig. 1). This suggests substantial gains still ahead as the field moves toward increasingly realistic scenarios. Third, we observe that COMPASS the leading strategy across tasks and base algorithms, and that SABLE remains the SOTA base policy even when using inference-ting search. Interestingly, under a small time budget, stochastic sampling outperforms online fine-tuning We nevertheless show in the following section that, given more budget, this result can be nuanced, and we share our interpretation of these findings.
0.0
Figure Performance obtained by inference strate- gies over the benchmark. Each base policy is evalu- ated with each possible inference strategy. We report the inter-quartile mean over tasks with 95% stratified bootstrap confidence intervals.
Mapping performance with compute and time budget
A recurrent limitation in previous work on inference strategies is the use of a fixed budget during evaluation, creating a narrow view on methods, and often creating a bias towards certain types of methods. In this section, we aim at providing much broader perspective over inference-time search by reporting performance over grid of time and compute budgets.
Experiments We choose maximum time of 300 seconds, and evaluate all inference strategies using the leading base policy (SABLE), with compute budget {4. 16, 32, 64, 128, 256). All in all, we have 4 inference strategies, 7 possible compute budgets, 17 tasks, and 128 seeds per task, leading to 60 928 evaluated episodes. This constitutes largest study released on inference strategies. We report these results using contour plots, where the æ-axis is time, y-axis the number of parallel attempts allowed (our proxy for compute) and colour corresponds achieved (win-rate when accessible or min-max normalised return) going from dark purple (min) to yellow (max). We keep the hardest tasks, the lower half based on the zero-shot performance the converged SABLE checkpoints (see Fig. 4), on Fig. 6 and defer remaining tasks to Appendix A.
the performance
Discussion As a sanity check, we remark that performance always increases (colours become lighter) when time compute increases (going towards the upper right corner). We now highlight three main observations. First, COMPASS demonstrates impressive versatility and achieves significant gains over other inference strategies, dominating all maps, except for con- 10x10x1000 where it gets

---

J0x20x10a aar-15xbb226
CONMAIZ
Computa capacity (#paralien attempts)
Stadhattc
Normalised Relllieddeeeeee Return
Finettning
370
seconds
Figure 6: Performance contour plots of Sable with different inference strategies on a range of time budget (æ-axis) and compute capacities (y-axis)) for set of representative tasks. Colours indicates performance, with brighter colours indicating higher values.
slightly outperformed by online fine-tuning. Second, we observe high variance for online fine-tuning: getting close to COMPASS for large budgets on con-10x10x10a, yet struggling to match stochastic sampling on others (e.g., RWARE's large-8ag). This shows that fine-tuning can be detrimental by reducing the number of attempts made within the time budget. Plus, policy gradients can unstable (small batch size) or converge to local optima. This observation disproves the common belief that inference time search is as trivial as over- aver-fitting the problem instance. Finally, we observe that SGBS, despite failing on the connector tasks, achieves competitive performance overall and even leads in the low-budget regime (i.e., below 100 seconds and fewer than 10 parallel attempts) some scenarios (large-4ag and
4.3 Scaling with increasing budget
In practical applications, time is often more restricted than compute: a couple of seconds or minutes can be allowed, sometimes few hours (train scheduling for the next day). but rarely more. Being able to improve solutions' quality under a fixed time budget by increasing compute is desirable. In this section, analyse how different methods scale with additional compute.
Stochastic
Fine-tuning
SG85
COMPASS
Experiments We keep the time budget fixed, 300 seconds, and we plot the final per- formance for each possible compute budget (still using the number of parallel attempts proxy). We SABLE the base pol- icy and evaluate all the inference strategies across the 8 hardest tasks. We report results per strategy, and aggregated over the tasks on Fig. 7.
Normalised performence score (1QM)
1.0
0.8
0.6
0.4
0.2
Discussion As expected, stochastic sam- pling has the lowest scaling coefficient: its search solely relies on chance, no adaptation or additional search is happening. On the other hand, we can see that online fine-tuning benefits from more compute, probably due to better estimation of the policy gradient. nevertheless requires budget of 64 paral- lel attempts clearly outperform stochastic
0.0
32
128
256
Compute capacity (#parallel attempts)
Figure Scaling of performance with respect to compute capacity (fixed time budget). Compute capacity set by limiting the number of parallel at- tempts. SABLE +COMPASS demonstrates impressive scaling properties, reaching the performance bounds.

---

sampling. On the other hand, COMPASS consistently provides significant advantage. Its scaling trend seems linear at first, with a high coefficient, and only seems to decline for higher budgets, as performance limits are reached (over 95% win-rate when accessible, best-ever observed performance elsewhere). We have a two-fold explanation for these particularly impressive scaling properties: (i) the diversity contained in the latent space can be exploited by more parallelism, leading to a massive exploration of the solution space, even when the initial policy is far from optimal, and (ii) the higher the batch size, the better the CMA-ES search, enabling COMPASS to exploit even more information from any additional searching step allowed within the given time budget. We discuss this further in Appendix K.
5 Conclusion
In this work, we demonstrate that inference-time strategies are a critical and underutilized lever for boosting the performance of RL systems in complex tasks, using multi-agent RL as a representative test-bed. While training-time improvements have long dominated the field, our results show that cefference-time search may offer significant performance gains using only few seconds to minutes of additional wall-clock time during execution. We introduce a unified view of inference strategies, extend it the multi-agent setting, and empirically validate its effectiveness under varying compute and time budgets.
Our large-scale evaluation, the most comprehensive date inference time methods, reveals three key takeaways: (i) inference-tin time search with a relevant strategy yields significant improvements, even under tight time constraints; (ii) the gains depend the inference budget, and our contour maps provide practitioners with practical guidance based on their constraints; and (iii) SABLE +COMPASS not only dominates the benchmark but also exhibits the most favourable scaling trends, making particularly effective for increasingly complex decision-makinn problems.
Altogether, our findings call for shift in how decision makking models are evaluated and deployed: inference-timee strategies should be treated as core components of the solution pipeline, not as optional refinements. We hope results and open-source tools will encourage broader adoption and inspire further innovation in the design of scalable inference-tine time algorithms.
Limitations and future work We focus on multi-agent RL it better captures the complexity of real-world decision- makiin systems, where successful operation often relies on coordination among multiple agents. Nonetheless, we acknowledge that naturally single-agent tasks can also be highly complex. In the spirit of seeding a wider investigation for future work beyond the multi-agent setting, provide additional results the single-agent Craftax benchmark in Appendix C, showing that simple 30-second inference-time search using stochastic sampling can provide a 37% performance boost. This provides initial (but limited) evidence that claims hold more generally and we leave more thorough investigation for future work. Beyond the broader RL setting, we intend to investigate two main future research directions. First, studying how to best combine existing inference paradigms to leverage their complementary strengths. Second, investigating how inference strategies compare when evaluated out-of- out-o--distribution.
Acknowledgements
We would like to thank Guillaume Toujas-Bernate Jake Lourie and Thomas Lecat for useful discus- sions on the use of inference strategies in real-world applications. We thank our MLOps team for developing model training and experiment orchestration platform Alchor. We thank the Python and JAX communitics for developing tools that made this research possible. We thank the anonymous reviewers for their constructive feedback and valuable suggestions. Finally, thank Google's TPU Research Cloud (TRC) for supporting our research with Cloud TPUs.
10

---

References
R. Agarwal, M. Schwarzer, P. S. Castro, A. Courville, and M. G. Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021. URL https: /taps:///rxiii /wwxivxivvv./////// wwg. httg/abb/2108 13264.
T. Ahmad, D. Zhang, C. Huang, H. Zhang. N. Dai, Song, and H. Chen. Artificial intelligence in sustainable energy industry: Status quo, challenges and opportunities. Journal of Cleaner Pro- duction, 1898145,34, 2021. ISSN 0959-6526. doi:
125833.
URL https: /www.//ww///oo////////// sciencedirect.
http:///aaaaaerrra.a//////////// com wttpeet.cooccocceeeeeeeeeeeeeeee
I. Bello*, H. Pham* Q. V. Le, M. Norouzi, and S. Bengio. Neural catpabiinatoriallppiiiiiaan with reinforcement learning, 2017. URL htt.t:ttt///ettt..tt httptt:ss///pperveeeee..e http://rwww.....efoooooo///////s
D. S. Bernstein, S. Zilberstein, and N. Immerman. The complexity of decentralized control of markov decision processes. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, UA0000, page 32-37, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607099.
C. Bonnet, D. Luo, D. Byrne, S. Abramowitz, V. Coyette, Duckworth, D. Furelos-Blanco, N. Grin- sztajn, T. Kalloniatis, V. Le, O. Mahjoub, L. Midgley, S. Surana, C. Waters, and A. Laterre. Jumanji: a suite of diverse and challenging reinforcement learning environments in jax, 2023, URL https: //git/ggtttt//t/// com/ instaddeeaaaaa... jumanjji..
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, VanderPlas, Wanderman- Miine,, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: http://gttttt..oooo jax///xxmmppp/ faammmmja....
L. Canese, G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Re, and S. Spanò. Multi-agent reinforcement learning: A review of challenges and applications. Applied Sciences, 11(11), 2021. ISSN 2076-3417, doi: 10.3390/appl 1 114448. URL https: http://wwwwwppp...//m wwt.:/ddp..co///0/// cwmm///006-1//1111 11/4948.
Chalumeau, R. Boige, B. Lim, V. Macé, M. Allard, A. Flajolet, A. Cully, and T. Pierrot. Neurocvo- lution is competitive alternative to reinforcement learning for skill discovery. In International Conference on Learning Representations, 2023a. URL http://///peeeeeeeeeeeeee net/ wwt.wiwwwwwwewrnnt//// forum?ide 6BH1ZgyPOZY.
F. Chalumeau, S. Surana, C. Bonnet, N. Grinsztajn, A. Pretorius. A. Laterre, and T. D. Barrett. Combinatorial optimization with policy adaptation using latent space search. In Thirtty-eventh Conference Neural Information Processing Systems, 2023b.
F. Chalumeau, B. Lim, R. Boige, M. Allard, L. Grillotti, M. Flageat, V. Macé, G. Richard, A. Flajolet, T. Pierrot, et al. Qdax: A library for duality-diversity and population-based algorithms with hardware acceleration. Journal of Machine Learning Research, 25(108):1-16, 2024.
Chalumeau, R. Shabe, N. D. Nicola, A. Pretorius, T. D. Barrett, and N. Grinsztajn. Memory- enhanced neural solvers for routing problems. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL httppa////ppervie...n.. p7WHZy8TCG.
J. Choo, Y.-D. Kwon, J. Kim, J. Jae, A. Hottung, K. Tiemey, and Y. Gwon. Simulation -eciattionnnuieed beam search for neural combinatorial optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https: /t///aax..aa aarxiv..vvv. org/ 2227 06190.
R. Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pages 72-83. Springer, 2006.
A. Cully and Y. Demiris. Quality and diversity optimization: unifying modular framework. IEEE Transactions Evolutionary Computation, 22(2):245-255 2017.
R. de Kock, O. Mahjoub, S. Abramowitz, W. Khlifi, C. R. Tilbury, C. Formanek, A. Smit, and
A. Pretorius. Mava: research library for distributed multi-agent reinforcement learning in jax. 2021.
arXiv preprint

---

C. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson. Is independent learning all you need in the starcraft aulti- challenge?, 2020. URL https: //arxiv. org/ abs/2011
2011. 20t./2011..........
DeepMind, I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Bud- den, Cai, A. Clark, Danihelka, A. Dedieu, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, T. Hennigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, Norman, G. Papamakarios,
Quan, R. Ring, Ruiz, A. Sanchez, I.. Sartran, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, M. Stanojević, W. Stokowice, L. Wang, G. Zhou, and F. Viola. The DeepMind JAX Ecosystem, 2020. URL http://git com/ hoom/goooleeeooo deepmind.
J. Dona, A. Flajolet, A. Marginean, A. Cully, and T. Pierrot. Quality-diversity for one-shot biological sequence design. In JCML 2 Workshop for Life and Material Science: From Theory to Industry Applications, 2024. URL https: : //pp//epppreeeeeee/eeeeeeeeeeeee
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. 2016. URL https: org/ abs /1w...11.....0 02779.
G. Dulac-Aroldd N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. An empir- ical investigation of the challenges of real-world reinforcement learning. CoRR, abs/2003.11881, 2020. URL https: / arxiv. org/ 22000 11881.
B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:37577-37593, 2023.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 11 123-1135. JMLR.org, 2017.
N. Grinsztajn, D. Purelos-Blanco, Surana, C. Bonnet, and T. D. Barrett. Winner takes it all: Training performant rl populations for combinatorial optimization. In Advances in Neural Information Processing Systems, 2023.
R. Hammond, D. Crages, M. Guo, J, N. Foerster, and I. Reid. Symmetry-breaking augmentations for ad hoc teamwork. In ICLR 2025 Workshop on Bidirectional Human-AI Alignment, 2025. URL h www.:/eeoooooiiddenneeeoo/eeeeee
N. Hansen and A. Ostermeier. Completely derandomized self aelftadapt in evolution strategies. Evolutionary Computation, 9(2):159-195, 2001. doi: 10. TB2///0066601700100000
T. Hayes, R. Rao, H. Akin, N. J. Sofroniew, D. Oktay, Z. Lin, R. Verkuil, V. Q. Tran, J. Deaton, M. Wiggert, R. Badkundri, Shafkat, J. Gong, A. Derry. R. S. Molina, N. Thomas, Y. A. Khan, C. Mishra, C. Kim, L. J. Bartie, M. Nemeth, P. D. Hsu, T. Sercu, S. Candido, and A. Rives. Simulating 500 million years of evolution with a language model. Science. 387(676) 2025. doi: 10. 121///ienne..ads000 8. URL https: : //www//ww/.// science. org borg/doi//nn ab. science.. ads0018.
Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee. Flax: neural network library and ecosystem for JAX, 2024. URL http:// gi thub. com/ wwt..//wwo/////eeeeeeeo google/flll..
A. Hottung, Y.-D. Kwon, and K. Tierney. Efficient active search for combinatorial optimization problems. In International Conference on Learning Representations, 2022.
A. Hottung, M. Mahajan, and K. Tierney. Polynet: Learning diverse solution strategies for neural combinatorial optimization. arXiv preprint 240. Feb 2024.
H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "other-play for zero-shot coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML'20, 20. JML..org, 2020.
12

---

A. Hundt, B. Killeen, H. Kwon, C. Paxton, and G. Hager. "good robot!": Efficient reinforcement learning for mteti-step visual tasks via reward shaping, 09 2019.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
R. M. Karp. On the computational complexity of combinatorial problems. Networks, 5(1):45-68, 1975.
S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances Neural Information Processing Systems, 33:81 33:8198-8210 2020.
A. Laterre, Y. Fu, M. K. Jabri, A.-S. Cohen, D. Kas, K. Hajjar, T. S. Dahl, A. Kerkeni, and K. Beguir. Ranked reward: Enabling self-play reinforcement learning for combinatorial optimization, 2018. URL https:tpp/.. / /wwpxvaxvvvv///////////ee//a/aaa org/ abs /807. 01672..
W. Long, W. Wen, P. Zhai, and L. Zhang. Role play: Learning adaptive spec-fic strategies in multi-agent interactions, 2024. URL https //arxiv. orgooogg/aa. ha//aabs/24111 01166.
A. Lupu, B. Cui, H. Hu, and J. Foerster. Trajectory diversity for zero-shot coordination. M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7204-7213. PMLR, 18-24 Jul 2021. URL https:::: http///prcceee...cc//c///o/ mlr, ppwss/vvsae http//uppllaaaaaa/aae/ html.
M. Macfarlane, Toledo, D. J. Byme, Duckworth, and A. Laterre. SPO: Sequential monte carlo policy optimisation. In The Thirty- eighth Annual Conference Neural Information Processing Systems, 2024. URL https: opwnreeeeeeeeeeeeeeeeeeeeee net net
O. Mahjoub, S. Abramowitz, R. de Kock, W. Khlifi, S. du Toit, J. Daniel, L. B. Nessir, L. Beyers, C. Formanek, L. Clark, and A. Pretorius. Sable: performant, efficient and scalable sequence model for marl. In Proceedings of the 42th International Conference Machine Learning. ICML'25, 2025.
M. S. Mark. T. Gao, G. G. Sampaio, M. K. Srirama, A. Sharma, C. Finn, and A. Kumar. Policy-agnostic RL: Offline RL and online RL fine-tuning of any class and backbone. In 7th Robot Learning Workshop: Towards Robots with Human-Level Abilities, 2025. URL https://openreet
net/ orrmmid--VoFxxdeG..
M. Matthews, M. Beukman, B. Ellis, M. Samvelyan, M. Jackson, S. Coward, and J. Foerster. Craftax: a lightning-fast fast benchmark for open-ended reinforcement learning. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
R. Mirsky, Carlucho, A. Rahman, Fosong, W. Macke, M. Sridharan, Stone, and V. Albrecht. A survey of ad hoc teamwork research. In Multi-Agent Systems: 19th European Conference, EUMAS 2022, Düsseldorf, Germany, September 14-16, 2022, Proceedings, page 275-293, Berlin, Heidelberg, 2022. Springer- Verlag. ISBN --226111----------------- doi: 10.1007/978-3-311-20611 16. URL https: h/tp:://ooo.oooooo 3-11 20612-6614466 6_16.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wienstra, S. Legg, and D. Hassabis. Human-level control through deep reinforce- ment learning. Nature, 518:529-533 2015.
N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi. L. Zettlemoyer, P. Liang, E. Candès, and Hashimoto. sl: Simple test-time scaling, 2025. URL https//////a org/ abs/2501. 19393.
M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and Levine. Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL GcEIvidYSs.
net/ ftr//oormm?i
13

---

M. Olivecrona, T. Blaschke, O. Engkvist, and H. Chen. Molecular de novo design through deep reinforcement learning. Journal of Cheminformaties, 9, 09 2017. doi: 10.1186611322--17-022353
L. Ouyang, Wu, X. Jiang, D. Almeida, C. L. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, Leike, and R. Lowe, Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information P Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 978171387 10888.
G. Papoudakis, F. Christianos, L. Schäfer, and S. V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. URL http: /arxiv. org/ htg/abs/2000 07869.
M. Popova, O. Isayev, and A. Tropsha Deep reinforcement learning for de novo drug design. Science Advances, a(7)=eaap7885, 2018. doi: 10.1 00.11266scia URL https: //www science.
org/doi hatooi//ass//ta 1126/sciady aap7885.
R. Qiu, Z. Sun, and Y. Yang. DIMES: A differentiable meta solver for combinatorial optimization problems. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Infor- mation Processing Systems, 2022. URL https: / opwnreviee.eee.eeee net//forurr/ce--955220nn
K. Rao, C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari. Rl-cyclegan: Reinforcemeet learning aware simulation-to-real pages 1114-11163, 06 2020. doi: 10.1 10010
C. Ruhdorfer, M. Bortoletto, A. Penzkofer, and A. Bulling. The overcooked generalisation challenge, 2025. URL https: //tp://aaiivvv//a org/ 2220/ 17949.
A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, A. Khan, C. de Witt, A. Souly, et al. Jaxmarl: Multi-agent rl environments in jax. arXiv preprint arXiv:2311.10090, 2023.
M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint 1atp:9992.04433 2019.
A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657 2019.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484-48) 2016.
C. V. Snell, J. Lee, K. Xu, and A. Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In The Thirteenth International Conference on Learning Representationss 2025. URL https:///ppnss eview net www//eeuunn/// (aworuuunidd44AAAtaaaa0
N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback. In Proceedings of the 34th Interna- tional Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
C. Wang, A. Rahman, I. Durugkar, E. Liebman, and P. Stone. N-agent ad hoc teamwork. In The eiihty- Annual Conference on Neural Information Processing Systems, 2024a. URL https: /ttp:/://///eeeeeeeeeee net. ftnt/forum?it wwwoormiidiiTtateeeee -wt.//dxTTTIII.D....
R. Wang. E. Zelikman, G. Poesia, Y. Pu, N. Haber, and N. Goodman. Hypothesis search: In- ductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024b. URL https: wptprevieeeeeeeeeeeeeee/ee net/
Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, Xia, E. H. Chi, Q. V. Le, and D. Zhou. tffin-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, 1088.
USA, 2022. Curran Associates Inc. ISBN
14

---

Y. Wu. Z. Sun, S. Li. S. Welleck, and Y. Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models, 2025. URL https: // //arxiv..o org/ abs//abb..000000 00724.
E. S. Yourdshahi, T. Pinder, G. Dhawan, L. S. Marcolino, and P. Angelov. Towards large scale ad-hoc teamwork. In 2018 JEEE International Conference on Agents (ICA), pages 44-49, 2018. doi:
10.1109/GEETSS.001... 206688461..6.
C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: / net/ forum?id=WVXXxxBi....
L. Zintgraf, S. Schulze, Lu, L. Feng, M. Igl, K. Shiarlis, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: variational bayes- adaytivv deep rl via meta-learning J. Mach. Learn, Res., 22(1), Jan. 2021. ISSN 1532-4435.
15

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: /
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Code and model checkpoints provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Seerides/CooSSubmisse wwwiisssssoolooioooooo icy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /CodeSubmic / //nips. wttpccccpppccccceeececceccccccct /tt.:///ggiiieeeeeeeuuubiiiiii ssisnnoillll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, values are given in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrpseee/ee//ee//e cc hc/pppblitt/thicccsueeeeeee..eee
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not relevant.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
20

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21

---

Appendix
A Additional results
Section 4.2 presents the contour plots of Sable with all inference strategies on the hardest 8 tasks of the benchmark. We report the contour plots of the other remaining tasks on Fig.
Nor Mormalised Return
Compute
Time in seconds
Figure 8: Performance contour plots of Sable with different inference strategies remaining tasks on a range of time budget (z-axis) and compute capacities (y-axis). Colours indicates performance, with brighter colours indicating higher values.
Fig. 8, shows similar trends to those illustrated in Section 4.2. SABLE +COMPASS leads to the best overall performance. Stochastic sampling provides a good and robust baseline overall. Online fine-tuning suffer from variance: despite being able to compete close to COMPASS on couple tasks, it often get outperformed by stochastic sampling. Finally, SGBS obtains good performance on numerous scenarios, and even leads one scenario (smacv2_ 10_ units).
B Details about the tasks
Figure 9: Environment rendering for Connector. Task name: con 1om-10xx0-10a Image from Mahjoub al. (2025).
Connector is an environment designed to model the routing of printed circuit board (PCB). Agents start at randomly generated initial positions and have to reach goal position without overlapping with each other's 's paths (indicated as lower opacity coloured cells in Figure 9). Each agent has
22

---

partial view with fixed field of view around itself as well as its current (z, coooddinte on the grid and the (z, cooordiaaee of goal location. At each timestep, agents receive small negative reward -0.03 and receive reward of 1 for successfully connecting to goal location. The particular diffcculty in this environment stems from the fact that agents have to select actions carefully so that they not only reach their goal greedily but so that all agents can ultimately reach their goals.
Figure 10:
Environment rendering for Robot Warehouse. from Mahjoub et al. (2025).
Task name: tiny-2ag.
Image
is an environment where agents need to coordinate to deliver shelves (green cells in Figure 10) goal location (dark grey cells Figure 10) in warehouse. The reward sparse since agents only receiving a reward of for a successful shelf delivery to the goal location and O otherwise. This sparsity makes the environment particularly difficult since agents have complete long sequence of correct actions in order to receive any reward signal.
RWARE
Step
20
25
30
Figure 11: Environment rendering for SMAX. Task name: 2s3z. Image from Mahjoub et al. (2025).
SMAC is environment where agents need to coordinate and develop the correct micro strategies in order to defeat an enemy team. was shown that the original benchmark was overly simplistic and for this reason SMACv2 (Ellis et al., 2023) was developed in order to address the shortcomings of the original benchmark. SMACv2 randomly generates a team of allied units by sampling from a selection of unit types and randomly generates ally starting positions at the start of each episode. The need for generalisation is what gives this benchmark its difficulty.
JAX-based implementations. Since Connector was developed part of Jumanji (Bonnet et al., 2023), we make use the JAX-based implementation directly. For RWARE we use the JAX- based reimplementation from Jumanji and for SMAC and SMACv2 we make use of the JAX-based reimplementation from JaxMARI. (Rutherford al., 2023), named SMAX. For a detailed discussion on task naming conventions we refer the reader to the Appendix of Mahjoub et al. (2025).
23

---

C Results and discussion about the single-agent case
C.1 Additional results on single-agent challenge: Craftax
To demonstrate that our core message is not limited to the multi-agent case, we selected single-agent task that is still far from being solved by RL algorithms, Craftax (Matthews al., 2024).
We trained a SABLE base policy and matched the current state-of-the-art performance reported on the Craftax-1B leaderboard, which is 18.3% of the maximum score. We then 30-second inference-tinee search with the simplest inference strategy (stochastic sampling). The results are reported in Table 1. Strikingly, and similar to our main results, using only 30-second inference-time search yields a 37% performance gain, reaching 25.8% of the maximum score. To further put this into context, this performance boost of the same magnitude that of the last 300 million steps of training the base policy.
These results confirm that the significant improvement enabled by inference-time search is not limited to multi-agent RL and is applicable to RL in general. the next subsection, we discuss existing results (reported in the literature) further support the importance of inference-time search in single-agent agent settings.
Table 1:
Performance comparison on Craftax- 1B between zero-shot inference and stochastic sampling with 30-second budget. Values indicate mean performance with 95% confidence intervals.
Scenario
Zero-shot
Stochastic Sampling (30s budget)
Craftax-IB 18.8 (18.1, 19.5)
25.8 (25.7, 26.1)
C.2 Published results on single-agent tasks
To further demonstrate the impact of inference-time strategies beyond the multi-agent case, we discuss previous results reported the literature. These results not show the same impact because they often close saturation, with zero-shot. They nevertheless demonstrate that inference- time strategies can be successfully applied to a wide range of problems and provide noticcable improvement.
We refer to the experimental results reported the seminal works that introduced the inference strategies used in our paper (Chalumeau et al., 2023b; Choo et al., 2022; Bello et al., 2017). All of these evaluate on ageett tasks (TSP, CVRP, JSSP, Knapsack, FFSP) and show that inference strategies bring statistically significant improvements over zero-shot performance.
However, we recall the limitations have identified these experimental results: the limited breadth of the time/compute settings, methods' reliance on problem-specific tricks and saturated benchmarks where zero-shot performance is already very high.
D Hyperparameters
In this section, we report the hyper-parameters used in our experiments. They can also be found in the submitted code files.
Training base policies Hyper-parane parameeasseserss used to train the base policies for this benchmark are taken from Mahjoub et al. (2025). These hyper-parameters were tuned on each task with a budget of 40 trials using the Tree-structured Parzen Estimator (TPE) Bayesian optimisation algorithm, and are reported in Mahjoub et al. (2025).
Hyper-parand prrper-parametess COMPASS Training Hyper-parameters used to train all the COMPASS checkpoints can be found in Table 2. Most of them were kept close to the original hyper-parameters used in Chalumeau et al. (2023b).
Hyper-parameters: Inference-time search Most values are directly taken from original works or kept close to these. When different, tried to use guidance from original work to decide how
24

---

Table Hyper- Hlpee-parametees used for COMPASS' training phase. The same values are used for all three base policies (i.e., SABLE, IPPO, MAPPO).
Parameter
Value
Instances batch size
128
Latent space sample size
64
Latent space dimension size
16
Latent amplifier
Padding with random weights
True
Weight noise amplitude
0.01
to set them. The values for the CMA-ES search used with COMPASS, for all base policies, be found in Table 3. The values for online fine-tuning (all base policies) are reported in Table 4. Hyper-parameters for SGBS (all base policies) reported in Table 5, and for stochastic sampling Table 6.
Table hypee-parameters of the CMA-ES process used to search COMPASS" latent space inference time. The same parameters are used for all tasks and all base policies (i.e., SABLE, MAPPO and IPPO).
Parameter
Number of attempts
4
16
32
64
128
256
Latent sample size
4
16
32
64
128
256
Number of clites
32
128
Covariance matrix step size
1.0
1.0
1.0
1,0
1.0
1.0
1.0
Initial sigma Num. components
1.0
1.0
1.0
1.0
1.0
1.0
1.0
1
1
Table 4: Hyper-parameters used when doing online fine-tuning inference-time with SABLE, MAPPO and IPPO.
Parameter
Task Name
Sable
IPPO
MAPPO
Max. trajectory size
All tasks
64
64
64
con-10x10x10a
0.001
0.00025
0.0001
con-15x15x23a
0.001
0.0001
0.0001
large-4ag
0.001
0.0005
0.0001
large-4ag- -ard
0.001
0.0005
0.00025
large-8ag
0,001
0.00025
0.0001
large-8ag-ban -ard
0.001
0.00025
0.0005
medium-4ag
0.001
0.00025
0.00025
medium-4ag-hare
0.0005
0.0005
0.00025
Learning rate
medium-6ag
0.0005
0.0001
0.00025
smacv2_10_uniz
0.00025
0.0005
0.00025
smacv2 20__200_taaa__
0.001
0.0005
0.0005
smacv2_5 Saacvv_5_unitt units
0.001
0.00025
0.00025
small-4ag
0.0005
0.0005
0.0005
small-4ag-hh hard
0.001
0.0001
0.00025
tiny-2ag hard
0.001
0.00025
0.00025
tiny-4ag-hand hard
0.0005
0.0005
0.0005
xlarge-4ag
0.001
0.0005
0.0005
xlarge-4ag hard
0.0001
0.0005
0.0001
Connector
0.05
0.00
0.00
Entropy coefficient
RWARE
0.0
0.0
0.0
SMAX
0.0
0.0
0.0
25

---

Table Syyper-arameters, for Simulation Guided Beam Search (SGBS). The same values are used for SABLE, IPPO and MAPPO. Values are chosen such that expansion factor and beam width (top-k) use fully the number of attempts allowed by batch.
Parameter
Number of attempts
4
16
32
64
128
256
Expansion factor
4
4
4
4
4
4
Beam width (Top k)
32
64
Table 6: To sample stochastically from policy (SABLE, IPPO MAPPO), always the same temperature as the one used during training: 1.0.
Parameter
Value
Evaluation greedy
False
Temperature
1.0
E Experimental stack
All algorithmic implementations are built by extending the JAX-hased research library, Mava (de Kock et al., 2021). Our version of CMA-ES used for searching COMPASS's 's latent space comes from the JAX-based quality diversity library, QDAX (Chalumeau et al., 2024). Both these librarics are built to leverage the DeepMind JAX ecosystem (DeepMind et al., 2020) and use Flax (Heek et al., 2024) for building neural networks, optax for optimisers and orbax for model checkpointing.
F Additional details about training
F.1
Base policy training
In Section 3, we explain that in setting, it is assumed that all methods be trained until convergence. For transparency and comparison with previous results in the literature, we report in Table 7 the training budget given to the methods reach convergence, in environment steps.
Table 7: Training budgets (in environment steps) for each method across tasks.
Task Name
Sable
MAPPO
IPPO
con-10x10x10a
100M
200M
200M
con-15x15x23a
100M
200M
200M
large-4ag
600M
200M
200M
large-4ag-hard
600M
400M
200M
large-8ag
600M
400M
200M
large-8ag-hand
600M
200M
200M
medium-4ag
400M
200M
200M
medium-4ag-hart -ard
600M
200M
200M
medium-6ag
600M
200M
200M
small-4ag
200M
200M
200M
small-4ag-hare
400M
200M
200M
tiny-2ag-hard
100M
200M
200M
tiny-4ag-hard
200M
200M
200M
xlarge-4ag
200M
400M
200M
xlarge-4ag- hard
200M
200M
200M
mnacv2_10_unii
100M
200M
200M
600M
200M
200M
26

---

F.2 COMPASS training
In this section, we provide details about COMPASS and our implementation. For a more in-depth explanation, refer the reader to the original paper (Chalumeau et al., 2023b). Our implementation available in the submitted code files.
COMPASS is method, which (i) uses a pre-trained base policy and modifies it to be conditioned by a latent vector (ii) re-trains it to enable latent vectors to create policies that are diversified and specialised for distinct training sub-distributions. At inference-time, this enables to search for the latent vector which performs best on the new instance to be solved.
Reincarnating a pre-trained base policy COMPASS checkpoints are not trained from scratch, they reincarnate an existing base policy, add parameters this policy in order process the latent vector which can be given as additional input. Hence, one must choose where to inject the latent vector in the existing architecture, and must modify the existing parameters of the neural policy to account for the new shape of the input. The latent vector must be input in a way that enables to diversify the actions produced for the same observation (i.e., making sure that different latent vectors create different policies). In practice, we concatenate the latent vector to the observations. When relevant, we make sure that this is done after observation normalisation layers.
In the original work, the additional weights added to the base neural network are initialised with zeros. In our case, we observed that this could not provide enough diversification in the first training step, preventing any emergence of specialisation, leading to no benefit. We found that initialising these parameters with random uniform values between -0.01 and 0.01 fixed the issue across all tasks.
Creating diversity and specialisation the latent space Similarly to the original method, the latent space is not learned, it is a fixed prior, and the network's weights are learned to create diversity from this prior space. We also use similar prior, which uniform distribution between -1 and 1, over 16 dimensions. We do not need any amplification of the latent vector (i.e., we multiply by 1, whereas the original work multiplies it by 100). At each training step, a batch of instances is sampled from the training distribution, a batch of latent vectors is sampled from that latent space, the conditioned policy is evaluated for each instance, for each latent vector, and only the best performing latent vectors, for each instance, used in the computation of the loss. This creates the specialisation (diversification). For additional motivation and mathematical formulation, we refer the reader to Chalumeau et al. (2023b). Like all training processes in experimental study, the method is allowed to train until convergence. In practice, each COMPASS checkpoint was trained for 100 million steps.
G Pseudo-algorithms
In this section, we provide algorithmic descriptions for the inference strategies used in the paper, showing how the policy is used at inference time, under the time and budget constraints. Stochastic sampling is explained in Algorithm 1, SGBS in Algorithm 2, online fine-tuning in Algorithm 3 and COMPASS in Algorithm 4.
Algorithms use the terminology trajectory т, which is equivalent to solution. The symbol o is used for the concatenation of new step transition a partial trajectory.
H Real-world examples where inference-time search is applicable
Applying RL in real-world applications is key driver of this research. Below, we describe three real-world cases where we have first-hand experience. In all of these, the user allows for a time budget, and a proxy of the solution's quality is accessible to inform the search.
Printed Circuit Board Routing The user submits the specification of a PCB board they want to route. This task can take up to several days even weeks, even for an expert, depending on the complexity of the board. Therefore, users are typically satisfied with allowing an model to take at least a few minutes even hours for search to obtain better solution (routed board). Additionally,
27

---

Algorithm Stochastic Sampling
1: Input: policy TH, instance time budget maxx compute capacity Bmax, reward function R
2: Initialize best trajectory 7* <- 0, best score It"
3: while elapsed time < maxx do
4: for parallel rollout
to mmax do
5: Initialize trajectory 7 f- 0
6: for step 1
H do Observe from environment copy Pb
7:
8:
Sample d T ot)
9:
Execute a, in pb. observe 0++1
10:
Append (Or, ar) to Th
11:
Rs R((r.)
RR > Rº then
13:
T" Tb.
R t- R
14:
return 7"
Algorithm 2 Simulation-Guided Beam Search
1: Input: policy Te. instance p. time budget Timax compute capacity Buaxx beam width K, reward function R
2: Initialize beam B + {0}, best trajectory 7" + 0, best score R2 < -00
3: while elapsed time Timax do
4: Bnew +- 0
5: for partial trajectory т in B do
6:
for sample 6 = 1 to Bmax/ B do
7:
Let o, be the observation at the end of Sample
8:
TT(- or)
9:
Simulate greedy rollout from a, using To
10:
Let Ag be the resulting full trajectory
Rs < R(fx)
12:
Add (T ar, Ra) to Brew
RG > R then
14:
7" 4 t.. RC 4- Rs
Prune top-K trajectories from Brew into
16:
return 7"
Algorithm 3 Online Fine-Tuuning
1: Input: base policy we, instance p, time budget Tmax. compute capacity Bmax- reward function
R, learning rate a
2: Initialize best trajectory 7" +- 0, best score Rt -00
3: Initialize adapted parameters 0' +- 0
4: while elapsed time Tiaxx do
5: for parallel rollout = 1 to Bmax do
6: Rollout тΙ Te on
7:
Rs + R(D,)
8:
R4 > R then
9:
R" <- Rs
10:
Compute gradient:
Vo f- B Buns d=1
1. Σ
Buax Vo VologTo(T))) Rs
11:
Update parameters: 0' + 0' VaV
12:
return 7"
28

---

Algorithm 4 COMPASS + CMA-ES search
1: Input: latent-conditionet policy TT(- o, 2), instance time budget maxx, compute capacity
Bmax, reward function
2: Initialize CMA-ES: mean μ., covariance E
3: Initialize best trajectory 7* <- 0, best score R 6- 00
4: while elapsed time < max do
5: Sample { Jb=1 Bmax ~ (1,
6: for each 2n parallel do
7: Rollout 7ь TT((- o, 2b) on
8:
Rs + R(T))
9:
if Rs > R then
10: T + Tbe
R f- Rs
11: Update (j,, Σ) using CMA-ES with {(zb, Rb) Ro)
12:
return 7"
the user will often find it beneficial to leave more time and to grant more computation in order to improve the board further, which could then casier to manufacture or reduce its production cost.
Bin Packing When logistics company wants to load containers with objects that need to be shipped overseas, operators typically have few hours days to plan which objects need to be transported, and can hence allow inference-time search. Here well, the energy and money gains from having better object dispatching (thus fewer containers to ship for the same amount of objects) will always justify few hours of search, or the allowance of more computing spend.
Train Scheduling Designing train schedules meet demand forecast, while proposing a feasible schedule (c.g., no collisions between trains), is another relevant example. This process usually happens once or twice year and can take months to complete. Hence, there is no need to rely on single rollout of the trained policy, and an inference-tie -ime search can instead be conducted over several days.
I Parameter counts for all base policy checkpoints
Creating a COMPASS checkpoint from pre-trained base policy requires adding new parameters to process the latent vectors (see Appendix F.2), hence increasing the model size. However, this increase small, never exceeding 2% of the total base policy model size.
We report the parameter count for all base policy checkpoints and all corresponding COMPASS checkpoints for IPPO (Table 9), MAPPO (Table 8) and Sable (Table 10). We observe that (i) for IPPO and MAPPO, the increase in model size stays below 1.6%, and (ii) for Sable, it typically adds 0.5% to the total parameter count. Consequently, this model size increase has a minor impact on the memory and computational footprint of the method, which is confirmed by our study of the attempts achieved over time (reported in Appendix J).
J Discussion about the inference-time budget
Choice of the budget used in the experiments We create our experimental setting to be similar to practical use cases. Hence, we use time as the main budget constraint, instead of number of attempts (contrary to most literature on the topic). This enables to account for the cost of search and adaptation. Even though we always evaluate methods on 128 instances for each task, ensure that each instance is solved fully independently. to avoid that the vectorisation process creates bias towards one method or another. We choose to study methods with time budgets ranging from 30 to 300 seconds. The lower bound is taken to be small, to show that even short inference-tin -ime searches can provide significant benefit. The higher bound is chosen to high enough to see which method can really benefit from longer search budget. Concerning the compute capacity, we choose a range of values that seems achievable in practice. In real-world scenarios, simulations can get more complex and require CPU core for one simulation. We hence believed that most use cases would
29

---

Table 8: Parameter count comparison for MAPPO.
Scenario
Base Policy
COMPASS Policy
% Increase
small-4ag-haid
125061
127109
1.64%
medium-64e -6ag
125317
127365
1.63%
tiny-4ag-hard
125061
127109
1.64%
xlarge-4ag
125061
127109
1.64%
xlarge-4ag-hard
125061
127109
1.64%
large-8ag
125573
127621
1.63%
tiny-2ag-hard
124805
126853
1.64%
large-8ag-hard
125573
127621
1.63%
medium-4ag
125061
127109
1.64%
large-4ag
125061
127109
1.64%
large-4ag-hard
125061
127109
1.64%
medium-4ag-har 4ard
125061
127109
1.64%
smacv2_29 _niittts
187417
189465
1.09%
smacv2_ _anuit_
151567
153615
1.35%
small-4ag
125061
127109
1.64%
con-10x 10a
124293
126341
1.65%
con- con-15x15x25a
125957
128005
1.63%
Table 9: Parameter count comparison for IPPO.
Scenario
Base Policy
COMPASS Policy
% Increase
large-8ag-hard
125573
127621
1.63%
medium-6ag
125317
127365
1.63%
large-8ag
125573
127621
1.63%
tiny-4ag-han
125061
127109
1.64%
small-4ag-hait
125061
127109
1.64%
tiny-2ag-hard
124805
126853
1.64%
xlarge-4ag-hard
125061
127109
1.64%
large-4ag-hard
125061
127109
1.64%
large-4ag
125061
127109
1.64%
xlarge-4ag
125061
127109
1.64%
medium-4ag
125061
127109
1.64%
medium-4ag-hard
125061
127109
1.64%
smacv2_22 20_units
187417
189465
1.09%
smacv2_10 10 10__uits
151567
153615
1.35%
small-4ag
125061
127109
1.64%
con- 10x 10x 10a
124293
126341
1.65%
con- 15x 15x22a
125957
128005
1.63%
fit within the limit of 256 simulations in parallel. Nevertheless, code works for higher values and can be used to obtain results in new settings (both terms of time and parallel attempts).
Total number of attempts achieved within time budget To remain consistent with previous literature, and provide better understanding of the search and adaptation cost of each inference strategy and each base policy, we provide the number of attempts achieved within the time limit, for all tasks and compute capacity (i.e., number of parallel attempts allowed). Values for SABLE within 300 seconds are reported on Table 11 for COMPASS, Table 12 for online fine-tuning, Table 13 for stochastic sampling. SGBS builds solutions from existing partial solutions, hence the count of the attempted solution cannot be obtained in the same way as other methods.
Values for all base policies within 30 seconds reported Table 14, Table 16 and Table 15.
30

---

Table 10: Parameter count comparison for SABLE.
Scenario
Base Policy
COMPASS Policy
% Increase
large-8ag
390096
392144
0.52%
smacv2 20 _nits
43363
43875
1.18%
con- 10x 10x 10a
388422
390470
0.53%
con-15x15x22
389907
391955
0.53%
umacv2_10_uml
201435
202459
0.51%
medium-6ag
389454
391502
0.53%
medium-4ag-hard
389580
391628
0.53%
large-4ag-hard
389004
391052
0.53%
large-Sag-hard
389520
391568
0.53%
small-4ag-hard
389004
391052
0.53%
tiny-4ag-hard
1080524
1082572
0.19%
small-4ag
389196
391244
0.53%
xlarge-4ag
389004
391052
0.53%
medium-4ag
389580
391628
0.53%
xlarge-4ag-hard
1078796
1080844
0.19%
large-4ag
389004
391052
0.53%
tiny-2ag-hard 388746
390794
0.53%
Table 11: Number of attempts (solutions generated) when using SABLE + COMPASS during 300 seconds of inference- search.
Task Name
Number of attempts
8
16
32
64
128
256
con-10x10x10a
9960
19448
37520
67296
131456
248320
422144
con-15x15x23a
1608
3112
7744
14144
27456
53248
80896
large-4ag
2608
5208
9360
16896
30464
44032
59392
large-4ag-har
2596
5000
9360
16832
29568
44288
59904
large-8ag
1612
3200
6240
10304
17280
23424
30976
large-8ag-haan -ard
1708
3144
6096
10624
18880
29056
39680
medium-4ag
2088
4072
7904
15104
29376
48256
87296
medium-4ag-hard
2072
4040
7888
16992
29632
48384
87552
medium-6ag
1536
3016
5728
10816
21440
37760
74240
smacv2_10_unit units
7628
13248
26176
45088
74304
122624
174336
smacv2_20_unil
5824
10428
18528
34112
64064
111232
180992
small-4ag
1964
3880
8672
14112
28096
61568
83456
small-4ag hard
1876
4216
7104
15488
30464
52096
93696
tiny-2ag-Ma hard
2736
6072
11856
22496
44544
77440
147200
siny-4ag-han hard
1384
2648
5024
10624
16640
32256
45568
xlarge-4ag
1816
3568
7952
15264
24320
45440
81152
xlarge-4ag-bat hard
1180
2264
5344
9696
17792
28544
51200
K
Additional discussion about COMPASS
In Section 4.3 we mention explanations to why COMPASS gets good performance and scaling trends. In this section, we elaborate on these, and highlight three main aspects.
First, COMPASS trains "multiple attempts" objective, unlike other methods, which are training for "single attempt" performance. Thus, COMPASS 's training objective anticipates that what matters is the maximum performance over multiple attempts, rather than the average performance over these attempts. While its average solution quality may be lower, it has higher probability of producing one excellent solution. This makes inherently better suited for inference-time search where multiple attempts are allowed.
31

---

Table 12: Number of attempts (solutions generated) when using online fine-tuning and SABLE during 300 seconds of inference- time search.
Task Name
Number of attempts
4
8
16
32
64
128
256
con- von-10x10x10s
10608
20256
37744
64448
115392
226560
395776
con- 15x11x23
2136
3952
6864
10496
16592
32000
60928
large-4ag
1792
3296
6560
9920
13376
24448
49664
large-4ag-bas
2064
3416
6448
8832
13248
24448
46592
large-8ag
1416
2336
4592
7552
11136
19200
34304
large-8ag-ban -ard
1140
2576
4064
6752
12736
26624
44032
medium-4ag
2000
4176
7472
9920
15744
28160
48384
medium-4ag-haa
2024
3776
7232
10816
14848
28160
48384
medium-6ag
1700
2872
6448
8704
21120
32896
65792
amacv2_10_un's units
6848
12576
20176
37536
68160
156672
235776
seaav2_20__uii1
5700
10584
18176
25216
47488
104382
155904
small-4ag
1940
3688
7072
9600
14272
30336
49408
small-4ag-hart
1940
3688
7072
9600
14272
30336
49408
tiny-2ag-han hard
2572
5600
9824
13344
22848
48768
86272
tiny-4ag-han
2572
5568
9808
13344
22784
48768
86272
xlarge 4ag
1768
3856
6368
9792
13760
24448
46080
xlarge-4ae Srrge-4ag-hard har-hard
1280
2024
3776
6368
11456
21504
40192
Table 13: Number of attempts (solutions generated) when sampling stochastically from SABLE during 300 seconds of inference-time search.
Task Name
Number of attempts
4
8
16
32
128
256
con- con-10x10x10a
10632
20504
39524
70720
138624
264960
448512
2048
4056
7840
14144
27648
54144
101376
large-4ag
2092
3624
7808
15456
30336
46464
82944
Large-4ag-hard
2108
3696
8112
13344
26816
46592
82176
large-8ag
1508
2552
5824
9568
17984
30720
52480
large-Bag-hard
1324
2256
4432
8416
16576
34048
32208
medium-4ag
2388
4576
8064
15296
29760
53120
81664
mediun-4ag-hard
2144
4616
7968
16992
29696
52992
88576
mediun-6ag
1344
3496
6768
10848
25280
66048
66048
smacv2_10_unie units
7168
11056
25376
47088
94720
148608
261888
1macv2_20_unis
4848
9144
17584
34688
55232
102144
228608
small-4ag
1472
3896
7072
16448
27840
61056
84224
small-4ag- hard
1812
3704
7072
16448
26880
59776
84224
tiny-2ag-hen hard
2824
6168
10432
20192
45504
77952
165632
tiny-4ag-hard
1600
2672
5888
9184
16960
33152
58880
xlarge-4ag
1832
3624
7888
13248
27584
50944
81152
xlarge-4ag hard
1164
2280
4480
8064
15296
33152
58880
Second, COMPASS creates dense space of diverse and specialised policies, hence creating strong exploration capacity. While most other methods rely on the entropy the action distribution to explore (i.e. sampling stochastically from the pre-trained policy), COMPASS can use the wide diversity encoded in its latent space to build much more diverse policies.
Finally, the Covariance Matrix Adaptation mechanism (CMA-ES) used by COMPASS at inference time enables to navigate the latent space and identify the most adapted (suitable) latent for the problem instance at hand. Using a CMA-ES search in 16-dimensional space is more efficient and less prone to getting stuck in local optima compared to using gradient descent in the entire policy parameter space (i.c. online fine-tuning).
32

---

Table 14: Number of attempts (solutions generated) when using COMPASS with SABLE, IPPO and MAPPO during 30 seconds of inference- seference-time search.
Task Name
Number of attempts Sable
PPO
16
32
64
128
256
64
con-10x10x100
996
1872
3744
6720
13120
24832
42240
65152
1m1115x55x23a
160
384
768
1408
2752
5376
8192
36608
large-4ag
260
464
928
1664
3072
4480
5888
5056
Garge-4ag-hard
256
464
928
1664
2944
4480
6144
4928
large-8ag
160
312
624
1024
1728
2304
3072
3520
large-8ag- hard
168
304
608
1056
1856
2944
4096
3584
medium-4ag
208
392
784
1504
2944
4864
8704
5120
medium-4ag-has
208
392
784
1696
2944
4864
8704
5056
medium-6ag
152
288
576
1088
2112
3840
7424
4416
smacv2_10_um units
760
1304
2608
4480
7424
12288
17408
30464
mmacv2_20..m units
584
928
1856
3392
6400
11136
18176
26880
small-4ag
196
432
864
1408
2816
6144
8320
5248
small-4a small-4ag-hard
188
352
704
1536
3072
5120
9216
5248
ciny-2ag-herl hard
272
592
1184
2240
4480
7680
14592
6848
tiny-4ag-hare
136
248
496
1056
1664
3200
4608
5312
xlarge-4ag
180
392
784
1536
2432
4480
8192
4160
xlarge-4ag-hant hard
116
264
528
960
1792
2816
5120
4288
Table 15: Number of attempts (solutions generated) when using stochastic sampling with SABLE, IPPO and MAPPO during 30 seconds of inference-ting search.
Task Name
Number of attempts Sable
PPO
4
16
32
64
128
256
64
con-10x10x10a
1060
2048
3940
7040
13760
26240
44800
103680
con-15x15x23a
204
400
780
1408
2752
5376
10112
51456
large-4ag
208
360
768
1536
3008
4608
8192
5312
large-4ag-hon hard
208
368
800
1312
2688
4608
8192
5184
large-8ag
148
248
576
928
1792
3072
5120
3904
Sarge-8ag-hard
132
224
432
832
1664
3328
3200
3712
medium-4ag
236
456
800
1504
2944
5248
8128
5120
medium-4ag-harz
212
456
784
1696
2944
5248
8832
5312
medium-6ag
132
344
672
1056
2496
6528
6528
4480
smacv2_10 2__02_11__uu_ units
716
1104
2528
4704
9408
14848
26112
33408
smacv2_22 20_units
484
912
1744
3456
5504
10176
22784
31808
small-4ag
144
384
704
1632
2752
6016
8384
5184
small-4ag-han hard
180
368
704
1632
2688
5888
8384
5248
tiny-2ag- hard
280
608
1040
2016
4544
7808
16512
4544
tiny-4ag-hard
160
264
576
896
1664
3328
5888
5312
xlarge-4ag
180
360
784
1312
2752
5120
8128
4416
alarge-4ag-Ba hard
116
224
448
800
1536
3328
5888
4352
L Additional experimental details and error bars
In Fig. 4, we report the zero-shot performance of SOTA method SABLE, and compare with the best results observed for 30 seconds of inference-time search. We report the error values in Table 17 and the figure with error bars in Fig. 12.
We report the mean and 95% bootstrap confidence intervals. The 20M timestep values aggregated over 10 independent runs and computed directly from the benchmarking data provided along with the Sable paper (Mahjoub al., 2025), for the policy trained convergence and for COMPASS, give the results aggregated over 128 independent runs.
33

---

Table 16: Number of attempts (solutions generated) when using online fine-tuning with SABLE, IPPO and MAPPO during 30 seconds of inference-ting search.
Task Name
Number of attempts Sable
PPO
4
16
32
64
128
256
64
con-10x10x106
1060
2024
3760
6432
11520
22656
39424
23680
con-15x15x23a
212
392
672
1024
1600
3200
5888
5888
1arge-4ag
176
328
656
992
1280
2432
4864
3136
large-4ag-hie hard
204
336
640
864
1280
2432
4608
2944
large-8ag
140
232
448
736
1088
1920
3328
2304
large-8ag-hand
112
256
400
672
1216
2560
4352
2112
medium-4ag
200
416
736
992
1536
2816
4608
2880
medium-4ag-han -ard
200
376
720
1056
1472
2816
4608
2880
medium-6ag
168
280
640
864
2112
3200
6400
4160
smaav2_10_unni 20p_v_2_0__mmmm
684
1256
2016
3744
6784
15616
23552
8960
amacv2_20__rm 2ni
568
1056
1808
2496
4736
10368
15360
7936
small-4ag
192
368
704
960
1408
2944
4864
2816
small-4ag-haa
192
368
704
960
1408
2944
4864
2816
Liny-2ag-hard
256
560
976
1312
2240
4864
8448
3904
tiny-4ag-hard hard
256
552
976
1312
2240
4864
8448
3904
xlarge-4ag
176
384
624
960
1344
2432
4608
2944
xlarge-4ag hard
128
200
368
608
1088
2048
3840
2432
*
20 M steps
X
Converged
X
COMPASS
1.0
Normalised performance
0.8
0.6
0.4
0.2
0.0
con-10x10x10a
con-15x1 con-15x15x23a
large-4ag
large-4ag-hard
large-Bag
large-Bi Bag
large- large-Bag-han Bag-hard
medium-4a0
medium-Nt 4ag-hard
medium-Aag-hare
medium-6ag
smacv2 10, units
smacv2 20 units
small-4ag
small-4ag-Hn 4aa-had hard
tiny- tiny-2ag-hare 2ag-hard
tiny hag-hard hard
xlarge-de 4ag
Marge-Aag-haare
Figure 12: Training the SOTA algorithm Sable to convergence is not enough to achieve optimal zero-shot performance, but using 30 seconds of search enables to consistenly reach above 85% normalised performance across the whole benchmark. We report the mean and 95% bootstrap confidence intervals.
34

---

Table 17: Performance comparison between the base policy (after 20M timesteps and after conver-
gence) and COMPASS. Values indicate mean performance with 95% confidence intervals.
Scenario
20M Timesteps
Converged
COMPASS
con-10x 10x 10a
0.388 (0.372, 0.404)
0.500 (0.414, 0.586)
0.953 (0.914, 0.984)
con-15x15x233
0,075 (0.067, 0.083)
0.078 (0.039, 0.125)
0.867 (0.805, 0.922)
large-4ag
0.246 (0.200, 0.283)
0.642 (0.618, 0.664)
0.949 (0.940, 0.957)
large-4ag-hare
0.175 (0.113, 0.225)
0.609 (0.581, 0.633)
0.933 (0.922, 0.943)
large-8ag
0.203 (0.197, 0.208)
0.496 (0.464, 0.526)
0.953 (0.947, 0.959)
large-8ag-hard
0.209 (0.202, 0.216)
0.516 (0.489, 0.541)
0.957 (0.950, 0.963)
medium-4ag
0.376 (0.346, 0.396)
0.666 (0.642, 0.686)
0.947 (0.941, 0.953)
medium-4ag-han
0.253 (0.220, 0.282)
0.701 (0.677, 0.721)
0.949 (0.943, 0.955)
medium-6ag
0.240 (0.227, 0.250)
0.702 (0.668, 0.733)
0.958 (0.954, 0.963)
smacv2_10 _i__ut__0_uunts
0.648 (0.623, 0.675)
0.852 (0.789, 0.914)
0.977 (0.945, 1.000)
smacv2_20 _nnitttts
0.599 (0.548, 0.639)
0.719 (0.641, 0.797)
0.969 (0.938, 0.992)
small-4ag
0.384 (0.243, 0.484)
0.728 (0.706, 0.746)
0.963 (0.956, 0.969)
small-4ag-hait
0.349 (0.327, 0.371)
0.707 (0.687, 0.725)
0.955 (0.948, 0.962)
tiny-2ag-hard
0.664 (0.646, 0.684)
0.810 (0.799, 0.821)
0.978 (0.967, 0.989)
tiny-4ag-hard
0.548 (0.529, 0.566)
0.783 (0.756, 0.806)
0.958 (0.951,
xlarge-4ag
0.193 (0.118, 0.261)
0.485 (0.465, 0.504)
0.905 (0.893, 0.916)
xlarge-4ag-hard
0.065 (0.001, 0.161)
0.394 (0.373, 0.413)
0.892 (0.879,
35


Reviews:
[{"review_id": "INyfXWZD0k", "reviewer": "Reviewer_EG7m", "summary": "The authors found that using small inference time budgets can significantly improve the performance of MARL agents, compared to the zero-shot setting. The typical RL literature usually focuses on a zero-shot setting, meaning the RL agent can one attempt at each time step during evaluation, whereas making multiple attempts before giving a final result has become more popular in the LLM space. \n\nThe authors took inspiration from making multiple attempts during inference and applied it in MARL. They conducted experiments on a wide range of MARL tasks, from the Multi-Robot Warehouse, the StarCraft Multi-Agent challenge, to Connector. In all the experiments, the authors found that inference strategies can push up the final performance higher.", "questions": "I am not sure I follow how the inference strategies work in MARL. The model is performing actions at every time step. How do inference strategies work? Do the models consider multiple actions before deciding on a single action at each time step?", "limitations": "N/A", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nExperiments seem comprehensive, covering a wide variety of tasks. The gains from inference strategies are solid. It is also surprising that even a small inference budget (30 seconds) could make a big impact on the model's performance. Charts seem reasonable, too. For example, Figures 5 and 6 show nicely how more parallel attempts can unlock greater performance.\n\nWeaknesses:\n\nIt feels like the benchmark environments are already somewhat saturated.", "was_revised": true, "final_justification": "The author's responses are well-written and their explanation on why COMPASS is better for Reviewer Ptix is well elaborated. Given these clarifications, I am raising my score.", "rebuttal": "We thank the reviewer for their positive feedback and comments. We provide additional insight on the benchmark used and explanations concerning inference strategies, which we will include in the revised manuscript.\n\n\\\n**> W1: it feels like the benchmark environments are already somewhat saturated.**\n\nOur benchmark actually focuses on tasks from the literature that are still far from being saturated. We chose tasks from the benchmark introduced in [1] (Neurips 2021) where the most recent state-of-the-art method Sable [2] (ICML 2025) reports an average performance below 50% normalised return (see grey crosses on Figure 4 of our main paper). To isolate the effect of a potential lack of training, we re-train Sable until convergence on these tasks and we observe that the zero-shot performance is plateauing under 70% (see orange crosses on Figure 4), and reaching only 5% on the hardest task (con-15x15x23a), hence demonstrating that the zero-shot performance is reaching a ceiling on these tasks.\n\nWe discuss this aspect of the benchmark in paragraph “Tasks” (Section 4, line 233) and paragraph “Training base policies” (Section 4, line 263/264). We can further clarify by updating the sentence on line 263/264: “... the converged policy stays below 70% normalised performance, demonstrating that the benchmark is still far from saturated”.\n\n[1] Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, Neurips 2021 \\\n[2] Sable: a Performant, Efficient and Scalable Sequence Model for MARL, ICML 2025\n\n\\\n**> Q1: How do inference strategies work? Do the models consider multiple actions before deciding on a single action at each time step?**\n\nSection 3 defines an inference strategy as a function that uses the base policy and any additional inference-time search, adaptation, storage, or optimisation methods under the time and compute budget to produce the best possible solution to the problem instance (lines 169 to 171). Inference strategies represent a broad concept, encompassing various approaches. The effectiveness of these strategies is contingent on specific contextual factors, such as the size of the pre-trained policy, the problem's inherent structure, the duration of the episode, the nature of the reward function, and the available time and computational budget.\n\nConcerning the action selection, all four inference strategies studied in the paper use a pre-trained policy which outputs an action distribution at each time step. The next action is always selected by sampling from this action distribution. In other words, any (allowed) action can be selected, with a probability which corresponds to the logits given by the pre-trained policy to this action.\n\nThese inference strategies differ in the way they incorporate outcomes of previous attempts to impact the way future actions will be sampled. Online fine-tuning uses the past attempts to derive a policy gradient and update the policy’s parameters. COMPASS uses the past attempts to update the distribution used to sample the latent vectors which condition the policy. SGBS (tree search) re-starts from the partial solutions which led to the best performance so far (whereas other methods always re-sample solutions from scratch). In contrast, stochastic sampling does not take past attempts into account.\n\nWe will add these details at the beginning of Section 3.2 in order to better link the methods under study with the definitions introduced in Section 3.1. We thank the reviewer for their comments which will help clarify the paper."}, {"review_id": "ouInv7QnBm", "reviewer": "Reviewer_Ptix", "summary": "The paper applied different searching strategies to multi-agent RL and showed that the COMPASS is the best compared to greedy, stochastic, tree search and fine-tuning.", "questions": "1. It would be good to explain details of how the specialized policy in COMPASS is trained. It will be good to show the comparison of the model size.\n2. It would be good to explain the reason behind why the COMPASS is better", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well motivated and the experimental results are promising. \n2. The paper is well written, the contribution is fair\nWeakness:\n1. It will be good to explain details of how the specialized policy in COMPASS is trained. It will be good to show the comparison of the model size.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for highlighting the contributions of our work and for their thoughtful comments, which we address in the responses below. We will update the manuscript to clarify the points raised in the review.\n\n\\\n**> Q1: Explain details of how the specialized policy in COMPASS is trained.**\n\nAt each training step, a batch of latent vectors is sampled from the latent space, using a fixed random uniform distribution. The latent-conditioned policy is evaluated on the same problem instance, with each of the sampled latent vectors. The episode returns obtained for each latent vector are compared, and only the best trajectory is kept to train on. In line with the original implementation of COMPASS [3] (Neurips 2023), we use REINFORCE to estimate the policy gradient, using a batch of instances evaluated in parallel.\n\nOnly the best latent vector in the batch is used to train the policy, thus creating a specialisation within the latent space. In doing so, different areas of the latent space learn to specialise for different subdistributions of the training instances. \n\nThis training process is explained in Appendix E.2 (lines 945 to 976), which we omitted to refer to in the main paper. We will revise the manuscript to add this reference at the end of the paragraph “Diversity-based approach” (line 204, Section 3.2).\n\nWe can also extend this paragraph if the reviewer(s) feel this would be useful, or use the additional page allowed for the camera-ready version to include it directly in the main paper.\n\n\\\n[3] Combinatorial Optimization with Policy Adaptation using Latent Space Search, Neurips 2023\n\n\\\n**> Q2: Comparison of the model size.**\n\nCreating a COMPASS checkpoint from a pre-trained base policy does indeed require adding new parameters to process the latent vectors (line 953, App. E.2), hence increasing the model size. However, this increase is small, never exceeding 2% of the total base policy model size.\n\nWe report the parameter count for all base policy checkpoints and all corresponding COMPASS checkpoints in the table below. We observe that (i) for IPPO and MAPPO, the increase in model size stays below 1.6%, and (ii) for Sable, it typically adds 0.5% to the total parameter count. Consequently, this model size increase has a minor impact on the memory and computational footprint of the method, which is confirmed by our study of the attempts achieved over time (reported in Appendix G).\n\nWe thank the reviewer for highlighting this aspect of COMPASS, which was not yet discussed in the paper. We will add this table to our appendix, along with a paragraph to comment on these values.\n\n\\\n**> Q3: Explain the reason behind COMPASS being better.**\n\nWe highlight three main reasons why COMPASS is better. \n\nFirst, COMPASS trains a “multiple attempts” objective, unlike other methods, which are training for “single attempt” performance. Thus, COMPASS’s training objective anticipates that what matters is the maximum performance over multiple attempts, rather than the average performance over these attempts. \n\nSecond, COMPASS creates a dense space of diverse and specialised policies, hence creating a strong exploration capacity. While most other methods rely on the entropy of the action distribution to explore (i.e. sampling stochastically from the pre-trained policy), COMPASS can use the wide diversity encoded in its latent space to build much more diverse policies.\n\nFinally, the Covariance Matrix Adaptation mechanism (CMA-ES [4]) used by COMPASS at inference time enables it to navigate the latent space and identify the most adapted (suitable) latent for the problem instance at hand. Using a CMA-ES search in a 16-dimensional space is more efficient and less prone to getting stuck in local optima compared to using gradient descent in the entire policy parameter space (i.e. online fine-tuning).\n\nSome of these elements are discussed in a paragraph in Section 4.3 to interpret the scaling capacity of COMPASS. We will add a section in the Appendix to discuss the elements that were not yet mentioned.\n\n\\\n[4] Completely derandomized self-adaptation in evolution strategies, Evolutionary Computation 2001\n\n\\\n\\\nThe following tables present the parameters count for the checkpoints used in our experiments.\n\n\\\nMAPPO: \n\nScenario  |  Base Policy  |  COMPASS Policy  |  Increase\n---|---|---|---\nsmall-4ag-hard  |  125061  |  127109  |  1.64%\nmedium-6ag  |  125317  |  127365  |  1.63%\ntiny-4ag-hard  |  125061  |  127109  |  1.64%\nxlarge-4ag  |  125061  |  127109  |  1.64%\nxlarge-4ag-hard  |  125061  |  127109  |  1.64%\nlarge-8ag  |  125573  |  127621  |  1.63%\ntiny-2ag-hard  |  124805  |  126853  |  1.64%\nlarge-8ag-hard  |  125573  |  127621  |  1.63%\nmedium-4ag  |  125061  |  127109  |  1.64%\nlarge-4ag  |  125061  |  127109  |  1.64%\nlarge-4ag-hard  |  125061  |  127109  |  1.64%\nmedium-4ag-hard  |  125061  |  127109  |  1.64%\nsmacv2_20_units  |  187417  |  189465  |  1.09%\nsmacv2_10_units  |  151567  |  153615  |  1.35%\nsmall-4ag  |  125061  |  127109  |  1.64%\ncon-10x10x10a  |  124293  |  126341  |  1.65%\ncon-15x15x23a  |  125957  |  128005  |  1.63%\n\n\\\nIPPO:\n\nScenario  |  Base Policy  |  COMPASS Policy  |  Increase\n---|---|---|---\nlarge-8ag-hard  |  125573  |  127621  |  1.63%\nmedium-6ag  |  125317  |  127365  |  1.63%\nlarge-8ag  |  125573  |  127621  |  1.63%\ntiny-4ag-hard  |  125061  |  127109  |  1.64%\nsmall-4ag-hard  |  125061  |  127109  |  1.64%\ntiny-2ag-hard  |  124805  |  126853  |  1.64%\nxlarge-4ag-hard  |  125061  |  127109  |  1.64%\nlarge-4ag-hard  |  125061  |  127109  |  1.64%\nlarge-4ag  |  125061  |  127109  |  1.64%\nxlarge-4ag  |  125061  |  127109  |  1.64%\nmedium-4ag  |  125061  |  127109  |  1.64%\nmedium-4ag-hard  |  125061  |  127109  |  1.64%\nsmacv2_20_units  |  187417  |  189465  |  1.09%\nsmacv2_10_units  |  151567  |  153615  |  1.35%\nsmall-4ag  |  125061  |  127109  |  1.64%\ncon-10x10x10a  |  124293  |  126341  |  1.65%\ncon-15x15x23a  |  125957  |  128005  |  1.63%\n\n\\\nSable:\n\nScenario  |  Base Policy  |  COMPASS Policy  |  Increase\n---|---|---|---\nlarge-8ag  |  390096  |  392144  |  0.52%\nsmacv2_20_units  |  43363  |  43875  |  1.18%\ncon-10x10x10a  |  388422  |  390470  |  0.53%\ncon-15x15x23a  |  389907  |  391955  |  0.53%\nsmacv2_10_units  |  201435  |  202459  |  0.51%\nmedium-6ag  |  389454  |  391502  |  0.53%\nmedium-4ag-hard  |  389580  |  391628  |  0.53%\nlarge-4ag-hard  |  389004  |  391052  |  0.53%\nlarge-8ag-hard  |  389520  |  391568  |  0.53%\nsmall-4ag-hard  |  389004  |  391052  |  0.53%\ntiny-4ag-hard  |  1080524  |  1082572  |  0.19%\nsmall-4ag  |  389196  |  391244  |  0.53%\nxlarge-4ag  |  389004  |  391052  |  0.53%\nmedium-4ag  |  389580  |  391628  |  0.53%\nxlarge-4ag-hard  |  1078796  |  1080844  |  0.19%\nlarge-4ag  |  389004  |  391052  |  0.53%\ntiny-2ag-hard  |  388746  |  390794  |  0.53%"}, {"review_id": "ySBCeoxp39", "reviewer": "Reviewer_LAZy", "summary": "The work studies different inference strategies and their effectiveness in improving multi-agent RL algorithms performances in a variety of difficult benchmarks. To this end, the work curates a set of difficult MARL problems and proposes adaptations of inference strategies for the MARL setting.", "questions": "* How do you see the relation to few-shot meta-RL approaches and fine-tuning in offline RL?\n* Do you expect that inference strategies always need full episodes or would it be possible to only use short episode stubs?\n* Can you provide real-world examples in which a user might permit inference time adaptation?", "limitations": "See my comments on the relationship to offline RL and meta-RL methods.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is well written and motivates the use of inference strategies for (MA)RL convincingly. The empirical evaluation is solid and highlights that the use of inference strategies can vastly improve MARL solutions, given that inference time adaptation is permissible.\nOverall I believe the work can be the basis of interesting discussions of the RL community at NeurIPS.\n\nI only see small weaknesses. The limitation to the MARL case seems the most restrictive limitation to me. It would have been interesting to see for at least some single-agent case how inference strategies perform. Further, in meta-RL few-shot learning is often done and a discussion of how this research is distinct to the use of inference strategies would be instructive. Similarly, offline RL often permits fine-tuning with real environment interactions after having learned from an offline dataset and a discussion of how these lines of work are related would put the presented work better into the overall RL context.", "was_revised": true, "final_justification": "The work was already easy to follow and convincingly showed the utility of using inference strategies for RL. The authors rebuttal response better connects the work to related RL research areas and highlights the practicality in real-world scenarios. Further, the additional experiment in the single-agent case solidifies that these strategies are not just helpful in the MARL setting.", "rebuttal": "We are grateful for the reviewer's feedback, especially their observation regarding the potential discussions our work could spark within the RL community. Below, we address the choice of benchmark, the connection to meta-RL and offline RL, and provide real-world examples, alongside proposed revisions for our manuscript.\n\n\\\n**> W1: Limitation to the MARL case. It would have been interesting to see for at least some single-agent case how inference strategies perform.**\n\nWe agree that discussing the single-agent case is valuable. In the following reply, we discuss published results on single-agent tasks, we provide new empirical evidence on a complex single-agent RL task (Craftax), and we reaffirm why our current benchmark is enough to sufficiently support our main message.\n\nFirst, we can refer to the experimental results reported in the seminal works that introduced the inference strategies used in our paper [3, 5, 6]. All of these evaluate on single-agent tasks (TSP, CVRP, JSSP, Knapsack, FFSP) and show that inference strategies bring statistically significant improvements over zero-shot performance. However, we recall the limitations we have identified on these experimental results (Section 2, lines 103 to 109): the limited breadth of the time/compute settings, methods’ reliance on problem-specific tricks and saturated benchmarks where zero-shot performance is already very high. \n\nWe however provide additional results to support our core message in the single-agent case within the time allowed for the rebuttal. Specifically, we trained a Sable base policy on Craftax [7] (ICML 2024) and matched the current state-of-the-art performance reported on the Craftax-1B leaderboard, which is 18.3% of the maximum score. We then ran a 30-second inference-time search with the simplest inference strategy (stochastic sampling). The results are reported in the table below. Strikingly, and similar to our main results, using only a 30-second inference-time search yields a 37% performance gain, reaching 25.8% of the maximum score. To further put this into context, this performance boost is of the same magnitude as that of the last 300 million steps of training the base policy.\n\nWe are happy to add these discussions and the additional results on Craftax in a new section in the Appendix of the revised manuscript.\n\nScenario  |  Zero-shot |  Stochastic Sampling  (30 Seconds budget)\n--- | --- | ---\nCraftax-1B | 18.8 (18.1, 19.5) | 25.8 (25.7, 26.1)\n\nWe nevertheless believe that our current benchmark and experimental protocol provides enough evidence in the main paper to support our message because: (i) it is peer-reviewed [1] and recognised in the RL community (accepted at Neurips 2021, widely cited), (ii) even the current SOTA method Sable [2] (ICML 2025) is far from optimal on a third of this benchmark, hence providing a 17-task suite to test our hypothesis, (iii) its formulation as a Dec-POMDPs makes it generic and closer to real-world RL settings.\n\nWe appreciate the opportunity to engage with this thoughtful discussion and hope that our response and additional results provide valuable insights.\n\n\\\n[1] Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, Neurips 2021 \\\n[2] Sable: a Performant, Efficient and Scalable Sequence Model for MARL, ICML 2025 \\\n[3] Combinatorial Optimization with Policy Adaptation using Latent Space Search, Neurips 2023 \\\n[5] Neural Combinatorial Optimization with Reinforcement Learning, ICLR 2017 \\\n[6] Simulation-guided Beam Search for Neural Combinatorial Optimization, Neurips 2022 \\\n[7] Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning, ICML 2024\n\n\\\n**> Q1: Relation to few-shot meta-RL and fine-tuning in offline RL.**\n\nWe acknowledge that relating meta-RL and offline-to-online RL to inference strategies is an interesting addition to the paper, and we are happy to add these to our related work (Section 2).\n\nInference strategies, just like meta-RL and offline-to-online RL, benefit from policy adaptation methods in order to produce better solutions. Consequently, methods developed in these fields can benefit the others. Interestingly, COMPASS shares similar concepts with VariBAD [8], a seminal work in meta RL: both condition a policy on a latent space and search this space to adapt. Other inference strategies from the literature have close links to meta learning. For instance, MEMENTO [9] shares similarities with RL2 [10], using a memory to search (respectively to adapt). And DIMES [11] is an inference strategy which explicitly does meta RL (following MAML’s methodology [12]). We do not compare to DIMES because it is highly domain-specific (only applicable to graphs, e.g. TSP and MIS). While we are not aware of inference strategies directly inspired by the offline-to-online RL literature, methods such as Cal-QL [13] and PA-RL [14] introduce adaptation mechanisms that would likely be beneficial during inference.\n\nWe will also recall some fundamental distinctions between these fields. In particular, inference strategies care about (i) the solutions found rather than the policy obtained, (ii) the maximum performance rather than average performance, and (iii) treat single instances at a time, whereas most meta-RL and offline-to-online RL methods adapt from a distribution to another distribution.\n\n\\\n[8] VARIBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning, ICLR 2020 \\\n[9] Memory-Enhanced Neural Solvers for Efficient Adaptation in Combinatorial Optimization, arxiv (2024) \\\n[10] RL^2: Fast Reinforcement Learning via Slow Reinforcement Learning, arxiv (2016, cited 1000+) \\\n[11] DIMES: A Differentiable Meta Solver for Combinatorial Optimization Problems. Neurips 2022 \\\n[12] Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks, ICML 2017 \\\n[13] Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning, Neurips 2023 \\\n[14] Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone, ICLR 2025\n\n\\\n**> Q2: Would it be possible to use short episode stubs?**\n\nWhen per-timestep rewards are accessible, it is indeed possible to use short episode stubs instead of entire episode rollouts. We stayed as close as possible to the existing methods published in the literature; hence, we only ever used full episode information, but we do not see any restrictions to extend these methods to use smaller episode stubs since the benchmarks we are using provide per-timestep rewards.\n\n\\\n**> Q3: Can you provide real-world examples in which a user might permit inference time adaptation?**\n\nApplying RL in real-world applications is a key driver of our research. Here are three such real-world cases where we have first-hand experience: in all these, the user would allow for a time budget, and a proxy of the solution’s quality is accessible to inform the search:\n- Printed Circuit Board Routing. The user submits the specification of a PCB board they want to route. This task can take up to several days/weeks, even for an expert, depending on the complexity of the board. Therefore, users are typically completely satisfied with allowing an AI model to take at least a few minutes or even hours for search to obtain a better solution (routed board). Additionally, the user will find it beneficial to leave more time and to grant more computation in order to improve the board further, which could then be easier to manufacture or reduce its production cost. \n- Bin Packing. When a logistics company wants to load containers with objects that need to be shipped overseas. The operators can typically have a few hours or days when planning which objects need to be transported, and can hence allow inference-time search. Here as well, the energy and money gains from having a better object dispatching (thus fewer containers to ship for the same amount of objects) will always justify a few hours of search (or the allowance of more computing spend).\n- Train scheduling: designing train schedules to meet some demand forecast, while proposing a feasible schedule (e.g., no collisions between trains). This only happens once or twice a year, and takes months, hence no need to rely on a single rollout of the trained policy, and an inference-time search can be conducted over several days.\n\nWe appreciate the reviewer's suggestion; these examples will be added to a new appendix section to illustrate the link of this work with real-world applications."}, {"review_id": "FLVLmsX8h1", "reviewer": "Reviewer_RVBa", "summary": "This paper investigates several test-time inference strategies for solving complex multiagent tasks using RL. Over an extensive suite of chosen benchmarks, it was shown that test-time inference consistently improved over baselines that didn’t use the technique, where a diversity-driven approach previously proposed for combinatorial optimization performed the best overall both in task scores and scalability.", "questions": "- Please provide more experiment details and/or report error bars for figure 4.\n- Since the experiments were only done on multi-agent domains, perhaps the title is a bit of an overly stated claim. Changing it to emphasize the multiagent aspect would be preferred. Either that, or include additional single-agent RL experiments", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strength\n- The paper is clearly written and easy to follow.\n- The empirical performance of the approaches is very impressive, especially given the wide ranging selection of tasks.\n- The analysis experiments in figure 6 and 7 give very practical takeaways on future usage of this technique.\n- Experiment results of figure 5 has nice statistical significance.\n\n### Weakness\n- The algorithmic novelty is rather limited, as it is simply applying a now widely-known principle from works on LLM inference-time compute. The application domain, however, is novel.\n- Experiment results of figure 4 did not report any quantification of statistical significance.", "was_revised": true, "final_justification": "This was a very solid empirical paper to begin with. Both my issues of the \"single-agent claim\" and lack of uncertainty quantification in figure 4 have been properly addressed; I gave both of these issues equal weighting. The author's provided context on the single-agent setting was nicely done as well. Hence I recommend rating of 5.", "rebuttal": "We thank the reviewer for their feedback and for highlighting the strengths of our paper.\n\n\\\n**> W1: The algorithmic novelty is rather limited, as it is simply applying a now widely-known principle from works on LLM inference-time compute.**\n\nOur main message is that practitioners often miss an opportunity to achieve significant performance improvements. Hence, this message is made stronger by the fact that these methods are already accessible, and often ignored despite having been published in top-tier conferences several years ago: online fine-tuning [5] at ICLR 2017, SGBS [6] at Neurips 2022, and COMPASS [3] at Neurips 2023.\n\nMethods presented in the paper are not all applicable to LLMs (e.g. online fine-tuning is intractable for LLMs) and are more closely related to the RL for CO literature. Even though the algorithmic contribution is not the core contribution of our paper, it is worth mentioning that we are the first to generalise COMPASS to the multi-agent setting.\n\nIn summary, we agree with the reviewer that our algorithmic novelty is rather limited; however, we emphasize that this aspect reinforces the significance of our main message.\n\n\\\n[3] Combinatorial Optimization with Policy Adaptation using Latent Space Search, Neurips 2023 \\\n[5] Neural Combinatorial Optimization with Reinforcement Learning, ICLR 2017 \\\n[6] Simulation-guided Beam Search for Neural Combinatorial Optimization, Neurips 2022\n\n\\\n**> Q1: Please provide more experiment details and/or report error bars for Figure 4.**\n\nWe thank the reviewer for pointing this out. Please see the table below where we give the mean and 95% bootstrap confidence intervals. The 20M timestep values are aggregated over 10 independent runs and computed directly from the benchmarking data provided along with the Sable [2] paper, for the policy trained to convergence and for COMPASS, we give the results aggregated over 128 independent runs. \n\nTo avoid overcrowding Figure 4 in the main paper, we propose to include these detailed confidence intervals and error bars in a dedicated table and an extended version of the plot, both placed in the Appendix. We will clearly reference this supplementary information in the caption of Figure 4.\n\n\\\n[2] Sable: a Performant, Efficient and Scalable Sequence Model for MARL, ICML 2025\n\n\\\nValues and error bars of results reported in Figure 4:\n\nScenario  |  20M Timesteps  |  Converged  |  COMPASS\n--- | --- | --- | ---\ncon-10x10x10a | 0.388 (0.372, 0.404) | 0.500 (0.414, 0.586) | 0.953 (0.914, 0.984)\ncon-15x15x23a | 0.075 (0.067, 0.083) | 0.078 (0.039, 0.125) | 0.867 (0.805, 0.922)\nlarge-4ag | 0.246 (0.200, 0.283) | 0.642 (0.618, 0.664) | 0.949 (0.940, 0.957)\nlarge-4ag-hard | 0.175 (0.113, 0.225) | 0.609 (0.581, 0.633) | 0.933 (0.922, 0.943)\nlarge-8ag | 0.203 (0.197, 0.208) | 0.496 (0.464, 0.526) | 0.953 (0.947, 0.959)\nlarge-8ag-hard | 0.209 (0.202, 0.216) | 0.516 (0.489, 0.541) | 0.957 (0.950, 0.963)\nmedium-4ag | 0.376 (0.346, 0.396) | 0.666 (0.642, 0.686) | 0.947 (0.941, 0.953)\nmedium-4ag-hard | 0.253 (0.220, 0.282) | 0.701 (0.677, 0.721) | 0.949 (0.943, 0.955)\nmedium-6ag | 0.240 (0.227, 0.250) | 0.702 (0.668, 0.733) | 0.958 (0.954, 0.963)\nsmacv2_10_units | 0.648 (0.623, 0.675) | 0.852 (0.789, 0.914) | 0.977 (0.945, 1.000)\nsmacv2_20_units | 0.599 (0.548, 0.639) | 0.719 (0.641, 0.797) | 0.969 (0.938, 0.992)\nsmall-4ag | 0.384 (0.243, 0.484) | 0.728 (0.706, 0.746) | 0.963 (0.956, 0.969)\nsmall-4ag-hard | 0.349 (0.327, 0.371) | 0.707 (0.687, 0.725) | 0.955 (0.948, 0.962)\ntiny-2ag-hard | 0.664 (0.646, 0.684) | 0.810 (0.799, 0.821) | 0.978 (0.967, 0.989)\ntiny-4ag-hard | 0.548 (0.529, 0.566) | 0.783 (0.756, 0.806) | 0.958 (0.951, 0.964)\nxlarge-4ag | 0.193 (0.118, 0.261) | 0.485 (0.465, 0.504) | 0.905 (0.893, 0.916)\nxlarge-4ag-hard | 0.065 (0.001, 0.161) | 0.394 (0.373, 0.413) | 0.892 (0.879, 0.906)\n\n\\\n**> Q2: Since the experiments were only done on multi-agent domains, perhaps the title is a bit of an overly stated claim.**\n\nWe agree with the reviewer that the inclusion of single-agent experiments is helpful to substantiate our title. In the following reply, we discuss published results on single-agent tasks, we provide new empirical evidence on a complex single-agent RL task (Craftax), and we reaffirm why our current benchmark is enough to sufficiently support our main message.\n\nFirst, we can refer to the experimental results reported in the seminal works that introduced the inference strategies used in our paper [3, 5, 6]. All of these evaluate on single-agent tasks (TSP, CVRP, JSSP, Knapsack, FFSP) and show that inference strategies bring statistically significant improvements over zero-shot performance. However, we recall the limitations we have identified on these experimental results (Section 2, lines 103 to 109): the limited breadth of the time/compute settings, methods’ reliance on problem-specific tricks and saturated benchmarks where zero-shot performance is already very high. \n\nWe however provide additional results to support our core message in the single-agent case within the time allowed for the rebuttal. Specifically, we trained a Sable base policy on Craftax [7] (ICML 2024) and matched the current state-of-the-art performance reported on the Craftax-1B leaderboard, which is 18.3% of the maximum score. We then ran a 30-second inference-time search with the simplest inference strategy (stochastic sampling). The results are reported in the table below. Strikingly, and similar to our main results, using only a 30-second inference-time search yields a 37% performance gain, reaching 25.8% of the maximum score. To further put this into context, this performance boost is of the same magnitude as that of the last 300 million steps of training the base policy.\n\nWe are happy to add these discussions and the additional results on Craftax in a new section in the Appendix of the revised manuscript.\n\nScenario  |  Zero-shot |  Stochastic Sampling  (30 Seconds budget)\n--- | --- | ---\nCraftax-1B | 18.8 (18.1, 19.5) | 25.8 (25.7, 26.1)\n\nWe nevertheless believe that our current benchmark and experimental protocol provides enough evidence in the main paper to support our message because: (i) it is peer-reviewed [1] and recognised in the RL community (accepted at Neurips 2021, widely cited), (ii) even the current SOTA method Sable [2] (ICML 2025) is far from optimal on a third of this benchmark, hence providing a 17-task suite to test our hypothesis, (iii) its formulation as a Dec-POMDPs makes it generic and closer to real-world RL settings.\n\nWe appreciate the reviewer raising this important point and hope the additional single-agent results provided here demonstrate the generality of our main message.\n\n\\\n[1] Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks, Neurips 2021 \\\n[2] Sable: a Performant, Efficient and Scalable Sequence Model for MARL, ICML 2025 \\\n[3] Combinatorial Optimization with Policy Adaptation using Latent Space Search, Neurips 2023 \\\n[5] Neural Combinatorial Optimization with Reinforcement Learning, ICLR 2017 \\\n[6] Simulation-guided Beam Search for Neural Combinatorial Optimization, Neurips 2022 \\\n[7] Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning, ICML 2024"}]

Author rebuttal:

Meta-review: The paper studies RL for combinatorial optimization problems, and convincingly demonstrates through extensive experiments that employing an inference phase with a time & compute budget allows finding significantly better solutions than those found zero-shot by a trained RL policy. All of the reviewers agreed that the experimental evidence is comprehensive & sound, the problem is very well motivated, and the exposition is clear. During the discussion phase, the authors contributed additional results on a single-agent RL problem (to go beyond only the multi-agent settings in the initial version), provided error bars on their main results using bootstrapping, and clarified the increase in parameter count in having the policy be able to process latent conditioning vectors. These results and the discussion situating the work in the broader topic of RL policy adaptation (e.g. meta-learning) substantially strengthen the paper. All the reviewers agreed that the paper is above the bar for publication, and likely to spark follow-up work and interesting conversations and so I recommend an oral slot for the paper.
Decision: Accept (oral)