Title: Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategies
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

inference strategies

complex decision-making
Abstract: Reinforcement learning (RL) systems have countless applications, from energy-grid management to protein design. However, such real-world scenarios are often extremely difficult, combinatorial in nature, and require complex coordination between multiple agents. This level of complexity can cause even state-of-the-art RL systems, trained until convergence, to hit a performance ceiling which they are unable to break out of with zero-shot inference. Meanwhile, many digital or simulation-based applications allow for an inference phase that utilises a specific time and compute budget to explore multiple attempts before outputting a final solution. In this work, we show that such an inference phase employed at execution time, and the choice of a corresponding inference strategy, are key to breaking the performance ceiling observed in complex multi-agent RL problems. Our main result is striking: we can obtain up to a 126% and, on average, a 45% improvement over the previous state-of-the-art across 17 tasks, using only a couple seconds of extra wall-clock time during execution. We also demonstrate promising compute scaling properties, supported by over 60k experiments, making it the largest study on inference strategies for complex RL to date. We make all of our experimental data and code available.


Peer reviews:
comments: SUMMARY:
This paper investigates several test-time inference strategies for solving complex multiagent tasks using RL. Over an extensive suite of chosen benchmarks, it was shown that test-time inference consistently improved over baselines that didn’t use the technique, where a diversity-driven approach previously proposed for combinatorial optimization performed the best overall both in task scores and scalability.

STRENGTHS AND WEAKNESSES:
### Strength
- The paper is clearly written and easy to follow.
- The empirical performance of the approaches is very impressive, especially given the wide ranging selection of tasks.
- The analysis experiments in figure 6 and 7 give very practical takeaways on future usage of this technique.
- Experiment results of figure 5 has nice statistical significance.

### Weakness
- The algorithmic novelty is rather limited, as it is simply applying a now widely-known principle from works on LLM inference-time compute. The application domain, however, is novel.
- Experiment results of figure 4 did not report any quantification of statistical significance.

QUESTIONS:
- Please provide more experiment details and/or report error bars for figure 4.
- Since the experiments were only done on multi-agent domains, perhaps the title is a bit of an overly stated claim. Changing it to emphasize the multiagent aspect would be preferred. Either that, or include additional single-agent RL experiments

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 1
DATE: 2025-06-27

comments: SUMMARY:
The work studies different inference strategies and their effectiveness in improving multi-agent RL algorithms performances in a variety of difficult benchmarks. To this end, the work curates a set of difficult MARL problems and proposes adaptations of inference strategies for the MARL setting.

STRENGTHS AND WEAKNESSES:
The paper is well written and motivates the use of inference strategies for (MA)RL convincingly. The empirical evaluation is solid and highlights that the use of inference strategies can vastly improve MARL solutions, given that inference time adaptation is permissible.
Overall I believe the work can be the basis of interesting discussions of the RL community at NeurIPS.

I only see small weaknesses. The limitation to the MARL case seems the most restrictive limitation to me. It would have been interesting to see for at least some single-agent case how inference strategies perform. Further, in meta-RL few-shot learning is often done and a discussion of how this research is distinct to the use of inference strategies would be instructive. Similarly, offline RL often permits fine-tuning with real environment interactions after having learned from an offline dataset and a discussion of how these lines of work are related would put the presented work better into the overall RL context.

QUESTIONS:
* How do you see the relation to few-shot meta-RL approaches and fine-tuning in offline RL?
* Do you expect that inference strategies always need full episodes or would it be possible to only use short episode stubs?
* Can you provide real-world examples in which a user might permit inference time adaptation?

LIMITATIONS:
See my comments on the relationship to offline RL and meta-RL methods.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
The paper applied different searching strategies to multi-agent RL and showed that the COMPASS is the best compared to greedy, stochastic, tree search and fine-tuning.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The paper is well motivated and the experimental results are promising. 
2. The paper is well written, the contribution is fair
Weakness:
1. It will be good to explain details of how the specialized policy in COMPASS is trained. It will be good to show the comparison of the model size.

QUESTIONS:
1. It would be good to explain details of how the specialized policy in COMPASS is trained. It will be good to show the comparison of the model size.
2. It would be good to explain the reason behind why the COMPASS is better

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
The authors found that using small inference time budgets can significantly improve the performance of MARL agents, compared to the zero-shot setting. The typical RL literature usually focuses on a zero-shot setting, meaning the RL agent can one attempt at each time step during evaluation, whereas making multiple attempts before giving a final result has become more popular in the LLM space. 

The authors took inspiration from making multiple attempts during inference and applied it in MARL. They conducted experiments on a wide range of MARL tasks, from the Multi-Robot Warehouse, the StarCraft Multi-Agent challenge, to Connector. In all the experiments, the authors found that inference strategies can push up the final performance higher.

STRENGTHS AND WEAKNESSES:
Strengths:

Experiments seem comprehensive, covering a wide variety of tasks. The gains from inference strategies are solid. It is also surprising that even a small inference budget (30 seconds) could make a big impact on the model's performance. Charts seem reasonable, too. For example, Figures 5 and 6 show nicely how more parallel attempts can unlock greater performance.

Weaknesses:

It feels like the benchmark environments are already somewhat saturated.

QUESTIONS:
I am not sure I follow how the inference strategies work in MARL. The model is performing actions at every time step. How do inference strategies work? Do the models consider multiple actions before deciding on a single action at each time step?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 1
ORIGINALITY: 2
DATE: 2025-07-03