YRC-Bench: A Benchmark for Learning to Coordinate with Experts
Authors: Mohamad H. Danesh
Khanh Xuan Nguyen
Tu Trinh
Benjamin Plaut
Venue: Accepted by TMLR ()

Abstract: When deployed in the real world, AI agents will inevitably face challenges that exceed their individual capabilities. A critical component of AI safety is an agent’s ability to recognize when it is likely to fail in a novel situation and to yield control to a more capable expert system. Leveraging such expert assistance can significantly improve safety and performance in such situations. Since expert assistance is costly, a central challenge is determining when to consult an expert. In this paper, we explore a novel variant of this problem, termed YRC-0, in which an agent must learn to collaborate with an expert in new environments in an unsupervised manner–that is, without interacting with the expert during training. This setting motivates the development of low-cost, robust approaches for training expertleveraging agents. To support research in this area, we introduce YRC-Bench, an open-source benchmark that instantiates YRC-0 across diverse environments. YRC-Bench provides a standardized Gym-like API, simulated experts, an evaluation pipeline, and implementations of popular baselines. Toward tackling YRC-0, we propose a validation strategy and use a proposer-validator decomposition as a diagnostic framework to evaluate a range of learning methods, offering insights that can inform future research. Codebase: https://github.com/modanesh/YRC-Bench

Published in Transactions on Machine Learning Research (11/2025)
YRC-Bench: A Benchmark for Learning to Coordinate with Experts
Mohamad H. Danesh"
mohomad danesh @eaiilesemmmmlmmmmm dangill
McGill University
Mila Quebec Al Institute
Nguyen X. Khanh
University of California, Berkeley
Tu Trinh
Universily of California, Berkeley
Benjamin Plaut
University of California, Berkeley
Reviewed on OpenReview:
Abstract
When deployed in the real world, AI agents will inevitably face challenges that exceed their individual capabilities. A critical component of Al safety is agent's 's ability to recognize when it is likely to fail in a novel situation and to yield control to a more capable expert system. Leveraging such expert assistance can significantly improve safety and performance in such situations. Since expert assistance is costly, a central challenge is determining when to consult an expert. In this paper, we explore novel variant of this problem, termed YRC-0, in which an agent must learn to collaborate with an expert in new environments in unsupervised manner that is, without interacting with the expert during training. This setting motivates the development of low-cost, robust approaches for training expert- leveraging agents. To support research in this arca, we introduce YRC-Bench, an open-source benchmark that instantiates YRC-0 across diverse environments. YRC-Bench provides a standardized Gym-like API, simulated experts, an evaluation pipeline, and implementations of popular baselines. Toward tackling YRC-0, we propose a validation strategy and use proposer-validator decomposition a diagnostic framework to evaluate a range of learning methods, offering insights that can inform future research.
Codebase: github.com/mnodaeeshy http://mooodeee///CBeBeeehc
1 Introduction
Deploying AT agents in real-world environments presents a critical challenge: agents must operate effectively dynamic and unpredictable settings, where their individual capabilities are often insufficient to ensure success (Amodei et al., 2016; Leike al., 2017; Zhou al., 2024). promising solution to equip agents with the ability to leverage assistance from more capable (human AT) experts (Sadigh al., 2016; Reddy et al., 2018; Nguyen et al., 2021; Ren et al., 2023). While providing expert assistance be costly-rrquirrng trained personnel or complex, resource-intensive systems it is often a worthwhile investment, given its potential to prevent catastrophic failures and outperform purely autonomous agents. Moreover, expert costs can be significantly reduced if an agent can intelligently docide when it requires assistance. To build such agents, we study the problem Vielu-or-Requet Control (YRC), where the goal is to train an agent to decide when to
Work done while interning UC Berkeley's 's Center for Hamman-Comaatblle AT (CHAT).

---

Published in Transactions on Machine Learning Research (11/2025)
consult expert in order to succeed with minimal assistance. In this work, we deliberately use nearly-ideal expert policies. This methodological choice allows us to isolate the foundational challenge of an agent's 's ability to assess its own competence under distribution shift, separating it from the orthogonal and equally complex-challenge of modeling and coordinating with a fallible, non-stationary human partner. In many safety-critical applications, AT systems must operate under strict constraints that prevent online learning during deployment. These constraints motivate our problem setting: an agent must learn to recognize its own incompetence and yield control before deployment, as it will have no opportunity to learn from failures in the field.
The YRC problem have been explored in various contexts. Previous research often assumes expert availability during the training phase (Nguyen et al., 2019; Nguyen & Daumé III, 2019; Thomason et al., 2020; Xie et al., 2022). Yet, in many real-world scenarios, experts are unavailable or prohibitively costly to employ or simulate. Other settings focus on in-distribution ovaluations, assessing agents in environments highly similar to those seen during training (Chernova & Veloso, 2009; Natarajan et al., 2024; Hu et al., 2020) Such scttings risk encouraging brittle solutions which fail under drastic distribution shifts.
Toward developing sample-efficient and robust YRC methods, we introduce a novel and practical variant of the YRC problem called YRC-0 (Fig. 1). Our problem motivates the search for unsupervised approaches to YRC. Specifically, in this setting, an agent undergoes a fully autonomous training phase in which it learns a set of tasks independently, without any communication with an expert. At test time, the agent faces novel tasks and may request expert assistance to enhance performance.
Developing unsupervised solutions to YRC is essential given the widespread adoption and inherent vulnerability of fully autonomous training paradigms, such reinforcement learning (RL) and behavior cloning, and the high costs of training and managing domain experts. Even in scenarios where experts are available, unsupervised approaches could serve as an effective pre-training step, significantly reducing the costs of expert queries during subsequent fine-tuning steps.
In this paper, we present YRC-Bench, an open-source benchmark that provides comprehensive infrastructure for conducting empirical research YRC in general, and YRC-0 in particular. The extensive collec- tion of environments in YRC-Bench supports a wide range of research rumning simple experiments to validate theoretical claims to developing large-scale, multi-environment learning and evaluation approaches. In particular, multi-cnvironment evaluation encourages the development of robust methods that do not exploit. the idiosyncrasies of any single environment. Furthermore, automated evaluation enabled by simulated experts makes experimentation cost-effective, efficient, safe, and reproducible. In developing this benchmark, we not only overcome engineering challenges but also introduce novel solutions to problems in training and evaluating policies. We relcase an extensible codebase to support future contributions, enabling easy integration of new environments and methods.
Training
Testing
Figure 1: Illustration of the YRC-0 problem. Left: agent learns tasks on its own (e.g., RL with environment rewards). Right: at test time, it has to perform novel tasks with the help of an expert. While learning in isolation, how can the agent develop a collabo- ration strategy that will be effective at test time?
Utilizing YRC-Bench, we take initial steps toward developing robust solutions to YRC-0. We conduct large-scale empirical study, training over 2,600 policies and comparing 10 learning methods across 19 envi- ronments. This study expends over 1,300 GPU days (NVIDIA A6000),
yielding several noteworthy findings: (1) no method consistently outperforms others, (2) a substantial performance gap remains between current methods and an oracle approach, and (3) this gap stems largely from the simplicity of the policy class considered by these methods, rather than our proposed validation strategy. We distill these findings into concrete recommendations to inform future research.
2

---

Published in Transactions on Machine Learning Research (11/2025)
2
Related work
Human-AI collaboration has seen growing interest in recent years (Shnciderman, 2022; Wu et al., 2022; Pflanzer et al., 2023; Fragiadakis et al., 2024; Vals et al., 2024). Various settings have been explored in this area. Closest to our work are approaches that enable agents to collaborate with partners who have superior knowledge or skills in a domain (whom we call "experts" in this work). Several works enable this capabiliiy to reduce supervision during training (Chernova & Veloso, 2009; Judah et al., 2014). At test time, however, the agent in these settings functions autonomonsly. Other works focus on building truly collaborative agents which can leverage expert assistance at test time. However, these approaches often assume certain forms of supervision during training time, such as the presence of the expert for interaction (Nguyen & Daumé III, 2019; Nguyen et al., 2019; 2021), an offline dataset recording interactions with experts (Thomason et al., 2020; Padmakumar et al., 2022), or labels of dangerous states (Xic et al., 2022). Our proposed setting takes the challenge to the extreme by assuming no supervision during training, with the goal of developing unsupervised methods that can complement existing supervised approaches. Moreover, it tests the generalizability of the learned policy to novel environments an aspect overlooked by work embracing the traditional single-enviromment RL setting (Natarajan et al.; 2024; Da Silva & Costa, 2019). Our work differs from online learning settings where an expert is available for queries during training to minimize regret (Cohen et al., 2022), as we focus on the zero-shot case where no expert interaction is possible before deployment.
RL Generalization and Self- Assessment in OOD Settings. The core challenge in YRC-0-a 0-a0 agent detecting its own incompetence under an out-of-iitributioo (OOD) shift is deeply connected to several established research areas. Our work relates to a broad literature on RL generalization under distribution shifts, which has typically focused on single-agent settings (Danesh & Fern, 2021; Liu et al., 2021; Paudel, 2022; Haider et al., 2023; Yang et al., 2024; Nasvytis et al., 2024). More specifically, YRC-0 interseets with Safe RL, where the goal is to identify unsafe states and revert to a safe policy (Xue-Song et al., 2023), and Uncertainty Quantification, where methods estimate model confidence to trigger fallback action (Lockwood & Si, 2022). YRC-Bench provides benchmark that integrates these challenges into a sequential decision-making problem with cost-sensitive expert queries. It is distinct from prior work on zero-shot coordination (Hu el al., 2020), which focuses on coordinating with novel partners but not in novel environments. Finally, YRC-0 serves as a challenging testbed for methods from meta-learning, where an agent must learn an adaptation strategy applicable to new tasks (Hospedales et al., 2021).
Active learning seeks to minimize annotation cost for training a model. Common methods include uncertainty-based (Lewis, 1995; Settles, 2009), query-by-commiteer (Roy & McCallum, 2001), submodular maximization (Hoi et al., 2006; Fujii & Kashima, 2016), and deep Bayesian (Gal et al., 2017). While active learning can be cast as an YRC problem, it typically considers a supervised learning problem where data points are identically independently distributed. In contrast, YRC-0 is a sequential decision-making problem with correlated input data. Although sequential variants of active learning have been proposed (Chernova & Veloso, 2009; Judah et al., 2014), the query policy in these settings is learned and used only during training (i.e. no distribution shift for this policy), whereas an YRC-0 policy is primarily used at test time and evaluated under out-of-distributio (OOD) setting.
3 The YRC-0 problem
YRC-0 is concerned with making a sequence of decisions, each of which asks: in this situation, should on agent make decision on ils own or seek advice from an expert? A reward function is specified to evaluate these decisions. As in any RL problem, the goal is to maximize the expected return (cumulative reward) under distribution of states and actions induced by the learned policy.
Formally, we define a lask as a Markov decision process (MDP) with state space S, action space A, reward function R. x A + R, initial state distribution Po: and transition function P: x A + ASS), where ((X) denotes the probability simplex over a set X. A task distribution E is a distribution over all possible tasks with the same action and state spaces. novice policy Tn : S S ((A) is trained with tasks sampled from distribution Erain and evaluated on tasks sampled from a different distribution Exst # Etriin- An
3

---

Published in Transactions on Machine Learning Research (11/2025)
ezpert with policy Fe S - A(A) is present only at test time to assist the novice on the test tasks. Importantly, we assume that the novice achieves high performance on Errain and low performance on &rest, and the expert achieves high performance on Eteat. Our setting therefore simulates a typical fully autonomous training procodure, after which the trained agent excels on tasks similar to those it is trained on, but experiences performance degradation when presented with OOD tasks. We assume the novice achieves high performance on Etrain and low performance on Étest not as a hard requirement, but as the standard consequence of significant distribution shift the very scenario that makes expert assistance necessary. If the novice performed well on Erest; expert assistance would be unnecessary.
The goal of YRC-0 is to Iearn a coordination pol- icy a : S x 4n + A((n, that decides at each time step t, whose policy (novice or expert) will be used for deciding the next action. Here, 4 rep- resents the space over internal representations ex- tracted from n. during its decision-making process. Specifically, in state 5t, while the expert computes Tn (S:), representation St) E 4r is extracted (e.g., the activations of neural network). The co- ordination policy ja then makes a binary decision Z4 ~ p(st; ((Tn, st) E {n,e), This decision is trans- lated into an environment action at ~ (8t) which is then executed in the environment, generating the next state $1+1- Hence, j induces an MDP with state space S and action space {n, e). Fig. 2 illustrates this MDP.
St
S Tux $1
2 ?
novice
I1 E {n.e)
o
coord
expert
Figure 2: Simulation of the YRC-0 problem. A coordination environment encapsulates two policies: novice and expert. A coordination policy decides which policy will be used to generate the next action. The coordination policy's decision is then translated into an environment action and executed.
With this formulation, we do not assume any specific form of expert advico; it can be demonstration, language utterance, etc. However, to focus on the problem of when asks for advice, we assume that the novice's understanding of advice is perfect; each piece of advice has been interpreted into a sequence of executable actions which are stored in the novice's memory. Whenever x, 2,=6 e, the novice retrieves the next action from memory and executes it.' The novice does not necessarily bother the expert every time it decides 22 c.
At test time, is evaluated on E where Te is present to assist T., For the learning of μe, however, We is unavailable. The challenge of YRC-O is to construct a learning method that can find an "effective" coordination policy having access to only Tn and Eurain: where the effectiveness is determined by a reward function described $4.2. YRC-0 is an OOD generalization challenge, where the distribution shift is caused by two factors: the novel environment dynamics and the introduction of the expert. The latter factor adds a zero-shot coordination challenge to the problem, distinguishing it from RL generalization challenges that focus only on environmental changes Cobbe et al. (2020); Yuan et al. (2023); Pumacay et al. (2024).
It is important to clarify why we forbid the novice or coordination policy from updating parameters during the test phase (online learning). In many high-stakes real-world applications, such as medical diagnosis or autonomous driving, online exploration is prohibited due to safety risks. An agent cannot "try and fail" to learn in a novel environment. It must reliably identify when its pro-trained capabilities are insufficient and yield control immedialely. Furthermore, even in settings where online learning technically feasible, regulatory frameworks often require pre-deployment certification, making online modifications problematic. Finally, regarding the novice's performance, we assume it performs poorly on Etest not a hard requirement, but as the standard consequence of significant distribution shift- the very scenario that makes expert assistance necessary. YRC-0 specifically isolates the safety-critical capability of recognizing this competence gap before a failure occurs.
By forbidding interaction with the expert during training, we encourage the development of unsupervised learning methods for YRC. Such methods, even if imperfect, can greatly reduce the cost of expert involvement. From a pragmatic point of view, removing the presence of the expert makes it casy instantiate the problem
1 One can imagine that the novice stores these actions in a queue, and whenever xt = e, pops the next action from the quene, If no action available, requests a new advice from the expert to refill the queue.
4

---

Published in Transactions on Machine Learning Research (11/2025)
in various environments. In contrast, allowing interactions with experts requires specifying an interaction budget for each environment- -a non-trivial task as a single value cannot represent various real-world scenarios.
The proposer-valication decomposition. In this work, consider a class of learning methods that can be decomposed into two components: a policy proposer P and a policy validator V. During training, P considers a policy class and generates finite set of candidate policies (2I, from this class which are then evaluated by V, predicting their test performances. The best candidate is chosen for testing. For example, a deep RL method considers policies parameterized by neural networks. It employs a gradient- optimizer as the policy proposer, which continuously updates the set of paramcters generate candidates for validation. Another example is to query the experl with probability p al each time step. A grid search over finite subset of [0, 1] can be used as the policy-proposiig approach. We introduce this decomposition primarily as diagnostic framework for the class of threshold-based and selection-based methods studied in this work. While gradient-based methods (where the "proposer" is the optimizer and the "validator" is the loss function) can theoretically be viewed through this lens, our focus here is on explicit candidate generation and selection. This separation allows us to isolate failure modes: is the method failing because it cannot generate a good policy (proposcr failure), or because it cannot identify the good policy among candidates (validator failure)? This distinction is crucial for understanding why current. unsupervised baselines si. struggle.
4 YRC-Bench: Robust evaluation of learning methods across diverse environments
Prior work on YRC employs diverse training and evaluation settings, making it difficult to compare results. To the best of our knowledge, no study has systematically evaluated these approaches across a large collection of environments. Is there a general approach to YRC that performs well across many domains, or does the no-froc-lunch principle apply? We develop YRC-Bench as a research tool to address this question. In addition to assembling a collection of environments, we address practical challenges to enable rigorous yet cost-effective evaluation. Our goal is to enable researchers to easily implement new methods, quickly compare them against existing approaches across diverse domains, and draw generalizable insights into their strengths and limitations.
4.1 Overview
YRC-Bench built on top of three environment suites: MiniGrid (Chevalier- -oiisverr et al., 2023), Procgen (Cobbe et al., 2020), and CLIPort (Shridhar et al., 2021), each offering unique challenge. MiniGrid features grid-based tasks with abstract state represontations and partial obscrvability. These environments customizable, making them suitable for simple, controlled experiments to illustrate a phenomenon or validate a theoretical claim. Procgen consists of procedurally generated video games. The main challenges of these tasks are pixel-based observations, stochastic dynamics, and long horizon (some task requires ~880 steps to complete). CLIPort is a suite of grgguagg-giided robotic manipulation tasks that require grounding language instructions in an RGB-D observation space and continuous action space. A common feature of these environments is that they support the creation of novel environment dynamics (Fig. 3a). Especially, CLIPort requires understanding compositionally novel task instructions. Test performance of standard approaches is lar from perfect (Fig. 3b), showing the difficulty of the generalization challenge.
highly
YRC-Bench is designed for extensibility. It supports easy integration of any environment following the gym3 interface. 3 We also defined standard interfaces for the model, policy, and trainer classes to facilitate modular development and ensure compatibility across components.
4.2 Resolving evaluation challenges: expert, evaluation metric, and tracking progress
Simulated expert. YRC-Bench provides expert policies that emulate real-world experts, enabling large-scale evaluation without the cost, risk, and complexity of employing human operators or heavy AI systems. To
Foor Proegen and CLIPort, we use the approaches proposed by the original work. For Minigrid, we use 3-layer convolutional max-pooling policy to encode the observation and pass the encoded vector through a GRU RNN 3 hhttps:// gi thub, con/openai/gym3
5

---

Published in Transactions on Machine Learning Research (11/2025)
ND bud deda
AUC
10
(a)
(b)
(c)
Figure 3: (a) Training (top) and test (bottom) tasks in DoorKey (Minigrid), CoinRun (Procgen), and stack- block- pyramid (CLIPort). (b) The generalization gaps of the novice: its average return on test lasks, normalized by average return of expert. (c) To evaluate policies, compute the mean and standard deviation of the under the curve defined by the average return at varying values a and random seed.
construct these experts, we train PPO (Schulman et al., 2017) policies on &test. for MiniGrid and Proegen, and employ a rule-based oracle for CLIPort. These experts are nearly ideal: they perform well and respond to all help requests. Nevertheless, our experiments show that current methods struggle even in this simplified setting. This highlights the fundamental difficulty of the problem of identifying risk states. We argue that it is important first address this core challenge before extending to more complex scenarios, which introduce additional, orthogonal difficulties.
Reward function. YRC-0 is fundamentally multi-objective problem: the coordination policy should maximize the task's return (cumulative reward) while minimizing the cost of expert assistance. To enable direct comparison of methods, it is essential to convert this problem into singlo-objective problem. Facing similar issues, previous work employs a hard-constraint approach: either maximizing the reward given a budget of assistance cost (Nguyen et al., 2019), or minimizing expert assistance to achieve a target reward (Ren el al., 2023). In practice, hard constraints may not be intuilive to specify and need to be frequently adjusted as the learning method improves.* Alternatively, one can adopt a soft-constraint approach by maximizing linear combination of return and assistance cost. In this approach, a user neods to specify weight hyperparameter reflecting a tradeoff between the two quantities. The challenge of this approach is to provide interpretable formulation that allows users to easily express their preferences through the weight hyperparameter. While many cost formulations are possible, adopt this approach for its clear interpretability and its ability to automatically adapt to cnvironments with vastly different reward scales, a crucial feal ure for a diverse benchmark.
We propose a soft-constraint approach where the weight hyperparameter has an interpretable meaning. The approach can be universally applied to any RL environment and expert policy. Let G. = E,,, [Σ> ra] be the average return of expert on test. task and T. be the corresponding average episode length. Our idea is quantify the cost of assistance as reduction in refurn, and the reduction amount proportional to the expert return G.. Specifically, we define the following reward function
G. Te
re(a)= = R(s., ar) - 11 = =e- a.
(1)
where R(st, 0z) is the environment reward obtained for the taken action a and E [0. is a user-specified hyperparameter. If the expert takes control in time steps, the total amount deducted from the environment. return is of T. Ge. This quantity is adaptive to the specific environment and export policy. Morcover, the penalty has a clear and interpretable meaning-2 it corresponds to a fraction of the expert's performance-whiee which facilitates intuitive tuning of a. For example, choosing a = 0.5 means that the expert makes decisions
4If hard constraint easy satisfy, hardly affects the optimization of the main objective. When that happens, one needs enforce stricter constraint (e.g. requiring the method to achieve higher performance or lower assistance cost).

---

Published in Transactions on Machine Learning Research (11/2025)
in every time step (making T = Te), the achieved return is approximately 50% of the expert return. In other words, the return loses 50% of its value due to requesting help.
In practice, users may specify a wide range of a values. Hence, it is crucial to evaluate with multiple values of a, simulating diverse scenarios. To summarize performances with multiple a values by a single mumber, we propose an area-under-the-curve (AUC) metric. As the name suggests, this metric estimates the AUC formed by the points {a,, G(a:))) where G(az) denotes the average return of the evaluated policy for a given aq (Fig. 3c). We present bootstrap procedure to compute the mean and error bars of this metric in Alg. 1.
Tracking progress toward solving YRC-0 problem. In ML benchmarks, an "oracle performance" " is typically provided to measure progress. However, establishing an oracle performance for a YRC-0 problem is non-trivial. One option is to use the expert's average return, but this performance is unattainable under the assumption that the novice is imperfect on the test tasks. Meanwhile, unlike many ML, problems, humans are not oracles in YRC-0. Because the mental states of both the novice and the expert are unobservable to a third-party agent, is difficult for a human to derive an optimal coordination policy for them. Our solution is to run an RL algorithm at test time to learn a near-optimal coordination policy. This algorithm operates on the MDP induced by the coordination policy and requires access to the expert policy Te and the test environment Evest. We refer to this approach as RLORACLE.
5 Policy validation by simulating test conditions
Policy validation by simulating test conditions. As mentioned, a major challenge in solving YRC-0 is policy validation, i.e., how to predict the test performance of poliey during training, without access to the expert and test distribution. Without validation approach, no learning method can be applied to our problem, as the policy selected for testing is ill-defined. To establish our validation approach, let us first define the oracle validator, which evaluates a policy under the exact test conditions (*())= = EVAL(P, Rne Re, Etest). Here, the function uses μ to coordinate nn and Te to perform test. tasks sampled from &vett and returns a score capturing the quality of μ. Our solution constructs a simulated volidator V(u)= = EVAL(H, Tny me: Erest) that mimics V", where any #es and Eustt are referred to as the simulated novice, expert, and test distribution, respectively.
How should one choose an. Re. and Erst so that V closely mimics V*? First of all, we set Eurtt Errain, because we have access to only this distribution dur- ing training. Given this choice, we want to construct a simulated expert 7e that performs well on this dis- tribution and a simulated novice 7n that performs not as effective on it. A natural choice for Ree is Wn, as our setting assumes that the novice performs well un- der Eurain To construct poor policy on Etrain the simulated novice, we learn a weakened novice म, by rumning the same algorithm used to train Ta but with limited supervision (i.e., limiting the number of inter- active episodes in RL or demonstrations in behavior cloning). Put all together, our simulated validator is (p) = EVAL(H, n. nn #. T., Euest = Emain). We provide a detailed quantitative analysis of this simulated validator's effectiveness in App. D.8. An alternative approach to simulating distribution shift would be to hold out a subset of Etrain to serve as a validation set. However, we choose to use the full Ensin for the primary novice to maximize its capabilities. Instead, we use "limited supervision" (redueed training time/data) to create x,. This serves as a practi- cal heuristic: we model the performance drop caused by distribution shift (in the real setting) using the performance gap caused by under-training (in the simulated setting).
Table 1: Uncertainty measures used to construct the co- ordination policy. Here, (21,'. (1444) is the logits computed by the novice in state s (novice is тæ during training and n. during testing). p SOPTMAX(2) and p denote the elements of sorted in descending order.
Method
Mcasure g(s)
MAXLOGIT
max; max; Pi
MAXPROB (Lewis, 1995)
MARGIN (Scheffer et al., 2001)
pf pa
NEGENTROPY (Settles, 2009)
Σ, Pr In In E, exp(2,)
NEGENERGY (Liu et al., 2020)
Deep SVDD (Ruff et al., 2018)
neural notwork
7

---

Published in Transactions on Machine Learning Research (11/2025)
Let G( be the mean episode return of policy π tasks sampled from distribution E. To achieve laithful simulation of the test. conditions, we want the ratio G( Errain) ( Eust) to be close to 1, i.e., the performance of the simulated expert on train tasks is close to that of the real novice on test tasks. 5 Empirically, we already obtain good results if this ratio is not greater than 5.
We select Deep SVDD (Ruff et al., 2018) as a representative baseline for OOD detection- approaches. Unlike generative approaches that model the full input distribution (which can be computationally expensive brittle in high-dimensional RL observation spaces), Deep SVDD focuses specifically on learning compact hypersphere boundary around feature embeddings of the training data. This makes it particularly suitable for detecting deviations in the novicc's internal latent representations.
Policy proposal approaches. We employ methods that compute a. decision function (s) (g(()) 2 r for each state s, where g is uncertainty measure and 7 is threshold. If f(s) 1, the novice follows the expert policy (at, e); otherwise, it follows its own policy (It- = =n). We consider various choices for the uncertainty measure (shown in Table 1), inspired by active learning and OOD detection methods. For each method, we generate a set of candidates thresholds T= (TI, T2, ...)...................) Let frt be the coordination policy induced by fg.-- We select the threshold r" = argmax,e7 ((pr) which maximizes the performance computed by the simulated validator V proposed in the previous section. In this approach, generating the candidate set TT presents a challenge, as the ranges of some uncertainty measures are not fixed. For example, NEGENTR outputs a value in [-In In [A], 0J, where A is the action space of an MDP. This range varies across environments, making standard grid search difficult to conduct. We propose an adaptive method to address this problem. Using the simulated novice Hn, we generate K task episodes, where the tasks are sampled from the training distribution. We gather a pool of states from these episodes and pass them through the uncertainty measure to generate a pool of scores. We then use the 10)-th percentile of these scores as the candidates, where n = 0, 1.. 10.
6 Experiments
YRC-Bench presents a unique opportunity to systematically analyze and compare the behavior of diverse methods across broad spectrum of tasks, yielding more robust insights than evaluations restricted to narrow domains. In this work, we conduct large-scale study of ten YRC-0 approaches across 19 environments drawn from YRC-Bench. Our experiments are compulationally intensive, requiring total of 1,347 NVIDIA A6000 GPU hours. The detailed runtime for each method and environment suite is provided in D.6.
6.1 Setup
We implement the policy proposal and validation framework described in $5, combined with the uncertainty measures listed in Table 1, resulting in six distinct learning methods. In addition, we evaluate four rule-based baselines: ALNAYSEXPERI, which always yields control to the expert; ALwAYSNOVICE, which always retains control; ALWAYSRANDOM0.3 which flips a fair coin each step to decide whether to yield control; and RANDOM, which queries the expert with fixed probability p E [0. The first three basclines do not require validator. For RANDOM, the optimal value of p is selected using the simulated validation procedure described in $5. Finally, we include the RLORACLE approach ($4.)) as reference for oracle performance.
We select Deep SVDD Ruff et al. (2018) as a representative baseline for OOD detection- based approaches. Unlike generative approaches that model the full input distribution (which can be computationally expensive or brittle in high-dimensional RL obscrvation spaces), Deep SVDD focuses specifically on lcarning a compact hypersphere boundary around feature embeddings of the training data. This makes it particularly suitable for detecting deviations in the novice's internal latent representations. We acknowledge that other uncertainty quantification methods exist, including ensemble-base based approaches (Deep Ensembles) and Bayesian methods (MC Dropout), but these require eit training multiple models or architectural modifications that may not be feasible when working with pre-trained novice policies. Deep SVDD represents a practical baseline that. can be applied post-hoc to existing policies.
Note that we are assuming knowledge of G(Nn,Eiaa)) This minimal and reasonable assumption, as without any knowledge about the test. conditions, predicting the test, performance would be impossible.
8

---

Published in Transactions on Machine Learning Research (11/2025)
For both deep SVDD and RLORACLE, we attempt various types of input features. We try every possible (non-empty) combination of the raw environment. observation (obs), the hidden features computed by the novice policy (to account for the novice's uncertainty) (hidden), and the novice's softmax action distribution (dist).
For each environment, we evaluate each method on 1,600 test tasks and run 1,000 bootstrap simulations to compute the mean and standard deviation of the evaluation metric (AUC; Fig. 3c).
6.2 Results
Owing to space limitations, we report the principal findings in this scction and provide additional analyses $D. We note that the value of these findings lies not in their quantity, but in their rigor, each is grounded in extensive experimental evidence.
So aarauted VaRSaDY
2
Finding 1: There is no single best method. Fig. presents an overview comparison of methods, showing the number of environments in which each achieves the highest mean AUC. Our analysis reveals a lack of consistency across methods. Even the most successful ones achieve top performance in only 3 out of 19 environments. This result underscores the importance of thorough empirical evaluation when selecting a solution approach for specific YRC problem. It also suggests the necessity of having a comprehensive benchmark like YRC-Bench, which supports quick evaluation diverse methods by providing a unified interface, standardized evaluation pipeline, and off- the-shelf baseline implementations.
aleryy oxteet
always roplly
always ardden 1.
cansoor
cne lopel raargir aort entropy
everpy
cot out hosbert
can widdeo cop
ond ete
and can ott
and histen dts and oku hutton dicc
Figure 4: Number environments where a method attains the highest mean AUC; solid bars denote use of our validation method.
Finding 2:
Our simulated validation approach is effective. Methods leveraging this approach collectively outperform their counterparts in 14 out of 19 enviromments. Furthermore, 3 out of the 4 most successful methods employ the simulated validator.
Finding 3: Existing Uncertainty-Base Methods Are Brittle, Often Failing to Outperform Random Baseline. Our analysis reveals stark indictment of the brittleness of current uncertainty-based methods when faced with distribution shifts. Sophisticated approaches like deep SVDD, which learn heuristics from the training distribution, often fail catastrophically, leading them to make confidently incorrect decisions about when to request help. This failure so significant that in multiple environments, these methods are oulperformed by a simple RANDOM baseline that requests help by flipping coin. Rather than being a flaw in the benchmark, this result underscores the severity of the OOD assessest challenge: methods designed for uncertainty cstimation can perform worse than random chance because they learn systematic biases that do not generalize. The RANDOM approach, in its simplicity, avoids these biases. This poor performance across the board, highlighted by the comparison to non-adaptive baseline, stands in stark contrast to the oracle performance shown in Fig. 5, confirming that there remains significant room for improvement, especially in challenging environments like Procgen and CLIPort.
Finding 4: Improving the policy proposer offers significantly more potential for performance gain than improving the validator
To offer more specific guidance for future development, we introduce systematic diagnostic method based on the proposer-validator decomposition of each algorithm. As a reminder, the policy proposer generates candidate coordination policies, while the validator evaluates these candidates to select the best one. Ideally, we want policy proposer that identifies the optimal policy as a candidate, and a validator that ranks it above all other policies. When an approach falls short, either the policy proposer, or the validator, or both are deficient.
9

---

Published in Transactions on Machine Learning Research (11/2025)
Best methed
ID +oratle validator
Z
+oracle proposer
minigrid
procgen
cliport
69-
68-
65 Di9 Doiieey LavaGap bossfight averyer
chaser
dinber cohirnn dodgeball
1e
Jamber
maze
nina purder assembing packing
JEaCk
separadig ples
Byranid
500
Figure Test performance of learning methods across environments, normalized by the performance of the best. RLORACLE method. For each enviromment, show three variants: the best performing method with simulated validation (i.e., excluding AnWAYSNOVICE, AlwaysExperT, and AIWAYSRANDOM0.5) the same method with an oracle validator (toracle validator). and the same method with oracle proposer, which employs a deep RT approach (toracle proposer). The gaps between the latter two variants and the original indicate potential performance gains that could be achieved by improving the replaced components. Error bars represent 2x standard deviation.
The proposer-validatoo decomposition enables us to identify components limiting the performance of an algo- rithm by replacing each with oracle counterpart and measuring the resulting improvement. A performance boost after replacement indicates that the replaced component is deficient and requires enhancement.
As shown in Fig. 5, replacing the simulated validator with an oracle validator yields minimal improvement in most environments. contrast, roplacing the policy proposer with its oracle counterpart produces substantial performance improvements 10 out 19 environments. This stands stark contrast. the minimal gains observed when replacing the simulated validator with its oracle counterpart (Fig. 5). While the overlapping error bars prevent a definitive conclusion that the validator is not bottlenock, our results strongly suggest that the policy proposer is the primary limiting component for the methods studied. Therefore, we conclude that improving the policy proposer offers a much larger potential for performance gain than improving the validator.
Our finding suggests that future research should focus on methods capable of exploring richer policy spaces. We discuss a notable exception to this trend the "plunder" environment in App. D.7.
Taken together, our results reveal fundamental limitation of current approaches: their search strategy does not identify effective candidate policies, it operates within overly restricted policy space. Our finding suggests that future research should focus methods capable of exploring richer policy spaces.
Drawing from the findings obtained from our experiments, we compose list of concrete recommendations for practitioners who want to develop or deploy solutions to YRC problems:
Explore diverse methods, good performance reported in one environment may not transfer another.
2. Do not overlook simple, computationally cheap baseline like RANDOM, which may yield surprisingly good performance.
Implement an oracle approach, like RLORACLE, estimate the remaining room for improvement. Remember that human-based policies often not oracles in this problem.
4. Replace the policy proposer or policy validator of the current method with an oracle counterpart to pinpoint the method's primary limiting component.
7 Conclusion & limitations
In this work, we formalize the learning to Yield and Request Control (YRC) problem, a critical challenge for AI agents operating in dynamic, safety-critical environments. Solving this problem is an important first step toward tackling more complex human-AT collaboration challenges. Our findings highlight significant.
10

---

Published in Transactions on Machine Learning Research (11/2025)
potential room for improvement, highlight opportunities for the research community to contribute in this area. Developing robust methods for this problem paves the way for safe, effective human-AT collaborative systems in the future.
While our work proposes a cost-effective and computationally efficient simulation of real-world scenarios, several limitations warrant consideration. First, the simulated experts YRC-Bench may not fully capture the variability and cognitive biases of human operators. While a full human study was beyond the scope of this foundational work, we consider it crucial direction for future research to validate our findings with real human participants. Second, although our benchmark incorporates distribution shifts across environments, real-world shifts may involve more complex, multimodal dynamics that are not yet modeled in existing simulations. Third, the cost model assumes fixed query costs, whereas practical deployments often face context-dependent or time-varying costs. By focusing on rule-based (always/randoom) bogit-based, OOD-based and oracle-RI, methods, we intentionally cover the most widespread coordination paradigns, extracting clean, genoralizable patterns that serve as performance basclines. Although we do not exhaust every possible algorithmie lamily (e.g., meta-learning, offline RL, human-in-Che-loop adaptation), our findings reveal fundamental proposer-validator bottlenecks that will directly inform and accelerate the design of these future approaches. In particular, evaluating stronger basclines from related fields, such as ensemble or Bayesian methods for more robust uncertainty quantification, is an important next step. Finally, our evaluation focuses on episodic tasks, leaving open questions about lifelong coordination in non-stationary settings. We choose PPO-trained and rule-based 'experts" to guarantee scalable, deterministic evaluation, sacrificing the idiosyneratic biases of real humans for consistency and speed. This strategic trade-off establishes a stable, low-variance baseline. Our modular framework can seamlessly swap in human-in-the-loop st stdees richer learned expert models in future work without altering its core mechanics.
In terms of methodology, we have so far explored only simple pallly-poopoiing and validation approaches. Threshold- based methods perform well with our proposed validator because they operate within restricted policy spaces, which effectively mitigate overfitting However, as we have shown, such a constrained policy space also limits performance. As one transitions toward methods that operate on richer policy spaces (e.g., deep RL methods), the risk of overfitting increases, and our proposed validator may not be sufficiently reliable to prevent it. We believe that addressing the current limitations of the simulation and developing a more faithful simulated validation approach are promising directions for future research.
References
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 1606.06666, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeicch Zaremba. Openai gym, 2016.
Sonia Chernova and Manuela Veloso. Intoractive policy learning through confidenco-base based autonomy. Journal of Artificial Intelligence Research, 34:1-25, 2009.
Maxime Chevalicr-Boisee
Bolun Dai, Mark Towers, Rodrigo VicentV, Lucas Willems, Salem Lahlon, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems 36, New Orleans, LA, USA, December 2023.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Mochine Learning Research, pp. 1048-2056. PMLR, 13 18 Jul 2020. URL https: pttps/////oocc.c..../ooo mlr cttsss/vv19///bbbbb
Michael K Cohen, Marcus Hutter, and Neel Nanda. Fully general online imitation learning. Journal of Machine Learning Research, 23(334):1-30, 2022.
Felipe Leno Da Silva and Anna Helena Reali Costa. survey on transfer learning for multiagent reinforcement learning systems. Journal of Artificial Intelligence Research, 64:645-700 0064-700,,60 2019.
11

---

Published in Transactions on Machine Learning Research (11/2025)
Mohamad H. Danesh and Alan Fern. Out-of-distribution dynamics detection: Rl-relevant benchmarks and results. arXiv preprint 0rXim:2107.449822 2021.
Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal misgeneral- ization in deep reinforcement. learning. In International Conference Machine Learning, pp. 12004-12019. PMLR, 2022.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
George Fragiadakis, Christos Diou, George Kousiouris, and Mara Nikolaidou. Evaluating human-ai collabora- tion: A review and methodological framework. arXiv preprint arXiv:2407.19000 2024.
Kaito Fujii and Hisashi Kashima. Budgeted stream-based active learning via adaptive submodular maximiza- tion. Advances in Neural Information Processing Systems, 29, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep Bayesian active learning with image data. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1183-1192. PMLR, 06-11 Aug 2017. URL https://proceedii wal.smml..pprerr petr.pres///70/gl1111
Tom Haider, Karsten Roscher, Felippe Schmoeller da Roza, and Stephan Günnemann. -itto--diitii detection for reinforcement. learning agents with probabilistic dynamics models. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pp. 851 1L-95,9 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770 778, 2016.
Steven CH Hoi, Rong Jin, Jianke Zhu, and Michael R Lyu. Batch mode active learning and its application to medical image classification. In Proceedings of the 23rd international conference on Machine learning, pp. 417-424, 2006.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amas Storkey. Meta-learning in neural networks: A survey. IEEE transactions pattern analysis and machine intelligence, 44(9):5149-516%) 2021.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pp. 4399 4410. PMLR, 2020.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jelf Braga, Dipam Chakraborly, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single- file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274): 18, 2022. URL http://jmmr.or orgll..oog//eeeee orgpp/pers////////////p v23/21-1342. html.
Shengyi Huang, Quentin Gallouédec, Florian Felten, Antonin Raffin, Rousslan Fernand Julien Dossa, Yanxiao Zhao, Ryan Sullivan, Viktor Makoviychuk, Denys Makoviichuk, Mohamad H. Danesh, et al. Open rl benchmark: Comprehensive tracked experiments for reinforcement learning. arXin preprint ffw:2402.03444 2024.
Kshitij Judah, Alan Paul Fern, Thomas Dicttcrich, and Prasad Tadopalli. Active Imitation learning: formal and practical reductions to iid learning. J. Mach. Learn. Res., 15(1):3925 3963, 2014.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. Ai safety gridworkds. arXiv preprinl orXiv: 171:::19109983 2017.
David D Lewis. A sequential algorithm for training text classifiers: Corrigendum and additional data. In Acm Sigir Forum, volume 29, pp. 13-19. ACM New York, NY, USA, 1995.
Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of- distribution generalization: A survey. arXiv preprint ar Xiv:2108.. 19624, 2021.
12

---

Published in Transactions on Machine Learning Research (11/2025)
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural informalion processing systems, 33:21464-21475, 2020.
Owen Lockwood and Mei Si. A review of uncertainty for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence and Inteructive Digital Entertainment, volume 18, pp. 155 162, 2022.
Linas Nasvytis, Kai Sandbrink, Jakob Foerster, Tim Franzmeyer, and Christian Schroeder de Witt. Rethinking out-of-distribution detection for reinforcement learning: Advancing methods for evaluation and detection. arXiv preprint arXio:2244.0700 2024.
Manisha Natarajan, Chunyue Xue, Sanne van Waveren, Karen Feigh, and Matthew Gombolay. Mixed- initiative human-robot teaming under suboptimality with online bayesian adaptation. arXiv preprint 2rX 16178 2024.
Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective encouraigg imitation learning. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan ), Proceedings of the 2019 Conference on Empirical Melhods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 684 695, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 111111/111Dv1--- URL https://aclanthott
org/D19-10633
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with languagebaaset assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), June 2019.
Khanh Nguyen, Yonatan Bisk, and Hal Daumé III. Learning when and what to ask: Ahierarchical reinforcement learning framework. arXiv preprint ar arXiv:21 10. 10.08258, 2021.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2017-20177 2025, 2022.
Abhishek Paudel. Learning for robot docision making under distribution shift: A survey. arXiv preprint arXiv:2203.075:0 2022.
M Pflanzer, Z Traylor, JB Lyons, V Dubljević, and CS Nam. Ethics in human ai teaming: principles and perspectives. ai and ethics, 3 (3), 917-919799 935, 2023.
Wilbert Pumacay, Ishika Singh, Jiafei Duan, Ranjay Krishna, Jesse Thomason, and Dieter Fox. The colosseum:
A benchmark for evaluating generalization for robotic manipulation. arXiv preprint "XXi:2242.08191, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adilya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PmLR, 2021.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforccment learning. Xixiv preprinl arxi::1802. 01744, 2018
Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In Proceedings of the Conference on Robot Learning (CoRL), 2023.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. In ICML, volume 1, pp. Citeseer, 2001.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Lukas Ruff, Robert Vandermenlen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2393-4402. PMLR, 10-15 Jul 2018. URL https::/pproceeddngs.n. press/v80/ruffi html.
Dorsa Sadigh, Shankar Sastry, Sanjit A Seshia, and Anca D Dragan. Planning for autonomous cars that leverage effects on human actions. In Robotics: Science and systems, volume 2, pp. 1-9. Ann Arbor, MI, USA, 2016.
Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International symposium intelligent data analysis, pp. 309 318. Springer, 2001.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint orXio:1707.0634" 2017.
Burr Settles. Active learning literature survey. Computer Sciences Technical Report 1648, University Wisconsin -iscoosin-Madioon, 2009. URL http://axon. cs. btw.aa//mmattiiie//ssesee//////e activelearning. pdf.
Ben Shneiderman, Human-centered Al. Oxford University Press, 2022.
Mohit Shridhar, Lucas Mamuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference Robot Learning (CoRL., 2021.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialle navigation. In Conference on Robot Learning, pp. 394 406. PMLR, 2020.
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.10033 2024.
Vanshika Vats, Marzia Binta Nizam, Minghao Liu, Ziyuan Wang, Richard Ho, Mohnish Sai Prasad, Vincent Titterton, Sai Venkat Malreddy, Riya Aggarwal, Yanwen Xu, et al. A survey on human-ai teaming with large pre-trained models. arXio preprint arXiv:2403. 0...24933,4491,,, 2024.
Tongshuang WU, Michael Terry, and Carrie Jun Cai. Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI conference human faclors in compuling systems, pp. 12, 2022.
Annie Xie, Fahim Tajwar, Archit Sharma, and Chelsea Finn. When to ask for help: Proactive interventions in autonomous reinforcement learning. Advances in Newral Information Processing Systems, 35:16918 16930, 2022.
Wang Xue-Song, Wang Automatica Sinica, 4999):1813-1835, 2023.
and Cheng Yu-Hu. Safe reinforcement learning: A survey. Acta
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of detection: A survey. International Journal of Computer Vision, 132(12):5635 5662, 2024.
Zhecheng Yuan, Sizhe Yang, Pu Hua, Can Chang, Kaizhe Hu, and Huazhe Xu. Rl-vigen: A reinforcement learning benchmark for visual generalization. Advances in Neural Information Processing Systems, 36: 6720-676 6747, 2023.
Qinhong Zhou, Sunli Chen, Yisong Wang, Haozhe Xu, Weihua Du, Hongxin Zhang, Yilun Du, Joshua B Tenenbaum, and Chuang Gan. Hazard challenge: Embodied decision making in dynamically changing environments. Xixiv preprint arXic:2401. 12975, 2024.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Appendix
Table of Contents
A YRC-Bench
16
A.1
Coordination Environment Wrapper
16
A.2 Training Novice and Expert Policies
16
A.3
Training Coordination Policy Eyaluation..
17
A.4
18
Policy Feature Extraction
C Environments
19
C.1 MiniGrid Enviromments
................................
19
C.2 Procgen
................................
19
C.3 CLIPort
19
D Detailed Results
20
D.1 Performance of RTOORACEE Methods
................................
20
D.2 Near-Optimal Coordination Achievements
................................
D.3 Comparison of Logit-based and OOD detection based Methods ................................
22
D.4 Best features for RLORACLE
22
D.5 Normalized Dovice-to-Fxpers Score Ratios
23
D.6 Computation Cost and Time
23
D.7 Analysis of Oracle Proposer Performance "plunder"
24
D.8 Analysis of the Simulated Validator
25
15

---

Published in Transactions on Machine Learning Research (11/2025)
Pseudocode Coordination environment interface
class CoordEnv(eym.. Env): Env):
def _.init__selff, config, base_env, novice, expert):
def rssef(::lf): # unlike gym3, actually reset to an initial state of a new task def step(self, action):
# like gym3, automatically reset upon end episode
Figure Interface the coordination policy's MDP class.
Pscudocode 2. Training coordination policy
1) Parse command-line flags
args maags
# 2) Configuration can be specified using a YAML file (args.config) or by flags (args) config = config_utils. (argllls. config, flags=args)
3) Make training, validation, and test environments
envs
make(config)
# 4) Initialize coordination policy
goord_policy
core. worre.polcc..oom make envs)
5) Create evaluator
evaluator YRC. core.
wet.:core..lllaee. vaauuatorcconfigg)
# 6) Initializ algorithm
ahgorithm = core. algorithh i. envs)
7) Train coordination policy
algori thm. train(coord_pollccy, envs, evaluator)
Figure Training a coordination policy with YRC-Bench.
A YRC-Bench
A.1 Coordination Environment Wrapper
To standardize coordination policy training and evaluation, we introduce the CoordEnv wrapper. This class converts any Gym-compatible environment (Brockman et al., 2016; Towers et al., 2024) into an MDP for the coordination policy that preserves the original state space but replaces the action space with two choices (n,e), representing the coordination policy's decisions of requesting control (novice acts) and yielding control (expert acts). At each timestep, the wrapper resolves the coordination policy p's decision into action in the base environment's 's action space: (s)). Subsequently, the next state 8,41 is generated following the base environment's transition dynamics: 8141 a))
Fig. describes the interface of CoordEnv. The class expects a base environment that follows the gym3 interface except that calling reset() actually resets to an initial state of a task (the reset method of an gym3 environment has no effect). This feature is employed during evaluation to ensure that all evaluation runs conducted with the same set of tasks.
Fig. 7 shows simple steps to train coordination policy using our codebase. Each step is highly customizable by extending the codebase's core classes.
A.2 Training Novice and Expert Policies
Each environment requires three policies for training and evaluation: expert Te, novice The weakened novice
16

---

Published in Transactions on Machine Learning Research (11/2025)
Expert construction. For MiniGrid and Procgen, we train ne using PPO Etest until convergence (Huang et al., 20r4)For CI.Poort, we use the rule-based oracles provided by the benchmark.
The novice is trained exclusively on Evrain MiniGrid and Procgen novices are learned by running PPO on Etrain until convergence CLIPort novices are checkpoints trained with 100 demonstrations.
Novice construction.
Weakened Novice Policy Training. We create " by deliberately limiting training exposure. For MiniGrid and Procgen, halve PPO training epochs while maintaining Erain exposure. CLIPort's n, uses checkpoints trained on only 10 demonstrations, reflecting partial task mastery. This mimics test-time performance degradation while preserving training distribution familiarity.
A.3 Training Coordination Policy
A.3.1 Logit-Based Methods
Let (21, #2, = be the logits output by the novice, and p= Softma((z) = (P1.P2.'.. PPAA) is the softmax distribution derived from 2. A logit-based method outputs a score u that captures the degree of uncertainty of the novice. Whenever the score falls below a pre-specified threshold, the novice yields control to the expert.
The score for each method defined as follows:
MAXLOGIT: u max; 25
MAXPROB: max, Ps
MARGIN: u = max; Ps maxj Pj where 1 = argmax, Pr (the difference between the highest and second highest softmax probabilities)
NEGENTROPY: u A Pt Inp;
NEGENERGY: = In
Σ exp(z/r). Following (Liu et al., 2020), we set the temperature
in all experiments.
Threshold selection. To determine the threshold, we conduct a grid search over a set of candidate thresholds, and select the one that maximizes validation performance. To generate these candidates, we rollout म,, for 64 episodes. In each episode, the policy executes task sampled from Evain This generates a pool of uncertainty scores, which each corresponds to the decision of the novice in state encountered in episode. We sort the scores and use the k-th percentiles as threshold candidates with k 0, 10, 20, 100. This method is data-driven and adaptive to the score range, which varies dramatically among methods.
A.3.2 Deep SVDD
Our implementation Deep SVDD is based on PyOD. All hyperparameters for Deep SVDD are set to their default values in PyOD. Similar to Logit-based methods, Deep SVDD computes an uncertainty score; the novice yields control if the score is below a threshold. We first generate executions of 64 Errai tasks using म, The generated data is used to both train the OOD-detection model and to determine the optimal threshold. The threshold is also chosen following similar procedure as that of logit-based method: generate pool of uncertainty scores and use the percentiles as candidates.
A.3.3 RLOracle
RLORACLE implements Proximal Policy Optimization (PPO) (Schulman et al., 2017). Our implementation largely follows CleanRL (Huang et al., 2022).6 To obtain our results, we run this algorithm with the coordination environment wrapper corresponding to Etest-
The underlying policy is an IMPALA model (Espeholt al., 2018). Depending on the chosen configuration, environment observation, hidden representation extracted from the
the imput features may include novice, or the softmax output distribution of the novice.
hhttps://g thub. con//wwyyjjn/cleann11 http:///eeeeeeillaaaaaeeeeaaaa/a
17

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm 1 Booostrap procedure to compute AUC metric. ArealnderCurve compules the area under the curve formed by the imput points. We use N = 1000, K = 6, M = 1600, m = 256 in our experiments.
x and G.,₃ is the return of the evaluated policy in the j-th Input: Data points {(a,, (G.,, = where episode, during which a = og.
Output: Mean estimation and its standard deviation
Initialize E for N simulations do Tnitialize D 0
4:
Dr 55
6:
for in 1.1.... 1...... K do
7:
Draw an m-element sample S; from {Gr,)== Compute G. = (Sa
8:
9:
D D ((a., E +- U (AreaUnder(uurve(D))
10:
return me) std(E)
We refer readers to our GitHub repository for detailed training and model hyperparameters.
A.4
Evaluation
Alg. presents our bootstrap procedure for computing the evaluation metric for each policy.
B Policy Feature Extraction
In MiniGrid, both the novice n. and weakened novice Th are trained using the GitHub repository: https: //github.com/ccsw//
files. To extract the hidden features (hidden) and the softmax action distribution (dist), we leverage the model's intermediate layers. For the former, we process the raw image observations through convolutional layers and text inputs through an embedding module, fusing these modalities into a unified hidden representation. Then, for the latter, we apply the actor network to this hidden representation compute unnormalized action scores (logits). Critically, this mirrors the forward pass of the original model but excludes the final step of constructing probability distribution (via the Categorical class), allowing direct access to the pre-softmax logits. This approach ensures alignment with the model's internal decision-making process while enabling targeted analysis of policy behavior.
For Procgen, we employ the same framework. Hidden features (hidden) are extracted via the ImpalaModel embedder (Espeholt et al., 2018), which processes high-dimensional visual observations through its residual convolutional architocture. Then these features are mapped through the policy network to produce unnor- malized action scores. While the model's forward pass applies log-sof normalization and constructs a categorical distribution for policy gradient updates, our analysis directly utilizes the raw outputs from the final policy layer. This preserves the model's original action preference rankings while bypassing probability normalization, a critical design choice that maintains numerical fidelity with the policy's internal decision logic while enabling direct comparison of action selection mechanisms across different, policy versions.
For CLIPort, we adapt the framework to accommodate its dual-stream architecture for robotic manipulation. Hidden features are derived through separate attention (pick) and transport (place) networks that process RGB-D observations and language goals through coordinated ResNet and CLIP-linguistic fusion pathways He et al. (2016); Radford et al. (2021). The attention stream first computes pixel-wise confidence maps for object selection, while the transport stream subsequently predicts placement locations and orientations conditioned on the chosen pick point. Unnormalized action scores emerge as spatial heatmaps encoding both positional and rotational preferences across the workspace. Though the operational pipeline constructs categorical distributions during training through spatial softmax normalization, our analysis directly utilizes the pre-normalization confidence values from both streams. This preserves the geometric relationships in the model's pick-and-place reasoning while maintaining fidelity the original visuolinguistic feature represcntations, crucial for interpreting the policy's physical interaction decisions in structured action spaces.
18

---

Published in Transactions on Machine Learning Research (11/2025)
C
Environments
YRC-Bench is built on top of three open-source benchmarks: MiniGrid (Chevalier-oiivert et al., 2023).
CLIPort (Shridhar et al., 2021), ProcgenAISC (commnt 7821f2c) (Di Langosco et al., 2022).
C.1 MiniGrid Environments
We use the following enviromments:
MiniGrid-Distthift- 1-v0 for training, and 2-v0 for testing:
MiniGrid-Doorkey: 5x5-vØ for training, 8x8-v0 for testing;
MiniGrid-LavaGap: S5-vØ for training, S7-v0 for lesting
All environments use partially observable grids with discrete actions. The test environments feature larger state spaces and more complex trajectory solutions.
C.2 Procgen
The suite includes 11 distinct platformer games with pixel-based observations and discrete actions:
bossfight: combat-focused game with escalating enemies
caveflyer: navigation through procedural caverns
chaser: avoidance of pursuing enemies
climber: vertical ascension challenge
coinrun: collcction-based platformer
dodgebal: projectile avoidance game
heist: stealth- based item retrieval
jumper: precision jumping challenges
maze: complex spatial navigation
ninja: timing-based obstacle course
plunder: resource gathering under threat
We use the easy distribution for training and the hard distribution for testing. The hard distribution
introduces stochastic elements and more complex terrains.
C.3 CLIPort
We experiment with five tasks:
Assembling- sequential object placement kits
Backing- pbject. pairing and containerization
Put Put-Block-in--owl. precise object-in-container placement
Stack-BlockPPyrmmid--ee vertical structure assembly Separating-Piles: object sorting and segregation
We use the seen split for training and the unseen split for testing. The unseen split (testing) introduces novel object geometries and color combinations not encountered in the seen split. The tasks require 6-DOF control with a continuous action space and spatial reasoning over maxel-based observations.
19

---

Published in Transactions on Machine Learning Research (11/2025)
obs-based
non-obs
Ervironment: DisShitt
Envionment: DoseKey
Envirenment: Lavafap
Environment:
Envirormen:: cavellyer
Performar
02-
Frvinorewet:
Enviroment: climber
Enddronment: cointun
Bruinonments dadgeball
Ervironmerts
Performanc
Environment: jumper
Ervironment: maze
Environment nina
Envirenment plundor
Environment: assembiing Mts sog
Performatee
122
Enviroement: parkig-bows-ppiis
Invironment: put-blec-ir-boll
E Environment: wtw.k-blcck--ao........ warr--lock-pyyannde pyramid--ee
Envionment separatingppiles
Performance
022
ub. hidderh ChC.
Nidden
150de RID0An CiAt
Sidee
de Hibden
Hiddln 31x
4.1
Figure 8: Per-environment performance of RL.ORACLE variants. Using observations as input features yields larger performance boost in more complex environments like Procgen and CI.roort.
D Detailed Results
D.1 Performance of RLOracle Methods
We further analyze the performance of individual RLORACLE algorithms across different environments, shown in Fig. 8. This detailed breakdown reveals that the advantage of raw observation-based policies is more pronounced in the Procgen and CLIPort environments, while it is less evident in the MiniGrid suitc. This discrepancy can be attributed to the nature of the environments: Procgen and CLIPort feature visually rich, high-dimensional observation spaces, where direct access to raw observations provides a clear advantage in learning nuanced coordination behaviors. In contrast, MiniGrid offers low-dimensional, symbolic representations, where the distinction between raw observations and the novice's internal features is less significant. In such structured environments, the novice policy's internal representations already capture most of the relevant task information, diminishing the benefit of using observations.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Figure 9: Comprehensive performance comparison across all learning methods and environments. RLORACLE represents the RL-computed upper bounds. logit and OOD methods approach the upper bound in several environments, mostly MiniGrid ones. Gray backgrounds denotes those environments.
D.2 Near- OptimOl Coordination Achievements
Fig. 9 illustrates the overall performance of cach algorithm and input fcature type across all environments studied this paper. It reveals an interesting pattern: logit-based and OOD detection-basee coordination policies achieve near-skyline performance in environments. We analyze these representative success cases:
DoorKey (MiniGrid): The 8 x 8 grid environment cxhibits deterministic dynamics but requires precise multi-step sequencing (find key, then unlock door, then navigate to goal). The MAXLOGIT policy matches skyline performance matches skyline performance by interfering the novice's potentially flawed decision-making, preventing costly mistakes and ensuring efficient completion of the task.
LavaGap (MiniGrid): This environment's lethal consequences (falling into lava) create clean separation between high-confidence navigation actions and uncertainty "cliff edges." " The OOD-based method with hidden-dist features and Margin logit policy are the closest skyline performance.
Climber (Procgen): Despite procedural generation, the logit-based methods are statistically the same as skyline methods. The policy successfully distinguishes between by novice) and truly novel gap configurations (referred to expert), despite being trained solely on the easy distribution.
but-scen obstacles (handled
Our analysis also reveals substantial performance gaps between RLORACLE and other methods. Notably, across all CLIPort manipulation tasks, no method approaches even the worst perss-performig RLORACLE. For example, in the mookinng-bxes---e task, the lowest-performing R.OORACLE variant. (using only the novice's
21

---

Published in Transactions on Machine Learning Research (11/2025)
Lagit cavellyer
QDD
Doerkey
LavaGap
bosstight
colnrun
codgeball
Perfo Performance
jumper
plunder
kRs-src
boses-parrs
In-how
mid
piles
Performance
Figure 10: Comparison logit-based and OOD detection methods. Error bars show the standard deviation across each method's variants.
action probability distribution as input) achieves a performance of 0.83, while the best non- non-RLORACLE methods (logit-based approaches) reach only 0.73, representing a 13.7% relative performance gap. Other CLIPort tasks exhibit even wider disparities, with RLORACLE outperforming alternatives by at least 30.7% on besembling-kits--eee 35.1% on put-block-in-bowl 40.3% on stack-block poreck-prrmid-seee and 20.9% on separating-pills. These substantial gaps highlight fundamental limitations in current coordination strategies for high-dimenssonal manipulation tasks, underscoring the urgent need for improved policy architectures that better leverage both environmental observations and novice uncertainty signals.
D.3 Comparison of logit-based and OOD detection-based Methods
Our experiments reveal an interesting insight in comparing logit-based methods with the Deep SVDD OOD detection approach, quantified in Fig. 10. Overall, in 1 out of 19 evaluated environments, logit-based methods oulperform Deep SVDD. In 2 environments the OOD detection-based method performs better. And in the remaining cases, they tie.
This suggests that practitioners may prefer computationally lightweight logit-based coordination unless operating in domains with known visual-semantic mismatch between observation space and task requirements. Based on our results, we suggest practitioners reconsider the prevailing assumption that complex OOD detection universally preferable for safety-critical coordination (Yang al., 2024). We demonstrate that simpler approaches often suffice when distribution shifts primarily affect agent behavior rather than environmental appearance.
D.4 Best features for RLOracle
While being an oracle in our setting, ILORACLE is a viable approach in life-long learning setting, where the novice contin- uously adapts to test conditions. We investigate the best recipe for this approach to provide helpful insights for researchers who want to tackle this setting.
ebs bused
nor obs
Our experiments reveal that including raw environment observa- tions as input the coordination policy consistently improves performance compared to using only its hidden representations or its logit outputs. This trend presents in 15 out of 19 envi- ronments (Fig. 11), suggesting that the novice does not acquire helpful, easily extractable uncertainty information if trained only to perform tasks autonomously.
hidden
hidden
abs
d8c
ded
Nodes
Figure 11: Number of environments in which each variant of RLORACLE achieve the highest AUC mean. Variants that take raw environ- ment observations as input yield superior per- formance.
Our results also highlight a relationship between environment complexity and observation-space utility. While raw obser- vations generally provide richer learning signals, their value
22

---

Published in Transactions on Machine Learning Research (11/2025)
diminishes in structured environments with comprehensive fea- ture representations. For instance, in Minigrid environments, the difference between using raw observations and structured feature representations is negligible. However, in more visually complex environments like Procgen or CLIPort (e.g., CaveFlyer or sta raw observations provide crucial information that significantly enhances performance (see App. D.1 for details). We thus suggest practitioners prefer observation-coondiioned coordination policies unless observations are complex to model and hidden representations are sufficiently rich.
D.5 Normalized Novice-to- Exppet Score Ratios
To quantify the gap between novice and expert policies on unseen test tasks, &tests we compute for each environment i the normalized score ratio
W, E,
r:
8r; 1
(W.. W.
+
E,
(2)
where Eg F F. and W; ow: are the expert's and novice's mean returns and standard deviations, respectively. Fig. 12 (left) shows with 2 x ôr; error bars for all 19 environments across MiniGrid, Procgen, and CLIPort.
We chose these suites to capture distinct evaluation challenges:
MiniGrid: a simple, highly customizable gridworld for quick proof-of-concept demonstrations on abstract state representations, highlighting decision-making challenges that generalist LLMs often face.
Procgen: morizon procedurally generated visual tasks that stress robust visual control and exploration.
vision-language manipulation benchmarks requiring integration of visual perception with high-level language instructions.
CLIPort:
Across individual tasks, novice policies achieve between approximately 12% and 43% of expert returns MiniGrid (mean 0.23), 17% 55% in Procgen (mean ss 0.32), and 24% 74% in CLIPort (mean 0.45). This systematic shortfall highlights the difficulty of generalizing to novel test environments.
We further aggregate these ratios at the suite level by
örg v Eiesorr S
rs
Σ SE
(3)
S
where S indexes environments in each suite. Fig. 12 (right) shows the average ratio rs with 2 x ōrg error bars. The consistent deficit across all suites, never exceeding 50% of expert performance, underscores that even our best-trained novices fall significantly short of expert performance when generalizing beyond training.
These patterns serve two purposes: (1) they illustrate that different environment familics pose unique generalization challenges, and (2) they motivate our coordination framework, which adaptively requests expert guidance most aggressively in those environments where r; is both lowest and most variable, thereby allocating expert queries where they yield the highest marginal benefit.
D.6 Computation Cost and Time
All experiments were executed on single NVIDIA A6000 GPU with 48 GB VRAM and 100 GB of host. memory. To highlight the significant computational cost, Table 2 summarizes the total wall-clock times for the training and evaluation phases, as well as their combined cost.
Table reveals that we expended a staggering total of over 1347 GPU-days of continuous run time on a single A6000 card. This corresponds to nearly 3.7 years of dedicated GPU usage, underscoring the high resource demands of largo-scale experiments under distribution shift. In particular, the Procgen suite alone accounted for almost 82% of total compute during training, reflecting the complexity and variability of those environments. Even the simplest MiniGrid tasks required several GPU-days achieve robust. evaluation.
23

---

Published in Transactions on Machine Learning Research (11/2025)
MiriGrid
Procgen
CLiPort
Dewttey
Lava
begulghtt
cawelye
doobat
sndg
Phonder
lmvenstig
pliking
Maak
orrand leperising
Mesordd
Poogge
CUPot
Figure 12: Left: Per-environment novice-to-expert score ratios r; with error bars x or; hatched by suite (no hatch: MiniGrid; dotted: Procgen; slashed: CLIPort). Right: Suite-averaged ratio with aggregated error bars 2 örs- Novices consistently achieve less than half of expert returns across all domains, and exhibit higher instability (wider error bars) stochastic tasks.
Table Wall-clock time on NVIDIA A6000 (48 GB) with 100 GB RAM. Times rounded to nearest hour for readability.
Category
Training
Evaluation
Total
By environment
MiniGrid
d-14 h
d-12 h
4 d-2
Procgen
1105 d-11 h
14 d-19 h
1120 d-6 h
CT.IPort
201 d-11 h
21 d-16 h
223 d-2
By algorittom
AtwaYs-based
3 d-0 h
6 d-9 h
9 d-9 h
TIRESHOLL-based
d-21 h
12 d-1 h
97 d-22
122 d-16 h
5 d-16
128 d-8 h
O.OORACLE
1082 d-0 h
d-17h
1087 d-17
Overall
1310 d-12 h
36 d-23 h
1347 d-10 h
By algorithm, the skyline RLORACLE dominated resource consumption (< 1087 GPU-days), while our OOD-based method consumed an additional 128 GPU-days. The relatively lower cost of the ALwAYs-based and TURESSOOLD-based methods (9 97 GPU-days respectively) highlights that naive baselines are cheaper but far less adaptive.
These figures make clear that replicating and extending our suite of experiments demands significant computational investments, reinforcing the importance of efficient coordination policies that can reduce expert. queries without incurring prohibitive compute costs.
D.7 Analysis of Oracle Proposer Performance in "plunder"
We note the interesting case of the plunder environment, where the oracle proposer slightly underperformed the best heuristic method (Fig. 5). We hypothesize this is due to the high variance inherent in both the procodurally generated environment and the deep RL training process for the RLORACLE proposer. For certain challenging random seeds, the complex oracle proposer may have converged to slightly suboptimal robust..
policy, whereas simpler heuristic method
24

---

Published in Transactions on Machine Learning Research (11/2025)
D.8 Analysis of the Simulated Validator
We provide additional details and analysis on our simulated validator setup to assess its reliability and sensitivity.
D.8.1 Holistic Evidence for Validator Effectiveness
To assess the effectiveness of our simulated validator, one could ideally analyze the direct quantitative correlation between its predicted policy rankings and the true rankings from the oracle validator. However, more practical and holistic mcasure of its real-world utility is demonstrated by its impact on final model performance across our entire suite of experiments.
Our primary evidence for the validator's effectiveness is presented in Finding and detailed Fig. 4. These results show that methods leveraging our simulated validation approach (denoted by solid bars Fig. 4) collectively outperform their counterparts in 14 out of 19 environments. Furthermore, 3 of the 4 most successful methods overall ones that employ the simulated validator. This demonstrates that simulated validator is an effective and reliable tool for hyperparameter selection in the challenging YRC-0 setting. The consistent performance improvement gained by using it provides strong empirical evidence of its ability to select policies that generalize well to the true test conditions, which its ultimate purpose.
D.8.2 Rationale for the Degree of Novice Weakening
Our simulated validator relies "weakened novice" (πх)- Our choice to weaken the novice by training it with 50% of the data/epochs of the full novice (TT) was deliberate methodological decision intended to strike crucial balance.
A more severely weakened novice (e.g., 25% of original training) might perform so ineffectively that fails to represent the subtle, near-distribution failures of the true novice, leading the validator to learn an overly pessimistic and simple strategy. Conversely, a less weakened novice (e.g., 75% of original training) would be too competent, providing an insufficient performance gap against the simulated expert (the full novice, u.) for the validator to learn meaningful distinction between reliable and unreliable actions.
Therefore, the 50% mark was chosen to create a challenging yet representative simulation where the weakened novice is prone to failure but still retains enough competence to model realistic decision-making challenges under distribution shift.
25


Reviews:
[{"review_id": "i1FVXwS9WL", "reviewer": "Reviewer_VUBK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers an MDP setting where a \"novice\" policy can perform well on tasks that were in the training distribution, but performs poorly on OOD tasks. However, there is an \"expert\" policy which performs well on OOD tasks. The goal of the paper is to learn a coordination policy, which will decide when to use the novice policy and when to defer to the expert policy (trying to minimize the use of the expert, as this is assumed to be costly). A key distinction from prior related work (on the YRC problem) is that in this paper, at train time there is no expert available (YRC-0).\n\nIn particular, the authors model the cost of using the expert as effectively a fraction of the expert's return (i.e. as they say, one might treat rewards gained under expert behavior as worth 50% of their actual reward). This is a reasonable, but very specific, way to model the issue of cost.\n\nThe authors present a new benchmark, YRC-Bench, which adds expert policies to a bunch of reasonable baseline MDPs. To model \"perfect\" performance, they can't exactly solve the problem, but can run costly RL training to get something like an optimal policy.\n\nThe authors frame their problem in terms of choosing a policy proposer (which creates a set of policies) and a policy validator (which picks one from the set). Their observation is that getting a policy validator is much more difficult given the lack of access to experts at train time. A clever but slightly ad-hoc solution to this is to treat the \"novice\" policy as an expert on the training tasks, create a deliberately degraded policy, and then train the coordinator to coordinate between those, hoping that OOD it will still generalize between the true expert and the true novice.\n\nExperimental results are positive in the sense that the benchmark and methodology seems to be reliable, but algorithmically there is much room for improvement (which is not necessarily a bad thing in a paper that mainly wants to present a new task and benchmark).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The problem setting is reasonable and connected well to previous work. Everything is explained clearly. Experiments are solid and support the claims in the paper. The authors are very honest about their limitations.", "audience_yn": "Yes", "audience_explanation": "The YRC task, and this variation of it, and related work in imitation learning etc., are of longstanding interest to the ML community; people will likely find the new benchmark useful.", "requested_changes": "The paper \"Fully General Online Imitation Learning\" (Cohen et al., JMLR 2022) takes a very different perspective but deals with a related problem setting. The authors may want to (or may reasonably decide not to) cite it.", "broader_impact_concerns": "There are no broader impact concerns; this is a paper fairly detached from real-world deployment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hQcABPoOnO", "reviewer": "Reviewer_8Tux", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The primary contribution is the introduction of YRC-0, a new, challenging variant of the YRC problem. In YRC-0, an agent must learn a coordination strategy for new environments/domains in an unsupervised setting i,e. without interacting with the domain expert during the training phase. This setting is motivated by the high cost of expert supervision and the need for robust, generalizable solutions.\n\nTo further enable work on YRC-0, the authors also introduce a new open source benchmark called YRC-Bench.\n\nThe authors provide a baseline implementation while conducting large scale evaluations to kick start work on YRC-0 and YRC-Bench.\n\nStrengths:\n- The problem description is relevant and timely for real world AI deployment. The constraints and framework of YRC-0 make sense. Expert abilities may improve, new experts may come up, while an agent is already deployed and unaware of those experts. Minimizing the cost of calling experts is also a sound real world assumption.\n- The proposed benchmark YRC-Bench appears to have all the elements that could drive its adoption, i.e. 3 environments, gym-like API, simulated experts, baseline implementation and an evaluation pipeline.\n- The baseline implementation and analysis from the authors seems to be very meticulous, and provides hints for future work, for e.g. further work needed on proposers over validators.\n- A bit of a meta point, but one of the strengths of the work is that it itself lists down a lot of limitations, assumptions, and directions for future work :) \n\nWeaknesses\n- To be fair, the authors list most of the weaknesses below in Section 7.\n- Proposer-validator framing. While this framework does make sense in the paper, the authors mention their bottleneck findings could be useful for future work on this problem involving other methods like offline RL or meta leaning. This point is not clear to me. How would this framing fit with these other methods? Not all potential solutions for YRC-0 can fit this framework. For e.g. either active RL  or meta leaning methods. In meta-learning, there is no clear distinction between proposing and validating.\n- Usage of simulated experts and validators. It is not clear how the results on this benchmark will transfer over to real world AI systems with experts. \n- Lack of human evaluation. Lack of human evaluation is understandable from the point of view of lack of resources, but any limited human eval would lend more weight to the results in the paper. Especially the results which the authors claim can guide further work in this direction.\n- The paper motivates the problem as one of human and AI coordination, but the nature of the coordination is simplified to choosing one of two policies. A more realistic form of coordination could be future work perhaps.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most of the claims made by the paper seem to be convincing. Here, I list some of the claims that need a bit more clarity:\n- The authors claim that YRC-Benchmark is open-source, but I don’t see any mention of the link to the open source implementation. Perhaps the authors are waiting to do so after submission, in order to support the anonymization of the review?\n- The claim made in the paper about proposer-validator bottlenecks being useful for future research of methods like meta-learning, offline RL. This might be true only for the narrow set of proposers and validators considered by this study. The proposers considered by this paper are fairly simple heuristics, and it is completely plausible that different / stronger proposers would reveal new challenges for validators.\n- Going on about the same claim above, looking at Figure 5, it seems to be hard to draw the conclusion about validators not being a bottleneck. A lot of the error bars overlap, so it’s hard to draw any conclusion either way. But yes, the point about replacement oracle validators being more impactful, is completely valid.\n- The claim that simulated validators are an effective way to tune hyperparameters. The authors further claim that replacing simulated validators with an oracle doesn’t change results significantly. While the simulated validator setup is clever, it would be informative to know more details here. Do the simulated validators correlate with true test performance? How does the setup depend on how much the novice is weakened? Details about how the novice is weakened would be useful to know.", "audience_yn": "Yes", "audience_explanation": "Yes, it is clear that the YRC-0 problem defined by the authors is challenging, novel, interesting and worth studying. The open-source (to be ?)  benchmark proposed is a great start towards more work in this direction, and hopefully TMLR’s audience can contribute towards better candidate submission and maybe removing some of the assumptions in the benchmark too.\n\nThe ablations in the paper, even though limited by some assumptions, are very thorough and can provide ideas about future work.", "requested_changes": "More details about the simulated validator setup would be great to have in the paper.\n- Some analysis around, how well the simulated validator correlates with true test time outcome.\n- Does the magnitude of the weakness of the simulated novice impact the effectiveness of the simulated validator.\n\nTighten the proposer-validation bottleneck claim. Looking at figure 5, why do some environments cause test performance to actually go down after using oracle proposer, for e.g. plunder? A deep dive into that might be interesting.", "broader_impact_concerns": "Research into AI and human collaboration is very important, and the authors attempt to advance this direction is commendable. In the light of recent advancements in AI and dangers of potential AI job automation, more work on AI and human collaboration is needed!", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O4FcuKr4fk", "reviewer": "Reviewer_qeVG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper makes three primary contributions to the field of human-AI collaboration and reinforcement learning. \n\n1. **Formal Definition:** It formally defines the *\"Yield-or-Request Control under Zero-Shot Generalization (YRC-0)\"* problem, where an AI agent must learn when to ask for help from an expert in new environments without any interaction with experts.  \n2. **Benchmark Contribution:** It introduces **YRC-Bench**, an open-source benchmark designed to facilitate research on the YRC-0 problem, complete with a variety of simulated environments, experts, and evaluation tools.  \n3. **Empirical Study:** Finally, the paper presents a large-scale empirical study on YRC-Bench, evaluating several baseline methods and offering insights into their performance and limitations.\n\n---\n\n**Key Strengths**  \n- **Addresses a Critical Problem:** The paper tackles the highly relevant and important challenge of how to build AI agents that can safely and effectively leverage expert assistance in novel situations. \n- **Comprehensive Benchmark:** The development of YRC-Bench is a contribution. It provides the research community with a set of environments for studying the YRC-0 problem.  \n- **Thorough Experimental Evaluation:** The authors have conducted an extensive set of experiments, which demonstrates a serious effort to understand the performance of the baseline methods and the challenges of the YRC-0 problem.\n\n\n---\n\n**Key Weaknesses**  \n- **Unrealistic Assumptions:** A major limitation is the benchmark's reliance on *“nearly ideal”* simulated experts. The experts for MiniGrid and Procgen are PPO policies trained directly on the test task distribution, and the CLIPort expert is a *rule-based oracle.* \nThis design choice means the experts have perfect, oracle-like knowledge of the exact environments the agent is being evaluated on — far from a realistic human expert, who would be suboptimal and biased. This assumption fundamentally limits the paper's claims about real-world human-AI collaboration, as the agent is learning to coordinate with a “god-like” oracle, not a realistic partner.  \n- **Uninformative Baselines:** The *“Always-Yield”* and *“Always-Request”* baselines are so simple that they are designed to be easily beaten and don’t represent any intelligent strategy. Their inclusion is standard, but they don't provide much insight. The fact that a *RANDOM* policy is *“surprisingly strong”* is a major red flag—it indicates that the other, more complex methods are failing to learn a meaningful policy.", "claims_and_evidence_yn": "No", "claims_explanation": "**Claim: The proposed \"proposer-validation\" decomposition is a framework for tackling the YRC-0 problem.**\n- **Evidence Provided:** The paper introduces a two-part framework where a \"proposer\" (the agent's policy) suggests an action and a \"validator\" decides whether to execute that action or query the expert. This is presented as a key methodological contribution.\n- **Why the Evidence is Unconvincing and the Framework is Inappropriate:**\n- **1.** The paper motivates its work by correctly identifying the challenge of deploying AI in \"real-world environments\" where they must collaborate with \"experts, whether humans or highly capable AI systems.\" It then implicitly argues that its \"proposer-validation\" framework is a useful model for understanding and tackling this real-world problem.\nHowever, the evidence for this framework's usefulness is unconvincing because the framework itself is an oversimplification of the problem it claims to address. The framework imposes a rigid, binary decision structure (either accept the agent's action or query the expert). A human expert might not just provide an alternative action; they might offer guidance, adjust the agent's plan, or take over for a sequence of actions. The framework's design is too restrictive to capture these richer forms of interaction.\n- **2.** The claim that the \"proposer-validation\" framework is a key contribution to tackling the YRC-0 problem is weak. The core problem, an agent needing to detect its own incompetence in a novel (OOD) environment and cede control to an expert, is already a central topic of research in fields like: 1) Safe Reinforcement Learning (e.g., identifying unsafe states and reverting to a safe policy); 2) Uncertainty Quantification (e.g., using model uncertainty to trigger a specific action); 3) Out-of-Distribution Detection (e.g., detecting covariate shift and flagging for human review).\nMeta-Learning RL (e.g., training agents to learn adaptation strategies that can be applied to new, unseen tasks, which is the very essence of the YRC-0 \"zero-shot\" challenge)\nResearchers in these areas are already developing algorithms (like deep ensembles, Bayesian methods, etc.) that are designed to solve this exact issue, and they can be applied without adopting the \"YRC-0\" or \"proposer-validation\" labels.\n\n**Claim: The experiments conducted on YRC-Bench provide a generalizable evaluation of methods for the YRC-0 problem.**\n- **Evidence Provided:** The paper presents a large-scale computational study across three distinct environments (MiniGrid, Procgen, CLIPort) with multiple baselines, evaluated on task performance penalized by expert query costs.\n- **Why the Evidence is Unconvincing and the Experiments are Inappropriately Designed:**\n- **1.** The most significant design flaw is the use of \"nearly ideal\" simulated experts. This is an explicit choice mentioned in the paper. The difficulty often lies in dealing with an expert who is fallible, inconsistent, has communication overhead, or whose strategy may differ from the agent's. By removing this complexity, the experiments test the agent's ability to recognize its own uncertainty, not its ability to coordinate with a realistic expert. The evidence gathered is only valid for a scenario that does not exist in the real world.\n- **2.** The paper reports that a \"RANDOM\" baseline (choosing randomly whether to yield or not) is \"surprisingly strong.\" A well-designed benchmark should present a challenge where intelligent methods clearly outperform naive or random ones.", "audience_yn": "Yes", "audience_explanation": "Researchers in AI safety, robustness, and human-AI Interaction, and the general reinforcement learning and decision-making community", "requested_changes": "**Would Strength the Work**\nAdjustment suggestions: \n- **1.** The paper's introduction and abstract explicitly motivate the work by highlighting the need for AI agents to collaborate with \"experts, whether humans or highly capable AI systems\" in \"real-world environments.\" This frames the paper's goal as understanding and improving this real-world human-AI collaboration. Given this stated goal, the complete absence of human participants is a critical weakness. While a large-scale real-world deployment is not expected, even a small, qualitative study with human participants interacting with an agent in one of the simpler environments (e.g., MiniGrid) would add significant value. This could provide a crucial sanity check on whether the findings from the simulated-expert experiments have any bearing on the complexities of real human interaction.\n- **2.** It is critical to include comparisons against more sophisticated methods from related fields, such as: baselines from Bayesian deep learning or ensemble methods that are specifically designed for more robust uncertainty quantification. Baselines from active learning or meta-learning that are designed to adapt quickly to new distributions. Without stronger baselines, the conclusion that the problem is difficult is not fully supported; it may be that the tested methods are simply not the right ones.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)