Reinforcement Learning from Imperfect Corrective Actions and Proxy Rewards
Authors: Zhaohui JIANG
Xuening Feng
Paul Weng
Yifei Zhu
Yan Song
Tianze Zhou
Yujing Hu
Tangjie Lv
Changjie Fan
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
imitation learning
corrective action
proxy reward
human-agent alignment
Primary area: reinforcement learning

Abstract: In practice, reinforcement learning (RL) agents are often trained with a possibly imperfect proxy reward function, which may lead to a human-agent alignment issue (i.e., the learned policy either converges to non-optimal performance with low cumulative rewards, or achieves high cumulative rewards but in an undesired manner). To tackle this issue, we consider a framework where a human labeler can provide additional feedback in the form of corrective actions, which expresses the labeler's action preferences although this feedback may possibly be imperfect as well. 
In this setting, to obtain a better-aligned policy guided by both learning signals, we propose a novel value-based deep RL algorithm called **I**terative learning from **Co**rrective actions and **Pro**xy rewards (ICoPro), which cycles through three phases: 
(1) Solicit sparse corrective actions from a human labeler on the agent's demonstrated trajectories; 
(2) Incorporate these corrective actions into the Q-function using a margin loss to enforce adherence to labeler's preferences; 
(3) Train the agent with standard RL losses regularized with a margin loss to learn from proxy rewards and propagate the Q-values learned from human feedback. Moreover, another novel design in our approach is to integrate pseudo-labels from the target Q-network to reduce human labor and further stabilize training. 
We experimentally validate our proposition on a variety of tasks (Atari games and autonomous driving on highway). On the one hand, using proxy rewards with different levels of imperfection, our method can better align with human and is more sample-efficient than baseline methods. On the other hand, facing corrective actions with different types of imperfection, our method can overcome the non-optimality of this feedback thanks to the guidance from proxy rewards.

Published as a conference paper at ICLR 2025
REINFORCEMENT LEARNING FROM IMPERFECT COR- RECTIVE ACTIONS AND PROXY REWARDS
Zhaohui Jiang, Xuening Feng & Yifei Zhu
University of Michigan-Shanghan Jiao Tong University Joint Institute, Shanghai Jiao Tong University (jiangzhachui, cindy fang2019 yifii. zhu) jtu. et.
Paul Weng
Digital Innovation Research Center, Duke Kunshan University wveggdukekurng
edu. en
Yan Song, Tiane Zhou, Yujing Hu, Tangjie Lv & Changjie Fan
NetEase Fuxi AI Lab
{songyan, choutianze, hug.r
wwwp://www.ooo..ee.eoooommm con
ABSTRACT
In practice, reinforcement learning (RL) agents are often trained with possibly imperfect proxy reward function, which may lead to a human-agent alignment is- sue (i.e., the learned policy either converges to non-optimal performance with low cumulative rewards, or achieves high cumulative rewards but in undesired man- ner). To tackle this issue, we consider framework where human labeler provide additional feedback in the form of corrective actions, which expresses the labeler's action preferences although this feedback may possibly be imperfect well. In this setting, to obtain a better-aligned policy guided by both learning signals, we propose a novel value-based deep RL algorithm called Iterative learn- ing from Corrective actions and Proxy rewards (ICoPro which cycles through three phases: (1) Solicit sparse corrective actions from human labeler on the agent's 's demonstrated trajectories; (2) Incorporate these corrective actions into the Q-function using a margin loss enforce adherence labeler's preferences; (3) Train the agent with standard RL losses regularized with margin loss to learn from proxy rewards and propagate the Q-values learned from human feedback. Moreover, another novel design in approach is to integrate pseudo-labeis from the target Q-network to reduce human labor and further stabilize training. We experimentally validate our proposition autonomous driving on highway). On the one hand, using proxy rewards with dif- ferent levels of imperfection, our method can better align with human preferences and is more sample-efficient than bascline methods. On the other hand, facing corrective actions with different types of imperfection, our method can overcome the non-optimality of this feedback thanks to the guidance from proxy rewards.
variety of tasks (Atari games and
INTRODUCTION
While reinforcement learning (RL) has proved its effectiveness in numerous application domains (Mnih et al.) 2015) Silver et al. 2017: Levine et al. 2016). its impressive achievements are only possible if a high-quality reward signal for the RL agent learn from is available. In practice, correctly defining such reward signal is often very difficult (e.g., in autonomous driving). If rewards are misspecified, the RL. agent would generally learn behaviors that are unexpected (Amodei al.) 2016b) and unwanted (Clark & Amodei, 2016)] Russell & Norvig 2016) by the system designer, notably due to overoptimization (Gao et al. 2023) or specification gaming (Randlov Alstrom 1998). This important issue has been well-recognized in the research community (Amodei et al. 60) and is an active research direction (Ouyang et al. 2022 Skalse et al. 2022
We have open-sourced its implementation: http://ggiiaaccmm//aa/aaaaaaaaat

---

Published as a conference paper at ICLR 2025
Various solutions have been proposed to avoid having to define reward function, for instance, be- havior cloning (Pomerleau 1989 Bain & Sammut 1995), inverse reinforcement learning (Ng & Russell 2000; Russell 1998), 3, (inverse) reward design (Hadield-Menell et al.] 2017), reinforce- ment learning from human feedback (Busa-Fekete et al. 2014] Christiano et al.) 2017). However, these approaches could be impractical and infficiintt since they may require perfect demonstrator, a reliable expert to provide correct labels, or assume that the agent only learns from human feedback, which would subsequently require too many (often hard-to-answer) queries for a human to consider.
As an alternative intermediate approach, we propose the following framework in which the RL agent has access to two sources of signals learn from: proxy rewards and corrective actions. A proxy reward function is an imperfect reward function, that approximately specifies the task to be learned. corrective action is provided by a (possibly unreliable) human labeler when s/he is queried by the RL agent: a trajectory segment is shown to the labeler, who can then choose to correct an action performed by the agent by demonstrating another supposedly-good action.
This framework is practical and easily implementable. Regarding proxy rewards, they are generally easy for system designers to provide. For instance, they can (1) learn proxy rewards from (pos- sibly imperfect) demonstration, (2) manually specify them to roughly express their intention (e.g.. supposedly- actions are rewarded and expected bad actions are penalized), or (3) only define sparse rewards (e.g.. positive reward for reaching a destination and penalty for crash). Regard- ing corrective actions, in contrast to typical demonstrations of whole trajectories, this feedback is usually much easier for the labeler to provide, since humans may not be able to complete task themselves but can readily offer action preference on some states.
While only learning with one of those two sources of signals has its own limitations, by proposing our framework, we argue (and experimentally demonstrate) that learning simultaneously from both of them, even though both may be imperfect, can be highly beneficial. More specifically, on the one hand, solely learning from proxy rewards would either lead to very slow learning (c.g,, when proxy rewards are well-aligned with ground-truth rewards but are very sparse) or yield policy whose performance is not acceptable to the system designer (e.g., when proxy rewards are dense, but misspecified). On the other hand, solely learning from corrective actions would require too many queries to the human labeler and may lead to suboptimal learned behaviors since the human may be suboptimal. In contrast, our key insight is that the two sources of signals can complement each other. Since they generally imperfect in different space regions, bad decisions learned from proxy rewards can be corrected by the human labeler, while the effects of suboptimal corrective actions may be weakened by proxy rewards. Therefore, learning simultaneously from the two imperfect signals can achieve better behaviors more aligned to the system designer's 's goals than using any one of the two alone and can be more sample-efficient (in terms of both environmental transitions and human queries).
As a proof of concept, we design a novel value-based RL algorithm (see Figure D. called Itera- tive learning from Corrective action and Proxy reward (ICoPro). ICoPro alternates between three the RL agent interacts with the environment to
steps. In the first step (datacollection-hhate collect transition data, and then the labeler provides corrective actions on them. In the second step teetuun--passee the RL agent learns to select actions according to this feedback via margin loss, which can be interpreted as an imitation learning (II.) loss. In the third step ppropagations phase), the RL agent is trained to maximize the expected cumulative proxy rewards while further enforcing margin loss. This latter loss is expressed on both observed labels (i.c., corrective actions given by the human labeler) and pseudo-labels generated by a trained model (target Q-network). Imitating such pseudo-labels can be interpreted in two ways: either to reduce the number of queries or to stay close to the previously-Leane methods (Schulman al., 2015 2017 Asadi al., (002)). By combining imitation learning and reinforcement learning in this third phase, the agent can learn from both proxy rewards and enforce the temporal consistency of the values learned from imitation learning.
policy, which can stabilize training like in trust region
The contributions of this paper can be summarized as follows: (1) We propose a practical human- in-the-loop reinforcement learning framework (Section 3) where the RL agent can learn from proxy rewards and (possibly suboptimal) corrective actions. (2) To train the agent from these two imperfect sources, we design an efficient and robust algorithm (Section 4 in which we demonstrate a simple but useful technique to reduce human feedback. (3) We experimentally validate our proof of concept (Section 5 by conducting an extensive number of experiments on various domains (e.g., image-
2

---

Published as a conference paper at ICLR 2025
Data collection Phase
Undesired Performance
A e
St
St
0
S2.7
>
Evaiuattonn rollouts
Agent
Sample queries
D
Den
Labeler
nt
attx
((x,a))
Corrective actions
Next iteration
Policy-Learningg Optimize
Propagati on Phase
Finetune Phase
Target Q
Q"
RL loss on Denn with proxy rewards
Margin loss on D
Margin loss on D
Margin Denvy D with target pseudo-labels
ICoPro is an iterative method with three phases in each iteration. It starts with the da a collect cn-phase to collect agent's rollouts. Segments are then sampled from these rollouts and used queries for the labeler to provide several corrective actions. Following this are two separate phases for policy updating, the finetune- and propaçat itgiaiiioon iitt-hhssee Then the updated policy is utilized in the data collections ioncthao of the next iteration.
Figure 1:
based Atari games and state-based highway driving with sparse and dense proxy rewards) under different conditions (e.g., simulated or real human labeler).
RELATED WORK
Given the difficulties in defining ground-truth rewards, a growing number of studies attempl to
corporate various types of human feedback to train agents, for example, demonstrations (Pomerleau
1989
Hester et al. 2018 Reddy et al.) 2019), preferences (Busa-Fekete el al. 2014 Christiano al.
2017 Brown al. 2020 Lee et al.) 2021a). scalar evaluation (Knox 2012 Knox & Stone [012
Saunders et al. 2018). action advising (Maclin Shavlik 1996 Da Silva et al. 2020: Ilhan et al.
2021a), interventions (Peng al. 2023 Luo al. (222)) example-states (Eysenbach al. 2021
or combination of multiple feedback types (Ibarz et al., 2018 Jeon et al. 2020 Yuan et al. 2024 Dong et 2024). Among them, demonstrations, action advising, and interventions, which discuss further below, are the most related to corrective actions. We roughly organize these works in three clusters. Preference signals can be provided an offline or online setting. In addition, we discuss related work that considers combining reward and other preference signals.
Offline setting One of the earliest approaches to circumvent the need of defining reward function imitation learning (aka learning from demonstration), where batch of demonstration assumed to be available. Two main approaches have been developed: Behavior Cloning (BC) and Inverse Reinforcement Learning (IRL). BC (Pomerleau 1989 Bain & Sammut 1995) requires significant amount of data to achieve the desired performance, while IRL (Ng & Russell 2000 Russell [998] Brown et al. 2020) is computationally expensive as it derives the policy from the rewards obtained from demonstrations, although some work (Ho & Ermon] 2016 |Reddy al.) 2019) aims over- come these limitations. The major drawback in methods designed in the offline setting stems from training policies on fixed limited dataset, which may yield policy that can perform badly when deployed due to the train-tes: distribution mismatch. A typical approach to overcome this issue is to resort to online policy training, as we do our method.
Online setting contrast to the offline setting, here the agent learns with a human the loop: either human actively oversees the agent's training and can intervene, the agent can query human to obtain feedback to lear from. The former case (Spencer et al.) 2020 Peng et al. 2023] Luo al. 2024] requires active participation and continuous attention from humans, while the latter may require too many queries, which can be too labor intensive for one or even multiple humans provide. Examples of the latter approach include methods in the DAgger family (Ross et al.
3

---

Published as a conference paper at ICLR 2025
2011 Sun al. 2017 Kelly et al. 2019) (with optimal action labeling), and reinforcement learning
from human feedback (RLHF) methods (Busa-Fekete et al. 2014 Christiano et al. 2017 Lee et al. 2021a)
(with pairwise comparisons).
Combining reward and other signals Since learning from one signal may be insufficient and/or inefficient, multiple works (Hester al. 2018 Ibarz et al., 2018) consider learning from demon- stration and rewards. In such work, demonstration is often provided offline and used for initializing an online training phase, which can be standard RL training (Hester et al. 2018) or use other types of feedback such as in RLHF (Ibarz et al. 2018), which can notably help for better human-agent alignment. In particular, demonstrations can be used to initialize a policy (lbarz et al., 2018 define
reward signal (Reddy et al., 2019)), or initialize a replay buffer for future RL training (Hester et al., 2018). In addition, they can also be queried online (Chen et al., 2020 Peng et al., 2023 Luo et al,, 2024. Most works assume a well-specified (ground-truth) reward function, then focus on balanc- ing the IL and RL objectives when human demonstrations are also usually assumed to be optimal (Shenfeld et al. 2023 Liu et al.) 2024), although some propositions have been made to deal with suboptimal demonstrations (Nair et al. 2018).
In contrast to previous work, we do not assume the availability of a ground-truth reward function but only an approximation of it (i.e., a proxy reward), which is easier to specify. Since learning from this imperfect signal is insufficient, we complement it with corrective actions. Providing cor- rective actions is less gegniivell-demandiin
for the human than generating whole demonstrations, especially if the agent and the human do not operate in the same sensori-motor space. Compared to action labeling (Ross et al. 2011, Kelly et al. 2019) or intervention (Peng et al. [0223] Luo et al.) 2024), which have happen on all unsatisfactory cases, our corrective action setting is less costly since the human chooses on which states s/he wants to give the feedback. Finally, compared to most previous work, we do not assume that the human is optimal. The goal of our paper to demonstrate as a proof of concept that the agent can learn more efficiently to reach a better performance from two imperfect sources of signals than with any of the two alone.
PROBLEM FORMULATION
We consider a reinforcement learning (RL) problem (Sutton & Barto 2018) where an RL agent repeatedly interacts with an environment: Given the state space S and action space A, at a time step 4, after observing an (observation) state 8: E S, it performs an action az E A, which yields an immediate reward r. E IR and brings the agent to new state 8t+1 ~ P(((st, where P the environmental transition function. The goal of the RI. agent is to learn, from interaction tuples (3;, de, Tr, 1). a policy x² : S - AA that maximizes the expected discounted sum of rewards. Recall that policy can be represented as the greedy policy with respect to Q-function, where Q SSS + R and Q(s, a) = E IE>= "rrs] with discount factor y, then π ((s) )) = arg max, Q(s, a).
In contrast standard RL, we do not assume that ground truth reward function (i.e., providing perfect description of the task that the RL agent needs to learn) is available. Instead, the agent only receives an approximation of it, called proxy reward function and denoted F. Indeed, while a precise reward function capturing all aspects of a desired behavior is hard to define for a system designer providing proxy reward function to roughly express the system designer's 's objectives is much easier to achieve (Reddy et al. Jeon:Jeon et al. 2020 Luo et al. 2024). However, since the proxy rewards are approximate, purely relying on learning from them may never achieve the desired performance. Therefore, we assume that the RL agent can also query a human labeler to obtain additional feedback under the form of corrective actions (see Figure [I]. Formally, a query corresponds to a T T-step seg- ment (i.e., sequence of state-action tuples) q (st; ot...., - 044T-1), which can for instance be sampled from trajectories generated from the interaction of the RL. agent with its environment. Given such query, the human labeler may select one state sy (such that t <t' < t T - 1) where s/he provides a corrective action af = π (sxr), where T² is the labeler's policy. The corrective action is supposedly a better choice than the peefooly- one ar,, Moreover, the labeler may choose to provide no feedback all the actions in the segment are good.
Note that we do not assume that the labeler is necessarily optimal. This means that it is not possible to solely rely on the corrective actions (without proxy rewards) to learn a policy to achieve the desired performance, which would furthermore be impractical in terms of label collection cost for
4

---

Published as a conference paper at ICLR 2025
the human labeler. Instead, by combining the two imperfect sources of learning signals, the RL agent could possibly learn a policy with a better performance and in a more sample-efficien way than using proxy rewards corrective actions alone. Intuitively, the imperfections of the two signals probably do not lie the same state-action space regions and therefore the two signals could correct each other. We confirm this intuition in our experiments (see Sections 5.2 and 5.3
METHOD
As first work demonstrating the benefit of learning from the two imperfect sources of feed- back. we assume the action space A is finite for simplicity. Building on the Rainbow algorithm (Hessel al. 2018). we design ICoPro, an iterative approach to learn Qo with desired policy ( = arg A (s.-). At each iteration i, ICoPro updates the current Q" to obtain the next Q according to the three phases: DataCollection, Finetune, and Propagati on, which we elaborate on next. Figure I outlines the proposed framework and Algorithm T is provided in Appendix B.
Data Collection-phase. Inspired by the popular RLHF setting (Christiano et al. 2017]. we adopt the following protocol for collecting two types of data: interaction tuples and corrective actions. For the former, the agent acts in its environment according to the e-greedy policy with respect to Q": denoted Ge (Q2)). to collect set De of H interaction tuples, (st, de, Tt, st+1), which are also stored in the transition replay buffer Deno For the latter, the RL agent can regularly during RL training issue batch of sampled queries from De to the labeler obtain set DÍ of pairs of state and corrective action (s, a²), which are also stored in the feedback replay buffer D4. The collected data, Dene and D are used to train the RL agent in the next two phases. As a side remark, for simplicity, querying the labeler is currently implemented in experiments in synchronous way (i.e., the agent waits for the labeler's feedback), but this process could possibly also be asynchronous.
Finetune-phan This phase is pure supervised training phase to learn from all the labeler's corrective actions collected so far in D4. Formally, O% is updated with the following margin loss Hester al. 2018 Ibarz et al. 2018):
CMG (Kim et al. 2013 Piot et al. 2014
M (0; D (s JEDE JEDe
max (s,a) ((a", a)) - (,(6, a
where a(a",a) 0 amounts to enforcing that the corrective actions' Q-values should not be smaller than those of any other actions. As common practice, Q is updated via mini-batch stochastic gradient de- scent using Equation ( Following classical methods (Christiano al. 2017) in RLHF, the updates end when the actions predicted by the updated Q" reaches a pre-defined accuracy Sace: PANDL [at arg max, (s. > Sace.
a², and a non-negative margin value C otherwise. This loss
Propagation-phan Since pure imitation learning requires a large number of human labels, we design the propagation-phat losses and margin loss (from actual labels in D but also pseudo-labels, which we explain next.
include the training of the updated Qo: with combination of RL
Training with RL Losses allows not only learning from proxy rewards, but also propagating the effect of human labels to more states (leared in the previous phase). The initial target Q in this phase the Q obtained at the end of the finetune-phane phase.. These RL losses are composed of two commonly used terms, LR and CRL, which are respectively defined as:
CRI(0, Denn, Q) = =Epenv (00e (st, ar)
Fr + max U(on),)))) and d'Ed
(2)
)
(0%
CR Denv Q) = Eponn
Trun + n" max ((Ltnn a') a'E.A
(3)
where Denr is the transition replay buffer, and Q is the target network from a historical version of 9² Note that in the context of standard RI., Equation ( is actually not theoretically-founde,,,
5

---

Published as a conference paper at ICLR 2025
since the goal is to learn greedy policy. However, in our context, it can help propagate faster the Q-values learned from the corrective actions.
Training with margin loss using both actual and pseudo-labels allows complement and correct the training with proxy rewards. Since the corrective actions in D are not exccuted in Denr, training with them using loss LMG prevents the agent from forgetting about the actual labels. In addition, training with pseudo-labels can reduce the cost of collecting human labels by leveraging the large number of unlabeled states in Dene, Pseudo-labels can be generated, using predicted greedy actions from the target Q, only on unlabeled states, since otherwise, it is better to enforce the actual label than pseudo-label. Formally, the margin loss using pseudo-label can be expressed as follows:
CMG CTG DTGT)= = - max (s, a) + 1aa TGT .a)] - (x, (4)
aEA
where DTGT = {(,,a= 8 E Den a TGT = arg ma Q(+,) .) and by abuse of notations, Denu denotes the states that have not received any actual corrective actions a In- tuitively, as the training process goes we expect that the quality of the pseudo-labels improves, enhancing further the benefit of enforcing LMG As a side note, LMG can also be understood as regularizing the online network's 's parameters to stay close to the target network, which has been shown to be beneficial in terms of learning efficiency (Asadi et al. (022). In contrast to this previous work, regularize via the margin loss in the policy space instead of the parameter space.
To sum up. the total loss in this phase is the sum of the RL losses and the margin losses
LL+TCT Dene ,Dh), where LMG Li+ (0, Denn DD) = (1 C. (0, D + w LMGT (Dmm))
Prop f (0: Denv Q) + CRL (0; Denn Q) LMG
(5)
and hyperparameter v controls how much weight is put on pseudo-labels. In our implementation, CRI. LRT. and LTGT works same minibatch from Dene in each gradient step, while L works another one of the same size from D' to relieve the unbalanced issue of these two datasets.
4.1 OTHER IMPLEMENTATION DETAILS
Updating schedule for Q" and Q. Since a TGT will change every time we update the target Q update Q with low frequency to stabilize training. Concretely, there F. epochs in one propagation-phast
that each epoch optimize Prop through Den' At the end of each epoch, Q will be updated to the online parameter Q": If E 1, pseudo-labels generated form the policy at the end of f Sinetune-phasse If E > 1, pseudo-labels are changing after each epoch and can come from the (potentially) improved Q" after optimizing Prop.
propagation-pin on-phase are
Optimization for ICoPro's iterative scheme. As for the optimizer, we use use a same optimizer through the two learning phases in ICoPro. Since ICoPro uses an iterative learning scheme, we follow the research from Asadi et al. (2024) that reset the 1st and 2nd order moment inside the Adam optimizer the beginning of each learning phase. The finetune- and propagation-phan use the same learning rate a, which remains unchanged throughout the training procedure.
5 EXPERIMENTS
We first explain our general experimental settings in Section 5.1 To compare ICoPro with basclines and analyze its design components on large scale of experiments, we use simulated labelers to demonstrate and analyze ICoPro's performance in Sections 5.2 and 5.3] Then in Section 5.4 we further verify the alignment performance of ICoPro using real humans' action feedback.
5.1 GENERAL EXPERIMENTAL SETUPS
To ensure reasonable labeling effort, only at most 1% of the environmental transitions are labeled in our experiments. Hyper-parameter values are given in Appendix B.2 unless otherwise specified. Detailed evaluation configurations can be checked in Appendix D.I
In Equation give equal weight to the II. and RL losses, but unequal weights could be provided if we expect one source to be suboptimal than the other.

---

Published as a conference paper at ICLR 2025
Baselines. To compare with the performance of learning from only proxy rewards, we use Rain- bow (Hessel et al.) 2018) as a baseline. To compare with the performance of learning from only corrective actions using 22 2. (Equation (I)), we evaluate behavior cloning (BC-L) on all collected labels D4 at the end of training of ICoPro as the one in offline style, and adapt HG-DAgger (Kelly et al. 2019) (HGDAgger-L) by removing the propagation-plee
from ICoPro as the one in on- line style. As for baselines involving learning from both signals, since previous work considers the exact same learning scheme as explained in Section [3 we adapt two state-of-the-ar art methods that were initially designed on normal RL methods into our iterative learning setting with the same labeling schedule and budget, DQID (Hester et al., 2018)) (DQfD-I) and PVP (Peng et al. 2023) (PVP-I), to compare the performance when learning from both corrective actions and proxy re- wards. Basically, the adaption is canceling the finetune-
in ICoPro and replacing their loss functions with ICoPro's Prop (Equation (5) in the propagation-
DQAD-I can be seen as an ablation of ICoPro by removing L.G TTG from Prrop. PVP-I's loss replaces LMG L+TGT - 1|2 + (s, 1I but using zero rewards in CRT. and CRE. We also test in PVP-I, the effects of using proxy rewards instead of zeros (PVP-IR). More details about the baselines and other ablations mentioned in Section 5.2 can be checked in Appendix C.T
to CPVP ()) [sQ
Environments. Our experiments involve both the state-based highway (Leurent 2018 environment
and imaged-based Atari (Aitchison et al. 2023) environments.
Highway is an environment that allows for flex- ible design of diverse proxy rewards and evalua- tion metrics, but is relatively simpler than Atari in terms of action dimension (IAA = 5) and en- vironmental complexity. The basic goal in this environment is to drive a car on a straight road with multiple lanes. The final performance of the trained vehicle can be measured with dif- ferent metries, e.g., the ratio of episodes with a crash (%Crash), the average total forward dis- tance in a given time limit (Distance-Avg), the average speed (Speed-Avg), and the ratio of tak- ing change lane action DeLLanChhage-Aver We design representative proxy rewards in differ- ent levels of imperfection by associating events with different rewards as shown in Table
Table I: We normalize the rewards into the range [-1, 1] in each step by min-max scaling if Normaliza- tion is not None, where the min (max) reward is the sum of negative (positive) event rewards.
Engineered proxy rewards on highway.
Proxy Rewards
Configurations
PRExp
PRI
PR2
PR3
PR4
Rewards Change lane
0.2
0.2
0
High speed
1.5
2
0,8
0
0
For
Low speed
-0.5
0
Events
Crash
-1.7
-1
-1
-1
-1
Normalization
[-1,1]
[-1.1] [-LI] [-L1] None
Dense(D) or Sparse(S)
D
D
- D
S
In contrast, Atari provides more complex environments with larger action dimensions and longer episode lengths, although with less flexibility in designing proxy rewards and diverse performance metrics. Since the dimension of action space can be seen as a measurement of hardness for one game, we first select 6 representative games with the full action dimension (I-4) 18) across var- ious categories suggested by Aitchison et al. (2023): (1) Combat: Battlezone and Seaquest, (2) Sports: Boxing, (3) Maze: Alien, (4) Action: Frostbite and Hero. We also include smaller-action- dimensional but classical games to complement our evaluation: MsPacman and Enduro (I.Al = 9), Pong ([A) 6). and Freeway ((A) = 3). In this setting, cumulative episodic raw rewards from ALE are used to measure agent performance, with signed raw rewards serving as proxy reward
Simulated labeler. We use Q-values trained with Rainbow simulate human feedback with a), where Q²(s, a) = Ep, (er== a] is the labeler's Q-function, and aL ~ (Q" -)) is the action issued from the labeler, a (Q" :)) is the executed action from the training agent. In practice, we select states with top NCF largest Qdiff to give corrective actions for querying segment Luo et al. (024)) where Ner = 1 is our default configuration. Basically, simulated labelers are agent checkpoints trained with engineered rewards by Rainbow but do not necessarily converge to optimal performances. For highway, labelers are trained with PRExp mentioned in Table [ For Atari, labelers are trained with the signed raw rewards. Concrete configurations to train labelers be checked in Appendix D.4
a)lsu = s, Ap- =
RRa rewards TA in Atari can be seen as gnal-conditioned rewards, but not all desired performances (e.g.. achieve goals in the right way) are rewarded in r.A Further discussion is available in Appendix D3
7

---

Published as a conference paper at ICLR 2025
Prasy Rewards
PRFxp
PR1
PR2
PR
Methods
ICoPro
Rainbow
Labeler
1200
28
NM NLann Change Avg
Aug
1000
Spacc Avg
20
NiGrash
0.50
500
0.25
200
2
0.00
20
400
-0.25
18
120
150
15C
50
120
150
50
120
Step (X10K)
5tep (X10K)
Step (X10K)
Step [x
10K)
Compare ICoPro (solid lines) with Rainbow (dotted lines) using different proxy rewards (different colors).
Methods
ICoPro
DQID-
HGDAgger-
Reinbow
PVP-R
PVP4
BCLL
Labeler
1.25
1200
1.00
1.00
StansChnngge Avg
1000
50
0.75
0.75
Stop Avg
750
0.50
0.25
800
8
0.00
250
0.00
-0.25
0
30
50 Step (x10K)
60
120
150
30
6.0
90
1.20
150
30
60
90
120
150
120
Stap (xX0
Stop (x10K)
(x10K)
(b) Compare ICoPro with other baselines (different colors). Results are averaged over the set of proxy rewards.
Figure 2: Experiments on highway using the set of proxy rewards mentioned in Table 1 Each sub- plot compares the performance with respect to one representative performance metric. |1.5K.
Experimental results on Atari with different size of |DL]. Red fonts denote the best results with the largest mean score. Bold black fonts emphasize results whose mean+sstd covers the best mean score. Yellow backgrounds highlight results that match the labelers' performance in the sense that their mean scores are in the range of the labelers' ttd
Table 2:
IP*
Methods
Environments
Seaquest
Boxing
Battlezone
Froutbite
Alien
Hero
MsPacman
Labeler
1021 924-224
66.05++9.17
24485, 199 s0 5647.63 1647.63-109899
2908 97-0087
20866.07-61233
3539: $539:54+127009
Rainbow
207.04 96
1.99-071
17058.00=210..0
2766.40-518875
1319.83=457.066
19413.57=600.5 39
1773.98+3233 19
HGDAgger-L
790, 10.73398
24.95.12:22
16573.331
3207
527.35
2067.45126200
13258.79
3052.38
PVP-I
635.83
7,76=084
15183.02:139.17
1535,28.
596. 56:696999444 45
8503.46::1208.
1686. 13
PVP-IR
770.88+991.0 D0
13.36+2000
17378.18=2244
18-2256442
1664.62-414.00
798. 998.55+137:99
7729.12- 12:1122999
920,25+2216 76
Large
wfa nar
BC-L
758.401-43.1
1213.98+102.54
21.3911121
48. 19+7.0
18111.52::1499.0 68
20220.0012881.97
3410.27-479.08
3874.36-515.00
2360.401:42257
1895.06-267.55 85
17805.
14288.67:11673.
35+2241.33
127770771799
2579.86++ 186
wa Finetune
1234_39::13.44
51,83+22667
19266.67=285.25
4713:07=:211:91
2323,66-368.6
8398.37
2633.59#1544
DQDI-I
1155 43:-5585
41.71++500
18444 1279779.70
4176.27-659.1
2079 9979999
1688.1688
2734
ICoPrr
1274.76.110955 53
61.52.1247
20030.30.166200
4817.74_.60..4
65
2360.271.00.52
23344.0112353..
3188.20.1453 453. 13
508.34+77.91 HGDAggen-L
17.91+2.21
14682.22=1633.4
1561 1561.04-457.61
1582,74=467.55 88
10617.77+1664.62
2342.51+224.99
PVP-
439.961588 62
1,0092.05
9242
42:22222
15
509.45:237.66
402 532-533:7844 44
6454,25::1193.92
891.84:1251. 87
PVP-IR
131.731116.70
14.33
16510,30,1
168:000011688:100
1883.641_20..44
1136.07
8777.76.:1410776
1097.92.244 94
Small
wfa P0
BC-L
537.20 86
1065.65+164.01
9.09=4664
40.56++10.2 10
17273.33=:2605.40
16465
1348.80=226.24
2785,41-457.
1269 1269.06-2288.88 76
1711,000659..6
15420.19+22000.0
10252.73#1556.11
1676.97-+6523 21
2278.001::377
w/o Finesune
1081.1611311 98
19.351116.5
21253.33.2669166
3037,84_ 1152,2
891. 991.97.1148.5
7759902.13093.0
1830.23 (.00:233.559..0 33
DQID-1
1003 11:1844
39.27=20085
17003.64:1552.88
3146.25--418.00
1073.79=:24.17
16525. 11
1952 0952.64::227.79
[CaPro
1167.19+105.02
61.40023391
999990991555.40
3368.31-701.54
1736.52-513.91
17862.58+2161.00
2858.57+351. 18
5.2 SAMPLE--FFICIENT ALIGNMENT WITH IMPERFECT PROXY REWARDS
We use one scripted labeler for each environment to validate the alignment ability of ICoPro and compare it with baselines, using proxy rewards with different levels of imperfections. In the follow- ing of this part, we analyze ICoPro's performance based on results shown in Figure ( and Table More specifically, Figure evaluatts the performance with various performance metrics, while Ta- ble 2 evaluates the performance with small or large size of action label budger") Due to space constraints, results for easier Atari tasks (i.e., Pong. Freeway, and Enduro) and corresponding plots for the results in Table [ are put into Appendix A
'Considering that the action budgets required to align depend the complexity of the labeler's perfor- mance, the conerete values of [D') is not specified in the main text but can he checked in Table
8

---

Published as a conference paper at ICLR 2025
Learning from a combination of two imperfect signals performs better than learning from T or corrective actions alone. Compared to Rainbow, Figure 2a and Table 2 show that ICoPro with corrective actions converge quickly and stably to the labeler's performance using proxy rewards with different levels of imprefection, but Rainbow's performance is unstable and can easily converge non-optimal performance without the guidance from corrective actions. Compared to BC-L and HGDAgger-L. (eguuee2Jannd deblsrdemnnstrate that ICoPro's 's performance exceeds their performances substantially in most of the games, which confirms the effectiveness of integrating the proxy rewards with the RL losses to achieve better alignment than using corrective actions alone.
ICoPro achieves better alignment performance than baselines. As shown in Figure 2b and Ta- ble [ using the same sample budget in terms of both environmental transitions and label budget, ICoPro achieves the best performance terms of aligning with labelers' performance almost all settings, performs similarly with the best one otherwise. In Figure 2b except for Rainbow, all methods perform similarly in terms of aligning the labeler's Speed- Avg and %lLaneChange--N per- formance, while for the harder performance metrics % &Crash and Distance-Avg, ICoPro and DQID-I perform similarly, and both are significantly better than PVP-I, PVP-IR, and HGDAgger-L. In Ta- ble 2 with large D ICoPro does match the labelers' performances, except in the hardest Hero with full action dimensions and relatively long episode length. Specifically, compared with the two PVP methods, their performances are obviously worse than that of the methods using margin loss, since GPVP sets a pre-defined bound for Q-values making it fails to adapt to various reward settings. In some games, incorporating proxy rewards into their framework (PVP-IR) performs even worse than not (PVP-I). However, margin loss is suitable choice to incorporate the corrective actions with proxy rewards. As for DQID-I, in simple environments we perform similarly, but in harder Atari environments ICoPro outperforms it and shows more notable performance gaps than in highway.
ICoPro's integral design leads to a stable and robust performance. Considering that the two-phase learning and pseudo-labeling are the main novel designs in ICoPro, there are two additional settings to examined besides DQND-I and HGDAgger-L.: (1) w/o Fineture, which removes the Finetune-phase but retains the pseudo-labels, and (2) w/o TGT, which removes pseudo-label,5] In the highway environment, the two ablations perform similarly to ICoPro, and we include them into Appendix A.2 instead of Figure 2b to make it clearer to check. However, in Table [ we find that while the two ablation settings may perform well in individual environments when action feedback is abundant, they are not as robust as ICoPro across different environments, especially when feedback is limited and tasks are challenging. ICoPro consistently demonstrates superior performance in such scenarios, maintaining its lead even with fewer feedback instances.
Ablations:
5.3 OVERCOMING NON-OPTIMALITY OF CORRECTIVE ACTIONS WITH PROXY REWARDS
To simulate different types of imperfections inside corrective actions, the most natural and com- monly adopted (Lee et al.] 2021b Luo et al. 2024) one is to replace part of a labeler's ' corrective actions with random actions (Di ff Rand). In our setting, it simulates labelers who know which state-action pairs to correct but provide noisy corrective actions. We evaluate it in three represen- tative environments that cover various environmental complexities: highway, Boxing and Seaquest. As shown in Figures Ba and 3b ICoPro performs robustly within a reasonable ratio of random ac- tions (i.e., < 25%), and can perform significantly better than the labelers with large randomness (i.e., 50%). For Atari, we further consider a harder type that uses different labelers with worse perfor- mances (Di (Difflabler) to simulate labelers who are suboptimal on both where and how to provide corrective actions. As shown in Figure ICoPro still demonstrate strong capability to overcome the non-optimality inside such unreasonable action feedback in most of the cases. Performance of Highway using another labeler, and training plots with concrete budget sizes or proxy rewards, instead of the averaged ones in Figure 3 can be checked in Appendices A.1.3 A.2.2 and A.2.3
5.4
USER STUDY
To verify that ICoPro can align with real human's preferences when the desired performance is hard to be obtained with engineered rewards, we use highway (IAI = 5). Pong ((AAl = 6), and Seaquest (AA) = 18) as three representative environments to evaluate ICoPro with real human
*Considrrig that removing margin loss on DL from the propagat cn-phase is not reasonable, we do show this set of ablations in the main text, but the results can he checked in Appendix C3.1

---

Published as a conference paper at ICLR 2025
Neto Faadee-AArrr
10%
38%
Fakky
Labste
1.0
1200
0.8
Dietance- Ang
1000
0.4
0.6
0.3
Speed
0.4
600
13
0.2
0.2
400
Nu
120
200
90
60
0.0
80
120
Step (x (x10K)
Step (x101)
(a) Highway with DindRand. Performances are averaged over the set proxy rewards in Table
of Ronder Adflars
De
50%
Ronks Perfrraacc
(Dobadd)
Labster
Polayy
1CoPro
Labeler
Bertlin (tepae
(b) Atari (Boxing, Seaquest) with DiffRand.
(c) Atari (Boxing. Seaquest) with DiffLabler.
Figure 3: ICoPro facing different types of non-optimality of corrective actions (different colors).
Performances are averaged over large/small D in Figures 3b and 3c
(ICoPro-HHee Humra)) with less than 500 action labels. For highway, we use PR4 as the proxy reward in (CoPro-Humal since does not introduce any extra performance shaping, and obtain a vehicle that
knows how to take over cars with super fast average speed ([Speed-Avg. SCrash]=[299000 0.22I). which is a behavior that we could never obtain with Rainbow using PR4 alone. For Pong, ICoPro- Human obtains an agent that performs more human-like than the simulated optimal labelers (see Figure in Appendix E. for details). For Seaquest, we use it to further verify the applicability of ICoPro-Hunni to environments with larger action dimensions. In this game, saving divers is desired performance that has never been correctly rewarded in its proxy rewards. CCoPro-Humar obtains an agent that rescues 1.5 times more divers than the scripted labeler (3.2 vs. 1.9, averaged over evaluation episodes), even though the scripted labeler costs 46 times more timesteps (0. IM vs. 4.6M). Moreover, compared to ICoPro-Humann performance due to the limited data and non-optimality of human feedback (highway: [29.02, 0.22] [27.90, 0.41)), Pong: 7.83 vs. -11.26, Seaquest: 3.2 2.1), which confirms again ICoPro's ability to learn from two imperfect signals in sample-efficient way. User interfaces and instructions for each environment, examples of corrective actions from humans, well as the training plots and evaluation videos of ICoPro-Human, are provided in Appendix
BC-L on those real human labels gets poor
CONCLUSIONS AND LIMITATIONS
We present a human-in-the-lone framework where an RL agent can learn from two potentially unre- liable signals: corrective actions provided by labeler and a pre-defined proxy reward. The motiva- tion for combining the two learning signals is threefold: (1) guide the agents' training process with corrective actions, (2) use proxy rewards help reduce human labeling efforts, and (3) possibly compensate for the imperfection of one signal with the other. Our value-based method trains pol- icy an iterative way, with two separate learning phases inside each iteration: the f finetune-piae uses margin loss to update the policy to better align with action feedback, then the propagation- phase incorporates RL losses as well as the margin loss expressed with pseudo-labels to generalize the improved values. Our experiments validate algorithmic design.
While we demonstrated the feasibility and the benefit of learning from two imperfect signals, we plan future work to provide more theoretical analysis to reveal what assumptions about the misspecification proxy rewards and/or the suboptimality of the corrective actions could guarantee their synergetic combination.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
This work has been supported in part by the National Key R&D Program of China (Grant No. 2024YFC3017000 and the program of the National Natural Science Foundation of China (No. 62176154).
REFERENCES
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5:
Distilling the areade learning environment down to five games. In International Conference on Machine Learning, pp. 421- 438. PMLR, 2023.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in safety. ArXiv 1606.06565. 2016a.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint arXiv: (oo:::60666500 2016b.
Kavosh Asadi, Rasool Fakoor, Omer Gottesman, Taesup Kim, Michael Littman, and Alexander J Smola. Faster deep reinforcement learning with slower online network. Advances in Neural Information Processing Systems, 35:19944-19955 2022.
Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep rl: An empirical study. Advances in Neural Information Processing Systems, 36, 2024.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Daniel S. Brown, Wonjoon Goo, and Scott Nickum. cester-tha-deemonstee imitation learning via automatieally--anked demonstrations. In Proceedings of the 3rd Conference Robot Learning. 2020.
Róbert Busa-Fekete, Balázs Szörenyi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference- Reinforcement Learning: Evolutionary Direct Policy Search using a Preference- based Racing Algorithm. Machine Learning, 97(3):327-351, 2014.
Si-An Chen, Voot Tangkaratt, Hsuan- Tien Lin, and Masashi Sugiyama. Active deep q-learning with demonstration. Machine Learning, 109:1699-1725, 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing sys- tems, 30, 2017.
Jack Clark and Dario Amodei. Faulty reward functions in the wild. Internet: https://bllg. openai. confffulty-revaardfuuuninin 2016.
Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pp. 5792-5799, 2020.
P Kingma Diederik. Adam: A method for stochastic optimization. International Conference Learning Representations, 2014.
Zibin Dong, Yifu Yuan, Jianye HAO, Fei Ni, Yao Mu, YAN ZHENG, Yujing Hu, Tangjie Lv, Changjie Fan, and Zhipeng Hu. Aligndiff: Aligning diverse human preferences via behavior- customisable diffusion model. In The Twelfth International Conference on Learning Representa- tions, 2024. URL htttss:: http::::///eeeeeiieeeee www.eee.eeeeeeeee www.nee//ooummmdaoommm.m hett:/foouum20OOxxXXXEEEYY
Ben Eysenbach, Sergey Levine, and Russ R Salakhutdinov. Replacing rewards with examples: Example-based policy search via recursive classification. Advances in Neural Information Pro- cessing Systems, 34:1 1541-11552, 2021.
11

---

Published as a conference paper at ICLR 2025
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Leo Gao. John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- wwtppeewuwr....cppppree..pep files/ http::/00.........22222666566661
ran Associates, Inc., 2017. URL /tLpattppp/ppprrceeenn....ooe
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Ercument Ilhan, Jeremy Gow, and Diego Perez. Student-initiated action advising via advice novelty. IEEE Transactions Games, 14(3):522 532, 2021a.
Ercument Ilhan, Jeremy Gow, and Diego Perez Licbana. Action advising with advice imitation in deep reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 629-637,20 2021b.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415- 4426, 2020.
Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. Hg- dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8077-8083, 2019. doi: 10. R0.1109/AA.2211..879988
Beomjoon Kim, Amir-massoue Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL ptttpp:://proceedinggt neur http://ss..rrrrrroo...p..ppppe paper/2013/EF e /ttp::/000ccccc33333.3aaba.
files/ ee2bl -aper..app..pdr..
W Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, volume 1004, pp. 475-482, Valencia, 2012.
William Bradley Knox Learning from human human-genennn generated reward. PhD thesis, The University of Texas at Austin, 2012.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficieen interactive reinforcement learning via relabeling experience and unsupervised pre- vre-training. In 38th International Confer- ence on Machine Learning, ICMI. 202/. International Machine Learning Society (IMLS), 2021a. a.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking peeference-based reinforcement learning. In 35th Conference on Neural Information Processing Systems (NeurIPS). Neural Information Processing Systems Foundation, 2021b.
12

---

Published as a conference paper at ICLR 2025
Edouard Leurent. An environment for autonomous driving decision- making. httpkiin.httttt////////// htttps///gthhu: com/ http///eeeeeiggaaaeaeeeeeeeeeeee
2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. Journal of Machine Learning Research, 17(39):1-40, 2016.
Xuefeng Liu, Takuma Yoneda, Rick Stevens, Matthew Walter, and Yuxin Chen. Blending imitation and reinforcement learning for robust policy improvement. In The Twelfth International Confer- ence on Learning Representations, 2024. URL httpp::/ net/ wwwp:nnrteeoottteteeee forum?id=
eJCdzPJq1F
Jianlan Luo, Perry Dong, Yuexiang Zhai, Yi Ma, and Sergey Levine. RLIF: Interactive imitation learning as reinforcement learning. The Twelfth International Conference on Learning Repre-
sentations, 2024. URL httplhtttp//ppeeeeeeeeee httpt///..peeaeeeeeeeeee///oodad
net/ E
Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
R. Maclin and W. Shavlik. Creating advice-taking reinforcement learners. Machine Learning, 22:
251-282, 1996.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature. 518(7540):52--333 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292-6299. IEEE, 2018.
Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning. pp. 663-670, 2000.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35; 27730-27744, 2022.
Zhenghao Mark Peng, Wenjie Mo, Chenda Duan, Quanyi Li, and Bolei Zhou. Learning from active human involvement through proxy value propagation. Advances in neural information processing systems, 36, 2023.
Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part 11 14, pp. 549-564. Springer, 2014.
D. Pomerlcau. Alvinn: An autonomous land vehicle in a neural network. In NIPS, 1989.
J. Randlov and P. Alstrom. Learning to drive a bicycle using reinforcement learning and shaping. In ICML, (1998)., 1998.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil:
Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2019.
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction no-regret online learning. Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Laud- erdale, FL, USA, 11-13 Apr 2011. PMLR.
Stuart Russell. Learning agents for uncertain environments. In COLT, 1998.
Stuart Russell and Peter Norvig. Artificial intelligence: modern approach. Pearson, 2016.
13

---

Published as a conference paper at ICLR 2025
William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans. Trial without enror: To- wards safe reinforcement learning via human intervention. In Proceedings of the 17th Interna- tional Conference on Autonomous Agents and MultiAgent Systems, pp. 2067-2069, 2018.
Schulman, S. Levine, P. Abbeel, M.I. Jordan, and P. Moritz. Trust region policy optimization. In ICML, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 .0300,, 2017.
Idan Shenfeld, Zhang- Beenn-WWii Hong, Aviv Tamar, and Pulkit Agrawal. TGRL: guided reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31077-31093. PMLR, 23-29 Jul 2023. URL htttps://procestt saateeddinns.m...... mlr. p w202/shenfeld23a.c
An algorithm for teacher
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):554335, 2017.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-944 01140-9977,,, 2022.
Jonathan Spencer, Sanjihan Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Siddhartha Srinivasa. Learning from interventions: Human-robot interaction as both explicit and implicit feedback. In 16th Robotics: Science and Systems, RSS 2020. MIT Press Journals, 2020.
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and Andrew Bagnell. Deeply ag- grevated: Differentiable imitation learning for sequential prediction. In International conference machine learning, pp. 3309-3318 18. PMLR, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When use parametric models in rein- forcement learning? Advances in Neural Information Processing Systems, 32, 2019.
Yifu Yuan, HAO Jianye, Yi Ma, Zibin Dong. Hebin Liang, Jinyi Liu, Zhixin Feng, Kai Zhao, and YAN ZHENG. Uni-rlhf: Universal platform and benchmark suite for reinforcement learning with diverse human feedback. In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2025
A MORE EXPERIMENTAL RESULTS AND PLOTS
A.1
ATART
A.1.1 LEARNING CURVES FOR TABLE INCLUDING PONG, FREEWAY, AND ENDURO
Figure Ishow plots comparing ICoPro with various baselines, and Figure 3 show ablation study for ICoPro. Lines represent the average episodic return in terms of the raw reward. Shadows represent the standard deviation. For Rainbow, the x-axis should read in terms of steps instead of iterations. For other methods, the x-axis can be read in terms of steps or iterations.
Larger N2 (More feedback)
Smaller N2 (Less feedback)
Larger N₂ (More feedback)
Smaller Ng (Less feedback)
Sanguet
200
-50
Dlt BullbiCome Mayrrt
Haro-Ro-221
Herrs (% (43))
Bessiion Steperncit
Allan 33,550
CoN Exodurs (fg-500)
Elstaro (44
Molfhecmen 75,-31
1300
A0 suntiee 2000
40
Trenway 2202
Plano (14-223
Parno (56-55)
100
1.5
75
153
Figure 4: Compare baseline methods on Atari in terms of the averaged episode return measured with the raw reward AA- The shadow indicates the standard deviation over seeds. N₂, in titles refer to the number of queries per iteration, and the larger (resp. smaller) ones correspond to the large (resp. small) DL in Table 2
A.1.2 COMPARE ICOPRO WITH MORE RELATED WORKS.
Compare with Ihan et al. (2021)). In Table we compare the performance of ICoPro with Actionlmitation (Ihan et al. 2021b), which is method that also uses action advice but is augment
15

---

Published as a conference paper at ICLR 2025
Larger No (More feedback)
Smaller No (Less feedback) Deargue
Larger No (More feedback)
Smaller N₂ (Less feedback) Boding (N
(2223)
Ocaing 1M 220
120)
Boddng (N(e5)
Plano Wi-
(K,)23)
83 IncetionS Froomity (M-121)
20
200
Frewway 13.51
Fling (3L-223)
Coll
Figure 5: Ablation study on Atari to test the effect of ICoPro's 's two-phase scheme and target pseudo label. Ng in titles refer to the number of queries per iteration, and the larger (resp. smaller) ones correspond to the large (resp. small) D.. in Table 2
the replay buffer, on their experimental environments. Results for Actionlmitation come from their paper. Our simulated labelers in the three games achieved the same score as theirs (full score on Pong and Freeway, and 1200 score on Enduro), we put ICoPro's results from Figure A directly to compare with. Our method uses significantly less data than the Actionlmitation method.
Table 3: Compare the number of feedback actions ((Nioh)) ) and environmental interaction timesteps (Tewe) needed to reach the labelers' score in each environment.
Niab Freeway
T Tenv Freeway
Method
Pong
Enduro
Pong
Enduro
ActionImitation
10K
10K
10K
3M
3M
3M
ICoPro
375
2K
8k
375K
500K
2M
16

---

Published as a conference paper at ICLR 2025
taato Pefferinn
Carra
Lobdle
Palay
Coffe
Lebdle
Badiy R2-201
Rix 14
Zce (2,)20)
Suargeet
Seccunt Wy (00
Sungustt 15,-51
1400
1200
1200
Pang (Mc231)
18
110
(a) Di ffkand correspond Figure
3b
(b) Di ffiiabler correspond to Figure
Figure 6: Detailed plots with large/small budget size for the averaged plots in Figure 3b although the plots for Pong are not shown in the main text due to the space limit. Larger (resp. smaller) N₂ indicates the large (resp. small) budget D
A.1.3 DETAILED RESULTS FOR DIFFRAND AND DIFFLABLER WITH DIFFERENT LABEL BUDGET SIZE
Figure aare separated plots using different size of label busget in Figures 3b and 3c can overcome the two kinds of imperfections from the labeler (i.e., checkpoint) in most cases. A larger label budget tends to lead performance with less variance in both of the two settings.
Overall, ICoPro
For DiffRand, which involves replacing part of the same labeler's labels with different ratios of random actions, larger number of imperfect labels does not necessarily result in worse perfor- mance. This suggests ICoPro's 's capability to effectively utilize those good labels and overcome the effects of bad labels with proxy rewards, as illustrated in Figure 6aa
For Di Difflabler, which employs labelers of varying performance levels, we have three main ob- servations from the results shown Figure 6b (1) For ICoPro, a worse labeler does necessarily decrease its performance. For example, in Figure 6b) the two labelers with performance ranks 0 and (in green and nacarat) in Boxing-N, 20, ICoPro labeled by them exhibit similar performances, de-
5, or the two labelers with performance ranks 2 and 3 (in
purple and pink) in Seaquest-NN, spite the obvious performance gap between these two labelers. (2) Labelers with lower performance indeed tend to lead to worse performances of the trained agent from ICoPro, which is also widely noted by related works that involve human feedback (c.g.. PVP (Peng et al. 2023), RLIF (Luo al., 2024)) (3) ICoPro demonstrates a strong capability to overcome the non-optimality of the labeler in most cases: For example, in Pong, even though the worst labeler only achieves scores around -15, ICoPro trained with this labeler
still achieve scores larger than +10. But if the labeler's performance bad, a larger number of "corrective" " actions (i.e., experiments using larger N may lead to poorer performance of ICoPro.
17

---

Published as a conference paper at ICLR 2025
A.2 HIGHWAY
A.2.1 EXTRA RESULTS FOR ABLATION STUDY
Figure 7 shows the ablation study highway. For such environment with relatively small num- ber of action dimensions, ICoPro can outperform HGDAgger-L significantly but only outperform other ablations slightly. Similar performance also be observed in Atari games with small action dimension like Pong and Freeway in Figure 5
ICoPro
DOID-
HGDAggarL
WD
wis Finetune
Labeler
1.00
Dietance- Avg
1000
Ncreah
SLannhhugge
1233
0.2
12
0.25
0.1
400
0.00
0.0
60
120
100
30
80
130
30
50
120
160
30
60
00
120
160
(x10K)
Step (x10
Step (0K)
Step (x10K)
Figure 7: Extra ablation results on highway with the labeler used in Figure averaged over the set of proxy rewards mentioned Table
Performances are
A.2.2 EXTRA EVALUATION WITH ANOTHER SCRIPTED LABELER (DIFFLABLER)
In Table we compare the two simulated labelers used in our experiments. The first labeler (Labeler-CL, CL), which is the labeler illustrated in our main text, prefers driving faster and changing to different lanes more often than the second one, but with a larger crash rate well. The second one (Labeler-RI.) drives in a safer way at lower speed and prefers to drive lanes on the right. Then, in Table we list the 2 sets of proxy rewards for the two labelers.
Figure shows experimental results with Labeler-RL, which confirm our analysis in Section 5.2 again that ICoPro can align to the desired performance better than baselines.
Table 4: Performance metric for the 2 simulated labelers in highway.
Labeler
&cCrahh
Step-Avg
Step-Min
Distance-Avg
Distance-Min Speed-Avg LancPosition-Arg LLaneChange-AAg
Labeler-CL. 0.10
48.51=00.39 20 80
1163.689+12.85 1099.38.19.02
183.09-227.88 23.97=0.06
0,57=00.5 0.91
0.18=:0.22 0.04110.1
Labeler-RL 0.02
49.29.10.4
28.95119.92
608.931404.97 22.30.0.12
Table 5: Reward weights for engineered proxy rewards highway for the two labelers mentioned in Table
Labeler-CI.
Proxy Rewards
PRExp
PRI
PR2
PR3
PR4
Change lane action
0.2
0.2
0
Normalized lane index
0
0
0
Evnets
High speed
1.5
0.8
0
Low speed Crash
-0.5
-1
0
0
-1.7
-1
-1
-1
Normalize
[-1.1]
[-11]]
[-1.1]
[-1.1]
None
Labeler-RL
Proxy Rewards
PRExp
PRI
PR2
PR3
PR4
Change lane action
0.2
0
Normalized lane index
0.5
0.5
0.2
Evnets
High speed
1.7
1.5
0.8
Low speed
-0.5
-0.5
Crash
-1.9
-1.5
-1
Normalize
[-1.1]
[-1.1]
[-1.]]
[-1.1]
None
18

---

Published as a conference paper at ICLR 2025
Prosy
PRF*p
Methods
ICoPro
Rainbow
Labeler
1000
-AVg
Avg
0.8
No
0.50
800
144
0.20
600
13
0.6
400
80
03
50
120
150
30
03
80
120
100
60
30
120
150
120
150
Step (X10K)
Step (x10K)
Step (X10K)
Step (x1010
(a) Compare agent's performance learned from ICoPro and Rainhow.
Methods
ICoPro
DQID-
HGDAgger-L
Reinbow
PVP-RR
PVP4
BCL
Laboler
1200
1000
0.6
Streih
800
24
Poslion-
Distance-
0.8
800
04
23
62
200
18
30
60
90
120
150
30
80
50
120
150
30
60
90
120
150
0
30
80
80
120
150
Step (X10K)
Step (x10K)
Step (x100K
Step (X101)
(b) Compare ICoProwith other baselines.
The learning curves are averaged over the set proxy rewards
mentioned in T
Figure 8: Experiments on highway with Labeler-RL using the set of proxy rewards mentioned in 3 Each plot compares the performance with respect to one representative performance metric. Compared with Figure 2 the different performance metric LanePosition-Av in the 4th subplot. |D4|= =1.5K at the end.
A.2.3 EXTRA RESULTS FOR DIFFRAND
In Figure we show the whole detailed plots for the averaged ones mentioned in Figure 3a ICo- Pro overcome such non-optimality inside the non-optimal corrective actions no matter which proxy reward is applied.
B IMPLEMENTATION DETAILS FOR ICoPRO
B.1 PSEUDO-CODE FOR ICoPRO
Algorithm show the concrete learning procedure ICoPro.
B.2 SYPER-PARAMETERR
The basic hyper-parameters for ICoPro in all environments are listed in Table 6
In Table 7 we list hyper-parameters with slight differences in different environments. The reasons that we do use same configuration is:
Choices of in Equation 3 are the same training the Rainbow labeler. We find that in Boxing and Enduro we can not obtain meaningful checkpoints with the default value 20 and therefore using smaller values.
E defaults to 2, except for the 3 hard Atari games with 18 actions. We observed that setting ICoPro and DQfD-I reach labelers' performance, but setting E 1 performs better.
2 will be hard
Since Enduro's episode length can reach 25K when the performance approach to our la- belers'. this case, set H as the default 5K can introduce bias in our iterative feedback schedule. Therefore we set H = 20K in this game.
Moreover, we compare the design elements between ICoPro and the original Rainbow's (Hessel al. 2018) in Table
19

---

Published as a conference paper at ICLR 2025
Ratoo handon
Deteuas
255
Islky
Katro
Labeter
1200
0.0
AAS
1.00
28
SLaneChunge Ave
Noreh
0.8
800
Speed
600
0.2
0.2
403
200
60
150
30
60
90
120
150
50
60
50
120
1.0
90
80
80
120
150
(x100K)
Step (x10K)
Step
1x1)
Step
(x10K)
(a) PRExp: F FCrass)) = - (17.pe))i)Sppeee) = 15, F(LowSpeed)
0.55, (ChamgeLane) = 0.2, Norm-[-1.A
1200
8
0.8
Distance Aug
1000
Speed Avg
0.4
0.6
800
-4
% SLareGhange
800
0.2
0.2
400
200
62
80
150
[X
Step (x10K)
Step (=10K)
(b) PR1: PC1:a(Crash)
( F(iighSee)))) FLooSSpeed) (Changgelam) Norm-[-1.1]
1.0
1200
0.5
NCrah
0.8
Distance AAgg
1000
800
Spondd Augg
MLameChange
0.3
600
0.2
0.2
400
60
[X10K)
Step (x10K)
Step [x10KK
(c) PRZ: + (Crash)
((ig 0.8, (LowSSped)) 0, (Changelane)) 0.2, Norm-|-1,1]
1200
SLansChunge Avg
0.8
Distance A0g
1000
0.8
800
800
400
0.0
30
60
50
120
150
50
60
90
120
150
30
50
120
100
30
50
120
150
Step Ix 10K)
Step
(X10K)
Step (x10K)
Step (x10K)
(d) (Crush)
(HighSpeedd 0, (LowSpeed) 0, F(Changelane)
0, Norm-[-1.)
0.6
26
Avg
0.8
Speed Avg
0.4
600
02
22
0.2
400
0.0
200
0.0
30
Step [X10K]
Step (X10K))
(e) PR4: FCrras)) ((Higheee)) = 0, F(LowSpeed) = 0, FCChangeLame) = 0, Norm-None
Figure 9: Detailed performance for each proxy rewards that used to show the average performance in Figure 3a
B.3 EXPERIMENTS COMPUTE RESOURCES
For experiments in Atari, we only need one GPU card to launch experiments, including GPUs GeForce RTX 3060 12G GPU + 48GB memory + Intel Core 17-10700F, GeForce RTX 3060 12G GPU 64GB memory + Intel Core i7-12700, GeForce RTX 2070 SUPER 32GB memory + In- tel Core i7-9700. GeForce RTX 2060 + 64GB memory + Intel Core i7-8700. For experiments in highway, CPUs.
20

---

Published as a conference paper at ICLR 2025
Algorithm
ICoPro
Require: Initial policy of with randomly initialized Qf, oracle policy x² with Q4 Require: # training iteration Nitr, # rollout steps H per iteration, # queries per iteration N₂
Require: finetune-phant phase: Target Accuracy dace
Require: In proppagttion-phase # update epochs E, data from the recent / iterations
Require: D D +
Nsr do 2: Collect rollout D.
1: for
{(8), F()) with of = Ge (arg m Q ().)))) {qx 1 from Dine
3: Sample segments D.
4: for each q E Di do
b Data Collection-phase
5:
Give corrective actions D. = {(}) d NNr Ner D' DD D.D.
6:
7:
end for
8: Reset optimizer's 1st and 2nd order moment
9:
while E,-DE [I la = arg max, (s. < 6 do Update O with L (Equation (1) minibatch from D end while
> Finetune-
10:
11:
12:
Reset optimizer's 1st and 2nd order moment
13:
for j + 1 E do
Propagation-phan
14:
Q Q+
15:
for minibatchs Dene in Der [i-T.i] do on DenU
16:
Calculate LRL, L'n and LM
Calculate LAG in DL D4, IDL Dene
18:
Update Q? with Prop (Equation 55)
19:
end for
end for
21:
Q4+1 + Q
end for
23:
return 2%
Table 6: Default hyper-parameters.
Syyper-arammtter
Value
training batch size B
128
General
margin C'
0.05
Architecture for Neural Network same with DERainbow Van Hasselt et al. 2019
type
Adam(Diederts 2014
learning rate
0.0001
Optimizer
eps
0.01/B
betas
(0.9, 0.999)
Ner
1
0.01
DataCollection-plass
T
Atari: 25, highway: 10
query sampling
uniformly sample segments without overlap
Finetune-phe thees
Accuracy target 8
0.98
discount factor
0.99
Propagation-phase
weight for Crom: weight for LMG, w
0.5
0.5
21

---

Published as a conference paper at ICLR 2025
Table 7: Specific configurations and hyper-parameters for each environment. Checkpoint-step is the timestep to obtain our simulated labelers (i.e., policy checkpoints). Meanings for other config- urations are consistent with Algorithm| 1 The total environmental steps equals to Niwr H, [D
equals to N x Ng x NCF, [D 1-1,1 7, equals to I x H.
Configuration
Atari-Giames
Highway-1 abbeles
Seaquest
Boxing
Battlezone Frostbite
Alica
Haro
MaPacman Freeway
Pong
Enduro
Labala-CC Labele-RL
Checkpoins-step
4.6M
10M
4.6M
4.6M
5M
5M
4M
1.8M
4.8M
330K
990K
Step
10
20
E
1
2
1
1
2
2
2
2
2
64
64
64
64
100
SK
5K
5K
[pp
32K
32K
32K
32K
32K
32K
32K
32K
32K
32K
100K
Nos
Small N, (DF))
(2K)
(500)
(2K)
5 (2K)
5 (2K)
20 (8K)
5 (2K)
5 (750)
5 (750)
(8K)
Large N, (IDE)
(8K)
(2K)
(8K)
20 (8K)
20 (8K)
(16K)
(8K)
(8K)
(3K)
160 (16K)
Nrer
400
100
400
400
400
400
400
150
150
100
150
Total Row
500k
2M
C MORE DISCUSSIONS ABOUT RELATED WORKS AND BASELINES
C.1 MAIN BASELINES AND ABLATIONS
BC-L. For BC, we ran seed for the replay buffer obtained at the end of training of ICoPro. therefore the performance is averaged over 5 different label buffers. In Atari, we select the best result during training with respect during training the accuracy to reach dasc 0.999. In highway, since we have multiple performance metrics and it's not easy to measure which one is the best, we present the performance when it reaches our default dace 0.98.
Rainbow. We compare ICoPro with Rainbow in Table g See Table 12 for hyper-parameters of Rainbow. We the same setting with data-efficient version of Rainbow, except for the step and the training timesteps (see Table(7)
Table 8: Compare the different component between ICoPro and Rainbow (Hessel et al. 2018
Components Replay
Method
I-step
N-step
Duel
Double Q
Distrib
Exploration
ICoPro
Yes
Yes
Yes
No
Uniform
Yes
Feedback guided
Rainbow
Yes
Yes
Yes
Yes
Prioritized
Yes
Noisy-net
Compare design choices with baselines and ablations. DQID (Hester et al. 2018) is a method that learns from human demonstrations, and PVP (Peng al. (023) learns from human interven- tional control. Both of the two methods are designed by augmenting the standard RL loss with their extra IL loss into normal RL methods like DQN (Mnih et al., 2015) and TD3 (Fujimoto al. 2018). We compare the learning scheme and corresponding losses for each baselines and ablation settings Table other hyper-parameter keep the same with Table have a fair comparison.
C.2 OTHER POTENTIAL BASELINES
C.2.1 RLIF
Besides baselines mentioned in the main text, RLIF (Luo et al.) 2024) is another potential baseline for comparison. However, adapting RLIF's design to learn from offline corrective actions instead of the original intervention feedback leads to unsatisfactory performance. Therefore, it is not consid- ered primary baseline for comparison, and the results are only included in this appendix.
Recall RLIF's method design. RLIF is a framework that lets humans oversee the training trajec- tories in real time, intervene in unsatisfactory states and take over control from unsatisfactory states to satisfactory ones. RLIF learns from these interventions by labeling the replay buffer with rewards of -1 for intervened states and otherwise.
22

---

Published as a conference paper at ICLR 2025
Table Compare the design choices between ICoPro, different basclines, and extra ablation groups. Backslashes mean there is no such phase.
Methods
ICoPro
DQFD-I
Finetune
LMG 0.5 L/O 0.5
Phases
Propagation
TGT
L + CRL
C/ C C
Methods
PVP-IR
PVP-
Finetune
Phases
Propagation
CLVP CRL + LRL
CLVP 1(F=0) + m(r=0)
Methods
DAgger
w/o Finetune
Finetume
LNG
Phases
Propagation
0.5- LNG 0.5 TGT CRL + CRL
Methods
w/o TGT
RLIF-1
Finetune Propagation
CMG
Phases
CNC + C CR
CRI
Methods
w/o Prop's a
w/o Prop's MG
Finetune
LMG
LMG
Phases
Propagation
CMG TTT
+ C - CRL
CRC CRL
Experimental setups to compare with RLIE. Although the feedback type used in RLIF is not the same as our corrective actions, we adapt their core idea (i.e., rewarding -1 for intervened states and 0 otherwise) into framework, named RLIF-I, to have fair comparison with ICoPro and other baselines. RLIF-I treats offline corrected states as RLIF's online intervened states. Concretely, RLIF-I implemented by modifying ICoPro in three ways: (1) Disable the Pinezune-phase in ICoPro, (2) Remove the margin loss used in ICoPro's Propagat inn-phase, and (3) Replacing the proxy rewards used in ICoPro's RL losses with (RLIF(s, a) -Le((s, a)}. where a is the action taken by the training agent, Te[(s, a)} if (s. a) has been corrected (i.e., intervened) by the labeler and otherwise. We showease this setting into Table yas well to provide a more clear comparison with others.
Experimental results and analysis. As shown in Figure 10 under the same setting as ICoPro, RLIF's learning strategy can not learn anything. This result reasonable since the offline cor- rective actions setting in ICoPro is quite sparse compared with RLIF's: note that there are up to 0.8% transitions being labeled under ICoPro's labeling schedule, which can be calculated with I/T oot EneSteps in Table 7 while RLIF requires to intervene all unsatisfied states. Such sults indicate that making RLIF work requires intervening in many more states than ICoPro, which is too label-intensing to be scaled for more complex and practical tasks.
C.3 OTHER POTENTIAL ABLATIONS
C.3.1 EXTRA ABLATIONS ON MARGIN LOSSES IN THE PwwpaaatiippopPP......SP
For the margin losses with at (Equation and a TGT (Equation used in our propagation- phase, we mainly focus on the ablation that removing Equation with a TCT but keeping Equa- tion with " in our main text. As discussed in Section removing Equation ( from the ioopagation-hhaee
is problematic since the corrective actions in D4 are not executed in Denv, that is, RI. losses (Equations (2) and (3) are optimized with off-policy trajectories and the newly provided corrective actions a from the labeler are never taken in those trajectories.
To make the ablation study more thorough, we show two extra ablations related to the margin loss with at used in the propagate wwwragaatiinioo tinn-pass: (I) removing at but keeping a (denoted as w/o Prop's 's a,), or (2) removing both " and a" TGT (denoted as w/o Prop's MG). Table compares the two ablation settings with other experimental ablations baselines. As shown in Figure IT] both the three ablations perform worse than ICoPro, indicating the necessity of keeping the margin loss with both a and a TGT in the propagat inteeattooo-hhss.se -hase.
23

---

Published as a conference paper at ICLR 2025
Prasy
PRExp
PR1
MR2
PR3
1984
-T,(55 4))
Methods
ICoPro
Labeler
1.25
0.5
100
1083
SLaneChangee Avg
08
800
0.4
600
0.2
400
22
0.0
200
0.2
60
150
150
60
Itoration (8tspex10K)
Iteration (Stage
Neration (Stopex 101)
(Stegmm:
(a) Highway with the same setting shown in the main text (e.g., Figure 2a
Boong
Sempuaes (1,-28)
W
woll
136
(b) Two Atari games (Boxing and Seaquest) with the same setting shown in the main text (e.g. Table
Figure 10: Compare RLIF-I with ICoPro under the same setting (e.g., simulated labeler, feedback schedule and budget).
Methods
ICoPro
wib 8
wto Prop's MG
wto Propls
Labeler
1200
1.00
SL Change----eee Aug
Distanoe .Ag
1000
0.75
800
0.2
500
22
0.25
400
160
30
1500
100
120
30
60
90
120
150
Heration 10K)
Heeation (Stepexc 10K)
Ieration (Stepax 10K)
Beraaion (Stepex100
1010
(a) Highway with the same setting as shown in the main text (e.g., Figure
2a
Pooror Lorke
43
83
130
Emeatere (009
100
200 30
200 Teraatonn (Pappen10))
(b) Two Atari games (Boxing and Seaquest) with the same setting as shown in the main text (e.g.. Table
Figure 1I: Ablations with respect to the margin losses with at and TGT in the propagation- phase. Compared with the default setting of ICoPro's propagation propagation-phaas ion-phase, w/o a TGT refers to removing a TGT but keeping ", w/o Prop's refers to removing a" but keeping a TGT w/o Prop's MG refers removing both a and TGT Note that of our main ablation settings, w/o a TGT has been tested besides
D MORE DETAILS ABOUT EXPERIMENTAL SETTINGS
D.I CONFIGURATIONS FOR EVALUATION
We use 5 seeds for ICoPro and baselines for evaluation, and 3 seeds for ablations. For all experiments we perform, we use 50 episodes in each evaluation. When we ran experiments for ICoPro and baselines, we used 5 seeds for each setting. For ablation studies, use seeds. Although ICoPro
24

---

Published as a conference paper at ICLR 2025
and its' ablation w/o Finetune involve separate phase in each iteration, their performance are still evaluated at the end of each iteration, which is the propagation-phan
D.2 HIGHWAY
Highway Leurent 2018) is an environment with state-based inputs in 35 dimensions and available action choices to control the vehicle's 's speed and direction. The input states contain information about both the controlled vehicles and the top nearest vehicles' positions, speeds, and directions. The basic goal in this environment is to drive car a straight road with multiple lanes within limited time. One episode is terminated when the time used up, or a crash happens.
We use the configuration listed in Table TO in highway.
Table 10: Hyper-parameter for environment in highway.
Saree--arameter
Value
Available speed range
[19,30]
High speed range
221
Low speed range
<21
Env
Lanes count
5
Vehicles count
40
Time limit
50
Policy frequency
Type
Kinematics matrix
Observation
Number of observed vehicles
5
Features for each vehicle
[presence, x, y. Mr. Vy, COSkrading sitheadsigg
Action
Type
[Right, Left, Faster, Slower, IDLE)
D.3 ATARI
In Atari, we use the signed raw reward from the environment as the proxy reward in our setting: FA sign(rA), where r' is the raw reward in Atari and FA is the proxy reward. The average cumulative A of episodes serves as a performance metric.
Both F and F. have different levels of imperfection. The imperfection of FA mainly comes from two aspects: (1) Missing rewards, c.g. in Seaquest, TA (LoooseLivees) = 0, but give a negative reward better; (2) Numerical mismatch between FΛ and r', e.g., in Hero, P (RescueMinee Miners) 1K, TA (ShootCritters) = 50 but A = 1 for both the two cases. More information about the ground- truth A can be checked in the official website
Our environment configurations follow standard requirements Machado et al. (2018) Table the related details.
lists
Table 11: Hyper-paramette parameters for the Atari environment wrapper.
Hyper-pun Sarerr-arrmettr
Value
Grey-scaling
True
Observation down-sampling
(84, 84)
Frame stacked
Frame skipped
Action repetitions
Max start no ops
30
Reward clipping
[-1, 1]
Terminal on loss of life
True
Max frames per episode
108K
25

---

Published as a conference paper at ICLR 2025
D.4 SIMULATED LABELERS
We use Rainbow to train labelers for all environments, with the general hyper-parameters shown in Table T2 and concrete training timesteps Table 7 For highway, script labelers are trained with PRExp mentioned in Tables and [ For Atari, scnpt labelers are trained with the signed raw rewards. Note that these script labelers are not necessarily perfect ones since their performance could be improved further with longer training.
Table 12: Hyper-parametert for Rainbow.
Hyper-parameter
Value
Update period target network
per 2000 updates
Atoms of distribution
0.99
Batch size
32
Optimizer
Same with Table
Max gradient norm
10
Prioritized replay
exponent: 0.5, correction: 0.4-+1
Noisy nets parameter
0.1
Warm-up steps
1.6K
Replay bufler size
1M
step
Same with Table
Sample steps update
CNN encoder: channels [32. 64), kernel size strides [5, 5). paddings (3,
Netowrk-Atari
MLP policy: hidden size [128]
MLP encoder: hidden size [128, 128]
Netowrk-Highway
MI.P policy: hidden size [128]
E MORE DETAILS ABOUT USER STUDY
Appendix E. and Appendix F.2 demonstrate the detailed evaluation results of ICoPro-Human. Then Appendix
shows some examples of the corrective actions provided by humans. Finally, Ap- pendix E.4 gives an introduction of tasks and corresponding user interface.
E.1
VISUALIZATION FOR AGENTS TRAINED WITH ICOPRO-HUMAN
we visualize an action segment from Pong to validate the human-like performance trained by (CoPro-Human comparing with the scripted labeler.
Figure 12
Scripted
Labeler
iCoPro [Human)
LEFTFIRE
LEFTFIRE
LEFTFIRE
LEFTFIRE
LEFTFIRE
Figure 12: Performances of the scripted labeler and /CoPro-Human in Pong. Each shows sequence state-action pairs.
Other evaluation videos for ICoPro-Human are sorted in Suppl 1 ComentaryMatent / [Env] UserStudy, which learns to perform more like human players compared with evaluation videos shown in Supplemeettaryatteriiee s/Vieos [Env] Oracles.
E.2 TRAINING PLOTS
Figure shows some statistics of CCoPro-Human during training.
For highway, we show the two related performance metrics (i.e., %Crash and Speed Aug) following the instruction described in Table 15
26

---

Published as a conference paper at ICLR 2025
For the two Atari games, only show their game scores. However, the intentions described in Table [3 are not entirely consistent with the game scores, as the human labeler aims to teach additional performance aspects that are not captured by the proxy rewards.
Highway 11))
Highway (1,)
Seaquest (M, +20)
Pong (NGad5)
Ww
Spaa Ave
REw Roward.
200
130
Reward
100
100
2
-20
10
20
25
50
100
Iberation (50opmx: 10K)
Heration [Stepex: 10K)
Iteration (5tepmxx9K)
Iteration (StopmxXK)
Figure 13: Performance of ICoPro-Human during training. Solid lines (resp. dashed lines) are the performance of ICoPro-Human (resp. BC-L), with shadows indicating the standard deviation measured over 50 episodes.
E.3 EXAMPLES HUMAN CORRECTIVE ACTIONS
In Figure 14 we showcase some visualizations of the states that human labelers selected, the original agent actions, and the provided corrective actions. Contextual information (i.e., what happens before and after a state) is available to the human labelers (e.g.. with our simple interface described in Figure
15) to help them select state(s) and provide corrective action(s). For example:
In Pong (Figure 14a: : human labelers can judge an agent action is reasonable or not by knowing the movement of the ball.
In Seaquest: the left-most example of Figure 14b shows that human labelers deter- mine which state to correct by assessing whether the agent's actions will result in a fire hitting fish. The other three examples of F demonstratt that human labelers can effectively guide the agent by knowing the movement of fish and divers before a state.
In Highway: the left two examples of Figure 14c show that human labellers can provide corrective actions by knowing the movement of cars, while the right two examples show that human labelers can provide corrective actions by knowing the consequences (e.g., car crash in the two examples) of agent actions on those states.
20
Torm
LEFTTRIRE !! RIGHTFRE
NDOP LEFTTREE
(a) Pong. The agent controls the right paddle.
(b) Seaquest. The agent controls the yellow submarinc.
a: SLOWER
SLOWER
RIGHT
LANTLLEFET
SLOWER
(c) Highway. Cars are driving from left to right. The agent controls the green car.
Figure 14: Examples of human actions provided for ICoPro-Human. Each image indicates a state, while and a refer to the agent action and human action in that state, respectively. In Figure 14a we add yellow dashed arrows denote the movement direction of the ball avoid confusion. In Figure 14b we add yellow text and dashed arrows to denote the role of objects.
27

---

Published as a conference paper at ICLR 2025
E.4 PROCEDURE OF USER STUDY
No need to let humans play the game by themselves before providing feed- back. Before letting the human labelers give feedback, tion about the task they will participate (see Tables T3 and [5). as well as an example video to let them become familiar with the environment dynamics (see PATH/TO/
show them instruc-
/ /Video (EnvName Oracler-Exaaaeeaeeeeeeeeeeeeeeae
We show our simple interface used in our user study in Figure 13
Table 13: User instruction for Pong.
User Instructions Pong
Description:
You control the right paddle, you compete against the left paddle.
You each try to keep deflecting the ball away from your goal and into your oppo- nent's goal.
Reward:
player scores +1 when the opponent hits the ball out of bounds or misses hit, or -1 when you hits the ball out of bounds or misses hit.
The first player or team to score 21 points wins the game.
Actions:
Value
Meaning
Value
Meaning
Value
Meaning
NOOP
FIRE RIGHTFIRE
2
RIGHT
3
LEFT
4
5
LEFTFIRE
NOOP: random action
LEFT: move your paddle down
RIGHT: move your paddle up
FIRE: add some speed to the return ball or put sharper angles on your return hits when the ball contacts with your paddle
Other actions combined effects as described above.
28

---

Published as a conference paper at ICLR 2025
Table 14: User instruction for Seaquest.
User Instructions Seaquest
Description:
You control submarine to (1) save divers, and (2) shoot fish.
Pay attention the oxygen bar: you must float the surface before using up your oxygen, otherwise, you will lose a life.
Each time you surface, you must bring at least one diver with you, or you will lose life.
Get in touch with the divers then they will enter your submarine.
Pay attention to the fish: you shoot them, but if you touch them are hit by their fire, you will lose a life.
You have three lives in total.
Reward:
+1 for shooting fish and 0 otherwise
Actions:
NOOP: random action
LEFT / RIGHT UP DOWN: control the frontal orientation the submarine
FIRE: shoot a fire from the front of the submarine
Other actions (UPRIGHT UPLEFT / DOWNRIGHT / DOWNLEFT DOWNRIGHT UPFIRE / DOWNF IRE LEFTFIRE RIGHTF IRE UPRI CHTFFIRE / UPLEFTFIRE IRE DOWNRIGHTF IRE/ DOWNLEFTFIRE) combined effects described above.
Table 15: User instruction for highway.
User Instructions highway
Description:
You control the green vehicle. There are other blue vehicles around.
You need drive speed large as you can while avoiding crash.
You need to take other vehicles if possible.
Reward:
-1 for collision and 0 otherwise
Actions:
LANE_LEFT Change lane the left, effect if already in the leftmost lane.
LANE RINHT Change lane to the right, effect if already in the rightmost lane.
IDLE: No action, keep the current speed and heading direction.
FASTER; Faster.
SLOWER: Slower.
29

---

Published as a conference paper at ICLR 2025
Ned--10er interiee
1h pwort auged
STPL. Lemmon
12P1, Lemmerr
onn Hommar
STPOLRCHTINE
LOTP
LETPA
JOYA
LOTF
LTPN
JOPA
LOPR
LETRL
(a) User interface for Pong.
Mer Uaee Besdlint quiry soaat
Hhe pone
20rt Lengermes
Irann pran JaP parr Scmeron
2012 DCveeae
XPPLIP aP1
RAMF
RAP
PAP
Pus
Jowwlo
cin ma
Ci LLWNNIS TT WWW.NINIT MTw.
(b) User interface for Seaquest.
pendii
(c) User interface for highway.
Figure 15: User Interface. We put the video of that segment in the top left comer. If the labeler is satisfied with the whole segment's 's state-action pairs, they can choose to pass this segment with the radio box at the top near the video window. Otherwise the labeler will give corrective actions other windows.
30


Reviews:
[{"review_id": "5roeCgfBA0", "reviewer": "Reviewer_BJFG", "summary": "In this paper, a framework is proposed where a human labeler can offer additional feedback in the form of corrective actions, reflecting the labeler's action preferences, even though this feedback may also be imperfect. Within this context, a novel value-based deep RL algorithm, Iterative Learning from Corrective actions and Proxy rewards (ICoPro), is introduced. This algorithm follows a three-phase cycle: (1) Eliciting sparse corrective actions from a human labeler on the agent's demonstrated trajectories; (2) Integrating these corrective actions into the Q-function using a margin loss to ensure adherence to the labeler's preferences; (3) Training the agent with standard RL losses, augmented with a margin loss to learn from proxy rewards and propagate the Q-values acquired from human feedback.", "questions": "see above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper explores an intriguing imitation learning problem in which human demonstrations are imperfect, although some corrective actions may be available. Additionally, there is no explicit reward signal, but certain action preferences (raw reward) are provided.", "weaknesses": "1) The performance of the human feedback policy would significantly impact the total performance of the proposed method. For example, if the human feedback is from an expert policy, then the cloning from its behaviors is reasonable and helpful for improving the performance. But if the human feedback is from a sub-optimal or even a random policy, then its behaviors may be harmful. Therefore, what we concern is how the level of sub-optimality of the human feedback policy would impact the performance of the algorithm.\n\tCurrently, in Section 5.3 the authors have simulated the sub-optimal human feedback by inserting noisy actions, but we think such simulation is not enough for the explanation for this problem. Maybe the authors could utilize a half-trained Rainbow policy to generate the imperfect behaviors.\n\n\t2) The positions (states) of human feedbacks are also important for the efficiency and effectiveness. We observe that the authors selected the feedback positions with the method in RLIF [1], but we does not see the comparison and discussion with RLIF.\n\tBesides, to be fair, maybe this trick should also be utilized to other methods for comparison introduced in this paper.\n\n[1] RLIF: Interactive imitation learning as reinforcement learning.\n\n\t3) The working scenarios. In our opinion, this method could be seen as a method that online learning while cloning from a specific behavior policy (human feedback). So why such combination could solve the problem of sub-optimality of the proxy reward and the human feedback? For example, if the proxy reward and human feedback make the similar mistake, then such combination would not work. Clearly, in such scenarios, the proposed method could achieve the claimed performance.\n\n\t4) In this paper, the authors raised an interesting and inspiring problem. However, the problem lacks of mathematical formulation and theoretical evidence for its effectiveness.\n\n       5)  I found that the methodology adopted in this paper is closely related to those references cited in line 229 and the difference to each of those should be clearly stated. In addition, if the range of reward is not limited, then the loss of  eq.1 could be negative, as this expression is different from that of the traditional margin loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYxD4H9HeX", "reviewer": "Reviewer_CFrc", "summary": "This paper presents a new pipeline to learn from corrective actions and proxy rewards, which is interesting and useful for the practical use of RL algorithms. The pipeline of the algorithm is reasonable. The experiments show the effectiveness of the proposed methods.", "questions": "Could you please elaborate more practical examples that involving human labelers is necessary?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The general idea of the paper is interesting. Involving human labelers in the training process is a very promising way to effectively enhance the training process. The paper is easy to follow.", "weaknesses": "One major weakness is that the experiments are kind of weak to support learning from corrective actions. The paper only provides two environments that are involved by real human labeler (Pong and Highway). Most of the environments are using simulated labeler. More environments that show the necessity to involve human labels would be great to support the idea of the paper.\n\n\n\nThe writing of the paper could be improved. \n\n- The abstract could be further condensed. \n\n- Line 178~185: $\\mathcal{S}, \\mathcal{A}$ is not defined. The $Q$ function is not formally defined.\n\n- Line 182: Is $\\pi^*$ a typo? Do you refer $\\pi^*$ to an optimal policy, or greedy policy, or arbirtrary policy?\n\n- Line 353: You should use $a^L \\sim $ rather than $a^L=$ if the action is sampled from some $\\epsilon$-greedy distribution.\n\n  \n\nThe motivation for combining the two sources of signals is good, but the description in the paper (from Line 66~80) does not make sense to me. The paper stated 'In contrast, our key insight is that the two sources of signals can complement each other.. while the effects of suboptimal corrective actions may be weakened by proxy rewards'.  However, the proxy rewards could also be suboptimal, as it's only an approximation of the real reward function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RunF6nCMS", "reviewer": "Reviewer_QvY8", "summary": "This paper proposes to use proxy reward (the reward that is easily defined and might not accurately reflects the desired behavior) and the offline corrective action (labeller assigns correct actions to the rollouts generated by the agent, and those corrective actions will not be used to the environment as the labelling process is offline) to train policies.\n\nThe training can be split into 3 phases: data collection, finetune-phase, and propagation-phase. In the data collection phase, agent rolls out trajectories with epsilon-greedy exploration and human provide offline corrective actions. In finetune phase, marginal loss is applied to the Q values to make sure the Q values for corrective actions are high. In propagation phase, RL loss (TD loss with the proxy reward) and the marginal loss (the imitation loss that enforces the learned policy to be closed with previous agent policy) are used together.", "questions": "1. Why the marginal loss in propagation phase works? IIUC the \"w/o aTGT\" ablation is the experiment ablating the marginal loss in propagation correct? It seems that adding that term can help in terms of the final results. But more analysis is needed. For example, will adding the term improve the learning efficiency and reduce human interventions? In Sec 5.2 the ablation studies only show the performance in tasks but the other metrics such as \"human efficiency\" is not compared.\n2. How to ensure the label quality? In more complex task like driving, human labeler is hard to understand the outcome of their \"corrective actions\". For example to rescue a car from crashing left car, the human labeler might steer the car all to the right hand side. And this action though seems to be correct, but it's impossible to be applied in real world (you can't rotate the steering wheel immediately from left to right), and it will also cause unknown outcome such as car rollover. There is no guarantee on the quality of the data especially in the offline setting. In motivation the authors suggest this paper can work with suboptimal human labeler but **no experiment shows the method still work on suboptimal experts**.\n3. I will expect more discussion on human cost. What designs in the method can improve human efficiency (less corrective actions)? Line 259-261 says \"In addition, training with pseudo-labels can\nreduce the cost of collecting human labels by leveraging the large number of unlabeled states in\nDenv\" No experiment supports this claim and I will consider this to be a overclaim. Is there any evidence showing that the agent converge to same performance with less human (here the simulated expert) involvement?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Good to see the author provides source code.\n2. Human-in-the-loop with offline corrective feedback is a quite interesting setting. Previous work either focus on online demonstration (HGDagger, PVP etc) or offline labeling (PbRL, TAMER), but this work focus on offline corrective action (not the reward labeling but the action).\n3. Human-in-the-loop methods suffer from the cost of human subjects, as they need to be very attentive if in a online human-AI shared control setting. This paper adopts the offline setting and utilize the proxy reward, which is an interesting way to scale up the human-in-the-loop method.", "weaknesses": "1. Further analysis will greatly improve the technical depth of the paper. Further analysis on ablations and why the method work is needed. These include: why marginal loss in propagation phase works? How well the method work with human labeler in different optimality? The ablations study are presented in Section 5.2 and Table 2 while in Appendix A, while there is no discussion on the reasons. The author just put the results there. See more on \"Questions\".\n2. All ablation studies are conducted in the Atari, and further discussion is needed. Why ablations fail? What insights can we get from the ablation studies?\n3. As a paper studying human-in-the-loop policy learning, there is no enough weights on real-human experiments. Section 5.4 shows the experiment with real human subject. I will expecting far more information about this experiment. Why don't put the evaluation results of the trained policies in each iteration of the training and compare it against the baselines? Is there qualitative results showing the human's corrective actions and the agent's actions?\n4. Follow question 3, providing corrective actions is actually very boring and time-consuming. The setting might be even more exhausting than online human supervision. I doubt whether the method is applicable at all in real-human setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rLhU3gCaMI", "reviewer": "Reviewer_kqSo", "summary": "This paper introduces a novel method that integrates additional (imperfect) corrective actions into the reinforcement learning (RL) process to address limitations in the imperfect proxy reward function.", "questions": "1. What interface is used for the corrective action process?\n2. The entire approach is based on the Q-function, which is limited to discrete action spaces. Can this method be extended to continuous action spaces (without discretization)?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and easy to follow, and the experimental section provides sufficient evidence to support the authors' claim that their approach improves the learning of an optimal policy.", "weaknesses": "1. The reviewer is curious about the performance information available to the labeler that allows them to decide whether an action is good or bad, considering that the queries are sampled from a dataset rather than occurring in real-time. In the experiments, a trained Q-network is used as the labeler, which simplifies the collection of corrective actions. Although there is a user study involving human labelers in two environments (Highway and Pong), this does not fully address my concern.\n\n2. The term \"sample-efficient\" is mentioned frequently throughout the paper, yet no clear metrics are provided to quantify this efficiency. It would be helpful to include some metrics to validate this claim.\n\n3. Some relevant literature is missing from the discussion. For instance, ranking-based reinforcement learning [1]. The authors could also refer [2], which might offer more effective corrective actions than relying solely on the target Q-network, especially during the early stages when the model is undertrained.\n\nThere are a few minor issues with the writing. For example:\n\nIn the related work section, the discussion on the limitations of the offline setting is mentioned but not clearly explained. Providing more clarity here would be helpful. Table I includes some abbreviations, such as PR1 and PR2, which I assume stand for Proxy Reward. However, additional explanation would improve understanding for readers unfamiliar with these terms.\n   \nReference:\n[1] Brown, Daniel S., Wonjoon Goo, and Scott Niekum. \"Better-than-demonstrator imitation learning via automatically-ranked demonstrations.\" Conference on robot learning. PMLR, 2020.\n[2] Zhan, Huixin, Feng Tao, and Yongcan Cao. \"Human-guided robot behavior learning: A gan-assisted preference-based reinforcement learning approach.\" IEEE Robotics and Automation Letters 6.2 (2021): 3545-3552.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)