Reinforcement Learning from Imperfect Corrective Actions and Proxy Rewards

## Abstract

Abstract In practice, reinforcement learning (RL) agents are often trained with a possibly imperfect proxy reward function, which may lead to a human-agent alignment issue
(i.e., the learned policy either converges to non-optimal performance with low cumulative rewards, or achieves high cumulative rewards but in undesired manner).
To tackle this issue, we consider a framework where a human labeler can provide additional feedback in the form of corrective actions,
which expresses the labeler’s action preferences although this feedback may possibly be imperfect as well.
In this setting, to obtain a better-aligned policy guided by both learning signals, we propose a novel value-based deep RL algorithm called I terative learning from Co rrective actions and Pro xy rewards (ICoPro) 1 1 1 We have open-sourced its implementation: https://github.com/JiangZhaoh/RLHF_CF . , which cycles through three phases:
(1) Solicit sparse corrective actions from a human labeler on the agent’s demonstrated trajectories;
(2) Incorporate these corrective actions into the Q-function using a margin loss to enforce adherence to labeler’s preferences;
(3) Train the agent with standard RL losses regularized with a margin loss to learn from proxy rewards and propagate the Q-values learned from human feedback.
Moreover, another novel design in our approach is to integrate pseudo-labels from the target Q-network to reduce human labor and further stabilize training.
We experimentally validate our proposition on a variety of tasks (Atari games and autonomous driving on highway).
On the one hand, using proxy rewards with different levels of imperfection, our method can better align with human preferences and is more sample-efficient than baseline methods.
On the other hand, facing corrective actions with different types of imperfection, our method can overcome the non-optimality of this feedback thanks to the guidance from proxy rewards.

## 1 Introduction

While reinforcement learning (RL) has proved its effectiveness in numerous application domains (Mnih et al., ; Silver et al., ; Levine et al., ),
its impressive achievements are only possible if a high-quality reward signal for the RL agent to learn from is available.
In practice, correctly defining such reward signal is often very difficult (e.g., in autonomous driving).
If rewards are misspecified, the RL agent would generally learn behaviors that are unexpected (Amodei et al., ) and unwanted (Clark & Amodei, ; Russell & Norvig, ) by the system designer, notably due to overoptimization (Gao et al., ) or specification gaming (Randlov & Alstrom, ).
This important issue has been well-recognized in the research community (Amodei et al., ) and is an active research direction (Ouyang et al., ; Skalse et al., ).

Various solutions have been proposed to avoid having to define a reward function, for instance,
behavior cloning (Pomerleau, ; Bain & Sammut, ), inverse reinforcement learning (Ng & Russell, ; Russell, ), (inverse) reward design (Hadfield-Menell et al., ), or reinforcement learning from human feedback (Busa-Fekete et al., ; Christiano et al., ).
However, these approaches could be impractical and inefficient since they may require a perfect demonstrator, a reliable expert to provide correct labels, or assume that the agent only learns from human feedback, which would subsequently require too many (often hard-to-answer) queries for a human to consider.

As an alternative intermediate approach, we propose the following framework in which the RL agent has access to two sources of signals to learn from: proxy rewards and corrective actions.
A proxy reward function is an imperfect reward function, that approximately specifies the task to be learned.
A corrective action is provided by a (possibly unreliable) human labeler when s/he is queried by the RL agent:
a trajectory segment is shown to the labeler, who can then choose to correct an action performed by the agent by demonstrating another supposedly-good action.

This framework is practical and easily implementable.
Regarding proxy rewards, they are generally easy for system designers to provide.
For instance, they can
(1) learn proxy rewards from (possibly imperfect) demonstration,
(2) manually specify them to roughly express their intention (e.g., supposedly-good actions are rewarded and expected bad actions are penalized), or
(3) only define sparse rewards (e.g., positive reward for reaching a destination and penalty for a crash).
Regarding corrective actions, in contrast to typical demonstrations of whole trajectories, this feedback is usually much easier for the labeler to provide, since humans may not be able to complete a task themselves but can readily offer action preference on some states.

While only learning with one of those two sources of signals has its own limitations, by proposing our framework, we argue (and experimentally demonstrate) that
learning simultaneously from both of them, even though both may be imperfect, can be highly beneficial.
More specifically, on the one hand, solely learning from proxy rewards would either lead to very slow learning (e.g., when proxy rewards are well-aligned with ground-truth rewards but are very sparse) or yield a policy whose performance is not acceptable to the system designer (e.g., when proxy rewards are dense, but misspecified).
On the other hand, solely learning from corrective actions would require too many queries to the human labeler and may lead to suboptimal learned behaviors since the human may be suboptimal.
In contrast, our key insight is that the two sources of signals can complement each other.
Since they are generally imperfect in different state-space regions, bad decisions learned from proxy rewards can be corrected by the human labeler, while the effects of suboptimal corrective actions may be weakened by proxy rewards.
Therefore, learning simultaneously from the two imperfect signals can achieve better behaviors more aligned to the system designer’s goals than using any one of the two alone and can be more data efficient (in terms of both environmental transitions and human queries).

As a proof of concept, we design a novel value-based RL algorithm (see Figure 1), called Iterative learning from Corrective action and Proxy reward (ICoPro).
ICoPro alternates between three steps.
In the first step (${\tt datacollection}$-phase),
the RL agent interacts with the environment to collect transition data, and then the labeler provides corrective actions on them.
In the second step (${\tt finetune}$-phase), the RL agent learns to select actions according to this feedback via a margin loss, which can be interpreted as an imitation learning (IL) loss.
In the third step (${\tt propagation}$-phase), the RL agent is trained to maximize the expected cumulative proxy rewards while further enforcing a margin loss.
This latter loss is expressed on both observed labels (i.e., corrective actions given by the human labeler) and pseudo-labels generated by a trained model (target Q-network).
Imitating such pseudo-labels can be interpreted in two ways: either to reduce the number of queries or to stay close to the previously-learned policy, which can stabilize training like in trust region methods (Schulman et al., ; ; Asadi et al., ).
By combining imitation learning and reinforcement learning in this third phase, the agent can learn from both proxy rewards and enforce the temporal consistency of the values learned from imitation learning.

The contributions of this paper can be summarized as follows:
(1) We propose a practical human-in-the-loop reinforcement learning framework (Section 3) where the RL agent can learn from proxy rewards and (possibly suboptimal) corrective actions.
(2) To train the agent from these two imperfect sources, we design an efficient and robust algorithm (Section 4) in which we demonstrate a simple but useful technique to reduce human feedback.
(3) We experimentally validate our proof of concept (Section 5) by conducting an extensive number of experiments on various domains (e.g., image-based Atari games and state-based highway driving with sparse and dense proxy rewards) under different conditions (e.g., simulated or real human labeler).

Figure: Figure 1: ICoPro is an iterative method with three phases in each iteration. It starts with the data collection-phase to collect agent’s rollouts. Segments are then sampled from these rollouts and used as queries for the labeler to provide several corrective actions. Following this are two separate phases for policy updating, the finetune- and propagation-phase. Then the updated policy is utilized in the data collection-phase of the next iteration.
Refer to caption: x1.png

## 2 Related work

Given the difficulties in defining ground-truth rewards, a growing number of studies attempt to incorporate various types of human feedback to train agents, for example,
demonstrations (Pomerleau, ; Hester et al., ; Reddy et al., ),
preferences (Busa-Fekete et al., ; Christiano et al., ; Lee et al., ),
scalar evaluation (Knox, ; Knox & Stone, ; Saunders et al., ), action advising (Maclin & Shavlik, ; Da Silva et al., ; Ilhan et al., ),
interventions (Peng et al., ; Luo et al., ), example-states (Eysenbach et al., ),
or combination of multiple feedback types (Ibarz et al., ; Jeon et al., ; Yuan et al., ; Dong et al., ).
Among them, demonstrations, action advising, and interventions, which we discuss further below, are the most related to corrective actions.
We roughly organize these works in three clusters.
Preference signals can be provided in an *offline* or *online setting*.
In addition, we discuss related work that considers *combining reward and other preference signals*.

##### Offline setting

One of the earliest approaches to circumvent the need of defining a reward function is
imitation learning (aka learning from demonstration). In this setting where a batch of demonstration is assumed to be available, two main approaches have been developed:
behavior cloning (Pomerleau, ; Bain & Sammut, ) and inverse reinforcement learning (Ng & Russell, ; Russell, ), with recent instantiations (Ho & Ermon, ; Reddy et al., ) trying to overcome the limitations of these two approaches.
Most of these methods require demonstrations to be (near) optimal.
Since a batch of demonstrations is provided, they may also suffer from train-test distribution mismatch.
A typical approach to tackle this latter issue is to resort to the online setting.

##### Online setting

In contrast to the offline setting, here the agent learns with a human in the loop:
either a human actively oversees the agent’s training and can intervene, or the agent can query a human to obtain feedback to learn from.
The former case (Spencer et al., ; Peng et al., ; Luo et al., ) requires active participation and continuous attention from humans, while the latter may require too many queries, which can be too labor intensive for one or even multiple humans to provide.
Examples of the latter approach include methods in the DAgger family (Ross et al., ; Sun et al., ; Kelly et al., ) (with optimal action labeling), and reinforcement learning from human feedback (RLHF) methods (Busa-Fekete et al., ; Christiano et al., ; Lee et al., ) (with pairwise comparisons).

##### Combining reward and other signals

Since learning from one signal may be insufficient and/or inefficient, multiple works (Hester et al., ; Ibarz et al., ) consider learning from demonstration and rewards.
In such work, demonstration is often provided offline and used for initializing an online training phase, which can be standard RL training (Hester et al., ) or use other types of feedback such as in RLHF (Ibarz et al., ), which can notably help for better human-agent alignment.
In particular, demonstrations can be used to initialize a policy (Ibarz et al., ), define a reward signal (Reddy et al., ), or initialize a replay buffer for future RL training (Hester et al., ).
In addition, they can also be queried online (Chen et al., ; Peng et al., ; Luo et al., ).
Most works assume a well-specified (ground-truth) reward function, then focus on balancing the IL and RL objectives
when human demonstrations are also usually assumed to be optimal (Shenfeld et al., ; Liu et al., ), although some propositions have been made to deal with suboptimal demonstrations (Nair et al., ).

In contrast to previous work, we do not assume the availability of a ground-truth reward function but only an approximation of it (i.e., a proxy reward), which is easier to specify.
Since learning from this imperfect signal is insufficient, we complement it with corrective actions.
Providing corrective actions is less cognitively-demanding for the human than generating whole demonstrations, especially if the agent and the human do not operate in the same sensori-motor space.
Compared to action labeling (Ross et al., ; Kelly et al., ) or intervention (Peng et al., ; Luo et al., ), which have to happen on all unsatisfactory cases, our corrective action setting is less costly since the human chooses on which states s/he wants to give the feedback.
Finally, compared to most previous work, we do not assume that the human is optimal.
The goal of our paper is to demonstrate as a proof of concept that the agent can learn more efficiently to reach a better performance from two imperfect sources of signals than with any of the two alone.

## 3 Problem Formulation

We consider a reinforcement learning (RL) problem (Sutton & Barto, ) where an RL agent repeatedly interacts with an environment:
At a time step $t$, after observing an (observation) state $s_{t}\in{\mathcal{S}}$, it performs an action $a_{t}\in{\mathcal{A}}$, which yields an immediate reward $r_{t}\in\mathbb{R}$ and brings the agent to a new state
$s_{t+1}\sim{P}(\cdot|s_{t},a_{t})$ where ${P}$ is the environmental transition function.
The goal of the RL agent is to learn, from interaction tuples $(s_{t},a_{t},r_{t},s_{t+1})$, a policy $\pi^{*}:{\mathcal{S}}\to{\mathcal{A}}$ that maximizes the expected discounted sum of rewards.
Recall that this policy can be represented as the greedy policy with respect to a $Q$-function, $Q:{\mathcal{S}}\times{\mathcal{A}}\to\mathbb{R}$, $\pi^{*}(s)=\operatorname*{arg\,max}_{a}Q(s,a)$.

In contrast to standard RL, we do not assume that a *ground-truth* reward function (i.e., providing a perfect description of the task that the RL agent needs to learn) is available.
Instead, the agent only receives an approximation of it, called *proxy reward* function and denoted $\widetilde{r}$.
Indeed, while a precise reward function capturing all aspects of a desired behavior is hard to define for a system designer
providing a proxy reward function to roughly express the
system designer’s
objectives is much easier to achieve (Reddy et al., ; Jeon et al., ; Luo et al., ).
However, since the proxy rewards are approximate, purely relying on learning from them may never achieve the desired performance.
Therefore, we assume that the RL agent can also query a human labeler to obtain additional feedback under the form of *corrective actions* (see Figure 1).
Formally, a query corresponds to a $T$-step segment (i.e., sequence of state-action tuples) $q=\left(s_{t},a_{t},\dots,s_{t+T-1},a_{t+T-1}\right)$, which can for instance be sampled from trajectories generated from the interaction of the RL agent with its environment.
Given such query, the human labeler may select one state $s_{t^{\prime}}$ (such that $t\leq\!t^{\prime}\leq\!t+T-1$) where s/he provides a corrective action $a^{L}_{t^{\prime}}={\pi^{L}}(s_{t^{\prime}})$,
where ${\pi^{L}}$ is the labeler’s policy.
The corrective action is supposedly a better choice than the actually-performed one $a_{t^{\prime}}$.
Moreover, the labeler may choose to provide no feedback if all the actions in the segment are good.

Note that we do not assume that the labeler is necessarily optimal.
This means that it is not possible to solely rely on the corrective actions (without proxy rewards) to learn a policy to achieve the desired performance, which would furthermore be impractical in terms of label collection cost for the human labeler.
Instead, by combining the two imperfect sources of learning signals, the RL agent could possibly learn a policy
with a better performance and in a more sample-efficient way
than using proxy rewards or corrective actions alone.
Intuitively, the imperfections of the two signals probably do not lie in the same state-action space regions and therefore the two signals could correct each other.
We confirm this intuition in our experiments (see Sections 5.2 and 5.3).

## 4 Method

As a first work demonstrating the benefit of learning from the two imperfect sources of feedback,
we assume the action space ${\mathcal{A}}$ is finite for simplicity.
Building on the Rainbow algorithm (Hessel et al., ), we design ICoPro, an iterative approach to learn $Q^{\theta}$ with desired policy $\pi^{\theta}(s)=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}Q^{\theta}(s,\cdot)$.
At each iteration $i$, ICoPro updates the current ${Q}^{\theta_{i}}$ to obtain the next ${Q}^{\theta_{i+1}}$ according to the three phases: ${\tt DataCollection}$, ${\tt Finetune}$, and ${\tt Propagation}$, which we elaborate on next.
Figure 1 outlines the proposed framework and Algorithm 1 is provided in Section B.1.

##### Data Collection-phase.

Inspired by the popular RLHF setting (Christiano et al., ), we adopt
the following protocol for collecting two types of data: interaction tuples and corrective actions.
For the former, the agent acts in its environment according to the $\epsilon$-greedy policy with respect to ${Q}^{\theta_{i}}$, denoted $\mathcal{G}_{\epsilon}({Q}^{\theta_{i}})$, to collect a set
$\mathcal{D}^{env}_{i}$ of $H$ interaction tuples, $(s_{t},a_{t},\widetilde{r}_{t},s_{t+1})$, which are also stored in the transition replay buffer $\mathcal{D}^{env}$.
For the latter, the RL agent can regularly during RL training issue a batch of sampled queries from $\mathcal{D}^{env}_{i}$ to the labeler to obtain a set $\mathcal{D}^{L}_{i}$
of pairs of state and corrective action $(s,a^{L})$, which are also stored in the feedback replay buffer $\mathcal{D}^{L}$.
The collected data, $\mathcal{D}^{env}$ and $\mathcal{D}^{L}$, are used to train the RL agent in the next two phases.
As a side remark, for simplicity, querying the labeler is currently implemented in our experiments in a synchronous way (i.e., the agent waits for the labeler’s feedback), but this process could
possibly
also be asynchronous.

##### Finetune-phase.

This phase is a pure supervised training phase to learn from all the labeler’s corrective actions collected so far in $\mathcal{D}^{L}$.
Formally, ${Q}^{\theta_{i}}$ is updated with the following margin loss ${{\mathcal{L}}_{L}^{MG}}$ (Kim et al., ; Piot et al., ; Hester et al., ; Ibarz et al., ):

$$ ${{\mathcal{L}}^{MG}}(\theta_{i}\mid\mathcal{D}^{L})={\mathbb{E}}_{(s,a^{L})\in \mathcal{D}^{L}}\left[\max_{a\in{\mathcal{A}}}\left[{Q}^{\theta_{i}}(s,a)+l(a^ {L},a)\right]-{Q}^{\theta_{i}}(s,a^{L})\right],$ (1) $$

where $l(a^{L},a)=0$ if $a=a^{L}$, and a non-negative margin value $C$ otherwise.
This loss amounts to enforcing that the corrective actions’ Q-values should not be smaller than those of any other actions.
As common practice, ${Q}^{\theta_{i}}$ is updated via mini-batch stochastic gradient descent using Equation 1.
Following classical methods (Christiano et al., ) in RLHF, the updates end when the actions predicted by the updated ${Q}^{\theta_{i}}$ reaches a pre-defined accuracy
$\delta_{acc}$:
$\mathbb{P}_{s\sim\mathcal{D}^{L}}\left[a^{L}=\operatorname*{arg\,max}_{a}{Q}^{
\theta_{i}}(s,a)\right]>\delta_{acc}\,.$

##### Propagation-phase.

Since pure imitation learning requires a large number of human labels,
we design the ${\tt propagation}$-phase to include the training of the updated ${Q}^{\theta_{i}}$ with a combination of RL losses and margin loss (from actual labels in $\mathcal{D}^{L}$ but also pseudo-labels), which we explain next.

- *Training with RL Losses* allows not only learning from proxy rewards, but also propagating the effect of human labels to more states (learned in the previous phase).
The initial target ${\bar{Q}}$ in this phase is the ${Q}^{\theta_{i}}$ obtained at the end of the ${\tt finetune}$-phase.
These RL losses are composed of two commonly used terms, ${{\mathcal{L}}_{1}^{RL}}$ and ${{\mathcal{L}}_{n}^{RL}}$, which are respectively defined as:

$$ $\displaystyle{{\mathcal{L}}_{1}^{RL}}(\theta_{i}\mid\mathcal{D}^{env},{\bar{Q} })={\mathbb{E}}_{\mathcal{D}^{env}}\left[\left({Q}^{\theta_{i}}(s_{t},a_{t})- \left(\widetilde{r}_{t}+\gamma\max_{a^{\prime}\in{\mathcal{A}}}{\bar{Q}}(s_{t+ 1},a^{\prime})\right)\right)^{2}\right]\mbox{ and}$ (2) $\displaystyle{{\mathcal{L}}_{n}^{RL}}(\theta_{i}\mid\mathcal{D}^{env},{\bar{Q} })={\mathbb{E}}_{\mathcal{D}^{env}}\left[\left({Q}^{\theta_{i}}(s_{t},a_{t})- \left(\sum_{k=0}^{n-1}\gamma^{k}\widetilde{r}_{t+k}+\gamma^{n}\max_{a^{\prime} \in{\mathcal{A}}}{\bar{Q}}(s_{t+n},a^{\prime})\right)\right)^{2}\right],$ (3) $$

where $\mathcal{D}^{env}$ is the transition replay buffer, and
${\bar{Q}}$ is the target network from a historical version of ${Q}^{\theta_{i}}$.
Note that in the context of standard RL, Equation 3 is actually not theoretically-founded, since the goal is to learn a greedy policy.
However, in our context, it can help propagate faster the Q-values learned from the corrective actions.

- *Training with margin loss using both actual and pseudo-labels* allows to complement and correct the training with proxy rewards.
Training with observed corrective actions in $\mathcal{D}^{L}$ using loss ${{\mathcal{L}}^{MG}}$
prevents the agent to forget about the actual labels.
In addition, training with pseudo-labels can reduce the cost of collecting human labels by leveraging the large number of unlabeled states in $\mathcal{D}^{env}$.
Pseudo-labels can be generated, using predicted greedy actions from the target ${\bar{Q}}$, only on unlabeled states, since
otherwise, it is better to enforce the actual label than a pseudo-label.
Formally, the margin loss using pseudo-labels can be expressed as follows:

$$ ${{\mathcal{L}}_{TGT}^{MG}}(\theta_{i}\mid\mathcal{D}^{TGT})={\mathbb{E}}_{(s,a ^{TGT})\in\mathcal{D}^{TGT}}\left[\max_{a\in{\mathcal{A}}}\left[{Q}^{\theta_{i }}(s,a)+l(a^{TGT},a)\right]-{Q}^{\theta_{i}}(s,a^{TGT})\right],$ (4) $$

where $\mathcal{D}^{TGT}=\{(s,a^{TGT})\mid s\in\mathcal{D}^{env}\backslash\mathcal{D}
^{L},a^{TGT}=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}{\bar{Q}}(s,\cdot)\}$
and by
abuse of notations, $\mathcal{D}^{env}\backslash\mathcal{D}^{L}$ denotes the states that have not received any actual corrective actions $a^{E}$.
Intuitively, as the training process goes on, we expect that the quality of the pseudo-labels improves, enhancing further the benefit of enforcing ${{\mathcal{L}}_{TGT}^{MG}}$.
As a side note, ${{\mathcal{L}}_{TGT}^{MG}}$ can also be understood as regularizing the online network’s parameters to stay close to the target network, which has been shown to be beneficial in terms of learning efficiency (Asadi et al., ).
In contrast to this previous work, we regularize via the margin loss in the policy space instead of the parameter space.

To sum up, the total loss in this phase is the sum of the RL losses and the margin losses(^2^22In Equation 5 we give equal weight to the IL and RL losses, but unequal weights could be provided if we expect one source to be more suboptimal than the other.):

$$ $\displaystyle{\mathcal{L}^{Prop}}={{\mathcal{L}}_{1}^{RL}}(\theta_{i}\mid \mathcal{D}^{env},{\bar{Q}})+{{\mathcal{L}}_{n}^{RL}}(\theta_{i}\mid\mathcal{D }^{env},{\bar{Q}})+{{\mathcal{L}}^{MG}_{L+TGT}}(\theta_{i}\mid\mathcal{D}^{env },\mathcal{D}^{L})\mbox{, }$ $\displaystyle\mbox{where }{{\mathcal{L}}^{MG}_{L+TGT}}(\theta_{i}\mid\mathcal{ D}^{env},\mathcal{D}^{L})=(1-{\bar{w}}){{\mathcal{L}}_{L}^{MG}}(\theta_{i}\mid \mathcal{D}^{L})+{\bar{w}}\cdot{{\mathcal{L}}_{TGT}^{MG}}(\theta_{i}\mid \mathcal{D}^{env}),$ (5) $$

and hyperparameter ${\bar{w}}$ controls how much weight is put on pseudo-labels.
In our implementation,
${{\mathcal{L}}_{1}^{RL}}$, ${{\mathcal{L}}_{n}^{RL}}$ and ${{\mathcal{L}}_{TGT}^{MG}}$ works on a same minibatch from $\mathcal{D}^{env}$ in each gradient step,
while ${{\mathcal{L}}_{L}^{MG}}$ works on another one of the same size from $\mathcal{D}^{L}$ to relieve the unbalanced issue of these two datasets.

### 4.1 Other implementation details

Updating schedule for $Q^{\theta}$ and ${\bar{Q}}$.
Since $a^{TGT}$ will change every time we update the target ${\bar{Q}}$, we update ${\bar{Q}}$ with low frequency to stabilize training.
Concretely, there are $E$ epochs in one ${\tt propagation}$-phase that each epoch optimize $\mathcal{L}^{Prop}$ through $\mathcal{D}^{env}$.
At the end of each epoch, ${\bar{Q}}$ will be updated to the online parameter ${Q}^{\theta_{i}}$.
If $E=1$, pseudo-labels in ${\tt propagation}$-phase are generated form the policy at the end of ${\tt finetune}$-phase.
If $E>1$, pseudo-labels are changing after each epoch and can come from the (potentially) improved $Q^{\theta}$ after optimizing $\mathcal{L}^{Prop}$.

Optimization for ICoPro’s iterative scheme.
As for the optimizer, we use use a same optimizer through the two learning phases
in ICoPro.
Since ICoPro uses an iterative learning scheme, we follow the research from Asadi et al. () that reset the 1st and 2nd order moment inside the Adam optimizer at the beginning of each learning phase.
The ${\tt finetune}$- and ${\tt propagation}$-phase use the same learning rate $\alpha$, which remains unchanged throughout the training procedure.

## 5 Experiments

We first explain our general experimental settings in Section 5.1.
To compare ICoPro with baselines and analyze its design components on a large scale of experiments, we use simulated labelers to demonstrate and analyze ICoPro’s performance in Sections 5.2 and 5.3.
Then in Section 5.4, we further verify the alignment performance of ICoPro using real humans’ action feedback.

### 5.1 General experimental setups

To ensure reasonable labeling effort, only at most 1% of the environmental transitions are labeled in our experiments.
Hyper-parameter values are given in Section B.2 unless otherwise specified.
Detailed evaluation configurations can be checked in Section D.1.

Baselines.
To compare with the performance of learning from only proxy rewards,
we use Rainbow (Hessel et al., ) as a baseline.
To compare with the performance of learning from only corrective actions using ${{\mathcal{L}}_{L}^{MG}}$ (Equation 1),
we evaluate behavior cloning (BC-L) on all collected labels $\mathcal{D}^{L}$ at the end of training of ICoPro as the one in offline style,
and adapt HG-DAgger (Kelly et al., ) (HGDAgger-L) by removing the ${\tt propagation}$-phase from ICoPro as the one in online style.
As for baselines involving learning from both signals, since no previous work considers the exact same learning scheme as explained in Section 3,
we adapt two state-of-the-art methods that were initially designed on normal RL methods into our iterative learning setting with the same labeling schedule and budget,
DQfD (Hester et al., ) (DQfD-I) and PVP (Peng et al., ) (PVP-I),
to compare the performance when learning from both corrective actions and proxy rewards.
Basically, the adaption is canceling the ${\tt finetune}$-phase in ICoPro
and replacing their loss functions with ICoPro’s ${\mathcal{L}^{Prop}}$ (Equation 5) in the ${\tt propagation}$-phase.
DQfD-I can be seen as an ablation of ICoPro by removing ${{\mathcal{L}}_{TGT}^{MG}}$ from ${\mathcal{L}^{Prop}}$.
PVP-I’s loss replaces ${{\mathcal{L}}^{MG}_{L+TGT}}$ to ${{\mathcal{L}}_{L}^{PVP}}={\mathbb{E}}_{(s,a,a^{E})\sim\mathcal{D}^{L}}\left[|
Q^{\theta}(s,a^{E})-1|^{2}+|Q^{\theta}(s,a)+1|^{2}\right]$, but using zero rewards in ${{\mathcal{L}}_{1}^{RL}}$ and ${{\mathcal{L}}_{n}^{RL}}$.
We also test in PVP-I, the effects of using proxy rewards instead of zeros (PVP-IR).
More details about the baselines and other ablations mentioned in Section 5.2 can be checked in Appendix C.

Environments.
Our experiments involve both the state-based highway (Leurent, ) environment and imaged-based Atari (Aitchison et al., ) environments.

**Table 1: Engineered proxy rewards on highway. We normalize the rewards into the range $[-1,1]$ in each step by min-max scaling if Normalization is not None, where the min (max) reward is the sum of negative (positive) event rewards.**
| Configurations | Proxy Rewards |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PRExp | PR1 | PR2 | PR3 | PR4 |  |  |
| Rewards<br>for<br>Events | Change lane | 0.2 | 0 | 0.2 | 0 | 0 |
| High speed | 1.5 | 2 | 0.8 | 0 | 0 |  |
| Low speed | -0.5 | -1 | 0 | 0 | 0 |  |
| Crash | -1.7 | -1 | -1 | -1 | -1 |  |
| Normalization | [-1,1] | [-1,1] | [-1,1] | [-1,1] | None |  |
| Dense(D) or Sparse(S) | D | D | – | D | S |  |

Highway is an environment that allows for flexible design of diverse proxy rewards and evaluation metrics, but is relatively simpler than Atari in terms of action dimension ($|{\mathcal{A}}|=5$) and environmental complexity.
The basic goal in this environment is to drive a car on a straight road
with multiple lanes.
The final performance of the trained vehicle can be measured with different metrics, e.g.,
the ratio of episodes with a crash (%Crash),
the average total forward distance in a given time limit (Distance-Avg),
the average speed (Speed-Avg),
and the ratio of taking a change lane action (%LaneChange-Avg).
We design representative proxy rewards in different levels of imperfection by associating events with different rewards as shown in Table 1.

In contrast, Atari provides more complex environments with larger action dimensions and longer episode lengths, although with less flexibility in designing proxy rewards and diverse performance metrics.
Since the dimension of action space can be seen as a measurement of hardness for one game,
we first select 6 representative games with the full action dimension ($|{\mathcal{A}}|=18$) across various categories suggested by Aitchison et al. ():
(1) Combat: Battlezone and Seaquest,
(2) Sports: Boxing,
(3) Maze: Alien,
(4) Action: Frostbite and Hero.
We also include smaller-action-dimensional but classical games to complement our evaluation: MsPacman and Enduro ($|{\mathcal{A}}|=9$), Pong ($|{\mathcal{A}}|=6$), and Freeway ($|{\mathcal{A}}|=3$).
In this setting, cumulative episodic raw rewards from ALE are used to measure agent performance, with signed raw rewards serving as proxy rewards(^3^33Raw rewards ${r_{A}}$ in Atari can be seen as goal-conditioned rewards, but not all desired performances (e.g., achieve goals in the right way) are rewarded in ${r_{A}}$. Further discussion is available in Section D.3.).

Simulated labeler.
We use Q-values trained with Rainbow to simulate human feedback with $Q^{diff}(s,a,a^{L})={Q^{L}}(s,a^{L})-{Q^{L}}(s,a)$,
where ${Q^{L}}(s,a)={\mathbb{E}}_{{P},{\pi^{L}}}[\sum_{t=0}^{\infty}\gamma^{t}r^{L}(s
,a)|s_{0}=s,a_{0}=a]$ is the labeler’s Q-function,
and
$a^{L}=\mathcal{G}_{\epsilon}({Q^{L}}(s,\cdot))$
is the action issued from the labeler,
$a=\mathcal{G}_{\epsilon}(Q^{\theta}(s,\cdot))$
is the executed action from the training agent.
In practice, we select states with top $N_{CF}$ largest $Q^{diff}$ to give corrective actions for a querying segment Luo et al. ()
, where $N_{CF}=1$ is our default configuration.
Basically, simulated labelers are agent checkpoints
trained with engineered rewards by Rainbow but
do not necessarily converge to optimal performances.
For highway, labelers are trained with PRExp mentioned in Table 1.
For Atari, labelers are trained with the signed raw rewards.
Concrete configurations to train labelers can be checked in Section D.4.

Figure: (a) Compare ICoPro (solid lines) with Rainbow (dotted lines) using different proxy rewards (different colors).
Refer to caption: x2.png

### 5.2 Sample-efficient Alignment with Imperfect Proxy Rewards

We use one scripted labeler for each environment to validate the alignment ability of ICoPro and compare it with baselines, using proxy rewards with different levels of imperfections.
In the following of this part, we analyze ICoPro’s performance based on results shown in Figure 2 and Table 2.
More specifically, Figure 2 evaluates the performance with various performance metrics, while Table 2 evaluates the performance with small or large size of action label budget(^4^44Considering that the action budgets required to align depend on the complexity of the labeler’s performance, the concrete values of $|\mathcal{D}^{L}|$ is not specified in the main text but can be checked in Table 7.).
Due to space constraints, results for easier Atari tasks (i.e., Pong, Freeway, and Enduro) and corresponding plots for the results in Table 2, are put into Appendix A.

**Table 2: Experimental results on Atari with different size of $|\mathcal{D}^{L}|$. Red fonts denote the best results with the largest mean score. Bold black fonts emphasize results whose mean$+$std covers the best mean score. Yellow backgrounds highlight results that match the labelers’ performance in the sense that their mean scores are in the range of the labelers’ mean$\pm$std.**
| $|\mathcal{D}^{L}|$ | Methods | Environments |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seaquest | Boxing | Battlezone | Frostbite | Alien | Hero | MsPacman |  |  |
| $\backslash$ | Labeler | 1071.92$\pm$224.88 | 66.05$\pm$9.17 | 24485.19$\pm$4987.60 | 5647.63$\pm$1038.91 | 2908.97$\pm$908.32 | 26866.07$\pm$612.29 | 3539.54$\pm$1270.95 |
| Rainbow | 707.04$\pm$98.96 | 1.99$\pm$0.71 | 17058.00$\pm$2310.60 | 2766.40$\pm$518.78 | 1319.83$\pm$457.06 | 19413.57$\pm$600.39 | 1773.98$\pm$323.19 |  |
| Large | HGDAgger-L | 790.10$\pm$73.98 | 24.95$\pm$2.22 | 16573.33$\pm$1262.82 | 3207.35$\pm$527.35 | 2067.45$\pm$262.04 | 13258.79$\pm$2128.13 | 3052.38$\pm$370.50 |
| PVP-I | 635.83$\pm$65.74 | 7.76$\pm$0.84 | 15183.03$\pm$1739.17 | 1535.28$\pm$528.88 | 596.56$\pm$99.45 | 8503.46$\pm$1205.13 | 1686.13$\pm$608.96 |  |
| PVP-IR | 770.88$\pm$91.00 | 13.36$\pm$2.00 | 17378.18$\pm$2245.64 | 1664.62$\pm$414.02 | 798.55$\pm$137.99 | 7729.12$\pm$1282.99 | 920.25$\pm$216.76 |  |
| BC-L | 758.40$\pm$41.31 | 21.39$\pm$1.21 | 16020.00$\pm$2081.92 | 3410.27$\pm$479.08 | 2360.40$\pm$422.57 | 14288.67$\pm$1673.48 | 2877.07$\pm$279.24 |  |
| w/o $a^{TGT}$ | 1213.98$\pm$102.54 | 48.19$\pm$7.90 | 18111.52$\pm$1409.68 | 3874.36$\pm$515.30 | 1835.06$\pm$267.85 | 17805.35$\pm$2471.35 | 2579.86$\pm$186.22 |  |
| w/o ${\tt Finetune}$ | 1234.39$\pm$113.44 | 51.83$\pm$26.67 | 19366.67$\pm$285.28 | 4713.07$\pm$1211.91 | 2223.66$\pm$368.38 | 8398.37$\pm$2729.85 | 2633.59$\pm$154.34 |  |
| DQfD-I | 1155.43$\pm$85.98 | 41.71$\pm$5.09 | 18444.85$\pm$1279.70 | 4176.27$\pm$659.14 | 2079.99$\pm$293.53 | 17814.10$\pm$1688.55 | 2734.25$\pm$176.22 |  |
| ICoPro | 1274.76$\pm$109.53 | 61.52$\pm$2.47 | 20030.30$\pm$1620.36 | 4817.74$\pm$605.45 | 2360.27$\pm$301.52 | 23344.01$\pm$2353.67 | 3188.20$\pm$453.13 |  |
| Small | HGDAgger-L | 508.34$\pm$77.91 | 17.91$\pm$2.21 | 14682.22$\pm$1633.46 | 1561.04$\pm$457.61 | 1582.74$\pm$467.88 | 10617.77$\pm$1664.62 | 2342.51$\pm$264.97 |
| PVP-I | 439.96$\pm$68.62 | 1.09$\pm$2.05 | 9242.42$\pm$2056.15 | 509.45$\pm$237.64 | 402.53$\pm$78.44 | 6454.75$\pm$1193.92 | 891.84$\pm$251.87 |  |
| PVP-IR | 733.73$\pm$116.70 | 14.33$\pm$3.98 | 16510.30$\pm$1684.40 | 1883.64$\pm$270.40 | 1136.07$\pm$301.34 | 8777.76$\pm$1410.76 | 1097.92$\pm$246.94 |  |
| BC-L | 537.20$\pm$51.86 | 9.09$\pm$4.64 | 17273.33$\pm$2605.40 | 1348.80$\pm$236.28 | 1711.00$\pm$659.06 | 10252.73$\pm$1556.18 | 2278.00$\pm$337.84 |  |
| w/o $a^{TGT}$ | 1065.65$\pm$164.01 | 40.56$\pm$10.22 | 16465.45$\pm$1434.87 | 2785.41$\pm$452.83 | 1269.06$\pm$288.76 | 15420.19$\pm$2108.08 | 1676.97$\pm$652.21 |  |
| w/o ${\tt Finetune}$ | 1081.16$\pm$131.98 | 19.35$\pm$16.53 | 21253.33$\pm$2694.61 | 3037.84$\pm$1152.24 | 891.97$\pm$148.50 | 9759.02$\pm$3093.64 | 1830.23$\pm$559.33 |  |
| DQfD-I | 1003.21$\pm$128.45 | 39.27$\pm$20.85 | 17003.64$\pm$1532.05 | 3146.25$\pm$418.00 | 1075.79$\pm$249.17 | 16525.18 $\pm$1915.10 | 1952.64$\pm$427.78 |  |
| ICoPro | 1167.19$\pm$103.02 | 61.40$\pm$23.91 | 19969.09$\pm$1555.69 | 3368.31$\pm$701.54 | 1736.52$\pm$513.91 | 17862.58$\pm$2181.02 | 2558.57$\pm$331.18 |  |

Learning from a combination of two imperfect signals performs better than learning from $\widetilde{r}$ or corrective actions alone.
Compared to Rainbow, Figure 2(a) and Table 2 show that ICoPro with corrective actions can converge quickly and stably to the labeler’s performance using proxy rewards with different levels of imprefection, but Rainbow’s performance is unstable and can easily converge to non-optimal performance without the guidance from corrective actions.
Compared to BC-L and HGDAgger-L, Figure 2(b) and Table 2 demonstrate that ICoPro’s performance exceeds their performances substantially in most of the games,
which confirms the effectiveness of integrating the proxy rewards with the RL losses to achieve better alignment than using corrective actions alone.

ICoPro achieves better alignment performance than baselines.
As shown in Figure 2(b) and Table 2, using the same sample budget in terms of both environmental transitions and label budget, ICoPro achieves the best performance in terms of aligning with labelers’ performance in almost all settings, or performs similarly with the best one otherwise.
In Figure 2(b), except for Rainbow, all methods perform similarly in terms of aligning the labeler’s Speed-Avg and %LaneChange-Avg performance, while for the harder performance metrics %Crash and Distance-Avg, ICoPro and DQfD-I perform similarly, and both are significantly better than PVP-I, PVP-IR, and HGDAgger-L.
In Table 2, with large $\mathcal{D}^{L}$, ICoPro does match the labelers’ performances, except in the hardest Hero with full action dimensions and relatively long episode length.
Specifically, compared with the two PVP methods,
their performances are obviously worse than that of the methods using margin loss,
since ${{\mathcal{L}}_{L}^{PVP}}$ sets a pre-defined bound for Q-values making it fails to adapt to various reward settings.
In some games, incorporating proxy rewards into their framework (PVP-IR) performs even worse than not (PVP-I).
However, margin loss is a suitable choice to incorporate the corrective actions with proxy rewards.
As for DQfD-I, in simple environments we perform similarly, but in harder Atari environments ICoPro outperforms it and shows more notable performance gaps than in highway.

Ablations: ICoPro’s integral design leads to a stable and robust performance.
Considering that the two-phase learning and pseudo-labeling are the main novel designs in ICoPro,
there are two additional settings to be examined besides DQfD-I and HGDAgger-L: (1) w/o ${\tt Finetune}$, which removes the ${\tt Finetune}$-phase but retains the pseudo-labels, and (2) w/o $a^{TGT}$, which removes pseudo-labels.
In the highway environment, the two ablations perform similarly to ICoPro, and we include them into Section A.2 instead of Figure 2(b) to make it clearer to check.
However, in Table 2, we find that while the two ablation settings may perform well in individual environments when action feedback is abundant, they are not as robust as ICoPro across different environments, especially when feedback is limited and tasks are challenging. ICoPro consistently demonstrates superior performance in such scenarios, maintaining its lead even with fewer feedback instances.

### 5.3 Overcoming Non-optimality of Corrective Actions with Proxy Rewards

Figure: (a) Highway with DiffRand. Performances are averaged over the set of proxy rewards in Table 1.
Refer to caption: x4.png

To simulate different types of imperfections inside corrective actions, the most natural and commonly adopted (Lee et al., ; Luo et al., ) one is to replace part of a labeler’s corrective actions with random actions (DiffRand).
In our setting, it simulates labelers who know which state-action pairs to correct but provide noisy corrective actions.
We evaluate it in three representative environments that cover various environmental complexities: highway, Boxing and Seaquest.
As shown in Figures 3(a) and 3(b). ICoPro performs robustly within a reasonable ratio of random actions (i.e., $<25\%$), and can perform significantly better than the labelers with large randomness (i.e., $50\%$).
For Atari, we further consider a harder type that uses different labelers with worse performances (DiffLabler) to simulate labelers who are suboptimal on both where and how to provide corrective actions.
As shown in Figure 3(c), ICoPro still demonstrate a strong capability to overcome the non-optimality inside such unreasonable action feedback in most of the cases.
Performance of Highway using another labeler, and more plots for Atari can also be checked in Appendix A.

### 5.4 User study

Figure: Figure 4: Performances of the scripted labeler and ICoPro-Human in Pong. Each row shows a sequence of state-action pairs. While the scripted labeler prefers to catch the ball with the corner of the paddle, ICoPro-Human prefers to catch the ball with a larger part of the paddle, which is more human-like.
Refer to caption: x7.png

To verify that ICoPro can align with real human’s preferences when the desired performance is hard to be obtained with engineered rewards,
we use Pong and highway as two representative environments to evaluate ICoPro with real human (ICoPro-Human) with 500 action labels.
For Pong, ICoPro-Human obtains an agent that performs more human-like than the scripted optimal labelers (see Figure 4 for details).
For highway, we use PR4 as the proxy reward inICoPro-Human since it does not introduce any extra performance shaping, and obtain a vehicle that knows how to take over cars with super fast average speed
([Speed-Avg,%Crash]=[29.02, 0.22]), which is a behavior that we could never obtain with Rainbow using PR4 alone.
Moreover, compared to ICoPro-Human, BC-L on those real human labels gets poor performance due to the limited data and non-optimality of human feedback (Pong: 7.83 vs. -11.26, highway: [29.02, 0.22] vs. [27.90, 0.41])), which confirms again ICoPro’s ability to learn from two imperfect signals in a sample-efficient way.
More details about our user study and the whole evaluation videos are provided in Appendix E.

## 6 Conclusions and limitations

We present a human-in-the-loop framework where an RL agent can learn from two potentially unreliable signals: corrective actions provided by a labeler and a pre-defined proxy reward.
The motivation for combining the two learning signals is threefold: (1) guide the agents’ training process with corrective actions,
(2) use proxy rewards to help reduce human labeling efforts, and
(3) possibly compensate for the imperfection of one signal with the other.
Our value-based method trains a policy in an iterative way, with two separate learning phases inside each iteration:
the
${\tt finetune}$-phase uses a margin loss to update the policy to better align with action feedback,
then the ${\tt propagation}$-phase incorporates RL losses as well as the margin loss expressed with pseudo-labels to generalize the improved values.
Our experiments validate our algorithmic design.

Although our method achieves better and robust performance compared with baseline methods, it could be further improved by replacing uniform query selection
by more advanced sampling methods, which could help reduce the number of needed feedback further.
In addition
, while we demonstrated the feasibility and the benefit of learning from two imperfect signals, we plan as future work to provide a more theoretical analysis
to
reveal what assumptions about the misspecification of proxy rewards and/or the suboptimality of the corrective actions could guarantee their
synergetic
combination.

## Appendix A More experimental results and plots

### A.1 Atari

#### A.1.1 Learning Curves for Table 2 Including Pong, Freeway, and Enduro

Figure 5 show plots comparing ICoPro with various baselines,
and Figure 6 show ablation study for ICoPro.
Lines represent the average episodic return in terms of the raw reward. Shadows represent the standard deviation.
For Rainbow, the x-axis should read in terms of steps instead of iterations.
For other methods, the x-axis can be read in terms of steps or iterations.

Figure: Figure 5: Compare baseline methods on Atari in terms of the averaged episode return measured with the raw reward ${r_{A}}$. The shadow indicates the standard deviation over 5 seeds. $N_{q}$ in titles refer to the number of queries per iteration, and the larger (resp. smaller) ones correspond to the large (resp. small) $\mathcal{D}^{L}$ in Table 2.
Refer to caption: x8.png

Figure: Figure 6: Ablation study on Atari to test the effect of ICoPro’s two-phase scheme and target pseudo label. $N_{q}$ in titles refer to the number of queries per iteration, and the larger (resp. smaller) ones correspond to the large (resp. small) $\mathcal{D}^{L}$ in Table 2.
Refer to caption: x9.png

#### A.1.2 Compare ICoPro with More Related Works.

##### Compare with Ilhan et al. ( 2021b ) .

In Table 3, we compare the performance of ICoPro with ActionImitation (Ilhan et al., ), which is a method that also uses action advice but is to augment the replay buffer, on their experimental environments.
Results for ActionImitation come from their paper.
Our simulated labelers in the three games achieved the same score as theirs (full score on Pong and Freeway, and 1200 score on Enduro), so we put ICoPro’s results from Figure 5 directly to compare with.
Our method uses significantly less data than the ActionImitation method.

**Table 3: Compare the number of feedback actions ($|N_{lab}|$) and environmental interaction timesteps ($T_{env}$) needed to reach the labelers’ score in each environment.**
| Method | $|N_{lab}|$ | $T_{env}$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Pong | Freeway | Enduro | Pong | Freeway | Enduro |  |
| ActionImitation | 10K | 10K | 10K | 3M | 3M | 3M |
| ICoPro | 375 | 2K | 8k | 375K | 500K | 2M |

#### A.1.3 Detailed Results for DiffRand and DiffLabler with Different Label Budget Size

Figure 7 are separated plots using different size of label busget in Figures 3(b) and 3(c).
For DiffLabler, a larger number of imperfect labels tends to lead to a worse performance, as shown in Figure 7(b).
But for DiffRand, a larger number of imperfect labels tends to lead to performance with less variance but may not necessarily to a worse performance, as shown in Figure 7(a).

Figure: (a) DiffRand correspond to Figure 3(b).
Refer to caption: x10.png

### A.2 Highway

#### A.2.1 Extra Results for Ablation Study

Figure 8 shows the ablation study on highway. For such an environment with a relatively small number of action dimensions, ICoPro can outperform HGDAgger-L significantly but only outperform other ablations slightly. Similar performance can also be observed in Atari games with small action dimension like Pong and Freeway in Figure 6.

Figure: Figure 8: Extra ablation results on highway with the labeler used in Figure 2. Performances are averaged over the set of proxy rewards mentioned in Table 1.
Refer to caption: x12.png

#### A.2.2 Extra Evaluation with Another Scripted Labeler ( DiffLabler )

In Table 4, we compare the two simulated labelers used in our experiments.
The first labeler (Labeler-CL), which is the labeler illustrated in our main text, prefers driving faster and changing to different lanes more often than the second one, but with a larger crash rate as well.
The second one (Labeler-RL) drives in a safer way at a lower speed and prefers to drive in lanes on the right.
Then, in Table 5, we list the 2 sets of proxy rewards for the two labelers.

Figure 9 shows experimental results with Labeler-RL, which confirm our analysis in Section 5.2
again that ICoPro can align to the desired performance better than baselines.

**Table 4: Performance metric for the 2 simulated labelers in highway.**
| Labeler | %Crash | Step-Avg | Step-Min | Distance-Avg | Distance-Min | Speed-Avg | LanePosition-Avg | %LaneChange-Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Labeler-CL | 0.10 | 48.51$\pm$0.39 | 20.80$\pm$9.78 | 1163.689$\pm$12.85 | 483.09$\pm$227.80 | 23.97$\pm$0.06 | 0.57$\pm$0.05 | 0.18$\pm$0.02 |
| Labeler-RL | 0.02 | 49.29$\pm$0.74 | 28.95$\pm$19.92 | 1099.38$\pm$19.02 | 608.93$\pm$404.97 | 22.30$\pm$0.12 | 0.91$\pm$0.02 | 0.04$\pm$0.01 |

**Table 5: Reward weights for engineered proxy rewards on highway for the two labelers mentioned in Table 4.**
| Labeler-CL |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Proxy Rewards | PRExp | PR1 | PR2 | PR3 | PR4 |  |
| Evnets | Change lane action | 0.2 | 0 | 0.2 | 0 | 0 |
| Normalized lane index | 0 | 0 | 0 | 0 | 0 |  |
| High speed | 1.5 | 2 | 0.8 | 0 | 0 |  |
| Low speed | -0.5 | -1 | 0 | 0 | 0 |  |
| Crash | -1.7 | -1 | -1 | -1 | -1 |  |
| Normalize | [-1,1] | [-1,1] | [-1,1] | [-1,1] | None |  |
| Labeler-RL |  |  |  |  |  |  |
| Proxy Rewards | PRExp | PR1 | PR2 | PR3 | PR4 |  |
| Evnets | Change lane action | 0.2 | 0 | 0 | 0 | 0 |
| Normalized lane index | 0.5 | 0.5 | 0.2 | 0 | 0 |  |
| High speed | 1.7 | 1.5 | 0.8 | 0 | 0 |  |
| Low speed | -0.5 | -0.5 | 0 | 0 | 0 |  |
| Crash | -1.9 | -1.5 | -1 | -1 | -1 |  |
| Normalize | [-1,1] | [-1,1] | [-1,1] | [-1,1] | None |  |

Figure: (a) Compare agent’s performance learned from ICoPro and Rainbow.
Refer to caption: x13.png

#### A.2.3 Extra Results for DiffRand

In Figure 10 we show the whole detailed plots for the averaged ones mentioned in Figure 3(a). ICoPro overcome such non-optimality inside the non-optimal corrective actions no matter which proxy reward is applied.

Figure: Figure 10: Detailed performance for each proxy rewards that used to show the average performance in Figure 3(a).
Refer to caption: x15.png

## Appendix B Implementation details for ICoPro

### B.1 Pseudo-code for ICoPro

Algorithm 1 show the concrete learning procedure of ICoPro.

Figure: Algorithm 1 ICoPro

### B.2 Hyper-parameters

The basic hyper-parameters for ICoPro in all environments are listed in Table 6.

**Table 6: Default hyper-parameters.**
|  | Hyper-parameter | Value |
| --- | --- | --- |
| General | training batch size $B$ | 128 |
| margin $C$ | 0.05 |  |
| Architecture for Neural Network | same with DERainbow(Van Hasselt et al., ) |  |
| Optimizer | type | Adam(Diederik, ) |
| learning rate $\alpha$ | 0.0001 |  |
| eps | $0.01/B$ |  |
| betas | (0.9, 0.999) |  |
| ${\tt DataCollection}$-phase | $N_{CF}$ | 1 |
| $\epsilon$ | 0.01 |  |
| $T$ | Atari: 25, highway: 10 |  |
| query sampling | uniformly sample segments without overlap |  |
| ${\tt Finetune}$-phase | Accuracy target $\delta_{acc}$ | 0.98 |
| ${\tt Propagation}$-phase | discount factor $\gamma$ | 0.99 |
| weight for ${{\mathcal{L}}_{TGT}^{MG}}$: ${\bar{w}}$ | 0.5 |  |
| weight for ${{\mathcal{L}}_{L}^{MG}}$: $1-{\bar{w}}$ | 0.5 |  |

In Table 7, we list hyper-parameters with slight differences in different environments.
The reasons that we do not use a same configuration is:

- •
Choices of $n$ in Equation 3 are the same one as training the Rainbow labeler.
We find that in Boxing and Enduro we can not obtain meaningful checkpoints with the default value 20 and therefore using smaller values.
- •
$E$ defaults to 2, except for the 3 hard Atari games with 18 actions.
We observed that setting $E=2$ will be hard to let ICoPro and DQfD-I reach labelers’ performance, but setting $E=1$ performs better.
- •
Since Enduro’s episode length can reach 25K when the performance approach to our labelers’.
In this case, set $H$ as the default 5K can introduce bias in our iterative feedback schedule. Therefore we set $H=20K$ in this game.

Moreover, we compare the design elements between ICoPro and the original Rainbow’s (Hessel et al., ) in Table 8.

**Table 7: Specific configurations and hyper-parameters for each environment. Checkpoint-step is the timestep to obtain our simulated labelers (i.e., policy checkpoints). Meanings for other configurations are consistent with Algorithm 1. The total environmental steps equals to $N_{Itr}\times H$, $|\mathcal{D}^{L}|$ equals to $N_{Itr}\times N_{q}\times N_{CF}$, $|\mathcal{D}^{env}_{[i-I,i]}|$ equals to $I\times H$.**
| Configuration | Atari-Games | Highway-Labelers |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Seaquest | Boxing | Battlezone | Frostbite | Alien | Hero | MsPacman | Freeway | Pong | Enduro | Labeler-CL | Labeler-RL |  |
| Checkpoint-step | 4.6M | 10M | 4.6M | 4.6M | 5M | 5M | 4M | 2M | 1.8M | 4.8M | 330K | 990K |
| Step $n$ | 20 | 10 | 20 | 20 | 20 | 20 | 20 | 20 | 20 | 3 | 20 |  |
| $E$ | 1 | 2 | 1 | 1 | 2 | 2 | 2 | 2 | 2 | 2 | 2 |  |
| $I$ | 64 | 64 | 64 | 64 | 64 | 64 | 64 | 64 | 64 | 16 | 100 |  |
| $H$ | 5K | 5K | 5K | 5K | 5K | 5K | 5K | 5K | 5K | 20K | 1K |  |
| $|\mathcal{D}^{env}_{[i-I,i]}|$ | 32K | 32K | 32K | 32K | 32K | 32K | 32K | 32K | 32K | 32K | 100K |  |
| $N_{CF}$ | 1 | 1 |  |  |  |  |  |  |  |  |  |  |
| Small $N_{q}$ ($|\mathcal{D}^{L}|$) | 5 (2K) | 5 (500) | 5 (2K) | 5 (2K) | 5 (2K) | 20 (8K) | 5 (2K) | 5 (750) | 5 (750) | 80 (8k) | 10 |  |
| Large $N_{q}$ ($|\mathcal{D}^{L}|$) | 20 (8K) | 20 (2K) | 20 (8K) | 20 (8K) | 20 (8K) | 40 (16K) | 20 (8K) | 20 (8K) | 20 (3K) | 160 (16K) | 10 |  |
| $N_{Itr}$ | 400 | 100 | 400 | 400 | 400 | 400 | 400 | 150 | 150 | 100 | 150 |  |
| Total Env Steps | 2M | 500k | 2M | 2M | 2M | 2M | 2M | 750K | 750K | 2M | 150K |  |

### B.3 Experiments Compute Resources

For experiments in Atari, we only need one GPU card to launch experiments, including
GPUs GeForce RTX 3060 12G GPU + 48GB memory + Intel Core i7-10700F,
GeForce RTX 3060 12G GPU + 64GB memory + Intel Core i7-12700,
GeForce RTX 2070 SUPER + 32GB memory + Intel Core i7-9700,
GeForce RTX 2060 + 64GB memory + Intel Core i7-8700.
For experiments in highway, we use CPUs.

## Appendix C More details about baselines and ablations

##### BC-L.

For BC, we ran 1 seed for the replay buffer obtained at the end of training of ICoPro, therefore the performance is averaged over 5 different label buffers.
In Atari, we select the best result during training with respect to during training the accuracy to reach $\delta_{acc}=0.999$.
In highway, since we have multiple performance metrics and it’s not easy to measure which one is the best, we present the performance when it reaches our default $\delta_{acc}=0.98$.

##### Rainbow.

We compare ICoPro with Rainbow in Table 8.
See Table 12 for hyper-parameters of Rainbow. We use the same setting with a data-efficient version of Rainbow, except for the step $n$ and the training timesteps (see Table 7).

**Table 8: Compare the different component between ICoPro and Rainbow (Hessel et al., ).**
| Method | Components |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1-step | N-step | Duel | Double Q | Replay | Distrib | Exploration |  |
| ICoPro | Yes | Yes | Yes | No | Uniform | Yes | Feedback guided |
| Rainbow | Yes | Yes | Yes | Yes | Prioritized | Yes | Noisy-net |

##### Compare design choices with baselines and ablations.

DQfD (Hester et al., ) is a method that learns from human demonstrations,
and PVP (Peng et al., ) learns from human interventional control.
Both of the two methods are designed by augmenting the standard RL loss with their extra IL loss into normal RL methods like DQN (Mnih et al., ) and TD3 (Fujimoto et al., ).
We compare the learning scheme and corresponding losses for each baselines and ablation settings in Table 9, other hyper-parameters keep the same with Table 6 to have a fair comparison.

**Table 9: Compare the design choices between ICoPro, different baselines, and extra ablation groups. Backslashes mean there is no such phase.**
| Methods | ICoPro | DQfD-I |  |
| --- | --- | --- | --- |
| Phases | ${\tt Finetune}$ | ${{\mathcal{L}}_{L}^{MG}}$ | $\backslash$ |
| ${\tt Propagation}$ | $0.5\cdot{{\mathcal{L}}_{L}^{MG}}+0.5\cdot{{\mathcal{L}}_{TGT}^{MG}}+{{\mathcal<br>{L}}_{1}^{RL}}+{{\mathcal{L}}_{n}^{RL}}$ | ${{\mathcal{L}}_{L}^{MG}}+{{\mathcal{L}}_{1}^{RL}}+{{\mathcal{L}}_{n}^{RL}}$ |  |
| Methods | PVP-IR | PVP-I |  |
| Phases | ${\tt Finetune}$ | $\backslash$ | $\backslash$ |
| ${\tt Propagation}$ | ${{\mathcal{L}}_{L}^{PVP}}+{{\mathcal{L}}_{1}^{RL}}+{{\mathcal{L}}_{n}^{RL}}$ | ${{\mathcal{L}}_{L}^{PVP}}+{{\mathcal{L}}_{1(r=0)}^{RL}}+{{\mathcal{L}}_{n(r=0)<br>}^{RL}}$ |  |
| Methods | DAgger | w/o ${\tt Finetune}$ |  |
| Phases | ${\tt Finetune}$ | ${{\mathcal{L}}_{L}^{MG}}$ | $\backslash$ |
| ${\tt Propagation}$ | $\backslash$ | $0.5\cdot{{\mathcal{L}}_{L}^{MG}}+0.5\cdot{{\mathcal{L}}_{TGT}^{MG}}+{{\mathcal<br>{L}}_{1}^{RL}}+{{\mathcal{L}}_{n}^{RL}}$ |  |
| Methods | w/o $a^{TGT}$ |  |  |
| Phases | ${\tt Finetune}$ | ${{\mathcal{L}}_{L}^{MG}}$ |  |
| ${\tt Propagation}$ | ${{\mathcal{L}}_{L}^{MG}}+{{\mathcal{L}}_{1}^{RL}}+{{\mathcal{L}}_{n}^{RL}}$ |  |  |

## Appendix D More Details about Experimental Settings

### D.1 Configurations for evaluation

We use 5 seeds for ICoPro and baselines for evaluation, and 3 seeds for ablations.
For all experiments we perform, we use 50 episodes in each evaluation.
When we ran experiments for ICoPro and baselines, we used 5 seeds for each setting.
For ablation studies, we use 3 seeds.
Although ICoPro and its’ ablation w/o ${\tt Finetune}$ involve 2 separate phase in each iteration, their performance are still evaluated at the end of each iteration, which is the ${\tt propagation}$-phase.

### D.2 Highway

Highway (Leurent, ) is an environment
with state-based inputs in 35 dimensions and 5 available action choices to control the vehicle’s speed and direction.
The input states contain information about both the controlled vehicles and the top 5 nearest vehicles’ positions, speeds, and directions.
The basic goal in this environment is to drive a car on a straight road with multiple lanes within a limited time.
One episode is terminated when the time is used up, or a crash happens.

We use the configuration listed in Table 10 in highway.

**Table 10: Hyper-parameters for environment in highway.**
| Hyper-parameter | Value |  |
| --- | --- | --- |
| Env | Available speed range | [19,30] |
| High speed range | $\geq$21 |  |
| Low speed range | $<$21 |  |
| Lanes count | 5 |  |
| Vehicles count | 40 |  |
| Time limit | 50 |  |
| Policy frequency | 1 |  |
| Observation | Type | Kinematics matrix |
| Number of observed vehicles | 5 |  |
| Features for each vehicle | [presence, x, y, $v_{x}$, $v_{y}$, $\cos_{heading}$, $\sin_{heading}$] |  |
| Action | Type | [Right, Left, Faster, Slower, IDLE] |

### D.3 Atari

In Atari, we use the signed raw reward from the environment as the proxy reward in our setting:
${\widetilde{r}_{A}}=sign({r_{A}})$, where ${r_{A}}$ is the raw reward in Atari and ${\widetilde{r}_{A}}$ is the proxy reward.
The average cumulative ${r_{A}}$ of episodes serves as a performance metric.

Both ${r_{A}}$ and ${\widetilde{r}_{A}}$ have different levels of imperfection.
The imperfection of ${\widetilde{r}_{A}}$ mainly comes from two aspects:
(1) Missing rewards,
e.g. in Seaquest, ${r_{A}}(LooseLives)=0$, but give a negative reward is better;
(2) Numerical mismatch between ${\widetilde{r}_{A}}$ and ${r_{A}}$,
e.g., in Hero, ${r_{A}}(RescueMiners)=1K$, ${r_{A}}(ShootCritters)=50$ but ${\widetilde{r}_{A}}=1$ for both the two cases.
More information about the ground-truth ${r_{A}}$ can be checked in the official website(^5^55[https://gymnasium.farama.org/environments/atari/](https://gymnasium.farama.org/environments/atari/)).

Our environment configurations follow standard requirements Machado et al. ().
Table 11 lists the related details.

**Table 11: Hyper-parameters for the Atari environment wrapper.**
| Hyper-parameter | Value |
| --- | --- |
| Grey-scaling | True |
| Observation down-sampling | (84, 84) |
| Frame stacked | 4 |
| Frame skipped | 4 |
| Action repetitions | 4 |
| Max start no ops | 30 |
| Reward clipping | [-1, 1] |
| Terminal on loss of life | True |
| Max frames per episode | 108K |

### D.4 Simulated labelers

We use Rainbow to train labelers for all environments, with the general hyper-parameters shown in Table 12 and concrete training timesteps in Table 7.
For highway, script labelers are trained with PRExp mentioned in Tables 1 and 5.
For Atari, script labelers are trained with the signed raw rewards.
Note that these script labelers are not necessarily perfect ones since their performance could be improved further with longer training.

**Table 12: Hyper-parameters for Rainbow.**
| Hyper-parameter | Value |
| --- | --- |
| Update period for target network | per 2000 updates |
| Atoms of distribution | 51 |
| $\gamma$ | 0.99 |
| Batch size | 32 |
| Optimizer | Same with Table 6 |
| Max gradient norm | 10 |
| Prioritized replay | exponent: 0.5, correction: 0.4$\rightarrow$1 |
| Noisy nets parameter | 0.1 |
| Warm-up steps | 1.6K |
| Replay buffer size | 1M |
| step $n$ | Same with Table 7 |
| Sample steps to update | 1 |
| Netowrk-Atari | CNN encoder: channels [32, 64], kernel size & strides [5, 5], paddings [3, 1] |
| MLP policy: hidden size [128] |  |
| Netowrk-Highway | MLP encoder: hidden size [128, 128] |
| MLP policy: hidden size [128] |  |

## Appendix E More Details about User Study

### E.1 Evaluation Videos

Evaluation videos for ICoPro-Human are sorted in SupplementaryMaterials/Videos[Env]UserStudy, which learns to perform more like human comparing with evaluation videos shown in SupplementaryMaterials/Videos[Env]Oracles.

### E.2 Procedure of User study

No need to let humans play the game by themselves before providing feedback.
Before letting the human labelers give feedback, we
show them an instruction about the task they will participate (see Sections E.2 and 14),
as well as an example video to let them become familiar with the environment dynamics (see PATH/TO/SupplementaryMaterial]/Video[EnvName]Oracles-Examples).

We show our simple interface used in our user study in Figure 11.

Figure: (a) User interface for Pong.
Refer to caption: x16.png

Table: Table 13: User instruction for Pong.

Table: Table 14: User instruction for highway.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: see above
soundness: 3
strengths: This paper explores an intriguing imitation learning problem in which human demonstrations are imperfect, although some corrective actions may be available. Additionally, there is no explicit reward signal, but certain action preferences (raw reward) are provided.
summary: In this paper, a framework is proposed where a human labeler can offer additional feedback in the form of corrective actions, reflecting the labeler's action preferences, even though this feedback may also be imperfect. Within this context, a novel value-based deep RL algorithm, Iterative Learning from Corrective actions and Proxy rewards (ICoPro), is introduced. This algorithm follows a three-phase cycle: (1) Eliciting sparse corrective actions from a human labeler on the agent's demonstrated trajectories; (2) Integrating these corrective actions into the Q-function using a margin loss to ensure adherence to the labeler's preferences; (3) Training the agent with standard RL losses, augmented with a margin loss to learn from proxy rewards and propagate the Q-values acquired from human feedback.
weaknesses: 1) The performance of the human feedback policy would significantly impact the total performance of the proposed method. For example, if the human feedback is from an expert policy, then the cloning from its behaviors is reasonable and helpful for improving the performance. But if the human feedback is from a sub-optimal or even a random policy, then its behaviors may be harmful. Therefore, what we concern is how the level of sub-optimality of the human feedback policy would impact the performance of the algorithm.
	Currently, in Section 5.3 the authors have simulated the sub-optimal human feedback by inserting noisy actions, but we think such simulation is not enough for the explanation for this problem. Maybe the authors could utilize a half-trained Rainbow policy to generate the imperfect behaviors.

	2) The positions (states) of human feedbacks are also important for the efficiency and effectiveness. We observe that the authors selected the feedback positions with the method in RLIF [1], but we does not see the comparison and discussion with RLIF.
	Besides, to be fair, maybe this trick should also be utilized to other methods for comparison introduced in this paper.

[1] RLIF: Interactive imitation learning as reinforcement learning.

	3) The working scenarios. In our opinion, this method could be seen as a method that online learning while cloning from a specific behavior policy (human feedback). So why such combination could solve the problem of sub-optimality of the proxy reward and the human feedback? For example, if the proxy reward and human feedback make the similar mistake, then such combination would not work. Clearly, in such scenarios, the proposed method could achieve the claimed performance.

	4) In this paper, the authors raised an interesting and inspiring problem. However, the problem lacks of mathematical formulation and theoretical evidence for its effectiveness.

       5)  I found that the methodology adopted in this paper is closely related to those references cited in line 229 and the difference to each of those should be clearly stated. In addition, if the range of reward is not limited, then the loss of  eq.1 could be negative, as this expression is different from that of the traditional margin loss.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Could you please elaborate more practical examples that involving human labelers is necessary?
soundness: 3
strengths: The general idea of the paper is interesting. Involving human labelers in the training process is a very promising way to effectively enhance the training process. The paper is easy to follow.
summary: This paper presents a new pipeline to learn from corrective actions and proxy rewards, which is interesting and useful for the practical use of RL algorithms. The pipeline of the algorithm is reasonable. The experiments show the effectiveness of the proposed methods.
weaknesses: One major weakness is that the experiments are kind of weak to support learning from corrective actions. The paper only provides two environments that are involved by real human labeler (Pong and Highway). Most of the environments are using simulated labeler. More environments that show the necessity to involve human labels would be great to support the idea of the paper.



The writing of the paper could be improved. 

- The abstract could be further condensed. 

- Line 178~185: $\mathcal{S}, \mathcal{A}$ is not defined. The $Q$ function is not formally defined.

- Line 182: Is $\pi^*$ a typo? Do you refer $\pi^*$ to an optimal policy, or greedy policy, or arbirtrary policy?

- Line 353: You should use $a^L \sim $ rather than $a^L=$ if the action is sampled from some $\epsilon$-greedy distribution.

  

The motivation for combining the two sources of signals is good, but the description in the paper (from Line 66~80) does not make sense to me. The paper stated 'In contrast, our key insight is that the two sources of signals can complement each other.. while the effects of suboptimal corrective actions may be weakened by proxy rewards'.  However, the proxy rewards could also be suboptimal, as it's only an approximation of the real reward function.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why the marginal loss in propagation phase works? IIUC the "w/o aTGT" ablation is the experiment ablating the marginal loss in propagation correct? It seems that adding that term can help in terms of the final results. But more analysis is needed. For example, will adding the term improve the learning efficiency and reduce human interventions? In Sec 5.2 the ablation studies only show the performance in tasks but the other metrics such as "human efficiency" is not compared.
2. How to ensure the label quality? In more complex task like driving, human labeler is hard to understand the outcome of their "corrective actions". For example to rescue a car from crashing left car, the human labeler might steer the car all to the right hand side. And this action though seems to be correct, but it's impossible to be applied in real world (you can't rotate the steering wheel immediately from left to right), and it will also cause unknown outcome such as car rollover. There is no guarantee on the quality of the data especially in the offline setting. In motivation the authors suggest this paper can work with suboptimal human labeler but **no experiment shows the method still work on suboptimal experts**.
3. I will expect more discussion on human cost. What designs in the method can improve human efficiency (less corrective actions)? Line 259-261 says "In addition, training with pseudo-labels can
reduce the cost of collecting human labels by leveraging the large number of unlabeled states in
Denv" No experiment supports this claim and I will consider this to be a overclaim. Is there any evidence showing that the agent converge to same performance with less human (here the simulated expert) involvement?
soundness: 3
strengths: 1. Good to see the author provides source code.
2. Human-in-the-loop with offline corrective feedback is a quite interesting setting. Previous work either focus on online demonstration (HGDagger, PVP etc) or offline labeling (PbRL, TAMER), but this work focus on offline corrective action (not the reward labeling but the action).
3. Human-in-the-loop methods suffer from the cost of human subjects, as they need to be very attentive if in a online human-AI shared control setting. This paper adopts the offline setting and utilize the proxy reward, which is an interesting way to scale up the human-in-the-loop method.
summary: This paper proposes to use proxy reward (the reward that is easily defined and might not accurately reflects the desired behavior) and the offline corrective action (labeller assigns correct actions to the rollouts generated by the agent, and those corrective actions will not be used to the environment as the labelling process is offline) to train policies.

The training can be split into 3 phases: data collection, finetune-phase, and propagation-phase. In the data collection phase, agent rolls out trajectories with epsilon-greedy exploration and human provide offline corrective actions. In finetune phase, marginal loss is applied to the Q values to make sure the Q values for corrective actions are high. In propagation phase, RL loss (TD loss with the proxy reward) and the marginal loss (the imitation loss that enforces the learned policy to be closed with previous agent policy) are used together.
weaknesses: 1. Further analysis will greatly improve the technical depth of the paper. Further analysis on ablations and why the method work is needed. These include: why marginal loss in propagation phase works? How well the method work with human labeler in different optimality? The ablations study are presented in Section 5.2 and Table 2 while in Appendix A, while there is no discussion on the reasons. The author just put the results there. See more on "Questions".
2. All ablation studies are conducted in the Atari, and further discussion is needed. Why ablations fail? What insights can we get from the ablation studies?
3. As a paper studying human-in-the-loop policy learning, there is no enough weights on real-human experiments. Section 5.4 shows the experiment with real human subject. I will expecting far more information about this experiment. Why don't put the evaluation results of the trained policies in each iteration of the training and compare it against the baselines? Is there qualitative results showing the human's corrective actions and the agent's actions?
4. Follow question 3, providing corrective actions is actually very boring and time-consuming. The setting might be even more exhausting than online human supervision. I doubt whether the method is applicable at all in real-human setting.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What interface is used for the corrective action process?
2. The entire approach is based on the Q-function, which is limited to discrete action spaces. Can this method be extended to continuous action spaces (without discretization)?
soundness: 3
strengths: The paper is well-organized and easy to follow, and the experimental section provides sufficient evidence to support the authors' claim that their approach improves the learning of an optimal policy.
summary: This paper introduces a novel method that integrates additional (imperfect) corrective actions into the reinforcement learning (RL) process to address limitations in the imperfect proxy reward function.
weaknesses: 1. The reviewer is curious about the performance information available to the labeler that allows them to decide whether an action is good or bad, considering that the queries are sampled from a dataset rather than occurring in real-time. In the experiments, a trained Q-network is used as the labeler, which simplifies the collection of corrective actions. Although there is a user study involving human labelers in two environments (Highway and Pong), this does not fully address my concern.

2. The term "sample-efficient" is mentioned frequently throughout the paper, yet no clear metrics are provided to quantify this efficiency. It would be helpful to include some metrics to validate this claim.

3. Some relevant literature is missing from the discussion. For instance, ranking-based reinforcement learning [1]. The authors could also refer [2], which might offer more effective corrective actions than relying solely on the target Q-network, especially during the early stages when the model is undertrained.

There are a few minor issues with the writing. For example:

In the related work section, the discussion on the limitations of the offline setting is mentioned but not clearly explained. Providing more clarity here would be helpful. Table I includes some abbreviations, such as PR1 and PR2, which I assume stand for Proxy Reward. However, additional explanation would improve understanding for readers unfamiliar with these terms.
   
Reference:
[1] Brown, Daniel S., Wonjoon Goo, and Scott Niekum. "Better-than-demonstrator imitation learning via automatically-ranked demonstrations." Conference on robot learning. PMLR, 2020.
[2] Zhan, Huixin, Feng Tao, and Yongcan Cao. "Human-guided robot behavior learning: A gan-assisted preference-based reinforcement learning approach." IEEE Robotics and Automation Letters 6.2 (2021): 3545-3552.

Meta-review: This paper proposes that human labelers can provide corrective actions and proxy rewards. This is interesting and useful for improving RL algorithms. The algorithm's pipeline is reasonable. The experiments are effective. This algorithm has three phases: (1) human labeling of demonstration trajectories; (2) nudging the corrective action into value estimation; (3) RL with the augmented Q value. The overall framework is novel. The quality is above the bar of ICLR. I hope the authors could improve the presentation and formulation in the camera-ready version.

Decision: Accept (Poster)