ARC - Actor Residual Critic for Adversarial Imitation Learning
Authors: Ankur Deka
Changliu Liu
Katia P. Sycara
Venue: CoRL 2022 Poster (2022)
Keywords: Adversarial Imitation Learning (AIL)
Actor-Critic (AC)
Actor Residual Critic (ARC)

Abstract: Adversarial Imitation Learning (AIL) is a class  of popular state-of-the-art Imitation Learning algorithms commonly used in robotics. In AIL, an artificial adversary's misclassification is used as a reward signal that is optimized by any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings, the reward in AIL is $differentiable$ but current model-free RL algorithms do not make use of this property to train a policy. The reward is AIL is also $shaped$ since it comes from an adversary. We leverage the differentiability property of the shaped AIL reward function and formulate a class of Actor Residual Critic (ARC) RL algorithms. ARC algorithms draw a parallel to the standard Actor-Critic (AC) algorithms in RL literature and uses a residual critic, $C$ function (instead of the standard $Q$ function) to approximate only the discounted future return (excluding the immediate reward). ARC algorithms have similar convergence properties as the standard AC algorithms with the additional advantage that the gradient through the immediate reward is exact. For the discrete (tabular) case with finite states, actions, and known dynamics, we prove that policy iteration with $C$ function converges to an optimal policy. In the continuous case with function approximation and unknown dynamics, we experimentally show that ARC aided AIL outperforms standard AIL in simulated continuous-control and real robotic manipulation tasks. ARC algorithms are simple to implement and can be incorporated into any existing AIL implementation with an AC algorithm.

ARC- Actor Residual Critic for Adversarial Imitation Learning
Ankur Deka' Changliu Liut, Katia Sycara? Robottcs Institute, Carnegie Mellon University "Intel Labs adekaCalumni cmu. edu, {cliu6 katia) cmu. edu
Abstract: Adversarial Imitation Learning (AIL) is class of popular state-of-the art Imitation Learning algorithms commonly used in robotics. In AIL, an artificial adversary's ' misclassification is used reward signal that is optimized by any standard Reinforcement Learning (RL) algorithm. Unlike most RL settings, the reward in AIL is differentiable but current model-free RL algorithms do not make use of this property to train a policy. The reward is AIL is also shaped since it comes from adversary. We leverage the differentiability property of the shaped AlL reward function and formulate class of Actor Residual Critic (ARC) RL algorithms. ARC algorithms draw parallel to the standard Actor-Crite (AC) al- gorithms in RI. literature and uses a residual critic, C function (instead of the stan- dard Q function) to approximate only the discounted future return (excluding the immediate reward). ARC algorithms have similar convergence properties as the standard AC algorithms with the additional advantage that the gradient through the immediate reward is exact. For the discrete (tabular) case with finite states, actions, and known dynamics, verges to an optimal policy. In the continuous case with function approximation and unknown dynamics, we experimentally show that ARC aided AIL outper- forms standard AIL in simulated continuous-conter and real robotic manipulation tasks. ARC algorithms are simple to implement and can be incorporated into any existing AlL implementation with an AC algorithm. Video and link to code are available at: sites. google. wtt.teew.occceeeeeeeeeeeeeeeeeee http://ttoorrrrua-rrrrreeeee
prove that policy iteration with C' function con-
Adversarial Imitation Learning (AIL), Actor-Critic (AC), Actor Residual Critic (ARC)
Keywords:
1 Introduction
Although Reinforcement Learning (RL) allows us to train agents to perform complex tasks without manually designing controllers [1, 2, 3], is often tedious to hand-craft dense reward function that captures the task objective in robotic tasks [4, 5, 6]. Imitation Learning (IL) or Learning from Demonstration (LfD) is a popular choice in such situations [4. 5, 6, 7J. Common approaches to IL are Behavior Cloning (BC) [8] and Inverse Reinforcement Learning (IRL.) [9].
Within IRL, recent Adversarial Imitation Learning (AIL) algorithms have shown state-of-the-a performance, especially in continuous control tasks which make them relevant to real-world robotics problems. AIL methods cast the problem an adversarial game between a policy and learned adversary (discriminator). The adversary aims to classify between agent and expert trajectories and the policy is trained using the adversary's mis-classification the reward function. This encourages the policy to imitate the expert. Popular AlL algorithms include Generative Adversarial Imitation Learning (GAIL) [10], Adversarial Inverse Reinforcement Learning (AIRL) [11] and f-MAX [12].
The agent in AIL is trained with any standard RL. algorithm. There are two popular categories of RL algorithms: (i) on-policy algorithms such as TRPO [13], PPO [2], GAE [14] based the policy gradient theorem [15, 16]; and (ii) oof-policy Actor-Critic (AC) algorithms such as DDPG [17], TD3 [18], SAC [3] that compute the policy gradient through critic (Q function). These standard RL algorithms were designed for arbitrary scalar reward functions; and they compute an approximate gradient for updating the policy. Practical on-policy algorithms based the policy gradient theorem use several approximations to the true gradient [13, 2, 14] and off-policy AC
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

algorithms first approximate policy return with a critic (Q function) and subsequently compute the gradient through this critic [17, 18. 3]. Even if the Q function is approximated very accurately, the error in its gradient can be arbitrarily large, Appendix A.
Our insight is that the reward function in AIL has 2 special properties: (i) it is differentiable which means we can compute the exact gradient through the reward functiin instead of approximating it and (ii) it is dense/Shaped as it comes from an adversary. As we will see in section 3. naively computing the gradient through reward function would lead to a short-sighted sub-optimal policy. To address this issue, we formulate a class of Actor Residual Critic (ARC) RL algorithms that use a residual critic, C' function (instead of the standard Q function) to approximate only the discounted future return (excluding immediate reward).
The contribution of this paper is the introduction of ARC, which can be easily incorporated to replace the AC algorithm in any existing AlL algorithm for continuous-control and helps boost the asymptotic performance by computing the exact gradient through the shaped reward function.
2
Related Work
Minimized f-Divergence
Algorithm
r(s, a)
Name
Expression
GAIL [10]
Jensen-Shannon
20 2 Egen log gr0tp"
20" + Bp- log poo+p"
log (s,,a)
AIRL [11]. M-MAX-KKI [12]
Reverse KL
Bpn log
o- p
log
D(s.a) D(aa))))
Table I: Popular AIL algorithms, f-iieergence metrics they minimize and their reward functions.
The simplest approach to imitation learning is Behavior Cloning [8] where agent policy directly regresses on expert actions (but not states) using supervised learning. This leads to distribution shift and poor performance at test time [19, 10]. Methods such as DAgger [19] and Dart [20] eliminate this issue but assume an interactive access to an expert policy, which is often impractical.
Inverse Reinforcement Learning (IRL) approaches recover a reward function which can be used to train an agent using RL [9, 21] and have been more successful than BC. Within IRL., recent Ad- versarial Imitation Learning (AIL) methods inspired by Generative Adversarial Networks (GANs) [22] have been extremely successful. GAIL [10] showed state-of-the-arr results in imitation learn- ing tasks following which several extensions have been proposed [23, 24]. AIRL [11] imitates an expert as well as recovers a robust reward function. [25] and [12] presented a unifying view on AlL methods by showing that they minimize different divergence metries between expert and agent state- action distributions but are otherwise similar. [12] also presented generalized AIL method f-MAX which can minimize any specified f-divergence metric [26] between expert and agent state-action distributions thereby imitating the expert. Choosing different divergence metrics leads to different AIL algorithms, e.g. choosing Jensen-Shannon divergence leads to GAIL [10]. [27] proposed method that automatically learns a f-divergence metric to minimize. Our proposed Actor Residual Critic (ARC) can be augmented with any of these AlL algorithms to leverage the reward gradient.
Some recent methods have leveraged the differentiable property of reward in certain scenarios but they have used this property in very different settings. [28] used the gradient of the reward to improve the reward function but not to optimize the policy. We on the other hand explicitly use the gradient of the reward to optimize the policy. [29] used the gradient through the reward to optimize the policy but operated in the model-based setting. If we have access to a differentiable dynamics model, we can directly obtain the gradient of the expected return (policy objective) w.r.t. the policy parameters, Appendix E.5. Since we can directly obtain the objective's ' gradient, we do not necessarily need to use either critic (Q) as in standard Actor Critie (AC) algorithms or a residual critic (C) in our proposed Actor Residual Critic (ARC) algorithms. Differentiable cost (negative reward) has also been leveraged in control literature for long time compute policy, e.g. in LQR [30] and its extensions; but they assume access to a known dynamics model. We on the other hand present model-free method with unknown dynamics that uses the gradient of the reward to optimize the policy with the help of a new class of RL algorithms called Actor Residual Critic (ARC).
3 Background
Objective Our goal is to imitate an expert from one or more demonstrated trajectories (state-action sequences) in continuous-control task (state and action spaces are continuous). Given any Adver-
2

---

sarial Imitation Learning (AIL) algorithm that uses an off-policy Actor-Critic algorithm RL algo- rithm, we wish to use our insight on the availability of a differentiable reward function improve the imitation learning algorithm.
Notation The environment is modeled as Markov Decision Process (MDP) represented as tuple
with state space S, action space A, transition dynamics P S x A x S - [0, 1], reward function r(s, a), initial state distribution po(s), and discount factor y. <-|ss), T (-[s) denote policies and p", p" : S x A + [0, denote state-action occupancy distributions for agent and expert respectively. T {51, 01, $2, ........
. ST, or } denotes a trajectory or episode and (s, s', a') denotes a continuous segment in a trajectory. A discriminator or adversary D(s, a)
tries to determine whether the particular (s, a) pair belongs an expert trajectory or agent trajectory,
Le. (ss, a) P(expert|s a). The optimal discriminator is D(s,a) (1o
0f ("9())))
[22].
Adversarial Imitation Learning (AIL) In AIL, the discriminator and agent are alternately trained. The discriminator trained maximize the likelihood of correctly classifying expert and agent data using supervised learning, (I) and the agent is trained to maximize the expected discounted return, (2).
max E Ea log D(s, a) + Esawp" [log(1 - D(s, D(s,a))l
(1)
max E Es.arpo P Errr(s,aa))
Here, reward rs (s, p (s, a)) a function of the discriminator which varies between differ- AIL algorithms. Different AIL algorithms minimize different f-divergence metrics between ex- pert and agent state-action distribution. Defining f-divergence metric instantiates different reward functions [12]. Some popular divergence choices are Jensen-Shannon in GAIL [10] and Reverse Kullback-Leibler in f-MAX-RKL [12] and AIRL [11] as shown in Table 1.
Any RL algorithm could be used to optimize (2) and popular choices off aff-policy Actor-Critic algorithms such as DDPG [17]. TD3 [18]. SAC [3] and on on-policy algorithms such as TRPO [13], PPO [2]. GAE [14] which based on the policy gradient theorem [15, 16]. We focus off-policy Actor-Critic algorithms as they are usually more sample efficient and stable than on-policy policy gradient algorithms [18, 3).
Continuous-conte using off-policy Actor-Critic The objective in off-policy RL algorithms is to maximize expected Q function of the policy, Q" averaged over the state distribution of a dataset D (typically past states stored in buffer) and the action distribution of the policy æ [31]:
max B amm BewD (s.
(3)
where, (s, = Es
Σ k20
*rth St 8,
The critic and the policy denoted by Q, respectively are approximated by function approximators such as neural networks with parameters & and 0 respectively. There is an additional target Q th.n function parameterized by drang There are two alternating optimization steps:
1. Policy evaluation: Fit critic (Q& function) by minimizing Bellman Backup error.
min s,, (80 a) u(s,a))²
(5)
where, y(8, a) = r(8, a) + 7 ten (s', a') and a' ~ TT(-6"")
(6)
Qø is updated with gradient descent without passing gradient through the target y(8, a).
2. Policy improvement: Update policy with gradient ascent over RL objective.
En-D [VQo Th (.(s))
All off-policy Actor Critic algorithms follow the core idea above ((5) and (7)) along with additional details such as the use of a deterministic policy and target network in DDPG double Q networks and delayed updates in TD3 [18], entropy regularization and reparameterizatioo trick in SAC [3].
3

---

Naive-Diff' and why it won't work Realizing that the reward in AlL is differentiable and shaped, we can formulate a Naive-Diff RL algorithm that updates the policy by differentiating the RL objec- tive (2) with respect to the policy parameters 0.
RT Vor(s1,a)) + Vor(s2, 02) + Vor(os, 4a) + +...]
(8)
T [$1. 01, $2, .2.............................. } is a sampled trajectory in D. Using standard autodiff packages such as Pytorch [32] or Tensorflow [33] to naively compute the gradients in (8) would produce incorrect gradients. Apart from the immediate reward r(s1, a1), all the terms depend on the transition dynam- ics of the environment ((s+11 at), which is unknown and we cannot differentiate through it. So, autodiff will calculate the gradient of only immediate reward correctly and calculate the rest 0's. This will produce short-sighted sub-optimal policy that maximizes only the immediate reward.
4 Method
Q Approximates return
f3
Immediate reward
C- Approximates future return (residue)
Figure 1: Visual illustration of approximating reward via Q function function.
The main lesson we learnt from Naive-Diff is that while we can obtain the gradient of immediate reward, we cannot directly obtain the gradient of future return due to unknown environment dynam- ies. This directly motivates our formulation of Actor Residual Critic (ARC). Standard Actor Critic algorithms use Q function approximate the return described in Eq. 4. However, since we can directly obtain the gradient of the reward, we needn' 't approximate it with a Q function. We, there- fore, propose to use C' function to approximate only the future return, leaving out the immediate reward. This is the core idea behind Actor Residual Critic (ARC) and highlighted word "Residual" refers to the amount of return that remains after subtracting the immediate reward from the return. As we will see in Section 4.3, segregating the immediate reward from future return will allow ARC algorithms to leverage the exact gradient of the shaped reward. We now formally describe Residual Critic (C' function) and its relation to the standard critic (Q function).
Fig. 1. The
4.1 Definition of Residual Critic (C' function)
The Q function under a policy π, Q* a), defined as the expected discounted return from state taking action a, (9). The C' function under policy म, C* (s, (s. is defined as the expected discounted future return, excluding the immediate reward (10). Note that the summation in (10) starts from instead of O. Q function can be expressed in terms of C' function as shown in (11).
as) = BBsaappiT
*rrth
St =8,0
(9)
k20
("((sa) a) = =Ea,ampo
ritk
8t = 8,
(10)
A21
(s, r(s, C* (5,a)
(11)
4.2 Policy Iteration using C function
Algorithm 1: Policy Iteration with C' function
Initialize (s a)vs,, while not converged do // Policy evaluation
for n=1,2,.. until Cx converges do
(s, < E. (('lls,))E. ((alls)) (r(s', +) +0
Vs,a
// Policy improvement
1, if argmax,
(s.a) a)
Vs,a
0. otherwise
Using C' function,
formulate Policy Iteration algorithm shown in Algorithm 1, which is guaranteed to converge to an optimal policy (Theorem 1), similar to the case of Policy Iteration with Q or V function [16]. Other properties of C' function and proof' are presented in Appendix B.

---

4.3 Continuous-control using Actor Residual Critic We can easily extend the policy iteration algorithm with C' function (Algorithm 1) for continuous- control tasks using function approximators instead of discrete C' values and discrete policy (similar to the case of Q function [16]). We call any RL algorithm that uses a policy, π and a residual critic, C' function existing Actor Critic algorithms, we can formulate analogous ARC algorithms. For example, using deterministic policy and larget network as in [17] we can get ARC-DDPG. Using double C net- works (instead of Q networks) and delayed updates as in [18] we can get ARC-TD3. Using entropy regularization and reparameterization trick in [3] we can get ARC-SAC SARC (Soft Actor Residual Critic).
Actor Residual Critic (ARC) algorithm. Using the specific details of different
4.4 ARC aided Adversarial Imitation Learning
To incorporate ARC any Adversarial Imitation Learning algorithm, we simply replace the Ac- tor Critic RI algorithm with an ARC RI. algorithm without altering anything else in the pipeline. For example, we can replace SAC [3] with SARC to get SARC-AIL as shown in Algorithm 2. Implementation-wise this is extremely simple and doesn't require any additional functional parts in the algorithm. The same neural network that approximated Q function can be now be used to approximate C' function.
Algorithm 2: SARC-AIL: Soft Actor Residual Critic Adversarial Imitation Learning
Intialization: Environment (env), Discriminator parameters 3/. Policy parameters A C-function parameters Φ1, 62, dataset of expert demonstrations D replay buffer 'D Target parameters Gurgt 61, Ourga <- d2, Entropy regularization coefficient at; while Max no. of environment interactions not reached do a ~ mo( (.))):):
s', s'r,d= rrd== env.step(a); d= 1 if s' is terminal state, 0 otherwise Store (s, a, s', s',d) in replay buffer D: Update interval reached then
for no. of update steps do
Sample batch B = (s.
Sample batch of expert demonstrations 8 (s, a) ~ Dep; Update Discriminator parameters (1)) with gradient ascent.
v E(an)ee=> (log Dp a]] + E(sn, JeE [log(1 - D (s.a))) a))]
Compute C' targets (ss.a,s'dd B
y(s, a, d) =7 (ro a")s,,") min Coun (8',ā)) a log (a''s'')), á' To(-Is"), ro .') = h(Ds(s,,,
U Update C-functions parameters (21, with gradient descent.
Ve [&] E(na,, . (Co, (s, a) y(s, a, d.))), for i = 1,2
Update policy parameters (0) with gradient ascent.
Ve ELED (rw ā) + min==1.2 (s, - a log ma(ls)) ) ā a~
[o(-)s), (s, (s, a) (s)
Update target networks.
Ouagi
Pourii + (1 - C). for = 1,2; ( controls polyak averaging
4.5
Why choose ARC over Actor-Critic in Adversarial Imitation Learning?
The advantage of using an ARC algorithm over an Actor- Critic (AC) algorithm is that we can lever- age the exact gradient of the reward. Standard AC algorithms use Q₂ to approximate the immediate reward+ + future return and then compute the gradient of the policy parameters through the Qs func- tion (12). This is approximate gradient with no bound on the error in gradient, since the Qs function is an estimated value, Appendix A. On the other hand, ARC algorithms segregate the immediate reward (which is known in Adversarial Imitation Learning) from the future return (which needs to be estimated). ARC algorithms then compute the gradient of policy parameters through the immediate reward (which is exact) and the C' function (which is approximate) separately (13).
Standard AC EEnD (s))
a ~ mo(ss)
(12)
ARC (Our) E.-D (or(s, (VCo
a a~)sss
(13)
In Appendix A.2, we derive the conditions under which ARC is likely to outperform AC by per- forming (Signal to Noise Ratio) SNR analysis similar to [34]. Intuitively, favourable conditions
5

---

for ARC are (i) Error in gradient due to function approximation being similar or smaller for C' as compared to (ii) the gradient of the immediate reward having high negative correlation with the gradient of C' (E War(s, a V. a)] is not highly negative). Under these conditions, ARC would produce higher SN R estimate of the gradient to train the policy. We believe that AlL. likely to present favourable conditions for ARC since the reward is shaped.
ARC would under-perform AC if the error in gradient due function approximation of C' network is significantly higher than that of Q network. In the general RL setting, immediate reward might be misleading (i.e. E Varss, V.C(s, a)] might be negative) which might hurt the performance of ARC. However, we propose using ARC for AIL where the adversary reward measures how closely the agent imitates the expert. In AlL. the adversary reward is dense/shaped making ARC likely to be useful in this scenario, as experimentally verified in the following section.
Results
In Theorem 1, we proved that Policy Iteration with C function converges to an optimal policy. In Fig. 2, experimentally validate this example grid world. The complete details are presented in Appendix 1. In the following sections (5.2, 5.3 and 5.4) we show the effectiveness of ARC aided AIL in Mujoco continuous-control tasks, and simulated and real robotic manipulation tasks. In Appendix D.2, we experimentally illustrate that ARC produces more accurate gradients than AC using a simple ID driving environment. The results are discussed in more detail in Appendix F.
5.1 Policy Iteration on a Grid World
0.59
0.65
0.73
0.81
0.59
0.65
0.73
0.81
0
0
0
0
0.65
0.73
0.81
0.90
0.65
0.73
0.81
0.90
r
0
0.73
0.81
0.90
0.00
0.73
0.81
0.90
1.00
G
0
1
G
0.81 0.90 0.00
G
0.81
0.90
1.00
G
(a) m
(b) r²
(0)
(d) Q"
+
Figure 2: On Grid World, the results of running two Policy Iteration (PI) algorithms Pl with C' function (Algorithm and the standard PT with Q function (Appendix C. Algorithm 3). Both algorithms converge in policy improvement steps to the same optimal policy n as shown in a. The optimal policy gets the immediate reward shown shown b. The C' values at the convergence of PI with C function and the Q values d at the convergence of PI with Q function are consistent with their relation Q' = + C (11). Details are in E.I.
5.2 Imitation Learning in Mujoco continuous-control tasks We used 4 Mujoco continuous-conte environments from OpenAI Gym [35], as shown in Fig. 3. Expert trajectories were obtained by training a policy with SAC [3]. We evaluated the benefit of using ARC with two popular Adversarial Imitation Learning (AIL) algorithms, (ariiihhmssMMMAAA f-MAX-RKL [12] and GAIL [10]. For each of these algorithms, evaluated the performance of standard AIL algorithms (f (f-MAX-RKS GAIL), ARC aided AIL algorithms LARC-f-MAX ---A---XKLLL ARC-GAIL) and Naive-Diff algorithm described in Section (Naive-Diff- fiff--ffffff------ f-MAX-KKL
Difi---ADA......... We also evaluated the performance Behavior Cloning (BC). For standard AIL algorithms (GAIL and f-MAX-RKL) and BC, we used the implementation of [28]. Further experimental details presented in Appendix E.
5.3 Imitation Learning in robotic manipulation tasks
We used simplified 2D versions of FetchReach (Fig. 5a) and FetchPush (Fig. 5b) robotic manipu- lation tasks from OpenAl Gym [35] which have a simulated Fetch robot, [36]. In the FetchReach task, the robot needs to take it's end-effector to the goal (virtual red sphere) quickly possible. In the FetchPush task, the robot's 's needs to push the block to the goal as quickly as possible. We used hand-coded proportional controller to generate expert trajectories for these tasks. Further details are presented in Appendix E.3.
Fig. 4 shows the training plots and Table 2 shows the final performance of the different algorithms. Across all environments and across both the AlL algorithms, incorporating ARC shows consistent improvement over standard AIL algorithms (Table 2). BC suffers from distribution shift at test time [19, 10] and performs very poorly. As we predicted in Section 3, Naive-Diff algorithms don't. perform well as naively using autodiff doesn" 't compute the gradients correctly.

---

(d) Hopper-v2 Figure 3: OpenAl Gym's [35] Mujoco certinuous-connter environments used for evaluation.
(a) Ant-v2
(b) Walker-v2
HalfCheetah-v2
Expert
ARC---Max-RKL (Our)
f-Max-RKL
Naive-Diff f-Max-RKL
BC
ARC-GAIL (Our)
GAIL
Naive-Diff GAIL
Ant
Walker2d
HaltCheetah
Hopper
000
Figure 4: Episode return versus number of environment interaction steps for different Imitation Learning algo- rithms on Mujoco continuous-come control environments.
Method
Ant
Walker2d
HalfCheetah
Hopper
Expert relurn
5926.18 1: 124.56
5344.21 +: 84.45
12427.49 1: 486.38
3592.63
19.21
ARC-/-MaX-RKL (Our)
6306.22 + 95.91
4753.63 = 88.89
12930.51 + 340.02
3433,45
49.48
f-Max-RKL
5949.81 +: 98.75
4069.14 + 52.14
11970.47 +: 145.65
3417.29 + 19.8
Naive-Diff J-Max-RKL
998.27 + 3.63
294.36 31.38
357.05 732.39
154.57 34.7
ARC-GAIL (Our)
6090.19 99.72
3971.25 70.11
11527.76 + 537.13
3392.45
GAIL
5907.98 + 44.12
3573.26 = 98.18
11075.31 + 255.69
3153,84
53.61
Naive-Diil GAIL
998.17 +: 2.22
99.26 + 76.11
277.12 = 523.77
105.3 48.01
BC
615.71 + 109.9
81.04 # 119.68
-392.78 + 74.12
282.44 + 110.7
Table 2: Policy return on Mujoco environments using different Imitation Learning algorithms. Each algorithm is run with 10 random seeds. Each seed is evaluated for 20 episodes.
Fig. 6a shows the training plots and Table 3 under the heading 'Simulation' shows the final per- formance of the different algorithms. In both the FetchReach and FetchPush tasks, ARC aided AIL algorithms consistently outperformed the standard AIL algorithms. Fig. 6b shows the magni- tude of the 2nd action dimension vs. time-step in one episode for different algorithms. The expert initially executed large actions when the end-effeccto/bllock was far away from the goal. As the cnd-eeffecto//block approached the goal, the expert executed small actions. ARC aided AIL algo- rithms
RKL and ARC-GAIL) showed similar trend while standard AIL algorithms (f-Max-RKL and GAIL) learnt a nearly constant action. Thus, ARC aided AIL algorithms were able to better imitate the expert than standard AlL algorithms.
5.4
Sim-to-real transfer of robotic manipulation policies
For testing the sim-to-real transfer of the different trained AIL manipulation policies, we setup JacoReach (Fig. 5c) and JacoPush (Fig. 5d) tasks with a Kinova Jaco Gen 2 arm, similar to the FetchReach and FetchPush tasks in the previous section. The details are presented in Appendix E.4.
Table 3 under the heading 'Real Robot' shows the performance of the different AIL algorithms in the real robotic manipulation tasks. The real robot evaluations showed similar trend as in the simulated tasks. ARC aided AIL consistently outperformed the standard AIL algorithms. Appendix D Fig. visualizes the policies in the JacoPush task showing that ARC aided AIL algorithms were able to push the block closer to the goal compared the standard AIL algorithms. Supplementary slide for these tasks (both our methods
shows videos of the same. Since we didn't tune and the basclines, details in Appendix E.3), is likely that the performances would improve with further parameter tuning. Without fine-tuning hyper-parameters for these tasks, ARC algorithms showed higher performance than the baselines. This shows that ARC algorithms are parameter robust and applicable to real robot tasks without much fine tuning.
7

---

(a) FetchReach Figure 5: Simulated and real robotic manipulation tasks used for evaluation. Simplified 2D versions FetchReach and FetchPush b tasks from OpenAl Gym, [35] with a Fetch robot, [36]. Conreponning JacoRe- ach and JacoPush d tasks with real Kinova Jaco Gen arm, [37].
(b) FeschPush
(e) JacoRcach
(d) JacoPush
ARC-f-Max-RK (Our)
ARC-GAIL (Our)
FMax-RKL
GAIL
Expert
FatchReach
FetchPush
FetchReach
FetchPush
-4
0.0
(1) Episade return vs. interaction steps
(b) Action vs. time step
Figure 6: Episode return vs. number of environment interaction steps for different Adversarial Imitation
Learning algorithms on FetchPush and FetchReach tasks. Magnitude the 2nd action dimension versus time step in single episode for different algorithms.
Simulation
Real Robot
Method
FetchReach
FetchPush
JacoReach
JacoPush
Experi return
.0.58 1:
-1.18 1 0.04
.0.14 1: 0.01
00.7 1: 0.01
ARC- ARC-f-Max-RKe MMMa--KLL (Our)
-143 + 0.08
-2,91 +: 0.25
-0.38 + 0.02
-1.25 :: 0.06
f f-Max
.2.22
ARC-GAIL (Our)
-1.53 0.06
-2.64 + 0.07
-0.46 + 0.01
-1.56 0.08
GAIL
-2.88 0.09
-4.53 1 0.01
-1.05
1 0.06
-2355
0.06
Table 3: Policy return simulated (FetchReach, FetchPush) and real (JacoReach, JacoPush) rohotic
nipulation tasks using different AlI. algorithms. The reward each time step negative distance hetween end-effector & goal for reach tasks and block & goal for push tasks. The reward in the real and simulated tasks are on different scales due to implementation details described in Appendix E.4.
Limitations
Three main limitations in our work are: (1) While many AlL algorithms can be trained using expert "states' only, ARC-AIL can only be trained with 'state-action" (s, a) pairs. There are several scenar- ios where obtaining (s, a) pairs challenging (e.g. kinesthetic teaching). such scenarios, ARC not directly applicable. People often use tricks to mitigate this issue and using (s, a) pairs to train policy is a popular choice [38, 39, 40, 41, 42). (2) ARC-AIL can only work with continuous action space. Most real world robotic tasks have or can be modified to have continuous action space. (3) We haven't explored how the agent-adversee interaction in AIL affects the accuracy of the reward gradient and leave that for future work.
7 Conclusion
We highlighted that the reward in popular Adversarial Imitation Learning (AIL) algorithms are dif- ferentiable but this property has not been leveraged by existing model-free RI algorithms to train policy. Further, they usually shaped. We also showed that naively differentiating the policy through this reward function does not perform well. To solve this issue, we proposed class of Actor Residual Critic (ARC) RL algorithms that use C function as an alternative standard Actor Critic (AC) algorithms which use a Q function. An ARC algorithm can replace the AC algorithm in any existing AIL algorithm. We formally proved that Policy Iteration using C' function converges to an optimum policy in tabular environments. For continuous-contro tasks, using ARC can compute the exact gradient of the policy through the reward function which helps improve the performance of the AIL algorithms in simulated cistinuous-contrur and simulated & real robotic manipulation tasks. Future work can explore the applicability of ARC algorithm to other scenarios which have differentiable reward function.

---

Acknowledgments
We are thankful to Swaminathan Gurumurthy and Tejus Gupta for several insightful discussions on the idea. We are also thankful to Rohit Jena, Advait Gadhikar for their feedback on the manuscript and Dana Hughes, Sushmita Das for their support with some logistics of the project.
Finally, we thankful to the reviewers for their constructive feedback through the rebuttal period, which we believe helped strengthen the paper.
This work has been supported by the following grants: Darpa HR001 120C0036, AFRL/AFOSR FA9550-18-1-0251 and ARL W91 199
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:/3/2.5602 2013.
[2] Schulman, Wolski, Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[3] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[4] C. G. Atkeson and S. Schaal. Robot learning from demonstration. ICML, volume 97, pages 12-20. Citeseer, 1997.
[5] Schaal. Learning from demonstration. In Advances in neural information processing sys- tems, pages 1040-1046, 1997.
[6] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
[7] P. Abbeel, A. Coates, and A. Y. Ng. Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 29: 1608-1639, 2010.
[8] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
[9] A. Y. Ng. S. J. Russell, et al. Algorithms for inverse reinforcement learning, In Icml, volume 1, page 2000.
[10] and S. Ermon. Generative adversarial imitation learning. In Advances neural infor- mation processing systems, pages 4565-4573 2016.
[11] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint 17 12248 2017.
[12] S. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective on imita- tion learning methods. Conference on Robot Learning, pages 1259-1277. PMLR, 2020.
[13] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897 PMLR, 2015.
[14] J. Schulman, Moritz, Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:/506.0244 2015.
[15] R. Williams. Simple statistical gradient-follovie algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256) 1992.
[16] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[17] P. Lillicrap, J. J. Hunt, A. Pritzel, Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02771, 2015.

---

[18] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
[19] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[20] M. Laskey, J. Lee, W. Hsich, R. Liaw, Mahler, R. Fox, and K. Goldberg. Iterative noise injection for scalable imitation learning. In Ist conference on robol learming (Co,L) Sergey Levine and Vincent Vanhoucke and Ken Goldberg), Mountain View, CA, USA, pages 13-15, 2017.
[21] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforce- ment learning. Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[22] I. J. Goodfellow, J. Pouget-Abadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets.
[23] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In NIPS, 2017.
[24] R. Jena. C. Liu, and K. Sycara. Augmenting gail with be for sample efficient imitation learning. In Conference on Robot Learning, pages 80-90. PMLR, 2021.
[25] L. Ke, M. Barnes, W. Sun, G. Lee, Choudhury, and S. Srinivasa. Imitation learning as f-divergence minimization. arXiv preprint arXiv: 1905. / 12888, 2019.
[26] J. Lin. Divergence measures based on the shannon entropy. IEEE Transactions Information theory, 37(1):145-1511 1991.
[27] X. Zhang, Y. Li, Z. Zhang, and Z.-L.. Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. arXiv preprint arXiv:2010.0/2077 2020.
[28] T. Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforce- ment learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021.
[29] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: imagination. In International Conference on Learning Representationss 2019.
Learning behaviors by latent
[30] A. Bemporad, M. Morari, V. Dua, and E. N. Pistikopoulos. The explicit linear quadratic regu- lator for constrained systems. Automatica, 38(1):3 20, 2002.
[31] D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395, PMLR, 2014.
[32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. De Vito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024 8024-8035. Curran Associates, Inc., 2019.
[33] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al. Tensorflow: A system for large-scale machine learning. In /2th (USENIX) symposium on operating systems design and implementation (LOSDI) 16), pages 265-283, 2016.
[34] W. Roberts and Tedrake. Signal-to-noiss ratio analysis of policy gradient algorithms.
[35] G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint vrXiv:1606.01555 2016.
10

---

[36] M. Wise, M. Ferguson, D. King, E. Diehr, and D. Dymesich. Fetch and freight: Standard platforms for service robot applications.
[37] A. Campeau-Lecove
Lamontagne, S. Latour, P. Fauteux, Maheu, Boucher, C. Deguire, and L.-J. C. L'Ecuer. Kinova modular robot arms for service robotics applications. In Rapid Automation: Concepts, Methodologies. Tools, and Applications, pages 693-719. IGI global. 2019.
[38] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto. Visual imitation made easy. In Conference on Robot Learning (CoRL), 2020.
[39] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00758 2020.
[40] Y. Lu, Hausman, Chebotar, Yan, E. Jang, Herzog, T. Xiao, Irpan, M. Khansari, D. Kalashnikov, et al. Aw-opt: Learning robotic skills with imitation andreinforcement at scale. In CoRL., 2021.
[41] O. Scheel, L. Bergamini, M. Wolczyk, B. Osiński, and P. Ondruska. Urban driver: Learning to drive from worldworld demonstrations using policy gradients. In 5th Annual Conference Robot Learning, 2021.
[42] R. Hoque, A. Balakrishna, E. Novoseller, A. Wilcox, D. S. Brown, and K. Goldberg. Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning. In Conference on Robot Learning, pages 598-608. PMLR, 2022.


Reviews:
[{"review_id": "smWL-Eb9VGh", "reviewer": "Reviewer_AjRj", "summary": "In adversarial imitation learning (e.g. GAIL), the reward function is expressed in terms of the discriminator output and thus differentiable. When an actor-critic method is used for optimizing this reward, the analytic and differentiable form of this reward can be exploited by expressing the Q function as $Q^{\\pi}(s, a) = r(s, a) + C^{\\pi}(s, a)$ and only learning the future return $C(s, a)$ during policy evaluation. The motivation behind this parameterization is that during policy improvement, the known gradient of $r(s,a)$ is used, while approximation errors only affect the gradient of the future return. The paper presents a theorem to show convergence of policy iteration on the C-function, a theoretical justification arguing that the signal-to-noise ratio of the gradient is larger when learning the C-function, and experimental results on Mujoco and Sim-to-Real transfer to a Fetch robot.\n ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well-written and proposes a simple modification to existing AIL method that can be easily implemented for slightly improving the performance.\n\nHowever, the quality of submission suffers significantly from overselling, which resulted in 1) (a quite blatant) error, 2) overcomplicating the approach, 3) blurring the contributions:\n1) (_Error_:) The paper aims to prove that learning the C function is better (lower variance) than learning the Q function, in general. I argue, that this is actually not the case, as we can easily construct counter-examples, where the gradient of the immediate reward function is misleading, for example when the agent needs to perform several bad-reward-actions to finally reach a highly-desirable final state. I conjecture, that in such settings, learning the C-Function would rather hurt the performance than improve it. In the misguided effort to prove the general advantage of the proposed decomposition in A.2, the paper wrongfully assumes that the SNR of a combined signal corresponds to the sum of the individual signal-to-noise ratios.\n\n2) (_Overcomplication_:) The paper contains a theorem for the sake of containing a theorem. I can not see, why we need a special Bellman-iteration for iterating over $C$. For all, I can tell, we can simply assume the proposed modification as imposing a certain structure on the Q-function ($Q(s,a) = r(s,a) + C(s,a)$) and performing standard updates on the Q-function. This simplicity is a strong selling point! We can modifiy existing Actor-Critic-AIL methods, by adding one line of code in the Q-network. Why do the authors scare people off, by introducing a \"novel\" policy iteration?\n\n3) (_Blurring the Contribution_:) In my opinion, the main insight is not that we can use knowledge of the reward gradient during policy iteration. I would argue that the reason that people did not propose this in a reinforcement learning setting (where often enough, we also have differentiable reward functions), is that learning the C-function is not beneficial per se. The intriguing bit about the current submission is, that *in the adversarial imitation learning setting, learning the C-function might indeed be beneficial because the reward function is typically shaped ([11])*. Interestingly, also the 1-D example in D.2 which demonstrates lower approximation errors when using the $r$-$C$ decomposition, uses a shaped reward function that is given by the likelihood of the expert polciy (with Gaussian action noise). Based on my interpretation of the mechanics of the proposed method, reward-shaping is a key factor for the benefits of the proposed method that is not even touched in current submission.\n\nThe limitations are not well discussed:\n- I think that the paper should mention that it does not provide insights on when the C-decomposition help, and when it doesn't. The paper does not provide a solid theoretical understanding on the effects of the decomposition on the sample efficiency. Providing formal results based on certain assumptions would be an important future work. I could, for example, imaging that decreased gradient variance could be formally proven when assuming shaped rewards of the form $r(s, a) = \\log \\pi^{\\star}(a|s)$ \n\n- The limitation section mentions the problem that actions must be observed, but is quick to dismiss this with the assertion that actions can be typically provided in robotics. I do not think that this is a just assessment. Arguably the most relevant scenarios for providing demonstrations are 1) observing human demonstrations and 2) kinesthetic teaching. For human demonstrations, it is already difficult to map the states to the robot (e.g. by mapping marker positions), but obtaining control commands is hardly possible. For kinesthetic teaching we can measure the joint positions of the robot during demonstrations, but even if the robot is equipped with joint torque sensors, accurately distinguishing external and internal torques is typically not possible. Demonstrating by teleoperation, and demonstrating by running a given controller are to my knowledge the most realistic settings for imitation learning with action, but they are, arguably, of less practical relevant compared to the aforementioned scenarios.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While I believe that the paper touches an interesting idea, the current presentation is too misleading and seems to contain a major error in the theoretical motivation (SNR). \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- The paper should provide at least empirical results on when the decomposition helps and **when it does not help**. It would be for example interesting to perform and discuss an experiment that is similar to the car-experiment but with a reward function that has action costs and a final state reward. \n- It needs to be clarified if the proposed method differs in any way from using a Q-network parameterized as $Q_{\\theta}(s,a) = r(s,a) + C_{\\theta}(s,a)$. If that is the case, it probably does not make sense to introduce a new policy iteration (Theorem 1) could be dropped.\n- The limitation section should be improved.\n"}}, {"review_id": "6SswZQXypOz", "reviewer": "Reviewer_XgGH", "summary": "This paper proposes a method for updating policies using the gradient of the reward function in adversarial imitation learning. The Q function is represented as the sum of the immediate reward and the C function approximates the sum of later rewards, and the gradient of this Q function is used to update policies. Simulations verified the proposed method to outperform standard adversarial imitation learning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Proposal for adversarial imitation learning using gradients of an immediate reward in policy updating with model-free RL\n* Basic validation with simple toy tasks\n* The proposed method's performance was investigated in various simulation tasks\n\nWeaknesses:\n* The proposal in this study is not a method for solving a real robot task problem. It is a proposal for general adversarial imitation learning.\n* There is not enough explanation as to why the proposed method improves performance. Since the implementation simply considers the gradient of the immediate reward, a little more explanation of the reason is needed.\n* I do not understand why GAIL's performance is too low in the Fetch and Jaco tasks in Sections 5.3 and 5.4. I think these tasks are not very difficult and GAIL can learn if the parameter settings are appropriate. I checked the readme of the code to confirm the GAIL parameters, but there was no description of config description for these reaching tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed method is applicable to a wide range of applications because it improves general adversarial imitation learning. The implementation of the proposed method is simple. The appendix is extensive.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "0s84YcDo-ws", "reviewer": "Reviewer_iG9x", "summary": "The authors introduce Actor Residual Critics (ARC) as an alternative to the typical critic structure used in actor-critic RL, where the residual critic learns to estimate only the discounted future reward in exclusion of the immediate reward, which is assumed to be representable by a differentiable function. To aid in imitation learning, they further use ARC with adversarial imitation learning techniques such as those pioneered by Generative Adversarial Imitation Learning (GAIL) where immediate rewards can be estimated by a neural network-based differentiable function. The authors demonstrate that their ARC-based imitation learning methods can outperform prior work in a number of continuous control tasks in both simulation and on real robots.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The core premise of the work is fairly clearly laid out and the paper itself is well written and relatively easy to follow, given at least a cursory understanding of the field.\n- I appreciate that the authors start by demonstrating the value of their approach on a simple toy problem in section 5.1 as it helps better establish the fundamental utility of their work.\n- There is relatively strong evidence to support the efficacy of the introduced method\n\n## Weaknesses:\n- The argument that maximizing the immediate differential reward in Equation 8 can result in a sub-optimal short-sighted policy seems sound to me for the general RL case, however, for the imitation learning case - particularly with methods that attempt to maximize occupancy matching, would the immediate reward not be a measure of how closely the policy matches the expert's behavior? It is curious to me that naive-diff seems to perform so poorly in such cases and furthermore, it is surprising that ARC-based techniques so significantly outperform the expert (which was presumably used for demonstration) on the ant and cheetah tasks. How is such an improvement over the expert even possible if imitation learning was the goal? These are aspects of the method, results and discussion that I feel bear more thorough examination.\n- There is a rather significant drop in GAIL's performance on the Hopper task results in Fig. 4, though, unless I missed something, this isn't discussed much. Given my own experience with the algorithm and also its performance on the other tasks, the large drop is concerning and a cause ought to be clearly identified so as to rule out a fundamental issue with the training structure, reference data and/or implementation.\n- ARC-AIL being limited only to training with state-action pairs is something that the authors recognize, however, I feel that this limitation is being understated -- the assumption that one typically has access to both for practical imitation learning can be a weak one, particularly if the demonstrator is not action-compatible with the trained agents, i.e. if the action spaces for the expert are different from the agent (eg. in the case of a human demonstrating a tool path for a robot to follow without directly guiding the robot). In such cases, methods that perform occupancy matching with (state, next-state) pairs (such as Adversarial Motion Priors [Peng et al., SIGGRAPH 2021]) might be more effective.\n- Training with only 5 random seeds does not seem statistically strong, particularly when the RL community is moving towards improving the statistical significance of results. Results would be stronger if more tests can be conducted, though I also understand that resource and time restrictions may limit this.\n- Unless I missed it, it seems that the source(s) of the expert demonstrations is never clearly addressed in the main paper. This is potentially important information for a reader to judge the the algorithm on the merits of the reported performance relative to an expert.\n- While I understand that the authors seem to want to draw a clear distinction from their work to differentiable model-based inverse reinforcement learning/imitation learning, I believe more discussion on the subject is warranted in order to give audiences a clearer picture of the methods and relative performances of that class of algorithms.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe that the paper is overall sufficiently technically sound and that results are mostly convincing. However, there are a number of issues with the presentation of the work which limit my confidence in it and leave the assertions with clear vulnerabilities that perhaps ought to be better addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Please better address how the trained agents could perform significantly better than the experts.\n2. Weaken assertion that having expert state-action pairs is something that is typically reasonable to expect.\n3. Offer more clarity in the main paper on how expert data is obtained and used as well as more thorough discussion (in the appendix if space doesn't allow) on the relative performances of the baseline algorithms.\n4. Strengthen discussion on related works for fully differentiable model-based imitation learning as this will provide stronger context for readers."}}, {"review_id": "-SZ5BJ6xJbh", "reviewer": "Reviewer_FH78", "summary": "This paper proposes to decompose the Q-value as immediate rewards and residual returns (i.e., the sum of rewards from t=1 to T). This decomposition makes the actor can leverage the gradient from the immediate rewards, which is likely to improve performance. The experimental results show a slight performance gain.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\nThe idea proposed in this paper is sensible and novel to me.\n\n## Weakness\nHow residual critic performs better than the actor-critic method is still unclear to me. The major difference between ARC and AC is that ARC leverages the gradient of the immediate rewards. The author believes that the use of a gradient of immediate rewards is the primary reason for performance gain. Yet, it is hard to believe that the gradient of one step is influential.\n\nOne suggestion may be comparing the gradients in a simpler policy class to see how the gradient of ARC and AC differ from the true gradient. That said, I don't demand the author to provide this evidence during the rebuttal. This is more or less a suggestion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a reasonable idea and shows positive results. Thus, I recommend for \"Weak Accept\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "I don't have specific questions for this paper. ", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)