Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes to decompose the Q-value as immediate rewards and residual returns (i.e., the sum of rewards from t=1 to T). This decomposition makes the actor can leverage the gradient from the immediate rewards, which is likely to improve performance. The experimental results show a slight performance gain.
strengths_and_weaknesses: ## Strength
The idea proposed in this paper is sensible and novel to me.

## Weakness
How residual critic performs better than the actor-critic method is still unclear to me. The major difference between ARC and AC is that ARC leverages the gradient of the immediate rewards. The author believes that the use of a gradient of immediate rewards is the primary reason for performance gain. Yet, it is hard to believe that the gradient of one step is influential.

One suggestion may be comparing the gradients in a simpler policy class to see how the gradient of ARC and AC differ from the true gradient. That said, I don't demand the author to provide this evidence during the rebuttal. This is more or less a suggestion.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes a reasonable idea and shows positive results. Thus, I recommend for "Weak Accept".
issues: I don't have specific questions for this paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors introduce Actor Residual Critics (ARC) as an alternative to the typical critic structure used in actor-critic RL, where the residual critic learns to estimate only the discounted future reward in exclusion of the immediate reward, which is assumed to be representable by a differentiable function. To aid in imitation learning, they further use ARC with adversarial imitation learning techniques such as those pioneered by Generative Adversarial Imitation Learning (GAIL) where immediate rewards can be estimated by a neural network-based differentiable function. The authors demonstrate that their ARC-based imitation learning methods can outperform prior work in a number of continuous control tasks in both simulation and on real robots.
strengths_and_weaknesses: ## Strengths:
- The core premise of the work is fairly clearly laid out and the paper itself is well written and relatively easy to follow, given at least a cursory understanding of the field.
- I appreciate that the authors start by demonstrating the value of their approach on a simple toy problem in section 5.1 as it helps better establish the fundamental utility of their work.
- There is relatively strong evidence to support the efficacy of the introduced method

## Weaknesses:
- The argument that maximizing the immediate differential reward in Equation 8 can result in a sub-optimal short-sighted policy seems sound to me for the general RL case, however, for the imitation learning case - particularly with methods that attempt to maximize occupancy matching, would the immediate reward not be a measure of how closely the policy matches the expert's behavior? It is curious to me that naive-diff seems to perform so poorly in such cases and furthermore, it is surprising that ARC-based techniques so significantly outperform the expert (which was presumably used for demonstration) on the ant and cheetah tasks. How is such an improvement over the expert even possible if imitation learning was the goal? These are aspects of the method, results and discussion that I feel bear more thorough examination.
- There is a rather significant drop in GAIL's performance on the Hopper task results in Fig. 4, though, unless I missed something, this isn't discussed much. Given my own experience with the algorithm and also its performance on the other tasks, the large drop is concerning and a cause ought to be clearly identified so as to rule out a fundamental issue with the training structure, reference data and/or implementation.
- ARC-AIL being limited only to training with state-action pairs is something that the authors recognize, however, I feel that this limitation is being understated -- the assumption that one typically has access to both for practical imitation learning can be a weak one, particularly if the demonstrator is not action-compatible with the trained agents, i.e. if the action spaces for the expert are different from the agent (eg. in the case of a human demonstrating a tool path for a robot to follow without directly guiding the robot). In such cases, methods that perform occupancy matching with (state, next-state) pairs (such as Adversarial Motion Priors [Peng et al., SIGGRAPH 2021]) might be more effective.
- Training with only 5 random seeds does not seem statistically strong, particularly when the RL community is moving towards improving the statistical significance of results. Results would be stronger if more tests can be conducted, though I also understand that resource and time restrictions may limit this.
- Unless I missed it, it seems that the source(s) of the expert demonstrations is never clearly addressed in the main paper. This is potentially important information for a reader to judge the the algorithm on the merits of the reported performance relative to an expert.
- While I understand that the authors seem to want to draw a clear distinction from their work to differentiable model-based inverse reinforcement learning/imitation learning, I believe more discussion on the subject is warranted in order to give audiences a clearer picture of the methods and relative performances of that class of algorithms.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I believe that the paper is overall sufficiently technically sound and that results are mostly convincing. However, there are a number of issues with the presentation of the work which limit my confidence in it and leave the assertions with clear vulnerabilities that perhaps ought to be better addressed.
issues: 1. Please better address how the trained agents could perform significantly better than the experts.
2. Weaken assertion that having expert state-action pairs is something that is typically reasonable to expect.
3. Offer more clarity in the main paper on how expert data is obtained and used as well as more thorough discussion (in the appendix if space doesn't allow) on the relative performances of the baseline algorithms.
4. Strengthen discussion on related works for fully differentiable model-based imitation learning as this will provide stronger context for readers.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In adversarial imitation learning (e.g. GAIL), the reward function is expressed in terms of the discriminator output and thus differentiable. When an actor-critic method is used for optimizing this reward, the analytic and differentiable form of this reward can be exploited by expressing the Q function as $Q^{\pi}(s, a) = r(s, a) + C^{\pi}(s, a)$ and only learning the future return $C(s, a)$ during policy evaluation. The motivation behind this parameterization is that during policy improvement, the known gradient of $r(s,a)$ is used, while approximation errors only affect the gradient of the future return. The paper presents a theorem to show convergence of policy iteration on the C-function, a theoretical justification arguing that the signal-to-noise ratio of the gradient is larger when learning the C-function, and experimental results on Mujoco and Sim-to-Real transfer to a Fetch robot.
 
strengths_and_weaknesses: The paper is well-written and proposes a simple modification to existing AIL method that can be easily implemented for slightly improving the performance.

However, the quality of submission suffers significantly from overselling, which resulted in 1) (a quite blatant) error, 2) overcomplicating the approach, 3) blurring the contributions:
1) (_Error_:) The paper aims to prove that learning the C function is better (lower variance) than learning the Q function, in general. I argue, that this is actually not the case, as we can easily construct counter-examples, where the gradient of the immediate reward function is misleading, for example when the agent needs to perform several bad-reward-actions to finally reach a highly-desirable final state. I conjecture, that in such settings, learning the C-Function would rather hurt the performance than improve it. In the misguided effort to prove the general advantage of the proposed decomposition in A.2, the paper wrongfully assumes that the SNR of a combined signal corresponds to the sum of the individual signal-to-noise ratios.

2) (_Overcomplication_:) The paper contains a theorem for the sake of containing a theorem. I can not see, why we need a special Bellman-iteration for iterating over $C$. For all, I can tell, we can simply assume the proposed modification as imposing a certain structure on the Q-function ($Q(s,a) = r(s,a) + C(s,a)$) and performing standard updates on the Q-function. This simplicity is a strong selling point! We can modifiy existing Actor-Critic-AIL methods, by adding one line of code in the Q-network. Why do the authors scare people off, by introducing a "novel" policy iteration?

3) (_Blurring the Contribution_:) In my opinion, the main insight is not that we can use knowledge of the reward gradient during policy iteration. I would argue that the reason that people did not propose this in a reinforcement learning setting (where often enough, we also have differentiable reward functions), is that learning the C-function is not beneficial per se. The intriguing bit about the current submission is, that *in the adversarial imitation learning setting, learning the C-function might indeed be beneficial because the reward function is typically shaped ([11])*. Interestingly, also the 1-D example in D.2 which demonstrates lower approximation errors when using the $r$-$C$ decomposition, uses a shaped reward function that is given by the likelihood of the expert polciy (with Gaussian action noise). Based on my interpretation of the mechanics of the proposed method, reward-shaping is a key factor for the benefits of the proposed method that is not even touched in current submission.

The limitations are not well discussed:
- I think that the paper should mention that it does not provide insights on when the C-decomposition help, and when it doesn't. The paper does not provide a solid theoretical understanding on the effects of the decomposition on the sample efficiency. Providing formal results based on certain assumptions would be an important future work. I could, for example, imaging that decreased gradient variance could be formally proven when assuming shaped rewards of the form $r(s, a) = \log \pi^{\star}(a|s)$ 

- The limitation section mentions the problem that actions must be observed, but is quick to dismiss this with the assertion that actions can be typically provided in robotics. I do not think that this is a just assessment. Arguably the most relevant scenarios for providing demonstrations are 1) observing human demonstrations and 2) kinesthetic teaching. For human demonstrations, it is already difficult to map the states to the robot (e.g. by mapping marker positions), but obtaining control commands is hardly possible. For kinesthetic teaching we can measure the joint positions of the robot during demonstrations, but even if the robot is equipped with joint torque sensors, accurately distinguishing external and internal torques is typically not possible. Demonstrating by teleoperation, and demonstrating by running a given controller are to my knowledge the most realistic settings for imitation learning with action, but they are, arguably, of less practical relevant compared to the aforementioned scenarios.


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: While I believe that the paper touches an interesting idea, the current presentation is too misleading and seems to contain a major error in the theoretical motivation (SNR). 

issues: - The paper should provide at least empirical results on when the decomposition helps and **when it does not help**. It would be for example interesting to perform and discuss an experiment that is similar to the car-experiment but with a reward function that has action costs and a final state reward. 
- It needs to be clarified if the proposed method differs in any way from using a Q-network parameterized as $Q_{\theta}(s,a) = r(s,a) + C_{\theta}(s,a)$. If that is the case, it probably does not make sense to introduce a new policy iteration (Theorem 1) could be dropped.
- The limitation section should be improved.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for updating policies using the gradient of the reward function in adversarial imitation learning. The Q function is represented as the sum of the immediate reward and the C function approximates the sum of later rewards, and the gradient of this Q function is used to update policies. Simulations verified the proposed method to outperform standard adversarial imitation learning.
strengths_and_weaknesses: Strengths:
* Proposal for adversarial imitation learning using gradients of an immediate reward in policy updating with model-free RL
* Basic validation with simple toy tasks
* The proposed method's performance was investigated in various simulation tasks

Weaknesses:
* The proposal in this study is not a method for solving a real robot task problem. It is a proposal for general adversarial imitation learning.
* There is not enough explanation as to why the proposed method improves performance. Since the implementation simply considers the gradient of the immediate reward, a little more explanation of the reason is needed.
* I do not understand why GAIL's performance is too low in the Fetch and Jaco tasks in Sections 5.3 and 5.4. I think these tasks are not very difficult and GAIL can learn if the parameter settings are appropriate. I checked the readme of the code to confirm the GAIL parameters, but there was no description of config description for these reaching tasks.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed method is applicable to a wide range of applications because it improves general adversarial imitation learning. The implementation of the proposed method is simple. The appendix is extensive.
issues: See weaknesses above.
review_title: null

Meta-review: metareview: While all reviewers appreciated the idea, there are severe concerns in particular from R3 concerning the validity of the statements and the contribution: 
- In many standard RL settings, the  immediate reward might be misleading. The proposed algorithm seems only to have a benefit if a properly shaped reward signal is used (which is arguably the case for many standard RL benchmarks). Yet, this is not properly discussed in the paper.
- The additional theorem for the new value iteration algorithm seems to unnecessary and an overcomplication
- The contribution is mixed between RL and IRL, where in my opinion the paper makes a stronger case for the IRL setting as here the reward is always shaped. The contribution should be more focused to avoid "washing out the message".
- More seeds are needed
- Comparisons with GAIL need to be clarified (e.g. poor performance of GAIL in some environments). 

The paper is currently borderline so all these issues need to be addressed very carefully.
Rebuttal Update: The paper has been improved considerably after an intensive discussion with the reviewers in the rebuttal phase. All issues have been addressed and reviewers have now a positive opinion on the paper and think it should be published. 

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)