Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations
Authors: Albert Wilcox
Ashwin Balakrishna
Jules Dedieu
Wyame Benslimane
Daniel S. Brown
Ken Goldberg
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
sparse reward reinforcement learning
reinforcement learning from demonstrations
online reinforcement learning

Abstract: Providing densely shaped reward functions for RL algorithms is often exceedingly challenging, motivating the development of RL algorithms that can learn from easier-to-specify sparse reward functions. This sparsity poses new exploration challenges. One common way to address this problem is using demonstrations to provide initial signal about regions of the state space with high rewards. However, prior RL from demonstrations algorithms introduce significant complexity and many hyperparameters, making them hard to implement and tune. We introduce Monte Carlo Actor-Critic (MCAC), a parameter free modification to standard actor-critic algorithms which initializes the replay buffer with demonstrations and computes a modified $Q$-value by taking the maximum of the standard temporal distance (TD) target and a Monte Carlo estimate of the reward-to-go. This encourages exploration in the neighborhood of high-performing trajectories by encouraging high $Q$-values in corresponding regions of the state space. Experiments across $5$ continuous control domains suggest that MCAC can be used to significantly increase learning efficiency across $6$ commonly used RL and RL-from-demonstrations algorithms. See https://sites.google.com/view/mcac-rl for code and supplementary material.

Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations
Albert Wilcox
Ashwin Balakrishna UC Berkeley
UC Berkeley albertwilcoxxberkeeee.y. edu
edu
Jules Dedieu UC Berkeley gilles_dedieubberkeee edu
Wyame Benslimane
UC Berkeley
wyame bes.eeeeeesmmmmneeeeeeeeelll/// edu
Daniel S. Brown University Utah dsbrown@cs www.tthacccm edu
Ken Goldberg
UC Berkeley
goldberg@berrkeloy. edu
Abstract
Providing densely shaped reward functions for RL algorithms is often exceedingly challenging, motivating the development RL algorithms that can learn from casier-to-specify sparse reward functions. This sparsity poses new exploration challenges. One common way to address this problem is using demonstrations provide initial signal about regions of the state space with high rewards. However, prior RL from demonstrations algorithms introduce significant complexity and many hyperparameters, making them hard implement and tune. We introduce Monte Carlo augmented Actor-Critt Critic (MCAC), parameter free modification to standard actor-critic algorithms which initializes the replay buffer with demonstra- tions and computes a modified Q-value by taking the maximum of the standard temporal distance (TD) target and Monte Carlo estimate of the reward-to go. This encourages exploration in the neighborhood of high-performing trajectories by encouraging high Q-values in corresponding regions of the state space. Exper- iments across 5 continuous control domains suggest that MCAC can be used to significantly increase learning efficiency across 6 commonly used RL and RL-from- demonstrations algorithms. See https: :ttps://ssii /sites....s////////////e/////tt/ google. com/ wwwi//wwwwwceieeeeeeiieiie/ee//e rl for code and supplementary material.
Introduction
Reinforcement learning has been successful learning complex skills in many environments [Mnih et al., 2015, Silver et al., 2016, Schulman et al., 2017], but providing dense, informative reward functions for RI. agents is often very challenging [Xie et al., 2019, Krakovna al., 2020, Wu et al., 2020]. This is particularly challenging for high-dimensional control tasks, in which there may be large number of factors that influence the agent's objective. In many settings, may much easier to provide sparse reward signals that simply convey high-level information about task progress, such as whether an agent has completed a task or has violated a constraint. However, optimizing RL policies given such reward signals can be exceedingly challenging, sparse reward functions may not be able to meaningfully distinguish between a wide range of different policies.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

This issue can be mitigated by leveraging demonstrations, which provide initial signal about desired behaviors. Though demonstrations may often be suboptimal in practice, they should still serve to encourage exploration in promising regions of the state space, while allowing the agent to explore behaviors which may outperform the demonstrations. Prior work has considered a number of ways to leverage demonstrations to improve learning efficiency for reinforcement learning, including by initializing the policy to match the behavior of the demonstrator [Rajeswaran et al., 2018, Peng et al., 2019], using demonstrations to explicitly constrain agent exploration [Thananjeyan et al., 2020, 2021, Wilcox et al., 2021], and introducing auxiliary losses to incorporate demonstration data into policy updates [Nair et al., 2018, Hester et al., 2018, Gao et al., 2018]. While these algorithms have shown impressive performance in improving the sample efficiency of RL algorithms, they add significant complexity and hyperparameters, making them difficult to implement and tune for different tasks.
We present Monte Carlo augmented Actor-Critic (MCAC), which introduces an pasy-to-implement, but highly effective, change that can be readily applied to existing actor-critic algorithms without the introduction of any additional hyperparameters and only minimal additional complexity. The idea to encourage initial optimism in the neighborhood of successful trajectories, and progressively reduce this optimism during learning so that it can continue to explore new behaviors. To operationalize this idea, MCAC introduces two modifications to existing actor-critic algorithms. First, MCAC initializes the replay buffer with task demonstrations. Second, MCAC computes a modified target Q-value for critic updates by taking the maximum of the standard temporal distance targets used in existing actor critic algorithms and a Monte Carlo estimate of the reward-to- -togo. The intuition is that Monte Carlo value estimates can more effectively capture longer-ten term reward information, making it possible to rapidly propagate reward information from demonstrations through the learned Q-function. This makes it possible to prevent underestimation of values in iighpeeforming trajectories early in learning. as high rewards obtained later in a trajectory may be difficult to initially propagate back to earlier states with purely temporal distance targets [Wright al., 2013]. Experiments on five continuous control domains suggest that MCAC is able to substantially accelerate exploration for both standard RL algorithms and recent RL from demonstrations algorithms in sparse reward tasks.
2 Related Work
2.1 Reinforcement Learning from Demonstrations
One standard approach for using demonstrations for RL first uses imitation learning [Argall et al., 2009] to pre-train policy, and then fine-tunes this policy with on-policy reinforcement learning algorithms [Schaal et al., 1997, Kober and Peters, 2014, Peng et al., 2019, Rajeswaran et al., 2018). However, initializing with suboptimal demonstrations can hinder learning and using demonstrations to initialize only a policy inefficient, since they can also used for Q-value estimation.
Other approaches leverage demonstrations to explicitly constrain agent exploration. Thananjeyan et al. [2020] and Wilcox et [2021] propose model-based RL approaches that uses suboptimal demon- strations to iteratively improve performance by ensuring consistent task completion during learning. Similarly, Jing et al. [2020] also uses suboptimal demonstrations to formulate a coft-constraint on exploration. However, a challenge with these approaches is that they introduce substantial algo- rithm complexity, making it difficult to tune and utilize these algorithms in practice. For example, while Thananjeyan et al. [2020] does enable iterative improvement upon suboptimal demonstrations, they require learning a model of system dynamics and a density estimator to capture the support of successful trajectories making it challenging to scale to high-dimensional observations.
Finally, many methods introduce auxiliary losses to incorporate demonstrations into policy up- dates [Kim et al., 2013, Gao et al., 2018, Kang et al., 2018J. Deep Deterministic Policy Gradients from Demonstrations (DDPGID) [Vecerik et al., 2018] maintains all the demonstrations in separate replay buffer and uses prioritized replay to allow reward information to propagate more efficiently. Nair et al. [2018], and Hester al. [2018] use similar approaches, where demonstrations are main- tained separately from the standard replay buffer and additional policy losses encourage imitating the behavior the demonstrations. Meanwhile, RL algorithms such as AWAC [Nair et al., 2021] I] pretrain using demonstration data to constrain the distribution of actions selected during online exploration. While these methods often work well in practice, they often increase algorithmic complexity and introduce several additional hyperparameters that are difficult and time consuming to tune. By
2

---

contrast, MCAC does not increase algorithmic complexity, is parameter-free, and can easily be wrapped around any existing actor-critic algorithm.
2.2 Improving Q-Value Estimates
The core contribution of this work is an mas-tto-implement, yet highly effective, method for stabilizing actor-critic methods for sparse reward tasks using demonstrations and augmented Q-value target. There has been substantial literature investigating learning stability challenges in off-policy deep Q-learning and actor-critic algorithms. See Van Hasselt et al. [2018] for more thorough treatment of the learning stability challenges introduced by combining function approximation. bootstrapping, and off- policy learning, as well as prior work focused on mitigating these issues.
One class of approaches focuses on developing new ways to compute target Q-values [Konidaris et al., 2011, Wright et al., 2013, Van Hasselt et al., 2016, Schulman et al., 2016]. Van Hasselt et al. [2016] computes target Q-values with two Q-networks, using one to select actions and the other to measure the value of selected actions, which helps to prevent the Q-value over-estimation commonly observed in practice in many practical applications of Q-learning. TD3 [Fujimoto et al., 2018] attempts to address overestimation errors by taking the minimum of two separate Q-value estimates, but this can result in underestimation of the true Q-value target. Kuznetsov et al. [2020] uses an ensemble of critics to adaptively address estimation errors in Q-value targets, but introduces number of hyperparameters which must be tuned separately for different tasks. Bellemare et al. [2016] and Ostrovski et al. [2017] consider linear combination of a TD-1 target and Monte Carlo target. Wright et al. [2015], Schulman et al. [2016] and Sharma et al. [2018) consider a number of different estimators of policy's value via n-step returns, which compute Q-targets using trajectories with contiguous transitions followed by a terminal evaluation of the Q-value after n steps. Each of these targets make different bias and variance tradeoff's that can affect learning dynamics.
Similar to our work, Wright et al. [2013] explore the idea of taking a maximum over a bootstrapped target Q-value (TD-1 target) and Monte Carlo estimate of the return-to-go improve fitted Q- iteration. However, Wright et al. [2013] focuses on fully offline Q-learning and only considers simple low-dimensions control tasks using Q-learning with linear value approximation. There numerous reasons to extend these ideas to online deep RL. First, deep RI. algorithms are often unstable, and using the ideas in Wright et al. [2013] to improve Q estimates is promising way to alleviate this, as we empirically verify. Second, while offline learning has many important applications, online RI. is far more widely studied, and we believe is useful study the effects these ideas have in this setting. To the best of our knowledge, MCAC is the first application of these ideas to online actor-critic algorithms with deep function approximation, and we find that it yields surprising improvements in RL performance on complex high-dimensional continuous control tasks.
3 Problem Statement
We consider a Markov Decision Process (MDP) described by tuple (S,A,,, with state set S, an action set A, a transition probability function p : S A x S + [0, 11, a reward function r: S x A- + R. discount factor 7, and finite time horizon T. In each state 8, E S the agent chooses an action a, E A and observes the next state 8t-1 ~ p(- a,) and a reward r(s,, 4:) E R. The agent acts according T (apls;). The agent's 's goal is to find the policy x² which at any given St E S maximizes the expected discounted sum of rewards,
policy π, which induces probability distribution over A given the state,
T
Ert(ss an)
π arg max ENow π
(3.1)
where 7 = (sp, ao, 81, ai..........................aa ST) and T ~ x indicates the distribution of trajectories induced by evaluating policy in the MDP.
We make additional assumptions specific to the class of problems we study, First, we assume that all transitions in the replay buffer are elements of complete trajectories; this is a reasonable assumption as long as all transitions are collected from rolling out some policy in the MDP. Second, we assume the agent has access to an offline dataset Dattine of (possibly suboptimal) task demonstrations. Finally, we focus on settings where the reward function sparse in the sense that most state transitions do not induce change in reward, causing significant exploration challenges for RL algorithms.
3

---

4 Preliminaries: Actor-Critic Algorithms
For given policy x, its state-action value function Q" is defined as
T
= ETOE
(kk ax)
(4.1)
Actor-critic algorithms learn a sample-based approximation to Q", denoted Q5. and a policy ne which selects actions to maximize Q7, with function approximator (typically a neural network) parameterized by 0 and e respectively. During the learning process, they alternate between regressing Q7 to predict Q" and optimizing & to select actions with high values under Q7.
Exactly computing QT targets to train Q is typically intractable for arbitrary policies in continuous MDPs, motivating other methods for estimating them. One such method is to simply collect trajec- tories (st, at, rt. $t+1.... ST oT 11 ST) by executing the learned policy Tz. from state st. computing a Monte Carlo estimate of the reward-to-go defined as follows:
(5t; 01)
Σ kunt
* trkk,
(4.2)
and fitting Q7 to these targets.
However, Que can be a high variance estimator of the reward-to-go [Schulman et al., 2016, Sutton and Barto, 2018]. motivating the popular one-step temporal difference target (TD-1 target) to help stabilize learning:
Queeett (st, ar) r(s ae) + 7Qg (s1+1, 0c+1);
(4.3)
where ar+1 To (5(()+)), )), which is recursively defined based only single (st, az, 8t+1,7e) Te) transi- tion. Here is the parameters also been interest in computing TD-n targets, which instead sum rewards for timesteps and then use Q-values from step n + 1 [Wright et al., 2015, Schulman et al., 2016, Sharma et al., 2018].
lagged target network as in [Van Hasselt et al., 2016]. There has
5
Monte Carlo augmented Actor-Critic
5.1 MCAC Algorithm
The objective of MCAC is to efficiently convey information about sparse rewards from suboptimal demonstrations to Q in order to accelerate policy learning while still maintaining learning stability. To do this, MCAC combines two different methods of computing targets for fitting Q-functions to enable efficient value propagation throughout the state-action space while also learning a Q-value estimator with low enough variance for stable learning. To operationalize this idea, MCAC defines new Q-function target for training Q by taking the maximum of the Monte Carlo Q-target (eq 4.2) and the temporal difference Q-target (eq 4.3): max QQD (st, at), Q (st. ot)]-
The idea here is that early in learning, Q-function trained only with temporal difference targets will have very low values throughout the state-action space as may be very difficult propagate information about delayed rewards through the temporal difference target for long-horizon tasks with sparse rewards [Van Hasselt et al., 2018]. On the other hand, the Monte Carlo Q-target can easily capture long-term rewards, but often dramatically underestimates Q-values for poorly performing trajectories [Wright et al., 2013]. Thus, taking a maximum over these two targets serves to initially boost the Q-values for transitions near high performing trajectories while limiting the influence of underestimates from the Monte Carlo estimate.
MCAC can also be viewed as a convenient way to balance the bias and variance properties of the Monte Carlo and temporal difference Q-targets. The Monte Carlo Q-target is well known to be an unbiased estimator of policy return, but have high variance [Wright et al., 2013]. Conversely, temporal difference targets are known to be biased, but have much lower variance [Wright et al., 2013, Van Hasselt et al., 2016). Thus, as the temporal difference target is typically negatively biased (an underestimate of the true policy return) on successful trajectories early in learning due to the challenge of effective value propagation, computing the maximum of the temporal difference and

---

Algorithm Monte Carlo augmented Actor-Critic
Require: Offlin dataset Doftine-
Require: Total training episodes N, batch size M, Pretraining Steps Np
Require: Episode time horizon T. Initialize replay buffer R. := Dfffine
2: Initialize agent To and critic Q2 using data from Doffline
3: for E (I...... ........ N2 do
4: Sample B Ç R. such that - M.
5: Optimize Q7 on B to minimize loss in eq (5.4). Optimize policy T to maximize Qo-
6: end for
7:
for E (I,..., N} do
8: Initialize episode buffer & -0.
9: Observe state 8].
10:
for j E {1, (_,..., T do
11:
Sample and exccute of ~ To(s)), observing s)+1,
12:
+ ($), aj. +.,
13:
E (T
14:
Sample B R such that |B| M.
15:
Optimize Qī on to minimize loss in eq (5.4). Optimize policy as to maximize Qs.
16:
end for
17:
for Tj E do
18:
Compute Quees (5), a()) in eq (5.2).
19: Ti + (s), aj 8) $411 GMC (6). a)).
20: end for
21:
R < RUE.
22:
end for
Monte Carlo targets helps push the MCAC target value closer the true policy return. Conversely, the Monte Carlo targets are typically pessimistic on unsuccessful trajectories, and their high variance makes difficult for them to generalize sufficiently to distinguish between unsuccessful trajectories that are close to being successful and those that are not. Thus, computing the maximum of the temporal difference and Monte Carlo targets also helps to prevent excessive pessimism in evaluating unsuccessful trajectories. Notably, MCAC does not constrain its Q-targets explicitly based on the transitions in the demonstrations, making it possible for the policy to discover higher performing behaviors than those in the demonstrations as demonstrated in Section 6.
5.2 MCAC Practical Implementation
MCAC can be implemented as wrapper around any actor-critic RL algorithm; we consider options in experiments (Section 6). MCAC starts with an offline dataset of suboptimal demonstrations D Domline. which are used initialize a replay buffer R. Then, during each episode i, we collect full trajectory r', where the j"h transition (s), al,, +/+1- r3) in r is denoted by TJ.
Next, consider any actor-critic method using a learned Q function approximator Qo(s,, az) that, for a given transition 7/ (e), a'j 8)+1, rf) E ç updated by minimizing the following loss:
J(Ø) = E (Qo(s), az), Q Qtarged (s) a)).
(..1)
where is arbitrary differentiable loss function and Qiarget is the target value for regressing Qo. We note that Quarett is defined by the choice actor-critic method. To implement MCAC, we first calibrate the Monte Carlo targets with temporal difference targets (which provide infinite-horizon Q estimates) by computing the infinite horizon analogue of the Monte Carlo target defined in Equation 4.2, which assumes the last observed reward value will repeat forever and uses this add infinite sum of discounted rewards, and is given by
Cuanget (s), -
T-j+1
rt 1-7
+
r at))
(5.2)
5

---

Then, we simply replace the target with maximum over the original target and the Monte Carlo target defined in Equation 5.2, given by
MMCA (s), a}) max Q a²), Omecn (s) (s),
(5.3)
This results in the following loss function for training Qo:
[( = e (Qo(s), a²),, McAC az))
(5.4)
The full MCAC training procedure (Algorithm 1) alternates between updating Q3 using the method described above, followed by optimizing the policy Ts using any standard policy update method.
Experiments
In the following experiments study (I) whether MCAC enables more efficient learning when built on top of standard actor-critic RL algorithms and (2) whether MCAC can applied to improve prior algorithms for RL from demonstrations. See the supplementary material for code and instructions on how to rum experiments for reproducing all results in the paper and additional experiments studying the impact of demonstration quantity, quality, and other algorithmic choices such whether to pretrain learned networks on demonstration data before online interaction.
6.1 Experimental Procedure
All experiments were run set of 24 Tesla V100 GPUs through combination of Google Cloud resources and a dedicated lab server. We aggregate statistics over 10 random seeds for all experiments, reporting the mean and standard error across the seeds with exponential smoothing. Details on hyperparameters and implementation details are provided the supplementary material.
6.2 Domains
We consider the five long-horizon continuous control tasks shown in Figure 1. relatively sparse rewards, making demonstrations critical for performance. We found that without demonstrations, SAC and TD3 made little to no progress on these tasks.
All tasks have
Pointmass Navigation: The first domain is a pointmass 2D navigation task (Figure I(a)) with time horizon 7' = 100, where the objective is to navigate around the red barrier from start set S to a goal set & by executing 2D delta-position controls. If the agent collides with the barrier it receives reward of 100 and the episode terminates. At each time step. the agent receives reward of - if it not in the goal set and if it is in the goal set. To increase the difficulty of the task, we perturb the state with zero-mean Gaussian noise at each timestep. The combination of noisy transitions and sparse reward signal makes this a very difficult exploration task where the agent must learn to make through the slit without converging to the local optima of avoiding both the barrier and the slit.
The demonstrator is implemented starting set to the slit, through the slit, and to the goal set. The actions are clipped to fit in the action space, and trajectories are nearly optimal. The agent provided with 20 demonstrations.
series of proportional controllers which guide it from the
Object Manipulation in MuJoCo: We next consider two object manipulation tasks designed in the MuJoCo physics simulator [Todorov al., 2012], where the objective extract block from tight configuration on a table (Block Extraction, Figure 1(b)) and push each of 3 blocks forward on the plane (Sequential Pushing, Figure ((c)). In the Block Extraction task, the action space consists of 3D delta position controls and an extra action dimension to control the degree to which the gripper opened. the Sequential Pushing environment, this extra action dimension is omitted and the gripper is kept closed. In the Block Extraction domain, the agent receives a reward of -1 for every timestep that it hasn't 't retrieved the red block and 0 when it has. In the Sequential Pushing domain, the reward increases by 1 for each block the agent pushes forward. Thus, the agent receives a reward of -3 when has made no progress and 0 when has completed the task. The Block Extraction task is adapted from Thananjeyan al. [2021] while the Sequential Pushing task is adapted from Wilcox al. [2021]. J. We use a time horizon of = for the Block Extraction task and longer T 150 for the Sequential Pushing task since it is of greater complexity.

---

(a) Pointmass Navigation
(b) Block Extraction
(c) Sequential Pushing
(d) Door Opening
(e) Block Lifting
Figure I: MCAC Domains: We evaluate MCAC on five continuous control domains: a pointmass navigation environment, and four high-dimensional robotic control domains. All domains are associated with relatively unshaped reward functions, which only indicate constraint violation, task completion, or completion of a subtask.
The block extraction demonstrator is implemented as series of proportional controllers guiding the arm to position to grip the block, followed by an instruction to close the gripper and a controller to lift. We provide the agent with 50 demonstrations with zero-mean noise injected in the controls to induce suboptimality. For the sequential pushing environment, the demonstrator uses a series of proportional controllers slowly push one block forward, move backwards, line up with the next block, and repeat the motion until all blocks have been pushed. Because moves slowly and moves far back from each block it pushes, demonstrations are very suboptimal. For Sequential Pushing, the agent is provided with 500 demonstrations due to the increased difficulty of the task.
Robosuite Object Manipulation: Finally, we consider two object manipulation tasks built on top of Robosuite [Zhu al., 2020), collection of robot simulation tasks using the MuloCo physics engine. We consider the Door Opening task (Figure I(d)) and the Block L Lifting task (Figure I(e)). In the Door Opening task, Panda robot with 7 DoF and parallel-jaw gripper must door in order to open it. The door's ' location is randomized at the start of each episode. The agent receives reward of if it has not opened the door and a reward of if has. In the Block Lifting task, the same Panda robot is placed in front of a table with single block on its surface. The robot must pick up the block and lift above a certain threshold height. The block's location randomized for each episode and the agent receives reward of -1 for every timestep it has not lifted the block and reward of 0 when it has. Both Robosuite tasks use a time horizon of 50.
the handle of
For both Robosuite tasks, demonstrators are trained using SAC on a version of the task with a hand-designed dense reward function, in the Robosuite benchmarking experiments [7hu al., 2020]. In order to ensure suboptimality, we stop training the demonstrator policy before convergence. For each Robosuite environment we use the trained demonstrator policies generate 100 suboptimal demonstrations for training MCAC and the baselines.
6.3 Algorithm Comparisons
We empirically evaluate the following baselines both individually and in combination with MCAC. All methods are provided with the same demonstrations which are collected as described in Section 6.2. See the supplement for more in depth details on implementation and training.
Behavior Cloning: Direct supervised learning the offline suboptimal demonstrations.
Twin Delayed Deep Deterministic Policy Gradients (TD3) [Fujimoto et al., 2018]: State of the art actor-critic algorithm which trains a deterministic policy to maximize learned critic.
Soft Actor-Critic (SAC) [Haarnoja et al., 2018]: State of the art actor-critic algorithm which trains a stochastic policy which maximizes a combination of the Q value of the policy and the expected entropy of the policy encourage exploration.
Generalized Q Estimation (GQE): A complex return estimation method from Schulman et al. [2016] for actor-critic methods, which computes weighted average TD-i estimates for range i. i. GQE is implemented on top of SAC (see supplement for more details). We tume the range of horizons considered and the eligibility trace value (corresponding to À in GAB).
Overcoming Exploration from Demonstrations (OEFD) [Nair et al., 2018]: OEFD builds on DDPG [Lillicrap et al., 2015] by adding loss which encourages imitating demonstrations and
7

---

learned filter which determines when to activate this loss. Our implementation does not include hindsight experience replay since it is not applicable to most of our environments.
Conservative Q Learning (CQL) [Kumar et al., 2020]: A recent offline RI. algorithm that addresses value overestimation with conservative function. Here also update CQL online after pre- training offline on demonstrations in order to provide fair comparison with other algorithms,
Advantage Weighted Actor-Critic (AWAC) [Nair et al., 2021]: A recent offline reinforcement learning algorithm designed for fast online fine-tuning.
We also implement versions of each of the above RL algorithms with MCAC (TD3 + MCAC, SAC + MCAC, GQE MCAC, OEFD MCAC, CQL + MCAC, AWAC MCAC).
The behavior cloning comparison serves to determine whether online learning is heneficial in general, while the other comparisons study whether MCAC can be used accelerate reinforcement learning for commonly used actor-critic algorithms (TD3, SAC, and GQE, which essentially SAC with complex returns) and for recent algorithms for RL from demonstrations (OEFD, CQL and AWAC),
6.4 Results
In Section 6.4.1, we study MCAC on a simple didactic environment to better understand how it affects the learned Q-values. We then study whether MCAC can be used accelerate exploration on number of continuous control domains. In Section 6.4.2, we study whether MCAC enables more efficient learning when built on top of widely used actor-critic RL algorithms (SAC, TD3, and GQE), Then in Section 6.4.3, we study whether MCAC can provide similar benefits when applied recent RL from demonstration algorithms (OEFD, CQL, and AWAC). Additionally, in the supplement we provide experiments involving other baselines, investigating the sensitivity of MCAC to the quality and quantity of demonstration data, and investigating its sensitivity to pretraining.
6.4.1 MCAC Didactie Example
In order to better understand the way MCAC affects Q estimates, we visualize Q estimates when MCAC is applied to SAC after 50000 timesteps of training in the Pointmass Navigation environment, in Figure 2. Here we visualize Q-values for the entire replay buffer, including offline demonstrations,
When training without MCAC (top row), the agent is unable to learn a useful Q function and thus does learn complete the task (the only successful trajectories shown are offline data). However, even when this is the case, the MCAC estimate is able to effectively propagate reward signal backwards along the demonstrator trajectories, predicting higher rewards early on (top right). We see that the GQE estimates (top middle) are somewhat more effective than the Bellman ones at propagating
Extimate
SAC Agent
Demoustrator Escectoriss
Ageui
-000
Figure 2: MCAC Replay Buffer Visualization: Scatter plots showing Bellman, GQE and MCAC Q estimates on the entire replay buffer, including offline demonstrations, for SAC learners with and without the MCAC modification after 50000 timesteps of training. The top row shows data and estimates obtained while training a baseline SAC agent without MCAC, while the bottom row shows same when SAC trained with MCAC. The left column shows Bellman Q estimates on each replay buffer sample while the middle columa shows GQE estimates and the right column shows MCAC estimates. Results suggest that MCAC is helpful for propagating rewards along demonstrator trajectories.
8

---

-30
-10
-202
-10
-40
-102
-150
-45
-402
-50
20000 40000 60000
100000
50000
154000
200000
50000 1:00001 500000000000000000000
Timasinga
Timasdapa
Timeriap
(a) Pointmass Navigation
(b) Block Extraction
(c) Sequential Pushing
-15
-20-
-20
-25
-25
-40
100000
150000 250203
150000 250000
(d) Door Opening
(c) Black Lifting
Demos
BC
TD3
SAC
GQE
TD3 MCAC
SAC MCAC
GOE MCAC
Figure 3: MCAC and Standard RL Algorithms Results: Learning curves showing the exponentially smoothed (smoothing factor 0.9) mean and standard error across 10 random seeds. We find that MCAC improves the learning efficiency of TD3, SAC, and GQE across all environments.
reward, but not effective MCAC. When the agent is trained with MCAC (bottom row), the agent learns useful Q function that it uses to reliably complete the task (bottom left). As expected with a high-performing policy, its Bellman estimates, GQE estimates and MCAC estimates are similar.
6.4.2 MCAC and Standard RL Algorithms
In Figure 3. we study the impact of augmenting SAC, TD3 and GQE with the MCAC target Q- function. Note that all methods, both with and without MCAC, we initialize their replay buffers with the same set of demonstrations. Results suggest that MCAC is able to accelerate learning for both TD3 and SAC across all environments, and is able to converge to performance either on-par with or better than the demonstrations. In the Pointmass Navigation and Block Lifting tasks, SAC and TD3 make no task progress without MCAC. MCAC also accelerates learning for GQE for the Pointmass Navigation, Block Extraction, and Sequential Pushing environments. In the Door Opening and Block Lifting environments, MCAC leaves performance largely unchanged since GQE already achieves performance on par with the next best algorithm without MCAC.
6.4.3 MCAC and RL From Demonstrations Algorithms
In Figure 4, we study the impact of augmenting OEFD [Nair et al., 2018], CQL [Kumar et al., 2020] and AWAC [Nair al., 2021] with the MCAC target Q-function. Results suggest that MCAC improves the learning efficiency of OEFD on the Pointmass Navigation, Sequential Pushing, and Block Lifting tasks, but does have a significant positive negative affect on performance for the Block Extraction and Door Opening tasks. MCAC improves the performance of AWAC on the Pointmass Navigation and Sequential Pushing environments, stabilizing learning while the versions without MCAC see performance fall off during online fine tuning. On the other environments where AWAC able to immediately converge to a stable policy after offline pre-training, MCAC has no significant negative effect on its performance. In all tasks, MCAC improves the performance of CQL. In particular, for the Pointmass Navigation, Block Extraction and Sequential Pushing tasks, COL. makes almost progress while the version with MCAC learns to complete the task reliably.

---

-20
-30
-200
-150
-303
-10
-120
-50
20000 40000 60000
100000
50000
100000
150000
50000
300000
193000
Timasinga
Timastapa
(a) Pointmass Navigation
(b) Block Extraction
(c) Sequential Pushing
-15
-20
-25
-25
-45
100000
150003
100000
150000
(d) Door Opening
(c) Block Lifting
Demos
BC
OEFD
CQL
AWAC
OEFD MCAC
CQL MCAC
AWAC
MCAC
Figure 4; MCAC and RL from Demonstrations Algorithm Results: Learning curves showing the exponen- tially smoothed (smoothing factor 0.9) mean and standard error across 10 random seeds. When OEFD or AWAC achieve high performance almost immediately, MCAC has little impact performance. However, when OEFD and AWAC are unable learn efficiently, MCAC accelerates and stabilizes policy learning.
7 Discussion, Limitations, and Ethical Considerations
We present Monte Carlo augmented Actor-Critic (MCAC), simple, yet highly effective, change that can be applied to any actor-critic algorithm in order to accelerate reinforcement learning from demonstrations for sparse reward tasks. We present empirical results suggesting that MCAC often significantly improves performance when applied to three state-of-the-art actor-critic RL. algorithms and three RL from demonstrations algorithms five different continuous control domains.
Despite strong empirical performance, MCAC also has some limitations. We found that while encouraging higher Q-value estimates is beneficial for sparse reward tasks, when rewards are dense and richly informative, MCAC not helpful and can even hinder learning by overestimating Q-values. From ethical standpoint, reinforcement learning algorithms such MCAC can be used to automate aspects of variety interactive systems, such as robots, online retail systems, or social media platforms. However, poor performance when policies haven't 't yet converged, or when band-designed reward functions do not align with true human intentions, can cause significant harm to human users. Lastly, while MCAC shows convincing empirical performance, in future work it would be interesting to provide theoretical analysis on its convergence properties and on how MCAC can be used to learn Q functions which better navigate the bias variiner tradeoff.
Acknowledgements
We would like to thank all reviewers for their invaluable feedback, which helped us significantly strengthen the paper during the author response period. This research was performed at the AU- TOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab, and the CITRIS "People and Robots" (CPAR) Initiative. The authors were supported in part by donations from Google, Toyota Research Institute, and NVIDIA.
10

---

References
Brenna D ArgalI, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469 483, 2009.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Sebastian Dittert. Cql. https: http::///ggghuu...oo/////////// 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation enror in actor- critic methods. In International Conference on Machine Learning (ICML), pages 1582-1591, 2018.
Yang Gao, Huazhe Xu, Lin, Fisher Yu, Sergey Levine, and Trevor Darrell. Reinforcement learning from imperfect demonstrations. arXiv preprint arXiv: (802.0880.35111 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning (ICML), 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Oshand, et al. Deep q-learning from demonstrations. In Thirty-second AAAI conference artificial intelligence, 2018.
Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Chao Yang, Bin Fang, and Huaping Liu. Reinforcement learning from imperfect demonstrations under soft expert guidance. In Proceedings of the AAAI Conference on Artificial Intelligenee volume 34, pages 5109-511 16, 2020.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In International Conference on Machine Learning, pages 2479-2478. PMLR, 2018
Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In NIPS, pages 2859-2867. Citeseer, 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1442.6990,,1 2015.
Jens Kober and Jan Peters. Policy search for motor primitives in robotics. Learning Motor Skills, pages 83-117. Springer, 2014.
George Konidaris, Scott Niekum, and Philip S Thomas. Td gamma: Re-evaluating complex backups in temporal difference learning. Advances in Neural Information Processing Systems, 24:2402- 2410, 2011.
Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog, 2020.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1179-1191. Curran Associates, Inc., 2020. URL https: httpppss/////////ppeeeeeee /ppp//eeeednng...///////////..sp neurips wwterrpssccppppe///e/ee cc/papprr/002//// 022222661882225544221161666 1aa.aaaa23322111115556555661311 Paper.
Arsenii Kuznetsov, Pavel Shvechikov. Alexander Grishin, and Dmitry Vetrov. Controlling overesti- mation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556-5566. PMLR, 2020.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, hss/1500.02999 2015, URL http:// http::p//axxvvorrr/eee
029771.

---

Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (18(7540):529 533,, 2015.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference Roboties and Automation (ICRA), pages
IEEE, 2018.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets, 2021.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning. pages 2721-2730. PMLR, 2017.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advintago-weighted regression: Simple and scalable off-policy reinforcement learning, 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018.
Stefan Schaal et al. Learning from demonstration. Advances in neural information processing systems, pages 1040-1046, 1997.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. International Conference Learning Representations (ICLR), 06 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Sahil Sharma, Girish Raguvir J, Srivatsan Ramesh, and Balaraman Ravindran. Learning to mix n-step returns: Generalizing lambda-returns for deep reinforcement learning. 2018.
Harshit Sikchi and Albert Wilcox. PWAOC URL AWAC
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Joannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madelcine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587)4444 2016.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Pranjal Tandon.
srtorch-sof-aactor-riiii crtorch-soft-actee cecector-riiicc.
URL https://githin com/ wwt./o/oo/////////
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E. Gon- zalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks, 2020.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3): 4915 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033 IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
12

---

Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad, 2018.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning robotics problems with sparse rewards, 2018.
Albert Wilcox, Ashwin Balakrishna, Brijen Thananjeyan, Joseph E. Gonzalez, and Ken Goldberg. Ls3: Latent space safe sets for long-horizon visuomotor control of sparse reward iterative tasks, In Conference Robot Learning (CoRL). PMLR, 2021.
Robert Wright, Steven Loscalzo, Philip Dexter, and Lei Yu. Exploiting multi-step sample trajectories for approximate value iteration. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný, editors, Machine Learning and Knowledge Discovery in Databases, pages 113-128, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg, ISBN 978-3-642-40988-2
Robert William Wright, Xingye Qiao, Lei Yu, and Steven Loscalzo. Cfqi: Fitted q-iteration with complex returns. In AAMAS, pages 163-170. Citeseer, 2015.
Zheng Wu, Wenzhao Lian, Vaibhav Unhelkar, Masayoshi Tomizuka, and Stefan Schaal. Learning dense rewards for contact-rich manipulation tasks. arXiv preprint arXiv:2011.0845 2020.
Jiexin Xie, Zhenzhou Shao, Yue Li, Yong Guan, and Jindong Tan. Deep reinforcement learning with optimized reward functions for robotic trajectory planning. JEEE Access, 7:105669-105679 2019.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12 12293, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The Abstract and introduction claim that we introduce new reinforcement learning algorithm and experiments on number of domains suggesting that accelerates learning for sparse reward tasks. This is consistent with the algorithm presented in Section 5 and the experiments in Figures 2 and 3 in Section 6
(b) Did you describe the limitations of your work? [Yes] Yes, Section 7
(c) Did you discuss any potential negative socictal impacts of your work? [Yes] Yes, see Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes, this work conforms to the ethics guidelines in the sense that it does not have direct negative socictal impacts that we know and is fully reproducible, with released code and results reported over 10 random seeds. That being said, reinforcement learning research general, including our work, has possible ethical concerns related to misuse in systems which interact with humans, such robots, online retail systems, and social media platforms. Specifically, poorly specified reward functions can cause unexpected consequences when algorithms discover behaviors different from what the human reward designer intended. Suboptimal behaviors during policy optimization also have potential cause harm. We discuss these concerns in detail in Section 7.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] There are no theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] There are no theoreti- cal results.
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] A zip file containing all code is included in the supplementary material. This code is associated with README that can be used to replicate all experiments.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All training details are described at high level in Section 6 and in detail in the supplementary material for all experiments.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] All results in all plots are reported over random seeds, with the mean displayed in the plot with standard error shading to indicate the margin of error over the 10 runs.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Yes, see the beginning of Section 6.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, the sources of all experimental domains used cited in Section 6.2. We also cite the sources of all algorithms we build on in Section 6.3 and reference implementations used of these algorithms in the supplement.
(b) Did you mention the license of assets? [N/A] There were no explicit licences associated with the datasets/code we used, but we cited the papers they came from and the Github repositories associated with them.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Yes, we have included code and datasets to reproduce all results in this paper in the supplement.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] All data was collected in computational simulations, so no explicit human permission was required.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] No data collected was explicitly collected by human users, this was necessary.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] There were no human subject studies.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicable? [N/A] There were no human subject studies.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] There were no human subject studies.
14


Reviews:
[{"review_id": "n4RNKY_RlBz", "reviewer": "Reviewer_jXtd", "summary": "The paper proposes a method to incorporate prior expert (possibly sub-optimal) demonstrations in the learning procedure of any actor-critic algorithms. Furthermore, the authors propose also a modified target estimator for the critic updates, based on the maximum of an MC estimate and the standard estimators of the base AC algorithms. The resulting framework is fairly easy to implement and introduces no additional parametrization or computation at the cost of an initial set of demonstrations from the expert. The empirical evaluation show an improvement over SOTA RL algorithms on a set of continuous domains with sparse rewards.", "questions": "The main question I have is how important is the presence of the expert demonstrations in the replay buffer. Namely, have you measured the performance of the proposed algorithm only with the application of the MC targets in Eq. 5.4?\n\nRelated to this, a study on the effect of the sub-optimality of the demonstrations should be performed. How much is the performance impacted from more sub-optimal demonstrations. This could be studied in a simple environment, by varying the amount of noise added to the expert demonstrations. This is important, since the method, as the authors state, is to be applied exclusively to sparse reward environments, and often times in these environments it might be difficult to have demonstrations with a reward signal (e.g. goal based environments where a human might have difficulties in making a policy that reaches the goal)", "limitations": "The authors adequately address their limitations in the conclusions.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and easy to understand. The authors propose a novel method to for Q-target correction, which is fearly easy to combine with current RL algorithms. The experimental evaluation shows significant performance improvement without additional computational cost. \n\nOverall I like the paper and I think it should be accepted.  The authors propose a simple method, thoroughly evaluate it and compare with appropriate baselines and discuss the limitations and shortcomings of the method. \n\nThe only minor weakness I would point is the lack of clarity in the presentation of the confidence intervals in the plots. It seems like the confidence intervals are applied around the non-smoothed curves which in practice makes them difficult to understand, I would suggest to add the confidence intervals around the smoothed curves and remove the raw curves from the plot to improve readability.\n\nThe main weakness I see, is the requirement of demonstrations in sparse reward tasks. In these kind of tasks, often times (not always), it might be difficult to have demonstrations that achieve a high return.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NRmRTUQaBgB", "reviewer": "Reviewer_VGAN", "summary": "This paper proposes an Actor-Critic method (MCAC) to resolve the sparse reward problem in deep reinforcement learning (single-agent cases). Technically, the algorithm starts from a replay buffer with demonstrations and performs policy evaluation with a modified $Q$-value function. Specifically, the policy evaluation is the maximum of the standard critic function and a Monte-Carlo estimation. The idea behind using two value estimations is that the combination may share the advantages of these two estimations, such as low variance and unbias. The implementation is given under some assumptions (such as \"... the last observed reward value will repeat forever ...\") to make the algorithm adapt to cases with an infinite horizon. The authors demonstrate experiments and comparisons with previous work on five continuous control tasks, in which the results show that MCAC outperforms all baselines.", "questions": "1. As the MCAC requires pre-training with offline datasets. So, Is there any requirement on the size of the dataset? I think it is critical to the algorithm's performance.\n2. For $Q^{target}_{MC-\\infty}$ in the infinite cases, why don't we use the critic to approximate the $\\gamma^{T-j+1}\\frac{r^i_T}{1-\\gamma}$?\n", "limitations": "The authors figure out some unresolved limitations in the last section. Such as MCAC is not helpful and even hinders learning by overestimating Q-values in dense-reward cases. As I mentioned in the \"weaknesses\", the normalization of the maximum operation maybe necessary to handle this problem.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Update After Rebuttal \n\nI thank the authors for their in-depth responses also the supplemented experiments. I am mostly convinced by the authors and have updated my score from 5 to 6.\n\n### Strengths\n\n1. this paper is well-organized and easy to follow.\n2. this paper concentrates on an interesting problem in deep reinforcement learning and proposes a novel plug-in for Actor-Critic methods.\n\n### Weaknesses\n\n1. **Some typos and referring errors**: e.g., line 209: \"... provided in Appendix ??\"\n2. **The demonstration is not so serious**: the demonstration lacks a comparison or investigation of the effectiveness of value function combinations. As the authors claim in this paper, they aim to bridge the advantages of the standard value function and Monte-Carlo methods to improve the accuracy/efficiency of policy evaluation (or value estimation). However, there is no explanation (at least empirically) for why this method can achieve such a target.\n3. **Some technique flaws**:\n\n    - As introduced in this paper, MCAC modifies the estimation of the target value function as Eq-(5.3). However, there may be a potential issue that such a method is biased as $Q^{target}$ and $Q^{target}_{MC-\\infty}$ have different scales.\n    - The authors approximate the Monte-Carlo estimation of an infinite horizon as Eq-(5.2). However, I am concerned about this approximation since the authors neither refer to some previous work to support the correctness nor give empirical results to explain it.\n4. This point is more like a suggestion to highlight your motivation. As the authors claim that their method is the first application of Monte-Carlo estimation of the return-to-go to improve value estimation for online cases, while is no discussion about what will such an estimator bring to online cases. Thus, I think the authors can make a deeper discussion on that.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CNfpZk6iaWD", "reviewer": "Reviewer_rJu7", "summary": "This paper proposes a modified Actor-Critic algorithm (MCAC) for sparse reward problems. This modification is achieved by initializing the replay buffer with demonstration and computing the modified Q-value by combining temporal distance target and Monte Carlo estimation. The method is evaluated in five environments, and performance is compared with a few existing methods. ", "questions": "Standard RL algorithms usage demonstration (line 297) and is then compared with the MCAC version. However, the standard RL is not equipt to deal with such a demonstration, and having some sub-optimal demonstration might worsen the performance. Thus, RL algorithms without the sub-optimal demonstration should be compared as well.\n \nIn figure 3, MCAC barely improves performance in all the environments; in most scenarios, performance remains the same as the starting position (initial timesteps); why is that the case, especially AWAC and AWAC+MCAC?\n\nAdditionally, the results seem puzzling ( Figure 3); the AWAC results start with a high reward and then fall. In the cases where it recovers (with MCAC) and barely reaches the initial performance again. Thus though MCAC makes training stable, it barely gives any reward-level advantage. This improvement is the core of this paper. Though on OEFD, MCAC is performing better; overall, this method appears to be not as good as AWAC-based methods.\n\nHow was the demonstration collected? Was it collected by programming (automatic) or human-collected (e.g., line 239)? Adding zero-mean noise might make it a bit realistic; however, it may be beneficial to have humans collect data, which would be more realistic. \n\nI understand that the paper focus on suboptimal demonstration (e.g., line 258); however, is the proposed method tied to such an assumption? How does the performance stand with near-to optimal demonstration, which is often the case of expert (human) demonstration?\n\nLooking at Figures 2 and 3, it seems the demonstration-based method performs worse than the standard-baseline variant (e.g., GQE in Figure 2). It seems that the demonstration is not helping in these scenarios. Further discussion on this issue would help clarify the advantage of the proposed method. \n\nThe MCAC has two modifications, namely, replay buffer and Q-value modification. An ablation study on them would further clarify their significance.\n\nThere are many missing references (“?”) throughout the paper. A careful pass should fix them. There are too many environmental descriptions in the main paper, which can go to the Appendix.\n", "limitations": "Yes. Limitations and potential societal impacts are discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength:\n- Tackle an important problem of learning from sub-optimal demonstration in sparse environments.\n- Relevant literature is discussed, and a few baselines are compared in five environments.\n\n### Weakness (Details in Questions):\n- Lack of empirical evidence of the advantage of the proposed method\n- Some details of the method design and experiments are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8WrCQI9eo4O", "reviewer": "Reviewer_dRfP", "summary": "This paper proposes a simple yet effective method MCAC to assimilate the benefits of both TD Q-target and Monte Carlo Q-targets, which can be applied to any actor-critic algorithm to solve the sparse Reinforcement learning problem. The authors empirically prove the effectiveness of MCAC for both standard reinforcement learning and imitation learning.", "questions": "For the experimental parts, the authors claim in Figure 2 that when GQE already achieves very strong performance, MCAC may have little impact. I feel confused about the criterion of \"strong performance\". If strong performance refers to the performance being higher than the demos, we can still see that on the Sequential Pushing, although GQE achieves strong performance, MCAC can still accelerate the learning. \n\nI also have a similar concern for Figure 3 about OEFD, the criterion for \"unable to learn efficiently or have significant instability\".\n\nAnother stuff is for Figure 3 in the appendix, I cannot see a clear pattern as $\\lambda$ increases (e.g., block lifting), I feel the results are still dominated by the randomness rather than the hyperparameters.\n\nMy general intuition is that MCAC is not shown to consistently take effect in these 5 environments, I suggest the authors give some theoretical proof and more experimental results on other domains.", "limitations": "The authors address both societal impacts and limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe method proposed is simple and works effectively on various tasks. For some tasks, MCAC achieves a pretty solid performance improvement whereas the baselines work very badly. The authors also provide some intuitive explanations for MCAC which I think is inspiring for future work. \n\nCons:\nHowever, my largest concern is that either the current experimental results or the intuitive explanations cannot justify the consistent effectiveness of MCAC, and generally, I think this work does not show enough technical contribution. This method is just a simple extension the idea in [1]. Besides, the writing of this paper is very messy with many missing section references. I see the authors want to give some ablation study in section 6.4 but that part is finally in the appendix.  \n\n[1] Robert Wright, Steven Loscalzo, Philip Dexter, and Lei Yu. Exploiting multi-step sample trajectories\n419 for approximate value iteration. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and\n420 Filip Železný, editors, Machine Learning and Knowledge Discovery in Databases, pages 113–128,\n421 Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-40988-2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept