Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
Authors: Wei Zhou
Yiying Li
Yongxin Yang
Huaimin Wang
Timothy M. Hospedales
Venue:  (2020)
Keywords: off-policy actor-critic
reinforcement learning
meta-learning

Abstract: Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic’s action-value function is updated using temporal-difference, and the critic in turn provides a loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared to the vanilla critic, the meta-critic network is explicitly trained to accelerate the learning process; and compared to existing meta-learning algorithms, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-art reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to improvements in a variety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the state-of-the-art SAC. 

Under review as a conference paper at ICLR 2020
ONLINE META-CRITIC LEARNING FOR OFF-POLICY ACTOR-CRITIC METHODS
Anonymous authors Paper under double-blind review
ABSTRACT
Off-Policy Actor-Critic (Off-PAC) methods have proven successful in a variety of continuous control tasks. Normally, the critic's ' action-value function updated using remporal-difference, and the critic in turn provides loss for the actor that trains it to take actions with higher expected return. In this paper, we introduce a novel and flexible meta-critic that observes the learning process and meta-learns an additional loss for the actor that accelerates and improves actor-critic learning. Compared the vanilla critic, the meta-critic network explicitly trained ac- celerate the learning process; and compared to existing meta-learning algorithms,, meta-critic is rapidly learned online for a single task, rather than slowly over a family of tasks. Crucially, our meta-critic framework is designed for off-policy based learners, which currently provide state-of-the-arr reinforcement learning sample efficiency. We demonstrate that online meta-critic learning leads to im- provements in a variety of continuous control environments when combined with contemporary Off-PAC methods DDPG, TD3 and the nate-of-theeart SAC.
INTRODUCTION
Off-policy Actor-Critic (Off-PAC) methods are currently central in deep reinforcement learning (RL) research due to their greater sample efficiency compared on-policy alternatives. On-policy requires new trajectories to be collected for each update to the policy, and is expensive as the number of gradient steps and samples per step increases with cask-compyxxtty even for contemporary TRPO (Schulman et al., 2015). PPO (Schulman et al., 2017) and A3C (Mnih et al., 2016) algorithms. Off-policy methods, such as DDPG (Lillicrap et al.J 2016), TD3 (Fujimoto et al. 2018) and SAC (Haarnoja et al. (008b) achieve greater sample efficiency due to their ability to learn from randomly sampled historical transitions without a time sequence requirement, thus making better use of past experience. Their critic estimates the action-value (Q-value) function using differentiable func- tion approximator, and the actor updates its policy parameters in the direction of the approximate action-value gradient. Briefly, the critic provides loss to guide the actor, and is trained in turn to estimate the environmental action-value under the current policy via temporal lemporal-difference learning Sutton et al. 2009). In all these cases the learning algorithm itself is hand-crafted and fixed.
Recently meta-learning, or Seaarinn-tooleare has become topical as a paradigm to accelerate RL. by learning aspects of the learning strategy, for example, through learning f'st adaptation strategies (Finn et al., [017]] Rakelly et al. 2019 Riemer et al., 2019), exploration strategies (Gupta et al.) 2018), optimization strategies (Duan al. (016b)), losses (Houthoolt al. 2018), hyperparameters (Xu et al. 2018: Veeriah et al.) 2019). and intrinsic rewards (Zheng et al., [008).. However, the majority these works perform meta-learning on family of tasks or environments and amortize this huge cost by deploying the trained strategy for fast learning on a new task.
In this paper we introduce novel meta-critic network to enhance existing Off-PAC learning frame- works. The meta-critic is used alongside the vanilla critic to provide loss to guide the actor's s learning. However compared to the vanilla critic, the meta-critic explicitly (meta)-trained to ac- celerate the learning process rather than merely estimate the action-value function. Overall, the actor is trained by gradients provided by both critic and meta-critic losses, the critic is trained by temporal -iffeeeeee mance improvements in the actor. In our framework, both the critic and meta-critic use randomly sampled off-policy transitions for efficient and effective Off-PAC learning, providing superior sam-
usual, and the meta-critic is trained to generate maximum learning perfor-

---

Under review as a conference paper at ICLR 2020
ple efficiency compared existing on-policy meta-leamers. Furthermore, we demonstrate that our meta-critic can be successfully learned online within a single task. This is in contrast to the currently widely used meta-learning research paradigm where entire task families required to provide enough data for meta-learning, and to provide new tasks to amortize the huge cost of meta-learning.
Essentially framework meta-learns auxiliary loss function, which can be seen motivation towards optimum learning progress (Oudeyer & Kaplan 2009). As analogously observed in several recent meta-learning studies (Franceschi al.) 2018), our loss-learning can be formalized as a bi-level optimization problem with the upper level being meta-critic learning, and lower level being conventional learning. We solve this joint optimization by iteratively updating the meta- critic and base learner online while solving single task. Our strategy is thus related to the meta- loss learning in EPG (Houthooft et al. 2018). but learned online rather than offline, and integrated with Off-PAC rather than their on-policy policy-gradien learning. The most related prior work is LIRPG (Zheng et al.) 2018), which meta-learns an intrinsic reward online. However, their intrinsic reward just provides helpful scalar offset to the environmental reward for on-policy trajectory optimization via policy-gradient (Sutton et al. 2000). In contrast our meta-critic provides a loss for direct actor optimization just based sampled transitions, and thus achieves dramatically better sample efficiency than LIRPG reward learning in practice. We evaluate our framework on several contemporary continuous control benchmarks and demonstrate that online meta-critic learning can be integrated with and improve a selection of contemporary Off-PAC algorithms including DDPG, TD3 and SAC.
intrinsic
BACKGROUND AND RELATED WORK
Policy-Gradient (PG) Methods. On-policy methods usually update actor parameters in the di- rection of greater cumulative reward. However, on-policy methods need to interact with the en- vironment in a sequential manner to accumulate rewards and the expected reward is generally not differentiable due to environment dynamics. Even exploiting tricks like importance sampling and improved application of A2C (Zheng et al., 2018), the use of full trajectories is less effective than off-policy transitions, as the trajectory needs series of continuous transitions in time. Off-policy actor-critic architectures aim to provide greater sample efficiency by reusing past experience (pre- viously collected transitions). DDPG (Lillicrp et al.) 2016) borrows two main ideas from Deep Q Networks (Mnih et al. 2013 2015): a big replay buffer and a target Q network to give consistent targets during temporal difference backups. TD3 (Twin Delayed Deep Deterministic policy gradient algorithm) (Fujimoto et al. 2018) develops a variant of Double Q-learning by taking the minimum value between a pair of critics to limit over-estimation. SAC (Soft Actor-Critic) (Haarnoja et al., 2018a b) proposes a maximum entropy RI. framework where its stochastic actor aims to simulta- neously maximize expected action-value and entropy. The latest version of SAC (Haarnoja et al.) 2018b also includes the "the minimum value between both critics" idea in its implementation.
Meta Learning for RL. Meta-learning (a.k.a. learning to learn) (Santoro et al. 2016 Finn et al.) 2017) has received resurgence in interest recently due to its potential to improve learning perfor- mance, and especially sample-efficiency in RL (Gupta et al., 20188). Several studies learn optimizers that provide policy updates with respect known loss reward functions (Andrychowicz
al.
2016 Duan et al. 2016b] Meier et al. 2018). A few studies learn hyperparameters (Xu et al. [2018] Veeriah al. 2019). loss functions (Houthooft et al. 2018 Sung et al.) 2017) or rewards (Zheng et al. (0188) that steer the learning of standard optimizers. Our meta-critic framework is in the cat- egory of Joss-function meta-learning, but unlike most of these we are able to meta-learn the loss function online in parallel to learning single extrinsic task rather. No costly offline learning on a task family required as in Houthooft et al. (2018): [Sung al. (2017). Most current Meta-RU RL methods are based on on-policy policy-gradien limiting their sample efficiency. For example, while LIRPG (Zheng et al. 2018 one of the rare prior works to attempt online meta-learning, it is in- effective in practice due only providing a scalar reward increment rather than loss for direct optimization. A few meta-RL studies have begun to address off-policy RL, for conventional offline multi-task meta-learning (Rakelly et al. 2019) and for optimising transfer forgetting in continual learning of multiple tasks (Riemer et al. 2019). The contribution of our CretiCCiitic is to enhance state-of-the-as OfC-PAC RL with single-task online meta-learning.
Loss Learning. Loss learning has been exploited in 'learning to teach' (w et al. 2018) and surro- gate loss learning (Huang et al. 2019: Grabocka al. 2019) where teacher network predicts the
2

---

Under review as a conference paper at ICLR 2020
parameters of manually designed loss in supervised learning. In contrast meta-critic is itself differentiable loss, and designed for use in reinforcement learning. Other applications learn losses that improve model robustness to out of distribution samples (Li al. 2019 Balaji et al.] 2018). Our loss learning architecture related to [i et al. (2019), but designed for accelerating single-task Off-PAC RL rather than improving robustness in multi-domain supervised learning.
METHODOLOGY
We aim learn meta-critic that provides auxiliary loss L assist the actor's learning of task. The auxiliary loss parameters are optimized in meta-lcarning process, The vanilla critic main and meta-critic L losses train the actor Th off-policy via stochastic gradient descent.
3.1 REVIEW OF OFF-POLICY ACTOR-CRITIC RL
Reinforcement learning involves an agent interacting with the environment E. At each time t, the agent receives an observation St, takes (possibly stochastic) action a, based on policy S + A, and receives scalar reward and new state of the environment 8t+1- We call (st, ae, Tt, Sc-1) as single point transition. The objective of RL to find the optimal policy xd. which maximizes the expected cumulative return J.
In on-policy RL, defined as the discounted episodic return based on a sequential trajectory over horizon H: (so, ao, ro. ................................ sH ag TH). Er,,st-E Σn= 7'r In the usual implemen- tation of A2C, is represented by a surrogate state-value (ss:) from its critic. Since J is only scalar value, the gradient of J with respect to policy parameters & has to be optimized under the policy gradient theorem (Sutton al. 2000p: VØJ( E [J V log (=(a)ss)))-
M
In off- off-policy RL (e.g.. DDPG, TD3, SAC) which is our focus this paper, parameterized policies To can be directly updated by defining the actor loss in terms of the expected return (Ø) and taking its gradient (Ø)) where J(b) depends the action-value 2o(s,,a)) The main loss / main provided by the vanilla critic is thus
main - = -Esop, Qu(s, a)la=na(s))
(1)
where we follow the notation in TD3 and SAC that Φ and 0 denote actors and critics respectively.
The main loss is calculated by mini-batch of transitions randomly sampled from the replay buffer. The actor's policy network is updated as AS = a V L main following the critic's gradient to in- crease the likelihood of actions that achieve higher Q-value. Meanwhile, the critic uses Q-learning updates to estimate the ration-valuus function:
arg min (Qo(st, at) - ri (0((441
3.2 ALGORITHM OVERVIEW
Our meta-learning goal is to train auxiliary meta-critic network L that in turn enhances actor learning. Specifically,, it should lead to the actor ó having improved performance on the main task m when following gradients provided by the meta-critic well those provided by the main task. This can be seen as bi-level optimization problem (Franceschi et al. 2018 Rajeswaran et al. 2019) of the form:
argmin 7, m.mota (dvat: (²)
(3)
s.f. 6² argmin (I (derrn Φ)
(derni
+ (derni Φ));
where we can assume L metn)) m.) for now. Here the lower-level optimization trains the actor to minimize both the main task and meta-criti critic-provided losses some training samples. The upper-level optimization further requires the meta-critic w to have produced a learned actor d* that minimizes meta-loss that measures the actor's main task performance second set of validation
3

---

Under review as a conference paper at ICLR 2020
Algorithm Online Meta-Critic Learning for Off-PAC RL
6, 0, w, D 3 ( for each iteration do for each environment step do
// Initialize actor, critic, meta-critic and buffer
Or ~ (a((()))))
// Select action according to the current policy
St+1 ~ p(sps1111/ St, ac), D + DU ( ae Tr, 5t-1)
// Observe reward r; and new state 81+11
// Store the transition in the replay buffer
end for
for each gradient step do
0 --0-AV
Update the critic parameters
meta-train:
Sample mini-batch dren from D
m Eqs.
Main actor loss
L - Eqs.
OT
Auxiliary actor loss from meta-critic
Oold Φ
main
// Update actor according vanilla critic only Update actor according to meta-critic
Onew Oold a V L
meta-test:
Sample mini-batch dept from D L metal (deat: Gold )new) Eqq.
Meta-loss: Did meta-critic improve performance?
meta-optimization
7/( ( mami + V A
Update actor parameters
w - w V.
// Update meta-critic parameters
end for end for-0
samples, after being trained by the meta-critic. Note that in principle the lower-level optimization could purely rely on T. analogously to the procedure in EPG (Houthooft et al.] 2018), but we find that optimizing their linear combination greatly increases learning stability and speed. Eq. (3) is satisfied when the meta-critic successfully improves the actor's performance on the main task as measured by meta-loss. Note that the vanilla critic update also the lower loop, but as it updates as usual, so we focus on the actor and meta-critic optimization for simplicity of exposition.
In this setup the meta-critic a neural network (derni o) that takes as input some featurisation of the actor & and the states and actions in dern- This auxiliary neural network must produce scalar output, which we can then treat as loss T,uux == hws and must be differentiable with respect ø We next discuss the overall optimization flow, and discuss the specific meta-critic architecture later.
Meta-Oppimization Flow. optimize Eq. 3 we iteratively update the meta-critic parameters (upper-level) and actor and vanilla-critic parame- ters ø and 0 (lower-level). At each iteration, we perform: (i) Mcta-train: Sample a mini-batch of
drm
update
transitions and putatively update policy oi accord- ing to the main mamin and meta-critic T.nix losses. (ii) Meta-test:
critic
meta-critic
Sample another mini-batch of tran- sitions to evaluate the performance of the updated policy according to L meta (iii) Meta-optimization: Update the meta-critic parameters w to maximize the performance on the validation batch, and per- form the real actor update according to both losses. In this way the meta-critic trained online and in parallel the actor so that they co-evolve. Fig- ure [ and Algorithm ( summariie the process and the details of each step explained next.
imaiin
Σ
Pg Emoae
update
actór Oold
deas
actor 0 Omgg"
Fa (L.mato + 4,m0rr
Figure 1: Meta-critic for Oft-PAC. The agent uses data sampled from the replay buffer during meta- train and meta-test. Actor parameters are first up- dated using only vanilla critic, or both vanilla- and meta-critic. Meta-critic parameters updated by the meta-loss.
Updating Actor Parameters (o). During meta- train, we randomly sample mini-batch of transi- tions de,m. {(s, 0; Ta, 8441)} } with batch size from the replay buffer D. We then update the pol-
4

---

Under review as a conference paper at ICLR 2020
icy using both losses as: Onew = Φ - 1] B 1 (derm)
9 (deen))
0e
1)
00
We also compute a separate
update Potd = Φ - 1) " (dvva) that only makes use of the vanilla loss. If the meta-critic provided beneficial source of loss, Pnew should be better parameter than 6, and in particular it should be a better parameter than Gold. We will use this comparison in the next meta-test step.
Updating Meta-Critic Parameters (w). To train the meta-critic network. we sample another mini-batch of transitions: deat = {( at r s(41)) }} with batch size M. The use of a val- idation batch for bi-level meta-optimization (Franceschi et al. 2018 Rajeswaran et al., (019) ensures the meta-learned component does not overfit. Since our framework is off-policy, this does not incur any sample-efficiency cost. The meta-critic is then updated by a meta loss 6 argmin L ( &new), which could in principle be the same as the main loss L = L m
s However, find it helpful for optimization efficiency optimize the (monotonically related) dif- ference between the updates with- and without meta-critic's input. Specifically, we use
L metu tanh (I. [. (deat; (new) I, (deat: Oold));
(4)
which is simply re-centering and re-scaling of L This leads
w< t argmin (L main (deal: Onew) - L main (deal: dold)).
(5)
Note that here the updated actor Onew has dependence on the feedback given by meta-critic w and Gold does not. Thus only the first term is optimized for w. In his setup the L (dvat: @new) term should obtain high reward/low loss on the validation batch and the latter provides a baseline, analogous to the baseline commonly used to accelerate and stabilize tanh reflects the idea of diminishing marginal utiliiyy and ensures that the meta-loss range is always nicely distributed in 1-1, 1]. In essence, the meta-loss is for the agent to ask itself the question based on the validation batch, "Did meta-critic improve the performance?", and adjusts the parameters of meta-critic accordingly.
RL. The use of
Designing Meta-Critic (hw.). The meta-critic network h. implements the auxiliary loss for the actor. The design-space for hi has several requirements: (i) Its input must depend on the policy parameters é, because this auxiliary loss is also used to update policy network. (ii) It should be per- mutation invariant to transitions in dern, i.e., it should not make a difference if we feed the randomly sampled transitions indexed [1,2,3] or [3.2,1]. The most naive way to achieve (i) is given in MetaReg (Balaji et al.. 2018) which meta-learns a parameter regularizer: (Ø) >. w/ woxl Although this form of Ah acts directly on Φ. it does not exploit state information, and introduces a large number of parameters Φ, and then A/ may be high-dimensional neural network. Therefore, we design a more efficient and effective form of h that also meets both of these requirements. Similar to the feature extractor in supervised learning, the actor needs to analyse and extract information from states for decision-making and decomposed into the feature extraction क and decision-makeer policy network) modules. Thus the output of the penultimate layer of full policy network is just the output of feature extraction (s),, and such output of feature jointly encodes & and 8. Given this encoding, we implement h (derni ø) as three-layer multi-layer perceptron (MLP) whose input is the extracted feature from F. (s). Here we consider two designs for meta-critic (hw): using our joint feature alone (Eq. 66) or augmenting the joint feature with states and actions (Eq. 7):
We assume the policy network can be represented as x A (T (s))
& (i.e., the last layer of the full
1
(i)
Bew (dern: Ø)
MLP M (To(ss))
(6)
N
N
(id)
ha (dirns Φ)
MLP st.).
N
1=1
hhw is to work out the auxiliary loss based on such batch-wise set-embdedding (Zaheer et al. 2017) of our joint actor-state feature. That is to say, dirn is a randomly sampled mini-batch transitions from the replay buffer, and then the s (and a) of the transitions are inputted to the fiw network in a permutation invariant way, and finally we can obtain the auxiliary loss for this batch dern. Here, our design of Eq. (7) also includes the cues features in LIRPG and EPG where s; and a, are used as the input of their learned reward and loss respectively. We set softplus activation the final layer of h.s. following the idea in TD3 that the vanilla critic may ever-estimate and so the introduction of non-negative actor auxiliary loss can mitigate such over-estimation. Moreover, we point out that
5

---

Under review as a conference paper at ICLR 2020
only 8, (and a,) from denn are used when calculating m and L for the actor, while S,. ais ra and 91+1 are all used for optimizing the vanilla critic.
Implementation
DDPG, TD3 and SAC. Our meta-critic module can be incorporated in the main Off-PAC methods DDPG, TD3 and SAC. In our framework, these algorithms differ only in their definitions of I. mami and the meta-critic implementation is otherwise exactly the same for each. Further implementation details can be found in the supplementary material.
TD3 (Fujimoto et al., 2018) borrows the Double Q-learnin idea and use the minimum value between both critics to make unbiased value estimations. At the same time, computational cost is obtained by using a single actor optimized with respect to Qo, Thus the corresponding 7, mumiin for actor becomes:
m - Eg-p, Qo, (s, a)lamm
(8)
In SAC, two key ingredients are considered for the actor: maximizing the policy entropy and automatic temperature syper-parameter regulation. At the same time, the latest version of SAC Haarnoja et al. (018b) also draws lessons from "taking the minimum value between both critics".
The L Lmail for SAC actor is: m.main Espp, log (To(a)s)) - Q0(8, a) ()))
4 EXPERIMENTS AND EVALUATION
The goal of our experimental evaluation to demonstrate the versatility of our meta-critic module in integration with several prior Off-PAC algorithms,, and its efficacy in improving their respective performance. We use the open-source implementations of DDPG, TD3 and SAC algorithms as our baselines, and denote their enhancements by meta-critic as DDPG-MC, TD3-MC, SAC-MC respectively. All -MC agents have both their built-in vanilla critic, and the meta-critic that we propose. We take Eq. (6) as the default meta-critic architecture f and we compare the alternative in the later ablation study. For our implementation of meta-critic, we use a three-layer neural network with input dimension of म (300 in DDPG and TD3, 256 in SAC), two hidden ferwafo layers of 100 hidden nodes each, and ReLU non-linearity between layers.
We evaluate the methods interfaced through OpenAl Gym (Brockman et al., 2016) and HalfCheetah and Ant (Duan et al, 201 in rllab. We use the latest V2 tasks instead VI used in TD3 and the old implementation of SAC (Haarnoja et al. 2018a) 8a) without any modification to their original environment or reward.
suite of seven MuJoCo continuous control tasks (Toooov al. 2012)
Implementation Details. For DDPG, we use the open-source implementation "OurDDPG" "apror"Whii which is the re-tuned version of DDPG implemented in Fujimoto et al. (2018) with the same hyper- parameters of the actor and critic. For TD3 and SAC, we use the open-source implementations of TD3 and S In each case we integrate our meta-critic with learning rate 0.001. The specific pseudo-codes can found in the supplementary material.
4.1 EVALUATION OF META-CRITIC OFF-PAC LEARNING
DDPG Figure 2 shows the learning curves of DDPG and DDPG-MC. The experimental results corresponding to each task are averaged over 5 random seeds (trials) and network initialisations, and the standard deviation confidence intervals are represented as shaded regions over the time steps. Following Fujimoto et al. (2018), curves are uniformly smoothed (window size 30) for clarity. We run the gym-MuJoCo experiments for 1-10 million depen ding on to environment, and rllab experiments for million steps. Every 1000 steps we evaluate our policy over 10 episodes with no exploration noise.
From the learning curves in Figure 2, we can see that DDPG-MC generally outperforms the corre- sponding DDPG baseline in terms of the learning speed and asymptotic performance. Furthermore, it usually has smaller variance. The summary results for all nine tasks in terms of max average return are given in Table [ We selected the six tasks shown in Figure] 2 for plotting, because the

---

Under review as a conference paper at ICLR 2020
HalfChoctah
Hopper
10000
19 1750
2000
IEFG LEFORE
2500
IRFG L0FGW:
15:00
6000
0 1250
1000
2000
1000
500
0
0.5
0.6
0.5
1.0
2.5
Art v2
HoliCheetah trlob)
Art (rlob)
4000
EIPG
0000
25:00
LEIG
3000
8
5000
2000
4000
Average Awvrrste
1500
3000
Aver
2000
1000
1000
5:0
-1000
D0 MC
0.0
1.5
2.0
0.5
1.0
2.0
2.5
Time Stess (206)
Time Steps (2:66
Time Steps (206)
Figure 2: Learning curve mean and standard-deviatton of vanilla DDPG and meta-critic enhanced DDPG-MC for continuous control tasks.
Table 1: Max Average Return over 5 trials all time steps. Max value for each task is bolded.
Environment
DDPG
DDPG-MC
TD3
TD3-MC
SAC
SAC-MC
PPO
PPO-LIRPG
HalfCheetah
8440.2
10187.5
12735.7
15064.0
16651.8
16815.9
2061.5
1882.6
Hopper
2097.5
3253.6
3807 0
3854.3
3610.6
3738.4
3762.0
2750.0
Walker2d
2920.1
3753.7
5942.7
5955.5
6398.8
7164.9
4432.6
3652.9
Ant
2375.4
3661.1
5914.8
6280.0
6954.4
7204.3
684.2
23.6
Reacher
-3.6
-3.7
-3.0
-2.9
-2.8
-2.7
-6.08
-7.53
InvPend
1000.0
1000.0
1000.0
1000.0
1000.0
1000.0
988.2
971.6
InvDouPend
9307.5
9326.5
9357.4
9358.8
9359.6
9359.6
7266.0
6974.9
HaliCheetah(rllabb)
5860.8
6254.6
8029.6
8552.
10011.0
10597.0
Ant(rllab)
2300.8
2721.1
3672.6
4776.8
8014.8
8353,8
other MuJoCo tasks "Reacher". "InvertedPendulum" and "JaverredDDoubleeenduuue have envi- ronmental reward upper bound which all methods reach quickly without obvious difference between them. Table [ shows that DDPG-MC provides consistently higher max return for the tasks without upper bounds.
TD3 and SAC Figure reports the learning curves for TD3, For some tasks vanilla TD3 perfor- mance declines in the long run, while our TD3-MC shows improved stability with much higher asymptotic performance. Generally speaking, the learning curves show that TD3-MC providing comparable or better learning performance in each case, while Table [ shows the clear improvement the most
in the max average return. Figure report the learning curves of SAC. Note that cent update of SAC (Haarnoja et al. 2018b), which can be regarded as the combination SAC+TD3. Although this SAC+TD3 is arguably the strongest existing method, SAC-MC still gives clear boost on the asymptotic performance for several of the tasks.
Comparison vs PPO-LIRPG Intrinsic Reward Learning for PPO (Zheng et al. 2018 the most related method to our work in performing online single-task meta-learning of an auxiliary re- ward/loss via neural network. The original PPO-LIRPG study evaluated on modified environ- ment with hidden rewards. Here we apply it to the standard unmodified learning tasks that we aim to improve. The results in Table demonstrate that: (i) this conventional setting, PPO-LIRPG wors- rather than improves basic PPO performance. (ii) Overall Off-PAC methods generally perform better than on on-policy PPO for most environments. This shows the importance of our meta-learning contribution to the off- off-policy setting. In general Meta-Crithe is preferred compared to PPO-LIRPG because the latter only provides a scalar reward bonus only influences the policy indirectly via volicy-gradies updates, while Meta-Critic provides direct loss.
Summary TableTane Figure all the results in terms of max average return. We can see that SAC-MC always performs best; the Meta-Critic-enhaneee methods generally comparable

---

Under review as a conference paper at ICLR 2020
16000
HaltCheetahug?
Hopper-u2
Walker
14000
3500
5000
Avsaa Bewars
12000
Avsrage Reward
3000
4000
10000
8000
2000
Avsrage
3000
6000
2000
TD3
1000
TD2
1000
TDX-MG MC
TD3-MC
T13 MC
2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.2
0.4
0.5
0.8
1.0
Time Staps (1+6)
Time Steps (1e6)
Time Steps (1a6)
Ant-v2
HalfCheetah (rfabl
Ant (rliabl
5000
6000
Averagge leewent
8000
Nuera gi Rewward
4000
TB3
TD3-CC
4000
6000
5000
3000
4000
2000
2000
1000
2000
1000
105
103
103VC
2
10
0.0
1.0
1.5
2.0
2,5
3.0
0.0
0.5
1.0
1.5
2.0
3.0
Time Steps (1e6)
Time Steps (le6)
Time Steps (1e6)
Figure 3: Learning curve mean and standard--eviaatio of vanilla TD3 and meta-critic enhanced TD3-MC for continuous control tasks.
HaifChwetabh-22
Hoppar-u2
Walker2dac2
7000
14000
3000
10000
12000
2 2000
2500
5000
4000
8 1500
1000
2000
2000
500
SAC
1000
0
Time
HaifCheetah
(rllab)
7000
6000
Averag Reward
10000
7000
8000
8000
2000
6000
50000
3000
4000
5
1000
2000
1000
2000
MAG
SACVC
0.0 0.5 1.5 2.0 2.5 3.0
3.5
0.0
0.5
1.0
1.5
2.0
2.5 3.0
0.5
1.0
2.0
2.5
3.0
Time Stops (106)
Time Steps (106)
Time Steps (2:6)
Figure 4: Learning curve mean and standard-deviation of vanilla SAC and meta-critic enhanced SAC-MC for continuous control tasks.
or better than their corresponding vanilla alternatives; and Meta-Critic usually provides improved variance in return compared to the baselines.
4.2 FURTHER ANALYSIS
To analyse the learning dynamics of our algorithm, we take Walker2d as an example. repure the main loss main curve of actor and the loss curves of hww (i.e., Laxk) and meta over 5 trials for SAC. We can see that: (i) SAC-MC shows faster convergence to lower value of L muin demonstrating the auxiliary loss's ' ability to accelerate learning. Unlike supervised learning, where the vanilla loss is, e.g., cross-entropy vs ground-truth labels. The I main for actors in RL is provided by the critic which is also learned, so the plot also encompasses convergence of the critic. (ii) The meta-loss (which corresponds the success the meta-critic in improving actor learning) fluctuates throughout, reflecting the exploration process in RL. But it is generally negative, confirming that the auxiliary-traiin trained actor generally improves on the vanilla actor at each iteration. (ii)) The auxiliary loss converges smoothly under the supervision of the meta-loss.
Loss Analysis.
Alation on h. design. We also run Walker2d experiments with alternative h. designs as in Eq. or MetaReg (Balaji al. 2018) format (input actor parameters directly). As shown in Table 2 record the max average return and sum average return (regarded as the area under the average reward curve) of all evaluations during all time steps. Eq. ( achieves the highest max average return and
8

---

Under review as a conference paper at ICLR 2020
HaliChoetah.v2
Hopper-v2
Walker2d-v2
16000
6000
12000
3000
5000
10000
+
2000
3000
6000
1000
2000
2000
rasaic
6Ac
PR0
nong
703
SAC
mostnc
nonG
TosMC
sac.
mo
Ant-v2
11000
HaliCheetahrrllab)
9000
Ant(rilab)
8000
6000
7000
5000
9000
6000
8000
3000
1000
1000
6000
3000
0-
5000
2000
octo
Demcatlio cos
oac
sicuc
mouooo
conade
rossc
sac
sacHec
mo
Figure 5: Box plots the Max Average Return over 5 trials of time steps.
-100
0.95
0.03
0.50
-300
0.00
0.80
-0.03
-000
0.70
-790
0.5
1.0
2.0
Time stops (leG)
Time steas (1e6)
Time steps Cledt
(a) Main loss of actor
(b) Auxiliary loss of actor
(c) Meta-loss
Figure 6: Loss analysis of our algorithm.
our default hww (Eq. (6) attains the highest mean average return. We can also see some improvement for (Ø) using MetaReg format, but the huge number (73484) of parameters is expensive. Overall, all meta-critic module designs provides at least small improvement
vanilla SAC.
Ablation on baseline meta-loss. In Eq. (5). we use L main m (deat' Pota) as a baseline to improve numerical stability of the gradient update. To evaluate this design, remove the Gold baseline and optimize w + argmin tanh(Lmiin (duin )new))) The last column in Table shows that this barely
improves vanilla SAC, validating design choice to use bascline.
Table 2: Max and Sum Average Return over trials of all time steps under different designs of meta-crite (aux-loss) and meta-loss. Max value in each bolded.
m Bnew
Pold
m duea (TT)
SAC
kg (Fa)
h (F.. s,a)
(o)
Max Average Return
289.2
7164.9 151.3
780.2 57,364,405
6644.3
1815.6
424.8 52,446,717
Sum Average Retum
53,695,678
61,672,039
58,875,184
CONCLUSION
We present Meta-Critic, an auxiliary critic module for Off-PAC methods that can be meta-learned online during single task learning. The meta-critic is trained to generate gradients that improve the actor's s learning performance over time, and leads to long run performance gains in continuous control. The meta-critic module can be flexibly incorporated into various contemporary Off-PAC methods to boost performance. future work, we plan to apply the meta-critic to conventional offline meta-learning with multi-task and multi-domain RI..

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingfor, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain gener- alization using meta-regulariiztion. In NeurIPS, 2018.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. In arXiv preprint arXiv:1606.00 1606.61666.4)).. 2016.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Picter Abbeel. Benchmarking deep reinforcement learning for continuous control. In JCML, 2016a.
Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:16/ 1.02779, 2016b.
Chelsea Finn. Pieter Abbeel, and Sergey Levine. Model-agnostic meta- meta-learning for fast adaptation of deep networks. In ICML, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In arXiv preprint 1XXi:18806049100. 2018
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In ICMI., 2018
Josif Grabocka, Randolf Scholz, and Lars Schmidt-Thiee Learning surrogate losses. In arXiv preprint arXiv:1905.10108 2019.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In NeurIPS, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In arXiv preprint arXiv: 1801.01900 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. In arXiv preprint arXivv 2018b.
Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly C Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In NewrIPS, 2018.
Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Ángel Bautista, Shih- Sun, Carlos Guestrin, and Josh Susskind, Addressing the loss-metric mismatch with adaptive loss alignment. In ICMI,, 2019.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales. Feature-critic networks for het- erogeneous domain generalization. In ICMI., 2019.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
Franziska Meier, Daniel Kappler, and Stefan Schaal. Online learning of a memory for learning rates. In ICRA, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In arXiv preprint arXiv: arXiv:1312.55 13225602, 2013.
10

---

Under review as a conference paper at ICLR 2020
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane I Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533. 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML., 2016.
Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. In NeurlPS, 2019.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off off-polliy meta-reiiforremment learning via probabilistic context variables. In JCML, 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In ICLR, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmented neural networks. In ICMI., 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06344 2017.
Flood Sung, Li Zhang, Tao Xiang, Timothy Hospedales, and Yongxin Yang. Learning to learn: meta-critic networks for sample efficient learning. In arXiv preprint arXiv: 1706:99706.95599, 2017.
Richard Sutton, David McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIS, 2000.
Richard S Sutton, Hamid Reza Maei, Doina Precup, Shalabh Bhatnagar, David Silver, Csaba Szepesvri, and Erie Wiewiora. Fast gradient-descent methods for temporal-difference learning with linear function approximation. In ICML, 2009.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions auxiliary tasks. In NeurlPS, 2019.
Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie- Yan Liu. Learning to teach with dynamic loss functions. In NeurIPS, 2018.
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In NeurlPS, 2018.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems 30. 2017.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.
11

---

Under review as a conference paper at ICLR 2020
Supplementary Information
ALGORITHMS OF META-CRITIC FOR DDPG, TD3 AND SAC
We incorporate our Meta-Critic to the implementation of vanilla DDPG, TD3 and SAC, following their original implementations.
Algorithm DDPG-MC alggorthh
Initialize critic ((s, alO), actor (+)()) and auxiliary loss network hw
Initialize target network Q' and x' with weights 0' + 0, ø Initialize replay buffer R for episode = 1, ..., M do Initialize a random process A for action exploration Receive initial observation state 81
for t= T. do
Select action a; (() Ne according the current policy and exploration noise
Execute action observe reward r, and new state 8+41
Store transition (st, at, rt. 8t-1) in R
Sample random mini-batch of N transitions (8;, Oz, Tá, 8:+1) from R
Set y/ r, QQ'(st+,, 1s )
Update critic by minimizing the loss: L = N E, Qss a,))))
meta-train:
Calculate the old actor weights using the main actor loss:
V VoL main
-N11
V
Vor())))))
Dald
aV V main
Calculate the new actor weights using the auxiliary actor loss:
w N-1 Σ (5
sux V
&new Dold V LoLm
meta-test:
Sample random mini-batch of N syall from R Calculate the meta-loss using the meta-test sampled transitions:
meta tanh(L ( main (s, al0))==]"
main (s, a=
meta-optimization: Update the weight of actor and meta-critic network:
Φ (V main + VoL
I.
Update the target networks:
0' t 70 + (1 - -'
o T TO + (1 ()-
end for end for=0
12

---

Under review as a conference paper at ICLR 2020
Algorithm 3 TD3-MC algorithm
Initialize critics Qo Qz actor & and auxiliary loss network hw
Initialize target networks 04 < 01, 02 +- 02, of t- Φ
Initialize replay buffer B fort 1 to T do
Select action with exploration noise To(s) te, NO and observe reward and new state s'
Store transition tuple (s,a,r,7) in B
Sample mini-batch of N transitions (,,'), from B
à + (5') + e~ clip(N -c,e)
y g++t? min Q0 (s',a)
Update critics 0, f- arg ming, N 1- (v - (s, if mod d then
VaL -N-1 Σ V. Qo, (s, (s)m V (p() V =
= N-1 (VMMLP
meta-train
Calculate the old actor weights using the main actor loss: Pold Φ aV Φ) main Calculate the new actor weights using the auxiliary actor loss: &new = Dold VV Lin meta-test:
Sample mini-batch of val from B
Calculate the meta-loss using the meta-test sampled transitions: metn tanh L (a aa(0)))0lll)) a(01))a=x(""))0omm
L main (s (s a.0, (a=
meta-optinization:
Update the actor and meta-critic: Φ Ø ( V aL muin + V L
w t w V L
meta
Update target networks:
8 - 70: + 7)
8' T + (1 T)
end if
end for=0
13

---

Under review as a conference paper at ICLR 2020
Algorithm 4 SAC-MC algorithm
01. 02. ④i
// Initialize parameters
5 0, 02 + 02 D マt) 0
Initialize target network weights // Initialize an empty replay pool
for each iteration do for each environment step do
Og ~ Te (ap (8e) S1+1 1(8141184, ae)
Sample action from the policy Sample transition from the environment Store the transition in the replay pool
D t DUs ae (8t, az), s(41)}
end for
for each gradient step do
0, 0, Λo V Vo, (Q(e) for E {1,2) Update the Q-function parameters
meta-train
V VoT. N-1 Ee Vala log (ro(a)))) Qo s=)==))==)(e)) V a (
Oold = Φ a L
Calculate old weights of the actor
Vol. Voln NNNNI E. MLMLP (x))]= s=))
Onew Pold VoL
// Calculate new weights of the actor
meta-test:
L met tanh main (s, a s= a= beee..) L (s, a al0) (==(slll))
// Calculate meta-loss
meta-optimization:
Φ - Φ 17( V & main + V L
Update the actor parameters
w w A.
Update the meta-critic parameters
a XV AV (a)
Adjust temperature
0, - 70: - (1 +)0; for i (1,2)
Update target network weights
end for end for =0
14

---

Under review as a conference paper at ICLR 2020
7
AVERAGE REWARDS ON OTHER TASKS AND PPO-LIRPG EXPERIMENTS
Reacherv22
InvertedPart Panduku--22
InverssDautieeendilum
Reward
Avenage Roward
900
Average Reward
9000
-6-
700
7000
600
500
400
D3PG
5000
3696
300
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
Stops (1o6)
Stops (1.6)
Time Stops (2,66
Figure 7: Learning curve and standard-devvation of vanilla DDPG and meta-critic enhanced DDPG-MC for MuJoCo tasks with upper reward bound.
Reacher-v2
InvertedPendukmm-2
ImvertetDDubleeendduumz
-4.5
-5.0
Auvrage Retward
1000
900
800
Auee Average Reeard
8000
9000
-3.5
700
7000
-6.0
600
-6.5
-7.0
1039C
300
400
1059C
4000
5000
10
0.8
1.0
0.6
0.8
1.0
0.4
0.6
Steps (1e6)
Time Steps (le6)
Time
(1e6)
Figure 8: Learning curve mean and standard-deviation of vanilla TD3 and meta-critic enhanced TD3-MC for MuloCo tasks with upper reward bound.
Reacher-22
Inverte Pandalam-2222
InvertaDDubbeeendlluu--
1000
-4.0
-4.5
Average Reward
800
Avenge RvengeRewwrd
Reward
8000
-5.0
-3.5
600
7000
400
-65
SAC
200
SAC
5000
SBC
-7.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
Time Steps (1e6)
Time Steps (3e6)
Time Steps (1e6)
Figure 9: Learning curve mean and fenndardddevation of vanilla SAC and meta-critic enhanced SAC-MC for MuJoCo tasks with upper reward bound.
1200
6.0
400
200
2009
-320
200
Staze
Time Stge (304)
Stepe Cood)
Reacher-v2
Inverte wlum v2
InvertedDeubleeenduuuuee
1000
7000-
-20
Auerage Reward
Aver hrge Beward
800
5000
600
400
3000
-80
-100
1000
2
10
10
Time Steps (1e6)
Time Steps (2e6)
Time Steps (2e6)
Figure 10: Learning curve mean and standard standard-deviation of PPO and PPO-LIRPG for continuous control tasks.
15

---

Under review as a conference paper at ICLR 2020
Reacher-v2
InvertedPendie lum-22
InvertedDouble www.Douuubeeenuuuuun...
+
1000
-3
990
9000
8500
-5
960
-
950
7000
D0P3
0010
103
TD3MC
sac
m50
mouuto
Boo
203
tasacc
stc
19
193
suc
mo
Figure 11: Box plots of the Max Average Return over 5 trials of all time steps for MuloCo tasks with upper reward bound.
FURTHER ANALYSIS
8.1 META-CRITIC COMPUTATION
In terms of computation requirement, meta-critic takes around 15-30% more time per iteration, depending on the base algorithm. This is primarily attributable to the cost of evaluating the meta- loss T, and hence mamain
To investigate whether the benefit of meta-critic comes solely the additional compute expenditure, perform additional experiment where we increase the compute applied by the baselines to corresponding degree. Specifically, meta-critic takes % more time than the baseline, then we re- the baseline with K% more update steps iteration. This provides the baseline more samples while controlling the number environment interactions. Examples Figure 12 shows that increasing the number of update steps does not have straightforward link to pertormance. For DDPG, Walker2d-v2 performance increases with more steps, but stills performs worse than Meta-Critic. Meanwhile, for HalfCheetah, the extra iterations dramatically exacerbates the drop in performance that the baseline already experiences after around 1.5 million steps. Overall, there is no consistent impact of providing the baseline more iterations, and Meta-Critic's consistently good performance can not be simply replicated by a corresponding increase in gradient steps taken by the baseline.
HalfCheetah (rllab)
Walker2d-v2
8000
3500
Average Reward
Average Reward
3000
6000
2500
2000
4000
1500
2000
TD3
1000
TD3-MMC
500
DOPGIMC
TD3 moroupdatte
0
DOPG, meremmdeespsses
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Time Steps (1e6)
Time Steps (1e6)
Figure 12: Experiment controlling for compute time per method. Assigning more update itera- tions to the baselines so their running speed matches Meta-Critic. Left: Learning curves of TD3 HalfCheetah (rllab). Right: Learning curves of DDPG Walker2d-v2.
8.2 ADDITIONAL ENVIRONMENTS
In order investigate the impact of meta-critic harder environments, evaluated SAC and SAC-MC on TORCS and Humanoid(rllab). The results in Figure [3 show that meta-critic provides clear margin of performance improvement in these more challenging environments.
16

---

Under review as a conference paper at ICLR 2020
30000
SAC
10000
SAC
SAC-MC
SAC-MC
Average Reward
25000
Average Reward
8000
20000
6000
15000
4000
10000
5000
2000
0
40
100
0.5
1.0
1.5
2.0
2.5
Time Steps(1e3)
Time Steps (1e6)
Figure 13: Learning curves of SAC and SAC-MC on the TORCS driving (Left) and Humanoid (Right) environments
17


Reviews:
[{"review_id": "rkeydHcRKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am very torn about this paper as the proposed approach is a fairly straightforward extension of past work on the meta-critic approach to meta-learning and the results are pretty good, but nothing amazing. I tend to accept this paper because I like their general direction and think what they are proposing is pretty simple with broad applicability. It should be fairly straightforward to append this idea to most new off policy methods as they come out, so I find their consistent gains across 3 different popular models pretty convincing that this could have value to the community.  \n\nThat being said, the gains are not huge, which does make me think about the potential computational overhead. How much more run time per step does the meta-critic add to the models in the paper? I am a bit worried that the comparisons are not apples to apples from the perspective of the amount of computation/update steps per environment interaction due to the use of the validation data. I wonder if it changes the conclusion at all if the other approaches are given the same amount of computation on their replay buffer between interactions with the environment. For example, more optimization steps on the buffer is another plausible explanation why the meta-critic does better at optimizing the loss. \n\nI also should note that this paper is not the first to propose conducting online meta-learning over a replay buffer. A paper at last year's ICLR [1] did so in the context of lifelong learning, but does not need task labels or tasks and was tested on single non-stationary environments as well. The Meta-Critic approach of this work, of course, still is cool as it does for learning with a Meta-Critic what Meta-Experience Replay does for optimization based meta-learning. However, I thought this should be pointed out as the novelty can be a bit overstated at times. Additionally, the paper would be significantly improved by fleshing out the theoretical motivation for the Meta-Critic approach in more detail. What are the underlying reasons why we would expect it to generically improve single task RL? \n \n[1] \"Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference\". Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. ICLR-19. \n\nThoughts After Author Feedback:\n\nI really appreciate the response of the authors to my review, which included some interesting new experiments and explanations addressing concerns I raised. I do, however, also see where reviewer 3 is coming from with both major comments. \n\nI don't think that R3A1 is particularly clear and it is an important concern. I think reviewer 3 is saying that Delta(t) in R3A1i will eventually converge to 0 when the policy stops changing while the author argue that it \"need not converge if there are fluctuations at each iteration\". So it not converging seems to be tied to the existence of some source of non-stationarity in the problem. This doesn't seem to be coming from the environment as they are considering single task settings. As a result, I believe the source of the non-stationarity in this case is the fluctuating parameters. Looking at figure 6a it does not seem obvious that the policy has really converged in the traditional sense as its score does seem to be changing to some degree throughout the chart. My best guess is that this is the reason why the meta-loss does not converge. However, I still totally agree that this is a major concern that is very much under addressed. \n\nI also agree that I found the comments about what the meta-critic is doing unconvincing. The authors provided a few different kinds of explanations of what the model could potentially be doing, but this approach to the answer really highlights  how the theoretical benefits of this approach remain unclear. I think it should be possible to directly verify some of these theories with well designed experiments. It feels like a better explanation is necessary in light of the often small margin of difference with baselines before publication. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xRDNSNKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a meta reinforcement learning approach where a meta-critic is trained in addition to a conventional actor and critic, so that the meta-critic boosts the training of the actor over a single task. The approach is combined to DDPG, TD3 and SAC and is claimed to convey superior performance (or learning speed) over these state-of-the-art actor-critic algorithms.\n\nAt first glance, the paper looks convincing, but closer inspection reveals a potential issue that I would like the authors to discuss.\n\nThe first cue is that in Fig. 6, the meta-loss does not seem to converge to anything, as the authors say it just fluctuates. Shouldn't it converge once the actor converges to close to optimal performance? \n\nThe second cue is that appart from the rllab tasks (ant and half-cheetah), it is not clear that the meta-critic approach brings some gain in the end of training. Particularly in Reacher (Fig 7), the performance seems to collapse faster with DDPG-MC than with DDPG, and on the rest of curves of Figs 7 and 8 it is had to determine whether the MC approach brings something significant or not. And in Fig. 3, in Walker2D, TD3-MC looks rather unstable.\n\nThe third cue is that in Section 3.2, the paragraphs \"Updating MC parameters\" and \"Designing MC\" are rather unclear (I'll come back to that) and lack a theoretical justification.\n\nSo I'm wondering what exactly MC is doing and I would like to see a more detailled analysis. Couldn't a similar performance improvement be obtained by just increasing the actor learning rate and addiing some noise to the actor learning gradient? Isn't this more or less what the meta-loss does, when looking at Fig. 6?\n\nTo me, unless the authors give a clear answer to the points above, the paper should be rejected as it does not provide clear enough evidence and justification in favor of the proposed meta-learning approach.\n\nAs stated above, I found the paragraph \"Updating MC parameters\" lacking a principled justification.\n\nAbout \"Designing MC\", it could be much clearer. You first express two requirements (i) and (ii). Fine.\nThen you explain how to meet requirement (i), but you say this is not what you are doing (!), and then you try to explain what you are doing but without going back to the requirements. In particular, it is not clear at all why your design is \"permutation invariant\". You should be much more direct. Besides, the way to extract features and the relationship to batch-wise set-embedding should be made more explicit. Well, it is complicated and should be explained more clearly...\n\n\"TD3 borrows the Double Q learning idea...\": no, TD3 does more than this, as it uses the min between both critics, which double Q-learning and DDQN do not.\n\nYou use HalfCheetah-v2 from gym-mujoco and HalfCheetah from rllab. Why? One might suspect this is because the performance of the MC appraoch is only better with rllab...\n\nYou say you are using 10 million steps, but this is not always the case.\n\nIt is interesting to see that the performance of TD3 on Half-Cheetah collapses at around 4 million steps. Any idea why? It seems that in many papers experiments are stopped before performance collapses, and a strong study about this phenomenon is missing as the authors don't want to show this. This is an open call to readers: a paper on that would be great! :)\n\n\"SAC-MC\" gives a clear-boost for several tasks\": well, looking at the figures it is not so clear. Do you mean faster learning, higher final performance or both? I would like to see this claim backed-up by some proper statistical significance test and a clear specification of the number of seeds, etc. Fig.5 conveys the adequate data for such test if your claim is that this is the final performance that matters (but beware that curves are crossing eachother, so the final performance depends a lot on where you stop...).\n\ntypos:\n\np1 \"For example, ... (Zheng et al., 2018).\" is not a sentence (no main verb).\np2 \"the on-policy (approach?) needs to interact with (the) environment...\"\np2 \"is less effective than off-policy transitions.\" => unclear statement\np3 to assist(s)\n\nEquations (2), (6) and (8) should finish with a dot as they close a sentence.\n\nEq X => Eq (X) : use \\eqref{label} instead of just \\ref{label}\n\np5 It's input => Its\np5 two key technologies: I would not call this a technology. Ingredients?\np5 \"computational cost is ? by using\": missong word\n\np6: asmyptotic\n\np9: we removing\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Skgom5J6FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In actor-critic algorithms, the policy tries to optimize cumulative discounted rewards by a loss formed with components from the critic. In this paper, the authors propose a novel way, namely, meta-critic, which utilizes meta-learning to learn an additional loss for the policy to accelerate the learning process of the agent. There are several advantages of the proposed method, it's learned online for a single task, it's trained with off-policy data, it provides sample efficiency and it's generally applicable to existing off-policy actor-critic methods. \n\nOverall the proposed method is novel and the research direction is a very interesting one to explore. Eqn (3) and Eqn (4) explains the key idea of the proposed method. Eqn (3) describes the meta-learning problem as a bi-level optimization problem where the agent is updated with the main loss L^main with data d_train, in addition, it's updated with L^aux where the loss is learned and parameterized by \\omega. After the agent being updated, it uses L^meta and data d_val to validate the performance of the updated agent. Eqn (4) describes an explicit way to formalize the usage of the auxiliary loss, which is to accelerate the learning process. Thus the meta loss is whether the L^aux helps the learning process or not. \n\nHope that the authors could address the following issues in the rebuttal:\n1) Investigate why DDPG with meta-critic gets much more improvements than TD3/SAC;\n2) Show SAC (or TD3) could get better performance on harder task. (I can understand that they are strong baselines and hard to improve on the current environments, however, for harder environments, there might be rooms for improvements.);\n3) Investigate different ways to parameterize the meta-critic other than simple MLP;\n\nA few minor points:\n1) Page 2 first paragraph, \"This is in stark contrast to the\nmainstream meta-learning research paradigm – where entire task families are required to provide\nenough data for meta-learning, and to provide new tasks to amortize the huge cost of meta-learning.\". Try to revise it and avoid the words like \"mainstream\". Think about the paper being read 5 years later or even more;\n2) Consider introducing a weighting hyperparam between L^main and L^aux in Eqn (3), these two losses might have different scales and it might be better to weigh them differently;\n3) Minor literature detail: Page 7 \"Comparison vs PPO-LIRPG\" mentioned that \"Intrinsic Reward Learning for PPO (Zheng et al., 2018) is the only existing online meta-learning method that we are aware of.\", however, AFAIK, \"Meta-Gradient Reinforcement Learning\" in NeurIPS 2018 and \"Discovery of Useful Questions as Auxiliary Tasks\" NeurIPS 2019 are methods where meta-learning is applied online and for a single task;", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was extension discussion of the paper between the reviewers. It's clear that the reviewers appreciated the main idea in the paper, and the notion of an "online" meta-critic that accelerates the RL process is definitely very appealing. However, there were unanswered questions about what the method is actually doing that make me reticent to recommend acceptance at this point. I would refer the authors to R3 and R1 for an in-depth discussion of the issues, but the short summary is that it's not clear whether, if, and how the meta-loss in this case actually converges, and what the meta-critic is actually doing. In the absence of a theoretical understanding for what the modification does to accelerate RL, we are left with the empirical experiments, and there it is necessary to consider alternative hypotheses and perform detailed ablation analyses to understand that the method really works for the reasons stated by the authors (and not some of the alternative explanations, see e.g. R3). While there is nothing wrong with a result that is primarily empirical, it is important to analyze that the empirical gains really are happening for the reasons claimed, and to carefully study convergence and asymptotic properties of the algorithm. The comparatively diminished gains with the stronger algorithms (TD3 and especially SAC) make me more skeptical. Therefore, I would recommend that the paper not be accepted at this time, though I encourage the authors to resubmit with a more in-depth experimental evaluation.
Decision: Reject