Interpretable Meta-Reinforcement Learning with Actor-Critic Method
Authors: Xingyuan Liang
Xu-Ying Liu
Venue:  (2021)
Keywords: meta-reinforcement learning
actor-critic
deep learning
interpretable

Abstract: Meta-reinforcement learning (meta-RL) algorithms have successfully trained agent systems to perform well on different tasks within only few updates. However, in gradient-based meta-RL algorithms, the Q-function at adaptation step is mainly estimated by the return of few trajectories, which can lead to high variance in Q-value and biased meta-gradient estimation, and the adaptation uses a large number of batched trajectories. To address these challenges, we propose a new meta-RL algorithm that can reduce the variance and bias of the meta-gradient estimation and perform few-shot task data sampling, which makes the meta-policy more interpretable. We reformulate the meta-RL objective, and introduce contextual Q-function as a meta-policy critic during task adaptation step and learn the Q-function under a soft actor-critic (SAC) framework. The experimental results on 2D navigation task and meta-RL benchmarks show that our approach can learn an more interpretable meta-policy to explore unknown environment and the performance are comparable to previous gradient-based algorithms.

Under review as a conference paper at ICLR 2021
INTERPRETABLE META-REINFORCEMEN LEARNING WITH ACTOR-CRITIC METHOD
Anonymous authors
Paper under double-blind review
ABSTRACT
Meta-reinforcemmet learning (meta-RL) algorithms have successfully trained agent systems perform well on different tasks within only few updates. How- ever, in gradient-based meta-RL algorithms, the Q-function at adaptation step is mainly estimated by the return of few trajectories, which can lead to high variance in Q-value and biased meta- gettd-graiiet estimation, and the adaptation uses large number of batched trajectories. To address these challenges, we propose a new meta-RL algorithm that can reduce the variance and bias of the meta-gradient es- timation and perform few-shot task data sampling, which makes the meta-policy more interpretable. We reformulate the meta-RL objective, and introduce con- textual Q-function as a meta peetcy critic during task adaptation step and learn the Q-function under soft actor-critic (SAC) framework. 'hee experimental re- sults on 2D navigation task and meta-RL. benchmarks show that our approach can learn an more interpretable meta-policy to explore unknown environment and the performance are comparable to previous gradient-base algorithms.
INTRODUCTION
Reinforcement learning problems have been studied for a long time and there are many impressive works that achieved human-level control in real world tasks (Mnih et al. 2013 (iiver et al., 2017 Vinyals et al. [219]] Schrittwieser et al. 2019). These agents are trained separately on each task and may require huge sampled data and millions of trails. However, the cost of sampling data is not negligible, thus we cannot give agent large number of trails in environment. In contrast, human can laverage past experiences and learn new tasks quickly few trails, which is very efficient. Many tasks in fact share similar structures that can be extracted as prior knowledge, e.g., shooting games aims to eliminate enemies with weapons different environments, which can help agent generalize quickly through different tasks. Meta-learn (Thrun & Pratt 2012) reinforcement learning tasks can be a suitable chioce.
many real world tasks,
Meta- Meta-reinforceement learning (meta-RL) aims to learn policy that can adapt to the unknown vironment within few interactions with environment. Meta-policy can be seen as a policy that can derive new a policy maximizes the performance in the new environment. Gradient-based algorithms in nI. (Finn et al. 2017] Stadie et al.) 2018, Rothfuss et al./ [0188: Liu et al.) (219)) showed that meta-policy can be obtained by reinforcement learning a policy adapted by few reinforcement learning steps. The experiment results suggests that gradient-based methods can learn to sample and utilize sampled data in some extent. Nevertheless, the learning style and learned meta-policy are still far from human. Human learns new task by interacting with the task sequentially and efficiently. With the obtaining of environment data, human gradually understanding where to sampling data and how to utilize the sampled data to adjust the policy, while gradient-based algorithms use parallel sampling neglecting the relations between data. Sampling independently is not data-efficient, usu- ally needs number of stochastic trajectories to do plicy adaptation. This causes the agent relying on the stochasticity to sample and only learns how to utilize data.
Inspired by the human behavior, we propose K-shot meta-RL problem that constrains on the data amount accessed by agent, e.g., adapting policy within only two trails. Low resource environment simulates the real world tasks that have high costs data obtaining, therefore. requires agent to leam a stable strategy to explore environment. To address the K-shot problem, we also propose a contextual gradient-based algorithm using actor-critic method. The adptation step uses trail buffer

---

Under review as a conference paper at ICLR 2021
D to store all the transitions in K-shot sampling and optimizes expected value for the states in D. The meta-learning step optimizes the expected return performed by adapted policy while learning the value functions and context encoder using soft actor-critic (Haarnoja et al. 2018) objectives. We learn the policy with reparameterized objective that derives an unbiased meta-gradient estimation and reduces the estimation variance for Q-value. Our contribution can be summarized as follows:
We reformulate and propose the K-shot meta-RL problem to simulate the real world envi- ronment.
We propose new gradient-based objective to address the K-shot problem.
We introduce context based policy and value functions perform efficient data sampling.
We use actor-critic method to reduce the variance and bias of estimation in Q-value and meta-gradien.
RELATED WORK
Meta-reinforce learning algorithms mainly have three different categories: gradient-based motheds (Finn al. 2017: Stadie al.) 2018; Rothfuss al., 2018 Liu al. 2019 Nichol et al.) 2018) recurrent meta- learners (Wang et al. 2016 Duan et al., 2016), multi-task learners (Fakoor et al,, 2019 Rakelly et al. 2019). Gradient-based algorithms like MAMI (Finn et al. 2017) optimizing the policy updated by one step reinforcement learning, aiming at learning a good initialization of the policy weights. E-MAML (Stadie et al., 2018) considered the impact that the data obtained by meta-policy can influence the adapted policy's performance and assigned credit for meta-policy. While ProMP (Rothfuss et al. 2018) modified the adaptation gradient estimator to be low variance on second-order gradient. Recurrent meta-learners (Wang et al., 2016 Duan et al.) 2016) use RNN as a meta-learner that can learn new task from environment data while exploring. The RNN learn- ers are optimized with sequentially performed episodes end-to-end, which is more similar to the learning process of human and more interpretable meta-policy. Multi-task learners (Fakoor et al.) 2019 Rakelly et al. 2019) aim at learning multi-task objective solve meta-learning problems. They argue that meta-learning can be done by explicitly resuing the learned features through context variable. MQL. (Fakoor al., 2019)
even perform well without adaptation. PEARL (Rakelly jet al. 2019) constructs context encoder to infer the latent task variable and also learns a multi-task objective. The trained policy can perform structured exploration by inferring the task while inter- acting with environment. Our approach is related closely to the gradient-based researches which also tries to reduce the estimation variance and bias of the second-order gradient, however, we estimate the second-order gardient with value functions, and we still want perform structured exploration in data expensive environments.
3 BACKGROUND
This section focuses on the problem definition and notation of reinforcement learning and meta- reinforcement learning problems.
3.1 REINFORCEMENT LEARNING
Reinforcement learning (RL) problems aim to maximize the expectation of episode returns
E [R(T] = ET-P(rr)) E ((r(s(,
(1)
with single task and agent, where 7 (so, Cp, .o. } is the trajectory performed by the agent, so ~ pd is the initial state, a; ~ To(a, (st) is the action sampled from the policy π that parame- P( la,, st) is the state at timestep t, and P(* 5z) is the transition probability. The problem can be represented by a Markov Desicion Process (MDP) with tuple M = (S, A. R, lo, y, H), where S R²" is the set of states, A C Rm is the set of actions, P(s'ss, a) E R is the system transition probability, R(s, a) E is the reward function the task, and H is the horizon.
terized by 0, St+1
2

---

2017 Stadie et al. 2018) are mainly based on the Under review as a conference paper at ICLR 2021
Optimizing usually gradient descent and the gradient is estimated using vanilla policy gra- dient (VPG) estimator (Williams 1992)
VoBr~ [R(T)) E, P(r (T)
(2)
Σ
Volog (o Σ R(s.,,aa.)) (.r))
3.2 GRADIENT-BASHR BASED META REINFORCEMENT LEARNING
rentrrreinfore learning (meta-RL) aims to learn fast adaptation procedure that leverage the learned prior knowledge from training tasks and adapt to new tasks with few steps. A task T in meta-RL can also be defined by MDP Mr (S, A, Pr. RT,P' po Y, H). The task is drawn from a distribution ~ P(T), for simplicity, we only consider tasks with different reward functions or system transitions but the same state and action space.
Gradient-based meta-RL algorithms (Finn et al.
basic meta-objective (Rothfuss et al. 2018)
J(O) Br- PCT)) (
[R(7))),, 0' U = 0 + a
[R(T)],
(3)
where 0 is the weights of meta-policy, and 0' is the adapted weights after one step gradient descent. The meta-objective (0) optimizes the expectation of episode return sampled from the adapted policy Te. The meta-gradient can be written as
VoJ(0)= = P(T))EE Pr(11)) Port /Vor log Pr(()10)
(4)
Vo0' I + a
METHOD
4.1 REFORMULATE CETTA-REINFRRCEEERN LEARNING PROBLEM
Different tasks have different features in MDP, a task can be inferred from few important states and transitions in the environment, e.g.. different friction coefficients on floor, different rewards for the same state and action, or some states only exists in certain environments. We name these states and transitions the feature points of the environment. Humans usually learn task sequentially and efficiently since they can easily recognize the feature points in an environment. The exploration policy of a human changes significantly after obtaining data from the envioronment, thus they can decide where to explore and learn task quickly. However, formula (3) fast adaptation ((, T) usually refers to few gradient descent steps in initial weights 0, and unlike humans, the updating is performed in batched style normal reinforcement learning. Batched sampling usually contains a large number of trajectories in parallel, which can be inefficient for inferring the task. E-MAML (Stadie al., (0188) also tried to improve the sampling efficiency of meta-policy by accounting for the fact that samples drawn from meta-policy will impact the adapted policy. Inspired by the learning procedure of human, we reformulate the meta-RL problem K-shot reetrreeinforrement learning.
Given a task T ~ P(T), the agent samples data in trail phase and perform good policy in test phase. In trail phase, the agent can only sequentially sample K trajectories in total adjust its policy, with each trajectory of H length. In test phase, the agent is required to perform only one trajectory and make the return as high as possible.
Definition.
K-shot meta-RU problem defined above constrains the amount of data that can be accessed by agent, and is more similar the real world meta-RL problem, e.g., super mario maker. In K-shot setting, meta-policy can still be updated using U00 T) with batched trajectories, since they can be seen as sampled independently in sequence. However, the variance of the gradient estimation grows as K descends, which means the performance becomes more unstable. To optimize the problem, propose a new meta-objective
jK-shot (0) Pr
0' U(0, D)
(5)
= 0 + a VV (V
3

---

Under review as a conference paper at ICLR 2021
V
seoe
0eae
Figure 1: Left: The whole computation graph for K-shot meta-RL, where is the meta-policy parameter, is the trail buffer storing transitions and context in trail phase, T is the task sample from task distribution P(T), V is the value function evaluating policy æ. Straight lines represent for forward path and curved lines are backward path. K-shot Meta-RL optimizes the average return after policy adaptation using trail buffer. Right: The LSTM context encoder structure. A transition first embeded by a linear layer and fed through LSTM to form context &r at timestep (. The hidden state will be kept between different episodes.
for the K-shot setting. Here D is the state buffer sampled by meta-policy trail phase, and (s|c) the expected return of policy state under context (see 4.2 for details). The state buffer D contains H states as described in definition, which means the agent can only use few states to update its policy. Due the constraint on availble environment information, the agent is encouraged to learn to explore more important states that can help performing well test phase.
4.2 INTRODUCING CONTEXT
In meta-RL, the task T sampled from the task distribution is not given to the agent and can be thought of latent variable of the task MDP. The latent variable can be inferred and has strong correlation with the context variable e which is encoded by the experience (st, Pt, 81+1) ) collected until time step f. The context variable contains the information of two aspects. First, the experience is one step of transition and reward that represents the system dynamics and reward function of the environment. Second, the decision history {($, Jn represents the agent policy in the current environment. Q-function uses the state action tuple (st, ae) to evaluate the future discounted return of the policy at state & taking action a, which also need the same two-aspect information about policy and dynamics. Therefore, we introduce contextual Q-function Q(s, alc) that evaluate policy in unknown environment. To encourage the agent to learn how to sample efficiently and infer the task from the unknown environment, the agent should also context depended policy Te(als, c) to memorize past states.
Encoding the context variable e uses Long Short Term Memory (LSTM) (Hochreiter Schmid- huber 1997). The context encoder takes as input the history experience so far and output a context variable deterministicly. LSTM encoder has advantage of dealing with sequential data like his- tory transitions, thus can give a good representation of context. Addationally, LSTM context can represent for the same current state with different history states, which helps agent to explore more states and Q-function to evaluate the discounted return correctly.
We follow the setting in (Duan et al. 2016) to design the context encoding. Transitions are continu- ously fed into LSTM encoder while agent performing trajectories. The initial context is a zero vector and the context will not be reset after episode ends. This means agent can keep the information be- tween episodes and decide how to explore in next steps. With setting, the adaptation procedure is divided into two parts. First, the agent samples important states for itself in environment according to the data collected
far. Second, the agent uses all data available in buffer D to adapt policy. Through this process, agent can learn how to explore environment and how to utilize the transition data, which is more structured learning scheme.
4

---

Under review as a conference paper at ICLR 2021
4.3 LEARNING WITH ACTOR- CRITIC METHOD
Solving the K-shot problem in 4.1 requires value functions to evaluate the future expected return of policy и, therefore, training the agent in an actor-critic style can be good choice. The adaptation step in (3) uses reward term to estimate the Q-value. Even it is an unbiased point estimation of Qs+ 0t), the variance have can be very high (Konda Tsitsiklis 22000), and may lead to an unstable learning process. Actor-critic algorithms can trade-off between variance and bias of the estimation and the learned value functions can be used do adaptation.
To learn the value functions, use soft actor-critic (SAC) (Haarnoja al. 2018) framework. SAC an off-policy RL algorithm that tries to learn a policy with maximized entropy, thus the agent can trad-off between exploration and expolitation. We modified the SAC objective as
JSMC (0) D DV*
= Bg~ (ola [Q" (s, alc) - a log (alas,)
(6)
Q" (c, ))}.
-BE-dl D (To(-
|| exp(
adding context dependency value functions and policy, and the value functions also satisfies Bell- man equation
(" al))) = R(se,ay az) +
and
(sele)) Ea (Q" arlez) - a log (a:
(8)
where Ct+1 = enc(st, ae, relct) and enc is the LSTM context encoder mentioned in 4.2
Learning Q-function, V-function and LSTM encoder requires minimizing loss
_ = E "(s,,,,)~~ alene(((()))))) (r(s. (s(e((((((((())))))
(9)
and
LV --Ea-a((V*(s)) Bo=xotels.e
- a log (a(sss,)))))
(10)
where D is the replay buffer (Mnih et al.) 2015) storing the transitions experienced by agent, is the state, is the action taken state 8, is the next given state and action (s, a), r(s, a) is the reward at state after taking action T1:t-1 and Tı: represents the trajectory before state and including state s, and 1/ is the target value function to stable value iteration.
Substitue the adaptation objective in with (6) we have
U D) = Vo (aaaaaaaa IQ" alc) a log (o(a)s, c)),
(11)
where e refers the context state The gradient estimation in second term using VPG estimator
is
(s, VoBa-DDa 1Q" alc) log (a)ss,
(12)
Volog ( alc) - a log mo(als,c)
Here 1. means stop gradient. However, the second-order gradient of the analytical form and the Monte Carlo approximation form are not the same, which are
BgwD DewoDaevro log To(als e) + Vo lo mogao e) (QQ log (alss,))
(13)
and
N Σ v log () -olemllll)laomoollls.oooomo (o(
(14)
This will cause biased estimation in meta-gradient. Suppose policy mo is Gaussian distribution, the action can be rewritten as deterministic form a s(c), where ~ N 1), and the gradient term in (TI) can be reparameterized as
IQ" ( mo(e slo)le) log (ogmouu(sssssl)ssso.
(15)
5

---

Under review as a conference paper at ICLR 2021
Algorithm K-shot Meta-Reinforcente Learning
Require: trials K, horizon H, task distribution P(T), learning rates a, 3, ò Initialize trail buffer D' and replay buffer D' for each training task Initialize weights of μıa. enco, Qv, Vn, Vi
1: while not done do
2: for .. do
3: Clear trial buffer D
4: Sample T; from P(T)
5:
Sample trajectories from T, with μg while encoding experiences, and add D' and D
6:
Compute adapted policy using 0' = (0, D in 17
7:
Run adapted policy 0' for several turns to estimate average return R(7'),
Compute tett-ggradiet VoJ,K-ehot (0)
8:
9:
Sample batch of transitions D
10:
Compute gradients VOL. V V using sampled batch
end for
0 - 0 +0+B3 B E. VoJ.K-shot (0)
12:
13:
サ + サ B Σ. V
ワ- N Σ. V,
15:
o+ Σ, VoL.
Soft update target of f- (1 - 8)n/ 6n
17:
end while
Thus the second-order gradient of Monte Carlo approximation
Σ Vl + val(ssale)Vauutei
(16)
a
c) Vopolessle + Va logamalaass V.logmaaas.eyyaaaccss.t.
an unbiased estimation the analytical form, and from (4) we know that meta veta-ggadieet estimation be unbiased using this adaption form. To utilize all the available data in D, deterministic form for adaptation step in (5) and rewrite K-shot meta-objective as
RK-shot hot (0) ((r) ( [R(T')]].
0' = D)
(17)
= N Σ Q (st (o a log ve(ua(es scles) c:)
where 3;, 6,, C are the jth data in replay buffer collected at trail phase. The meta-RL problem proposed in can be directly optimized by (17) while learning value functions with ( and (10)
EXPERIMENTS
To evaluate our algorithm proposed above, we implemented our approach in several different meta- reinforcement learning environments, including 2d navigation task from (Rothfuss et al. 2018) and meta-RL benchmarks previously used by (Rakelly et al. 2019) in mujoco (Todorov al. 2012).
5.1 ENVIRONMENT SETUP
First we introduce the 2d navigation task. This task requires the agent to explore sparse reward environment, infer the goal point in an unbounded plane. The plane is divided into four parts with each goal in part. The agent starts from center of the plane and tries to obtain task data in trail phase and reach the goal test phase. The observation to be received is its coordinate concatenating the remained available steps. The reward is sparse and be the difference between the distances

---

Under review as a conference paper at ICLR 2021
2D Nanigatonn
Figure 2: The 2d navigation task result. The policy converged, and the performance is close to max score in task.
to the goal within two steps when near the goal, otherwise is set to zero. We use this environment to test whether the agent can learn to sample different states and use these states to adapt to the right policy. Second, we describe the mujoco benchmarks. Mujoco tasks are environments for controlling robot in simulated physical world to learn task adaptation. We tested three mujoco environments: MiriChheetaFFowwddBBa.
AntRandDir, HalfCheetahRanddel HalfChectahForwardde the agent to run forward or backward as fast as possible, AntRandDir requires agent to run in two random selected directions fast possible, and HalfCheetahRandVo requires agent run with certain speeds.
requires
5.2 RESULTS
In this section we will show the experiment results of our approach. In 2d navigation task, the meta- policy learning curve converged easily early training steps and the trained meta-policy is shown in figure 3 The agent have three trails on each task, then perform apdated policy for testing. In each trail, the agent perform 100 steps and total 300 steps in trail phase. This means our approach uses less steps to figure out the task than it is in Rothfuss et al. (2018) which used 2000 steps. As shown in figure agent chose very different states to explore. Each trail the agent will visit the states that have not been explored, and states are separated in plane with clear bounds. These states helps the agent to infer the task efficiently, and the meta-policy can performed data expensive environments.
Figure 3: meta-policy and adaptation in 2d navigation, where pink circles are the goals green circle zero reward area. Agent not in the right part corresponding the current goal or in green circle will get zero reward. each task agent has three trails to collect data, then perform the test phase.
We also evaluated our algorithm in mujoco meta-RL environments. The results are showed in figure 4 The performances of our algorithm slightly higher than the previous gradient-based algorithms. The data amount we sampled at trail phase are also less than is in MAML., ProMP and even PEARL (Rakelly al., 2019). Each trail we sampled 200 steps for total 2 trails, the data used to do adaptation in our algorithm is 10% of in PEARL and 5% in ProMP.
Thhe MAMI. and ProMP results obtained from published results Rakelly al. 2019)

---

Under review as a conference paper at ICLR 2021
Figure 4: meta-policy and adaptation in mujoco environments. The performance is compared with previous mudient-based algorithms. The performance is better than the previous algorithms.
CONCLUSION
In this paper, proposed and have given a new meta-RL algorithm that optimizing with contextual policy and actor-critic framework. Our approach can estimate unbiased meta-gradient and reduce the estimation variance of Q-function. From the experiments, we demonstrated that contextual policy can sample efficiently in data constrained environments. Finally, the experiments on mujoco environments suggested that our algorithm can have competitive performance with other gradient-based algorithms. Human behavior can usually bring us inspiration on designing the intelligent system and maybe is a key to AGI.
meta-RL. problem that contrains the data amount utilized by agent,
REFERENCES
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:/611.0277 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J Smola. Meta-q-leaming, arXiv preprint arXiv:1910.00123 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-
policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv: 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Vijay Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances neural information processing systems, pp. 1008-1014, 2000.
Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- 2019.
reinforcement learning. In International Conference on Machine Learning, pp.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, nature. 518(7540):529 533, 2015.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv: 1803:0099..99 2018.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off oof-py rentoreeifor learning via probabilistic context variables. In International conference on machine learning, pp. 5331-5340, 2019.
8

---

Under review as a conference paper at ICLR 2021
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784, 2018.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. arXiv preprini arXiv:191 1et.19911..2.... 2019.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676;354-357) 2017.
Bradly C Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv: 118, 2018.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEF/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEFE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaöl Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(77822:350-354 2019.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn arXiv preprint arXiv:1611.05760 2016.
Ronald J Williams.
Simple statistical foodient algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.


Reviews:
[{"review_id": "z0u9pYP6aro", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### **Summary and Contributions of Paper** \nThis paper proposes to improve the K-shot RL meta-learning problem by using an LSTM, whose repeated inputs are (s,a,s') state-action transitions, and whose step-wise outputs are context vectors. The policy, during the K-shot phase, then additionally observes the current context vector along with the standard state, in order to produce an action.\n\nIn order to train both the policy and the LSTM in an end-to-end fashion, the paper modifies the MAML objective appropriately, since the LSTM's context vector is used throughout the entire K-shot process. \n\nExperiments are performed on the Nav-2D 4-corner exploration task (proposed by ProMP, 2018) and a few standard Mujoco meta-learning tasks.\n\n### **Strengths**\n- Seems to be novel in the sense of allowing an RNN to control how the policy explores the task during the K-shot phase. This is in contrast with previous works which used manual methods for the policy exploration.\n- Once I understood Figure 3 (past the presentation issues), the LSTM's context vector seems to signal to the policy which areas to explore next (rather than entropy-dependent exploratory movement in the default MAML algorithm) in the K-shot phase, which is good exploration behavior.\n\n\n### **Weaknesses**\n- One of the most obvious issues are the numerous grammar issues/awkward phrasings. The writing of the paper can be significantly improved. This also includes the mathematical notation, as variables are sometimes used abruptly without introduction. I do think this paper has promise, but really needs to be cleaned up in its presentation.\n- Experiments can be more comprehensive. So far, there are only 3 Mujoco tasks used, whereas there are 8+ to choose from (see T-MAML, 2019). More Mujoco benchmarks will make the results convincing. Furthermore, the authors only plotted the recorded scores from previous papers (as horizontal lines) rather than run the baselines themselves. One issue here is that in many cases, there are multiple task definitions/tweakings that are not the same across papers. For instance, the original MAML paper and T-MAML used completely different scalings for the alive bonus, movement reward, and the energy expenditure penalties for the task \"ForwardBackwardAnt.\" The authors need to run the baselines themselves in order to have rigorous metrics. This should not be difficult, as there are already very clean implementations, e.g. see (https://github.com/lhao499/taming-maml).\n\nI think the paper proposes some promising ideas. However, due to the weaknesses mentioned, I think the paper needs to be cleaned up a lot more before being ready for submission, and thus I propose rejection.\n\n\n### **Clarity Questions**\n- Can you provide some explanations for pros/cons against PEARL (which also uses a context vector approach, but uses a feed-forward VAE-like network for the context vector?) PEARL seems to be the most direct competitor to the proposed paper's approach, as both are using context vectors.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "whfj3opi86_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors introduce a new meta-RL algorithm based on SAC. It uses a context variable $c$ that they condition the Q-function on and the adaptation mechanism which is based on the values of the value function (ie. $\\mathbb{E_a} Q(\\dot, a)$) instead of the true returns. Authors claim their method reduces variance and bias of the meta-gradient estimation, is closer to human learning, encourages the agent to learn to explore, is more data-efficient in test-time and has competitive performance among gradient-based algorithms.\n\nI found many of the claims of the paper are unjustified:\n1. Authors claim \"we reformulate and propose the K-shot meta-RL problem to simulate the real world environment\". The formulation that follows is the standard one (cf. MAML (Finn et al. 2017): \"In K-shot reinforcement learning, K rollouts from $f_\\theta$ and task $T_i$ (...) may be used for adaptation on a new task $T_i$\". It is unclear how does the authors' definition relate to the real world.\n2. They suggest their method is able to \"learn where to explore\", but nothing in the method specifically addresses this part (compared to eg. Learning to reinforcement learn (Wang et al. 2016), which also has a learned context that influences the policy).\n3. They suggest their method learns more \"human-like\", which is understood as \"not using batched sampling\" (as, arguably, humans learn more sequentially). However, their method also uses batched sampling (see eq. (17)). In another part of the paper human-like learning is associated with the usage of LSTM, which is also neither novel (see Wang et al. (2016) again) nor grounded.\n4. They claim their method decreases bias and variance. Gradient estimation in a typical meta-RL method is unbiased, so that's hard to decrease. On the other hand, I believe the variance of the proposed method is increased compared to E-MAML and others due to approximating rewards with V, which, due to the fixed-capacity of the model, will make their adaptation procedure inherently biased and thus estimating gradients in incorrect places.\n5. Their method \"makes the meta-policy more interpretable\", yet no interpretation attempt was presented.\n6. \"Agent can learn how to explore environment and how to utilize the transition data, which is a more structured learning scheme\". The learning scheme is basically the same as in PEARL (Rakelly et al. 2019). It's not clear what exactly makes the learning scheme \"more structured\" nor what it means.\n7.  \"Each trail we sampled 200 steps for total 2 trails, the data used to do adaptation in our algorithm is 10% of in PEARL and 5% in ProMP.\". According to Rothfuss et al. (2018), app. D.2, they used a single trajectory of 200 steps for adaptation for HalfCheetahFwdBack, which is half of what the authors' method used. On the other hand, PEARL achieves much better results than the authors' method, what is not mentioned in the paper.\n\nDetails of the method are not very clear, I assume they follow SAC with online/target networks, but am not really sure nor what $\\mu$, $\\phi$ and $\\eta$ mean. Mechanics of meta-testing also weren't described in enough detail.\n\nThe writing of the paper is terrible, to the point it's not always clear what the authors mean. Most of the problems are simple grammar errors, which are easily solvable by a tool like grammarly or google docs correction. As such, I consider sending the paper to the review in its current state a disrespect for the reviewers who have to spend the time to decipher the writeup.\n\nGrammar problems (first page only):\n1. However, in many real world tasks\n2. Derive a new policy **which** maximizes\n3. The experiment results suggest that\n4. Utilize sampled data **to** some extent.\n5. **While** obtaining environment data\n6. Human gradually understand**s** where to sample data\n7. Stochastic trajectories to do p**o**licy adaptation\n8. \"This causes agent (...) and only learns how to utilize data\" - not clear what was meant\n9. That have high cost o**f** obtaining data", "extra_scores": {}, "extra_text": {}}, {"review_id": "wFytLGTzGrA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper proposes a new approach for meta-RL. The paper claims that the proposed method reduces variance and bias of the meta-gradient estimation by only a few samples. In addition, this paper claims that their method is more interpretable. \n\nMy comments:\n\nThere are lots of unknown, unwarranted claims about this paper in addition to no thorough experiments and comparison with previous works :\n \n1. Paper claimed that their proposed method not only reduces the variance of meta-gradient but also reduces bias. Reading through this method and experiments, I don't see any theoretical or empirical justification why that should be the case.\n\n2. The method claims this method has a more interpretable meta-gradient. Again, there is nothing in this paper that verifies this claim.\n3. Paper proposed to use fewer samples for the adaptation phase. I don't understand why this can help at all.\n4. Experiments are incomplete and there is no rigorous evaluation to analyze this method.\n5. The proposed context in this paper has been already proposed in previous work, not sure what is new here.\n6. There are lots of things that need to defined like trail, few-shot, etc. \n\nIn summary, this paper is not ready at all and needs lots of works. Hence, I'd recommend this paper to be rejected.", "extra_scores": {}, "extra_text": {}}, {"review_id": "tPfDdRCR_gX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new meta reinforcement learning algorithm that uses trajectory information to create a contextual embedding for each task. The context is used as a given condition for learning the Q function and policy using soft actor critic algorithm. \n\nSuggestions to improve the quality of the paper:\n- The title states \"interpretable\" but there is no interpretability discussed anywhere in the paper. A more relevant title would be appropriate.\n- The idea of contextual embedding using trajectory and LSTMs have been explored in multiple papers (e.g. PEARL (Rarely et al. 2019), MQL (Fakoor et al., 2019). It is not clear how the proposed method is different from these. \n- It is unclear how adding context information to the Q function leads to structured exploration. SAC does not use Q function to explore, it uses the probability distribution of the policy network output for exploration. Prior work has shown that SAC exploration can be improved with dual Q function and upper confidence bounds, but that is not being used here: https://papers.nips.cc/paper/8455-better-exploration-with-optimistic-actor-critic \n- It is unclear how the K-shot version of the meta-RL algorithm is different from other meta-RL algorithm. The proposed algorithm also samples from the tasks uniformly, same as prior algorithms. \n- There are numerous typographical and grammatical errors throughout the paper. These can be easily fixed with modern text editors. Some examples - trails, data in some extent, where to sampling data, plicy, meta-gradien, want perform, availble, help performing, subistitue, contrains, etc. \n- Figure 3 and 4 are very difficult to read, even on a computer. The graph title, axes labels, axes tick marks, legend are all ineligible. The quality of these graphs need to be improved for readability. \n- The evaluation results are over two trials, need more of them for a convincing argument that the proposed method is better than prior methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "s1YkYEeSYYG", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n----------\n\nThis paper presents an approach to meta-RL based on combining gradient-based updating with recurrence-based meta-learning. The approach is based on combining gradient-based policy updating with recurrence-based updating of the value function. The authors evaluate the method of simple standard meta-RL benchmarks. \n\n\nComments\n----------\n\nOverall, the direction of the paper is interesting but the paper has numerous shortcomings. \n\nFirst, the proposed method is a fairly straightforward combination of existing techniques. In particular, the approach consists of a fairly simple combination of gradient-based and recurrence-based methods. While this is not necessarily a limitation, it necessitates a thorough set of experiments to justify the combination of approaches. This is the second limitation of the paper: the experimental evaluation is very limited. The authors test of very simple benchmark problems compared to recent work in meta-RL. Moreover, there are few comparisons to baseline methods (especially PEARL) and the authors should include ablation experiments in which they examine the performance of their method relative to strictly gradient-based and strictly recurrence-based methods, using the SAC algorithm as an underlying algorithm. \n\nFinally, the writing of the paper is extremely sloppy. Trials is misspelled as trails throughout the paper. There are also numerous typos, such as \"gardient\", \"meta-reinforce\", \"Substitue\", \"apdated\", etc. The presentation of the algorithm is quite unclear and the discussion of related work is quite limited. \n\nOverall, the paper should be tested on a wider range of environments and against more competitive baselines to warrant acceptance. The authors should also improve the presentation of the paper to improve clarity. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A meta-RL algorithms that aims to improve meta policy interpretability by reducing the meta-gradient variance and bias estimation. The method is evaluated on an exploration in 2d navigation and meta-RL benchmarks.

Despite an important topic of research, the reviewers are unanimous that the paper is an early version and requires further work to be suitable for publishing. Specifically, the future versions of the manuscript should address the novelty by better distinguishing from the prior work, improve the evaluations, presentation of the work.

Decision: Reject