Meta Attention For Off-Policy Actor-Critic
Authors: Jiateng Huang
Wanrong Huang
Long Lan
Dan Wu
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
meta learning
Attention Mechanism

Abstract: Off-Policy Actor-Critic methods can effectively exploit past experiences and thus they have achieved great success in various reinforcement learning tasks. In many image-based and multi-source tasks, attention mechanism has been employed in Actor-Critic methods to improve their sampling efﬁciency. In this paper, we propose a meta attention method for state-based reinforcement learning tasks, which combines attention mechanism and meta-learning based on the Off-Policy Actor-Critic framework. Unlike previous attention-based work, our meta attention method introduces attention in the actor and the critic of the typical Actor-Critic framework rather than in multiple pixels of an image or multiple information sources. In contrast to existing meta-learning methods, the proposed meta-attention approach is able to function in both the gradient-based training phase and the agent's decision-making process. The experimental results demonstrate the superiority of our meta-attention method in various continuous control tasks, which are based on the Off-Policy Actor-Critic methods including DDPG, TD3, and SAC.

Under review as a conference paper at ICLR 2022
META ATTENTION FOR OFF-POLICY ACTOR-CRITIC
Anonymous authors
Paper under double-blind review
ABSTRACT
Off-Policy Actor-Critic methods can effectively exploit past experiences and thus they have achieved great success in various reinforcement learning tasks. In many image-based and multi-source tasks, attention mechanism has been employed in Actor-Critic methods to improve their sampling efficiency. In this paper, we propose a meta attention method for state-based reinforcement learning tasks, which combines attention mechanism and meta-learning based on the Off-Policy Actor-Critic framework. Unlike previous attention-baset work, our meta atten- tion method introduces attention in the actor and the critic of the typical Actor- Critic framework rather than multiple pixels of an image or multiple informa- tion sources. In contrast to existing meta-learning methods, the proposed meta- attention approach is able to function in both the gradient-based training phase and the agent's decision-making process. The experimental results demonstrate the superiority of our meta-attention method in various continuous control tasks, which are based on the Off-Policy Actor-Critic methods including DDPG, TD3, and SAC.
INTRODUCTION
Reinforcement Learning (RL) algorithms based the Actor-Critic framework have achieved con- siderable success in many areas such as games, robot control, and planning. Compared with on- policy methods, off-policy methods possess more efficient sampling since they do not require new samples to be collected for each gradient step and make better use of experience (Haarnoja et al., 2018a). However, even for off off-policy methods, traditional reinforcement learning algorithms still have extremely low sample efficiency Yu 2018). Recently, meta-leaming (Hospedales et al.) 2020) has become topical as a paradigm to accelerate RI. by learning an inductive bias from past expe- rience. By learning aspects of the learning strategy, such as fast adaptation strategies (Finn et al.) 2017) RRakelly et al.] 22019), losses (Zhou et al. 2020) (Bechtle et al.. 2020), optimization strate- gies Duan al. 20166), exploration strategies (Gupta al.) 2018), hyperparameters (Bechtle al.) 2020 and intrinsic rewards (Zheng et al., 2018), meta-leamiin has significantly improved sample efficiency over standard RL algorithms.
Improving sample efficiency through attention mechanism has also been proved to be very effective in image-based reinforcement learning (Bratt & Chen, 2019 Chen al., 2019). The application of attention mechanism in multi-agent system Tesm al. 2021) also shows its powerful capabilities in information processing. However, in the existing works, attention mechanisms often have clear application scenarios, such as image- based control or multiple sources of information (multi-agent multi-target). The effective combi- nation of attention mechanisms with current algorithms in state- state-based single-agent environment is still a problem to be investigated
et al. 2002 Igbal & Sha 2019) and multi-object
In this paper, we propose meta attention method based on the attention mechanism. In the human decision-making process, people often modify their concepts based on the feedback and results to obtain a better decision. Inspired by this decision-making process, we use meta attention to adapt the features generated by the policy network based on the evaluation of value network. Our work differs from current attention-based work in that our meta attention approach works only within the Actor-Critte framework and does not depend on a specific scenario. We formalize the meta- learning process as a bi-level optimization problem. Our approach can be flexibly combined with various algorithms by using meta attention as a meta-learner and optimizing meta attention in the outer layer. Unlike the existing meta-learning methods, our meta attention approach can improve the

---

Under review as a conference paper at ICLR 2022
performance of agent through gradients in the training stage and obtain better actions by adjusting the features in the execution stage.
We evaluated the proposed meta attention method in series of continuous control tasks in Gym and Roboschool, including three 3D robot control tasks, two 2D control tasks, and one classic control task based on DDPG 2018b). Besides, we also discussed the changes and impact caused by modifying the actor features through meta attention. Experimental results how that our meta-attention approach is not only ef- fective in accelerating the learning progress of the agent in the training phase, but also improves the actions in the execution phase, further enhancing the performance of the agent.
al. 2016). (Fjimmoo et al. 2018) and SAC(Haarnoja
al.)
2 RELATED WORK
Attention Mechanism Attention is a behavioral and cognitive process of selectively attending to discrete aspect of information, whether subjective objective, while ignoring other perceptible informationEde Santana Correia & Colombini 2021). Typically, attention mechanism is mainly applied in the fields of computational vision and natural language processing. Accordingly, although the implementation methods are different, the application of attention mechanism in reinforcement
2 Chen et al. 2019 Barati & Chen 2019) Mott et al.) 22T99) et al. (219)). Other works such as Peng et al. (2020) proposed a dynamic attention model with a dynamic encoder-decode architecture, which dynamically explores node features to efficiently exploit hidden structural information at different construction steps. Li et al. (202)) applied the attention mechanism to generate feature vectors and input them into value and policy head during the feature extraction phase of PPO and PPG & Lu (2018); Iqbal & Sha (2019): [Ma et al. (2019) employed multi-head attention mechanism to make one agent selectively pay attention information from other agents. Team al. (202T) used the attention mechanism to match multiple goals with the hidden state of the current state to obtain the goal-attention hidden state under different goals. This approach allows the agent to predict the expected return obtained by attending to goal at the end of a scene.
learning is mainly focused on video games Wu et al.] 202T]
Meta Reinforcement Learning Meta-learning is most often understood learning to learn, which refers to learning from historical information or multiple learning episodes to improve learn- ing algorithms. Since works which fed historical trajectories into Recurrent Neural Network to flesh
out task-level information Wang et al. 2016; Duan et al. 2016). various meta-learning methods have been proposed to strengthen agent performance. [Houhoooft et al. (2018): 8); Kirsch et al. (2020);
Zhou et al. (2020) used meta- meta-learning methods to learn a loss function rather than artificial design improve performance of agent in single multiple tasks Gupta et al. (2018); (ttdie et al. (2018): Xu et al. (0) employed meta-learning methods to learn exploration instead of traditional ex- ploration methods/Finn al. (017)) meta learned a good model initialization of the model that can be quickly adapted to different tasks. Xu et al.] (2018b) improved the performance of the agent by meta-learning discount factors Rakelly et al. (2019 Fakoor et al. (020) treats meta information as an unobservable state of Partially Observable Markov Decision Process, further improves the agent's s performance in multi task learning. Although the attention mechanism may not have an explicit meta-learning object, it can also be considered the king of the meta-learning method.
Bi-level Optimization Generally, meta-learning can be formalized as a bi-level optimiza- tion(BLO) problem. However, the solution of BLO problems is often challenging. |Fraccechii et al.. (2018) proposed framework for approximating the solution of BLO problems using the gradient method. Since then, many works based on the gradient methods to optimize meta leaner have suc- cessfully proved the feasibility of gradient method, such a<Li et al. (2019): Finn & Levine (2018);
Lian et at. (2020) Flennerhag et al. (2020). For the RL problem. /Zhou et al. (020) optimize the meta critic, the leper-level of a BLO, as an intrinsic motivation by gradient method. Kirsch et al. (2020) enables population of agents to use and improve single parameterized objective function through gradient learning on different tasks. Rajeswaran et al. (2019) proposed an implicit MAML algorithm by drawing upon implicit differentiation that effectively decouples the meta- greederrddnntt computation
from the selection of an inner loop optimizer. [iu et al.] (2019) proposed a surrogate objective function TMAML, which incorporates control variables into gradient estimation through automatic
2

---

Under review as a conference paper at ICLR 2022
differentiation and improves the quality of the gradient estimation by reducing the variance without introducing bias.
3 METHODOLOGY
3.1 OFF-POLICY ACTOR-CRITIC
In general, the reinforcement learning task can be considered as finding the optimal policies in Markov Decision Processes (MDPs). The MDP defined by a tuple (S, A, PR), where set of states, A is set of actions, IP is a set of probabilities to switch from a state to s' for given set of action a, and S x - is scalar reward function. the Actor-Critic framework, the policy (ectork(Actor) (s) and the value network(Critic) Qo a) are parameterized by each time I, the agent receives an observation 8, and takes a action aj based its policy S - A. then receives reward r; and new state S1+1- The tuple (31, ae, Tt, St+1) describes a state transition and will be stored in a reply buffer D for off off-policy learning. The objective of RL is to find the optimal policy n to maximizes the expected cumulative return
neural network respectively.
(Ø) = E
Σ r R (sc, at) a T (- se)
1=
Where y is the discount factor, and (Ø) also can be written as the expected value for the Q-function:
í( = Ex-p- Qo(s, a)la.>mp( st)
(2)
Where Q-function is the expected discounted sum of rewards following visitation at state s and execution of action a, and pr is the state distribution induced by policy For off- off-pooicy Actor- Critic architectures such as DDPG, TD3 and SAC, the loss for the actor provided by the critic may be different, but the Q-function will be learned by minimizing the loss of the below equation as same:
Lg Critic Barpparrr [Qu (St, ar) y²)²
yt Qu (s1+1, Tι (sc+1)),
where ø' and represent the target network for the critic and the actor respectively. The actor loss usually differs in details according to different algorithms, however, they all follow form of the following formula:
T. Actor J(() Eapp. Q0(s, (0 (ammm()
3.2
META ATTENTION METHOD
The attention mechanism has been widely used in image recognition and natural language processing since it model the human pattern recognition process. It allocates attention the important part of information while automatically ignores low-value features. According to (Vaswani et al. (017)), attention function can be described as mapping query and a set of key-value pairs to an output, where the output is computed as a weighted sum of the values, with the weight assigned to each value being computed by compatibility function of the query with the corresponding key:
Attention(Query Key, Value)
Σ
Similarity (Queryi, Key;) * Value
In order to introduce the attention mechanism into the Actor-Critic framework, we first split the actor and critic network into two parts We take the last layer of actor network t action net to produce actions and the rest as actor feature net मे ((s) to extract features, with the entire policy network being denoted (s)
*()(s)). Similarly, the value network also be divided into a critic net Q and critic feature net Q(s). and the whole value network can expressed as Qs(s) = Q(Q(s,a)) We use the feature net as encoder to obtain Query, Key and Value by the following formula:
Querry =s() Key Q(8, To(s)) Value ((s)
3

---

Under review as a conference paper at ICLR 2022
Ante
Ciise
Mecinni
Sunz
Action
5000
Oscry Viles
(Nita stentoo)
Action nzi
Fastao Sade
Sigeeit
Festure
Crttic net
hationne
Actien
Coilc
(a]
(6)
Figure 1: (a) Traditional Actor-Critic framework. (b) The meta attention Actor-Critic framework. The part circled by the orange box represents the actor, the part circled by the blue box represents the critic, and the part circled by green dashed line represents the attention method. There is two orange(blue) boxes in (b), but they refer to the same actor(critic). Lines with different colors represent the update process dominated by different losses.
We input Query(actor features) and Key(critic features) into meta attention network fo (Query, Key) (a three-layers MLP parameterized by vi) to calculate the similarity of each feature dimension. To enhance or reduce the features specifie corresponding dimensions, we multiply the output after the sigmoid function by to obtain the feature scale. In this paper, we use to denote the Hadamard product, and by calculating the Hadamard product of Value(actor features) and scale, obtain the attention features H' t'(s) after modifying some dimensions:
Attention Features मed (s) for ( Q( mT()))) 2 7(8)
This calculation process corresponds to the part in the green dotted line box in Figu
There are also two critical problems in the optimization process. 1) how to affect the agent's decision-making and training process through attention features; and how to optimize the meta attention network to achieve the correct matching Query(actor features) and Key(critic features) to generate proper feature scales. To address these issues, we formalize the entire optimization process bi-level optimization problem, referring to the meta attention method as the outer level, and to the task performed by the agent as the inner level:
arg min I Attention (b)
(6)
s.t. = min LL (D: øla To(S)) L Actor (Did: ((F((s)))
where L L Attentoon is the meta optimization objective, and L Actor the actor loss in the form of Eq.44
For the first problem, we first implement traditional back bockppropagate dominated by the actor loss A.Aorrr] (s) on training data d
Actor (dernla To(3))
Pold = x
(8)
do
Where 77 is the learning rate, this process corresponds to the upper part of Figure [ the first step of meta-training in Algorithn[]] Then, generate new attention action
(b) and

---

Under review as a conference paper at ICLR 2022
( (s)) from the attention features through action net and feed into the critic to get the loss A (D; 1b) (o))( for back-propagation:
Actor
(deri bbaa ((F)(s)))
Onew
Vold
11
ab
This allows the agent obtain better actions by simply modifying the features compared the orig- inal action without increasing the batch size, strengthening the agent's 's learning of good actions and reducing the probability of producing poor actions after the gradient step. This process corresponds to the lower part of (b) in Figure and the second step of meta-training in Algorithm
For the second problem, we have two basic assumptions: inevitably generate feature scales that can generate actions with higher value since it better associates relationship between actor features and critic features; 2) in the process of back-propggation, high-value actions enhance actors more than low-value actions because good actions will strengthen the actor's tendency to make such choices, while bad actions cause the actor to try other actions. Furthermore, since the meta attention network is involved in the back propagation process, the utility meta attention in this process as the attention loss L Attenton and the attention loss on validation data dval defined as:
good meta attention network will
I. Attemtinn tanh (L.guter (dwat Thnew (s)) A.te (drat Tobra (+))
(10)
Under this definition, when performing gradient descent updates, it ensured that the meta attention is always updated along the direction that improves the performance of the agent. This process corresponds to the blue line part of (b) in Figure and the meta-test Algorithm[]
3.3 BEYOND THE GRADIENT
Environment
Collic Resuree md
Asiin net
Fesure net
Chikk
Adarr
State
3
Criti (whrchh is better?)
Excution bettes sctionn
Amcettonn
Feanires
Mosa Atiantion
Figure 2: When the agent interacts with the environment, in addition to the action produced by the traditional Actor-critic framework, meta attention also attempts to make further adjustments to the initially produced results the hope of achieving better results.
Unlike previous gradient-based meta-learning approaches, our proposed meta attention method can affect the learning process of the agent through the gradient and participate in each of their decisions. In the traditional Aectr-Critic framework. the critic estimates each action's 's value based the current state, which provides criterion for evaluating the action produced by an actor. The Critic and the trained meta attention make possible for meta attention methods to participate in decision-making.
When people are given the rules for scoring in human docision-making they often modify their previous ideas to obtain better results. Similarly, we combine the attention method with the critic to introduce this decision-making method into the AC framework. For the deterministic policy, when an agent is given a state from the environment, we can get another attention action by adjusting the actor features in addition the actions directly produced by the traditional Actor-Critic framework. Then the action with the higher Q value given by the critic is executed. This decision process is shown in Figure and corresponds the environment step in Algorithm 1
5

---

Under review as a conference paper at ICLR 2022
This method of action selection is like actor dancing once more after performance, getting another score by adapting his or her actions based on the feedback given by the critics, and choosing which set of actions to be performed formally by comparing the two scores.
For stochastic policies such as SAC, their actions under a continuous control task are typically sampled from Gaussian distribution. Although it is meaningless to compare sampled actions, can still compare and modify the actor features in terms of the mean value of Gaussian distribution. Another attentional action is then sampled from the Gaussian distribution and the parameters (mean and variance) generated by the attentional features are modified.
Although the Q-value given by the critic may be inaccurate in the early stage of training, and off- policy algorithms typically have serious over estimation problem, these issues are beyond the scope of this paper.
Algorithm Meta attention for off-policy Actor-Criti
Initialized parameters 0, v/; D 1 and learning rate 17, for each iteration do for each environment step do if the actor dances twice then T (8t) or A ( (8))
% Select the action with higher Q-value
else
at T (5t) end if
% Traditional action generation process
St+1 p( st, oz) rt D {(t) ae re, 86-1)}
% Observe reward Tr and new state 8 % Store the transition in the replay buffer
end for for each gradient step do
Sample mini-batch dern from DD
LCritic + Eq. 3)
0 0 - - V Critic
Critic
% Critic update
meta-training:
Bold < Eq.88
% Vanilla Actor update
Onew Eq.
Actor update based on attention action
meta-test:
I Attention <- Eq.10) 10
% Attention loss
meta-optimisation:
④ Onew
% Update the actor parameters
AV L Attention
% Update the meta attention parameters
end for end for
4 EXPERIMENT
4.1 SETUP
To evaluate our meta attention framework, selected four 3D continuous robot control tasks in Roboschool and two 2D continuous robot control tasks in Gym without modifying the original environments or rewards. Although MuJoCo is widely used to test various reinforcement learning algorithms, Roboschool provides a more realistic and difficult robot control task than MuloCo. Compared with MuloCo, Roboschool reduces the alive bonus and increases drive costs to encourage movement in low-energy manner. Roboschool also made other adjustments, such encouraging the Ant robot to have two or more legs on the ground to ensure that the robot moves in a more natural posture.
We applied our meta attention framework to DDPG, TD3 and SAC as the basic algorithms. Algo- rithms that use meta attention methods only in the training stage will be annotated as .MATT (such as DDPG MATT), while algorithm that uses the meta attention method both the training stage and agent decision-making process will be annotated MATT DTT(such as DDPG MATT DTT).

---

Under review as a conference paper at ICLR 2022
We also compare -ATTT and ..AAT-D'' with another SOTA work (Zhou al. 2020 which proposed Meta Critic (.MC) to enforce the basic algorithm.
All our codes are built the code provided by Zhou et al. (2020). We ran tasks for million or hundred thousands game steps depending on the environment, and we evaluate our policy over 10 episodes without exploration every 1000 game steps. All results for each task averaged over 10 random seeds (trials) and network initializations. 95% confidence intervals are shown as shaded regions over time steps. Following Fujimoto et al. (2018), curves smoothed evenly for clarity (window_siz==10),
Experiments in all environments had the same network structure and hyperparameters, with learn- ing rate of 0.003 for DDPG and TD3 but 0.001 for SAC. All experiments are performed on 4 servers with 4 NVIDIA-TITAN- GPU each and servers with NVIDIA-TITAN. GPUs. Each server is equipped by 2 Intel(R) Xeon(R) Gold 5218 CPUs.
4.2 EVALUATION OF META ATTENTION METHOD
#ctoehooliogaee
350
-200
Roboschos www.arrrcuuunoooooooooooooo
DOGS
-00
20
322
2.6
33
33
20000 20-00 VIROI
10000 2100
Figure 3: The learning curves of DDPG and enhancement algorithms. The shaded region represents the 95% confidence interval.
1480 3303
550
760
340
2001
100 marr
750 PNTT
-30
120
Figure 4: The learning curve of TD3 and enhancement algorithms. The shaded region represents the 95% confidence interval.
Deterministic policy Figure 3 and Figure 4 demonstrate the comparison between the vanilla
algorithms and their enhancive versions based on DDPG and TD3. From the figures,
7

---

Under review as a conference paper at ICLR 2022
observes that MATTT and MATT DT achieve significant performance improvements in all but the RoboschoolHalfCheeaahhy environment. DDPG_MATT perform almost as well as DDPG MATT DT under the environments BipedalWalker-v2, www.rhowwwwweeeeeeeedaaaaaiiiiii
Munarlanderonttnuouss But in these enviroment, TD3 MATT DT perform sig- nificantly better than TD3 TD3_MATT, and we speculate that this phenomenon may be related more accurate Q-values.
and
Similar Meta Critic, _MATT and .MC only work in the training stage and influence the ac- tor through gradients, but DDPG_MC performs even worse than the original algorithm in four environments, while DDPG_MATE shows performance improvements in all environments except JeboschhooHajjjheetthvr/ TD3 MATT also shows high performance in more environment than TD3 MC. From the data provided in Zhou et al. (2020), we can also find similar situation: the performance of _MC decreases in the first million steps in deterministic policy based algorithms. This also shows that our meta attention method completes the meta-learning process faster than Meta Critie.
1230
300
BCC
690
430
para
100
3.2
128:
112
Figure 5: The learning curve of SAC and enhancement algorithms. The shaded region represents the 95% confidence interval.
Stochastic policy Figure compares the SAC and its enhancive versions. Although SAC_MATT has only achieved performance improvement in environments Bipedal and SAC_MATT.ND DD still achieves performance improvements in all en- vironments other than RoboschoolAnt-vl and wwtpsshoollveere//enddunnniieine gests that our method is also effective in stochastic policy, but the improvement in stochastic policy less pronounced than in Deterministic policy. We speculate that, compared with deterministic strategies, sampled actions in random strategies bring more uncertainty to the learning of meta at- SAC_MATT and ACC in the four environments is similar, which is the same as or slightly higher than the original algorithm. The performance of the first million steps also matches the data provided in the Zhou et al. (202)).
This sug-
tention. The performance of SAC MATT
4.3 FURTHER ANALYSIS
In order to analyze the effect of meta attention on the features, we ran 5 random seeds(trails) in Roboschoo/Wallee22ddw environment. each trial, we ran a total of million game steps and evaluate our policy over 10 episodes without exploring every 1000 game steps. Figure 6 (a) shows an overview of this experiment, and the shaded region represents the 95% confidence interval.
We recorded the Q-values of attention and actor features during the evaluation stage verify the impact of meta attention on performance. In the evaluation phase, we took I million consecutive Q-values of the actions generated by attention features and actor features. These million samples correspond to million time steps in the evaluation stage or 10 evaluations. We used the error between Q-values of the attention action and vanilla action performance improvements brought
8

---

Under review as a conference paper at ICLR 2022
KoboschoolWalker22h.1
1000
800
mean reward
600
pine PM
(6) traininn steps
340
200
TD3
TD3_MATI MATT DT
140
0.0
0.4
0.6
0.8
game step
106
-M8
(e) lourning
(c) afber 120000 training steps
(6) afler 40000 training
afler 160000 training steps
Figure 6: (a) Learning curves, the shaded part represents the 95% confidence interval: (b-e) The results obtained by sampling different stages (green circle in (a)) of training. The left part represent the difference in Q-values of actions generated by attention features and actor features, respectively The right part represents t-SNE visualization of attention features and actor features, where red dots represent attention features and blue dots represent actor features.
by meta attention. We sampled at four different time points in the training phase, and these results represented in Figure 6 (b-e).
To analyze the diversity of attention features and actor features, we sampled 250 attention features and actor features on continuous 250 time steps during evaluation on one trail. We used t-SNE to visualize these total 500 features. Similarly, we sampled at five different time points in the training stages, and these results are shown in the right part of Figure (b-c).
Figure 6 (b-e) shows the results of sampling at approximately 40000, 80000, 120000, 160000 train- ing steps on one trail of million time steps or 200000 training steps. In terms of improvement(left part of Figure 6 (b-e)), the actions obtained after the modification of our meta attention usually terms the distribution (right part of (igure(((b-e)), attention features have changed in most cases compared with actor features, as shown by the apparent separation of red points and blue points. This suggests that our meta-attention ap- proach is effective in modifying features and improving performance in the current state throughout the training phase.
achieved higher Q-values at different stages of training.
5 CONCLUSION
In this paper, we present the meta attention method, a derivative-based meta learner for off-policy Actor-Critic reinforcement learning methods. By modifying the actor features, we speed up the training process and enable agents to make better decisions when interacting with the environment. The experimental results demonstrate the effectiveness of our method in both stochastic policy and deterministic policy. Moreover, our method can be flexibly integrated into a variety of contemporary off- off-policy Actor-Critic methods to boost performance. In future work, will use the multi-heal attention to improve the meta learner and decouple the meta attention method from the value function to reduce the impact of the over estimation problem.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Elaheh Barati and Xuewen Chen. An actor-critic-atteniion mechanism for deep reinforcement learning in multi-view environments. In Sarit Kraus (ed.), Proceedings of the Twenty-Kighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 2002-2008. ijeai.org, 2019. doi: 0/4496///ccai.201//777 URL https: //doi.org/////
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gau- rav S. Sukhatme, and Franziska Meier. Meta learning via learned loss. In 25th Interna- tional Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021, pp. 4161-4168. IEEE, 2020. doi: 10.1 N0.109//CP488806.000 2021 2911210...00 URL 2021.
https://dof www.ooggg11... wtt.:11119//111B 242..9900112222...21.1 9412010
Yilun Chen, Chiyu Dong. Praveen Palanisamy, Priyantha Mudalige, Katharina Muelling, and John M. Dolan. Attention-based hierarchical deep reinforcement learning for lane change be- haviors in autonomous driving. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2019, Macau, SAR, China, November 3-8, 2019, pp. 3697-3703. wt.p/...r1111101 1109/
IEEE, 2019, doi: 1099//ROO0009777099966666666
IROS40397. 6968565
Alana de Santana Correia and Esther Luna Colombini. Attention, please! survey of neural at- tention models in deep learning. CoRR, abs/2103.16775, 2021. httttttt::///aa. org/
Abss2113.16777
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RI
Fast reinforcement learning via slow reinforcement learning. CoRR, abs/161, 2016 URL
http:://axiie httppp///axxiv.or//abbba.s
www./osg////..o..ooo.oo...o..../ 611 .07799
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-leaming. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httph: http:s:://openzeieeww net/
SJJDDCRFPP
Chelsea Finn and Sergey Levine, Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In 6th International Conference on Learn- ing Representations. ICLR 2018, Vancouver. BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpthttt///peeeee
ew net/ forum?id
Hy HyjC5yWCW
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th Interna- tional Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-1 1 August 2017, volume 70 of Proceedings of Machine Learning Reseurch, pp. 1126-1135. PMLR, 2017. URL mmlr.
http::/ppocced/eee
Sebastian Flennerhag, Andrei A. RUsu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Had- sell. Meta-learning with warped gradient descent. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net forum? ddrremmiddre
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riceardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Leurning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15. 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1572-1572. PMLR, 2018. URL httt:://proceedinn mlr. www.ppesssssse. http:///ww//aaccccchhhccceecc
html
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Learning, ICML 2018, Stockholmssässsan Stockholm, Sweden, July 10-15, 2018 volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. PMLR, 2018. htt.ttttt//////
ngs ml www.prsssssss. www.s//v0/eee./ fww::oo//fifmooooooooo 8a html httpsssswrrrrreeee.eeee..
10

---

Under review as a conference paper at ICLR 2022
Abhishek Gupta, Russell Mendonca, Yuxuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Samy Bengio, Hanna M. Wal- lach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- fion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 5307-5316, 2018. URL
neurips. http:///wwecaaaee.e..e..e.//aaa/ pctppppr//2//////ttt/// 2eee332222811cccbeeeeeeeeeeeeeee
Abstract. html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, 1bs/1812,, 2018b. URL
1812. 05905
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta-learning in neural networks: A survey. CoRR, abs/2004.05439, 2020. URL https::/arxxv..rgggg../o
2004. 05439
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Ad- vances in Neurul Information Processing Systems 31: Annual Conference Neural Informa- fion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. wwwpeediiiiin....i...o.ee....eee html
5405 2018 URL
htc//apprr2218///00
Shariq Iqbal and Fei Sha. Actor-attention-critiic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2961-2970. PMLR, 2019. URL http:// proceedings.oo , mlr. www.srrsssssee........ / /ttp9/////aalla0 html
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 31: Annual Con- ference on Neurul Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, neurips.cc/ .ttl
Montréal, Canada, pp. 7265-7275, 2018. URL httt:s::/proccceinnggg
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httptttpp///ppeeeeeeeee
net/ www.mmmmmdddeeeeeeeeeede
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M. Hospedales. Feature-critic networks for heterogeneous domain generalization. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 9924-3924. PMLR, 2019. URL http::/ppoceedding/ mlr. p html
Yunfei Li, Tao Kong, Lei Li, Yifeng Li, and Yi Wu. Learning to design and construct bridge without blueprint. CoRR, abs/2108.02439, 2021. URL htt.lttts//axxv..cooo http://axxx.....//baaa/aa/ 2108 02439
Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 20-30, 2020. OpenReview.net, 2020. URL http::tppp////ttt bttps:://penreviert net/ cettforrumid
eewAANFV
11

---

Under review as a conference paper at ICLR 2022
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
//arxiv.org/aa hbtpg//bb//100099 02971
Hao Liu, Richard Socher, and Caiming Xiong. Taming MAMI: efficient unbiased meta- reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 4061- 4071. PMLR, 2019. URL WRLLLttp://prcceediiiii
mlr wrw.sppsssse.......... / wttp//1/1ll/. 9g. html
Anthony Manchin, Ehsan Abbasnejad, and Anton van den Hengel. Reinforcement learning with attention that works: A self-supervised approach. In Tom Gedeon, Kok Wai Wong, and Minho Lee (eds.), Neural Information Processing- 26th International Conference, ICONIP 2019, Syd- ney, NSW, Australia, December 12-15, 2019, Proceedings, Part V, volume 1143 of Communi- cations in Computer and Information Science, pp. 230. Springer, 2019. doi: 10. 3 030 36802 9
000-3-030-36800--99 25. URL https::///// wtt.::.........o..oo..0
25
Hangyu Mao, Zhengchao Zhang, Zhen Xiao, and Zhibo Gong. Modelling the dynamic joint policy of teammates with attention multi-agent DDPG. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor (eds.), Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pp. 1108-1116 International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL http://dll acm. wttp://........t.tooooooooooiaa. cit
Alexander Mott, Daniel Zoran, Mike Chrzanowski, Daan Wierstra, and Danilo Jimenez Rezende. Towards interpretable reinforcement learning using attention augmented agents. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcéé-Bue, Emily B. Fox, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 12329-12338, 2019. URL -aaseaeeeeeeaaaaaoaaaaaaeeeeeeee 1hppr//211//aahh//11101111313
www.ris...... cc/ Abstt .ttml
Parnika, Raghuram Bharadwaj Diddigi, Sai Koti Reddy Danda, and Shalabh Bhatnagar. Atten- tion actor-critic algorithm for multi-agent constrained co-operative reinforcement learning. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé (eds.), AAMAS '21: 20th In- ternational Conference on Autonomous Agents and Multiagent Systems, Virtual Event, United Kingdom, May 3-7, 2021, pp. 1616-1618. ACM, 2021. URL http::ttt///dlt acm. org/doi/ 10.5555/3455 00155/3466955 3464178
Bo Peng, Jiahai Wang, and Zizhen Zhang. A deep reinforcement learning algorithm using dynamic attention model for vehicle routing problems. CoRR, abs/2002.03282, 2020. URL https:
03282
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine Meta-learning with implicit gradients. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence A'Aché--Buu Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- neur ips cc/ppapar///111//hatt Abstract
113-124, 2019. URL -Astaaaaa..tteeeeeeaaaaaaaaaa.. html
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcemmett learning via probabilistic context variables. In Kamalika Chaud- huri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019. 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5331-5340. PMLR, 2019. URL http: 7/proceed ngs. ml press wwwpppaww///ppppppppppp /atas//99//akke 1 9a. html
12

---

Under review as a conference paper at ICLR 2022
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and llya Sutskever. Some considerations on learning to explore via reinfrreinor learning. CoRR, abs/1803.01 2018. URL
http//aas.////e.ee/e/ 01118
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michaël Mathieu, Nat McAleese, Nathalie
Nathamiel Wong, Nicolas Porcel, Roberta Raileanu, Steph Hughes- Fitt, Valentin Dalibard, and Wojciech Marian Czarnecki. Open-ended learning leads to generally capable agents. CoRR, abs/2107.12808, 2021. URL httpp::/// http://axvvvcco wwwpxxi..rr//bb//11111 12408
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. neerrre wwtppppppe/////aaeeeeeepeeeeeeee 2017. URL ott
Abst
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn.
CoRR, abs/1611.0576. 2016. URL http:://aaxxv.orggabb////eeeeeee 05763
Haiping Wu, Khimya Khetarpal, and Doina Precup. Self-supervised attention-awart reinforcement learning. In F'iitty-Ffth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 10311-10319, AAAI Press, 2021. URL https::/tpsss.o.. www....aa.aa.................... org/ index.ndxxx.e php/ BAAI/article/view/
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to to with meta- meta-policy gradient. 2018a. URL
CoRR, abs/1
05000
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta- Meta-gradient reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2402-2413, 2018b. URL ht httt:////ccoeceeeeeeeeeeeeeeesee neurips cc/paper/ htmm
2018 /000////177751111100000000333320
Yang Yu. Towards sample efficient reinforcement learning. In Jérôme Lang (ed.), Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp. 5739-57433 ijcai.org, 2018. doi: 10.24963/ijcai.ii 2018//2000000 URL
.2..1...44..0 24963 tttp:::/ddoi.org100 jicii............ 2018/820
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for pol- icy gradient methods. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 4649 4659, 2018. URL htt.::::/pprceeeings.. neurips......................... 51de85add068 68 Obe 7ee.777766666 (71776611361111 - html
Wei Zhou, Yiying Li, Yongxin Yang, Huaimin Wang, and Timothy M. Hospedales. On- line meta-critic learning for off-policy actor-critic methods. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neu- ral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- tual, 2020. URL cosfffaas5333add3332251l......e3 Abstraatttttttttttttc.c. html
13


Reviews:
[{"review_id": "vCh_G6OQZfs", "reviewer": "Reviewer_qn99", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main weakness of this work is its clarity. This work would severely benefit from adding high-level motivation / justification behind the design choices of the proposed architecture. For example, it's unclear what the goal of adding attention to these actor-critic architectures is -- is it just to improve the expressivity of the model? Or are explicit inductive biases that this work is trying to add in? The rationale behind the design choices for the attention are also unclear. Why is it reasonable to attend over the actor features using the Q-values as the key? Without laying the high-level ground work, it's difficult to understand what this work is trying to accomplish, and exactly what it's doing. It's worth noting that the paper does try to provide some high-level motivation and states:\n\n> In the human decision-making process, people often modify their concepts based on the feedback and results to obtain a better decision. Inspired by this decision-making process, we use meta attention to adapt the features generated by the policy network based on the evaluation of value network.\n\nHowever, it's still not entirely clear to me how this justifies the proposed attention mechanism (feedback and results seem more closely related to reward signal than a critic), and the concrete design decisions remain unjustified.\n\nAdditionally, in several areas, it would be helpful to be more precise. For example, line 4 of the algorithm 1 pseudocode states \"if the actor dances twice,\" in reference to the top of page 6. It's somewhat difficult to understand precisely what the algorithm is doing here, although from the surrounding context, I gather that the algorithm just allows for sampling actions under two policy architectures and selecting the one with higher Q-value under the critic.\n\nAs a minor comment, the notation is also somewhat confusing. I would urge the authors not to overload $\\pi$ with several variants, such as $\\hat{\\pi}$ as the last layer of the actor network and $\\bar{\\pi}$ for the feature representation, which makes it difficult to keep track of what means what.\n\nThe experimental results seem promising, although the improvement of the proposed approach over baselines is fairly modest in many tasks. Additionally, the plots are cutoff while many of the curves appear to still be learning, so it's difficult to gauge if the proposed approach indeed achieves a higher final performance. I would recommend running these curves for slightly longer until they've actually plateaued.", "summary_of_the_paper": "This work proposes a new architecture that adds attention to actor-critic RL methods. This new architecture is evaluated on several Roboschool tasks.", "summary_of_the_review": "I do not believe this work is ready for publication in its current form due to significant clarity issues. I was not able to fully understand all the details of the proposed approach, so it is possible that I've missed some crucial details, but I don't think its current presentation is sufficiently clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lD7mVIw5-d", "reviewer": "Reviewer_R4EG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper brings an interesting perspective to RL. The attention mechanism combines the information from the actor and the critic to further refine the actor representation. This is also supported by better empirical performance from the proposed method.\n2. The paper is mostly clearly written and easy to follow, with informative diagrams for explaining the proposed method.\n\nWeakness:\n1. The proposed algorithm does not exactly match the bi-level optimization formulation from Eq. 6 and 7. In particular, Eq. 6 is broken down into two separate gradient steps. In addition, argmin from Eq. 7 is approximated by a single gradient step. Lastly, \\phi^* is not used during meta training. Instead, the algorithm requires \\phi_old and \\phi_new as input for meta training.\nThe issues above makes it challenging to understand the proposed method from a conceptual level: what exactly is meta attention optimizing for?\n\n2. I find the several design choices ad-hoc and not well motivated. Specifically,\n\n      a. The attention scaling factor of 2 is not explained.\n\n      b. The use of tanh in the meta-learning loss is not a standard choice and deserves more justification.\n\n      c. The meta-learning loss considers actions produced by \\phi_new and \\phi_old, but not the action produced by the attention mechanism. Could the authors further elaborate on this choice?\n\n3. The presentation for the empirical results is too dense (figures only), and without any numerical comparison. It is difficult to assess how much better than proposed method is compared to the baselines. It's also difficult to assess the stability of different methods, when the figures appear to show significant standard deviation of performance for the meta-trained methods. Could the authors provide numerical comparison of the experiments?\n\n4. It is difficult to interpret the results between _MATT and _MATT_DT variants. _MATT seems to outperform in some settings while under-perform in the rest. Most crucially, Neither _MATT nor MATT_DT exactly matches what the model is trained on (e,g. there is an extra selection step for _DT). Could the authors elaborate on the comparison between the two variants? Could we compare with the attention action without the extra selection step?\n\n5. There is no discussion on the additional computational complexity imposed by the meta-learning formulation. It would be good to compare the computational resources needed, as well as the sample complexity of different methods.", "summary_of_the_paper": "The paper introduces attention mechanism into actor-critics method, and formulates RL as a bi-level optimization to learn the (meta) attention parameters. The attention mechanism appears model agnostic and acts on the feature representation from the actor and critic models. Empirically, the proposed model shows improved performance over baseline methods.", "summary_of_the_review": "The proposed method brings an interesting perspective combining RL and meta-learning, and demonstrates better empirical results. However, the model design is not well motivated, and the optimization objective does not match the practical algorithm. Lastly, the empirical results are too concise for detailed performance assessment. I think the current paper would benefit from another revision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MCK-RuccSkd", "reviewer": "Reviewer_ihtm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The basic idea of this paper is quite interesting; it introduces an attention mechanism that tries to exploit differences in representations between the policy and the value function to improve the policy. The attention mechanism is also learned in an interesting fashion, as it is formulated as a meta-learning problem: the goal of the attention layer is to be such that it improves learning of other components of the agent. Empirically, the authors demonstrate that the algorithm does yield performance gains. \n\nWith that said, this paper is not ready for publication. A minor issue is that the writing makes several sweeping statements that cannot be verified or that have no clear meaning. As an example, the authors motivate their work in the abstract by the phrase \"our meta attention method introduces attention in the actor and the critic of the typical Actor Critic framework rather than in multiple pixels of an image or multiple information sources\". This is clearly false, as the most common way to use attention is either in the form of a transformer or as neural memory, neither of which operate directly on pixels.\n\nSimilarly the technical presentation is imprecise, which both calls in the question what the authors are actually proposing and whether the exposition is correct. Concretely, the presentation of the RL framework is not complete and somewhat incorrect; the true return in Eq. 1 cannot be written in the form presented in Eq. 2, as that is an approximation involving a learned critic. Further, it is odd to write the policy objective as directly maximising the learned Q-function–the policy-gradient maximises the observed reward via an advantage estimate A = Q' + r - Q. \n\nMore central to the proposal, I find the proposed attention mechanism rather different from what I expected. The authors cite the Attention is All You Need paper and use the same nomenclature, but rather than the usual softmax look-up operation, they use an MLP to compute the Query-Key look-up and match the values by an element-wise product. While that is valid, given that the attention mechanism is a well-established concept, I would expect such a departure from the norm to be well-motivated, but the authors only present it as is. \n\nThe imprecision of technical descriptions and lack of motivations become a serious problem when trying to parse the loss functions introduced in Eqs. 7-10. In Eq. 7, the authors minimise a loss function with respect to $\\phi$, but it is not clear to me where these parameters are used. I understand that they define the parameters of the policy, but looking at Eq.5, are they also part of the $\\bar{\\pi}$, or does that function have separate parameters? In Eq.8, the authors mention training data, but don't define it. How does training data and validation data differ? Is it just two separate draws from the replay, or is there a separate validation replay that is being used? Similarly, after Eq. 9, the authors motivate their two-stage approach by \"this allows the agent to obtain better actions by simply modifying the features compared to the original action without increasing the batch size, strengthening the agent's learning of good actions and reducing the probability of poor actions after the gradient step.\" I am still not sure what is meant by this, isn't it a policy-gradient step on the agent's policy parameters? For the attention loss in Eq. 10, could the authors please commend on why simply taking the policy-gradient of the post-adaptation actor-critic loss is not sufficient?\n\nOverall, while I do appreciate the novelty of the idea and some attempts on the author's part to empirically demonstrate that their attention mechanism does provide non-trivial benefits, I feel that the authors have mainly presented a new architecture without proper motivation for the involved design choices, nor any empirical or theoretical results to motivate the components. Empirical results on the method as a whole only indicate moderate improvement that is within the statistical confidence interval of the baseline. As such, they are not sufficient to conclude that the proposed method represents a significant improvement for actor-critic methods.\n\nSmaller nits: \n- The algorithm chooses an action according to the highest Q-value if the 'actor dances twice'. I'm assuming this is a typo.\n- I cannot parse the first sentence of paragraph 5.\n- Final sentence, second paragraph on page 1 is missing a period.", "summary_of_the_paper": "This paper proposes an attention-based actor-critic agent. The authors propose to parameterise the actor and the critic with two separate neural networks. Their algorithm operates in a two-stage fashion. In the first stage, the algorithm is a standard actor critic that produces an action probability distribution and a prediction of the action-value function. In the second state, the algorithm attends over the policy's and value function's hidden features. This output of the attention mechanism is a new feature representation that is fed into the policy's and value function's final layer, to produce an alternative action probability distribution and action value-function. The agent then takes the action that has the highest predicted return. The authors evaluate their proposed algorithm under DDPG and SAC on standard continuous control tasks and demonstrate that it can improve performance.", "summary_of_the_review": "This paper proposes an attention-based architecture for actor-critic algorithms. While the idea is novel and has interesting elements to it, this submission is not ready for publication. The presentation is imprecise, to the point of obfuscating what is being optimised, while the motivation is either non-existent or makes sweeping claims that has no clear support. Empirically, the results do not sufficient to conclude that the proposed method presents an improvement for actor-critic algorithms.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9hPQAcNCUX", "reviewer": "Reviewer_v8k3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe paper is generally well-written. Figure 1 and Figure 2 clearly present the main idea of this work and it helps understand this work.\nIt is relatively novel to consider the idea of applying attention mechanism between actor and critic instead of only for image feature extraction. \nThe proposed method can be easily used in any off-policy actor-critic network and the experiments show the advantage of the proposed method on continuous control tasks.\n\nWeaknesses:\n\nIt seems that this work positions itself in the literature of meta-learning. They call it \"meta-attention\" (in abstract), \"meta learner\" (in conclusion). Considering that most works in meta-learning investigate the multi-task setting. It might be better to add experiments of the proposed method on meta RL benchmarks? Please correct me if this impression or understanding of the proposed method is wrong.\n\nIt is unclear to me, what does the proposed attention mechanism learn to pay attention to? Consider the image-based attention mechanism or the multi-object attention mechanism, the attention is used to extract the important region of an image or an important object in the set of objects. In this work, the attention weight is calculated for each feature dimension. Then why should some dimensions be more important than the other dimensions? What information is represented in each dimension? Could you please provide an intuitive explanation about the meaning of attention in the proposed method?\n\nAnother question is also related to the attention mechanism. It is said: \"meta attention network to calculate similarity of each feature dimension\". Why a three-layer MLP could calculate the similarity of actor feature and critic feature? How is the similarity defined?\n\nIn equation 10, why we should use tanh instead of other possible functions to formulate the loss for attention? Is there any intuition behind it?  Did you try other design choices of attention loss?\n\nIt is said the challenges of value estimation in RL are \"beyond the scope of this paper\". This point is not very convincing to me. The proposed method employ critic features to adapt the actor features (through the attention mechaism) and expect that the new action features are better for actor neural network. This means the advantage of the proposed method heavily relies on good critic estimation. When the critic estimation is problematic or not accurate enough, will the proposed attention mechanism even harm the performance of the standard off-policy actor-critic? This point will definitely limit the application of the proposed method to more challenging RL problems. I think this might be an important weakness of the proposed idea.", "summary_of_the_paper": "This paper proposes to modify the off-policy actor-critic framework by introducing an attention mechanism in the actor and critic. The attention mechanism is used to adjust the actor features (i.e. intermediate features generated by the actor neural network) for better action selection in the continuous control tasks. The query is the actor feature. The key is the critic feature. The output from the attention network is the similarity of each feature dimension. The new actor feature is the attention-weighted action feature and it is used to generate new action in the actor neural network. \n\nExperiments demonstrated that the proposed attention mechanism between actor and critic network can improve actor-critic algorithms, DDPG, TD3, and SAC.", "summary_of_the_review": "The intuition and meaning of the attention mechanism need more explanations. The advantage of the proposed method seems to rely on an accurate critic network and this advantage may become a disadvantage when the critic is not good enough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript proposes a meta-attention based mechanism for improving off-policy actor-critic algorithms. Instead of introducing attention into networks at the level of pixels or multiple sources of information, this work focuses on using attention between features from the actor network (which become queries and values) and features from the critic network (which act as keys). Attention produces new features that are given as input to the action net, enabling it to potentially improve it's action selection. The attention is trained using a meta-learning objective that encourages outputting features that help other parts of the architecture to learn.
Reviewer note that there are some positive features of the paper. It is relatively well written and the figures are useful for spelling out the approach. In addition, so felt that the basic idea is an interesting one. However, there was general agreement that the manuscript is not ready for publication. 
Most reviewers noted that, the newly proposed architecture and learning rules were not well motivated by the manuscript. Why was this particular approach pursued? Is there any better theoretical justification than can be offered? These questions on their own are not problematic. However, the empirical work does not robustly demonstrate that the algorithm yields a clear performance gain over baseline actor-critic methods in the literature. Most of the tasks are relatively simple and those gains that are observed are marginal. This is especially difficult for the manuscript given the increased compute and complexity of implementation required by the method. Finally, several reviewers were concerned about the presentation of some of the technical aspects of the works, potentially making it difficult to replicate important aspects of the work. In sum, the manuscript is not ready for publication.
Decision: Reject