Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning
Authors: Kibeom Kim
Min Whoo Lee
Yoonsung Kim
JeHwan Ryu
Minsu Lee
Byoung-Tak Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: goal-aware
attention
reinforcement learning
multi-target environment
visual navigation
manipulation

Abstract: Learning in a multi-target environment without prior knowledge about the targets requires a large amount of samples and makes generalization difficult. To solve this problem, it is important to be able to discriminate targets through semantic understanding. In this paper, we propose goal-aware cross-entropy (GACE) loss, that can be utilized in a self-supervised way using auto-labeled goal states alongside reinforcement learning. Based on the loss, we then devise goal-discriminative attention networks (GDAN) which utilize the goal-relevant information to focus on the given instruction. We evaluate the proposed methods on visual navigation and robot arm manipulation tasks with multi-target environments and show that GDAN outperforms the state-of-the-art methods in terms of task success ratio, sample efficiency, and generalization. Additionally, qualitative analyses demonstrate that our proposed method can help the agent become aware of and focus on the given instruction clearly, promoting goal-directed behavior.

Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning
Kibeom Kim 1.2, Min Whoo Lee', Yoonsung Kim Je-Hwan Ryu', Minsu Lee Byoung-Tak Zhang Seouu National University. Surromind, *AIIS [kbkim, mwlee, yskim, jhryu, mslee, btzhang) Abi. snu.ac ac.kr
Abstract
Learning in multi-target environment without prior knowledge about the targets requires a large amount of samples and makes generalization difficult. To solve this problem, it is important to be able to discriminate targets through semantic understanding. In this paper, we propose goal-aware cross-entropy (GACE) loss, that can be utilized in a self-supervised way using auto-labeled goal states alongside reinforcement learning. Based on the loss, we then devise diariiiscieeeettive attention networks (GDAN) which utilize the goal-relevant information to focus on the given instruction. We evaluate the proposed methods on visual navigation and robot arm manipulation tasks with multi-target environments and show that GDAN outperforms the statc-of-the-ar methods in terms of task success ratio, sample efficiency, and generalization. Additionally, qualitative analyses demonstrate that our proposed method can help the agent become aware and focus on the given instruction clearly, promoting goal-directed behavior.
Introduction
Reinforcement learning (RL) has been expanding to various fields including robotics, to solve increasingly complex problems. For instance, RL has been gradually mastering skills such robot arm/hand manipulation on an object 2 48 36, 22 and navigation to a target destination [18, 39). However, benefit humans like the R2-D2 robot in the Star Wars, RL must extend to realistic settings that require interaction with multiple objects or destinations, which is still challenging for RL.
For this reason, it is important for multi-target tasks to be considered. We use the term multi-target tasks to refer to tasks that require the agent to interact with variable goals. In multi-target task, targets are possible goal candidates, which may be objects key entities that play decisive role in determining the success or failure of the task execution. The goals may be selected among the targets by the current multi-target task, specified with cup or specific objecr)" or "Go to the (kitchen, livingroom or specific destination)" The states in which the agent reaches the goal are called goal states.
instruction such "Bring (spoon,
Reinforcement learning allows learning multi-target or instruction-basee tasks an end-to-end manner. Latest reinforcement learning studies on these tasks mainly focus on prior knowledge about targets 227 30 (). Other studies focus on learning representation on the environment [46) or about 1
the targets only implicitly [44]. These methods lead to insufficient understanding of the goal, and sample-ineffiiiency and generalization problems arise.
For this matter, we propose a Goal-Aware Cross-Entropp (GACE) loss and Soaal-Disrimmnative Attention Networks (GDAN for multi-target tasks in reinforcement learning. These methods, unlike
" "Corresponding authors. !Al Institute, Scoul National University
"Code available at
GDAN
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Emhalling
get
Galee-amention
St
(a,:55 !) V)
Feature Extractor
ActorCrttk
LSTM
(()
Ke
Vt.
Goal Storage
gonal
Gool-lierrimintter
poall
GACE
W
V(v.
a()
Ptc
Sgoot,
+
Φ
ut
success time d:
WopfE
W
:Operator :Linear
Store (Sg, embed(Armar) Storage
Bonous Armor
Policy inference
Interence
qi
(a) Architecture from Goal-Aware Cross- Entropy and GDAN
(b) Attention in ActorCritic
Figure I: Overview of the proposed architecture. (a) A goal-aware visual representation learning method by LGACE from goal-aware cross-entropy loss. The loss updates the feature extractor and the goal-discriminator (b) In GDAN, the ActorCritic utilizes the information from the goal-diseriminator
as the query for goal-directed actions. See the details in Section 3.3 and 3.4
previous studies without prior knowledge, allow semantic understanding of goals including their appearances and other characteristics. The agent automatically labels and collects goal states data through trial-and-error in self-supervised manner. Based on these self-collected data, we use GACE loss as an auxiliary loss train a goal-discring -isccriminator that learns goal representations. Lastly, the GDAN extracts the information from the coal-discriminaoor as goal-relevant query, with which an attention performed to infer goal-dire direediee actions.
We additionally propose visual navigation and robot arm manipulation tasks as benchmarks for multi-target task experiments. These tasks involve targets which are the multiple types of objects randomly placed within the environments. In the benchmarks, we make these tasks visually complex using randomized background textures, allowing
evaluate generalization.
In summary, the contributions of this paper are as follows:
We propose Gwar-Aware Cross-Entropy loss for learning auto-labeled goal states in a self- supervised manner, solving instruction-base multi-target tasks in reinforcement learning.
We additionally propose Diseriiicriiiittvve Attention Networks that use the goal-relevant query from the goal-discriminttor to focus exclusively on important goal-related information.
Our method achieves significantly better performances in terms the success ratio, sample- efficiency, and generalization visual navigation and robot arm manipulation multi-target tasks. In particular, compared with the baseline methods, our method excels in Sample Efficiency Improvement metric by 17 times in visual navigation task and by 4.8 times in manipulation task.
We present instruction-based benchmarks for goal-based multi-target environments, to interact with multiple targets for realistic settings. These benchmarks are made publicly available, along with the implementation of our proposed method.
Related work
Multi-target tasks in reinforcement learning There has been sparsity of studies that aim to solve multi-target tasks with reinforcement learning. Wu et al. [44) proposes instruction- based indoor environments, as well as attention networks for multi-target learning to respond to the given instruction. Deisenroth Fox []] uses a model-based policy search method for multiple targets, instead of learning goal-relevant information. The aforementioned methods are for learning dynamics model or learning targets indirectly for multiple target tasks. our method, on the other hand, the policy directly learns to distinguish goals without the dynamics model.

---

The terms targets and goals have been used in diverse manners in the existing literature. There are subgoal generation methods that generate intermediate goals such as imagined goal 14 or random goals [3]] to help agents solve the tasks. Multi-goal RL 47 35 13 10 aims to deal with multiple tasks, learning to reach different goal states for each task. Scene-driven visual navigation tasks 44 12 30] specify the goal by an image. However, these tasks require the agent only to search for visually similar locations or objects, rather than gaining semantic understanding of goals and promoting generalization. Instruction-bart instructions. These tasks have a different purpose from our tasks, where we focus on having appropriate interactions with multiple targets depending on the instruction.
tasks [ 40] focus on learning to follow detailed
Representation learning in reinforcement learning To improve performance in RL, several recent works have used variety of representation learning methods. Among these methods, many make
use of auxiliary tasks taught via unsupervised learning techniques such as VAE (24 19 for learning
dynamics [6 45 38
and contrastive learning method [ for learning representations [26] of given environment. Nair et al. [32 uses goal states, specified as the last states of the trajectories. to learn the difference between future state and goal state for redistributing model error in model- based reinforcement learning. The aforementioned methods are for learning representations of the environment rather than for learning those of key states for solving the task. In contrast, our method directly learns goal-relevant state representations.
Jaderberg et al. 21 proposes various auxiliary tasks for representation learning in RL. One such method is reward prediction, which learns the environment by predicting the reward of the future state from three classes: possiiiee, O, negative]. This method focuses on myopic reward prediction, and not on ultimately learning the goal state to solve the task. Consequently, it is not suitable method to apply to multi-targe environments, where the goal can be selected among diverse range of objects.
Attention methods in reinforcement learning The attention method was initially introduced for natural language processing [3] but is now being actively studied in various fields such as computer vision (33. There are also attention-based methods for reinforcement learning which use inputs from different parts of the state [4 9.. Other methods include the use of encoding information of sequential frames up to the last step 229] well the application of self-attention [6]. Unlike these methods, our approach explicitly extracts and focuses solely on goal-related information for solving a task.
3. Method
3.1 Preliminary
Reinforcement learning (RL) from Sutton and Barto 41) aims maximize cumulative rewards by trial-and-error in Markov Decision Process (MDP). An MDP is defined by tuple (S, R, P, 7). where 8 is the P: Sx x A 1x S + IR is the transition probability distribution, and 7 E (0,1) is the discount factor. At each time step f, the agent observes state S; S. selects an action o; AA according to its policy S - A. and receives a reward r2 and next state St+1. In finite-horizon MDPs, return R2 k==0" > is accumulated discounted rewards, where T is the maximum episode length. State value function V(s) E [[R s = s] the expected return from state s.
of states, A is the set of actions, R S A R is the reward function,
Instruction-basee multi-target reinforcement learning Universal value function approximators 37 estimate state value function as V (s, æ). for jointly learning the task from embedded state and goal information x. This is relevant to the notions of multi-target RL that we define as below.
Multi-targer environments contain N targets, or goal candidates T {21. 22, .... 2 These environments provide an instruction Ia that specifies which target the agent must interact with, or the goal 2 E T. The instruction is given randomly every episode, in the form such as "Get the Bonus" or "Reach the Green Box" as in our benchmark tasks, in which cases the goals are Bonus and Green Box respectively. Hence, the state value function in such task is V(s, [2) E[R, 5, = (s. =)]. where 8, is state conditioned on the goal z. The policy и is also conditioned on the instruction T as (ace Ie) where a, is action at time given state s, and 1a
3

---

3.2 Auto-labeled goal states for self-supervised learning
Prior to describing the main method in Sec. 3.3 and 3.4, this section explains the automatic collection of goal data for self- learning. Suppose that the instruction I specifies the target as the goal and the episode ends when the goal is reached at time step t'. We refer to the state Se as goal state, which we assume is highly correlated with the reward or the rewarding state. Throughout the training. the reached goal label and the corresponding goal state sy are automatically collected as tuple (Str, soe_hot(=)) called storage data. Rather than being manually provided the goal information, the agent actively gathers the data needed to learn the goals, relying only on the instruction 1s given by the environment. This allows the agent to learn in an end-to-end, self-supervised manner. The states for the failed episodes are also stored as a negative case with low probability ey. We clarify that the storage data does not serve as the prior during the learning and is instead used for training alongside the multi-target reinforcement learning task.
3.3 Goal-diseriminator by goal-aware cross-entropy
Our proposed method adds an auxiliary task to the main reinforcement learning task as shown in Figure Ta A general reinforcement learning model consists of a feature extractor (-)) which converts state to encoding e,, and an ActorCritic f(-) that outputs policy and value V:
e, = (s.)
(1)
π (aylst, I), V(s; f(e,
(2)
where I is the instruction. The details of the ActorCritic f(.) and the resulting loss Lat depend on the specific reinforcement learning algorithm that chosen, In our visual navigation experiments, we use asynchronous ad advantage actor-critic (A3C) (28) as the main algorithm, where the loss L is defined as the following
Cp V log π (a. st, (R, V(s, +)) BVH
(3)
C, (R. - V(,, 1))²
(4)
LAL = LASC = Lp 0.5
(5)
where Cp and C. respectively denote policy and value loss, Rs denotes the sum of decayed rewards from time steps to T, and H and B3 denote the entropy term and its coefficient respectively. See Appendix [ for algorithm details.
The vanilla base algorithm is inefficient and indirect at gaining an understanding of goals. Suppose that we have set of goal states S₂ C 8 corresponding to each goal z. Intuitively, given an instruction Ia that specifies z, the agent must learn the value function V (s, I) that discriminates the goals, such that V(s ((5go (+)=
V (isoout 1+) for "goat Sz and s gool E Sy. Vz j..
To achieve this effect, we propose Goal-Aware Cross-Entropy (GACE) loss as our contribution, which trains the dial-discriminator that facilitates semantic understanding of goals alongside the policy in Figure Ta The inference process is as follows. First, the storage data collected Sec. 33.2 are sampled as batch of ggoat,''s 's (with i as data index within the batch) and fed into the feature extractor (-) for state encoding in Eq. Next, the encoded input data exguat, is fed into the goal-discriminator d(-). , a multi-layer perceptron (MLP), in Eq. 7 This yields a prediction ggoal, a vector that contains probability distribution over possible goals that $gant,s belongs to.
gogut. = (8gontt,))
(6)
ggoal, = d(c d((Cajoat))
(7)
From the output i we calculate the Goal-Aware Cross- Pross-Entropy loss LGACE as Eq. where M is the batch size, and z; is the automatic label corresponding to state goal,i.
LGACE = Σ one hot_hot(2;) log([gool,s) ==0
(8)
We complete the training procedure by optimizing the overall loss Listal as the weighted sum of the two losses in Eq. g We focus on improving the policy for performing the main task and assign weight 77 to LGACE for performing goal-aware representation learning for the feature extractor o (-).

---

We note that the coal-discriminator d(-) is updated according to LGACE only during training, not during inference, excluding the ActorCritic.
Liotal RL + 1LGACE
The explained procedure forms& visual representation learning method, where the GACE loss makes the coal-discrim become goal-aware without external supervision. Such goal-awarenee is udvantageous for sample-efficienee multi-target environments, as well as generalization, as makes the agent robust in noisy environment. Such effects are demonstrated in our quantitative results and qualitative analyses.
3.4 Goal-diseriminative attention networks
The goal-discriminator discussed so far can influence the policy inference. However, in order to effectively utilize the discriminator to enhance the performance and efficiency of the agent, we propose Goal Dina-iisrimiiteive Attention Networks (GDAN). Overall, the goal-aware attention described in Figure [b involves innivolless a goal-relevant query q from within the -isrriiirrm d(-). ((-), and the key kn and value Ut from encoded state in the ActorCritic f()).
During the inference, the state encoding vector e₁ is passed through the first linear layer of the goal- discriminator, which yields the query vector qi that implicitly represents the goal-relevant information. Meanwhile, the vector e, is inferenced through gated-attention 44 with instruction I for grounding.
The gated-attention vector is passed through LSTM (Long Short-Term Memory) (20] and splits in half into key k and value Uz. Suppose the vectors qi ki, and v, are dly de- and d, dy-dimensional, respectively. The query and key are linearly projected to dy-diimnnioona space using learnable parameters Wg q of dimensions dy dg and Wk of dimensions de x dg. respectively. The activation function ø, which is tanh in our case, is applied to the resulting vectors to yield goal-aware attention vector ui in Eq. I0 The attention vector uf contains the goal-relevant information in st. Lastly, Hadamard product is performed between uf and in Eq. I1 yielding the iitentiio-dependden state representation vector h4 used for caleulating the policy and value. The vector h, concatenated with gated-attention vector from a state encoding vector e₁ in ActorCritic f(-).
" = o ((Wg que + Wkk))
(10)
h; O ui
(11)
We underscore two points about the proposed networks. First, while the goal dise-iiicriiieeeiee feature extractor a--) trained by GACE loss may be sufficient, by constructing attention networks, the implicit goal-relevant information from the goal-diseriminttor directly affects the ActorCritic f(-) that determines the policy and value. Second, the query vectors from a part of the goal-diseriminator, which allows the ActorCritic to actively query the input for goal-relevant information rather than having to filter out large amounts of unnecessary information. These two design choices enable the agent to selectively allocate attention for goal-directed actions (Figure 5c). making full use of the GACE loss method. Details of our architecture are covered in Appendix B
4 Experiments
The experiments are conducted to evaluate the success ratio, sample-efficiency, and generalization of our method, as well as compare our work competitive baseline methods in multi-target environments. We develop and conduct experiments on (1) visual navigation tasks based on ViZDoom 23 18), and (2) robot arm manipulation tasks based on MuJoCo 42 These multi- target tasks involve N targets which are placed at positions P2.
Experimental setup
Pv. and provide the index of one target as the goal 2 E 21, ................................ N). At time step 1, agent receives state 8, normalized to range [0,1]. The agent receives a positive reward l'success if it gets sufficiently close to the goal (i.e. ||x, - Pall S <) where x, the agent position at time On the other hand, the agent penalized by Tnongoal if it reaches non-goal object, l'timoout if it times out at t 2 T. Lastly, the agent receives a time step penalty Tstep. which is empirically found to accelerate training. Reaching either a goal or non-goal object terminates the episode. All experiments are repeated five times.
Details of sample efficiency metrics We introduce Sample Requirement Ratio (SRR) and Sample Efficiency Improvement (SEI) metrics measure the sample efficiency of each algorithm. For the
5

---

(a) Visual navigation task ViZDoom
(b) Robot manipulation task
Figure 2: Two kinds of multi-target tasks in our experiments. The object positions and background are randomly shuffled in (a) egocentric visual navigation tasks and (b) robot arm manipulation tasks.
given task, we select a reference algorithm A and its reference success ratio X %. Suppose this algorithm reaches the success ratio X within n updates. We measure the SRR metric of another algorithm B as SRRg- = nB/nAA where B reaches % within ng updates. Also, SEI metric of B is measured as SEIB (nA ng)/ng. Lower SRR, higher SEI indicate higher sample efficiency.
4.1 Visual navigation task with discrete action
We conduct experiments on egocentric 3D navigation tasks based on ViZDoom (Figure 2ap. The RGB-D state provided as an input to the agent, and three different actions (MoveForward, TurnLeft, TurnRight) are allowed. One item from each of the four different classes of objects each class (e.g. Bonus) containing two different items (e.g. HealthBonus and ArmorBonus) is placed within the map, with one class randomly selected as the goal by the instruction for each episode like "Get the Bonus". In every episode, the agent is initially positioned the center of the map. The rewards are set as ssccesss 10, nonggoo 11, lttmmouu 0011, Tssep" 0.011 Full details of the environment are provided in the Appendix C
Details of each task To evaluate the performance of our method on tasks of varying difficulties, we set up four configurations. The V1 setting consists of closed fixed rectangular room with walls the map boundaries, and object positions are randomized across the room. V2 is identical to VI, except that the textures for the background, such as ceiling, walls, and floor, are randomly sampled from a pool of textures every episode. 40 textures are used for the seen environment, while 10 are used for the unseen environment which is not used for training to evaluate generalization. Hence the tasks allow 40 and 10 different state variations in seen and unseen environments respectively. V3 more complicated, with larger map size, shuffled object positions, and additional walls within the map boundaries to form a maze-like environment. "Shuffled positions" indicates that the object positions are chosen as random permutation of a predetermined set of positions. Lastly, V4 is equivalent to V3 with the addition of randomized background textures for the seen and unseen environments. V2 and V4 allow to evaluate the agent's 's generalization to visually complex, unseen environments. V1 and V3 can be regarded as the upper bound of performance on V2 and V4 respectively.
Baselines for comparison As the base algorithm. we use A3C, an on-policy RL algorithm, with the
model architecture that uses gated- attenttoen S 33 with LSTM. This baseline is proposed in A4
for multi-target learning on navigation. Other competitive general RL methods are added onto the base multi-target learning algorithm, due to the rarity of appropriate multi-target RL algorithms for comparison. A3C+VAE [6. 38 learns VAE features for all time steps, unlike our methods which learn only for goal states. A3C+RAD applies augmentation the random crop method, known to be the most effective in [25] to input states in order to improve sample efficiency and generalization. A3C+GACE and A3C+GACE&GDAN are methods applied to the A3C as covered in Sec. 3.3 and 3.4
Results Performance is measured as the success ratio of the agent across 500 episodes in all tasks. As shown in Figure 4 most baselines successfully learn the V1 task, albeit at different ratios. In V4,
The cited paper also introduces multi-target tasks, but those unavailable due to license issuee.

---

0.0
0.8
0.8
S.0cem Datts
0.66
S.c00s Ratto
0.6
0.6
Abc
+VAC
+840
FBAC Blour3
520K
304
1.5M
3M
3M
201
3M
116
2M
(a) V1
(b) V2 Seen
(c) V2 Unseen
0.8
0.5
Guscoea Satt
0.4
Succeg Rato
0.4
0.4
0.1
0.1
100%
1M
3.5M
254
2.5M
314
154
354
2M
Kumber Upcates
Number Updances
Number Liddstes
(d) V3
(c) V4 Seen
(f) V4 Unseen
Figure 3: Visual navigation learning curves. The solid lines show the average success ratio over the repeated experiments, and the shades indicate bounds given mean + standard deviation of success ratio. Orange and red lines are our methods, GACE loss and GACE&GDAN, respectively.
A3C+VAE (green) initially shows the steepest curve, probably because it reaches the object located near the initial agent position before the learning commences properly. A3C+RAD (blue) shows the steepest learning curve VI, but fails learn in visually complex environments such V2 and V4. In V3, A3C+GACE (orange) and A3C+GACE&GDAN (red) achieve 52.6% and 78.2% respectively. We attribute such discrepancy A3C+GACE&GDAN more direct and efficient usage of goal-related information from the goal-diseriminator Especially, A3C+ GACE&GDAN achieves as high as 86.6% and 54.9% in V2 seen and V4 seen respectively. Furthermore, in V2 unseen and V4 unseen tasks, the agents trained with our methods generalize well to unseen environments. Thus, our methods achieve the state-of-the of-the-art sample-efficiency and performance all tasks. The full details are covered the Appendix B
The SRR and SEI measurements of the models shown in Table For measuring sample efficiency, we select A3C as the reference algorithm and 56.55% as the reference success ratio in VI task. This is the highest performance of A3C within 2M updates. Our methods show the highest sample efficiency compared to the baselines.
4.2 Robot manipulation task with continuous action
To evaluate our method in the continuous-action domain, conduct experiments the URS robot arm manipulation tasks with degrees of freedom, based on MuloCo (Figure 2b). The environment comprises a robot arm and three five objects of different colors (red, green, blue, etc). The state is an RGB image provided from the fixed third-person perspective camera view that captures both the robot and the objects Figure 2b The rewards are Tsaccess Tnongool 00.3, l'timoout -..1,
r Tstep = -0011.
Details of each task Our method is evaluated on three different vision-based tasks. In the R1 task, the object positions are shuffled among three preset positions on grey background, and one of the three objects is randomly specified as the goal by the instruction like "Reach the Green Box". The robot arm must reach the goal object within the time limit T steps while avoiding other objects. R2 task complicates the RI task by replacing the grey background with random checkered patterns. The agent's performance is measured on the seen environment, in which the agent is trained, as well as the unseen environment. The seen and unseen environments each uses different set of colors of checkered background. R3 task is a more complex variant of RI setting, where five target classes are available, three of which are randomly sampled by the environment and randomly positioned among the three preset locations in RI. The full details of these tasks are outlined in the Appendix C
7

---

Table Success ratio (SR) and sample efficiency metrics visual navigation task V1. SRR (lower the better) and SEI (higher the better) are measured with A3C performance reference. "Number of Updates" indicates the number of updates required to reach the reference performance.
SR of V1 (%)
Number of Updates
Algorithm
SRR (%)
SEI (%)
A3C
2M
100
+VAE
810,086
40.50
146.89
+RAD
56.55
703,574
35.18
184.26
GACE (ours)
163,602
8.18
1122.48
+GACE & GDAN (ours)
110,930
5.55
1702.94
Table 2: Success ratio (SR) in robot arm manipulation tasks.
SR
SR R2
SR of R2
SR of
Algorithm
R1 (%)
Seen (%)
Unseen (%)
R3 (%)
SAC
63.1 6.9
60.5 + 5.7
53.4 6.9
61.7 + 5.4
+AE
67.2 - 5,0
72.8 + 5.9
59.4 + 5.5
62.3 5.1
+CURL
67.9 7.3
74.5 + 9.2
36.6
64.7 + 4.0
GACE
84.7 10.0
75.0 + 8.9
9.0
79.3 + 8.9
GACEE&DAN
89.3
78.2 8.7
73.3
79.6 + 8.4
Baselines for comparison As baseline method, use deterministic variant of Soft Actor-Critic (Pixe1-SAC [7), an off-policy algorithm commonly used in robotic control tasks. As baseline, we 45 which uses 3-VAE [99 for encoding and decoding states. Another baseline is SAC+CURL which performs data augmentation (cropping) and contrastive learning between the same images. These baseline methods are all competitive methods in the continuous action domain. We use SAC as a base algorithm to apply our methods SAC+GACE and SAC+GACE&GDAM, Note that the ActorCritic is as depicted in Figure Tb, without the L.TTM.
evaluate SAC+ AE
Results The results for robot manipulation tasks are shown in Table and Figure Appendix B). Performance is measured as the success ratio of the agent for 100 episodes. SAC shows higher difficulty in learning R2 tasks, which are more visually complicated than RI task. Unexpectedly, SAC+AE and SAC+CURL has improved performance in R2 seen. We speculate that the two algorithms that perform representation learning are more suitable for the task with diversity. Especially, SAC+CURL consistently achieves competitive performances in all environments except for R2 unseen task. However, it does not seem capable of learning with complex unseen backgrounds, showing huge deterioration in performance for R2 unseen. SAC+GACE consistently shows strong performance in all environments, as well as high generalization capability for R2 unseen task. Finally, SAC+GACE&GDAN attains state-of-the-ar performance in all environments including the unseen task for generalization. In particular, we observe the smallest performance discrepancy between R2 seen and unseen tasks. This shows that it can learn robustly in complex background acting noise. Furthermore. methods show the highest performance in R3 task, adapting well a more diverse set possible target objects. This supports that the GACE mechanism addresses the scalability issue well.
The SRR and SEI measurements of these methods are shown in Table 3 We select SAC as the reference algorithm and 63.1% the reference success ratio in R1 task. SAC+GACE shows the catte-of-hheaat sample efficiency compared to all baseline models, meaning that it learns faster than SAC+GACE&GDAN for relatively easy tasks.
Comparison with single-target task We conduct an ablation study, where effectively single-targe policies are trained on variant of RI task with the instruction fixed to a single target as goal. We train these policies for 120K updates, such that the total number of updates (360K) is roughly equal to the number of updates for training the multi-target baseline models. The average performance of these policies is (63.1 6.9%). Continuing the training, the single-target policies converge to 87.5+2.44% success ratio in 333,688 updates, serving competitive baseline. However, training individual single-target
which is noticeably lower than the performance of multi-target SAC agent

---

Table 3: Sample efficiency metrics for R1 task. SR indicates the reference performance in R1 task.
Number of Updates
Algorithm
SR (%)
SRR (%)
SEI (%)
SAC
314,797
100
+AE
230,339
73.17
36.67
+CURL
63.1
142,480
45.26
120.94
GACE (ours)
53,774
17.08
485.41
GACE&GDAN (ours)
63,140
20.06
398.57
1.0
GACE
0.9
0.6
GACE 20K
10K
Accuracy
0.8
Success Raboo
0.2
GACE
GACE
GACE 10K
50K
100K
150K
200K
100K 150K
200K
Number of Undates
Number Updates
(a) Goal discriminator accuracy
(b) Learning curve in task
Figure 4: Additional experiments analyze the effectiveness of GACE loss. The accuracy of the goal-discriminator (a) and learning curve of the agent (b) are measured with the goal-discriminator weights unfrozen (red), frozen 10K (purple) and 20K (blue) updates.
policies for mulli-targee tasks poses scalability problem. It not only requires memory for weights that is directly proportional to the number of targets, but also uses large amount of samples to learn the targets separately. In contrast, the multi-target agent efficiently learn joint representation, improving memory and sample efficiency as well as allowing generalizable learning of many targets.
4.3 Analyses
Effectiveness goal-aware cross-entropy loss Additional experiments are conducted in the V1 task to investigate how method influences the agent's learning. To vary the extent to which the GACE loss may affect learning, the goal-discriminator weights are frozen after 10K and 20K updates such that the GACE loss LGACE does not contribute learning afterwards. We note that in Figure 4aj although the GACE loss (frozen weights) does not further contribute to learning, the discriminator accuracy improves only by updating the policy. This indicates that throughout the training, the agent gradually develops feature extractor a(.) that can discriminate targets. Our method makes possible for the feature extractor directly learn discriminate.
In addition, the learning curves in Figure i corroborte that such development of the feature extractor accelerated by the GACE loss. Even when the agent is trained with the GACE only temporarily (as with GACE 10K and 20K), the learning curve is steeper than that with vanilla A3C. Furthermore, the unfrozen GACE (red line) shows steeper learning curve than GACE 20K, which is again steeper than GACE 10K. Consequently, it can be seen that representation learning of feature extractor updated by GACE loss has positive influence policy performance than learning solely with policy updates.
Simple ActorCritic f(-) We perform an ablation study about attention model that simply con- catenates e from feature extractor (-)) with qi from ActorCritic f(.) in Figure Ib to verify the effectiveness of the attention method. We conduct the experiment in R1 task. As result, success ratio 81.5 10.0 % obtained, which is lower than that of GACE loss and GACE&GDAN. We corresponds to the goal such goal states are quite sparse the
speculate that since not every additional information of the goal-diseriminator acts as noise, hindering the efficient learning of the policy. This supports that, in contrast to such naive of goal-diseriminator features, our proposed attention method can effectively process goal-related information and is robust against noise.

---

Analysis using saliency maps To ascer- tain that an agent trained with GACE and GACE&GIAN indeed becomes goal-aware, we use saliency maps [T5] to visualize the operation of three agents within the V2 un- seen task, as shown in Figure [ The blue shades indicate the regions that the agent significantly attends to for policy inference, and the red shades indicate those regions for the value estimation. The overlapping blue and red regions appear purple. The three agents are trained with A3C, GACE, and GACE&GIAAN, respectively, for 4M up- dates.
(a) A3C Results
(b) GACE Results
Figure 5a shows the operation of the A3C agent. It shows overly high sensitivity to edges in the background, approaching the walls rather than searching for the goal. Also, in the rightmost image in (a), although it does attend to the goal object located on the right side, TurnLeft action is performed. This suggests that the agent cannot discern be- tween objects because it lacks understand- ing of targets. Figure 55 corresponds to the GACE agent, showing high sensitivity to all targets and intermittent edges. As its attention to targets demonstrates, the ability to distinguish the goal is far superior to that of the A3C agent. Finally, Figure 5c| shows that the GACE&GDAN agent exhibits high sensitivity responding to irrelevant edges. In addition. upon noticing the goal, the agent allocates attention only to the goal, rather than unnecessarily focusing on the non-goals. These results support that our method indeed promotes goal-directe behavior that is visually explainable.
(c) GACE & GDAN Results
Figure 5: Visualization of saliency maps in V2 unseen (goal and non-goal in yellow and green eircles respec- tively). (a) The agent is overly sensitive to edges in the background. (b) All goals and non-goals are detected successfully. (c) The agent shows sensitive reactions only the goal.
all goals while hardly
5 Conclusion
We propose GACE loss and GDAN for learning goal states in a self-supervised manner using reward signal and instruction, promoting a goal-focused behavior. Our methods achieve tfat--of-the-at sample-efficiens and generalization in two multi-target environments compared to previous methods that learn all states. We believe that the disparity between the performance improvements in the two benchmark suites is due to the degree of similarity between the goal states and the others.
The limitation of our method is that may not be well-applied in tasks where instructions do not specify a target (e.g. "Walk forward as far as possible") or tasks with little correlation between states and rewards based on success signal. Also, our methods assume that the total number of target object classes is known beforchand. Our method can be abused depending on the goal specified by user. Nonetheless, this paper brings to light the possible methods of prioritizing data for efficient training. especially in multi-target environment.
Acknowledgments and Disclosure of Funding
We would like to thank Dong-Sig Han, Injune Hwang, Christina Baek for their helpful comments and discussion.
This work was partly supported by the IITP (011500.0331--5...taaLaaaa/ N0162-HumanCare/105, Mind/10%%
2017-0- -1170-000772-VT//100 2018-0-00662-RM1/1993 2019-0-01371- A1168-AAHHH/100,, 10%, 100006720-ILLAS/1055 grant, and the NRF of Korea 0001RIA22101099/// grant funded by
(SNU)/0%) grants, the KIAT
the Korean government, and the BMRR Center (UDI90018ID/10%) funded by the DAPA and ADD.
10

---

References
[1] Anderson, P., Wu, Q. Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid, I., Gould, S., and van den Hengel, A. (2018). Vision-and-langee navigation: Interpreting visually-groume navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683
[2] Andrychowiez, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2017). Hindsight experience replay. arXiv preprint arXiv:/707.0144 049..
[3] Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:/409.000
[4] Barati, E. and Chen, X. (2019). An actor-eritic-atteniion mechanism for deep reinforcement learning in multi-view environments. In IJCAI, pages 2002-2008.
[5] Chaplot, D. S., Sathyendra, K. M., Pasumarthi, R. K., Rajagopal, D., and Salakhutdinov. R. (2018). Gated-attention architectures for task-oriented language grounding. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
[6] Chen, H., Liu, Y., Zhou, Z., and Zhang, M. (2020a). A2c: Attention-augmentedd contrastive learning for state representation extraction. Applied Sciences, 10(17):5902.
[7] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020b), A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709
[8] Cheng, R., Agarwal, A., and Fragkiadaki, K. (2018). Reinforcement learning of active vision for manipulating objects under occlusions. In Conference on Robot Learning, pages 422-431. PMLR.
[9] Choi, J., Lee, B.-J., and Zhang, B.-T. (2017). Multi-focus attention network for efficient deep reinforcement learning. arXiv preprint arXiv:1712.04603.
[10] Colas, C., Fournier, P., Chetouani, M., Sigaud, O., and Oudeyer, P.-Y. (2019). Curious: intrinsically motivated modular multi-goal reinforcement learning. In International conference on machine learning, pages 1331-1340. PMLR.
[11] Deisenroth, M. and Fox, D. (2011). Multiple-targer reinforcement learning with a single policy.
[12] Devo, A., Mezzetti, G., Costante, G., Fravolini, M. L., and Valigi, P. (2020). Towards general- ization in target-driven visual navigation by using deep reinforcement learning. IEEE Transactions on Robotics, 36(5):1546-1566
[13] Dhiman, V., Banerjee, S., Siskind, M., and Corso, (2018). Learning goal-conditioned value functions with one-step path rewards rather than goal-rewards. ICLR.
[14] Florensa, C., Held, D., Geng, X., and Abbeel, P. (2018). Automatic goal generation for reinforcement learning agents. In International conference on machine learning, pages 1515-1528. PMLR.
[15] Greydanus, S., Koul, A., Dodge, and Fern, A. (2018). Visualizing and understanding atari agents. In Proceedings of the 35th International Conference on Machine Learning, pages 1792-1801.
[16] Ha, D. and Schmidhuber, J. (2018). World models. arXiv preprint arXiv:1803.10122.
[17] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pages 1861-1870.
[18] Harries, I.,, Lee, S., Rzepecki, J., Hofmann, K., and Devlin, S. (2019). Mazeexplorer: A customisable 3d benchmark for assessing generalisation in reinforcement learning. Proceedings of the IEEE Conference Games.

---

[19] Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerehner, A. (2017). beta-vae: Learning basic visual concepts with constrained variational framework. ICLR.
[20] Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 18):1735-7780.
[21] Jaderberg, M., Mnih, V., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu. K. (2017). Reinforcement learning with unsupervised auxiliary tasks. International Conference in Learning Representations.
[22] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Vanhoucke, V., and Levine, S. (2018). Scalable deep reinforcement learning for vision-basee robotic manipulation. In Billard, A., Dragan, A., Peters, J., and Morimoto, J., editors, Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pages 651 657- PMLR.
[23] Kempka, M.. Wydmuch, M., Runc, G., Toczek, J., and Jaškowski, W. (2016). Vizdoom: A doom-based research platform for visual reinforcement learning. In IEEE Conference on Computational Intelligence and Games (CIG), pages 1-8.
(24) Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv: 14..
[25] Laskin, M., Lee, K., Stooke, A., Pinto, L.,, Abbeel, P., and Srinivas, A. (2020a). Reinforcement learning with augmented data. arXiv preprint 2004 /4990.
[26] Laskin, M., Srinivas, A., and Abbeel, P. (2020b). Curl: Contrastive unsupervised representa- tions for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning.
[27] Liu, L.,, Dugas, D., Cesari, G., Siegwart, R., and Dubé, R. (2020). Robot navigation in crowded environments using deep reinforcement learning. pages 5671-5677.
[28] Mnih, V.. Badia, A. P., Mirza, M.. Graves, A., Lillicrap. T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In Inter- national conference on machine learning, pages 1928-1937.
[29] Mott, A., Zoran, D., Chrzanowski, M., Wierstra, D., and Jimenez Rezende, D. (2019). Towards interpretable reinforcement learning using attention augmented agents. Advances in Neural Information Processing Systems, 32:1 12500-19123599
[30] Mousavian, A., Toshev, A., Fišer, M., Košecká, J., Wahid, A., and Davidson, J. (2019). Visual representations for semantic target driven navigation. In 2019 International Conference on Robotics and Automation (ICRA), pages 8846-8851 8852. IEFF.
[31] Nair, A. V., Pong, V., Dalal, M., Bahl, S., Lin, S., and Levine, S. (2018). Visual reinforce- ment learning with imagined goals. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi. N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
[32] Nair, S., Savarese, S., and Finn, C. (2020). Goal-aware prediction: Learning to model what matters. In International Conference on Machine Learning, pages 7207-7219.
[33] Narasimhan, K., Barzilay, R., and Jaakkola, T. (2018). Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874.
[34] Pardo, F., Levdik, V., and Kormushev, P. (2020). Scaling all- all-goals updates in reinforcement learning using convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5355-5362.
[35] Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., et al. (2018). Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint 0944.44.
12

---

[36] Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S. (2018). Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania.
[37] Schaul, T., Horgan, D., Gregor, K., and Silver, D. (2015). Universal value function approxima- tors. In International conference on machine learning, pages 1312-1320. PMLR.
[38] Shelhamer, E., Mahmoudieh, P., Argus, M., and Darrell, T. (2016). Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv: arXiv:16/2.07337.
[39] Shen, W. B., Xu, D., Zhu, Y., Guibas, L. J., Fei-Fei, L.,, and Savarese, S. (2019). Situational fusion of visual representation for visual navigation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2881-2890.
[40] Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zeutlemoer, I.., and Fox, D. (2020). Alfred: A benchmark for interpreting grounded instructions for everyday tasks. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages
10740-10749.
[41] Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
[42] Todorov, E.,, Erez, T., and Tassa, Y. (2012). Mujoco: A physies engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033.
[43] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, (2017). Attention is all you need. In Advances neural information processing systems, pages 5998-6008.
[44] Wu, Y., Wu, Y., Gkioxari, G., and Tian, Y. (2018). Building generalizable agents with a realistic and rich 3d environment. arXiv preprint arXiv: 100I. 018801002009
[45] Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J., and Fergus, R. (2019). Improving sam- plc efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.0174;
[46] Ye, X., Lin, Z., Lee, J.-Y., Zhang, J., Zheng, S., and Yang, Y. (2019). Gaple: Generalizable approaching policy learning for robotic object searching in indoor environment. IEEE Robotics and Automation Letters, 4(4):4003-401
[47] Zhao, R., Sun, X., and Tresp, V. (2019). Maximum eetroiy- multi- -oal reinforcement learning, In International Conference on Machine Learning, pages 7553-7562. PMLR.
[48] Zhu, H., Gupta, A., Rajeswaran, A., Levine, S., and Kumar, V. (2019). Dexterous manipula- tion with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3651-3657 IEEE.
[49] Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., and Farhadi, A. (2017). Target-driven visual navigation in indoor scenes using deep reinforcement learning. In /EEE international conference robotics and automation, pages 3357-3364.
13


Reviews:
[{"review_id": "zJgq05Ehxq_", "reviewer": "Reviewer_RRiW", "summary": "The paper addresses problem of sample efficiency and generalization in the multi-target reinforcement learning. The author proposes goal-aware cross-entropy (GACE) loss for auto-labeling goal states. The author also develops goal-discriminative attention network (GDAN) for multi-target reinforcement learning. They evaluate the proposed method on visual navigation robot arm manipulation with multi-target environment.  The proposed method in the paper compares with A3C, SAC.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The sample efficiency and generalization for multi-goal RL are interesting problems to be addressed.\n- The proposed GACE loss and GDAN method are nice contributions for research community.\n- The paper evaluates good range of experiments.\n- The proposed method in the paper achieves strong results against the baselines.\n- The paper is well written.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Could you please explain RS-D2 in line 18.\n- The multi-target setup has also a challenge of target changing or dynamic targets, could you talk about how the method would perform in dynamic targets situations?\n- In section 5, the paper talks about one of the limitation to the proposed method as target ambiguity. Could you please elaborate a little more?\n- I understand that with the pandemic situation it would be difficult to access lab to conduct real-world experiments. If possible could the authors talk about how the methods can be extended to real-world robotics tasks?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "v6TAjyjDx-", "reviewer": "Reviewer_TeEW", "summary": "This paper proposes an auxiliary loss function and a model architecture to facilitate RL agent’s semantic understanding. Specifically, a separate discriminator that partially shares weights with the policy is trained to select the correct goal given the input instruction, while another attention module is trained to better extract the goal-relevant features. Experiments show that the proposed method improves both the sample efficiency and generalization dramatically on instruction-following tasks in visual navigation and robot manipulation.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Originality*: The method proposed by this paper is simple and sound. The related works are adequately cited so the contribution is clear.\n\n*Quality*: I believe the proposed method is technically sound and used appropriately. The work is complete and both major contributions are backed up by clean ablation study. The authors are honest about their weaknesses too (e.g., it may not be well-applied to a task where the goal is not associated with a target). \n\nHowever, I wonder whether it’s true that the target also needs to be known beforehand? For example, one needs to first define a fixed set of targets that potentially can show up in the environment before training the agent. If this is true, then I think authors should add this as another weakness as it’s hard to achieve in an open-world.\n\n\n*Clarity*: The paper is pretty well-written and easy to understand. The analyses in section 4.3 are very informative.\n\n*Significance*: I think the results are cool and improves the state of the art in a demonstrable way. Additionally, other researchers will likely build on this approach to learn better representation for RL agents because it’s easy to implement.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "mP6p87Tl7C4", "reviewer": "Reviewer_PmsL", "summary": "This paper presents a method for training multi-target policies using reinforcement learning. It introduces an auxiliary classification-type loss for a feature encoder on which RL is performed, and additionally, uses an attention-based architecture to compute the policy and value function. In combination, these two elements seek to learn features which can distinguish between different goals in the environment, and then have the RL algorithm directly use these features to compute goal-relevant policy outputs. The authors show that adding these two components to existing RL algorithms improves performance and sample efficiency on simulated visual navigation and robotic manipulation environments compared to the base versions of these algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Taking features from the learned goal-discriminator and using it as an attention input is novel to me and seems quite effective. The overall method is able to demonstrate impressive performance on previously unseen visual navigation settings, likely through attending to only goal-relevant portions of the image observations. \n\nQuality: The method seems to provide significant improvements over the base algorithms in the experiments demonstrated. The qualitative results (Figure 5) are compelling, supporting the authors’ arguments. \n\nOne point which could be investigated a bit more thoroughly: the GACE loss function appears to be analogous to training a classifier on goal images, which is similar to methods such as CURL but assumes that one can identify goal states for multiple tasks, which CURL does not use. If contrastive learning were performed on goal images instead, would this result in better performance than the SAC+CURL baseline presented?\n\nClarity:\nMostly, section 3.2 is confusing, discussing auto-labeled goal states -- how are the reached goal label z and corresponding goal state automatically collected during training? It seems the way in which the goal state is identified is through the last timestep of the episode $t^\\prime$. But the termination condition seems to come from the environment, which then provides supervision. If this interpretation is correct, I feel that this is not quite completely self-supervised learning, and rather a multi-task sparse reward setting -- the environment still needs to provide the label for when the task is completed.\n\nIt would also be interesting to see how much using goal-discriminator features in the attention mechanism improves performance. What if the outputs of $\\sigma$ are simply used instead?\n\nAnother useful comparison point would be the performance of training separate policies on each task (e.g., use a different actor/critic each time z changes). Because the proposed method is not goal-conditioned but rather instruction-conditioned, and so has a fixed set of possible instructions, this could be a good baseline to compare to.\n\nNitpick: Is the description on L138 necessarily true? This seems to depend on the particular rewards chosen for each task.\n\nThe paper is generally well-written, although as mentioned above, Section 3.2 was slightly confusing to me. \n\nSignificance:\nUsing GACE+GDAN leads to vastly better sample efficiency for the navigation task, even in settings where the baseline algorithms can solve the task.\n\nThe experiments on the continuous robotic control domain are a bit weaker than the visual navigation task -- there are only three possible object positions, and the tasks are pushing and reaching, so all algorithms produce relatively strong performance. It seems that SAC+CURL is able to achieve almost the same performance (within 1 standard deviation) of GACE+GDAN, with very similar sample efficiency. Therefore it is hard to say that the method is improving much here. An even more complex manipulation task could be insightful.  \n\nRecommendation: The ideas presented in this paper are quite nice and it also has some impressive empirical results. However, the robotic manipulation environments don't seem to significantly improve, the method could use additional comparisons as mentioned above, and it may not be suitable to tackle completely self-supervised settings as the authors claim. Therefore I feel that in its current state the paper should not yet be accepted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As pointed out in the conclusion of the paper, there is some disparity between the performance on the two tasks, and the authors note that this is due to the degree of similarity between goal states and others (assuming it is meant the similarity between goal states and other states in the manipulation setting). I agree with this assessment. The authors also address an important limitation that tasks must specify targets, and have high reward-state correlation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "gfYPCyrw8RZ", "reviewer": "Reviewer_LzLd", "summary": "This paper introduces a novel approach for improving sample efficiency in what they call \"multi-target reinforcement learning\". This is essentially goal-conditioned reinforcement learning where the tasks are of the form \"bring me a spoon\" or \"go to the kitchen\". Sample efficiency is improved in two ways. Firstly, they introduce a new auxiliary task of training a goal-discriminator that takes as input a state embedding and predicts which goal this state achieves. Second, they introduce a new architecture that uses attention and information from the goal discriminator to better extract task-relevant features for the policy.\n\nSeveral experiments are provided to test the sample efficiency of agents trained using the method in both visual navigation tasks and robotic manipulation tasks. In both settings, their method outperforms on-policy baselines. Additional analysis is done, showing that: (1) while explicitly optimizing the state representation to improve the goal discriminator results in the fastest improvement in accuracy in the discriminator, the RL loss itself still causes the representations to contain information about the goal, just slower. This shows that the representation learning loss is helping the model learn faster what it would have to learn anyways. And finally, (2) saliency maps are provided which show that when using attention, the policy attends to more interpretable regions of the state space.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: While representation learning and goal-conditioned RL are popular these days, the idea of using pairs of observations and the tasks that they solve to improve learned representations and reduce sample complexity seems novel.\n2. Quality: I think this is a neat idea that makes better use of the assumptions we already make when we do goal-conditioned RL (like there being some idea of \"targets\" that we have access to during training). I had a few questions when reading this that I am curious about:\n    1. Is there a reason why on-policy RL is used rather than off-policy + HER? If a practitioner is looking for a sample-efficient algorithm for goal-conditioned RL, they will likely use some combination of off-policy RL and HER. It would be great to see whether this representation learning approach works well with those algorithms.\n    2. In Section 3.2, you describe the method for collecting $s_t$ and $z$ pairs. This seems like a bit of a chicken and egg problem, where with a random policy I would imagine that there would be a very low chance of actually achieving a goal and getting a positive example, but it seems positive examples are needed to learn a good representation to learn the task. It seems like this would be trivial if you made the assumptions made in HER (that you can get a goal label for any state during training in order to relabel unsuccessful trajectories). Is this something you considered? Is there a reason you do not assume access to something like this since it seems necessary in order to evaluate the reward anyways.\n3. Clarity: I think clarity is the main area where this paper is lacking. Some points:\n    1. The introduction does not do a great job of motivating the approach or describing it.\n    2. Section 3.3 for the most part makes sense (you are classifying states based on their achieved goal), but the motivation is not clearly described. For example, on line 137 you write \"[the] vanilla base algorithm is not sufficient for gaining direct understanding of goals\". What does it mean for an algorithm to be not sufficient at this task? What does gaining a \"direct understanding of goals\" mean? It seems like since these multi-target RL tasks are just a collection of RL tasks, that an algorithm like A3C should be able to eventually solve them?\n    3. Section 3.4 is very unclear and I am still confused about GDAN after thoroughly reading the paper. I think this needs to be set up with much better motivation and described in a more intuitive way. \n    4. In Section 4.3, I am wondering how the discriminator accuracy goes up when it is not being trained anymore? This result would make sense if the discriminator was still learning but just not passing gradients back to the state encoder, but this isn't how it is described.\n4. Significance: This type of representation learning approach seems applicable in many applications and seems simple to implement (at least for GACE).\n\nOverall, I think the ideas behind the paper are solid (at least for GACE, I still need more clarity on exactly what is happening with GDAN). I would like to see the writing cleaned up and better explanations for some parts listed above.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does a good job of listing some potential limitations of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the multi-target RL settings, where the agent is to reach specified and variable goals (called targets) depending on the task specification, is an important setup and the proposed extension of RL formulation -- a loss to predict goal states and a second to focus on the features relevant for the goal -- are novel, well motivated, and empirically supported. Some of the reviewers recommend to slightly improve the writing and further clarify the self-supervision setup, which we believe can be easy accomplished for the final version. Hence accept.
Decision: Accept (Poster)