Reinforcement Learning with Automated Auxiliary Loss Search
Authors: Tairan He
Yuge Zhang
Kan Ren
Minghuan Liu
Che Wang
Weinan Zhang
Yuqing Yang
Dongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement learning
Representation learning
Auxiliary Loss

Abstract: A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learning informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automatically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size $7.5 \times 10^{20}$ and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2-winner) significantly improves the performance on both high-dimensional (image) and low-dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability to different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance. 

Reinforcement Learning with Automated Auxiliary Loss Search
Tairan He Yuge Zhang?
Kan Ren Minghuan Liu
Che Wang Weinan Zhang'
Yuging Yang"
Dongsheng Li
Shanghai Jiao Tong University "Microsoft Research Asia NNew York University
whynot@sjtu. edu.
kan. renêmi crosoft.com
Abstract
A good state representation is crucial to solving complicated reinforcement learning (RL) challenges. Many recent works focus on designing auxiliary losses for learn- ing informative representations. Unfortunately, these handcrafted objectives rely heavily on expert knowledge and may be sub-optimal. In this paper, we propose a principled and universal method for learning better representations with auxiliary loss functions, named Automated Auxiliary Loss Search (A2LS), which automat- ically searches for top-performing auxiliary loss functions for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space size 7.5 x 1020 and explore the space with an efficient evolutionary search strategy. Empirical results show that the discovered auxiliary loss (namely, A2 A2-winner) significantly improves the performance both high-dimensional (image) and low- dimensional (vector) unseen tasks with much higher efficiency, showing promising generalization ability different settings and even different benchmark domains. We conduct a statistical analysis to reveal the relations between patterns of auxiliary losses and RL performance. The codes and supplementary materials available at
baragml.githun github......... io/a2ls a21s
1 Introduction
Reinforcement learning (RL) has achieved remarkable progress in games 31 50 financial trading [8] and robotics [ However, in its core part, without designs tailored to specific tasks, general RL paradigms are still learning implicit representations from critic loss (value predictions) and actor loss (maximizing cumulative reward). In many real-world scenarios where observations are complicated (e.g., images) or incomplete (e.g., partial observable), training an agent that able to extract informative signals from those inputs becomes incredibly sample-inefficient
Therefore, many recent works have been devoted obtaining a good state representation, which is believed to be one of the key solutions to improve the efficacy of RL 13 24]. One of the main streams is adding auxiliary losses to update the state encoder. Under the hood, it resorts to informative and dense learning signals in order to encode various prior knowledge and regularization [40] and obtain better latent representations. Over the years, series of works have attempted figure out the form of the most helpful auxiliary loss for RL. Quite few advances have been made, including observation reconstruction [51], reward prediction (20). environment dynamics prediction
[0 35),
etc. But we note two problems in this evolving process: (i) each of the loss designs listed above are obtained through empirical trial-and -errors--errors based on expert designs, thus heavily relying on human labor and expertise; (ii) few works have used the final performance of RL an optimization objective to directly search the auxiliary loss, indicating that these designs could be sub-optimal.
The work was conducted during Tairan He's internship at Microsoft Research. The corresponding author is Kan Ren.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Gradient
Loop (RL training)
Outer Loop (Evolution)
Data
RL Training Scores
Replay Buffer
Select Top 25%
Policy Network
State Encoder
Loss
Mutation
Q Network
Reject
Loss Rejection
at
Auxiliary Loss
Auxiliary Loss
BL+k
Ft
Auxiliary Loss Candidates
Figure Overview of A2LS. A2LS contains an inner loop (left) and an outer loop (right). The inner loop performs an RL training procedure with searched auxiliary loss functions. The outer loop searches auxiliary loss functions using an evolutionary algorithm to select the better auxiliary losses.
To resolve the issues of the existing handcrafted solution mentioned above, we decide automate the process of designing the auxiliary loss functions of RL and propose principled solution named Automated Auxiliary Loss Search (A2LS). A2LS formulates the problem as a bi-level optimization where we try to find the best auxiliary loss, which, to the most extent, helps train a good RL agent. The outer loop searches for auxiliary losses based on RL performance ensure the searched losses align with the RL objective, while the inner loop performs RL training with the searched auxiliary loss function. Specifically, A2LS utilizes an evolutionary strategy to search the configuration of auxiliary losses over a novel search space of size 7.5 1020 that covers many existing solutions. By searching on small set of simulated training environments of continuous control from Deepmind Control suite (DMC) 43,, A2LS finalizes loss, namely A2-winner.
To evaluate the generalizability of the discovered auxiliary loss A2-winner, we test A2 12-winner on a wide set of test envirouments, including both image-based and vector-based (with proprioceptive features like positions, velocities and accelerations as inputs) tasks. Extensive experiments show the searched loss function is highly effective and largely outperforms strong baseline methods. More importantly, the searched auxiliary loss generalizes well unseen settings such as different robots of control; (ii) different data types of observation; (iii) partially observable settings; (iv) different network architectures; and (v) even to a totally different discrete control domain (Atari 2600 games (]). In the end, we make detailed statistical analyses on the relation between RI. performance and patterns of auxiliary losses based on the data of whole evolutionary search process, providing useful insights future studies of auxiliary loss designs and representation learning in RL.
2 Problem Formulation and Background
We consider the standard Markov Decision Process (MDP) & where the state, action and reward time step are denoted as (st, rt). The sequence of rollout data sampled by the agent in the
episodic environment is (Su...., St, Qt, St+1, ST), where T represents the episode length. Suppose the RL agent is parameterized by w (either the policy or the state-action value function Q), with state encoder ge parameterized by 0 C w which plays a key role for representation learning in RL. The agent required maximize its cumulative rewards in environment by optimizing w, noted as [ww; E) = E, 1TT-1 rt).
In this paper, we aim to find the optimal auxiliary loss function L such that the agent can reach the best performance by optimizing under a combination of an arbitrary RL loss function LRI. together with an auxiliary loss LAux. Formally, our optimization goal is:
max Exu
R(min (RL E) + AC (a; ();
(1)
where À a hyper-parameter balancing the relative weight the auxiliary loss. The left part (inner loop) of Figure I illustrraees how data and gradients flow in RL training when an auxiliary loss is enabled. Some instances of CRI. and LAux are given in Appendix Unfortunately, existing auxiliary losses LAux are handcrafted, which heavily rely expert knowledge, and may not generalize well

---

Table Typical solution with auxiliary loss and their common elements.
Input Flements
Auxiliary Loss
Operator
Horizon
Source
Target
Forward dynamics
Rd 6
MSE
[st, Rt)
8141
Inverse dynamics
MSE
{or, B
{s
Reward prediction 20 6
MSE
[$t: at)
Action inference 60.6 6
MSE
(81,841)
CURL
Bilinear
{+t)
{8r]
ATC
Bilinear
(+t)
{8++1
88tk) 814k)
SPR
N-MSE
(8t, ar a141," a(-+-1)
(841,
in different scenarios shown in the experiment part. To find better auxiliary loss functions for representation learning in RL, we introduce our principled solution in the following section.
Automated Auxiliary Loss Search
To meet our goal of finding top-performing auxiliary loss functions without expert assignment, turn to the help of automated loss search, which has shown promising results in the automated Correspondingly,
machine learning (AutoML) community [27 PS 4
propose Automated Auxiliary Loss Search (A2LS), a principled solution for resolving the above bi-level optimization problem in Equation M A2LS resolves the inner problem
standard RL training procedure; for the outer one, A2LS defines finite and discrete search space (Section 3.1 and designs a novel evolution strategy efficiently explore the space (Section 3.2
3.1 Search Space Design
have argued that almost all existing auxiliary losses require expert knowledge, and we expect to search for better automatically. To this end, it is clear that should design search space that satisfies the following desiderata.
Generalization: the search space should cover most of the existing handcrafted auxiliary losses to ensure the searched results can be no worse than handcrafted losses;
Atomicity: the search space should be composed of several independent dimensions fit into any general search algorithm 30) and support an efficient search scheme;
Sufficiency: the search space should be large enough to contain the top-performing solutions.
Given the criteria, we conclude and list some existing auxiliary losses in Table and find their commonalities, as well as differences. We realize that these losses share similar components and computation flow. As shown in Figure when training the RL agent, the loss firstly selects sequence {st. oz, rt Tt
1i+k from the replay buffer, when & is called horizon. The agent then tries to predict some elements in the sequence (called target) based on another picked set elements from the sequence (called source). Finally, the loss calculates and minimizes the prediction error (rigorously defined with operator). To be more specific, the encoder part go of the agent, first encodes the source into latent representations, which is further fed into predictor h to get prediction y; the auxiliary loss is computed by the prediction y and the target ÿ that translated from the target by a target encoder ga. using an operator f. The target encoder is updated in momentum manner shown in Figure (details are given in Appendix C.I.2.. Formally,
C (0: ():E) = (h (((g0((eQ (Se(garee))
where seqsoure: Sequaget C {sr, rt, simplicity, we will denote go(st, at, Tr, ...1,....) short for (90(s;), Tts Tts Cor.eeggosssa (or+))),, for the rest of this paper (the encoder only deals with states {si)). Thereafter, we observe that these existing auxiliary losses differ in two dimensions, i.e., input elements and operator, where inpur elements are further combined by horizon, source and target. These differences compose our search dimensions of the whole space. We then illustrate the search ranges of these dimensions in detail.
both subsets of the candidate sequence. And for
Input elements. The input elements denote all inputs to the loss functions, which can be further

---

Search Space f(
Input Elements
Auxillary Loss
Operator /
Example of forward dynamics prediction (horizon k=1)
y
Operator
Source
Stargt Fiei
stop-grad
Predictor
Target
update
i
78
7/
Encoder go
Target Encoder 9p
Re
Horizon
Input Elements I
F--1
Ru+1
Fuol
%
Pe-13
Pa+1
Figure 2: Overview of the search space {I, /} and the computation graph of auxiliary loss functions. I selects candidate sequence {st, az, rt re with horizon k; then determine a source and a targer arbitrary subsets of the sequence; an encoder go first encodes the source into latent representations, which is fed into a predictor to get a prediction y; the auxiliary loss is computed over the prediction y and the ground truth ý that is translated from the target by a target encoder Iý- using a operator f.
1ith
disassembled as horizon, source and target. Different from previous automated loss search works, the larget here is not "ground-truth" because auxiliary losses in RL have no labels beforehand. Instead, both source and larget are generated via interacting with the environment in self-supervised manner. Particularly, the inpul elements first determine a candidate sequence {st, a, r() 14+k with horizon Then, it chooses two subsets from the candidate sequence source and targel respectively. For example, the subsets can {st), ; {sc, 8t+1), or (st, To+1; (t+2), {81, $141; a++1) etc.
Operator. Given prediction y and its target D. the auxiliary loss is computed by operator f. which often a similarity measure. our work, we cover all different operators f used by bilinear inner product (Bilinear) (2)).
the previous works, including inner product (Inner) 117 42
cosine similarity (Cosine) []. mean squared error (MSE)
35 and normalized mean squared error Additionally, other works also utilize contrastive objectives, e.g-, InfoNCE loss (33 incorporating the trick to sample un-paired predictions and targets as negative samples and maximize the distances between them. This technique orthogonal the five similarity measures mentioned above, so we make it optional and create 5 x 2 = 10 different operators in total.
(N-MSE) 39
Final design. In the light of preceding discussion, with the definition of input elements and operator, we finish the design of the search space, which satisfactorily meets the desiderata mentioned above. Specifically, the space generalizable to cover most of the existing handcrafted auxiliary losses; additionally, the atomicity is embodied by the compositionality that all inpul elements work with any operator; most importantly, the search space is sufficiently large with a total size of 7.5 1020 (detailed calculation can be found in Appendix B to find better solutions.
3.2 Search Strategy
The success evolution strategies exploring large, multi-dimensional search space has been proven in many works 19 4.
Similarly, A2LS adopts an evolutionary algorithm 3 to search for top- performing auxiliary loss functions over the designed search space. In its essence, the evolutionary algorithm (i) keeps a population of loss function candidates; (ii) evaluates their performance; (iii) eliminates the worst and evolves into a new better population. Note that step (ii) of "evaluating" very costly because it needs to train the RL agents with dozens of different auxiliary loss functions. Therefore, our key technical contribution contains how to further reduce the search cost (Section 3.2. and how to make an efficient search procedure (Section 3.2.2
3.2.1 Search Space Pruning
In our preliminary experiment, we find out the dimension of operator in the search space can be simplified. In particular, MSE outperforms all the others by significant gaps in most cases. So we effectively prune other choices of operators except MSE. See Appendix [D. for complete comparative results and ablation study the effectiveness of search space pruning.

---

Mask before mutation
Mask After mutation
Horizon decrease
Sep+
Rc-1
Pie1
Resa
Ps-a
Sip1
Re-
Pi+1
Horizon increase
Replacement
Pial
Selected
Not Selected
%
Rest
Pesl
Crossover
Ac+1
Topl
Changed
R--1
Figure 3: Four types of mutation strategy for evolution. We represent both the source and the larget of the input elements as a pair of binary masks, where each bit of the binary mask represents selected (green block) by I or not selected (white block) by 0.
3.2.2 Evolution Procedure
Our evolution procedure roughly contains four important components: (i) evaluation and selection: a population of candidate auxiliary losses is evaluated through an inner loop of RL training, then we select the top candidates for the next evolution stage (i.e., generation): (ii) mutation: the selected candidates mutate to form new population and move to the next stage; (iii) loss rejection: filter and skip evaluating invalid auxiliary losses for the next stage; and (iv) bootstrapping initial population: assign more chance to initial auxiliary losses that may contain useful patterns by prior knowledge for higher efficiency. The step-by-ster evolution algorithm is provided in Algorithm [ in the appendix, and an overview of the A2LS pipeline is illustrated in Figure We next describe them in detail.
Evaluation and selection. At each evolution stage, first train population of candidates with a population size = 100 by the inner loop of RL training. The candidates are then sorted by computing the approximated area under learning curve (AULC) [ 41 which is single metric reflecting both the convergence speed and the final performance 46 with low variance of results.
After each training stage, the top-25% candidates selected to generate the population for the next stage. We include an ablation study on the effectiveness AULC in Appendix D33
Mutation. To obtain new population of auxiliary loss functions, we propose novel mutation strategy. First, we represent both the source and the target of the input elements as pair of binary masks, where each bit of the mask represents selected by 1 or not selected by For instance, given candidate sequence {st, az, Tz, St+1. 1,+1, 1 the binary mask of this subset sequence (8t; 0t, P++1 is denoted as 110001. Afterward, we adopt four types of mutations, also shown in Figure 3 (i) replacement (50% of the population): flip the given binary mask with probability with the horizon length k; (ii) crossover (20%): generate a new candidate by randomly combining the mask bits of two candidates with the same horizon length in the population; (iii) horizon decrease and horizon increase (10%): append new binary masks to the tail or delete existing binary masks at the back. (iv) random generation (20%): every bit of the binary mask is generated from Bernoulli distribution ((0.5).
p= 2-(3++3
Loss rejection protocol. Since the auxiliary loss needs to be differentiable with respect to the parameters of the state encoder, we perform a gradient flow check on randomly generated loss functions during evolution and skip evaluating invalid auxiliary losses. Concretely, the following conditions must be satisfied to make a valid loss function: (i) having at least one state element in seqsoure to make sure the gradient of auxiliary loss can propagate back to the state encoder; (ii) seQuareet is not empty: (iii) the horizon should be within a reasonable range S k 10 in our experiments). If loss is rejected, we repeat the mutation to fill the population.
Bootstrapping initial population. To improve the computational efficiency so that the algorithm can find reasonable loss functions quickly, we incorporate prior knowledge into the initialization of the search. Particularly, before the first stage of evolution, we bootstrap the initial population with prior distribution that assigns high probability to auxiliary loss functions containing useful patterns like dynamics and reward prediction. More implementation details provided in Appendix OK
5

---

Evolution and Searched Results
1246
CUR
doegg
249
180
SAC
Aap03 29202 (nged
#lpe- augoo stppe 2ago7
29p01 2app2
zappo
slagod stapec
sugo?
xapel
saapo
113094
Cheetai-Run (Image)
Reacher-Easy (Image)
Walker-Walk (Image)
Figure 4: Evolution process in the three training tamage-based environments. Every white dot represents candidate of auxiliary loss, and y-axis shows corresponding approximated AULC score MΑ The horizontal lines show the scores of the baselines. The AULC score is approximated with the average evaluation score at 100k, 200k, 300k, 400k, 500k time steps.
As mentioned in Section we expect to find auxiliary losses that align with the RL objective and generalize well unseen test environments. To do so, we use A2LS search over small set of training environments, and then test the searched results on a wide range of test environments. In this section, we first introduce the evolution training environments and search results.
4.1 Evolution Training Environments
The training environments chosen three image-based (observations for agents images) continuous control tasks in DMC benchmark [3), Cheetah-Run, Reacher-Easy. and Walker- Walk. For cach environment, the search when the resource exhausted. Due to computation complexity, we only run one seed for each inner loop RL training, but try to prevent such randomness cross validation (see Section
set the total budget 16k GPU hours (on NVIDIA P100) and terminate
4.2 We use the same network architecture and hyperparameters config as CURL 2
(see
Appendix 4. for details) to train the RL agents. To evaluate the population during evolution,
measure A 22S as compared to SAC, SAC-wo-aug, and CURL, where we randomly crop images from 100 100 84 84 data augmentation (the same technique used in CURL 23 for all methods except SAC-wo-aug. The whole evolution process on three environments is demonstrated Figure Even in the early stages (c.g.. stage 1), some of the auxiliary loss candidates already surpass baselines, indicating the high potential of automated loss search. The overall AULC scores of the population continue improve when more stages come in (detailed numbers are summerized in Appendix D.I. Judging from the trend, we believe the performances could improve even more if we had further increased the budget.
4.2 Searched Results:
A2-winner
Although candidates in the population have achieved markable AULC scores in the evolution (Figure D. they were only evaluated with one random seed in environment, making their robustness under question. To ensure that we find a consistently-uset validation. We first choose the top candidates of stage-5 the evolution on Cheetah-Run (detailed top candidates during the whole evolution procedure are provided in Appendix F For each of the five candidates, we repeat the RL training on all three training environments, shown in Figure Finally, we mark the best among five (green bar in Figure as our final searched result. We call it A2-winney which has the following form:
auxiliary loss, conduct cross
Cheetab-Run
Reacher-Eee Easy
Walker-Woik
Figure 5: Cross validation on image- based training environments.
LAuX (0: = =11((g0((444 a+11 a442, (433) go(Te, 8142, s143)l2.

---

Table 2: Episodic rewards (mean & standard deviation for 10 seeds) on DMC100K (100K time steps)
and DMC500K (500K time steps). Note that the optimal score of DMC is 1000 for all environments.
The baseline methods are PlaNet [16], Dreamer (15. SAC+AE 51 SLAC [26], image-based SAC [1]] Performance values of all baselines are referred to 23
except for Image SAC. Learning
curves of all 12 DMC environments are included Appendix D.2
500K Steps Scores
A2-winner
CURL
PlaNet
Dreamer
SAC+AE
SLACv1
Image SAC
Chectah-Run'
613
518 28
305 + 131
570 + 253
550 34
640 19
99 = 28
Reacher-Easy
938
929 44
390
793
164
627 58
312 1 132
Walker-Walk'
917
902 + 43
351 + 58
897 + 49
847 48
842 = 51
76 + 44
Finger-Spin"
983 +
926 + 45
561 + 284
796 + 183
884 128
673 92
282 + 102
Cartpole-Swingup
864
841 + 45
475 71
762 + 27
735 -
344 = 104
Ball in cup-Catch"
970 +
959 + 27
460 + 380
897 + 87
794 +
852
200 = 114
100K Steps Scores
Cheetah- R
449 34
299 48
138 1 88
235 137
267 24
319 1 56
128 1. 12
Reacher-Fasy!
778 + 164
538 + 223
20 + 50
314 + 155
274 + 14
277 + 69
Walker- Walk
510 + 151
403 4: 24
224 1 48
277 1: 12
394 +: 22
361 + 73
127 1: 28
Singer-Spir'
872 + 27
767 + 56
136 - 216
341 + 70
740 + 64
603 : 141
160 = 138
Cartpole-Swingue
815 66
582
+ 146
297 + 39
326 +: 27
311 + 11
243 +: 19
Ball in Catchitchi"
862 1 167
769 43
0 0
246 1 174
391
82
512 L 110
100 T 90
t: Training environments. Unseen test environments. 9: Results reported in
23
5
Generalization Experiments
To verify the effectiveness of the searched results, we conduct various generalization experiments on wide range of fest environments in depth. Implementation details and more ablation studies are given Appendix
C and Appendix D
Generalize to unseen image-based tasks. We first investigate the generalizability of A2-winner to unseen image-based tasks by training agents with A2-winner on common DMC tasks and compare with model-based and model-free baselines that use different auxiliary loss functions (see Appendix C.5 for details about baseline methods). The results are summarized in Table 2 where A2-winner greatly outperforms other baseline methods most tasks, including unseen fest environments. This implies that A2-winner is a robust and effective auxiliary loss for image-based continuous control tasks to improve both the efficiency and final performance.
Generalize to totally different benchmark domains. To further ver- ify the generalizability of A2-winner on totally different benchmark domains other than DMC tasks, we conduct experiments on the Atari 2600 Games D. where we take Efficient Rainbow [4l] as the base RL algorithm and add A2-winner to obtain a better state representation. Results are shown in Table 3 where A2-winned outperforms all baselines, showing strong evidence the generalization and potential usages of A2-winner. Note that the base RL algorithm used in Atari is value-basse based method, indicating that A2-winner generalizes well to both value-based and policy- ppsecy-based RL algorithms.
Table 3: Mean and Median scores (normalized by human score and random score) achieved by A2LS and baselines on 26 Atari games benchmarked at 100k time-steps (Ataril00k).
CUBL
Bd. Rainbow
2X]
Random
Human
Meon Eluman Norm'd
0,568
0,381
0.285
0.357
0.000
1.000
Mediun Noraid
0.317
0,175
0,258
0.800
1.000
Generalize to different observation types. To see whether A2-winner (searched in image-based environments) is able to generalize to the environments with different observation types, we test A2-winner on vector-based (inputs for RL agents are proprioceptive features such as positions, velocities and accelcrations) tasks of DMC and list the results in Table 4 Concretely, we compare A2 12-winner with SAC-Identity, SAC and CURL, where SAC-Identity does not have state encoder while the others share the same state encoder architecture (See Appendix C.I.I and Appendix D for detailed implementations). To our delight, A2-winner still outperforms all baselines in 12 out of 18 enviromments, showing A2-winner can also benefit RL performance in vector-based observations. Moreover, the performance gain particularly significant more complex environments like Humanoid, where SAC barely learns anything at 1000K time steps. In order to get deeper understanding of this phenomenon, we additionally visualize the Q loss landscape for both methods in Appendix D.7
7

---

Table 4: Episodic rewards (mean & standard deviation for 10 seeds) on DMC100K (casy tasks) and
DMC1000K (difficult tasks) with vector inputs.
100K Steps Scores
A2-winner
A2-vinner-V
SAC-Identity
SAC
CURL
Chectah- Run"
529 76
472 + 30
237 +
172 + 29
190
Finger Spin"
790 1 128
837 :: 52
805 +: 32
785 4: 106
712 4: 83
Finger- Tum hard"
272 +
218 117
347
174 +
42
SartpoleSwingue
866 24
877 +
873 = 10
866 =
854 = 17
Caripole-S Swingup sparse'
634 +: 226
695 147
455 : 359
627 307
446 : 196
Reacher Easy"
818 1: 211
934 1 38
697 1 192
874 1: 87
749 1: 183
Walker-Stand"
935 + 32
948 7
940 + 10
862 196
767 + 104
Walker- Walk"
932
906 +
873
925 +
852
Walker-Run Run"
616 = 52
564 1 45
559 + 34
403 = 43
289 = 61
Ball in cup- Catch"
964
Fish- Uprighh"
586 + 128
498 + 88
471 + 62
400 + 62
295 + 117
Hopper-
177 + 257
311 177
14 16
26 40
1,000K Steps Scores
42-vinner-V
A2-vinner
SAC-Identity
SAC
CURL
Quadruped-Run Run!
863 = 50
838 + 58
345 + 157
707 + 148
497 + 128
Hopper-Hop'
213 = 31
278 =: 106
121 # 51
134 + 93
60 =
Pendulum- Swingup"
200 +: 322
579 410
506
379 191
363 366
Humamoid Stand"
329 35
286 1 155
9 9-1.2
7 7-1.
7 7:1:1
Humanoid- Walk"
299
2
Humanoid Run'
75 15 37
88
11:
14:0
1
t: Training environmentss. Unseen test ensironments.
Generalize to different hypothesis spaces. The architecture of a neural network defines a hypothe- sis space of functions be optimized. During the evolutionary search in Section 4.1 the encoder archi- tecture has been kept static 4-layer convolutional neural network. Since encoder architecture may have a large impact on the RL training process 34 2), we test A2- with three encoders with different depth of neural networks. The result is shown in 6 Note that even though the auxiliary loss is searched with a 4-layer encoder, the 6-layer convolutional encoder able to perform better both two environments. This proves that the auxiliary loss function of A2-winner is able to improve RL performance with a deeper and more expressive image encoder. Moreover, the ranking of RL performance (6-layer > 4-layer 2-layer) is consistent across the two environments. This shows that the auxiliary loss function of A2-winner does not overfit one specific architecture of the encoder.
500
933
230
Goayar
SE64
2004
20ck
Bell oup Goleh
Chestan-tas
Figure 6: Comparison of A2-winner with different depth of convolutional encoder in image-based DMC environments.
Figure
Generalize to partially observable scenarios. Claiming the generality of method based on con- clusions drawn just on fully observable environments like DMC is very dangerous. Therefore, we conduct an ablation study on the Partially Observable Markov Decision Process (POMDP) setting to see whether A2-winner is able to perform well POMDP. We random mask 20% of the state dimensions (c.g.. 15 dimensions -> 12 dimensions) to form a POMDP enviromment in DMC. As demonstrated in Figure 7 A2-vinner consistently outperforms CURL and SAC-DenseMLP in the POMDP setting in Hopper- Hop and Cheetah- Run, showing that wi-winner is not only effective in fully observable environments but also partially observable environments.
200
400
Sac Densett
AZediner
Culi..
1250 Stepe
Steat
Cheessh-Run
Heppen-Hop
Figure 7: Comparison of A2-winner and baselines in partially observable vector-based DMC environments.
To search or not? As shown above, the searched result A2-winner can generalize well all kinds of different settings. A natural question here is, however, for a new type of domain, why not perform new evolution search, instead of simply using the previously searched result? To compare these two solutions, we conduct another evolutionary search similar to Section 4.1 but replaced the three image-based tasks with three vector based ones (marked by in Table 4 from scratch. More details are summarized in Appendix D.5] We name the searched result as "A2-winner-v". As shown in Table A2-winner-v is a very strong-perfore peroon-peffommng loss for vector- based tasks, even stronger than A2 Ainwinner. Actually, A2-winner-v is able to outperform baselines in 16 out of 18 environments (with 15 unseen test environments) while A2-winner only outperforms baselines in 12 out of 18 environments. However, please note that it costs another 5k GPU hours (on NVIDIA P100) to
8

---

Table Statistical analysis on auxiliary loss functions. The number reported is the difference of the expected RL score when the auxiliary losses have one pattern compared to those do not have. The corresponding p-value from the t-test also reported. Positive numbers indicate that this pattern is beneficial. If the performance gain is statistically significant, the number is marked with the asterisk, indicating it is very likely to be helpful. Negative numbers indicate this pattern is detrimental.
The score difference between average performances w/ and w/o typical patterns (w/ - w/o)
Forward dynamics
Inverse dynamics
Reward prediction
Action inference State reconstruction
Checiah--uus Run (Image)
+1.28
-3.11
-31.16"
- -75.95
+
Reacher-Easy (Image)
+28 25"
+8.36
+37.80
+3.35
+70.72
Walker-Wall (Image)
+22.20
-48,59"
-8.11
+-29. 86"
+13.93
Cheetah Run (Vector)
+94.18** 18
-33.66
-33.28**
-109 1000.33*
-50. 15"
Hlopper-Hop (Vector)
+15.50
16.47*
-11.30*
-32,10"
-25.677 25.67
Quadruped Run (Vector)
-28.07
- 18.19
- -114.23
-105.37
-82.06*
p-value 0.05. **: p-value < 0.01
The score difference between two sets varying the number of elements source and target
State, Rigrget > Paaarrs
Action, Vegnget >>
Plaaures
Keward, nituree > Plyuurer
Chectab-Run (Image)
+-80.99*
+13.02
+3.
Reacher-Eas Easy (Image)
-1.98
-2.72
-63 66.
Walker-Walk (Image)
+73.56*
12 22
Cheetah-Run Run (Vector)
+ +188. (16
- -102 62
-KI 94""
Hopper-Hop (Vector)
+1
29. 70
-5.03
Quadruped-Rel Run (Vector)
+75.17"
-1.11
16. 16.60"
+: *-p-value < 0.05, en; p-value < 0.01
search for A2-winner-v while there is additional cost directly use A2-winner. It is a trade-off between lower computational cost and better performance.
6 Analysis of Auxiliary Loss Functions
In this section, we analyze all the loss functions we have evaluated during the evolution procedure as a whole dataset in order to gain some insights into the role of auxiliary loss in RL performance. By doing so, we hope shed light future auxiliary loss designs. We will also release this "dataset" publicly to facilitate future research.
Typical patterns. We say that an auxiliary loss candidate has a certain pattern if the pattern's source is subset of the candidate's 's source, and the pattern's 's target is subset of the candidate's 's target. For instance, loss candidate of { {st, }²} + { {st+1, $t+2} has the pattern {st, ar} + {$L+1} }, and does not have the pattern {ar, $ + { We then try to analyze whether a certain pattern is helpful representation learning in RL expectation.
Specifically, we analyze the following patterns: (i) forward dynamics {se; at} - { J: (ii) inverse dynamies {a,, 11+13 - {st]: (iii) reward prediction {st, a -> {re]: (iv) action inference St11 + {a;} } and (v) state reconstruction in the latent space {st} + { For each of these patterns, we categorize all the loss functions we have evaluated into with or (ii) without this pattern. We then calculate the average RL performances of these two categories, summarized in Table 5 Some interesting observations are as follows.
Forward dynamics is helpful in most tasks and improves RL performance on Reacher-Easy (image) and Cheetah-Run (vector) significantly (p-value-c0.05).
(ii) State reconstruction in the latent space improves RL performance in image-based tasks but undermines vector-based tasks. The improvements in image-based tasks could be attributed to the combination of augmentation techniques, which, combined with reconstruction loss, enforces the extraction of meaningful features. In contrast, no augmentation is used in the vector-hased setting, and thus the encoder learns useful representations. This also explains why CURL performs poorly in vector-based experiments.
(iii) In the vector-based setting, some typical human desiande patterns (e.g., reward prediction, inverse dynamics, and action inference) can be very detrimental to RL performance, implying that some renowned techniques in loss designs might not work well under atypical settings.
Number of Sources and Targets. We further investigate whether it more beneficial to use a small number of sources to predict large number of targets (Mtarget > Rssorross e.g., using 8, to predict 8141, 8 8443), or the other way around (Mearget <, Tssorees e.g., using 8t, 8,+1, 86+2 to predict 8143). Statistical results are shown in Table 5l where we find that auxiliary losses with more states
9

---

on the targer side have a significant advantage over losses with more states on the souree side. This result echoes recent works 42 39
predicting more states leads to strong performance gains.
Related Work
Reinforcement Learning with Auxiliary Losses. Usage of auxiliary tasks for learning better state representations and improving the sample efficiency of RI. agents, especially on image-based tasks, has been explored in many recent works. A number of manually designed auxiliary objectives
shown to boost RL performance, including observation reconstruction and contrastive learning objectives 2X
reward prediction [20]. It is worth noting that most of these works focus on image-based settings, and only limited number of works study the Although people may think that vector-based settings can benefit less from auxiliary tasks due to their lower-diment swwer-dimensionll state space, we show in our paper that there is still much potential for improving their performance with better learned representations.
dynamics prediction
vector-based setting 2 35
Compared to the previous works, we point out two major advantages our approach, (i) Instead of handcrafting auxiliary loss with expert knowledge, A2LS automatically searches for the best auxiliary loss, relieving researchers from such tedious work. (ii) A2LS a principled approach that be used in arbitrary RL settings. We discover great auxiliary losses that bring significant performance improvement in image-based and the rarely studied vector-based settings.
Automated Reinforcement Learning. RL training notoriously sensitive to perameerrametess and environment changes [8] Recently, many works attempted to take techniques in AutoML to alleviate human intervention, for example. hyper-parames
optimization 7 36 53 reward search y 45 and network architecture search [38 [10]. In contrast to these methods which optimize
new configuration for each environment, we search for auxiliary loss functions that generalize across different settings such as (i) different robots of control; (ii) different data types of observation; (iii) partially observable settings; (iv) different network architectures; (v) different benchmark domains.
Automated Loss Design. In the AutoML community, it has become trend to design good loss functions that can outperform traditional and handcrafted ones. To be specific, to resolve computer vision tasks, AM-LFS 27 defines the loss function search space parameterized probability distribution the hyper-parameters of softmax loss. A recent work, AutoLoss-Zero 28 proposes to search loss functions with primitive mathematical operators.
For RL, existing works focus on searching for better RL objective, EPG 19] and MetaGenRL 21 define the search space of loss functions parameters of low complexity neural network. Recently, [ defines the search space of RL loss functions as a directed acyclic graph and discovers two DQN-like regularized RL losses. Note that none of these works investigates auxiliary loss functions, which are crucial to facilitate representation learning in RL and to make RL successful in highly complex environments. To the best of our knowledge, our work is the first attempt to search for auxiliary loss functions that can significantly improve RL performance.
8 Conclusion and Future Work
We present A2LS, a principled and universal framework for automated auxiliary loss design for RL. By searching on training environments with this framework, we discover a top-performing auxiliary loss function A2-wi A2-winner that generalizes well to diverse of test environments. Furthermore, present an in-depth investigation of the statistical relations between auxiliary loss patterns and RL performance. We hope our studies provide insights that will deepen the understanding of auxiliary losses in RL, and shed light on how to make RL more efficient and practical. Limitations of our current work lie in that searching requires an expensive computational cost. In the future, we plan incorporate more delicate information such as higher-order information 122 of the inner-loop RI. training procedure derive more efficient auxiliary loss search methods.
10

---

References
[1] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 01:2553-77,, 2013.
[2] Johan Bjorck, Carla P Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning. arXiv preprint arXiv:2106.01151, 2021.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[4] John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Quoc V Le, Sergey Levine, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Conference on Learning Representations, 2020.
[5] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, and Joseph E. Gonzalez. Fbnetv3: Joint architecture- recipe search using predictor pretraining, 2020.
[6] Tim De Bruin, Jens Kober, Karl Tuyls, and Robert Babuška, Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters, 3(3):1394- 1401, 2018.
[7] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machine Learning, pages 1407-1416. PMLR, 2018.
[8] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie Yan Liu. Universal trading for order execution with oracle policy distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 107-115, 2021.
[9] Alcksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reinforce- ment learning, arXiv preprint arXiv: 1905.07650 2019.
[10] Jörg K. H. Franke, Gregor Köhler, André Biedenkapp, and Frank Hutter. Sample-efficient automated deep reinforcement learning. In 9th International Conference on Learning Represen- tations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[11] Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. arXiv preprint arXiv:2003.07417, 2020.
[12] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. In International Conference on Machine Learning. pages
PMLR, 2019.
[13) Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off off-policy updates. In 20 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
[14] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 1812.0881 2018.
[15] Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 19/2.0103,, 2019.
[16] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference Machine Learning, pages 3555-2565. PMLR, 2019.

---

[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[18] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
[19] Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5405-5414, 2018.
[20] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[21] Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[22] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint orXiv:2004.13649 2020.
[23] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In International Conference Machine Learning, pages 6639-5650. PMLR, 2020.
[24] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 01:19984-9999 2020.
[25] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in Neural Information Processing Systems, 33:19884-19888 2020.
[26] Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint 1907.0/9900009000 2019.
[27] Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am- Ifs: Automl for loss function search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 84 8410-8419, 2019.
[28] Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu, Autoloss-zero: Searching loss functions from scratch for generic tasks. arXiv preprint arXiv:2103. 200.::2003..0266 2021.
[29] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[30] M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp. Marben, P. Müller, and F. Hutter. Boah: A tool suite for multi-fidelity bayesian optimization & analysis of hyperparameters. arXiv: 1908.. 06756 [cs.
[31] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13 12.6002 2013.
[32] Jelle Munk, Jens Kober, and Robert Babuška. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 4667-4673. IEEE, 2016.
12

---

[33] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1xXiv:8077003444 2018.
[34] Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning, arXiv preprint arXiv:2102.07922 2021.
[35] Kei Ota, Tomoaki Oiki, Devesh Jha. Toshisada Mariyama, and Daniel Nikovski. Can increasing imput dimensionality improve deep reinforcement learning? In International Conference on Machine Learning, pages 7424-7433. PMLR, 2020.
[36] Supratik Paul, Vitaly Kurin, and Shimon Whiteson. Fast efficient hyperparameter tuning for policy gradients. arXiv preprint urXiv:/902.000
2019.
[37] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, 4789, Jul 2019.
[38] Frederic Runge, Danny Stoll, Stefan Falkner, and Frank Hutter. Learning to design RNA. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
[39] Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- ceff-rrdiitive representations. In Interna- tional Conference on Learning Representations, 2020.
[40] Evan Shelhamer, Parsa Mahmoudich, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv: 2016.
[41] Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint urXiv:1507.00814 2015.
[42] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[43] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint
2018
[44] Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
[45] Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Hanna M. Wallach, Hugo Larochelle, Alína Beygelzimer, Florence d' Alcch--Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 9317, 2019.
[46] Tom Viering and Marco Loog. The shape of learning curves: a review. arXiv preprint arXiv:2103. 10948, 2021.
[47] Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-33 354, 2019.
[48] Xiaobo Wang, Shuo Wang, Cheng Chi. Shifeng Zhang, and Tao Mei. Loss function search for face recognition. In International Conference on Machine Learning, pages 10029-10038. PMLR, 2020.
[49] Zhongwen Xu, Hado Philip van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. In Hugo Larochelle, Mare Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
13

---

[50] Guan Yang, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect information distillation. arXiv preprint arXiv:2203. 16406, 2022.
[51] Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019.
[52] Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search, 2019.
[53] Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. self-tuning actor-critic algorithm. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We include data, implementation details for reproduction.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We try to cover all the training details.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "euKAixEmLHn", "reviewer": "Reviewer_bXkd", "summary": "The paper proposes to use an evolutionary strategy to search the best auxiliary losses for reinforcement learning. The main technical contribution is the design of a search space that covers a set of common auxiliary losses such as the forward dynamics, inverse dynamics and reward prediction. Empirical experiments show that, a loss searched on 3 image-based DM Control tasks also brings better sample efficiency on other DM Control tasks and Atari games.", "questions": "Other suggestions:\n1. Eqn. 1 seems a bit odd. Maybe it's better to write it as $\\max_{\\mathcal{L}_{aux}} \\mathcal{R}(\\min_w \\text{losses})$\n\n2. The normalized AUC score is not explained. Normalized w.r.t. what?", "limitations": "Please see above. Since searching over auxiliary losses is computationally expensive and unlikely for other researchers to search for each task they are interested in, I think it is important to state clearly when does the searched auxiliary task transfer and when it does not.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* I like the overall direction of the paper. It did a good job exploring whether we can find a good set of auxiliary losses that transfer to different tasks. It also provides insights (through empirical experiments) what are important auxiliary tasks.\n* The experiments are extensive well presented to support the main claims of the paper. The generalization of the searched auxiliary loss are demonstrated on different tasks as well as on different RL algorithms (SAC and Efficient Rainbow) and different input observation.\n\n### Weakness\n1. A main question that is not fully addressed in the paper is, to what degree is the searched auxiliary loss (i.e. A2-winnder) universal? When does the auxiliary loss transfer and when does it not? To provide two angles to explore in this aspect:\n    * The searched loss in Eqn. 3 involves states up to 3 time steps into the future. Where does this horizon of 3 come from? In another environment where the time is discretized differently, either more frequently or less frequently, should this loss be adapted accordingly? One experiment that can be done is, for the currently tested environment, if we add action repeat i.e. repeat each action for two steps, would the optimal auxiliary loss change?\n    * The tasks evaluated in this paper are all 2D tasks. How about the more complicated 3D manipulation tasks, such as meta-world [1], \nwhich is also commonly used for evaluating RL algorithms? \n\n2. Some related works are missing. There are a body of works on balancing multiple auxiliary tasks for reinforcement learning [2,3,4]. These works discuss the interplay between the auxiliary tasks and the main task, as well as among the auxiliary tasks. Specifically, Lin et al. [2] also considers a set of auxiliary tasks are also considered, including the forward and inverse dynamics. Lin et al. [2] proposes to tune the weights of the auxiliary tasks based on the main task performance. In a way, it provides a more efficient way of evaluating an auxiliary loss, and also provides an extention of the search space in the submitted paper by including the weights of the auxiliary losses.\n\nI am willing to raise my score if these concerns can be sufficiently addressed.\n\n[1] Yu, Tianhe, et al. \"Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.\" Conference on robot learning. PMLR, 2020.\n\n[2] Lin, Xingyu, et al. \"Adaptive auxiliary task weighting for reinforcement learning.\" Advances in neural information processing systems 32 (2019).\n\n[3] Shi, Baifeng, et al. \"Auxiliary task reweighting for minimum-data learning.\" Advances in Neural Information Processing Systems 33 (2020): 7148-7160.\n\n[4] Chen, Zhao, et al. \"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks.\" International conference on machine learning. PMLR, 2018. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZxuiJkpYvt5", "reviewer": "Reviewer_qgb7", "summary": "This paper proposes an evolutionary strategy (ES)-based auxiliary loss search for training neural networks for RL. The main motivation is that while it is known that auxiliary losses can improve training by aiding useful representation learning, it is not clear which formulation of the auxiliary loss is most beneficial since the number of potential possibilities is very large. Therefore, it can be valuable to automate the search through this space in order to identify the most useful loss functions. The paper proposed to use binary-encoded selection variables for the inputs and targets (called *input elements*) of the auxiliary loss, with associated mutation operations for the ES. For the functional form of the loss itself (called *operator*), the paper does not use a search but instead uses simple random sampling to identify that the MSE loss works best when the inputs and targets selection are such that the auxiliary loss becomes forward dynamics. \n\nFixing the loss to MSE and using the ES to search for the input elements, the authors identify an auxiliary loss using 3 image-based control environments for training, that leads to improved performance on multiple image-based test environments. It is also able to improve performance on 12 out of 18 tested non-visual RL tasks showing that it can generalize to a variety of tasks. The authors also present aggregated analyses of the losses evaluated during the search procedure to identify common useful patterns among loss functions that improve or reduce performance.", "questions": "Did I understand correctly above that A2-winner was obtained with 5 generations of the ES? If so, can the authors offer convincing arguments that the proposed ES is necessary, and address related comments above?\n\nIs appears that at 500K, the performance with A2-winner might not be too different (statistically) from CURL. Is that correct?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this paper is that it is the first study to show the feasibility of automating the search of auxiliary loss functions for RL with results indicating that a useful and novel loss function was indeed discovered through the proposed method, with a relatively simple evolutionary strategy design.\n\nI also appreciate that the authors attempted extensive generalization experiments on several unseen environments and compared to strong baselines.\n\nI have some concerns that reduce my confidence in accepting this paper:\n\n- The biggest concern is that it appears that the identified A2-winner loss is obtained with just 5 generations of the proposed ES (the term *stages* appears to be used in the paper, which I found a little confusing without explanation). This makes me wonder if the ES is actually necessary, and whether a random sampling of 4 or 5 times the population size would already produce equally good results, while being more parallelizable. The authors comment in Sec. 4.1 that \"Judging from the trend, we believe the performances could improve even more if we had further increased the budget\", but it unclear to me from Fig. 4 that there really is a trend of increasing performance, especially in terms of the best individual in the population improving across the three environments.\n\n- In Table 2, many results for A2-winner at 500K steps are in bold, but their difference from CURL does not appear to be statistically significant.\n \n- The main paper omits any discussion of how the target encoder parameters are obtained. This is left completely to the supplementary material, and confused me for the first couple of readings. \n\n- The paper is notoriously missing references from the ES literature, focusing only on references to some recent papers that apply ES to RL.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G4TaqQjzN9E", "reviewer": "Reviewer_Jaq4", "summary": "The paper proposes a method to do automated search for reinforcement learning auxiliary losses. This is done by formulating a configuration for possible losses, which covers many of the common losses used in the field (e.g. contrastive, self-predictive), and searching using evolutionary search. This leads to a bi-level optimization: outer loop -- proposing, selecting and evolving candidate loss functions -- and inner loop, where the candidate loss function is evaluated by training an RL agent together with the candidate loss, and using the overall performance as a metric for the outer loop. \n\nThe method is applied to continuous control DeepMind Control Suite, where an auxiliary loss (A2-winner) is found that consistently outperforms the baselines. Moreover, this loss has similarly strong results on different test environments, including e,g, some with different observation types or discrete control domain (Atari).", "questions": "In addition to the points above (including information about the inner loop algorithm in the main paper and adding a short discussion on how the baselines were chosen), the following aspects might be useful to consider:\n- How does the method compare with other AutoML methods applied to the task? \n- Do the findings apply when tested in the setting of task generalization? \n- What values of lambda (eq 1) were tried and how do the results compare?\n- Section 3.2.1 provides a very specific method to prune the search space, seemingly inspired by empirical evidence, rather than a general assumption. More discussion on results/generality could make its scope clearer.\n- Does the search space include reconstruction losses at the moment? ", "limitations": "The authors have adequately noted and addressed the limitations of the proposed method. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has many strong aspects: clarity, thorough evaluation of the proposed candidate loss on many different test environments and originality in applying the AutoML framework to the space of auxiliary losses for RL and in, among others, proposing a loss configuration that covers most of the hand-engineered loss functions, as well as being interpretable (the proposed A2-winner seeming like a combination of reward prediction and forward dynamics). \n\nThe method is expensive, requiring tens of thousands of GPU hours, and there is room for improvements in efficiency, which is also remarked by the authors. However, this does not represent a weakness with respect to the work's significance, as important insights can already be drawn from the presented empirical evidence and, further, the results of A2-winner loss seems to generalize to environments on which the search was not performed. \n\nHowever, presenting more information about the inner loop and its comparison with the baselines (hyper-parameters, choosing criteria) could be highly beneficial to leveraging the work in the future and answering questions such as: do these findings apply to value-based, as well as policy-based methods? \n\nOverall, as most aspects are highly positive, the paper meets the acceptance criteria. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept