AARL: Automated Auxiliary Loss for Reinforcement Learning
Authors: Tairan He
Yuge Zhang
Kan Ren
Che Wang
Weinan Zhang
Dongsheng Li
Yuqing Yang
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement learning
Representation learning
Auxiliary Loss

Abstract: A good state representation is crucial to reinforcement learning (RL) while an ideal representation is hard to learn only with signals from the RL objective. Thus, many recent works manually design auxiliary losses to improve sample efficiency and decision performance. However, handcrafted auxiliary losses rely heavily on expert knowledge, and therefore lack scalability and can be suboptimal for boosting RL performance. In this work, we introduce Automated Auxiliary loss for Reinforcement Learning (AARL), a principled approach that automatically searches the optimal auxiliary loss function for RL. Specifically, based on the collected trajectory data, we define a general auxiliary loss space of size $4.6\times10^{19}$ and explore the space with an efficient evolutionary search strategy. We evaluate AARL on the DeepMind Control Suite and show that the searched auxiliary losses have significantly improved RL performance in both pixel-based and state-based settings, with the largest performance gain observed in the most challenging tasks. AARL greatly outperforms state-of-the-art methods and demonstrates strong generalization ability in unseen domains and tasks. We further conduct extensive studies to shed light on the effectiveness of auxiliary losses in RL. 

Under review as a conference paper at ICLR 2022
AARL: AUTOMATED AUXILIARY Loss FOR REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A good state representation is crucial reinforcement learning (RL) while ideal representation is hard to learn only with signals from the RL objective. Thus, many recent works manually design auxiliary losses to improve sample efficiency and decision performance. However, handcrafted auxiliary losses rely heavily on expert knowledge, and therefore lack scalability and can be suboptimal for boost- ing RL performance. In this work, we introduce Automated Auxiliary loss for Re- inforcement I Learning (A ARI., principled approach that automatically searches the optimal auxiliary loss function for RI. Specifically, based on the collected tra- jectory data, we define a general auxiliary loss space of size 4.6 x 101D and explore the space with efficient evolutionary search strategy. We evaluate AARL on the DeepMind Control Suite and show that the searched auxiliary losses have sig- nificantly improved RL performance in both pixel-based and based settings, with the largest performance gain observed in the most challenging tasks. AARL greatly outperforms state-of-the-arr methods and demonstrates strong generaliza- tion ability in unseen domains and tasks. We further conduct extensive studies to shed light on the effectiveness of auxiliary losses in RL.
INTRODUCTION
Reinforcement learning (RL) has been hot research topic and has shown significant progress in many fields (Mnih et al., 2013 Silver al.] 2016); Gu al.) 20177; Vinyals et al.] 2019). Recent RL research focuses on obtaining a good representation of states as is shown be the key to improve sample efficiency of RI. (Laskin et al.] 2020). This because in high-dimensional environ- ments, applying RL directly the complex state inputs is incredibly
(Lake al.) 2017 Kaiser et al. 2019), making RL hard to scale to real-world tasks where interacting with the environment is costly (Dulac-Amold et al.) 2019). Standard RL paradigm learns the representation from critic loss (value prediction) and / or actor loss (maximizing cumulative reward), which hardly extracts informative representations in challenging environments like pixel-based RL and complex robotics systems (Lake et al. 2017 Kober et al. 2013).
This motivates adding auxiliary losses in support of learning better latent representations of states. The usage of auxiliary loss encodes prior knowledge of RL and its environments, and puts regu- larization on training (Shelhamer et 2016). Typical auxiliary losses are manually designed by human experts, including observation reconstruction (Yarats et al. 2019), reward prediction (Jader- berg et al.) 2017) and environment dynamics prediction (Shelhamer et al. 2016 De Bruin et al. 2018 Ota et al.) 2020). However, such auxiliary loss designs and choices rely heavily on human knowledge what might be helpful for RL training, which requires extensive human efforts to scale to new tasks and can be suboptimal for improving RL performance (Yang & Nachum 2021).
Otta
In this paper, we rigorously treat auxiliary loss functions for RI. as a first-class problem to explicitly address the question: what is the universal approach find good auxiliary loss function for RL? Considering that the automated machine learning (AutoML) community has shown promising re- sults with automated loss search in computer vision tasks (Li al. 2019). propose automate the process of designing auxiliary loss functions of RL.
Specifically, we formulate the task as bi-level optimization, where try find the best auxiliary loss function, which, to the most extent, helps train a good RL agent. The inner-level problem is typical RL training problem, while the outer-level can be seen search problem. To tackle

---

Under review as a conference paper at ICLR 2022
Table 1: Existing auxiliary losses in our search space.
Loss inputs
Auxiliary loss
Loss operator
Horizon
Source {8t, (Ots St-1
Target
Forward dynamics
MSE
(8141
Inverse dynamics
Reward prediction
MSF
(
Action inference
MSE
I
{se, St-1)
CURL (Laskin el al.
2020
Bilinear
I
[+c]
[sc]
ATC (Stooke et al.
2021
Billnnar
k
{5t)
{
SPR (Schwarzer el al.
2020
N-MSE
k
{st, Or41s -044-11 1
this, we first design a novel search space of auxiliary loss functions, which combination of two components: loss input and loss operator. As shown in Table I the search space covers many existing handcrafted losses, and it is also substantially large (as large as 4.6 x 10 We then propose an efficient search strategy to explore the space. This is done in two-step manner. We first finalize the loss operator specification and then use an evolutionary strategy that performs mutations on configurations of loss input, identify the top-performing loss inputs quickly.
We evaluate our search framework on both pixel-based and state- baate environments in the Deep- Mind Control suite (Tassa al. 2018). Extensive experiments show that the searched auxiliary loss functions greatly outperform state-of-the-art methods and can easily transfer to unseen environments (that are never used during the search), showing that our proposed method is robust and effective.
We highlight the main contributions of this paper below:
We introduce a principled and universal approach for auxiliary loss design in RL. To the best of our knowledge, we are the first to derive the optimal auxiliary loss function with an automatic process. Our framework can be easily applied to arbitrary RL tasks to search for the best auxiliary loss function.
We demonstrate that AARL significantly outperforms sate--of-thearte methods in both pixel-based and state-based environments. The searched auxiliary loss functions show strong generalization ability to transfer to unseen environments.
We analyze the derived auxiliary loss functions and deliver some insightful discussions that we hope will deepen the understanding of auxiliary losses in RL. We will also open source code- base to facilitate future research.
2 METHODOLOGY
We consider the standard Markov Decision Process (MDP) setting where we specify the state, action and reward at time step (st, Tz). The sequence of interaction data is denoted as (st, Gt, T.. 8t+R), where represents the horizon of this sequence. Suppose we have an RL agent Ma parameterized by и and R(M,; is the agent performance (i.e., cumulative discounted reward) in environment E. The goal of AARL is to find the optimal auxiliary loss function L, such that, when optimized under an arbitrary RL objective function CRL (e.g., actor-critie loss) together with L, the agent achieves the best performance. Formally, we have
max R(Mo-(C);;),
S.L. w (C) = arg CRL (w; + AL(w;
where À is a hyper-parameter that controls the relative weight of the auxiliary task. We solve this bi-level optimization problem with AutoML techniques. The inner optimization is standard RL training procedure. For the outer one, we define finite and discrete search space (Section 2.1 and use variation of evolution strategy to explore the space (Section 2.2 We will explain the details in the rest of this section.
2.1 SEARCH SPACE
An auxiliary loss function can be viewed as a combination of two components: 1) loss input I and 2) loss operator We define search space as shown in Figure where I is pair of binary masks
2

---

Under review as a conference paper at ICLR 2022
Auxilaary
Search Space L(I,
Operator f
Example of forward dynamics prediction (horizon k=1)
Predictor
updant
Encoder
Target Encoder
Binar
Binary
Loss
Figure 1: Computation graph and search space of auxiliary loss functions.
(mi, m) that selects from interaction data to compute ý and and is operator that aggregates selected inputs into an expression the loss function with scalar output.
Loss Input Unlike supervised machine learning tasks with explicit prediction and ground truth for the loss function, auxiliary losses in RL have no ground truths beforehand. Instead, they are generated accordingly upon interaction with the environment itself self-supervised manner. As shown in Figure 1 we take some tokens from the sequence (st, dt, "t,"" .(+*k) with binary mask vector n and feed them into the encoder which maps states to latent representations. The predictor module then tries to predict targets (encoded with momentum encoder 0, as is typically done in recent works), which another few tokens select from the sequence with another binary mask vector m. Details about momentum encoder are given in Appendix A.122 In other words, the search space for loss input pair of binary masks (m, m). each of which up to length (3k + 1) if the length of an interaction data sequence, i.c. horizon, is limited to k steps. In our case, we set the maximum horizon length kanax 10.
Loss Operator We optimize ò to be as similar possible to y. Therefore, we make loss operator cover commonly-used similarity measures, including inner product (Inner) (He et al. 2020 Stooke
al.
202T), bilinear inner product (Bilinear) (Laskin et al. 2020), cosine similarity (Cosine) (Chen
et al. 2020). mean squared error (MSE) (Ota et al. 2020) De Bruin et al. 2018) and normalized mean squared error (N-MSE) (Schwarzer al. 2020p. Some existing works, e.g., contrastive objectives like InfoNCE loss (Oord al. 2018). also incorporate the trick to sample un-paired predictions and targets as negative samples and maximize the distances between them. find this technique applicable to all the loss operators mentioned above and thus incorporate the discriminative version of these operators in our search space.
Search Space Complexity As shown Table many existing manually designed auxiliary losses can naturally fit into our loss space thus are our special cases, which proves that this loss space is reasonable, flexible, and general. However, this is also at the cost that the space is substantially large. In total, the size of the entire space is 10 x E1e 20i+2 =s 4.6 x 10 The detailed derivation can be found in Appendix
2.2 SEARCH STRATEGY
Search Space Pruning Considering that the loss space is extremely large, an effective optimiza- tion strategy is inevitably required. Directly grid-searching over the whole space infeasible be- cause of unacceptable computational cost. Thus some advanced techniques such as space pruning and an elaborate search strategy are necessary. Our search space can be seen as a combination of the space for the input I and the space for the operator f. Inspired by AutoML works (Dai et al.) 2020 YYng et al., 2019) that search for parammeer first and then neural architectures, we ap- proximate the joint search of input and operator in Equation (T) in two-step manner. The optimal
3

---

Under review as a conference paper at ICLR 2022
Table 2: Normalized episodic rewards (mean standard deviation for 5 seeds) of 3 environments used in evolution on pixel- pixel-based based DMControl500K with different loss operators.
operator discrimination
Bilincar
Cosine
MSE
N-MSE
w/ negative samples
0.979 + 0.344
0.963 + 0.321
0.872 + 0.412
0.124 +01.25
0.333 + 0.360
w/o negative samples
0.669 0.311
0.707
0,299
0.959 + 0.225
1.000 + 0.233
0.983 + 0.229
auxiliary loss (I" can be optimized as:
max R(M= = max (Z, v max RMM I,f
where f R arg max Ey
To decide the best loss operator, for every ʃ the loss operator space, we estimate (z, with random sampling strategy. We run 15 trials for each loss operator and calculate the average of their normalized return on three environments. More details are available in Appendix A..13]] Surprisingly, as summarized in Table 2 the simplest MSE without negative samples outperforms all other loss operators with complex designs. Therefore, this loss operator is chosen for the rest of this paper.
Evolution Even with the search space pruning, the rest of the space still remains large. To efficiently identify top candidates from the search space, we adopt an evo-
Population (P=100)
lutionary algorithm (Real et al. 20T9p. The pipeline of
Binary Mook di
Mask
the evolution process is illustrated in Figure 2 In each stage, we first train and evaluate population of candi- dates with size P = 100, where each candidate corre- sponds pair of binary masks (m, m). The candidates then sorted by the approximated area under learn- ing curve (AULC) (Ghiassian et al. 20200, Stadie et al. 2013), which well captures both convergence speed and final performance (Viering & Loog 2021) and reduces the variance of RL evaluation.
Binryy Mook
Binray
Binaay da
Binray
Loss Roj-RRoeecionn
Protocol
Training
Reject
Mutation
Top 25% Candidates
Binary Meak
Binrry
The top-25% candidates are selected after each training stage, and perform four types of mutations the se- lected candidates to form a new population for the next stage: replacement (50% of the population); crossover (20%); horizon decrease and horizon increase (10%), as shown in Figure 3 Moreover, the last 20% of the new population is from random generation. As for each bit of masks, replacement operation flips the given bit with probability
Binray Book da
Binryy Mask
Biirary Mook da
Binry Mask
Figure 2: Overview of the evolution pipeline.
2-(344T). where & is the horizon length. Crossover generates new candidate by randomly combining the mask bits of two candidates with the same horizon length in the population. We also incorporate the knowledge from the existing auxiliary loss designs by bootstrapping the ini- tial population with prior distribution (Co-Reyes al. 2020). so that we can quickly converge a reasonable loss function. More implementation details are available in Appendix
Mask before mutation
Mask After mutation
Horizon decrease
Rest
Changed by Muation
Horizon Increase
Przz
Fu-a
Replacement
Rus
Aps
S
0
Pioz
Tros
Su.a
Figure 3: Mutation operations of aaxiiiaary loss functions.
4

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENTS
We evaluate our method and the obtained auxiliary loss functions on DMControl suite (Tassa al. 2018). DMControl suite is powered by MuJoCo physics engine (Todorov et al. 2012) with various challenging tasks. In both pixel-based (images) and state-based (proprioceptive features) settings, we choose Soft Actor-Critee (SAC) (Haarnoja et al. 2018) as our base RL algorithm since the state-of-the-ar model-free RI. algorithm. Implementation details are given in Appendix A
3.1 PIXEL-BASED RT.
Experiment Settings We follow the same network architecture as CURL with a 4-layer convolu- tional encoder for image input. In the search phase, compare method AARL to SAC, SAC (no aug), and CURL with the same ppper--rrameters reported in Appendix A All methods randomly crop images from 100 100 84 84 data augmentation apart from SAC (no aug). After searching, we test the generalization ability of the obtained auxiliary losses in other unseen environments in comparison with state-of-the-ane model-free and model-based methods.
450
300
800
403
200
300
Mage
Oagee
Cheetah-Run (Pixel)
Reacher-Easy (Pixel)
Walker-Walk (Pixel)
Figure 4: Evolution process in pixel-based environments. Every white dot represents a loss candi- date, and the score of y-axis shows its corresponding approximated AULC score (Ghiassian et al.) 2020 Stadie et al.) 2015). The horizontal lines show the scores of baselines. The AULC score is approximated with the average evaluation score at 100k, 200k, 300k, 400k, 500k time steps.
Meacher-Easy. and Malker-Walk which we call "Iraining environments". We choose these environments because they are more chal- lenging and there might be room for improvement with auxiliary losses. For each environment, we set the total budget of each experiment to 16k GPU hours (on NVIDIA P100) and terminate the experiment when the resource exhausted. The evolution process is demonstrated in Figure 4 where the y-axis shows the AULC of candidates. The results show that, during the evolution pro- cess, we can easily find candidates surpassing baselines in the first stage. the subsequent stages, the overall population score continues to grow, and most candidates can outperform baselines. This demonstrates the effectiveness of our search space and the efficiency of our search strategy.
Search Results We apply search algorithm Cheetah-Runs
Generalize to Other Environments To reduce the risk that auxiliary losses overfit to one par- ticular environment and avoid selecting candidates that perform well only due to randomness, run cross-validation on the three searched environments to decide the best candidate. The details of this process can be found in the Appendix A.4 After cross-validation, loss function, which we call "AAI"AARR--AA AARL--Piee (all the top candidates during evolution are reported in Appendix D). and we train agents with the obtained auxiliary loss compare with state-of-the-a model- free and model-based methods on DMControl100k and DMControl5001s We select these environments because they are common benchmarks for state-of ootee---------- algorithms of pixel-based RL. (Laskin et al.) (020). Note these comparisons are additionally made on three unseen environ- ments, that are entirely not accessible during evolution, to test the generalizability to ensure our comparisons are fair. The results are summarized in Table 3 where AARL greatly outperforms state-of-ther Reacher-Eass and Ball in cup-Catch
finalize auxiliary
our method and
methods on out 12 benchmark settings. Note that Finger-Spin, environments that never used during the search, while AARL still performs very well on them. This result implies that AARL-Pixel is robust and fective auxiliary loss that is potentially helpful RL under various settings. The performance gain
5

---

Under review as a conference paper at ICLR 2022
Table 3: Episodic rewards (mean & standard deviation for 10 seeds) on DMControll00K and DM- Control500K with pixel inputs. Note that the optimal score of DMControl is 1000 for all environ-
ments. The baselines methods are PlaNet (Hafner al. 2019b), Dreamer (Hafner et al.. 2 SAC+AE (Yarats et al.) 2019), SLAC (Lee et al. 2019), pixel-based SAC (Haarnoja et al. 2018 Performance values of all baselines are referred to Laskin et al. (2020), except for Pixel SAC. We
also show learning curves of all 12 DMC environments in Appendix B.
500K Steps Scores
AARI
CURLE
PlaNet
Dreamer
SAC+AFE
SLACCI
Pixel SAC
Singer-Spin"
983 4
926 45
561 4 284
796 + 183
884 128
673 +
282 102
Carpalo-Swingup
864 +
841 + 45
476 + 71
702 + 27
736 + 63
344 +: 104
Reacher-Easy"
938
518 14
210 390
793 161
627
132
Cheetah- Run!
613 39
305 + 131
570 253
550 + 34
640 + 19
99 + 28
Walker Walk!
917 18
902 43
351 5.58
897 1. 49
847 48
842 1.
76 44
Ball in Catch"
970
958 27
160 4: 350
897 1 87
794 =:
852 :
200
114
100K Steps Scores
Finger-Spin"
872 + 27
767
+ 56
136 + 216
341 + 70
740 + 64
693 141
160 + 138
Carpolee "wingupp
815 66
582
146
1.39
326
27
243
Reacher-Faage Fasy!
778 164
538
223
20 + 50
314 + 155
274 + 14
277 69
Cheetah Run!
449 34
299
1 18
138 1. 88
235 137
267
1. 24
319 1. 56
128 12
Walker !
151
277
391
361
127
Ball in cup-Catch'
862
167
769 + 43
0 0+ 0
246
+ 174
391
+ 82
512 110
100 = 90
Training environments. Unseen environments, 3) Results reported in Laskin et.l.. 2020
is obvious in DMControl100K, where AARL-Pixel (pixel-based input) has shown great sample efficiency.
3.2 STATE-BASED RL
Experiment Settings As for state-based RL, since the state is low-dimensional, simply use 1 1-layer densely connected MLP as the state encoder as shown in Figure(5] So, for this setting, we focus on this simple encoder structure. Additional ablations on state encoder architectures are given in Section 3.4 In the search phase, we compare AARL to SAC-Identity, SAC-DenseMLPP, CURL-DenseMLI To ensure fair comparison, all SAC related hyper-parameters are the same those reported in the CURL paper. Details can be found in Appendix [A.2.2] SAC-Identity vanilla SAC with no state encoder, while the other three methods (AARL, SAC-DenseMLP, CURL-DenseMLPP use the same encoder architecture. Different from the pixel-based setting, there is no data augmentation in the state- based setting. Note that many environments that are challenging in pixel-based settings become easy to tackle with state-based inputs. Therefore we apply our search framework to more challenging environments for state-based RL, including Cheetah- Run, Hlpper-HHop and Quadruped-Rund -un..
Embedding
Concat
MLP Encoder
Figure 5: Net- work architec- of 1-layer DenseMLP state encoder.
Search Results Similar to pixel-based settings, we approximate AULC with the average score agents achieve at 300k, 600k, 900k, 1200k and 1500k time steps"] For each environment, we early stop the experiment when the budget of 1,500 GPU hours is exhausted. The evolution process is shown in Figure 66 where find large portion of candidates outperform baselines (horizontal dashed lines). The performance improvement is especially significant on Cheetah-Run, where al- most all candidates in the population greatly outperform all baselines by the end of the first stage.
Generalize Other Environments Similar to pixel-based settings, we also use cross-validation to select the best loss functionn which we call "AARL-Sttate" here (all the top candidates during evolution are reported in Appendix D). and run thorough evaluation on all 18 environments. The results are summarized in Table 4 AARL-Statt again brings significant performance gain, achiev- ing much stronger sample efficiency than SAC. It is noteworthy that AARL-State is able to outper- form baseline methods in 16 out of 18 environments (with 15 unseen environments). These results show that even for tasks with lower-dimensional state space, there is still a huge potential for im- provement with better auxiliary objectives. Moreover, this performance gain is especially significant
As for Cheetah-Run still use average score agents achieve 100k, 200k, 300k, 400K and 500k time steps since agents converge close optimal score within 500k time steps.

---

Under review as a conference paper at ICLR 2022
1001
46E
1.00
480
540 54C CaILOOETNN
Cheetah-Run (Start)
Hopper-Hop (Starie)
Quadruped-Run (Starie
Figure 6: Evolution process in state-based environments. Every white dot represents a loss candi- date, and the score of y-axis shows its corresponding approximated AULC score. The horizontal lines show the scores of baselines. The AULC score is approximated with the average evaluation score at 300k, 600k, 900k, 1200k, 1500k time steps (Cheetah- Run at 100k, 200k, 300k, 400K).
Table 4: Episodic rewards (mean & standard deviation for 10 seeds) on DMControl100K (easy tasks) and DMControl 100K (difficult tasks) with state inputs. SAC-Identity has no state encoder while A ARI,, SAC and CURL use the same state encoder architecture. All four variants here use the same syper-parameters..
100K Steps
AARL-Soa Statt
SAC-ldenity
SAC
CURL
Finger Spin"
837 1: 53
805 52
785 105
712 1::3
Finger Tum hand"
218 1::1111 117
347 3: 1150
174
45:4 42
Steps
AARI-SSute
SAC-Mlenity
SAC
Cantelle-Swngipp
877
873 10
366 0:7
554 17
Canpate Swingup spune"
605 147
453 3: 3399
627 307
445 1: 196
Quadruped Run'
838 1: 58
345 4: 157
:: 148
897 128
Reacher Easy*
934 38
097 1: 199
574 1: 57
149 1: 183
Pandolum-Swingg Swingup"
579 + 410
506 + 374
539 + 301
363
366
Chectth-Ruu!
472 + 30
233 27
172 79
190- 9:33
Hopper- Hop"
278 106
121 5:51
134 =
00 22
Wolker- Sund"
948
940 10
862 196
067 104
Humanoid- Stand"
286 15
9:1-2
7+
7
Waiser Walk"
905
973
925 12
952 1.64
Humanoid-Waan Walk"
299 55
16 10+-30
2++0
2++00
Walker-RRnn
559
43
289 6.61
Humanoid Run"
88
110
110
14.0
cap-C
941
Fish Upsigat"
Training
Unseen
Hopper Stand"
311
16
t: Truinin covinamenss, Usseen costronments.
in complex environments like Humanoid, where SAC barely learns anything 1000K time steps, while AARL-State is able to achieve much better performance.
3.3 ANALYSIS OF AUXILIARY LOss FUNCTIONS
We have collected large number of data from the evolution process far. To make the best use of these data, we now conduct a comprehensive analysis to investigate statistical relations between patterns of auxiliary loss functions and RL performance.
Typical Patterns We first analyze how RL performance affected by typical patterns of auxil- iary loss functions. Define the set of the input to the encoder that is being trained as "source", and the set of prediction targets as "target". We say an auxiliary loss candidate has a certain pattern if the pattern's 's source is a subset of the candidate's source, and the pattern's 's target is a subset of the candidate's 's target. For instance, a loss candidate of {st, at} - {st41, 2 has the pattern {st, ) {st+1}. and does not have the pattern {az, St+1] + {st}. The patterns
consider
{8411), inverse dynamics {at, $t+1} + {st], reward pre- diction {st, a.} -> {re), action inference {st, $t-1} {a,) and state reconstruction in the latent space {5r} - }. For each of these patterns, we divide auxiliary loss candidates into two classes: 1) with this pattern 2) without this pattern. We then calculate the average performance of these two classes and compare their performances to conclude whether this pattern is helpful for RL. The results are summarized in Table 5 Here a positive number indicates that this pattern is beneficial, and if the performance gain is statistically significant, the number is marked with the asterisk, in- dicating very likely to helpful. negative number indicates that this pattern is detrimental and correlated to worse performance. We highlight some interesting observations as follows. 1) For- ward dynamics helpful most environments, and improves the RL performance on Reacher-Easy (Pixel) and Cheetah- Run (State) significantly (i.e., p-value-<0.05) 2) State reconstruction in the la- tent space is able to improve RL performance in pixel-based environments while it has significant negative effects state-based environments. We hypothesize that this is because in the pixel-based based setting, data augmentation encourages the encoder to learn augmentation invariant representations, while in the state-hased setting, augmentation is used, and thus the encoder learns useful rep-
include forward dynamics {st, {st,ae}
7

---

Under review as a conference paper at ICLR 2022
resentations. This also explains why CURL is underperforming in state-based experiments. Note that not all typical patterns bring performance gains. Some can even be very detrimental, and this shows that commonly auxiliary loss patterns do not work well in the state-based setting, further highlighting the fact that great research potential lies in this setting.
Table 5: Statistical analysis on auxiliary loss functions. The number reported is the difference of the mean score of two classes of auxiliary losses during evolution, and we report its corresponding p-value from the t-test.
Typical patterns (w/ - woo)
Forward dynamics
Inverse dynamics
Reward prediction
Action inference
State reconstruction
Cheetah-Run (Pixel)
+1.28
-3.51
-31. 16
-75.95"*
+12.411*
Reacher-Eary (Pisel)
+288
+8.36
+37.80
+3.35
+70.72**
Walker-Walk (Pixel)
+22.20
-28,59**
-8.11
+29.86"
+13.93
Cheetah (State)
+94 18
-23.66"
333.28
100.33"
-50.15
Hopper-Hop (State)
+15.50
-16.47*
-11,30
-32 12"100"
-25.67*
Quadruped- Run (Stale)
-28.07
-18.19
-114.23
-10.337*
-82.06"
p-valoe 0.05 p-value 0.01
Source larget
State,
Action
Reward,
Cheetah (Pixel)
80.00)
13.62
Reacher-Eau Easy (Pisel)
+1.98
-2.722
+65. 65..66"
Walker-Walk Walk (Pixcl)
+73.50
22"
-1
Cheetah- Run (Slate)
+188.00 06
-102 62"
-93.94
Hopper-Hop (State)
+19.80
-29.70 70
-5.03
Quadruped-He Run (State)
475.17
-4.31
-46.60
---value 0.05.
p-value < 0.01
Number of Sources and Targets We then investigate whether it is more beneficial to use a small number of sources to predict a large number of targets (Itargett > Ysmurrn: e.g., use to predict 8141, 8t+2; 8t+3), or the other way around (Ntarget < DIsources e-g., use 8t: 8,41, to predict 8143). The statistical results are shown in Table 5 where we find that auxiliary losses with more states on the target side have a significant advantage over losses with more states on the source side. This result echoes recent works (Stooke al.) [202 [Schwarzer et al. 2020) which show that predicting more states leads to strong performance gains.
3.4 ADDITIONAL ABLATION STUDY
Search Space Pruning As introduced in Section 2.2 we decompose the full search space into loss operator and loss inputs. Here we try to directly apply evolution strategy the whole space without the pruning step. The comparison results are shown Figure 7 We can see that pruning improves the evolution process, making it easier to find good candidates.
Encoder Architecture for State-based RL As shown in Figure 5 we choose I-layer densely connected MLP as the state encoder for state-based RL. Here, we con- duct an ablation study on different encoder architectures in the state-based setting. The results are summarized in Table [6 where AARI. with 4-layer encoders consis- tently perform worse than 1-layer encoders. We also note that dense connection is helpful in the state-based setting compared with naive MLP encoders.
Table 6: Normalized episodic rewards of AARL (mean & standard deviation for seeds environments) on state- based DMControl with different encoder architectures.
CURL-MLP(Illaaee 0.919 0.217
AARL-MLP (4-laer) 0,544 O. 0.360
AARL DenseMLP (I-la()(((arrr 1,000 0.129
0.813 0.218
22.5
wis pruting aipruringg
WIS prUTing praning
WD praning sruniigg
10.5
Score
Score
Score
Stage
Stage
Stage 3
Figure 7: Comparison of evolution with and without pruning by performance histogram.
8

---

Under review as a conference paper at ICLR 2022
4
RELATED WORK
4.1 REINFORCEMENT LEARNING WITH AUXILIARY LOSSES
Using auxiliary tasks to improve sample efficiency of RL, especially on pixel-based control tasks, has been explored in many recent works. A number of manually designed auxiliary objectives are shown to boost RL performance. including observation reconstruction Yaratt et al., 2019), reward prediction (Jadebbrg et al. (017). dynamics prediction (De Bruin et al. (218)) and contrastive learn- ing objectives (Laskin et al. 22200] Schwarzer et al. [020 Stooke al. 2022 It is noteworthy that these works mainly focus on pixel- based settings, while only a limited number of works study the state-based setting (Munk al.) 2016; Ota et al., (020). Although might seem that the state-based setting benefits less from auxiliary tasks due to their lower-dimensional state space, we show that there is in fact huge potential of improving state-based RL performance with auxiliary objectives.
Compared to the previous works, we point out two major advantages of our approach. 1) Instead of handcrafting an auxiliary loss with expert knowledge, AARL automatically searches for the best auxiliary loss, relieving researchers from such tedious work. 2) AARL is a principled approach that can be used in arbitrary RL settings. In both pixel- and the rarely studied state- state-based settings, we discover good auxiliary losses that bring significant performance improvement.
4.2 AUTOMATED REINFORCEMENT LEARNING
is well known that RL training is sensitive to pppee--rraeetess and environment changes (Hen- derson et al. 2018). Thus many works have attempted to use techniques in AutoML to alleviate human intervention. Exiting works focus on apper-paraette optimization (Espeholt et al.) 2018 Paul al., [019 Xu et al.] 2020, Zahavy al., 2020), reward search (Faust et al.] [209]] Veeriah et al.) 2019) and network architecture search (Runge et al.] 2019) Franke et al.) (021). ). In contrast, our method aims to search for auxiliary loss functions that generalize across different environments.
4.3 AUTOMATED LOSS DESIGN
A few recent works in the AutoML community have been automating the design of good loss func- tions that outperform traditionally handcrafted ones. Specifically, AM-LFS (Li et al. 2019) defines the loss function search space as a parameterized probability distribution of softmax loss hyper- parameters. AutoLoss--7en (Li et al. 202T) proposes to search loss functions with primitive math- ematical operators. These methodologies are proposed specifically for computer vision tasks.
For RL, existing works focus on searching for a better RL objective, EPG (Houthooft et al. 2018 and MetaGenRL (Kirsch et al. (2020) define the search space of loss functions parameters complexity neural network. Co-Reyes et al. (2020) defines the search space of RL loss functions as directed acyclic graph and discovers two DiN-like regularized RL losses. Note that none of these works investigate auxiliary loss functions, which are crucial to facilitate representation learning in RL and to make RL successful in highly complex environments. To the best of our knowledge, our work is the first attempt to search for auxiliary loss functions and greatly improve RL performance.
low
CONCLUSION AND FUTURE WORK
We present A AARL, a principled and universal approach for automated auxiliary loss design for RL. With this framework. we discover highly performant auxiliary loss functions that generalize beyond the training environments for both pixel-based and state-based settings. We present extensive experimental results that provide strong empirical evidence for the effectiveness of our method, and also conduct an in-depth investigation of the statistical relations between auxiliary loss patterns and RL performance. We hope our studies provide insights that will deepen the understanding of auxiliary losses in RL, and shed light on how to make RL more efficient and practical. One interesting future work direction is to further investigate the relationship between auxiliary loss patterns and RL performance, and to better understand how and why auxiliary losses can help to improve RL performance. Another direction is to study how to combine the RL objective and the auxiliary loss functions in more effective manner.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
All implementation details are introduced in the main text and appendix (Appendix A We will open source our codebase facilitate future research.
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
John D Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Quoc V Le, Sergey Levine, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Con- 2020.. 2020.
ference on Learning Representations,
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, and Joseph E. Gonzalez. Fbnetv3: Joint architecture-recipe search using predictor pretraining, 2020.
Tim De Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Integrating state representation learning into deep reinforcement learning. IEEE Robotics and Automation Letters, 3(3):1394- 1401, 2018.
Gabriel Dulac-Amold Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: 19t.. 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et Impala: Scalable distributed deep-rl with im- portance weighted actor- letoreeearer architectures. In International Conference Machine Learning, pp. 1407-1416. PMLR, 2018.
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards automate reinforcement learning. arXiv preprint 105.02290..07688, 2019.
Jörg K. H. Franke, Gregor Köhler, André Biedenkapp. and Frank Hutter. Sample-effiiient au- tomated deep reinforcement learning. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://ppnnne
net wwtp:/eeeeeeettteeeeeeeeeeeeeeee
Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. arXiv preprint arXiv:2003.07417, 2020.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-polic updates. In 2017 IEEE international confer- ence on robotics and automation (ICRA), pp. 3389-3396. IEEE, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint 1xXiv:/8/2.05000 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint 1rXiv:1912.01603 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International Conference on Machine Learning, pp. 2555-2565. PMLR, 2019b.
Kaiming He. Haoqi Fan, Yuxin WU, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
10

---

Under review as a conference paper at ICLR 2022
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly C. Stadie, Filip Wolski, Jonathan Ho, and Picter Abbeel. Evolved policy gradients. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- Bianchi, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp- 5405-5414, 2018. URL
wew.iiinnnsseuuugneee neur ips wwtppppppp.pceeccc saaper/2C1B/hat.. Abstract
(177acb666.00add11eeelll11111111 Abstssataa.t--otc.t...c......... html
Max Jaderberg. Volodymyr Mnih, Wojciech Marian Czamecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. peet/forunt 2o2uum?id=sJJyP5xg
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model-based reinforcement learning for atari. arXiv preprint arXiv:/903
2019.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net htt.//ormmmm/ id= dee levHerYPr
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in roboties: A survey. The International Journal of Robotics Research, 32(1 12111:123-11 2013.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR. 2020.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprini arXiv: 9997000953 2019.
Chuming Li, Xin Yuan, Chen Lin, Minghao Guo, Wei Wu, Junjie Yan, and Wanli Ouyang. Am-lfs: Automl for loss function search. In Proceedings of the IEEF/CVF International Conference on Computer Vision pp. 8410-84-8 8419, 2019.
Hao Li, Tianwen Fu, Jifeng Dai, Hongsheng Li, Gao Huang, and Xizhou Zhu. Autoloss-zeri zero: Searching loss functions from scratch for generic tasks. arXiv preprint arXiv:2103.14026, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.566 2013.
Jelle Munk, Jens Kober, and Robert Babuška. Learning state representation for deep actor-critic control. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 4667-4673. IEEE, 2016.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018
Kei Ota, Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforeement learning? In International Conference on Ma- chine Learning, pp. 7424-7433 PMLR, 2020.
11

---

Under review as a conference paper at ICLR 2022
Supratik Paul, Vitaly Kurin, and Shimon Whiteson. Fast efficient hyperparameter tuning for policy gradients. arXiv preprint arXiv:1902.0653 2019.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, 33: 4780-4789, Jul 2019. ISSN 2159-5399. doi: 10.1609/aaa http///aa.ii331..3333. (33101.3301478. URL http:
// //xx. doi. org/ 10. 1 1009/6000
33314113331478800 33014780
Frederic Runge, Danny Stoll, Stefan Falkner, and Frank Hutter. Learning to design RNA. In 7th International Conference on Leurning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL htttps://openrevee ew net www.mmrrrrmm//omm
ByfyHh05tQ
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2020.
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self- supervision for reinforcement learning. arXiv preprint arXiv: 161 2016.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 2016.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1rXiv:1507.00814.. 2015.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrranq, al. Deepmind control suite. arXiv preprint arXiv:1801. 00690, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Jun- hyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions as auxiliary tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buu,, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 9306-9317, 2019, URL https::/prroeeeinnggss. neurse cc cctppaaer//11///sss... .htrlltttttmlll..
Tom Viering and Marco Loog. The shape of learning curves: a review. arXiv preprint 1rX1::2103.10948 2021.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)3350-3555 2019.
Zhongwen Xu, Hado Philip van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:///////////////ooooooooo... 1el35555555552ce000000aaaa222232
12

---

Under review as a conference paper at ICLR 2022
Mengjiao Yang and Ofir Nachum. Representation matters: Offline pretraining for sequential deci- sion making, arXiv preprint arXiv:2102.05815, 2021.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:1910.01741, 2019.
Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, and Frank Hutter. Nas- bench-101: Towards reproducible neural architecture search, 2019.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Has- selt, David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:::/pproceeeiingss neurips. /c/ppper/2220/hastt E5220000000e5077f5555cba-a--ttt. html
13

---

Under review as a conference paper at ICLR 2022
A
IMPLEMENTATION DETAILS
A.1 ARCHITECTURE AND SEARCH SPACE
A.1.1 STATE ENCODER ARCHITECTURES
We demonstrate in Figure the overall architecture where our auxiliary loss is used. The architecture
is generally identical to those adopted in CURL (Laskin el al. 2020). "Pixel-based" and "State- based" are the architectures we used in our experiments. "MLP" P" and "4-layer DenseMLP" are for ablations.
State Erbeading
Sxeoil
Adter
Ontic
Acter
Crtc
Adar
Gridc
Driteciin
Cancet
Sura Enowediig
Cancat
Stas Erbedding
Donce:
Conool
Cony Eacodar
HRP Sacodar
HLP Escoca
NLR Enoader
Cennal
Encoder
(MIL))
(1-Layer DenseMLP))
State-Based (4-Layer DenseMLLP
Figure Network structures pixel-based RL and state-based RL with auxiliary losses.
A.1.2 SIAMESE NETWORK
For fair comparison with baseline methods, we follow the same Siamese networks structure for representation learning as CURL (Laskin et al. (020). As shown in Figure [ when computing targets y for auxiliary losses, we map states to state embeddings with a target encoder. We stop gradients from target encoder 0 and update 0 in the exponential moving averaged (EMA) manner where 6' r (1 r)0. This step, i.e., to freeze the gradients of the target encoder, is necessary when the loss is computed without negative samples. Otherwise, encoders will collapse to generating the same representation for any input. We have verified this in our early experiments.
A.1.3 LOSS OPERATORS
Instance Discrimination
Our implementation is based on InfoNCE loss (Oord et al. 2018):
exp(ol0. v+))
L= log
exp(ø(ū.v+)) Σ exp(o(ú.
The instance discrimination loss be interpreted the log-loss of K-way softmax classifier whose label is y+- The difference between discrimination based loss operators lies in the discrim- ination objective & used to measure agreement between (ý, y) pairs. Inner employs inner product ő(0. y) = ğ while Bilinear employs bilinear product @(8. y) ÿWy. where W is a learnable parameter matrix. Cosine uses cosine distance Φ(ÿ, v) for cross entropy based loss without negative samples, we only take diagonal elements for matrix M where Maj Φ(ū yj) for cross entropy calculation.
9 |ला for further matrix calculation. As
Mean Squared Error
The implementation of MSE-based loss operators are straightforward. 24
MSE loss operator = (ý )2 while normalized MSE |
11.. ) When combined with
negative samples, MSE loss operator (with negative pairs) = (0 y+ (g w- while normal- ized MSE (with negative pairs)
)
1
l
14

---

Under review as a conference paper at ICLR 2022
Evaluation of Performance Expectation To reduce the large search space consists of loss input and loss operator f. we run 15 trials for each loss operator to get an estimation of performance expectation. For each of 10 possible f in the search space (5 operators with optional negative samples), we run 5 trials on each of the 3 pixel-based environments (used in evolution) with the same found that forward dynamics is reasonable representative of
loss inputs (+) + (8++1) ), our search space with highly competitive performance.
A.2 TRAINING DETAILS
A.2.1 HYPER- PARAMTTRRS IN THE PIXEI. BASED SETTING
We use the same hyper-panee parameters for AARL, SAC (no aug), SAC and CURL during the search phase, to ensure perfectly fair comparison. When evaluating the searched auxiliary loss, we use slightly larger setting (e.g., larger batch size) to train RI. agents sufficiently. A full list is shown in Table
Table 7: Hyper-parameters used in pixel-based environments.
Hyperparameter Random crop
During Evolution
Final Evaluation of AARL-Pixel True
False for SAC (no aug);
True for others
Observation rendering
(84, 84) for SAC (no aug);
(100, 100)
(100, 100) for others
Observation downsampling
(84, 84)
(84, 84)
Replay buffer
100000
100000
Initial steps
1000
1000
Stacked frames
3
3
Actoin repeat
4 (Cheetah- Run, Reacher-Fasy) 2 (Walker- Walk):
(Cartpole-Swingup): (Others)
2 (Walker Walk, Finger-Spin) 1024
Hidden units (MLP)
1024
Hidden units (Predictor MLP)
256
256
Evaluation episodes
10
10
Optimizer
Adam
Adam
(0,,9g) for actor/criticicneoder
9999
(9,9999)
(3), (8)) for cntropy
(99)
.999
Learning rate for actor/critic
1o-3
2c-4 (Chectah-Run)
1e-3 (Others)
Learning rate for encoder
1c-3
3e-3 (Cheetab-Run. Finger-Spin, Walker-Walk); 1c-3 (Others) lc-4
Learning for a
lc-4
Batch size for RL loss Batch size for auxiliary loss
128
512 128 (Walker-Walk) 256 (Cheetah- Finger-Spin)
128
512 (Others); 0.01
Q function EMA Critic target update freq Convolutional layers
0.01
2
2
Number of filters Konn-iin Encoder EMA + Latent dimension
32
32
ReI.U
ReI.U
0.05
0.05
50
50
Discount
.99
.99
Initial lemperature
0.1
0.1
A.2.2 HYPER-PARAMETERS IN THE STATE-BASED SETTING
We use the same hyper- parammerseeetess for AARL, SAC-Identity, SAC-DenseMLP and CURL- DenseMLP, shown in Table 8 As training in state-baser environments is substantially faster than pixel-hased environments, there no need to balance training cost and agent performance. We use this setting for both the search phase and the final evaluation phase.
A.3 EVOLUTION STRATEGY
Horizon-changing Mutations There are two kinds of mutations that are able to change hori- zon length. One is to decrease horizon length. Specifically, we remove the last time step, i.e.,
15

---

Under review as a conference paper at ICLR 2022
Table 8: Hyper- paramerrasseeesss used in state-based environments.
Replay buffer size
100000
Initial steps
1000
Action repeat
4
Hidden units (MLP)
1024
Hidden units (Predictor MIP)
256
Evaluation episodes
10
Optimizer
Adam
(81, 32) for actor/critic/encoder
(.9, .999)
(3r, B2) for entropy
(.5, .999)
Learning rate for actor/critic/cncodeer
2c-4 (Chectah-Runj,
lc-3 (Others) Ic-4
Learning for a Batch size
512
function EMA
0.01
Critic target update freq
2
DenscMLP Layers
1
Non-lincarity
ReLU
Encoder EMA
0.05
Latent dimension of DenseMLP
40
Discount Initial temperature
.99
0.1
(a++k, Te*k; 81+++1) if the target horizon length is k. The other is to increase horizon length, in which we append three randomly generated bits to the given masks the end. We do not shorten the horizon when it becomes too small (less than 1), or lengthen the horizon when it is too long (exceeding 10).
Mutating Source and Target Masks When mutating candidate, the mutation on the source and the target masks are independent to each other except for horizon change mutation where two masks could either both increase horizon decrease horizon.
Loss-rejection Protocol The loss-rejection protocol makes sure that invalid loss functions do not go into the expensive computation of RL training. Concretely, the following conditions must be satisfied to make valid loss function: 1) having least one state embedding vi to make sure gradient of auxiliary loss backward propagates to the state encoder, and 2) target is not empty. If a loss rejected, we repeat the mutation to fill up the population.
Initialization At each initialization, we randomly generate 100 auxiliary loss functions (every bit of masks are generated from Bernoullipp) where p = 0.5.) and generate 25 auxiliary loss functions with prior probability, which makes the auxiliary loss have some features like forward dynamics prediction or reward prediction. The prior probability for generating forward dynamics pattern is: 1) every bit of states from target is generated from Bernoull(p) where actions from source is generated from Bernoullipp) where = 0.8; 3) every bit of states from target generated by flipping the states of source; 4) The other bits generated from Bernoulli(p) where 0.5. The prior probability for generating reward prediction pattern is: 1) every bit of rewards from target is generated from Bernoullipp) where 0.8; Every bit of states and actions from target are 0; 3) The other bits are generated from Bernoulli(p) where 0.5.
0.2; 2) every bit of
A.4 CROSS VALIDATION
After the evolutionary search is done, we choose top-5 candidates for cross- tross-validation across envi- ronments. This step is to avoid overfitting to one single environment. For each top-5 loss candidate. we rum 3 trials on each of these training environments (45 runs in total). Then we sort these top-5 candidates by cross-valls validation overall performance (mean AULC score three environments) and retrieve the top-1 candidate as our final searched loss. We do this in pixel-based environments and state-hased environments separately. For basel RI,, the top-5 candidates the top candidates
16

---

Under review as a conference paper at ICLR 2022
from stage of Cheetah-Run (Pixel). For state-based RL, the top-5 are the top candidates from stage and 5 of Cheetah- (State).
A.5 BASELINES IMPLEMENTATION
Pixel-based Setting CURL (Lskiin al. 2020 is the main baseline compare with in the txxel-baeed setting, which is considered to be the state-of-the-art pixel-based RL algorithm. CURL learns state representations with a contrastive auxiliary loss. PlaNet (Hafner al., 2019b) and Dreamer (Hafner et al. (019a) are model-based methods that generate synthetic rollouts with learned world model. SAC+AE (Yarats et al. 2019) uses reconstruction auxiliary loss of images to boost RL training. SLAC (Lee et 2019) leverages forward dynamics to construct a latent space for RL agents. Note that there are two versions of SLAC with different gradient Updates per agent step: SLACv1 (1:1) and SLACv2(3:1), We adopt SLACv1 for comparison since all methods only make one gradient update per agent step. Pixel SAC are just vanilla SAC (Haarnoja et al. 2018) agents with images inputs respectively.
State-based Setting As for the state-based setting, we compare AARL-State with SAC-Identity, SAC and CURL. SAC-Identity is the vanilla state-based SAC where states are directly fed to ac- tor/critic networks. SAC and CURL use the same architecture of 1-layer densely connected MLP as state encoder. Note that both AARL and baseline methods use the same hyper-parameter reported in Table A without additional hyper-parameter tuning.
B FURTHER EXPERIMENT RESULTS
B. LEARNING CURVES FOR AARL ON PIXEL-BASED DMCONTROL
We benchmark the performance of AARL to the best-performing pixel-based baseline (CURL). As shown in Figure the sample efficiency of AARL outperforms CURL on 10 out of 12 environments. Note that the learning curves of CURL may not match to the data in Table 3 this is because we use the data reported in CURL paper for tabular while we rerun CURL for learning curves plotting, where we find the performance of our rerunning CURL is sightly below the CURL paper.
3000
ECC
202
DUNL
437
AdR Fan
255
ANRL Plac
AMRL Flac
400, Fac
1166
2936
Stast
1024
2004
2886
2004
20.6
DAA
M4k
2024
300k 20.5
Ball cup-Catch
Certpcle-Swingup
Cartpale-Swingup sparse
Cheetah-Run
3002
200
QURL
Addiu aos
600
600
52.11
100
CURL
200
200
1824
203c
1230c
1300k
193k
2034
300c
430c
320c
steps
Finger Spin
Finger-Spin hard
Hopper-Hop
Hoopeer-Stnd
2000
CUN
1000
100
000
500
600
200
200
200
AMLL-ool
0
3006
5006
Do0K
3200s
J500e
190x
3008
4006
5936
100k
2006
2008
400%
500e
2004
2008
3034
4008
5000
Swnduuuu-Swiiggup
Easy
WalkerStand
Walker-Walk
Figure 9: Learning curves AARL-Pixel and CURL on 12 DMC environments. Shadow represents the standard deviation over five random seeds. The curves uniformly smoothed for visual display. The y-axis represents episodic reward and x-axis represents interaction steps.
17

---

Under review as a conference paper at ICLR 2022
B.2 EFFECTIVENESS AULC SCORES
To illustrate intuitively why we use the area under learning curve instead of other metrics, we select top-10 10 candidates with different evolution metrics. Iodeonsstsssss the usage of AULC score could well balance both sample efficiency and final performance. The learning curves of the top-10 10 candidates selected by AULC score look better than the other two metries (that select top candidates simply with 100k step score or 500k step score).
500
400
Score
300
200
100k
100
500k
AULC
100k
200k
300k
400k
500k
Steps
Figure 10: Learning curves of top-10 10 loss candidates selected with different metrics.
B.3 HISTOGRAM AUXILIARY Loss ANALYSIS
The histogram of each pattern analysis is shown in Figure
C SEARCH SPACE COMPLEXITY ANALYSIS
The search space size is the size of loss input space multiplied by the size of loss operator space.
For loss input, calculate separately for each possible horizon length When length k the interaction sequence length (st, ".. 23k-1 different options. There are also 23k+1 distinet binary mask m select targets. Therefore, there are 26k+2 combinations when horizon length is fixed to k. As our maximum horizon is 10, we enumerate from 1 to 10, resulting in ا 26i+2.
S(++k) ""..""..."""""""""""..." has length (3k + 1). For binary mask m, there are
For loss operator, we can learn intuitively from Table 2 that there are 5 different similarity measures with without negative samples, resulting in 5 2 10 different loss operators.
In total, the size of the entire space is
10
2 26+2
4.6 x 19
18

---

Under review as a conference paper at ICLR 2022
Cheetal-RRun (Piwal)
Reacher-Eaey (Paee))
Wakkr-WWik (Pixe0)
Cheelah-RRn (Stane)
Hooger-Hop Hop (State)
Quadruped- Fun (Slate)
(a) Forward dynamics
Cheetah-Run (Pive))
Bpacche-Easy (Piwel)
WalkerWaak IP (Piee))
Chectah-RRn (State)
Hooper-Htop (State)
Quadruped-Pun (State)
(b) Inverse dynamics
Cheetah Hun (Pwee)
Reacher Easy (Phool)
Walker Walk (Poel)
Cheotah Run (State)
Hoapan-Hop (Stato)
Quadruped- FRun (Stano)
(c) Reward prediction
Chestah-Run (Flool)
Reacher-Eady (Pixol)
Walker-Wall (Pixal)
Cheetsh-Run (Stato)
Hooper-Hop (Stato)
Quadrupad- Run (State)
Action inference
Cheetal-Fun (Pisel)
Reacher Essy (Powl))
Walker-Wak (Pixel)
Cheelsh-Run (Statee
Hooper-Hop (Starle)
Quedruped Run (Statie)
(c) State reconstruction
(Ptce)
Easy (Poell)
Walke-Wak (Picet)
Cheetsh-Run (State)
Hoopen-Hop (State)
Quadruped (State)
(I) State source & target
10.
Cheetah-Run (Pooel)
Ressche-Easy (Pixel)
(Pixel)
(State)
Hooper-Hop (State)
Qusdruped-Fuu (State)
(g) Action source & target
Run (Ptea))
Easy (Pxet)
Walker (Pbxal)
Run (Stato)
HoopenHHop (State)
Quadruped (Stato)
(h) Reward source & target
Figure 11: Histogram of statistical analysis of auxiliary loss candidates in 6 evolution process. The x-axis represents approximated AULC score while the y-axis represents the percentage of corre- sponding bin of population. Best viewed in color.
19

---

Under review as a conference paper at ICLR 2022
D TOP-PERFORMING AUXILIARY LOSSES
D.1 AARL-PIXEL. AND AARL-STATE
The auxiliary loss of AARL-Pixel is:
{st-1, 0++1, de+2, 0++3) (ri, 7441, S142, 8143) which is the third-best candidate of stage 4 in Cheetah-Ruu (Pixel). The auxiliary loss of AARL-State is: {s a2s 1 8e42; 0.+2, Qr-3, 3443 44444 T144, d2+5 4447, S148. ar 04-8:7+48)
(5)
+ {5t+1, 8143, a++4, $146, 81497
which is the fourth-best candidate of stage Cheetah-Run (State).
These two losses are chosen because they are the best-performing loss functions during cross- validation (Appendix A.4).
D.2 DURING EVOLUTION
We report all the top-5 auxiliary loss candidates during evolution in this section.
Table 9: Top-5 candidates of each stage in Cheetah-Run (Pixel) evolution process
Cheetah-Run (Pixel)
(T's dreis Rees fres, fees. de42, (ha422//4aa Reed Peea) (de, des 8642, dre-s, 814-1
de) [8/-1]
Stage-1
(9i (t.a))1772 e) = (110 fi 1:0 4212: (213)
ft 1642: Resai Ptsa: 0es4: Pessi Reses Re47)
(9,, Ges Stu 9102 2522: Pres/ Pe44 9646. Reai, 9647)
(4,, 9te1, Pe42/ -t (fe tsst, 8142. 0143)
St, deen; Fees fred:
19442,0ee0a reed, dhes, 8545)
(5s oz, dey 2242)) (Su7t 9612-
Stage-2
(at, rei Hua1-ff 1191 0242: P2:2: (214)
(91:118122) fe (eaa:
feat, feed, dea2: frea: 0e-9.0044) de44) (9e, (96,
(res dre1: Preal 3 (8,, dee tra2]
Nis 2e44/
(4h Ta 1.0 Paraa 51177 D: R21): - [8t 91 11111 Pap Po
Stuge:
9es des deeas Toea: Resa Pea2, fe44, Geea) (Rea frezy Rep4s (esn)
(4e, deet, Fres
Ro-g Ti-c] + (Ret) drea, Teed Read (p-s)
(Tr, Mea1 (8r Me deat Reag)
12:
(3:4)))) (A 22:2 P212:
(4e, Ozi desai Peed fra Resa: (6-a) - (8443 Pe43:
Stage 4
(9c) (de, Yt. 8epa, Teess deea, Bren,
(sg Th Reed Stan, dres Phea, Ti-., (e44) (8s, Mee1 Teets Thess Moads Resd seet
St 1. 2013 ui)
Ber 5012)
(at 9pa1 Pess 0ss2 Pts 2::2:0033 Peis) ( - (Ta Reaks Rea1 (194)
Bes deed Tesh 4es5. Peoc
(fe4s) Ree2; Geed Seads Oept; See-i 6445)
Stago-5*
I (feet dren, d4s) + (re, 814. 8143) (de) (Be depa Pe4a-
[0e4 Popa Tepa Reag 0e44) - (Fe, Peets Peasi Res-s: Re-4i Stsn)
Be, dews: Bred St-t Te-o) (det1: 8e42; Reed. Beads degel (145)
Stage-6 6
Tesal - (et Heeg Treg, 864-4 Se43]
Be, deed: Tees ased Poec) (Beal Reas, Beads Bead Oetts fees, Reaa)
(8s, de, Yt. Bess fees, dres, d444) -t [dr, Stuz, Peed, 8t deet: (est,8645)
Ne Th43, d143,
fer, Sest, St-g, Ti-g deed Tekds Redd Goed Teed. 8i-5)
Stage 7
(Seak (142, T442, 8543, de43, Feed, Sead] (re, des Paps 8442, T442, 8144, Read)
deess R243, au-2, Te-a, Nt-3 0r43, Tesd: Reed) (Si O2y3: Ru 4111) + (5c 3012 122200
814 mea. 842.. R443, 7243 N944 Reed.
*: Used for cross validation, Pixe.
20

---

Under review as a conference paper at ICLR 2022
Table 10: Top-S candidates of each stage in Reacher-Ease (Pixel) evolution process
Beaches Bay (Pice))
Stage:
(4u Peicere de dara thess forto fay Sive dee Aeraa Maodees fress Seip fae fava de Alyii for fays daci Plyni Kayp Aeccad Pheio
Dhecs the Ruco Faod Paus Nopp
de Payps fo df af He Eou th d fo ii days "vae To-e
the fee frem Chea
Reseedoon Torelles Aua Reass foa thae storr - Inses dhe
(1r.0) con (odiro ] (rudiy (or
Stage1
Seyl ta po dhys So (oy2) Io the tra to it's di the dh Ai to ih
Sage-1
a fay
Sage-5
Keel feetty
Suge-5
Suge-
Table 11: Top-5 candidates of each stage in Walker-War Walk (Pixel) evolution process
Wilker Stads Pr Te- deed Arod. Fean)
Yean]
Stage-1
Stage-2
(re,
Fousl) [tr,
[4x,
d [ Ta -n City
(4)
A
the
Stag-33
Ae Rosss Teae -i (nees)
Au Fas fo Greas Taods fo-ds Orea Roea Feaal -r (Ar, Or, faea feags Roep frod dest dess duary Paudy Faud deuel
(4) Pas dea de
As diy Hoal)
1980 Pe92)
Le Tel Feels Aoaz: Megg Taag: Te44) (1y Ty Tha fa-z: Psagi Ayad Fawas Food: (rae
Stage
(e) Aiy An du Fe
(e - Eig To the ti Naya drpa)
Le To Yeais Mi-d Mo-ay Mhs.o Mo Me (or, his Poniy Prese Tayas meee)
21

---

Under review as a conference paper at ICLR 2022
Table 12: Top-5 candidates of each stage in Cheetah- -unn (State) evolution process
Chectab-Ron (Row)
[te Preic teea ne [Reas. (to2))
(rea) (Au,
Stage11
Tean deam raam deess Bred, Trea) deals feass Ohuck feeds feed, deed, ftur fead)
(01-1- dedd died deee Meeds toede (es7) the feek, feed foek feedy dred Muetiteee Ne46)
for, deat that Paads thad
Ausd deads the dealt deatt desa Reeas Pryal (tuats Ready Aeady de Abudd fhe Rean)
(eee Tes Meats 2eas -h (taz)
Stago-2
fie Reeas Press deeas Frep Ruedu Faeds Reeas Taeas fi deadi Ta-ds Toed, dgary o Freal - foei feepi flod feedi fredi feed Toeo)
(o dags dhy yo depss Preae Rapas Ply dup Meyt Poy Oaps Pays) (Mauts Nayda Ptods (epeee
[Rees]
3
(9642)
mou 7 [Reas]
Ie deass Prant (s,, deas)
the de 7h teat desal
to ae, deeas Ptece des2 PrepoPted To fees, Bead, Foed, Reeds 9t- Geeas Ftoas (rea) -e [er, feedy dteg, Reepy freas frad Reees Atai)]
Stage 4"
Me mop fudd Paeds fayds thce taeds neyd faedy th fue Mayk Yurk) -
(oe Aredd fisk Royd Aeeo Raply Ator) (ora Trun) (r deacy
Reed mte)]
desti
Pepsy
Hee meaas Ptad) Shuas thats feags the Yeap Yead))
Stage-5"
the ous Reess feea? (
No au the to to faear deser
as da you the Paya)) Regpi Ane do Sepe)
Le dee Ty tear Peass fisel
Fooad -i [Mey Reads fougg Poad fe Rap du Paps) Peys) Hi (Ae.
th Reea) - (Fa
Stsge-6
Seps)
*e "er
tee, Res Reats thas? Repss mur)
(te Pe.-9e77.000 Rroci rees) - (sess-Pees
Suggo7
(ro Nuss)
(rr,0r, Ress) - (dt-1 Bepa f1az)
[9t Reezs Reed. Feegy 9643, Peeds Reads Poeas Ared Reed, Reen, Re47, Fess] -b (Reed Ai-g. Apads 9te1-00-4---- Reads 2roa)
4: Used for cross-aaaiian, AARL-Suk..
Table 13: Top-5 candidates of each stage in Hopper-Hopp (State) evolution process
Hogper- Hop (Raw)
[dr = (riidess)
(ar ity deels 0140s Teea, Rted, dredy Teea, Re-d, Aeat, Pied, dree diet: Teed Reeas Plas) (Aty Reed, Re-dy Oued, Stedy A6477 Re-d, Rean] An, Orea: Reeds Prety feee, Prao, Pual t> Prestt fe Peess Aeeds Proe Reets Reyd Rrass Aejo)
Singe11
Depa- Feaa Pead (Ar, Rey feems Seeas Teea the Tepn fees, Buee)
[A, any O1a1) "> Fre9)
[as,
Peaa] - (Feet: Sted, ai Asoo]
2tage-2
(4,, de Reed, ste3, Reed, Aseo, 0146]
(sres, 7tea: reea Pres Aeed, Ri-ty 4eeo 0uso: (ee))
(f dee Tagt Reed, Aesa Aesa desgi Tepes
di des feee deenn Dewl -t fheii Aesa)
(
Pre))
Stage-3
(dr. dee Aeaa Pa-s) -+ -a.. Repo, Bepo.
(ai, an Re-s) > [Aest: 9rea)
Ar, Reeds need (14s) (143)
(A Aes1:
[Pea)]
Suge-4
Au Fh Seads d Post Sen.. Neo)) + (epposs
be fiee Peea. - (o:4
(e, degd. 0143] [96-1.8142)
Ai, on doy do deen, f2-3) + [Au deple mes)
(a, and fests Rod, (e-2)ftss)
>> (Mo Repo
Stage-5
Le Teail (9t, Aqul
dagg Peas) (1p-1.0
(or deas
Ay any Re - (eo
be Seed dega, Poaa] (ft. (tg1
Stage-6
(4, an R2-3] (e4499-11
[A, 2t, Aests P2o3) -3 (7,, (641,
(dr, at, Grs Re9]] = [St, Sept. Po-1)
22

---

Under review as a conference paper at ICLR 2022
Table 14: Top-5 candidates each stage in Quadruped-Rus Run (State) evolution process
Quadraped-Rin Run (Raw)
{ax, Phi deeeee Reed,
Bee1, Reea]
Fry
3 [Ag Pe Readr Tenss Preae Prude Ouadd Psyi) (de
Stage-1
teed)
(1 2e Sess: Puass ((a4:)965)) s) Pess P Paeas Pesar (en du Ty Mo dests aesversd + (trssifug)
Re,
Pee - [Ae, Osel Piat fe-d, Aeed doeds Pred, deeds Teed Aeen Phet Reet Reads Foeds Reee) (Jees Reec, Read, fee fee Aeed Reeds Assa)
Slage-2
[ I- Aeggs Aeads Feea) [Pt, 2e, Reels Repar
Puid (144:050.773 Yeuto TesperP Pron Prars Peeay (ceo)) - Etes Resis Reads Aipie Rpsiy se Mes fevo)
Auss frsa) (f Rtag:
(ee, deads Gteas Teas Yese, (445:04777744 Ri-ss desa) (ft, fhess triz, stass Seess Cheas Phes deeds tham de Pi Rt-p)
ae, de-d Grea Pres Reed, Best Re-k) J - (di Aeed deec Grets treds the Oheed Steds Stop daye Peep (rie)
Stage-3
Preal [Rp
fres)
newd Feeds dree, Meek mrets deed Pie Mtss] (Rr, mies Bhell thod the dr dren, theth Meck te-e)
(a On Te Orago Paag Ressofu fru 91.4 Rraa)
Pec fas des Pead)
Fay
fe-d. Peea
- (e-22-
the
Stage-4
(az, dewd Bsea: Thea Red)) [fte): Thal, 1e-2. (1-2-11-2--8-aa)
Geyd feed Besds Teeds Repos 9ts5] [ [ar, Ro-is Rrwe)
Hegs Tea3 feed. Meads Teed Meadi Toua) (eot (aan Meeg Mee3)
Jec Fry Test Re-2, Tesay aoaa Aead, ae Te543 -t [Sc, 9191:9001, 4142 9e-2 Ges th (t.5)
Stage-5
(e, Be-
(8t, To Re-dy
Read)
Bewil Seads Arya Yeess Rreec Beeds Aopos 9ew7) Aeps, Rraa) (Ar Seeay Reads Reais
Prati See TeelyT
(Ar Ohy The Paele Reen, Reed Reads Poyd, At45) (0- 0144)
23


Reviews:
[{"review_id": "vNh3pGap7Uw", "reviewer": "Reviewer_xAxX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n- The paper is well written an easy to follow\n- Error measures are present in both the figures and the table report \n- The Appendix is well written and has many details\n\n\nCons:\n- In several parts of the paper, the authors claim that AARL discovers the “optimal auxiliary loss function for RL”. I think this statement is not supported by any mathematical derivation that proves the optimality. Even if we decide to forget about a proper mathematical derivation of optimality there is still the important issue that the authors only considered one domain: the DeepMind control suite. This is clearly not enough to support the broader claims made by the authors, in the abstract, introduction and conclusion.\n- Another issue of this approach is that the search space is done on the loss input (I) and the loss operator (f), but the encoder is kept constant. Changing the architecture of the encoder can have a massive impact on the auxiliary loss, especially the one that requires rollouts. For instance, would the conclusion hold if the CURL encoder is substituted with a Transformer? The authors have an ablation on Encoder Architectures in section 3.4, but it’s limited to the state-based case, which is less interesting than the pixel based one. I believe this is an important limitation as it could drastically change the conclusion of the paper.\n- Another significant limitation of using just the DeepMind control suite is that the environment is fully observable, at least in its normal implementation. I assume this is the case also for this paper as I don’t see any details in the main paper or in the appendix that go against this assumption. Claiming generality of a method based on conclusions drawn just on a fully observable environment is very dangerous. One known example of this is actually the action prediction auxiliary loss reported in the paper {s_t, s_t+1} -> {a_t}. Indeed, it is well know that this kind off loss works well in fully observable 2D environments, but it could have severe issues in partially observable 3D environments due to the increase state aliasing problem (e.g. Badia et al., 2020).\n", "summary_of_the_paper": "The paper introduces an approach to automatically discover optimal auxiliary loss for RL, the method is called (AARL). The authors claim that AARL outperform baselines on both pixel-based and state-based task on the DeepMind Control Suite. Also using their method the author analyse different auxiliary losses to identify common patterns.", "summary_of_the_review": "The authors combine different AutoML techniques to automatically derive the best auxiliary loss function for RL. Although AARL is an interesting approach, the claim in the paper are not supported by empirical findings. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "avvQhiWVyfC", "reviewer": "Reviewer_fH26", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. The motivation of automatically searching optimal auxiliary loss is good. Handcraft loss function highly depends on researchers' domain knowledge, and its performance can not be secured. The automatic search strategy can significantly reduce locality and improve the RL agent's performance.\n2. The design of auxiliary loss function search space, as well as the evolutionary pipeline, is reasonable. The pruning of search space can greatly help to reduce search complexity.\n3. The experiments are extensive and supportive. The critical experimental parameters are provided, and thus there should be no issues with the repeatability of the experiments.  \n\nDisadvantages:\n1. My major concern is about the efficiency of the proposed method AARL. The overall loss function is a combination of reinforcement learning loss and evolutionary loss, both of which are difficult to optimize. The combination of RL and evolution would be even more difficult to converge. What's more, the efficiency could be very low, as the difficulty of convergence would not be a plus, but be a multiplication of RL and evolution.\n2. The combination of RL and evolution could be improved. Equation (1) optimizes the two algorithms by a simple weighted sum of the two loss functions. A more practical solution is to alternatively train the two losses. There are two reasons for doing so: 1). The scalabilities of the RL loss, as well as the auxiliary loss, are different, and it is difficult to confirm the weight \\lambda; 2). The required training steps for the two algorithms may differ.\n3. The contributions are not significant. The major technical contribution is to design a search space and an evolution strategy to derive an optimal auxiliary loss. And also, there is no theoretical justification for the proposed method.", "summary_of_the_paper": "In this paper, the authors intend to automatically search for the optimal auxiliary loss. To achieve this goal, the authors propose Automated Auxiliary loss for Reinforcement Learning (AARL), which uses an evolutionary search strategy to explore the very large loss space. The paper conducts extensive experiments on the DeepMind Control Suite and shows that the searched auxiliary losses have significantly improved RL performance in both pixel-based and state-based settings.\n", "summary_of_the_review": "This paper has good motivation, a good solution for the problem, and extensive and supportive experiments. The major concerns are low efficiency, convergence difficulty, and non-significant contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aFs-QMp8L0", "reviewer": "Reviewer_fPxR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n* This paper is overall clearly written and easy to follow. The method is easy to understand and relatively straightforward. There are enough details to fully reproduce the training. \n* To my knowledge, evolving the optimal loss function for auxiliary RL tasks seems to be a novel approach. The central thesis of better representation learning in RL is a problem of good practical value.  \n\n**Weaknesses**\n\nMy major concern is the weak experimental results. To elaborate: \n\nFor pixel-based Deepmind Control Suite, the strongest baseline that the authors compare to is CURL (Laskin et al., 2020). However, this is quite an outdated baseline. The authors *ignore at least 2 recent strong baselines*:\n\n* Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. Yarats et al. ICLR 2021 Spotlight. https://openreview.net/forum?id=GY6-6sTvGaf. The algorithm is known as \"DrQ\"\n* Reinforcement Learning with Augmented Data. Laskin et al. NeurIPS 2020. Also known as \"RAD\". \n\nBoth papers (neither cited in the paper) are published on top ML conferences before June 2021, so it is fair to request comparison with these prior SoTAs per the ICLR review guideline. Both papers above are about a very simple idea - vanilla reinforcement learning with simple data augmentation can be an exceptionally strong baseline. In fact, if we compare table 1 of the \"DrQ\" paper (Yarats et al.) with table 3 of this paper, we will see that DrQ *beats AARL-Pixel in 7 out of 12 tasks. For the other 5 tasks, none of the gains of AARL-Pixel is statistically significant.*\n\nThis indicates that even **simple RL with image augmentation can outperform the complicated bilevel optimization and auxiliary loss in AARL**, which greatly undermines this paper's contribution. \n\nFurthermore, in section 3.1, Fig. 4, the author does mention \"SAC with data augmentation\", which appears to use the same scheme as \"Reinforcement Learning with Augmented Data. Laskin et al.\" (RAD). However, SAC + augmentation (blue dashed line) consistently underperforms CURL (blue dotted line) in Fig. 4, which contradicts the results in RAD. In addition, the numerical results indicated by the lines also disagree with Table 1 in the RAD paper. I believe there are factual errors in these plots. \n\nFor state-based DMControl experiments, the paper claims in section 3.2 that \"there is no data augmentation in the state- based setting.\" This is not true. Section 5.4 in the RAD paper discusses simple ways to augment low-dimensional states, and shows that they are highly effective in boosting performance. \n\nMinor comment: table 1 has a misnomer. \"Inverse dynamics\" typically means action inference, instead of predicting the previous state. ", "summary_of_the_paper": "Reinforcement learning agents can benefit from auxiliary tasks. This paper introduces an evolutionary algorithm to automate the design of auxiliary loss functions, based on prior approaches like forward dynamics, inverse dynamics, contrastive state representation learning, etc. The evolution solves a bilevel optimization problem in which the inner loop is regular RL training while the outer loop evolves the loss function. Experiments are conducted on the Deepmind Control Suite for both pixel and state-based observations. ", "summary_of_the_review": "The paper ignores at least 2 simple but important prior works, and do not outperform the results in those baselines. The contribution is greatly undermined by the fact that simple RL with image augmentation can outperform the complicated bilevel optimization and auxiliary loss in AARL. Some experiment plots may contain factual errors. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_eOb8uuwvI", "reviewer": "Reviewer_Fvi1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows the following advantages:\n\n(+) A method automatically selecting the auxiliary loss function is introduced in this paper. It avoids the hassle of manually designing and testing auxiliary loss functions, as well as provides a more efficient search method than grid-search when dealing with a huge search space of the auxiliary functions.\n\n(+) The experiments are done with either 5 or 15 seeds, providing reasonably reliable results, although more seeds could improve the accuracy of evaluation even more.\n\nThere also remain several concerns:\n\n(-) I am concerned about the accuracy of the first pruning step---randomly sample from the search space, then choose the similarity measure with the best performance which is averaged over 15 random seeds. I agree that this method is more computationally efficient than grid search. However, there exists a trade-off between computational efficiency and the accuracy of searching. There is no guarantee that a good candidate can be sampled in the random sampling step. The sampled mask for loss input may not be the best combination of the chosen operator (similarity measure). Thus, it introduces the risk that the best candidate is never selected and evaluated. It might be better to give an analysis on how large this risk is or the probability this situation happens.\n\n(-) In the experimental result section, I am not convinced by the result of Hopper-Hop and Quadruped-Run. With the chosen auxiliary loss function, there exists a high chance for the agent to perform worse than the SAC-DenseMLP baseline---the averaged value of the violin plot is slightly above the SAC-DenseMLP performance, leaving a lot of auxiliary loss function candidates performing worse than this baseline. This makes me worry that the risk of choosing auxiliary loss seems high. \n\n(-) The paper claims that the chosen auxiliary loss can generalize to unseen tasks. But it is not clear to me where this generalization ability comes from: the auxiliary loss selection method itself, which is a searching process, or the setting of training on multiple different and challenging tasks. Multi-source training itself can prevent overfitting. Therefore, if multi-source training is not part of the framework, it may be worth checking how the different number of training environments affects the generalization ability, especially in the case when there is one training task only.\n", "summary_of_the_paper": "This paper introduces a method for searching for the best auxiliary loss function from a huge search space automatically, while the best auxiliary loss function is defined as the one that encourages the agent to get a higher return. The searching space of auxiliary loss functions is finite and discrete. It is a combination of (1) manually designed similarity measures, which is used for encouraging the prediction and target to be similar in the auxiliary task, and (2) the binary mask, which is used for selecting auxiliary loss inputs. In this work, the size of the search space is around $4.6 \\times 10^{19}$. The space is pruned firstly by random sampling from the similarity measure space and choosing the one showing the highest averaged performance, then the evolutionary algorithm is applied to select top candidates for loss inputs, which is a method in the literature. The chosen auxiliary losses are empirically shown to be helpful with increasing the learning efficiency and have generalization ability to new tasks.\n", "summary_of_the_review": "I consider the weakness to outweigh the strength. This work does provide an interesting idea, which is selecting auxiliary loss functions automatically from a huge search space. The empirical results suggest that the average performance of applying the chosen auxiliary task is higher than baselines. However, the risk of using this method seems high---at the first step of pruning the search space, there remains a high chance that good candidates are never tested, thus the pruning result can be based on the evaluation of bad combinations of operators and inputs. Moreover, the experimental result also suggests that this risk exists.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use evolutionary methods to learn auxiliary loss functions, demonstrating superior performance vs. typical auxiliary losses previously proposed in the RL literature.

Demonstrating that it is possible to learn auxiliary losses by evolution, both for pixel and state representations, that help train significantly faster (even on new environments) is definitely a meaningful contribution, as acknowledge by the majority of reviewers.

Although many of the original reviews' concerns were addressed by the authors during the discussion period, two major ones were only partially answered, both related to the limited empirical evaluation of the proposed approach (which is crucial for such a contribution that aims to demonstrate an improvement over existing related techniques):
1. The limited set of environments used for evaluation (and in particular the lack of partially observable environments)
2. The fact that the baseline being compared to was CURL, which the paper describes as "the state-of-the-art pixel-based RL algorithm", while reviewers mentioned DrQ and RAD as two more recent (and better) algorithms that were known well ahead of the ICLR submission deadline (note that the more recent DrQ-v2 is now even better). Since the data augmentation techniques used by these algorithms help shape the internal representation, like auxiliary losses do, it would have been important to validate that the proposed technique could be useful when plugged on top of such baselines.

The authors did try their best to address these major concerns during the rebuttal period, but the discussion between reviewers and myself came to the conclusion that this wasn't quite convincing enough yet. I encourage the authors to investigate these points in more depth in a future version of this work so as to make the empirical validation stronger (NB: the links provided in the last comment by authors on Nov. 30th didn't work, but this wasn't the main factor in the decision).
Decision: Reject