RL-DARTS: Differentiable Architecture Search for Reinforcement Learning
Authors: Yingjie Miao
Xingyou Song
Daiyi Peng
Summer Yue
John D Co-Reyes
Eugene Brevdo
Aleksandra Faust
Venue: ICLR 2022 Submitted (2022)
Keywords: darts
differentiable
architecture
search
neural
nas
rl
reinforcement
learning
procgen
supernet
softmax
variable
ppo
rainbow
off-policy
on-policy
convolutional
autorl
automated
one-shot
efficient

Abstract: Recently, Differentiable Architecture Search (DARTS) has become one of the most popular Neural Architecture Search (NAS) methods successfully applied in supervised learning (SL). However, its applications in other domains, in particular for reinforcement learning (RL), has seldom been studied. This is due in part to RL possessing a significantly different optimization paradigm than SL, especially with regards to the notion of replay data, which is continually generated via inference in RL. In this paper, we introduce RL-DARTS, one of the first applications of end-to-end DARTS in RL to search for convolutional cells, applied to the challenging, infinitely procedurally generated Procgen benchmark. We demonstrate that the benefits of DARTS become amplified when applied to RL, namely search efficiency in terms of time and compute, as well as simplicity in integration with complex preexisting RL code via simply replacing the image encoder with a DARTS supernet, compatible with both off-policy and on-policy RL algorithms. At the same time however, we provide one of the first extensive studies of DARTS outside of the standard fixed dataset setting in SL via RL-DARTS. We show that throughout training, the supernet gradually learns better cells, leading to alternative architectures which can be highly competitive against manually designed policies, but also verify previous design choices for RL policies.

Under review as a conference paper at ICLR 2022
RL-DARTS: DiFFERENTIABLE ARCHITECTURE SEARCH FOR REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, Differentiable Architecture Search (DARTS) has become one of the most popular Neural Architecture Search (NAS) methods successfully applied in supervised learning (SL). However, its applications other domains, in particular for reinforcement learning (RL), has seldom been studied. This is due in part to RL possessing a significantly different optimization paradigm than SL, especially with regards to the notion of replay data, which is continually generated via inference in RL. In this paper, we introduce RL-DARTS, one of the first applications of end-to-end DARTS in RL search for convolutional cells, applied to the challenging, infinitely procedurally generated Procgen benchmark. We demonstrate that the benefits of DARTS become amplified when applied RL, namely search efficiency in terms of time and compute, as well as simplicity in integration with complex preexisting RL code via simply replacing the image encoder with a DARTS supernet, compatible with both off-policy and on-policy RL algorithms. At the same time however, we provide one of the first extensive studies DARTS outside of the standard fixed dataset setting in SL via RL-DARTS. We show that throughout training, the supernet gradually learns better cells, leading to alternative architectures which can be highly competitive against manually designed policies, but also verify previous design choices for RL policies.
INTRODUCTION AND MOTIVATION
Over the last decade, recent advancements in deep reinforcement learning have heavily focused on algorithmic improvements (Hessel al., 2018 Wang al.)
2016). data augmentation (Raileanu
et al. 2020; Kostrikov et al. 2020), infrastructure upgrades (Espeholt et al. [0188] Horgan et al.
2018), and even hyperparameter tuning (Zhang et al.) 2021 Faust et al. 2019). The automation
of such RL methods have given rise to the field of Automated Reinforcement Learning (AutoRL... Surprisingly however, one relatively underdeveloped and unexplored area is automating large-scale policy architecture design.
Recently, works in larger-scale RL suggest that designing a policy's architecture can be just as important as the algorithm quality of data, if not more, for various metrics such generalization, transferrability, and efficiency. One surprising phenomenon found on Procgen (Cobbe et al. 2020), procedural generation benchmark for RI., was that "IMPALA-CNN"
residual convolutional architecture from (Espeholt et al.] 2018), could substantially outperform "NatureCNN", the standard 3-layer convolutional architecture used for Atari (Mnih et al. (0133), both generalization and sample complexity under limited and infinite data regimes respectively (Cobbe et al. 2019). Furthermore, in robotics subfields such as grasping (Kalashnikov al. 2018 Rao al. 2020),, cameras collect very detailed real-world images (ex: 472 x 472, 4x larger than ImageNet (Russakovsky et al., 2015) for observations which require deep image encoder architectures, consisting of more than 15 convolutions, raising concerns on efficiency and speed in policy training and inference. As such RL policy networks gradually become larger and more sophisticated, so does the need for understanding and automating such designs.
Many efficient Neural Architecture Search (NAS) methods utilize the one-shot setup, where only one set of shared weight parameters is trained through an ensemble of models, otherwise known as a supernet. Differentiable Architecture Search (Liu et al. 2019). trains the supernet by inserting architecture selection variables inside the network in continuous relaxation approach, in order to be differentiable with respect to a loss. These architecture search parameters are then used to select a final discretized perception module to be retrained from scratch in evaluation phase.

---

Under review as a conference paper at ICLR 2022
The differentiable aspect of DARTS is highly attractive to RL, which possess two complications which SL does not:
1. RL is an entire system. In most RL algorithms involving auto-differentiation, there are multiple different phases: a collection/exploration phase which collects environment data via forward passes/inference of a policy, in addition to a training phase which updates parameters for potentially multiple network components (e.g. both the policy and value function in policy gradient methods). There can also even be a separate evaluation phase (disjoint from evaluation in NAS) different from the collection phase (e.g. using argmax greedy evaluation instead of the stochastic actor, for both Q-learning and policy gradient methods). Such phases are also commonly distributed over different machines, adding an extra layer of complexity via inter mier-machine communication.
2. RL training curves possess considerably more variance and noise than SL curves, with a much larger room for error due to sensitivity to hyperparameters (Andrychowicz et al. 2020), which originally led to the criterion of reporting mean and standard deviation for 3 seeded runs (Henderson et al. 2018). In fact, this noise can be nearly unbounded, as in many cases, unlucky and catastrophic seeded training run can simply lead to a trivial reward even with an optimal architecture. Such noise may potentially be problematic for fast convergence for blackboxxevolutionary optimization methods. Furthermore, the RL loss contains important information not conveyed by just pure reward (e.g. value function error and entropy penalty, as well as potentially exploration/novelty bonuses), whereas information may be lost when simply summarizing all progress to single scalar point of data.
DARTS solves (1) as it integrates naturally within the RL. system with minimal modifications to the code and pipeline, by simply replacing the image encoder in standard RI algorithms. For (2), architecture search parameters in the DARTS supemnet are joint trained alongside weight parameters, and can thus also use information from multiple auxiliary losses, which can be much richer signals than potentially sparse rewards. We represent this approach in Figure However, since there has been previous proof of DARTS's feasibility in standard RL, one may hypothesize a few subtle concerns. While DARTS in SL is simpler and can focus more on generalization, a core major challenge RL- DARTS is the feasability of training the supernet itself due to a non- -tationoonn data distribution. This is especially because using differentiable loss (ex: Bellman error) as the search objective is a double-edged sword in RL, as loss is defined with respect to the replay buffer, whose data is collected by the policy itself via inference on the architecture. In SL, the data distribution is independent of the architecture, and the loss (e.g. cross-entropy or negative log-liiel)hood) strongly correlated or even equivalent to the objective (e.g. accuracy or density estimation (Parmar let al. (018)). However in RL, a bad architecture could lead to a poor initial distribution of data, and thus a local optimum where the loss is zero while the policy is still poor. This is inherently due to the fact that the data in the replay buffer may not sufficiently cover the state and action space of the entire environment, and is common in cases such as sparse reward problems. This issue occurs even if the replay buffer data is fixed, which is the basis of offline RL. (Levine et al. 2020).
DARTS Supernet
Learner
Replay
ONT
Actor
Envreoment
Figure 1: Representation of our method, in which a DARTS supernet is simply inserted into the network components of standard RI. training pipeline, which may potentially he highly distrihuted.
The other question occurs during the discretization phase of DARTS, as any continuous relaxation method must deal with integrality gaps when rounding back to a discrete space, with this gap dependent on the optimization landscape. At the core of multiple discretization procedures (Liu) et al. 2019 Chu et al. 2020c. Hundt et al. 2019) lies the idea of op selection based on edge weights. However, RL's optimization landscape is significantly different (Ilys et al. 2020; Ahmed et al. 2019) from SL, and thus even if the supernet trains in RL, it is unclear if e.g. DARTS's 's default discretization procedure leads to a better discrete cell, which is the original goal of DARTS and NAS in general.
2

---

Under review as a conference paper at ICLR 2022
In this paper, carefully analyze the extent at which DARTS can optimize perception modules for RL environments. While there is a rich series of active study on the theoretical and empirical behavior of DARTS optimization in SL (Wang al. 2021b Chen & Hsich 2020 Li al. 202ID. our work is one of the first to study its optimization in RI., a completely different paradigm. Through extensive experiments, we ultimately validate that even vanilla DARTS is capable of architecture search in RL, laying a core foundation for future work in NAS for RL. In summary, our contributions are:
We demonstrate DARTS can be integrated with existing RL algorithms in a minimally invasive and easy-to-use manner, agnostic the specific pipeline. Remarkably, challenging Procgen benchmark featuring infinite data, the DARTS supernet is able to train from scratch and efficiently produce similar training curves baselines using both on-policy (PPO) and off--oolic (Rainbow) methods, both of which require inference on the network.
the
Due to the common issue of integrality gaps between supernet cells and discrete cells, we verify both qualitatively via visualizations and quantitatively via rewards, that discretized cells start with suboptimal architectures and gradually evolve to better architectures. How- ever, we also demonstrate how this can fail, especially if the corresponding supernet fails to train, with further extensive ablations in the Appendix.
By evaluating the final discrete cells the end of supernet training, discover that many environments benefit from learned, custom-tailored architectures, suggesting that human- designed architectures can be suboptimal. Furthermore, on multiple environments can potentially produce a single transferrable cell. This opens up questions for future study, large-scale RL can potentially benefit from more applications of NAS techniques.
find that joint-training supernet
RELATED WORKS
Most previous AutoRL methods can be thought of in terms of blackbox and evolutionary methods. For NAS on RI, these include (Song et al. [02T] Gaier & Ha] 2019 Stanley & Miikkulainen) 2002
Stanley et al. 2009) which utilize CPU workers for forward pass evaluations rather than exact gradient computation on GPUs Such methods are usually unable to train policies involving more than 10K+ parameters due to the sample complexity of zeroth order methods in high dimensional parameter space (Agarwal et al. 2010). However, such methods have proved useful on smaller search spaces
for RL, such as hyperparameter optimization (Zhang et al. 2021 Franke et al.) 20200 Parker-Holdde
et al.
2020;
al. 2017) and algorithm search/ learning to learn" methods (Faust et al.
2019
Co-Reyes et al. 2021).
In terms of differentiable search in RL, methods for tuning hyperparameters and environments include (Zahavy et al.) |020: Xu et al. 2018 Hu et al.] 2020). The only previous known application of
DARTS is (Akinola et al. 2021 which searches for the optimal way of combining the observation and action tensors together in off-policy QT-OPT (Kalashnikov et al. 2018). However, this approach does not modify the image encoder nor uses the supernet for inference, as off off-policy robotic data collected independently from the actual policy is used for training, which leaves the applicability of DARTS to inference inference-dependen dependent RL as an open question addressed in our work.
3 METHODOLOGY
3.1 VANILLA DARTS AND NAS PRELIMINARIES
We provide a brief review of DARTS which can be pictorially summarized in Figure although more details can be found in the original DARTS paper (Liu et al. 2019). DARTS optimizes substructures called cells, where each cell contains intermediate nodes organized in a directed acyclic graph, where each node ((()) represents a feature map. and each edge (i.j) consists of an operation (op) obd,) with later nodes z((i) merged (e.g. summation) from some previous (z((). A DARTS supernet is constructed by continuously relaxing selection of ops in O, via softmax weighting, i.e.
alid) (z(0)) EnEo (1.) o()(1)), where p!..))
The cell's output is by default Eoreo 5.05
3

---

Under review as a conference paper at ICLR 2022
the result of a Conv Ixl op on the depthwise concatenation of all intermediate node features, although this may be changed (e.g. simply output last intermediate node's 's features). We denote the collection of all architecture variables ab (KJ) as Since the total op space O must contain the Zero and Skip Connection ops, we mainly refer to the main op space as Obose where = Ohase U (Zero, Skip). In the standard optimization framework for SL where there is the notion of validation set, the learning procedure consists a bilevel optimization problem:
arg min Leat (f", a)) s.t. 0* = arg min Lirain (0.
0
where [(,, at) is a loss function defined with respect to the supernet. at is then discretized into final cell, by representing each edge (i. j) with the highest softmax weighted op arg max,eO (,o Po (1.f)
and then retaining only the top K 2 incoming edges for each intermediate node.
Following common NAS practices (Zoph et al. 2018: Pham et al. 2018 Zoph & Le| 2016),
((0,a)
AL RL-DARTS Call
construct our supernet by stacking both normal (N times) and reduction cells (R times) together into blocks, which are themselves also stacked together times (see Figure 2). Normal cells preserve the input shape on output, while reduc- tion cells apply a stride of to reduce height and width immediately on the input. Each block can have its own associated convolutional channel depth which is used throughout all cells in the block. Another common NAS practice we adopt is to train smaller supernet (i.e. depth 16) to reduce computation time, but evaluate final dis- cretized cells larger models (i.e. depth 64), with D = 3 layers for cheap large-scale runs and D 5 layers for fine-grained A/B testing.
Output
Normal Cell (xN)
Node
XD
Reduction Cell (xR)
Nods
Ieput
Figure 2: Illustration of our supernet via stacking nor- mal and reduction cells. Solid lines correspond to se- lected ops after discretizing the supernet, which contains all possible ops weighted using a. If R >> O, we add an inital Conv3x3 for preprocessing.
3.2 RL-DARTS
For notation, we use standard conventions in the RL literature: for given MDP MM, we denote st. des Fz as the state, action, reward respectively at time t. π is the policy and D is the replay buffer which contains collected trajectories 7 (so, ao, To, .... The goal of the RL algorithm is to maximize
((T) = E Σr Tt the
expected cumulative reward when using policy In most RL algorithms, there is the notion of neural network torso, or encoder which maps a state 8 (or sequence T) to a feature vector, which may be projected correct action dimensions, behind the scenes in forming In the DARTS case, use a supernet for this encoder instead, which leads to policy denoted as TT,
The original bilevel optimization framework (Eq. D is notoriously difficult and unstable, some- times requiring special techniques (Liu et al. 2018; Dong & Yang/ 2019 Hundt et al. 2019 Li [020 2019; Wang et al., 2021b) specific to SL. opti- mization. To avoid such confounding factors, especially when DARTS has not yet been fully validated in RL, we may simply optimize sample complexity and raw training performance which are common metrics in RL. We thus joint optimize both f and
Algorithm 1: RL-DARTS Procedure.
Supernet training: Compute a" from arg maxg, ).T.a)).
al. 2020b c Liang et al.
2. Discretization: Discretize a² to construct evaluation policy TT,((a)) Report max, J (T
3. Evaluation:
arg max (TTO.a)) 0,0
via arg min L(0, ox)
(2)
8.0
which leads to an optimal a". With slight abuse of notation for convenience, denote To.((a") as the policy using the discretized cell 3(a") and (o) to be the corresponding loss function, now dependent on only the new sparse weights o. Note that under standard NAS procedure, must then
evaluate the discretized cell via training from scratch again to obtain the final reward:
max J (TT,)((a)) via arg arg min Ls(a
(3)
4

---

Under review as a conference paper at ICLR 2022
This can be concisely summarized in Algorithm! [ although remind the reader that L(-) is defined with respect to replay buffer D and is not exactly equivalent to optimizing J(.). Unless specified, we by default use consistent hyperparameters (found in Appendix(F) for all comparisons found inside figure, although learning rate and minibatch size may be altered when training models of different sizes due to GPU memory limits. Thus, even though RL is commonly sensitive hyperparameters (Zhang) iet al., 2021). we find that once a pre-existing RL baseline has already been setup, incorporating DARTS requires no extra cost in tuning, as evidence of its ease-of-use.
EXPERIMENTS
4.1 EXPERIMENT SETUP
We perform key experiments to understand the behavior of RL-DARTS, and answer the following main questions in our analysis:
1. Does the supernet train end-to-end at all will fail even initially collect good online data? Does converge towards a sparse solution over time, and what is the computational cost?
2. Even if the supernet trains, do the corresponding discrete cells also improve in evaluation performance throughout a's training, despite common integrality gap issues? What kinds of failure modes occur?
3. How do the final discrete cells perform evaluation, and how does RL-DARTS com- pare against random search? Can a large-scale joint optimization over multiple diverse environments lead to single transferrable cell?
In order to study the behavior of RL-DARTS, we use the Proegen benchmark (Cobbe al. 2019 2020), which readily provides pre-existing baselines for both PPO (Schulman et al. 2017) and Rainbow DQN (Hessel et al. 2018) algorithms, over diverse selection of 16 games, each with infinite levels to benchmark RL-DARTS in large data regimes where episodes may drastically change, relevant for generalization. We use the standard "easy" difficulty commonly benchmarked by variety of other works (Raileanu et al. 2020 Raileanu & Fergus 2021 Parker-Holder et al.) 2021
We also the default IMPALA-CNN architecture (Espeholt al. 2018) strong hand-designed baseline when comparing both supernet and discrete cell performances. IMPALA-CNN can be seen as specific instance of the stacked cell design in Subsection 3.1 3..1aamm and Figure where its "Reduction Cell" consists of a Conv3x3 and MaxPool3x3 (Stride 2) with R = 1 and its "Normal Cell" consists of residual layer with Conv3x3's and ReLU's, with N 2. More details can be found in the original paper (Espeholt et al. 2018).
As for RL-DARTS's search space, we consider the following base ops basee corresponding algo- rithms, and (N, R, l):
Classic + PPO: Obase.N = (Conv3x3+ReLU, Conv5x5+ReLU, Dilated3x3+ReLU, Di- lated5x5+ReLU) for normal ops and O Obsee. n [Conv3x3, MaxPool3x3, AveragePool3x3] for reduction ops. which is standard in supervised learning (Liu et al.) 2019 Zoph et al. 2018 Pham al. 2018). We use (N, R, I) = (1, 1, be comparable the baseline.
Mícro Rainbow: Obase,N (Conv3x3, ReLU, Tanh), more fine-grained and novel search space which has not been used previously in SL. The inclusion of Tanh is motivated by its use previously for continuous control architectures (Salimans et al. 2017] [Song et al. 2020). In order to avoid hidden confounding effects from the reduction cell when displaying the normal cell figures, we use (N, R. I) = (2, 0, 4), where reduction ops default to IMPALA-CNN's
4.2 TRAINING THE SUPERNET
In SL, it is expected to see the supernet's asymptotic validation accuracy eventually reach a sufficient threshold, comparable to (although slightly worse than) discrete cell's validation accuracy (Liu ct al.) 2019 ei et al. 2020). The question is, can an RL supernet achieve the same result for training reward? As mentioned earlier in the introduction, this is unclear negative feedback loop can
5

---

Under review as a conference paper at ICLR 2022
occur where initially, suboptimal supernet collects poor data for training, which leads to an even worse supernet, as a may potentially upweight suboptimal ops. However, we find that training the supernet end-to-end works effectively, even with minimal hyperparameter tuning for both PPO (Figure 3 and Rainbow (Figure[4), although PPO may take longer (50M steps) to reach comparable scores IMPALA-CNN while Rainbow only needs 25M steps. However, this is non-issue during discretization, as the a has already converged to a sparse solution (via softmax weight magnitudes) by 25M steps on PPO.
Edge tt.
Bigfish
Dadgeball
Maze
30
Softmax Waight
25
8
Frdge
Edge 17,
0.3
DiComyboReu
20
0.1
0.2
ConvixSRelu DiComySiGRReu SkipConnection
6
0M
25H
SUN
CH
25M
SOM
Edge 13
Edge 17, 2:
Coge 13.
0.4
0.3
DARTS Superet
0.1
02
0
Vonila
UM
25N
2331
25M
50M
50M
25M
Steps
Environment Steps
Figure 3: Left: Softmax op weights over all edges in the cell when training the supernet with PPO + "Classic" search space on Dodgeball. Zero op weight is not shown to improve clarity. Note that by 25M steps, the op choices have already converged towards sparser solution. Right: Sanity check to verify that the supernet achieves regular training curve, using vanilla IMPALA-CNN as rough gauge. Both use depths 16 x 3.
Edge
Edge 1)
Edge (2,
Edge
Edge (3,
Bigfish
30
0.5
Softmax Weight
0.0
Conv2x3
Reward
20
Ralu
Edge 3)
Edge 14, 1)
Edge (4, 21
Edge 14, 3)
Edge (4,
ScipConnection
0.5
Zero
10
OM
10M
20M
DM
10M
20M
0M 10M 20M
0M 10M
20M
DM
10M
20M
20M
Environment Steps
Figure 4: Analogous settings with Figure 3 Dusing Rainbow + "Micro" search space. Left: Softmax weights when training Rainbow with infinite levels on Bigfish, also converging towards sparser solution. Right: Sanity check for supernet when using Rainbow.
Interestingly, on Dodgeball with PPO, we find that the process strongly downweights all base ops except for the standard Conv3x3+ReLUU with 5x5 ops possessing the lowest softmax weights. Conveniently, this provides opportu- nity to understand whether a downweights sub- optimal ops throughout training. We confirm this result in Table by performing evaluations over standard IMPALA--NN and demonstrating that the 3x3 setting outperforms the 5x5 setting (especially in limited data, e.g. 200-level training/test regime), suggesting the signaling ability of on op choice. Although the
Scenario
Conv 3x3
Conv 5x5
Train (Inf. levels)
15.1 + 2.5
13.2 + 2.3
Train (200 levels)
12.1 + 1.7
9.8 2.1
Test (from Train)
10.2 + 2.3
5.9 + 1.7
Table 1: PPO IMPALA-CNI evaluations (mean return
50M steps) on Dodgeball. Learning curves can be found in Appendix [C.I Figure
cells using either purely 3x3 or 5x5 convolutions for the whole network,
Rainbow (25M Steps)
PPO (50M Steps)
Scenario
Trainable a Uniform a
With ReLU
No ReLU
Training (Inf. levels) Reward
3.1 0.5
0.9 + 0.2
7 0.9
1.9 0.3
Table 2: Supernet training rewards on Dodgehall. Learning curves can be found Appendix
training abilities of supernets in RL have been established, an important contrasting question now surfaces: Can any supernet train? The supernet possesses an incredibly dense set of weights, and thus one might wonder whether trainability occurs with any search space or settings. We answer in the negative, where poorly designed supernet can fail. To show this clearly, we remove all ReLU nonlinearities from the "Classic" search space used for PPO (although ReLUs still exist in final MLP projections), well simply freeze to be uniform for Rainbow, and find both cases produce poor training in Table 2 Thus, the supernet in RL provides important search signals in terms of reward and during training, especially on the quality of search space.

---

Under review as a conference paper at ICLR 2022
We further provide computational efficiency metrics in Table where we find that the practical wall-clock time required for training the supernet (i.e. the search cost) is very comparable with DARTS in SL (Liu et al., 2019), requiring only few GPU days. We do note that unlike SL where the vast majority of the cost is due to the network, RL time cost is partially based on non-network factors such environment simulation, and thus wall-clock times may change depending specific implementation.
Network
Training Cost in GPU Days (w/ specific algorithm)
IMPALA-CNN
(PPO), 0.5 (Rainbow)
"Classic" Supernet
2.5 (PPO)
"Micro" Supernet
1.5 (Rainbow)
CIFAR-10 Supemet (Lu al. 2019)
(SL/Original DARTS)
Table 3: Computational efficiency terms of wall-clock time, achieved on a V100 GPU. For the cases (PPO + Rainbow), all networks use depths of 16 x 3. Training cost in RL defined as the wallelock time taken to reach 25M steps, rounded to the nearest 0.5 GPU day. We have also included reported time for DARTS in SL Liu et al 2019 comparison.
4.3 DISCRETE CELL IMPROVEMENT
Due to the existence of integrality gaps in SL between a continuously relaxed supernet and its discrete cell (Wang et al.] 2021b(a), a). this also raises the same question in RL. We demonstrate that even using the default è discretization from (Liu et al.] 2019) leads to both qualitative and quantitative improvements, although we leave potential improvements to future works.
In terms of qualitative improvement, we display discretized cells when training a supernet with Rainbow on the Starpilot environment in Figure As we can see, the earlier cell consists of only linear operators is clearly a poor design in comparison to the later cell. We provide more extensive C.2 displaying more sophisticated yet
cell evolution ablations for PPO in Figures still interpretable changes in cell topology.
15 in Appendix
Genvia3
Genwdad
Eeptotractoi
ippp:
Itaki
Commaad
Conv263
Rulu
SkyConnsrion
Figure 5: Evolution of discovered cells over a DARTS optimization process. Left: A cell discovered in the early stage which is dominated by skip connections and only linear ops. Right: A cell discovered in the end which possesses several reasonable local structures similar to Conv ReLU residual connections.
For quantitative improvement, as a changes during supernet train- ing, so do the outputs of the discretization procedure a. We collect all distinct discrete cells {8(a1). 8((2))...... into a sequence, and evaluate each cell's performance max ((a))) Vi via train- ing from scratch for 25M steps, displayed in Figure 6 As we can see, the performance generally improves over time, indicating that
Starpilot Evaluations
40
Reward
36
32
supernet optimization selects better cells. However, we find that such behavior can be anvironment-deeenden as some environments possess less monotonic evaluation curves (see Figure 16 in Appendix C).
5
7
Discovered Calls in Order
Figure 6: Evaluation of distinct discrete cells in order from the tra- jectory of on the Starpilot envi- ronment when using Rainbow.
One reason why discretized cell may underperform is if its cor- responding supernet fails to learn. One simple reason due
the inherent sensitivity and variance in RL training. For example, in Figure 7 we see that when using Rainbow, bad training setups (ex: setting D too high) can create an unlearnable "Supernet 2", subsequently leading to unlearnable corresponding discrete cell, while the inverse occurs with "Supernet 1" and its own discrete cell. The discrete cells' rewards can also be qualitatively explained (ex: see Figure 7s caption). We perform further studies in Appendix C.3 Figure [7] showing that supernet and discrete cell rewards are indeed correlated, but also have environment-dependeen integrality gaps, suggesting that search quality can be improved via both
7

---

Under review as a conference paper at ICLR 2022
better supernet training (Raileanu et al. 2020; Zhang al. 2021). as well as better discretization
procedures Liang et al. 2019; Wang et al. 2021b).
Dodgeball
6
Supemet
5
Supemet
Discretized
Poky
Reward
4
Discretized Cell
3
(b) Discretized Cell 1
2
Tanh
Ral.
oulpul
0
inpul
Tarh
Relu
OM
10M Environment Steps
20M
Tanh
(c) Discretized Cell 2
(a) Different supernet runs, with corresponding dis- cretized cell (depths 64 5) training curves.
Figure (a) Two different supernets trained the Dodgeball environment using Rainbow, with corresponding discretized cells evaluated using random seeds. (b) Discretized cell from Supernet Note similarity to regular Conv3x3 + ReLU designs. (c) Discretized cell from Supernet 2, which uses too many Tanh nonlinearities, known to cause vanishing gradient effects.
4.4 SINGLE GAME EVALUATIONS
Since the results above establish RL-DARTS's search capability, we now focus on comparisons between discrete architectures and baselines. We first present results in the standard scenario, where RL-DARTS is applied to each game independently. discretized at 25M steps.
Plunder
Climber
12
25
10
20
13
10
Conoiciee
Contuifly
Reward
00
10M Dadgeeblll
20M
0M
10M
20M
10
Heist
Sombbez
12
MacPas333
10
MesPoolluS
orpu
MasPaetba
Disoretied Cel
Nee
IVTALA CON
Ranatom Search
0M
10M
20M
OM
10M
20M
Environment Steps
Examples discrete cell evaluations using the "Classic" search space PPO, with depths 64 3. Right: Normal (Top) and Reduction (Bottom) cells found for "Plunder" which achieves faster training than IMPALA-CNI Note the interesting use of 5x5 convolutional kernel sizes later in the cell.
Figure Left 2x2 Plot:
IMPALA-CNN RL-DARTS (Discrete Cell) Random Search 0.708
Avg. Normalized Reward
0.709
0.489
Table 4: Average normalized rewards across all environments w/ PPO. using the normalization method from
Cobbe et al. 2020). Full details and results (including Rainbow) are presented in Appendix D
We use random search "Classic" total size is x 10" see Appendix G.1. For fair comparison, we ensure total wall-clock time (with same hardware) stays equal, common in (Liu al.) 2019). Since in Table 3 a PPO supernet takes 2.5x longer to reach 25M steps, this is rounded to ran- dom search budget of cells to be trained with depths 16 3 for 25M steps. The best
important baseline to gauge the search space's difficulty (ex:
8

---

Under review as a conference paper at ICLR 2022
of the 3 cells is used for full evaluation. In Table 4 average, random search under- performs significantly, which also shows IMPALA-CNT is a strong hand-designed baseline. However, in Figure 8 RL-DARTS
Random Cell Performances (Dodgeball)
capable of finding architectures from scratch which outperform IMPALA-CNN on few environ- ments such as Plunder and Heist,
DARTS Discrete Cell
Random Cell Counts
IMPALA-CNI
40
while maintaining competitive per- formance on others. On the Rain- bow side, in Figure we found that in certain cases, even 100 unique ran- dom cells underperform against RL-DARTS!
30
10
8
10 12
Reward
4.5 TRANSFER EVALUATION
Figure 9: Histogram of 100 random cells' rewards on Dodgeball with Rainbow "Micro" search space (depths 64 3), with 95%
Instead of performing the RL-DARTS by joint training a supernet across infi- nite levels within only one game, one may ask if it is possible to joint train a supernet across all games over all infinite levels to find single transferrable cell, which may be of interest to practitioners seeking to find common architecture general to a broad class of RL problems. We confirm this approach using Rainbow, where learner performs gradient updates over actor replay data (with normalized rewards) from all 16 games.
substantially worse and 55% unable even train. More results in Appendix
Caveflyer
Fruitbot
Input
Relu
20
1.5
Relu
2
Relu
10
Ralu
Relu
Relu
Reward
4
Jumper
Joint Training on 16 Environments
Normalize Reward
DARTS-Sugermet IMPMLA-CNN
0.3
0.2
0.1
0.0
DARTS
-0.1
IMPALACON
0M
10M Environment Steps
20M
0M
20M
0M
20M
Environment Steps
Figure 10: Left 2x2 Plot: Evaluation of the discrete cell joint-trained over all environments using depths 6-1 x emphasize comparison differences. Right (Top): Discrete cell found. Right (Bottom): Average Normalized rewards all 16 games during supernet baseline training.
CONCLUSION
Our research has demonstrated that in a minimally invasive way, it entirely possible to apply DARTS RL, a completely different optimization paradigm than SL. which DARTS was originally proposed for. We believe this work can open new doors to both NAS and RL communities alike. Especially for theorists and analysts of DARTS, RL a completely new frontier for which to understand softmax routing and continuous relaxation techniques. Previous DARTS techniques (Wang et al.) 2021b Chu et al. 2020a Liang et al. 2019) may be tested in the RL setting, and new RL-specific ways of discretization and training may also be proposed, increasing RL-DARTS's s utility even more. For
RL practitioners, RL-DARTS could be important to finding better and more efficient (Cai et al.
2019) architectures for large-scale robotics (James et al., 2019 Akinola Akinoll et al. (021). transferable architectures in offline RL (Levine et al. 2020). as well as RNNs for memory (Pritzel et al. 2017 Kapturowski et al." 2019) and adaptation (Duan al. 2016 Wang al. 2017
Fortunato al. 2019

---

Under review as a conference paper at ICLR 2022
6 STATEMENTS
The only relevant potential concerns in our work oceur normally with general NAS methods, which can sacrifice model interpretability in order to achieve higher objectives. For the field of RI. specifically, this may warrant more attention in Al safety when used for real world robotic pipelines. Furthermore, with any NAS research, the initial phase of discovery and experimentation may contribute to carbon emissions due to the computational costs of extensive tuning. However, this is usually means end, such an efficient search algorithm, which this paper proposes with no extra hardware costs.
Ethics Statement:
We base our statement on the "NAS Best Practices Checklist" (Lindauer & Hutter 2019). recently released to improve reproducibility in neural architecture search.
Reproducibility Statement:
In terms of raw reproducibility, we discuss the explicit hyperparameters in Appendix F as well as search
space details in Subsection 4.1 The original DARTS implementation is readily available across Github (e.g. htt.hhtss:/// htts://ggttub.... com/ ccm/ quwrku darts as it is a highly popular algorithm. Similarly, Procgen is publicly available (https:/ /ttt:::://gitubb. con. / procgen) along with its training code and details (Cobbe et al. 2020). For our case, we used Ag-Att for PPO, available at (it hhtpps: / www. tensorf low oorg/agnts and Acme for Rainbow, available gtltsps://gie hub...... com/ deepmi pndp/min//////m a cme] Since algorithm is relatively simple assemble given preexisting pipelines, we believe that our approach should require minimal effort to reproduce.
In terms of comparing NAS methods, we indeed controlled for confounding factors by using the same hardware. as shown in Table For ablation studies, we provided large set of studies as seen throughout Subsections 4.2 and [.3 as well as the Appendix. We also ensured the same evaluation protocol via making sure all depths remain the same (16 for supernets, 64 for discrete cells, and for fine- tne-grained A/B testing). In terms of performance comparisons over time, we showed the performance of RL--DARTS's 's discrete cells over the search pracedure/ / supermet training in Figure We extensively covered the comparisons against random search in Subsection 4.4 over all 16 games well as large scale 100 random cell comparison in Figure For seeded runs, as standard in RI., we performed 3-seeded runs for all experiments.
layers for cheaper large-scale runs or D
terms of hyperparameter tuning, we discuss this in Appendix F and also Subsection 3.2 The time reported for all components of the NAS method can be found in Table 3 and we explicitly reported all of the details of our experiments throughout the paper.
REFERENCES
Alekh Agarwal, Ofer Dekel, and Lin Xiao. Optimal algorithms for online convex optimization with multi-point bandit feedback. In Adam Tauman Kalai and Mehryar Mohri (eds.), COLT 2010 The 23rd Conference on Learning Theory, Haifa, Israel, June 27-29, 2010, pp. 28-40. Omnipress, 2010.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), P.sss,,,rroceede Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA. volume 97 Proceedings of Machine Learning Research, pp. 151-160. PMLR, 2019. URL
mlr. press/ 2rres//v77. 9 ahaaddl 9a. 99tmmm
Iretiayo Akinola, Anelia Angelova, Yao Lu, Yevgen Chebotar, Dmitry Kalashnikov, Jacob Varley, Julian Ibarz, and Michael S. Ryoo. Visionary: Vision architecture discovery for robot learning. In ICRA, 2021.
Marcin Andrychowizz, Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin. Raphaël Marinier, Léonard Hussemot, Matthieu Geist, Olivier Pietquin, Marcin Michalski. Sylvain Gelly, and Olivier Bachem. What matters in poliicy reinforcement lcarning? large-scale empirical study. CORR, Abs/2206.0599) 2020.
05990
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, ICMI. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 449-458. PMLR, 2017. URL. http:://ppccee/// ing.inns................m......m bel 1elleer 7a. h
ml r. www.ppres///v
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. htLLitttt://::ttrrttet wet.tiiiiwweeeeeeeeeeee.eeeeeeee wttt:/oormm.d--A11BBB...
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation-based regularization. In Proceedings of the 37th International Conference Machine Learning, ICMI. 2020, 13-18
10

---

Under review as a conference paper at ICLR 2022
July 2020, Virtual Event volume 119 of Proceedings of Machine Learning Rexearch, pp. 1554-1565. PMLR, 2020. htLLhttt::/prrceedlngg mllr. http://............p./ss///o.//.
Xiangxiang Chu, Xiaoxing Wang. Bo Zhang, Shun Lu, Xiaolin Wei, and Junchi Yan. DARTS- robustly stepping out of performance collapse without indicators. CoRR, ahs/2009.01027, 2020a. URI. https?
aaar///////// //arxiv.org/ abs 2009.01027
Xiangxiang Chu, Bo Zhang, and Xudong Li. Noisy differentiable architecture search. CoRR, abs/2005.03566. 2020b. URL
org/ abs/ 2005. 03566
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair DARTS; eliminating unfair advantages in differentiable architecture scarch. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow; UK. August 23-28, 2020, Proceedings, Part XV, volume 12360 of Lecture Notes in Computer Science, pp. 465-480. Springer, 2020c. doi: 3- 030 58555- 6_28
10. 10.1007/978-3-11 0-77788-3-00--555555 6), 28.
/ttp: wtt.:/o..011111110000000000000/0
JD Co-Reyes, Yingjie Miao, Daiyi Peng, Esteban Real, Sergey Levine, Quoc V. Le,,, Honglak Lee, and Aleksandra Faust. Evolving reinforcement learning algorithms. In International Conference on Learning Representations (ICLR), 2021. URL URLhttts::// //rxii wwg/ httpg/abb/221111 03958
Karl Cobbe, Oleg Klimov, Christopher Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforeement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pp. 1282-1289. PMLR, 2019. URL
http::/proceeddnngg mlr press/ v97 www.bbeell....... htm
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp- 2056. PMLR, 2020.
Xuanyi Dong and Yang. Searching for robust neural architecture in four GPU hours. IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 1761-1770. Computer Vision Foundation / TEEE, 2019. doi: 10.1 20.11099/V...219.00001
URL http://openaccess..u wtweenaccesss..eet..... wwweteeeeee...... com/ wtw.eetccceccccccceeeeeeeeeeeeee
CVPR 2019/ntnal /Dong Searching For Rebust Neural Architecture in Four GPU. Hours 2019 wal__paperrr... html
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbecl. RIS Fast
reinforcement learning via slow reinforcement learning. CoRR, abs/1611.00279, 2016. URL http:// arxiv hwwpgg/bbs/1111 0111161171119999
I Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu. Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-leamer architectures. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüsssun. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1406- 1415. PMLR, 2018. URL http: http:///ooceeeeng....c .ml wwwps wwtp///eeeeeo....e
html
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards to automate reifforcmet leamning. In 6th ICMI. Workshop on Automated Machine Learning, 2019. URL htt.:httpe::/rxxi..rrr/////
905
07628
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charlic Deck, Joel Z. Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), ), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information P Processing Systems 2019, NewrlPS 2019, December 8-14, 20/9, Vancouver. BC, Canada, pp. 12448- 2019. URL httptttss:///cceeeeeeee
wnw.ripppp...... cc/ wwtp://apppp////. wtthh/2ed81222222 2ateee6@122200005faabbeee wdddee/fwwaaabaddeddaddd wee.edddddddaodd-ooooo www.:eaaaaastttttttttaaaa html
Jörg K. Franke. Gregor Köhler, André Biedenkapp. and Frank Hutter. Sample-efficient automated deep reinforcement learning. CoRR, abs/2009.01555. 2020. URL wtt.tttts:///xxtttee org/ abs/ 2209.. 01555
Adam Gaier and David Ha. Weight agnostic neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox. and Roman Gamett (eds.). Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 5365-5379, 2019.
11

---

Under review as a conference paper at ICLR 2022
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro. Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Neal Wu, Efi Kokiopoulou, Luciano Sbaiz, Jamie Smith, Gábor Bartók, Jesse Berent, Chris Harris, Vincent Vanhoucke, and Eugene Brevdo. AFeggtt::: A library for reinforeement learning in tensorflow, bbtpss: httg///gttt..ccoo www.//eenooeeo oo flow//www//o /ww/agent. agents 2018. URLpttts:://g1tme com/
(tensorflw/ s [Online; accessed 25-June-2019).
Tuomas Haarnoja, Aurick Zhou, Pieter Abheel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssen, Stockholm, Sweden, July 10-15, 2018. volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMIR, 2018. URL http: http:://prrceeeee ngs. .m www.pr/ss//ee.....
ja 8b. html
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforeement learning that matters. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3207-3214, AAAI Press, 2018
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul. Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Sheila A. Mcllraith and Kilian Q. Weinherger (eds.). Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (1AAI-18), and the 8th AAAI Symposium Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. pp. 2215-3222. AAAI Press. 2018
Matt Hoffman, Bobak Shahriari, John Aslanides, Gabriel Martn, Feryal Behbahani, Tamara Norman, Abbas Abdolmaleki, Albin Cassirer, Fan Yang, Kate Baumli, Sarah Henderson, Alex Novikov, Sergio Gómez Colmenarejo, Serkan Cabi, Caglar Gulcehre, Tom Le Paine, Andrew Cowie, Ziyu Wang, Bilal Piot, and Nando de Freitas. Acme: A research framework for distributed reinforcement learning. arXiv preprint arXiv:2006. 20K,::9000000979, 2020. URL (alttttt///aaaat orw. / abs 20..0000000000000 00979
Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Yujing Hu, Weixun Wang, Hangtian Jia, Vixiang Wang, Yingfeng Chen, Jianye Hao, Feng Wu, and Changjie Fan. Learning to utilize shaping rewards: A new approach of reward shaping. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (cds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 1- 2020, virtual, 2020. URL BtRLbttps: http://pcceeeeenneoeeee
Andrew Hundt, Varun Jain, and Gregory D. Hager. sharpdarts: Faster and more accurate differentiable architecture search. CoRR, abs/1903.099900 09900, 2019. URL/attp://axxi wwwpxii.orrg//aeo abs /aab////99.. 1903. 09900
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. A closer look at deep policy gradients. In 8th International Conference on Learning Representaations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL lattos://cpenreviie net/ lttrmmmiddxyyxdeee..
Max Jaderberg, Valentin Dalibard. Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Popula- tion based training of neural networks. CoRR, abs/1711 1.0944, 2017. http: wtgpaiss/rr///////
1 09846
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized--o-ccanoiical adaptation networks. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20, 2079, pp. 12627-
12637. Computer Vision Foundation / IEEE, 2019. doi: 10. 1001109CCVPR..219900029 //openaccess. twweeheeecccccc com conte www.eaacccsss..cct...... hgr
URL pttpt
(tte_2219//mmmmmm /tapt//tmmmammma//a/ea/aaaa Sim- 1 To- Real siiiaaliiiiiiaa
Sim-To-Sin Sim--o--Smmaaaaaaaaaaaea Daataaaaaaaaaaaaaa -eataaakfffiinntt cocetttt_oeciiicooo Gra.pr wigviiaavvaaaao halivviaaaaaaaon Randor zed To-Connn cal
Adaptation Sdaptattio_Nettorkk.. CVPR wttp:p__111_ppperhhaa__
12

---

Under review as a conference paper at ICLR 2022
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. Scalable deep reinforcement learning for aision-based robotic manipulation. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 651-673. PMLR, 2018, http:/ httt::/ppoccee..nggg/
mlr. gress/v8 7 www.kaaaaaaaaaaaaaaaa kovl8a html
Steven Kapturowski, Georg Ostrovski, John Quan, Rémi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning, In 7th Internutional Conference on Learning Reprexentations. ICLR 2019. New Orleans, LA. USA, May 6-9, 2019. OpenReview.net 2019. URL httts:// net/ forum?iddrrlyyjaaaae
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. CoRR, abs/2004. 13649, 2020. htLL/tttss//axxiv http//awwvv..ga.aaaaa///a /bs/ 2004. 1004413649
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning Tutorial, review, and perspectives on open problems. CoRR, ftss/2005.01643 2020.
/
2005. 01 643
Guohao Li, Guocheng Qian, Itzel C. Delgadillo, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. SGAS: sequential greedy architecture search. In 2020 IEEE/CVF Conference on Computer Vision and Patter Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 1617-1627. IEEE, 2020. doi: 1099 CVPR42660.2020.0016 URL.tttps:///o..... oog/ 10. 1wt1/1100///260000 2020. 00169
Liam Li, Mikhail Khodak, Nina Balean, and Ameet Talwalkar. Geometry-aware gradient algorithms for neural architecture search. In 9th International Conference on Learning Reprexentations, ICLR 2021, Viriual Event, Austria, May 3-7, 2021. OpenReview.ner, 2021. URL URL htttp::/ www.n/ett///rr/...e id=
MuSYkal 1hxRP
Hanwen Liang. Shifeng Zhang, Jiacheng Sun, Xingqiu He, Weiran Huang, Kechen Zhuang, and Zhenguo Li. DARTS+: improved differentiable architecture search with early stopping. CoRR, abs/1909.06033 99009, 2019. URL htt.httpp//axii..ata
Marius Lindauer and Frank Hutter. Best practices for scientific research on neural architecture search. CoRR, abs/1909.02453, 2019. URL /Rtp:ptt://axii http://awwwwv....aaaaaaaa abs/ 19099.99..... 02453
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan L. Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.). Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, volume 1 11205 of Lecture Notes in Computer Science, pp. 19-35. Springer, 2018 doi: 10.00077978-3--30--00266-5.. 2. URL |atpp! 0124000--12665----
1007/ 9778
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Yan Liu. Neural architecture optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa- -isacBianchi, and Roman Gameet (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7827-7838, 2018 URL http://pppp//pooeeeeiiiii ngs www.:/eerennnnnnnn.nne........ meurips. http:/papr/211///heepp 993670f1acc3aa969332222 3tt.aa/aaaaaaaa3333332aaaaaa/e/r htw.aaaaaa.bbbbtttttttttt...
Joseph Mellor, Jack Tumerr Amos Storkey, and Elliot J. Neural architecture search without training.
CoRR, abs/2006.04647, 2020. URL
64647
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL
1112.. 5602
Jack Parker- Holder, Vu Nguyen, and Stephen J. Roberts. Provably efficient ouline hyperparameter optimization with population-based bandits. In Hugo Larochelle. Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and HSsuan-Ticn Lin (cds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 621 2020, virtual, 2020.
Jack Holder, Vu Nguyen, Shaan Desai, and Stephen J. Roberts. Tuning mixed input hyperparameters on the fly for efficient population based autorl. CoRR, abs/2100.11l (1006.18888 15883, 2021a. URL htt.s: httpp::/arxiv..rrt abs / 066 15883
13

---

Under review as a conference paper at ICLR 2022
Jack Parker-Holder. Vu Nguyen, Shaan Desai, and Stephen J. Roberts. Tuning mixed imput hyperparameters on the fly for efficient population based autorl. CoRR, abs/2106. 15883, 2021b. URLhtttss:/arxiv..or abs/2106. 15883
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 4052-4061. PMLR, 2018. URL http
\/proceecings. mlr. prwss vv0 ht////araaaaa raaarmar18e 1ht..
Hicu Pham, Melody Y. Guan, Barret Zoph, Quoe V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Jennifer G. Dy and Andreas Krause (eds.), Pruceedings of the 35th International Conference Machine Learning, ICML 2018, Stockholmsmüsssen, Stockholm, Sweden, July 10-15, 2018. volume 80 of Proceedings of Machine Learning Research. pp. 4092-4101. PMLR, 2018. URL http: pproseediigs mlr..rres. v80 pham1 8a. html
Alexander Pritzel, Benigno Uria, Sriram Srinivasan. Adrià Puigdomènech Badia, Oriol Vinyals. Demis Hassabis. Daan Wierstra, and Charles Blundell. Neural episodic control. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017 Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1827-2836 PMLR, 2017.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. CoRR, abs/2102. 10330, 2021. URL httph:/ts/ http:////wwwwww..///boeeeee/
10330
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmentation for generalization in deep reinforcement learning. CoRR, abs/2006. 12862, 2020. hctps:////////xi
org/abs 12862
Kanishka Rao, Chris Harris, Alex Irpan, Sergey Levine. Julian Ibarz, and Mohi Khansari. Rl-cyclegan: Reinforcement learning aware simulation-to-real. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 11154-11163. IEEE, 2020. doi: 10.1 (ttp///YY22222..200000.0 117. /tttttts:: //ddttts:://::::. .org/ 1199///PR42 2020. 01117
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. Int. Comput. Vis., 115(3)211122 252, 2015. doi: 10.1007/s 1211-1256-0112111
/RR/Attps:///c..orr crg. 100 100001000111331
0816-y
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as scalable alternative reinforcement learning. CoRR, abs/1 700643844, 2017. URL/bttp://arxxi.or wwg. abs abs/1/03 11011 03864
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http:: org/ /bs/
1515205952
John Schulman, Filip Wolski, Prafulla Dhariwal, Alcc Radford, and Oleg Klimov. Proximal policy optimization 16330006477
algorithms. CoRR, ftd/1700.06447 2017.
Xingyou Song. Yiding Jiang, Stephen Tu. Yilun Du and Behnam Neyshabur. Observational overfitting in reinforcement learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevicw.net, 2020. Uttltttppa///ppenssiiss net forum?
5d-HJ1i 2hNKDH
Xingyou Song. Krzysztof Choromanski, Jack Parker-Holder, Yunhao Tang, Daiyi Peng, Deepali Jain, Wenbo Gao, Aldo Pacchiano, Tamás Sarlós, and Yuxiang Yang. ES-ENAS: combining evolution strategies with neural architecture search at no extra cost for reinforcement leareinng CoRR, abs/2101.07415, 2021. URL
https: http///axvvvv//// org. abs/
Kenneth O. Stanley and Risto Mikkulainen. Efficient reinforcement learning through evolving neural network
topologies. In William B. Langdon, Erick Cantú-Paz, Keith Mathias, Rajkumar Roy, David Davis, Riccardo Poli, Karthik Balakrishnan, Vasant G. Honavar, Günter Rudolph, Joachim Wegener, Larry Bull, Mitchell A. Potter, Alan C. Schultz, Julian F. Miller. Edmund K. Burke, and Natasa Jonoska (eds.), GECCO 2002: Proceedings of the Genetic and Evolutionary Computation Conference, New York, USA, 9-13 July 2002, pp. 569-577 577. Morgan Kaufmann, 2002.
Kenneth O. Stanley, David B. D' D'Ambrosio, and Jason Gauci. A hypercube- encoding for evolving large-scale neural networks. Artif. Life, 15(2):185-112 2009. doi: 16.1 200. URL. https::/ddo wwwpw/..rr1..... 1162/art1 2009. 15.2. 15202
14

---

Under review as a conference paper at ICLR 2022
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1801.0000 .00690, 2018.
http://orrrrabaa/aaeeaae 00690
Huiqun Wang. Ruijie Yang. Huang, Yunhong Wang. idarts: Improving DARTS node nommalzaaiion and decorrelation discretization. CoRR, abs/2108.1 11014, 2021a. htRl//ttps.s/aarxx....//////////
11044
Jane Wang, Zeb Kurth-Nelson, Hubert Soyer, Joel Z. Leibo, Dhruva Tirumala, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matt M. Botvinick. Learning to reinforcement learn. In Glenn Gunzelmann, Andrew Howes, Thora Tenbrink, and Eddy J. Davelaar (eds.)) Proceedings of the 39th Annual Meeting of the
Cogmitive Science Society, CogSei 2017, London, UK, 16-29 July 2017. cognitivexceenceocieteycco pttp /dmoodddee ndmode ing. org/ /
2017.
ndex.htm]
Ruachen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsieh. Rethinking architecture selection differentiable nas. In International Conference on Learning Representations, ICLR 2021. OpenReview.net, 2021b.
Ziyu Wang, Tom Schaul, Mattco Hessel, Hado van Hasselt, Mare Lanctot, and Nando de Freitas. Ducling network architectures for deep reinforcement learning. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 1995-2003. JMLR.org. 2016.
Colin White, Arber Zela, Binxin Ru, Yang Liu, and Frank Hutter. How powerful performance predictors neural architecture search? CoRR, abs/2104.01177, 2021. /RLLitttps//ccxvv.oors
01177
Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicoló Cesa- Bianchi, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 2402-2413, 2018. [Rtsssss:://poooo.o. nnuu neur ips..pp... ww/pap/ppeeeeeeeeeeeeeeee.e /2018/ha0hhh 7715558833759993344233333335
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado Hasselt, David Silver, and Satinder Singh. self-tuning actor-critic algorithm. In Advances Neural Information Processing Systems, 2020.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakchi, Thomas Brox, and Frank Hutter. Under- standing and robustifying differentiable architecture search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababu, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htttss://qpenzevieww net. www.foorrriiiimmo -or.
Baohe Zhang, Raghu Rajan. Luis Pineda, Nathan O. Lambert, André Biedenkapp, Kurtland Chua, Frank Hutter. and Roberto Calandra. On the importance of hyperparameter optimization for model-based reinforcement learning. CoRR, 132.5 2021. URL https://axxxx http://wwwwvvvrrrrrr/reeree wttpg//b/b/////// 23651
Barret Zoph and Quoe Le. Neural architecture search with reinforcement learminng CoRR, abs/1611.01158.. 2016. /ttp://///axx.. URL.attpr//aaxxrrrrrrr/////11111
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V. Le. Learning transferable architectures for
scalable image recognition. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 8697 8710. IEEE Computer Society, 2018. doi:
URL ht tp: http://ppppaccccce/ee theevf.... com cntt eetpeeettttt__teeeeeeeeeeee 2018 html Zow_nooo__Laiinnggggee Trans wwwpaeeaaaaaaaaaaaaaaaaaeeeeeeee
10. 10.11000 (111////V..211..000000
CVp 2018 paper. html
15

---

Under review as a conference paper at ICLR 2022
APPENDIX
A REBUTTAL
A.1 TRAINING PROCEDURE (EXTENDED)
Below are PPO (Schulman et al. 2017) and Rainbow-DQN (Hessel et al. 2018) RL-DARTS variants. Exact loss definitions and data collection procedures can be found in their respective papers.
Algorithm 2: RL-DARTS with PPO
Supernet training:
Setup supernet encoder Je with weights Og.
Initialize policy and value head projection weights Wa E Rd W. e Ed4
Collect all trainable weights 0 {6e,, W=, W.).
Setup policy (s) ~ softmax W (s))
Setup value function (s) = W. (s).
Define standard PPO loss L(0, a)) using Te.. and Vo...
Perform PPO training via collecting data from TØ,a and SGD with Vo,a
Collect a from previous training procedure.
Discretization:
Let 5(a²)) be the discrete cell constructed via Algorithm 4
Evaluation:
Setup discretized cell encoder fe..((a')-
Initialize policy and value head projection weights W E R4.A W e { W W").
Collect all trainable weights
Setup policy (s,5(a-)(8) ~ softmax(W" (())
Setup value function (o. W (s).
Define standard PPO loss L using TT,5(ar) and Vo,5(a*).
Perform PPO training via collecting data from TT,((a*) and SGD with V VaCs(ar)
Report final policy reward.
Algorithm 3: RL-DARTS with Rainbow. Note that we do not noisy nets in this implementa-
tion.
Supernet training:
Setup supernet encoder fo..a with weights 0z.
Initialize dueling network projections W,, E RRd,1 W. E RdMM
Collect all trainable weights 0 = (e,, W., Wa).
Setup value network Va, W. (s).
Setup advantage network Ag (3, = W. (s).
Setup Q-network Qu.o a) = (s) + (s, TAT Eves Aa as)
Define standard Rainbow loss L(0, a)) using Q0.a-
Perform Rainbow training via collecting data from Qa,a and SGD with V C(0, a). Collect a* from previous training procedure.
Discretization
Let 5(a") be the discrete cell constructed via Algorithm 4
Evaluation
Setup discretized cell encoder f(a.) with weights de.
Initialize dueling network projections W E [Rd,1, W E R414
Collect all trainable weights
{ W W?
Setup value network (a.a(ar)(s) = W (s)))(a))s))
Setup advantage network Ag.5(a.) (8,, = W (o.)(a))(s)
Setup Q-network
Q0. a) = ((. + (s, [ En'e A
Define standard Rainbow loss (((a))())) using Qo,āaa))-
Perform Rainbow training via collecting data from Q0.((a)) and SGD with V VoLa(an)())
Report final policy reward.
16

---

Under review as a conference paper at ICLR 2022
Algorithm 4: Discretization Procedure.
Argmax:
For (i,j) across all edges:
Define edge strength Wa,j maxoo0, oz Po
[i:5]
Define edge op ((i)) arg 0a0 Po (5.5)
Prune:
For node in all intermediate nodes:
Sort input edge weights Wh., 2 W/z.j 2 ...
Retain only top K edges (11,J),.... (ik, (ik.j) and corresponding ops ((id)ss ( in final cell.
Note that both RL-DARRS procedures can also be summarized in terms of raw code as simple one-line edits to the image encoder used (compressing the rest of the regular RL training pipeline code):
def train (faastte____cooderr
"""Initial RL algorithm setup"""
extra_ variables = Wrap(feature_encode
all all_traimale__ariabb
@eeaturr_encoder- [
extra_variables)
"""Rest of RL algorithm setup"""
apply_gradients (loss, all all_trainable, (aaiiaabbeeaaribbessss__sl
Thus, the 3-step RL-DARTS procedure from Section 3.2 can be seen as:
DARTSSuperNet MakeSuperNet (ops, nun_nodess Setup
train (DARTS)uperNet)) & Supernet training
DiscretizedNet DARTsSuperNet. discretize () Discretization
train (DiscretizedNet. # Evaluation
A.2 POSSIBLE IMPROVEMENTS TO RL-DARTS
These include:
1. Discretization Changes: One may consider discretization based on the total reward J (TT,..)) a.). which may provide a better signal for the correct discrete architecture. This is due to the fact that the relative strengths of operation weights from a may not correspond the best choices during discretization. (Wang et al. (021b) considers iteratively pruning edges from the supernet based on maximizing validation accuracy changes. For RL, this would imply variant of discretization dependent multiple calculations of J[TT 5, (ar) J [ .2(a*)) where 8* consists the weights obtained during supernet training, as well as fine-tuning J (To- (a=))) at every pruning step. These changes, in addition to the inherently noisy evaluations of J+),, greatly increase the complexity of the discretization procedure, but are worth exploring future work.
2. Changing the Loss Regularization: Throughout this paper, we have found that vanilla DARTS able train by simply optimizing a with respect the loss, even though in principle, the loss is not strongly correlated to the actual reward in RL. Thus, it is curious to understand whether loss-based metrics or modifications may help improve RL-DARTS. One such modification is based on the observation that certain RL losses may not be required for training a. PPO, the entropy loss of TT may not be necessary or useful for improving the search quality of ax, and thus it may be better to perform two step update by providing a different loss for a. One may also consider searching for two separate encoders via two supernets, since both PPO and Rainbow feature separate networks, e.g. the policy mo, and value function Vol.oz for PPO and advantage function Ag, and value function Vil for Rainbow.
17

---

Under review as a conference paper at ICLR 2022
3. Signaling Metries and Early Stopping: Observing metrics throughout training allows for early stopping, which can reduce search cost and provide better discrete cells. This includes metrics such the strength of certain operation weights (Lingg et al.
2019) well the
Hessian with respect to a throughout training, i.e. C2 a) as found in /ela et al. 2020). Furthermore, inspired by performance prediction methods (Mellor al. 2020 Luo et al.) 2018), one may analyze metrics such as the Jacobian Covariance, via the score defined
to be the -E [log(a; + E) + (o; + =] where = 10-8 is stability constant and 0n ... 0B are the eigenvalues of the correlation matrix corresponding to the Jacobian J 01 dsB
0f T
with B input images (81.....88).
De1
This metric/predictor has been found to be strong signal for accuracy in SL NAS among many previous predictors (White et al. (02T). However, for the RL case, just like the loss, the mentioned metrics must be defined with respect the current replay buffer D, and thus raises the question of what type of data is to be used for calculating these metrics. When using reasonable variant where the data collected from pretrained policy, we find that methods such Jacobian Covariance do not provide meaningful feedback as shown in Figure RI.
Cell Performance Prediction
0.6
Cell Performance
0.5
0.4
0.3
0.2
0.1
0.0
1100016
1000014
100012
-100010
-100008
Cell Jacobian Covariance Score
Figure R1: Attempts to predict random cell performances
Dodgeball environment by using the Jacobian
covariance predictor. Jacobian covariance scores are computed as in (Mellor et al.) 2020) by evaluating the
networks (using the cells with depths 64 x 3) on mini-batches of B 16 random images sampled from the Dodgeball environment using pretrained policy's trajectories. For each cell, Jacobian Covariance scores across mini-batches are averaged, and the performance on the Dodgeball environment normalized by the maximum possible return.
4. Supernet Training: As seen from the results Figure 7 (main body) and Appendix C.3 there is correlation between supernet and discrete cell performances. However, this is affected by the environment used as well as integrality gaps between the continuous relaxation and discrete counterparts, and thus further exploration is needed before concluding that improving the supernet training leads to better discrete cell performances. In any case, reasonable methods of improving the supernet can involve DARTS-agnoshe modifications to the RL pipeline, including data augmentation (Kostrikov et al. 2020 Raileanu et al. 2020 well online hyperparameter tuning Parker-Holder et al.
2020
2021a). Simple
hyperparameter tuning (e.g. on the softmax temperature for calculating pa effective.
(1.)) ss) also can be
A.3 DM-CONTROL WITH SAC (EXTRA BENCHIARRK) RESULTS
We also include results via using a Soft Actor Critic (SAC) HHarrojja et al. 2018 variant of RL-DARTS on DM-Control (Tasa et al. 2018) with image-based observations resized to 64 x 64 with a frame-stacking of 3. The code can be found in https:: /ectps://qitaan //ghtttthuuuu com/ where disabled the predictive information loss to only reg- ular SAC. The baseline architecture a 4-layer convolutional architecture found in https::/ gi t.ub. www.co//////mmmm wtt.ccog//oooooooooooooooooooooo e reaea meaearch/ppl http:/sssaaa////aa
rgogle-reeearch/
Both
18

---

Under review as a conference paper at ICLR 2022
4, K 1 using the "Micro" search space, DARTS supernet and discrete cells use N = 3, with convolutional depths of 32 to remain fair to the baseline.
Our results in Figure R2 thus also provide evidence of RL-DARTS's flexibilitty with yet a 3rd algorithm (SAC), as well as on continuous control, where discrete cells outperform the baseline.
cheetah run
finger spin
hopper stand
walker stand
600
500
800
800
800
600
600
600
300
400
400
200
400
200
100
200
DARTS Discrete
1.0
100000
200000
0.0
0.5
1e6
Steps
Figure R2: Results with RL-DARTS discrete cells on DM- Control when using the SAC algorithm.
19

---

Under review as a conference paper at ICLR 2022
B WHAT AFFECTS SUPERNET TRAINING?
Given the positive training results we demonstrated the main body of the paper, one may wonder, any supernet, no matter how poorly designed setup, still train well in the RL setting? If so, this would imply that the search method would not be producing meaningful, but instead, random signals.
We refute this hypothesis by performing ablations understanding of what components affect its performance. We ultimately show that the search space and architecture variables play a very significant role in its optimization, thus validating our method.
our supernet training in order have better
B. ROLE OF SEARCH SPACE
We remove the ReLU nonlinearities from the "Classic" search space, that O 33onv3x3, Conv5x5, Dilated3x3, Dilated5x5) and thus DARTS cell consists only linear operations. shown in Fig. [1] this leads to dramatic decrease in supernet performance, providing evidence that the search space matters greatly.
Search Space Comparisons (Dodgeball)
10
Classic
Supernet Reward
Classic (No ReLU)
8
6
4
2
0M
25M
50M
Environment Steps
Figure 11: Supernet training using PPO on Dodgeball with infinite levels, when using the "Classic" search space with/without ReLU nonlinearities, under the same hyperparameters.
B.2 UNIFORM ARCHITECTURE VARIABLES
We further demonstrate the importance of the architecture variables a on training. We see that in Fig. 12, freezing a to be uniform throughout training makes the Rainbow agent unable to train at all. This suggests that it is crucial for properly route useful operations throughout the network.
Trainable a Comparisons (Dodgeball)
Trainable
Supernet Reward
Uniform
3
1
OM
10M
20M
Environment Steps
Figure Supernet training using Rainbow on Dodgeball with infinite levels, when using the "Micro" search space with/without trainable architecture variables o, under the same hyperparameters.
C WHAT AFFECTS DISCRETE CELL PERFORMANCE?
C.1 SOFTMAX WEIGHTS VS DISCRETIZATION
As seen from Figure in the main body of the paper, the DARTS supernet strongly downweights Rell operations when using the "Classic" search space with PPO. In order verify
20

---

Under review as a conference paper at ICLR 2022
the predictive power of the softmax weights, proxy, thus also performed evaluations when using purely 3x3 or 5x5 convolutions on large IMPALA-CNN with 64 depths. We see that the Conv3x3 setting indeed outperforms Conv5x5, corroborating the results in which during training, strongly upweights the Conv3x3+Rel.U operation and downweights Conv5x5+Rel.U.
Docgeball INF Levels
Dodgeball 200 Levels
20.0
16
17.5
14
15.0
12
12.5
Reward
10.0
8
1.5
4
2.5
2
Conv3x3
0
0M
25M
50M
0M
25M
50M
Environment Steps
Figure 13: Large IMPALA-CNNs evaluated on Dodgeball using either Infinite or 200 levels with PPO. For the 200 level setting, lighter colors correspond to test performance.
C.2 DISCRETE CELL EVOLUTIONS
Along with Figure in the main body of the paper, we also compare extra examples of discretizations before and after supernet training, to display reasonable behaviors induced by the trajectory a
In Figure 14 we use PPO with the "Classic" search space, but instead use (N, R, 7) (1,0, 0, 6) along with outputting the last node (instead of concatenation with a Conv for the output) allow larger normal cell search space and graph topology. In Figure [5] the discretized cell initially uses large number skip connections as well as dead-end nodes. However, at convergence, eventually utilizes all nodes to compute the final output. Curiously, we find that the skip connection between the input and output appears commonly throughout many searches.
Figure 14: Comparison of discretized cells before and after supernet training, on Starpilot using PPO with I = 6 nodes.
Figure 15: Comparison of discretized cells before and after supernet training, on Plunder using PPO with nodes.
For the Rainbow setting, in Figure 6 in the main body of the paper, saw that when the search process successful, the supernet's s training trajectory induces discretized cells which improve evaluation performance well. The cells discovered later generally perform better than cells discovered earlier in the supernet training process. In Figure 16 we show more examples of such evaluation eurves.
21

---

Under review as a conference paper at ICLR 2022
Rigfish Call Fvaluations
Rossfight Cell Fvalustions
Coinrun Fvaluations
Ninja Fvaluations
24
5.4
3
10
2
6
1
0
5
12
2
3
7
2
in Order
Figure 16: Evaluated discretized cells discovered throughout training the supernet with Rainbow. To save computation, we evaluate every 2nd cell that was discovered.
C.3 CORRELATION BETWEEN SUPERNET AND DISCRETIZED CELLS
Given that the discretized cell only explicitly depends on architecture variables a and not necessarily model weights 0, one may wonder: Is there relationship between the rewards of the supernet and of its corresponding discretized cell? For instance, the degeneente/umderperoommeee setting mentioned in Section 4.4 and Appendix can be thought of an extreme scenario. At the same time, there could be an integrality gap, where there the discretization process 5(()) produces cells which give different rewards than the supernet.
In order to make such an analysis comparing rewards, we first must prevent confounding factors arising from Rainbow's natural performance on an environment regardless of architecture. We thus first divide the supernet and discrete cell scores by the score obtained by the IMPALA-CNN baseline, where the baseline and discrete cells all used depths of 64 3.
In Figure 17 when using Rainbow and observing across environments, we find both high correlation and also integrality gaps: for some environments such Ninja and Coinrun, there is significant cor- relation between supernet and discrete cell rewards, while for other environments such as Dodgeball, there is significant gap. This suggests that search quality can be improved via both better supernet training such as using hyperparameter tuning or data augmentation (Raileanu et al. [2020). as well as better discretization procedures such as early stopping and stronger pruning (Chu et al.) 2020a Liang
et al. 2019
Wang et al. 2021b).
Supernet and Discrete Cell Relationship
1.4
Reward (Normalized by IMPALA-CNM (N-CNN)
IMPALA-CNN
Supernet
1.2
Discretized
1.0
0.8
0.6
0.4
0.2
0.0
Bigfish
Bossfight
Caveflyer
Chaser
Climber
Coinrun
Dodgeball
Fruitbot
Heist
Jumper
Leaper
Maze
Miner
Ninja
Plunder
Starpilot
Figure 17: Supernet and their corresponding discrete cell rewards across all environments Procgen using Rainbow, after normalizing using IMPALA-COE performances. Thus, the black dashed line at 1.0 corresponds
to IMPALA-CNN
22

---

Under review as a conference paper at ICLR 2022
D NUMERICAL SCORES
In Tables and 5b we display the average normalized reward after 25M steps, as standard in Procgen (Cobbe al. 2020), for subset of environments which RL-DARTS performs competitively. The normalized reward for each environment is computed as Roorm (R Rmin [Rmaa Rmin) where Raar and Rmin are calculated using combination of theoretical maximums and PPO-trained agents, and can be found in (Cobbe et al. 2020).
Env
IMPALA-CNN Baseline
RL-DARTS (Discrete)
Random Search
Bigfish
0.60
0.60
0.42
Bossfight
0.75
0.73
0.81
Caveflyer
0.75
0,47
0.85
Chaser
0.71
0.55
0.15
Climber
0,69
0.90
0.61
Coinrun
0.91
0.53
0.8
Dodgeball
0.53
0.59
0.29
Fruitbot
0.92
0.93
0,83
Heist
0.72
0.89
0.38
Jumper
0.62
0.76
1.0
Leaper
0.2
0.28
-0.28
Maze
1.0
1.0
0.0
Miner
0.74
0.85
0.70
Ninja
0.87
0.69
0.38
Plunder
0.57
0.76
0.43
Starpilot
0.71
0.73
0.40
(a) PPO + Classic
Env
IMPALA-CNN Baseline
RL-DARTS (Discrete)
Random Search
Bigfish
0.71
0.65
0.60
Bossfight
0.54
0.48
0.45
Caveflyer
-0.05
-0.03
-0.01
Chaser
0.30
0.26
0.22
Climber
-0.04
-0.02
-0.05
Coinrun
0.06
0.21
0.15
Dodgeball
0.57
0.59
-0.05
Fruitbot
0.68
0.68
0.70
Heist
-0.48
-0.49
-0.47
Jumper
0.21
0.18
0.17
Leaper
-0.07
-0.07
-0.03
Maze
0.76
0.74
0.51
Miner
0.35
-0.03
0.11
Ninja
0.03
0.13
0.28
Plunder
0.14
0.02
0.03
Starpilot
0.91
0.80
0.76
(b) Rainbow + Miero
Table 5: Normalized Rewards in ProcGen across different search methods, evaluated 25M steps with depths
64 x 3. Largest scores the specific environment (as well values within 0.03 of the largest) bolded.
23

---

Under review as a conference paper at ICLR 2022
E EXTENDED LARGE SCALE STUDY ON RANDOM CELLS
For some environments such as Dodgeball and Maze, the gap between DARTS and random search can be dramatic, For each game, we trained 100 unique random cells using Rainbow the "Micro" search space with depths 64 x 3 and found that all of the random cells underperformed against DARTS in Figure 18
Random Cell Performances (Dodgeball)
Random Cell Performances (Maze)
DARTS Discrete Call
20
DARTS Discrete Coll
Random Cell Cell Counts
IMPALA-CNN
Random Cell Counts
IMPALA-CNN
15
10
20
4
10
Reward
Reward
Figure 18: Histogram of the normalized rewards of 100 random cells evaluated on Dodgeball Maze, using the "Micro" search space.
F HYPERPARAMETERS
In our code, we entirely use Tensorflow 2 for auto-differentation, as well as the April 2020 version of Proegen. For compute, either used P100 or V100 GPUs based convenience and availability. Below are the hyperparameter settings for specific methods. For all training curves, we use the common standard for reporting in RL (i.e. plotting mean and standard deviation across seeds).
F.1 DARTS
Initially, we sweeped the softmax temperature in order to find a stable default value that could be used for all environments. For PPO, the sweep across the set (5.0, 10.0, 15.0]. For Rainbow, the sweep was across (10.0, 20.0, 50.0}.
For tabular reported scores in Figures 8 and 5b
we used a consistent softmax temperature of 5.0 for
PPO, and 10.0 for Rainbow.
F.2 RAINBOW-DON
10-5 We Acme (Hoffman et al. 2020 for code infrastructure. We learning
batch size 256, n-step size 7, discount factor 0.99. For the priority replay buffer (Schaul et al. 2016). we use priority exponent 0.8. importance sampling exponent 0.2, replay buffer capacity 500K. For particular environments (Bigfish, Bossfight, Chaser, Dodgeball, Miner, Plunder, Starpilot), we use n-step size and replay buffer capacity 10K. For C51 (Bellemare et al. (007], we use 51 atoms, with Umin = 0, Umaz 1.0. As preprocessing step, we normalize the environment rewards by dividing the raw rewards by the max possible rewards reported in (Cobbe et al. 2020
F.3 PPO
We use TF-Agents (Guadarrama et al. 2018) for our code infrastructure, along with equivalent PPO hyperparameters found from (Cobbe et | (020). Due necessary changes in minibatch size when applying DARTS modules networks with higher GPU memory usage, thus swept learning rate across {1 10 2.5 x 10-4, 4,5 x 10 10-4) and number of epochs across {1, 2,3..
maximum power of minibatch size before encountering GPU out-of- memory issues on a standard 16 GB GPU. Thus, for a 16 3 = [16, 16, 16] DARTS supemet, we set the minibatch size to be 256, which is also used for evaluation with 64 x [64, 64, 64) discretized CNN. Our hyperparameter gridsearch for the evaluation led to an optimal setting learning rate 10-1 and number of epochs 1.
For all models,
24

---

Under review as a conference paper at ICLR 2022
G MISCELLANEOUS
G.1 SEARCH SPACE SIZE
Let Ons | 1, the number of non-zero ops in O. For cell (normal reduction), the first intermediate node can only be connected to the input via single op, and thus the choice is only Onz However, later intermediate nodes use K = 2 inputs which leads to choice size of O x (:) where the index of the intermediate node. Thus the total number of possible discrete cells One Il (Omz (k)).
For the "Classic" search space, there are both normal and reduction cells to be optimized. with number non-zero normal ops Ons.N and number non-zero reduction ops One,R = with
I= 4, K = for both. This leads to total configuration size of Onn,N II (O x (2)) x
[Ons. IT (O x (2) x 10a.
25


Reviews:
[{"review_id": "zx5ljceuSTq", "reviewer": "Reviewer_mu9V", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors give a nice overview of common issues of using DARTS within the RL framework, specifically compared to supervised learning settings. An interesting point is the non-stationarity of the collected data, which depends on the initialization, which does not occur in SL.\n\nI found the explanation of the training procedure a bit unclear, and it would be beneficial to focus on this more, as this is one of the main contributions.\n\nThe performance does not seem to be better compared to the Vanilla benchmark in 4.2. I would like to see more discussion why this happens and maybe what are some potential improvements to supernet training and discretisation that could improve the overall performance.\n\nSome relevant references which I think should be discussed, in order to provide better context for the proposed approach:\n  - Luo, Renqian, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. \"Neural architecture optimization.\" arXiv, 2018.\n  - Stanley, Kenneth O., David B. D'Ambrosio, and Jason Gauci. \"A hypercube-based encoding for evolving large-scale neural networks.\" Artificial life, 2009\n", "summary_of_the_paper": "This paper studies the use of DARTS within the RL setting. Specifically, it investigates how DARTS can optimize perception modules for RL environments. RL-DARTS is optimised in a typical RL setting, using the standard RL loss function, in an end-to-end manner.\nThe proposed approach is evaluated using the Procgen benchmark, and compared to the IMPALA-CNN architecture. Several ablation studies are performed to further examine the contributions of the components of the proposed approach.\n", "summary_of_the_review": "- Discuss additional references\n- Provide some additional discussion how can the performance be improved\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IbYddzbY-2X", "reviewer": "Reviewer_wMrv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1 NAS in RL is a difficult problem. However has lot of promise. Designing suitable function approximators can affect convergence dynamics of deep RL. \n\n2. The proposed solution and its presentation, motivation and explanation is very lucid and clear. Notations are introduced succintly and resepctive intuitions are clearly stated as well. \n\n3. The experiments are indeed quite extensive and the authors have attempted to study the effect of apply end to end DARTS in RL from many possible dimensions including ablation studies on how the responses change in different kind of convolutions. \n\n\nWeaknesses/Concerns\n\n1. Novelty is a concern. Tts not that people have not thought about NAS for RL. but prohibitive search space becomes a bottleneck in most cases. And the authors here solve that simply by taking DARTS and applying it to learning function approximators for deep RL. \n \n   1a.  One clear example of this is the fact that authors point out some interesting challenges wil RL earlier in the paper such as \"RL training curves possess considerably more variance and noise than SL curves\". However I do not see any change in formulation in section 3.2 to address that concern. \n \n   1b. Another challenge with RL which the authors also have hinted at is that unlike SL feedback data in RL replay buffer suffers from slight distributional shift over time, until enough amount of exploration is done. This is quite a big problem and RL-DARTS formulation does take care of this as well. \n \n Inshort is is difficult to identify how the authors have extended DARTS formulation to suitably adapt to RL scenario. JUst stating that it was integrated with minimal code change is not sufficient. \n\n2. The main section needs to be expanded a bit more, the entire methodology is compressed into a few paragraphs. The formal algorithm provided here is also a surprise. It literally has 3 steps. A reader can already see these 3 things. An algorithm is meant to give the reader an idea of how to even implent the same without using any backbone codebase. For instance, this is an iterative setup. So the algo should outline at each itheration, how the components of the loss are computed. in what order the architecture parameters are updated etc... \n\n3. Diversity in domains is needed. Procgen is a great becnmark indeed. However no matter how we look at them and their different settings they are games. Other tasks such as robot locomition (simple version), or drone simulations, or treatment planning in clinical decision support systems will actually be able to realy test the validity of the claims made herein. \n", "summary_of_the_paper": "The authors aim to automatically design Reinforcement Learning Q function approximator via NAS, specifically by adapting DARTS framework. They main motivation behind this is that as RL state spaces become more complex simple function approximator models, even predesigned DNNs prove to be sub-optimal, since supervised models are not always suitably designed to learn in an online fashion from reward signals rather than labels. Beyond that there are also existing RL bottlenecks such as reward sparsity, sample efficiency etc. The authors use the whole DARTS supernet at the function approximator. To bypass the bilevel optimization target in DARTS, which could prove to be intractable for RL context, authors propose to optimize the loss on cumulative returns of the RL problem by optimizing the loss on architecture and weight parameters from the replay buffer samples. ", "summary_of_the_review": "This indeed and interesting work / study and has a lot of merit. However Authors are strongly encouraged to address the concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FKuMkn7Seow", "reviewer": "Reviewer_KUPC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: this paper discusses several main challenges of DARTS for RL. For example, the main challenge of applying DARTS to RL is that it needs to address the non-stationary data distribution arises due to the exploration process, where the data distribution depends on the network architecture used to generate the replay data. Another challenge is that it is unclear if DARTS’s default discretization procedure leads to a better discrete cell in the RL setting due to different optimization landscape in the RL case. In addition, the work also carried out extensive experiments to examine the behaviors of DARTS for RL.\n\nWeakness. The main weakness of the paper is its lack of novelty and insufficient technical contribution. The work is mainly an application of standard DARTS to the RL setting, without additional contributions in the algorithm side. Although the unique challenges of DARTS for RL are pointed out earlier in the paper, the paper does not propose any solution to address these challenges. In contrast, experiments in later sections seem to indicate that these are indeed not the case, and standard DARTS technique developed in SL could be directly applied to the RL setting with minimal modification. For example, experimental results seem to suggest that end-to-end DARTS works well on such non-stationary data distribution even without any additional tweaks. (Likewise, the concerns about the discretization has also been shown to be not an issue here in experiments.) While it is indeed interesting to observe that existing DARTS from SL could be readily applicable to RL, it does not play a major contribution for the current paper.\n\nIn summary, this paper can be regarded as an extensive experimental investigations, which find out that standard DARTS developed in SL are actually applicable to RL. Although the findings are interesting, it could be fit better as a workshop paper.\n\n- Minor comments\nNeed to give the precise definition / expression for the loss L(theta, alpha) in (2) in the RL-DARTS.\n", "summary_of_the_paper": "This paper considers the problem of differentiable neural architecture search for RL applications. It applies existing DARTS to the RL setting and conducts extensive experimental studies to examine the performance and behaviors of RL-DARTS. Experimental results show that the supernet learns alternative architectures that are highly competitive against manually designed policies.", "summary_of_the_review": "This paper can be regarded as an extensive experimental investigations, which find out that standard DARTS developed in SL are actually applicable to RL. Although the findings are interesting, it could be fit better as a workshop paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D_fe9N5rLZ-", "reviewer": "Reviewer_wzZ5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\nThis paper aims to tackle interesting but less explored questions: while there have been extensive studies on searching network architectures for supervised learning, it is generally less understood whether and to which extent architecture search would benefit reinforcement learning pipelines.\n\nThis paper has provided extensive experimental results exploring this direction and studied how different design choices would affect the learning procedure and final results.\n\n\n[Weakness]\n\nWhile I like the direction this paper is going, I have concerns regarding the novelty of this paper and whether the experiments show the full power of differentiable neural architecture search (DARTS).\n\nThe proposed method is a direct application of neural architecture search on reinforcement learning problems without any new technical innovations. I'm not against straightforward A+B types of research (in this paper, A=RL, B=DARTS). For example, the nice thing about the proposed method is that it is simple and can be a plug-and-play module on top of existing RL frameworks. However, I typically expect the combined method to accomplish tasks that neither A nor B can do, or at least demonstrate significant performance gain over existing methods. Yet from Figure 3, it seems that the proposed method is on par (in Maze) or worse (in Bigfish and Dodgeball) than the IMPALA-CNN baseline [1], which makes me worry about the significance of the proposed method and its impact on the field. For example, should we prefer IMPALA-CNN over this paper's method since IMPALA-CNN is supposed to be easier to train as the network structures are fixed? The authors may want to include more discussions or comparisons to show how the proposed method is better and in which scenario we should choose it over other state-of-the-art approaches.\n\nContinuing from my previous point, the IMPALA-CNN baseline also seems to have a smaller search cost, according to Table 3.\n\nOne great benefit of neural architecture search is that it allows a tradeoff between model accuracy and computational efficiency [2]. Are there any similar tradeoffs we can make in the RL setting, e.g., would smaller/sparser neural networks allow higher feedback control frequency such that control becomes easier? Showing examples demonstrating the unique benefit of neural architecture search can potentially make this paper much stronger and show potentials to enable new capabilities.\n\nIn the Heist task of Figure 8, why do random search and IMPALA-CNN seem to have a very good initial start but experience a sharp drop in performance at the beginning of the training processes? Why not directly keep the initial policy?\n\n\n\n[Minor points]\n\nThe authors made a claim in the intro that \"in RL, a bad architecture ... and thus a local optimum where the loss is zero while the policy is still poor.\" However, the statement may not be well-grounded without context, e.g., to which objective is the \"optimum\" referring? Is the \"loss\" defined over a very small replay buffer? If the loss is zero, why is it still a local optimum (assuming that the loss is nonnegative)? \n\nTypo in Page 7 first paragraph:\n\"RL time cost is partially is based on\": multiple \"is\"\n\n\n[1] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, Koray Kavukcuoglu, \"IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures\"\n[2] Han Cai, Ligeng Zhu, Song Han, \"ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware\"\n\n\n===============\n\n[Post Rebuttal]\n\nI thank the authors for the detailed feedback. I have read the reviews from other reviewers and decided to keep my score the same (5: marginally below the acceptance threshold).\n\nAs shared by most reviewers, the novelty of this paper is a bit limited, and as I have stated in my main review, I'm not against A+B types of research (in this paper, A=Reinforcement Learning, B=Differentiable Neural Architecture Search), but I typically have high expectations from such straightforward combination.\n\nHowever, I do not think this paper has shown the full benefit of the combination. It will greatly strengthen this paper if the authors show what new capabilities are enabled by the proposed method but are not possible with either A or B.", "summary_of_the_paper": "This paper proposes to combine differentiable neural architecture search (DARTS) with standard reinforcement learning (RL) frameworks by searching the model structures (convolutional cells) for the policy and value functions. The authors have applied the method to infinitely procedurally generated Procgen benchmark and demonstrated the benefits of DARTS in RL on search efficiency in terms of time and compute. In addition, the proposed method can be easily integrated with existing RL pipelines by simply replacing the image encoder with a DARTS supernet, compatible with both off-policy and on-policy RL algorithms. The authors further show that the supernet gradually learns better cells with more training iterations, leading to alternative architectures that can be highly competitive against manually designed policies and verify previous design choices for RL policies.", "summary_of_the_review": "While I like the direction this paper is going, I have concerns regarding the novelty of this paper and whether the experiments show the full power of differentiable neural architecture search (DARTS). Therefore I currently lean towards the rejection side. Please see my main review for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies network architecture search in the context of reinforcement learning. In particular it applies the DARTS method to the Procgen RL benchmark, and conducts extensive experimental evaluations. It identifies a number of issues that could potentially prevent DARTS from working well in the RL setting (such as nonstationarity and high variance), but in the end shows good performance without needing to modify DARTS substantially.

The reviewers agreed that a key strength of the paper is in its experiments. But they also identified a weakness in novelty: if a paper's main contribution is to combine two previously well-explored ideas (in this case, RL and DARTS) then there is a high bar for the quality of exposition and positioning, and the reviewers did not feel that this bar was met. (Though the authors' updates during the rebuttal period did help substantially with clarity and relationship to other methods -- thank you for these!)

Recommended decision: while the paper makes a worthwhile contribution, it does not in its current form rise to the level of novelty and general interest that is needed for publication in ICLR.
Decision: Reject