GDP: Generalized Device Placement for Dataflow Graphs
Authors: Yanqi Zhou
Sudip Roy
Amirali Abdolrashidi
Daniel Wong
Peter C. Ma
Qiumin Xu
Ming Zhong
Hanxiao Liu
Anna Goldie
Azalia Mirhoseini
James Laudon
Venue:  (2020)
Keywords: device placement
reinforcement learning
graph neural networks
transformer

Abstract: Runtime and scalability of large neural networks can be significantly affected by the placement of operations in their dataflow graphs on suitable devices. With increasingly complex neural network architectures and heterogeneous device characteristics, finding a reasonable placement is extremely challenging even for domain experts. Most existing automated device placement approaches are impractical due to the significant amount of compute required and their inability to generalize to new, previously held-out graphs. To address both limitations, we propose an efficient end-to-end method based on a scalable sequential attention mechanism over a graph neural network that is transferable to new graphs. On a diverse set of representative deep learning models, including Inception-v3, AmoebaNet, Transformer-XL, and WaveNet, our method on average achieves 16% improvement over human experts and 9.2% improvement over the prior art with 15 times faster convergence. To further reduce the computation cost, we pre-train the policy network on a set of dataflow graphs and use a superposition network to fine-tune it on each individual graph, achieving state-of-the-art performance on large hold-out graphs with over 50k nodes, such as an 8-layer GNMT.

Under review as a conference paper at ICLR 2020
GDP: GENERALIZED DEVICE DATAFLOW GRAPHS
PLACEMENT FOR
Anonymous authors Paper under double-blind review
ABSTRACT
Runtime and scalability of large neural networks can be significantly affected by the placement of operations in their dataflow graphs on suitable devices. With in- creasingly complex neural network architectures and heterogeneous device char- acteristies, finding reasonable placement is extremely challenging even for do- main experts. Most existing automated device placement approaches impracti- cal due to the sigmificant amount of compute required and their inability to gener- alize to new, previously held-out graphs. To address both limitations, we propose an efficient end-to-end method based on a scalable sequential attention mecha- nism over graph neural network that is transferable to new graphs. On diverse set of representative deep learning models, including Inception-v3, AmoebaNet, Transformer-Xi and WaveNet, our method average achieves 16% improve- ment over human experts and 9.2% improvement over the prior art with 15x faster convergence. To further reduce the computation cost, we pre-train the policy net- work on set of dataflow graphs and use superposition network to fine-tune it on each individual graph, achieving state-of-the-an performance on large hold-out graphs with over 50k nodes, such as an 8-layer GNMT.
INTRODUCTION
Neural networks have demonstrated remarkable achieved by training larger model on larger dataset (Hestness al., [017]] (Sazzerr et al.) [017]] Jozefowicz et al. 2016 Mahajan et al. 2018 Radford et al.). Training such large models efficiently while meeting device constraints, like memory limitations, necessitate partitioning of the underlying dataflow graphs for the models across multiple devices. However, devising a good partitioning and placement of the dataflow graphs requires deep understanding of the model architecture, optimiza- tions performed by domain-speciff compilers, well the device characteristics, and is therefore extremely hard even for experts.
performance can usually be
ML practitioners often rely their understanding of model architecture determine a reasonable partitioning and placement for graphs. However, relying solely on the model architecture while ig- noring the effect of the partitioning to sub-optimal placements and consequently under-utilization of available devices. The goal of automated device placement is find the optimal assignment of operations devices such that the end-to-end execution time for a single step is minimized and all device constraints like memory limitations are satisfied. Since this objective function is non-differentiable prior approaches (Mirho- seini et al. 2017 2018 Gao et al. 2018) have explored solutions based on reinforcement learning (RL). However, these RL policies are usually not transferable and require training a new policy from scratch for each individual graph. This makes such approaches impractical due to the signifi- cant amount of compute required for the policy search itself, times offsetting gains made by the reduced step time.
subsequent compiler optimizations like op-fusion lead
In this paper, we propose an end-to-end deep RL method for device placement where the learned pol- icy is generalizable to graphs. Specifically, the policy network consists of a graph-embedding network that encodes operation features and dependencies into a trainable graph representation, fol- lowed by scalable sequence-to sequence placement network based (Vaswani et al., 2017 Dai et al., 2019). The placement network transforms the graph representa- tions into placement decision with soft attention, removing hard constraints such as hierarchical
improved Transformer

---

Under review as a conference paper at ICLR 2020
grouping of operations (Mirhoseimi al. 2018) co-location heuristics (to reduce the placement complexity) (Mirhoseini et al., (017). Both of our graph-embedding network and placement net- work can be jointly trained in an end-to-end fashion using supervised reward, without the need to manipulate the loss functions at multiple levels. We empirically show that the network learns flexible placement policies at per-node granularity and can scale problems over 50,000 nodes.
To generalize to arbitrary and held-out graphs, our policy is trained jointly over a set of dataflow graphs (instead of one at a time) and then fine-tuned on each graph individually. By transferring the learned graph embeddings and placement policies, we are able to achieve faster convergence and thus use less resources to obtain nigh-quality placements. We also use super-positionis i.e., a feature conditioning mechanism based on the imput graph embeddings, to effectively orchestrate the optimization dynamies of graphs with drastically different sizes in the same batch.
Our contributions can be summarized as follows:
1. An end-to-end device placement network that can generalize to arbitrary and held-out graphs. This is enabled by jointly learning transferable graph neural network along with the placement network.
2. A scalable placement network with an efficient recurrent attention mechanism, which elim- inates the need for explicit grouping stage before placement. The proposed end-to-end network provides 15x faster convergence as compared to the hierarchical LSTM model used in earlier works (Mirhoseini et al. 2017 2018).
3. A new batch pre-training and fine-tuning strategy based on network superposition, which leads to improved transferability, better placements especially for larger graphs, and 10x reduction in policy search time as compared to training individual graphs from scratch.
4. Superior performance over a wide set of workloads, including Inception (Szegedy al., 2015), AmoebaNet (Real et al., 2018). RNNs, GNMT (Wu et al. (0166), Transformer-Xi (Dai al. 2019). WaveNet (van den Oord al. 2016 and more.
RELATED WORK
Device Placement Reinforcement learning has been used for device placement of given dataflow graph (Mirhoseini et al. 2017) and demonstrated run time reduction over human crafted place- ment and conventional heuristics. For improved scalability, a hierarchical device placement strategy (HDP) (Mirhoseini et al. 2018) has been proposed that clusters operations into groups before placing the operation groups onto devices. Spotlight (Gao et al., 2018) applics proximal policy optimization and cross-entropy minimization lower training overhead. Both HDP and Spotlight rely LSTM controllers that are difficult to train and struggle to capture very long-term dependencies over large graphs. In addition, both methods are restricted process only single graph at time, and cannot generalize to arbitrary and held-out graphs. Placeto (Addanki et al. 2019) represents the first at- tempt to generalize device placement using graph embedding network. But like HDP, Placeto also relies on hierarchical grouping and only generates placement for one node at each time step. Our approach (GDP) leverages a recurrent attention mechanism and generates the whole graph place- ment at once. This significantly reduces the training time for the controller. We also demonstrate the generalization ability of GDP over a wider set of important workloads.
Parallelization Strategy Mesh-TensorFloww is language that provides general class of dis- tributed tensor computations. While data-parallelism can be viewed as splitting tensors and opera- tions along the "batch" dimension, in Kesh-TensorFFow the user can specify any rensor-dimensions to be split across any dimensions of a miutii-iimenional mesh of processors. FlexFlow (Jia et al., 2018) introduces SOAP, a more comprehensive search space of parallelization strategies for DNNS which allows parallelization of DNN in the Sample, Operator, Attribute, and Parameter dimen- sions. uses guided randomized search of the SOAP space to find a parallelization strategy for a specific parallel machine. GPipe (Huang et al. 2018) proposed pipeline parallelism, by partitioning a model across different accelerators and automatically splitting a mini-batch of training examples into smaller micro-batches By pipelining the execution across micro-batches, accelerators can op- erate in parallel. Our GDP focuses on a general deep RL method for automating device placement on arbitrary graphs, and is therefore orthogonal existing parallelization strategies.
2

---

Under review as a conference paper at ICLR 2020
Adjccenc Matric
Transformer-X--based
Nxd
GraphSAGE
Placer Network
Nkh
Recesssrraaton
Node Feature:
Segment
Segment
Policy Output
Ope Type
Ouiput Shapo
Concadinated Nodes Facilunss
Appregator Approgator
Devics Placermnn Probab iteb
Figure 1: Overview of GDP: An end-to-end placement network that combines graph embedding and sequential attention. N: Number of Nodes, h: Hidden Size, d: Number of Devices.
Compiler Optimization REGAL (Paliwal al. 2019) uses deep RL optimize the execution cost of computation graphs in a static compiler. The method leverages the policy's s ability transfer to new graphs to improve the quality the genetic algorithm for the same objective budget. How- REGAL only targets peak memory minimization while GDP focuses graph time and scalability while also meeting the peak memory constraints the devices. Specifically, we general- ize graph partitioning and placement into single end-to-end problem, with and without simulation, which can handle graphs with over 50,000 nodes.
3 END-TO-END PLACEMENT POLICY
Given a dataflow graph G(V, E) where represents atomic computational operations (ops) and E represents the data dependency, our goal is to learn policy æ G r 'D that assigns placement D E 'D for all the ops in the given graph G E G, to maximize the reward ra.n defined based on the time. DD is the allocated devices that can be mixture of CPUs and GPUs. In this work, represent policy as a neural network parameterized by 0.
Unlike prior works that focus single graph only, the RL objective in GDP defined to simulta- neously reduce the expected runtime of the placements over a set of N dataflow graphs:
1 Σ
J(0) EG Dws x N
In the following, we refer to the case when N = 1 as individual training and the case when N >>>>> 1 as batch training. We optimize the objective above using Proximal Policy Optimization (PPO) (Schul- man et al. 2017) for improved sample efficiency.
Figure shows an overview the proposed end-to-end device placement network. Our proposed policy network consists graph embedding network that learns the graphical representation of any dataflow graph, and a placement network that learns a placement strategy over the given graph embeddings. The two components is applied to make a set decisions at each node. These decisions, denoted as a, for each e V across all nodes, form multi-bandit problem, and specifying the entire a corresponds to playing several arms together in single shot. Note the architecture is designed be invariant over the underlying graph topology, enabling us to apply the same learned policy to wide set of input graphs with different structures.
jointly trained in end-to-end fashion. The policy p(a[G)
action (avev). One decision corresponds to playing
arm of
3.1 GRAPH EMBEDDING NETWORK
We leverage graph neural networks (GNNs) (Hamilton et al. 2017; Xu et al. 2019; You et al. 2018) to capture the topological information encoded in the dataflow graph. Most graph embedding frameworks are inherently transductive and can only generate embeddings for a given fixed graph. These transductive methods do not efficiently extrapolate to handle unseen nodes (e.g.. g., in evolving graphs), and cannot learn to generalize to unseen graphs. GraphSAGE (Hamilton et al. 2017) is an inductive framework that leverages node attribute information efficiently generate representations
3

---

Under review as a conference paper at ICLR 2020
on previously unseen data. While proposed framework is generic, adopt the feature aggrega- tion scheme proposed in GraphSAGE to model the dependencies between the operations and build general, end-to-end device placement method for wide set of dataflow graphs
In GDP, nodes and edges in the dataflow graph are represented as the concatenation of their meta features (e.g., operation type, output shape, adjacent node ids) and further encoded by the graph embedding network into a trainable representation. The graph embedding process consists of multi- ple iterations, and the computation procedure for the I-th iteration can be outlined follows:
First, each node v E V aggregates the feature representations of its neighbors, {he Vu (V()),,
into single vecter h (i) "N This aggregation outcome is function of all previously generated representations, including the initial representations defined based on the input node features. In this work, we use the following aggregation function with max pooling:
(1) h (m) max(o ( + b()). Vu E N(())
where (W b()) define an affine transform and a stands for the sigmoid activation function. We then concatenate the node's current representation, n, with the aggregated neighborhood vector, (1) "M)" and feed this concatenated vector through a fully connected layer f(1+1)
h (++1) (concatth))) ()))))
Different from GraphSAGE. parameters in our graph embedding network are trained jointly with placement network via stochastic gradient descent with PPO, in a supervised fashion, as described Section 3
That is, we replace the unsupervised loss with our spek-sp objective.
3.2 PLACEMENT NETWORK
The graph neural network works feature aggregation network that learns trainable feature representation for the computational graph, we still need a policy network that produces actions on per node basis. Given h ss the policy network produces a, 's through conditionally independent predictions, where the prediction for one node does not depend on the prediction of other nodes.
p(a|G) = II G(a..G) II p(ar))(bb))
While I can be represented using multilayer perceptrons (MLPs), where the MLPs is shared across all nodes for prediction the placement output distributions. However, MPLs lack a dependency tracking mechanism across nodes. In practise, the placement of one node can be determined by the placement of another node, where the placed node may consume a large size of data produced by the other node. Intuitively, attention network can learn this dependency and the relative importance of dependencies across an entire graph. Therefore, we decide to use an attention-bant placement network to better track inter-node placement-related dependencies.
Designing a scalable placement network that can generalize to graphs with thousands of nodes is challenging, as the conventional GNMT models proposed for language tasks usually target a shorter sequence length. Hierarchical placement (Mirhoseini et al.) 2018)) has been proposed to address this however, the proposed grouper network comes with limited flexibility and generality. For example, the grouper network leverages an aggregated feature representation by averaging feature vectors for nodes within the same group. The -infeifferetti grouping procedure prevents train- ing the graph-embedding and placement networks end-to-end.
To remove the two-stage hierarchical workflow HDP for improved scalability, we propose to use a Transformer- baasd attentive network to generate operation placements in end-to-end fashion. As the graph embedding already contains spatial (topological) information for each node, we remove the positional embedding in the original transformer to prevent the model from overfitting node identifications. To capture long-term dependencies efficiently among a large set of nodes, we adopt segment- recurrence introduced in Transformer-Xi (Dai et al.) 2019 Dai 2019), where hidden states computed for the previous set of nodes are cached (with gradient flows disabled) and reused as an extended context during the training of the next segment. Besides achieving extra long context,
4

---

Under review as a conference paper at ICLR 2020
empirically find the segment-level recurrent attention much faster than conventional LSTM- based GNMT model. In our experimental evaluation, we compare both the performance and speed up of our placement network with that of the LSTM-based hierarchical device placement.
3.3 BATCI TRAINING WITH PARAMETER SUPERPOSITION
Since the parameterization for the architecture of the end-to-end policy is designed to be invariant over input graphs with different topologies, the same placement policy can be shared across wide set of workloads. We therefore propose batch training strategy, and further enhance the aforemen- tioned architecture handle such generalization across graphs.
Naïve batch training is challenging in our context as different dataflow graphs contain different num- ber of operations connected in different topologies. In addition, unlike previous device placement methods, GDP aims to handle graphs from potentially different application domains (e.g. computer vision, language, and speech), where the number of operations can range from few thousand to one million. These graphs have drastically different network architecture, in terms of computational operations, data shape, and network topology. As an example, recurrent networks have completely different operation types and connections compared to multi-branch convolutional networks that are widely used in computer vision. It would be highly desirable to train single shared network that maximizes information sharing across these heterogeneous tasks, without hurting the performance on each them due to their distinct learning dynamics.
Along similar direction of multi-task learning and few-shot learning (Oreshkin et al. 2018 we propose a feature conditioning mechanism similar to parameter superposition (Cheung et al. 2019). The idea is train one shared policy, but condition its parameters based the input features to mitigate the potentially undesirable interference among different input graphs. Since dense layers (affine transforms followed by nonlinearity) serve the fundamental building blocks in all of our network components, we introduce an additional conditioning layer to enable superposition in all dense layers the placement network:
[1+1)) - (c(s:!0)) (C(xx"))))
Ox 2x(1)
where (1) stands for dense layer in our policy network, stands for the feature conditioning layer, and (0) denotes the feature representation of the input graph generated by the graph-embedding network. The feature conditioning layer is implemented with minimum overhead by adding an additional transformer layer to placement network.
4 EXPERIMENT
4.1 EXPERIMENT SETUP
In this section, evaluate our training strategy on widely used machine learning models in com- puter vision, natural language processing, and speech domains. We compare our approach to human expert placement, TensorFlow METIS placement, and hicrarchical device placement (HDP) (Mirho- seiin et al. (2018). Our experiments are run on machines with one Intel Broadwell CPU and up to eight Nvidia PTO0 GPUs. Note that the prior work (Mirhoseini al. 2017 2018 Gao et al. 2018) were evaluated on different GPU devices, preventing direct comparison of results. Therefore, we re-evaluate HDP on our own system environment and report those numbers.
The performance of placement is evaluated by the resulted training step time (run time) of the neural network. We use the negative square root the normalized run time as the reward, where the run time is normalized with the best run time from a baseline. We use the average reward of all the previous trials as a bias term. The advantage value is computed by subtracting the reward by the average reward. During the search, apply large negative reward (-10) for invalid placements (e.g. a violation of co-location constraint, out of memory, etc.). For operation scheduling, we rely on the Tensorflow default FIFO scheduling.
4.2 PERFORMANCE ON INDIVIDUAL GRAPHS
We evaluate GDP by training the model separately on six important graphs, including RNN Lan- guage Modeling, GNMT (Sutskever et al. 2014) Transformer-XI, Inception, AmoebaNet, and
5

---

Under review as a conference paper at ICLR 2020
Table Run time comparison between GDP-one, human expert, Tensorflow METIS, and hicrarchi- cal device placement (HDP) on six graphs (RNNLM, GNMT, Transformer-XL, Inception, Amoe- baNet, and WaveNet). Graph runtime speed up is compared with Human Placement (HP) and Hi- erarchical Device Placement (HDP). Search speed up is the policy network training time speed up compared to HDP (reported values are averages of six runs).
Run time speed up
GDP-one (8)
HP (s)
METIS
HDP
Search speed up
Model (#ddevices)
(s)
(S)
over HP HDP
2-layer RNNLM (2)
0.234
0.257
0.355
0.243
9.8% / 4%
2.95x
4-layer RNNLM (4)
0.409
0.48
OOM
0.490
17.4% / 19.8%
1.76x
2-layer GNMT (2)
0.301
0,384
OOM
0.376
27.6% 7 24.9%
30x
4- 4-layer GNMT (4)
0.409
0,469
OOM
0.520
14.7% / 27.1%
58.8x
8-layer GNMT (8)
0.649
0.610
OOM
0.693
-6% 6.8%
7.35x
2-layer
Transformer-XL. (2)
0.386
0.473
OOM
0.435
22.5% / 12.7%
40x
4-layer
0,580
0.641
00M
0.621
11.4% / 7.11%
26.7x
Transformer-XL XL (4)
8-layer
0.748
0.813
OOM
0.789
8.9% / 5.5%
16.7x
Transformer-X XL (8)
Inception (2)
0.405
0.418
0,423
0.417
3.25 / 3%
13.5x
AmoehaNct (4)
0.394
0.44
0,426
0.418
26.1% /
58.8x
2-stack 18- 18-layer
0.317
0.376
OOM
0.354
18.6% / 11.7%
6.67x
WaveNet (2)
4-stack l6-layer
0.659
0.988
OOM
0.721
50% 9.4%
20x
WaveNet (4)
GEOMEAN
16 % 9.2%
15x
WaveNet. We name this approach GDP-one. For all the tasks, GDP-one consistently outperforms human expert placement, TensorFlow METIS (Karypis & Kumar 1998) placement, and HDP. For extremely large graphs, GDP-one is only 6% worse on 8-layer NMT (over 60k nodes), compared to human placement, but is 6.8% better than HDP. Overall, GDP-one achieves average more than 16% run time reduction across the evaluated 12 graphs, compared to human expert place- ment. Compared to hierarchical device placement, GDP-one achieves average 9.2% speed up, and scales better to large graphs such as 8-layer NMT and 4-layer RNNLM. Importantly, with the efficient end-to-end training and sample efficient reinforeement learning algorithm, GDP-one has 15x speed up in convergence time of the placement network over HDP.
4.3 GENERALIZATION
GDP enables the training of multiple heterogeneous graphs in a single batch, sharing parameters in the graph-embedding network and the placement network. We name this training strategy GDP- batch. We empirically show that GDP-batch generates better placements for many workloads such as transformer-XL. (7.6%), WaveNet (15%), and 8-layer GNMT (8%). Table 2 compares the run time of 11 tasks using GDP-batch, with the same end-to-end architecture as described in section 4.2 GDP-batch yields slightly better run time compared GDP-one in majority of the tasks, while being only slightly worse on AmoebaNet. Compared to training graphs separately. GDP-batch reduces network parameters and enables transfer learning among different graphs
We further evaluate the effect of transfer learning by mixing redundant tasks in a batch. We find that mixing different graphs such as RNNLM and GNMT models with different number layers results in both faster and better learning for RNNLM and GNMT with large number of layers (8- layer). As a matter of fact, both Placeto (Addanki et al. 2019 and HDP had problems matching human placement performance for 8-layer GNMT or 8-layer RNNLM. With batch training, GDP the first device placement work to match human expert performance for both 8-layer GNMT and 8-layer RNNLM. We also for the first time show that GDP- not only improves the search time (since we do not retrain the policy for every new graph), it can also improve the performance of the found placements. More detailed results shown in Appendix Table

---

Under review as a conference paper at ICLR 2020
Table 2: Run time comparison on GDP-batch vs. GDP-one.
Model
Speed up
Model
Speed up
2-layer RNNLM
0
Inception
0
4-layer RNNLM
5%
AmoebaNet
-5%
2-1ayer GNMT 4-layer GNMT
4-stack 36-layer WaveNet
3.3
0
2-stack 18-layer WaveNet
15%
2- 2-layer Transformer- XL
7.6%
8-layer Transformer-Xi
1.5%
4-layer Transformer-XI.
3%
Generalization to hold-out graphs: Here we show another set of experiments where we treat GDP- batch pre- vee-trriiing strategy and remove the target graph from the batch training dataset. We then fine-tune the pre-trained model on the hold-out graphs for fewer than 50 steps, which takes less than minute. We name this aoo--generalization-fnneen hold-out graphs outperforms human expert placement and HDP consistently on all six batch training datasets, and performs only slightly worse than GDP-one. 2-layer RNNLM and 2-stack WaveNet almost match the performance of GDP-one. We also run inference (generate placement) directly on the pre-traing pre-trained model for the target hold-out graphs, and name this aurr--eneraliiattoozerooeet We find that GDP- geweealizaaiinnzeeeeee zerossoo only marginally hurts performance as compared to GDP- generalization-finetun while being slightly better than human placement and HDP. This indicates that both graph embedding and the learned policies transfer and generalize the unseen data.
Figure 2 shows that GDP fine-tuning for
Human expert
Our methed inciicua train
twrd
3.4
2.3
0.2
0.1
2/agwr AJagnr
#layeer CNVTT
2. payye
Hayer THAML
Zustack
Unisen Madels
Figure Finetuning on hold-out graphs.
Comparisons with other generalized placement approaches: Placeto (Addanki al. 2019), to y our knowledge, is the only other method beside GDP that shows true (and non-simulated) gener- alized device placement results. Direct comparison is not possible since Placeto uses different hardware platform and different input graphs (Inception-V) V3, NMT, and NASNet). Placeto's search time is on average 2.65x faster than HDP, while GDP is on average 15x faster than HDP on our larger set of graphs. Apart from search time speed up. Placeto on average reduces placed graph run time by 3% (for its different graphs and hardware) while GDP on average reduces placed graph run time by 9.2%, compared to HDP. One advantage of GDP over Placeto is that it does not rely on any initial feasible placement. Providing a reasonable initial placement is often non-trivial for domain experts, especially for larger graphs such as 8-layer GNMT. As such, we are the first to report superhuman results on 8-layer GNMT (with GDP-batch).
4.4 ABLATION STUDIES
Attention and Superposition. We did an ablation study on the attention and the superposition layer in the transformer-XL. placer network. We find that attention improves placement run time by an average of 18% compared to placer network with attention, and superposition improves placement run time by an average of 6.5% where all the graphs are trained a single batch as described in Section 4.3 Without superposition network, batch training fails for AmoebaNet and Inception when mixing with larger RNNLM or GNMT models (4-layer).
Pre-training graph embeddings. We also evaluate fine-tuning strategy by pre-training the graph embedding and placement network and fine-tuning the network on the down stream tasks. The difference here compared Section 4.3 is that also include the target graphs in the pre-training

---

Under review as a conference paper at ICLR 2020
0.45
0.40
2.40
0:35
2.35
0.30
0.10
0:25
3.25
0.29
2.15
9.30
0.05
0.02
2.00
Doeepon
Amsebroitt
2dajer RMA.M
GROST
Jayer TYAL
Zatuck WoweNet
mocoption
AmootruNtt
2eaper AMAG
GNAT
Saper TilLL
Januck WaveNe
(a) Comparison wo attention.
(b) Comparison wo superposition.
Figure 3: Ablation Study on Attention and Superposition of the Placement Network.
normalized run time
normalized search time
rnnim 2-layer
rnnim 4-layer
rnlm 8-layer
nmt 4-layer
nmt 8-layer
trfxl 4-layer
8 8-layer
inception
amoebanet
wavenet
wavenet 4-
GEOMEAN
0.25
0.5
0.75
1.25
Figure 4: Normalized run time (step time for the generated placement) and normalized training time (search time) for fine-tuning. Time is normalized to GDP-one.
dataset. When GDP-batch is used as a pre-training strategy, the graph embedding and placement network assimilate meaningful graph representations and placement policies from a wide set of graphs, thus can be used as strong baseline network for fine-tuning compare the generated placement run time and the placement search time, normalized to GDe-one. We find that tuning further reduces the the placed graph run time by average of 5% and placement search time by an average of 86%, compared to GDP-one.
downstream tasks. We
CONCLUSION
In this paper, we present a generalized device placement strategy that uses a graph neural network and super-pon super-positionin to generalize to arbitrary and held out graphs. Through experimental eval- uation over a wide set of representative graphs from different domains including computer vision, speech, and NLP, we demonstrated over 15 times faster convergence while achieving a 16% and 9.2% reductions in step time over human expert placement and HDP, respectively.
ACKNOWLEEDMEENTS
TBD
8

---

Under review as a conference paper at ICLR 2020
APPENDIX
6.1 PROXIMAL POLICY OPTIMIZATION
In device placement, the objective is to minimize the training step time of given computational graph or batch of dataflow graphs for target system configuration (e.g. a 8-GPU cluster or a TPU pod), by placing operations onto different devices to enable model-level parallelism. This process corresponds to maximizing the expected performance in the MDP. For better sample efficiency, we adopted Proximal Policy Optimization (PPO) (Schulman et al. 2017 algorithm. The objective is to maximize a surrogate objective:
g. Ax (Sn, (8n) an) g(an (8n)
La Eojoin nut
N-1
min q (an|sn)
(R - R), clpp ((an(sa) - - - +c( R)M
Ln max
N
qaan spp)
g(an (n)
0.00
Within loop, GDP PPO continuously samples placements from the distribution and evaluates their training times in real systems. For a rollout of K, we perform a minimatch of m stochastic gradient ascent steps with respective to the objective proximal policy optimization, which makes incre- mental policy improvements. The rollout steps K and minibatch size m are hyper parameters for PPO. We find set of optimized hyper parameters and keep them fixed for all the experiments pre- sented. As the rewards are generated on-the-fly fly based on real system measurements, we no longer need to re-evaluate the placement solutions in a separate phase.
6.2 HYPERPARAMETERS
In this section, we list out all the selected hyperparameters in our experiments for reproducibility in Table 3 and Table
Table 3: Hyperparameters for Policy Network. gslaayers: GraphSAGE layers, gs jsn:: Graph- SAGE maximum neighbors, tr _d_model: Dimension of the TransformerXL model, tr ___head: Number attention heads, tr fayyers: Number of TransformerXL layers, tr headss Dimen- sion of each attention head, f _dinner: Dimension of inner hidden size in positionwise feed- forward.
Parameters
Value
Parameters
Value
gs layers gs knn
4
gs dim
128
5
tr lagers
tr d model
128
tr n head
tr head
25
tr _dinner
256
Table 4: Hyperparameters for PPO.
Parameters
Value
Parameters
Value
learing rate
0.5
murm of rollouts
400
minibatches
40
epochs
epsilon
0.2
entropy
0.05
6.3 INPUT GRAPHS
We used a variety of widely used workloads from computer vision, speech, and NLP. In this section, give detailed explanation the selected models and hyperparameters.

---

Under review as a conference paper at ICLR 2020
6.3.1 INCEPTION-VE
Inception-V3 (Szegedy et al. 2015) is multi-branch convolutional network used for variery of computer vision tasks, including classification, recognition, or generation. The network consists of blocks made of multiple branches of concolutional and pooling operations. Within block, the branches of ops can be executed in parallel. However, the model is mostly sequential as the outputs of each block concatenated together to form the input to the next block. We batch size of 64. The Tensorflow graph of this model contains 24,713 operations.
6.3.2 AMORBANET
AmoebaNet RReal al.. 2018) automatically designed neural network that yields SoTA perfor- mance on ImageNet. Similar to Inception-V3, 3, contains Inception-like blocks called cells, which receives direct input from the previous cell and skip input from the cell before it. The network is made of redundant cells stacked together, therefore is more modular than Inception-V3. We use batch size of 64. The Tensorflow graphs contains 9,430 operations.
6.3.3 RNNLM
Recurrent Neural Network Language Model (Zaremba et al., 2014: Jozefowicz et al., 2016) is made of many LSTM cells organized grid structure. The processing of each LSTM cel only depends on the results of 2 other cells (from the previous layer, and from the previous time step). which make the concurrent execution of many LSTM cells possible given enough hardware resources. We use batch size 64 and a hidden size of 2048. The corresponding TensorFlow graph contains 9,021 operations for a 2-layer model. The number of ops grow roughly proportional with the number of layers.
6.3.4 GNMT
Neural Machine Translation with attention mechanism (Bahdanau et al. 2015 Wu et al., 2016) has architecture similar to that of RNNLM, but its many hidden states make it far more computa- tionally expensive than RNNLM. To reduce the training time, prior work [Wu et al. 2016) propose placing each LSTM layer, well as the attention and the softmax layer, separate device. This strategy demonstrates early success in human placement, we show that GDP can find significantly better placements. We use batch size 64. The original 2-layer encoder-decoder consisting of 28 28,044 operations. An extended 4-layer version consisting of 46,600 operations, An even larger 8-layer version consisting of 83,712 operations.
6.3.5 TRANSFORMER XL
Transformer-XI. Dai et al. 2019) is an modified version of Transformer (Vaswani et al. 2017) that supports segement- level recurrence and novel positional encoding scheme. This innovation enables learning dependency that is 80% longer than RNNs, and 450% longer than vanilla Trans- formers. We use a transformer-Xl. with batch size of 64, sequence length of 256, segment length of 64, model hidden dimension of 500 and feed forward hidden dimension of 1000, 10 heads, and head dimension of 50. The 2-layer Transformer-X: contains 2,618 operations. The number ops grow roughly proportional with the number of layers.
6.3.6 WAVENET
WaveNet (van den Oord et al. 2016) is a generative model for speech synthesis. The model is fully probabilistic and autoregressive, with the predictive ditribution for each audio sample conditioned on all previous ones. Architecturally, WaveNet uses causal convolutions with dilations, to obtain large receptive field. We use WaveNet model with batch size 64 and receptive field size of 2048 (9-layers per stack). An 5-stack WaveNet contains 4,374 operations and a 10-stack WaveNet contains 8,516 operations.
10

---

Under review as a conference paper at ICLR 2020
Table Run time comparison GDP batch training vs, the best of related methods (human expert, METIS, HDP, and GDP no batch training).
Baach Setting
Model
speed (s)
Inception
AmoebaNet
2-yryrr RNNLM
Batch
2-layer GNMT
2-layer Transfema--T.
6 5%
18-layer Wavenet
2-layer RNNLM
4-layer RNNLM
8-layer RNNLM
4 5%
Batch
2-ayer GNMT
4-aaye GNMT
8-layer GNMT
888
Batch
3x8 layer GNMT
5.1%
3xd-Jayer
REFERENCES
Ravichandra Addanki, Shaileshh Bojja Venkatakriishann, Shreyan Gupta, Hongzi Mao, and Mo- hammad Alizadeh. Placeto: Learning generalizable device placement algorithms for distributed machine learning. CoRR, abs/1906.08879, 2019. URL
08879
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR 2015. 2015. URL httpss//arziv..rr/////
1409.0473
Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno A. Olshausen. Superpositon of many models into one. CoRR, abs/1902.05522, 2019. URL http:t:p//axxxx..rr/////
1902. 05522
Zihang Dai.
Improving deep generative modeling with applications. 2019. zihangdai. http::atttt....oc///oooooo.//
https://
pdf
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. ACL., 2019. URL http://arxis v. orgg abs / 901 02860
Yuanxiang Gao, Li Chen, and Baochun Li. Spotlight: Optimizing device placement for training deep neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1676-1684, Stockholmsmssan, Stockholm Sweden, 10-15 Jul 2018, PMLR. URL http: //proceed ngs. ml m w.w.l......./......... /vieess//vv/e/ //w//////../// 8a html
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. NIPS, 2017. URL
02216
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint
2017.
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V. Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. CoRR, ab./181 2018.
org. abs 1811111111 06965
Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. SysML, 2018. URL htthtt:: pettt:://artte //arxiv. org/ abs 1807. 1807.05358
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Exploring the limits of language modeling. arXiv preprint 102.02160. 2016.

---

Under review as a conference paper at ICLR 2020
George Karypis and Vipin Kumar. A fast and high quality multilevel scheme for parti- tioning irregular graphs. SIAM Sci. Comput., 20(1):359 392, December 1998. ISSN 1064-8275. doi: 10.1137/510648775553999999 URL http::/dke wttppxtcccccccoooooooooooooooc.. wtt..000.10110113333333 S F106482755527555
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Azalia Mirhoseini, Hieu Pham, Quoe V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean, Device placement optimization with reinforcement learning. JCML, 2017. URL
org abs////s///// 07940 04972
Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoc V. Le, and Jeff Dean. A hierar- chical model for device placement. ICLR, 2018.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task deepedeentadaptiii metric for improved few-shot learning. CoRR, abs/1805. htss/805..0022. 10123, 2018. URL http::/arxxx
Srg/abs/18055 10123
Aditya Paliwal, Felix Gimeno, Vinod Nair, Yujia Li, Miles Lubin, Pushmeet Kohli, and Oriol
Vinyals. REGAL: transfer learning for fast optimization of computation graphs. KDD, 2019.
URL httphhttt:/:xiiii org/ abs/1b000 02494
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized evolution for image 2018.
classifier architecture search. CoRR,
18C2.01544
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 707.06347, 2017. URL http:://rxxv..rrgg/
1707.00 06347
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprini arXiv: 770100653.. 2017.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to sequence learning with neural networks. CoRR, abs/1409.3215, 2014. htttht//p///xx harg///bs/4403 3215
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. CoRR, abs/1512.00067,, 2015. URL
wwwp:iiiirrg/ab...oo. /1512. 00567
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. CoRR, (bbs//09..0449, 2016. URL http:///xaxx wwtp://ww.wgaab//bb/./.oo/s /b60999..////.......///// 03499
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neu- ral Information Processing Systems 30, pp. 5998-6008. Curran Associates, Inc., 2017. URL http: http:://papprss.ee wwtpeeesssiiss.ccpaappppeeeeeeee wttp:c/pppppppeeeeeeeeeeeeeee.et / http://781-aattettiioo
all--al-ll-ooueeeeeee-eee pdf
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick. Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/16009066 44, 2016. URL /ttt:///arxt org/ abs 609. 081 14
12

---

Under review as a conference paper at ICLR 2020
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful graph neural
networks? ICLR, 2019. /R////ttp///xxx..oo/////////////
Jiaxuan You, Bowen Liu, Rex Ying, Vijay S. Pande, and Jure Leskovec. Graph convolutional pol- icy network for goal-directed molecular graph generation. CoRR, abs/1806.02273. 2018. URL
org/ harg/bbe aww/aasso00 02473
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. Recurrent neural network regularization. CoRR, abs/1409 409.2329, 2014. URLpptt://arxix
wttp://wwwwr/ob//////a////// http:/abs/111..2333 2329
13


Reviews:
[{"review_id": "SyeWKze-qS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This work proposes to use a combination of graph neural networks (GNNs) and proximal policy optimization (PPO) to train policies for generalized device placement in dataflow graphs. Essentially, (1) a GNN is used to learn representations of a dataflow graph (in an inductive manner), (2) a transformer is used to output a device placement action for each node in the graph, and (3) the entire system is trained end-to-end via PPO. Extensive experimental results show very impressive results compared to strong baselines.\n\nAssessment: Overall, this is a solid application paper. The authors GNNs, PPO, and Transformers in an effective, well-motivated, and sound manner. Moreover, the task is interesting and relevant. There is not significant methodological novelty, as the authors are essentially combining standard components in a straightforward way. That said, the results are strong and the paper is well-written, so it certainly has merits as an application paper. \n\nWill code be released? This is essential for reproducibility, as the paper does not contain sufficient technical details to allow for reproduction.\n\nReasons to accept:\n- Strong empirical results on an interesting application \n- Well-written paper\n- Thorough experiments\n\nReasons to reject:\n- Incremental methodological contribution\n- Likely difficult to reproduce", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx27H9AtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper the authors propose an end-to-end policy for graph placement and partitioning of computational graphs produced \"under-the-hood\" by platforms like Tensorflow. As the sizes of the neural networks increase, using distributed deep learning is becoming more and more necessary. Primitives like the one suggested by the authors are very important in many ways, including improving the ability of the NN to process more data, reduce energy consumption etc. The authors compared to prior work propose a method that can take as input more than one data flow graphs, and learns a policy for graph partitioning/placement of the operations in a set of machines that minimizes the makespan. This problem in principle is NP-hard as it entails both graph partitioning and graph scheduling as its components. The authors propose a heuristic that composes of two existing methods: graph neural networks are used to produce an embedding of the computation/data flow graph, and then a seq-2-seq placement network. The method is able to generalize to unseen instances.\n\nI vote for weak reject since some issues that I would like to see addressed by the author(s) are not. These include the fact that since the goal is to minimize the makespan,  scheduling within each machine the operations should be addressed in a better way. Also, while the objective J(\\theta) is reasonable, the distribution for the makespans could be very skewed (e.g., heavy tails over the dataflow graphs). Doesn't this affect the results? Finally, the novelty from a deep learning perspective is limited.\n\n- How do the authors address the issue of scheduling the operations within each machine? \n- How is \\mathcal{D} formally defined (i.e., the range of the mapping function)? Do you take into account the different number of machines, their memory footprints that can be significantly different, the different processing units they may have (GPUs, CPUs, TPUs)? Is the number of machines used for the partition automatically learned by the policy? That part was not very clear.\n- Since the authors compare with METIS, it is worth also comparing with Scotch https://www.labri.fr/perso/pelegrin/scotch/ that is also publicly available.\n- Can the authors comment on the scalability of their method as a function of n (number of nodes), and k (number of  devices)? \n\n\nUpdate: Thanks to the author(s) for the detailed feedback. I have upgraded my score accordingly. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJg8eLkaKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "GDP: Generalized Device Placement for Dataflow Graphs\n\nThis paper presents a method to assign the individual operations making up the dataflow graph of a deep neural network to a set of connected devices on which they should be executed, with the objective of maximizing runtime performance of inference. The paper’s proposed approach relies on graph neural networks to produce embeddings of the dataflow graph nodes that are claimed to be transferable to unseen graphs. These embeddings serve as input to a transformer-based sequence model that outputs the final assignment. The end-to-end model is trained with a reinforcement learning criterion minimizing the expected placement runtime. Experimental comparisons against alternative placement methods (including human expert) are presented for several large neural networks of different architectures.\n\nOverall, the paper is well written, easy to follow, and addresses an important concern is scaling up neural networks to large model sizes. The proposed approach combining graph neural networks with transformer-based placement network appears, at first glance, novel. \n\nMy main reservations with the paper is that it lacks many details in several key sections, preventing a full appreciation of the contributions, and making reproducibility of results impossible to contemplate. In particular:\n\n1. How the placement network is designed and used (section 3.2) is completely lacking.\n2. The details of how the training and testing datasets are obtained are also lacking. In particular, the specific model architectures for the likes of RNNLM, including hyperparameter choices, the runtime data fed to those models, etc. (This can be given in appendix). It is also not clear if the models listed in Table 1 are singular models with fixed values of hyperparameters, or if they correspond to distributions over models (e.g. with differences in hyperparameters).\n3. How PPO is used is glossed over: for instance, is the proposed placement of a given model tried « live » in the inner loop of PPO to generate the reward corresponding to its runtime? What runtime data is fed to the model in order to do this? Since this would be a somewhat unusual setup, an illustration of the overall training loop would certainly solidify understanding. Moreover, a few sentences or equations explaining how PPO is used in this context would also help. Is training with PPO imperative for getting good model performance? Have other methods been tried?\n\nIn section 3.3, the parameter superposition mechanism appears to perform a kind of meta-learning. Explicit connections to such should be made, such as « Tadam: Task dependent adaptive metric for improved few-shot learning » (NeurIPS 2018).\n\nMoreover, at the end of section 4.4, the paper claims to « report superhuman results on 8-layer GNMT ». Unless this reviewer misunderstands, the results in Table 1 (wherein GDP-one clocks in at 0.649 for 8-layer GNMT, versus 0.610 for Human Placement) would contradict this claim. This should be clarified.\n\nGiven these reservations, in spite of the potential of the proposed approach, the paper appears in its current form too immature to recommend acceptance at ICLR.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new reinforcement learning based approach to device placement for operations in computational graphs and demonstrates improvements for large scale standard models.

The paper is borderline with all reviewers appreciating the paper even the reviewer with the lowest score. The reviewer with the lowest score is basing the score on minor reservation regarding lack of detail in explaining the experiments.

Based upon the average score rejection is recommended. The reviewers' comments can help improve the paper and it is definitely recommended to submit it to the next conference.


Decision: Reject