Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper describes a machine learning method for mapping computational nodes in a neural network onto different levels of the memory hierarchy (DRAM, LLC, and SRAM) to minimize latency of inference. The proposed approach builds on CERL, combining policy gradient, reinforcement learning, and graph neural network, achieving 28-78% speed-up over the native NNP-I compiler on vision and language benchmarks (ResNet-50, ResNet-101, and BERT).

Overall, the paper was well-written, targets an impactful problem, and the reported improvements (28-78% over native compiler) are impressive.

In the related work section, I did have a concern, as the authors state “For example, previous work with manual grouping (sic) operate at most in 5^280 \~= 10^196 dimensional action space (Mirhoseini et al., 2020), compared to 10~ 20^358 for our BERT problem”. However, Mirhoseini et al., 2020 (“Chip placement with deep reinforcement learning”) places “a few thousand clusters” (>=2000 nodes) onto a grid with “an average of 30 rows and columns” (~900 cells), so wouldn’t the action space be at least 900^2000? Also, didn’t that work use a heuristic grouper (hMETIS), but maybe that’s close enough to “manual”? 

The authors only look at three benchmarks, but they were well-chosen (two representative vision models and one large language model). It’s also good that they compare against PG and EA alone as a form of ablation, given that their method is effectively a combination of these two. It would have been better if they also had compared with prior state-of-the-art (e.g. HDP, REGAL, Placeto, or (the unmentioned) GDP / GO), but it is somewhat understandable given that their code does not seem to be open-sourced.  

I liked that the authors report mean and standard deviation for the five runs, and measured “true” reward by running on hardware. I also thought they did a good job motivating their method (aside from the questionable statements about action spaces in prior work), and of analyzing and visualizing its performance. 

Nits:
In the Method section, “the compiler rectifies them and outputs a modified map, M_c, that is fully executable (Line 6).” It would probably be good to add “in Algorithm 1” so as not to confused the reader.

“It comprises of a single PG learner” -> “It is comprised of…”

“Both methods are known to produce highly performant and stable solutions but are also significantly slow compared to Deep RL” (“significantly slower than”?)

“While the transferred policies are clearly underperform those from scratch” -> “underperforming”
review_title: Well-written paper on memory mapping method that outperforms native NNP-I compiler by 28-78%

final_score: rating: 7: Good paper, accept
review_content: Optimizing the execution of deep neural networks has tremendous impact on the cost and performance in many industries due to the proliferation of "Deep Learning". There has recently been an interesting line of work of using learning to optimize policies related to placement and scheduling of the neural network computation graph outperforming tediously hand-crafted heuristics. The proposed paper would be a nice extension along this line.
The impact of memory placement for DNN has been clearly motivated and is easy to appreciate.
The paper overall is clear and easy to follow. The methodology is sound and justified. Experiments and results make a compelling case in supporting the claims.

I think one aspect that is insufficiently motivated is the need to use a hybrid approach between RL and evolutionary algorithms. Results show improvement in performance but it is not clear to me why. Perhaps this is addressed in the CERL paper which I am not familiar with.

In the results section, it would be nice to see the sample complexity of each of the methods. I see that the number of iterations are shown in Figure 3 but it is not clear to me if that also corresponds to the number of samples consumed by each of the approaches before they converge.
review_title: Interesting extension of the Graph Optimization using DRL line of work

final_score: rating: 7: Good paper, accept
review_content: The paper proposes Evolutionary Graph Reinforcement Learning to solve the memory placement problem. Main ideas are using GNN as the network architecture for reinforcement learning agents that look for more informed priors for evolutionary algorithms. Overall novelty of the paper comes from the neat combination of RL, EA, and GNN, and applying it to memory placement (ML for Systems).

The paper indeed tackles an important problem that can affect the overall performance and efficiency of the hardware. I believe the reorganization of various off-the-shelf ML techniques to solve real problems in the systems domain marks a large contribution, hence the positive overall rating.

One of the main drawbacks of the paper is that the paper only tests on a single type/configuration of hardware. While this is fine to some extent, this makes it hard to get confirmation about the generality of the overall method considering the large variance of the speedup.

Another related question comes from how this work relates to the optimizations of the dataflows [1,2]. As it is difficult to evaluate the overall memory communication without considering the order of operations, etc. the work in turn neglects the big question and focuses on only the partial view of the problem. It would provide a nice reference point if some of these points are discussed in the paper.

Last question comes from the baselines. While the previous works on tensor optimizations [3,4] are very closely related and many of the ideas provide a good comparison point, these have not been discussed nor cited. For example, I guess AutoTVM's way of approximating the search space using TreeGRU or XGBoost can help. Also, Chameleon's way of sampling the examples using adaptive sampling may provide an interesting reference point in terms of reduction of number of samples.

Overall, I have enjoyed reading the paper and I find the ideas in the paper interesting. I am currently weakly pro for the paper, and look forward to the authors' response :)

Questions
1. Could you provide and overview of the NNP-I's architecture in the appendix? Also, possibly ablation studies over different configurations of the hardware.
2. What are the relative communication speed of SRAM, LLC, and DRAM? 
3. It is well discussed in the computer architecture community that the memory communication is very much determined by the dataflows of the architecture. How are the results affected by these dataflows?
4. How does the work compare to the methods described in [3,4]?

[1] "Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks", ISCA 2016

[2] "Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators", ASPLOS 2020

[3] "Learning to optimize tensor programs", NeurIPS 2018

[4] "Chameleon: Adaptive code optimization for expedited deep neural network compilation", ICLR 2020
review_title: Official Blind Review #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper proposes a new algorithm called EGRL to improve computation graph running time by optimizing placement of the graph's components on memory. Specifically, the authors demonstrate the algorithm on the Intel Neural Networks Processor for Inference (NNP-I), which allows them to map neural network components on one of three memory hierarchies, each with different tradeoffs. The authors demonstrate that this technique provides speedups on BERT, Resnet-50 and Resnet-101.


Pros:

- Some past papers (for eg. [1]) in this domain evaluate their work in simulators instead of real hardware, and often, the simulators make assumptions that are not realisitc. The paper tests its technique on actual hardware, and this is definitely a plus.
- The authors promise that they will open-source their code. This is important since many of the efforts in this domain remain fragemented and difficult to reproduce. This is primarily due to the lack of open source code or a standard benchmark.
- The paper is well written.
- The visualisations of the learned policy vs the baseline in Figure 6 are quite good.
- EGRL directly builds upon CERL so it is not very novel, but it has not been applied before to this domain.

Cons:

- The paper evaluates the technique on just 3 workloads. This is in contrast to [1] who evaluate on 372 different workloads and [2] who evaluate on 96 synthetic graphs. [3] and [4] also evaluate on a very small number of workloads, but I believe they probably got a freepass since they were the earliest works in their domain.
- The baseline that the experiments are being compared against might be weak - in figure 3, it looks the policy from both EGRL and EA in iteration 0 itself beat the baseline for Resnet-101 and BERT! 
- How long does it take to perform one iteration? And how long does it take to train the policy? This would be useful to get an idea of how EGRL fairs against [3] and [4] which also trained on real hardware and took many hours to finish training the policy.
- The demonstration of generalizability is insufficient - it is difficult to conclude that EGRL can generalize to other workloads. For eg. in Figure 4 (left), the policy performs worse than the baseline for Resnet-50. Moreover, two of the workloads are from the Resnet family.
- If it takes a long time to train each policy and if the model also shows poor zero-shot generalizability, it makes me question if this approach is practical for a compiler setting where a user would typically want the compilation to be completed quickly.

Overall:

I felt that the paper has some interesting ideas but needs more experiments.


Questions and Clarifications:

- I believe that the related work section should add a clarification - [1], [2], [3] and [4] primarily deal with device placement, i.e., placing components of computation graph on different CPUs/GPUs to optimize run time via better parallelization. While this work is concerned with mapping components to different memory hierarchies on the same device. 

- While EGRL's action space is larger than [5], the action space in [1] is much larger - for a graph with 2000 nodes to be placed on 2 devices, there are 2^2000 possible choices ~ 10^603

- In Figure 6 (Bottom), is there any reason why you didn't show the result for BERT?


References:

[1] Aditya Paliwal,  Felix Gimeno,  Vinod Nair,  Yujia Li,  Miles Lubin,  Pushmeet Kohli,  and Oriol Vinyals. Reinforced genetic algorithm learning for optimizing computation graphs. arXiv preprint arXiv:1905.02494, 2020

[2] Ravichandra Addanki, Shaileshh Bojja Venkatakrishnan, Shreyan Gupta, Hongzi Mao, and Moham-mad Alizadeh. Placeto: Efficient progressive device placement optimization. In NIPS MachineLearning for Systems Workshop, 2018

[3] Azalia Mirhoseini, Anna Goldie, Hieu Pham, Benoit Steiner, Quoc V. Le, and Jeff Dean. A hierarchical model for device placement. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=Hkc-TeZ0W.

[4] Azalia Mirhoseini, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, NaveenKumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. Device placement optimization with reinforcement learning. arXiv preprint arXiv:1706.04972, 2017.

[5] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, Azade Nazi, Jiwoo Pak, Andy Tong, KavyaSrinivasa, William Hang, Emre Tuncer, Anand Babu, Quoc V. Le, James Laudon, Richard Ho,Roger Carpenter, and Jeff Dean. Chip placement with deep reinforcement learning. arXiv preprint arXiv:2004.10746, 2020.
review_title: Review

Meta-review: Most of the reviewers agree that this paper presents interesting ideas for an important problem. The paper could be further improved by having a thorough discussion of related works (e.g. Placeto) and construct proxy baselines that reflect these approaches.


The meta-reviewer decided to accept the paper given the positive aspects, and encourages the author to further improve the paper per review comments.


Thank you for submitting the paper to ICLR. 

Decision: Accept (Poster)