Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a joint placement and routing system based on deep reinforcement learning. They use two RL agents to do placement and routing and train them using a single reward function. This design and the reward structure enables them to reach closer to the global optimum. The results show that the learned policies are effective and beats the baselines they consider.
main_review: # Strengths

* Joint optimisation of placement and routing. This is enabled through their careful design of the reward function that guides the optimization policies to the true target. More specifically, their use of wire length (HPWL) enables the authors to make placement decisions which are cognizant of routing output.
* Use of both a CNN and GNN to model different aspects of the input. The netlist intrinsically has a graph structure, where as the die has a 2D structure. Their use of two different NN topologies to exploit these two different structures is commendable.
* Usage of reward distillation to enable more exploration and to somewhat mitigate the sparse reward problem: authors correctly identifies that the unavailability of intermediate rewards makes the convergence of their policies slower and hence introduces random distillation to mitigate the problem somewhat.
* Adoption of pre-training (tasks separately) and then fine-tuning 

# Weaknesses and questions for authors

* Some of the terminologies of the paper is not that approachable by a non-domain expert. For instance, what is a macro and a standard cell. To improve the clarity of the paper, the authors should have a relatively small background section to ground these keywords that are used over and over again in the text.
* Evaluation
  * Authors mention about ablation studies to validate the usefulness of a multi-view policy network. However, I could not find some of those results in the paper. Specifically, I would like to know how much does having both a GNN and CNN help in the final performance of the policy?
* The baselines are not explained properly. In order for a non-domain expert to appreciate the results, the baselines should be explained better. For instance, how is sequential learning placer different from DREAMPlace. A table with differences would be helpful.
* I am not familiar with the scale of the two benchmarks. Does ISPD-2005 have only 2 benchmarks?  Are these problems of the same scale as usual place and route problems encountered in normal chip design?
* Table 2 and 3 wire length statistics are of different magnitude orders. Is it because in Table 3 you only consider macros? Does that mean standard cell placement is more important than macros since it introduces more wires? Please explain.
limitations_and_societal_impact: Authors mention unemployment of engineers as a negative societal impact. I do not see any other major negative societal impact from this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a joint learning method to solve the placement and routing problems together. Such problems used to be studied separately.  For the placement, this paper is also solving the placement of macro and standard cells together. Moreover, both CNN and GNN are adopted to provide different embedding model.  Finally, the experimental shows that the proposed jointly method has notable performance margin than traditional separate pipeline. 
main_review: Pros:
	This paper is targeting an important challenge in IC design. The idea that jointly solving placement and routing is promising solution since they are strongly coupled. And the placement’s quality will severely impact the routing performance. 
	The related work section is exhaustive which gives the reader a good understanding of the existing works about placement and routing. Not only the traditional methods but also the recently learning-based methods.
	In the Methods part, the author clearly described the proposed method including the problem formulation, reward design, how to calculate wirelength and congestion,  how to utilize both CNN and GNN, and etc.

Cons:
	The major issues are lack of details and insufficient experiment.
	In the experiment, only 2 circuits are used to test the proposed method. The author shows the plot or table but failed to explain the reason behind the result. For example, the author mentioned pre-trained for 300 iterations achieves the best result. But why it achieves best and why only it has a big jump around 100K timesteps?  It is better if the author can re-organize the result, present them more clearly and analyze them in detail not only just show them.

	Minor issues:
	Some abbreviations are not explained such as DRL(Line 37, Page 2), SA(Line 66, Page 3). Although reader can guess their meaning, but it may cause unnecessary mis-leading. 

	The method section should be explained more than it is. Every sub-section only have one paragraph for how it is design. It is better if the author can explain why you design it like this? and how it works in the whole system?. And this will also help reader to follow your main story. 


---raised my score to 6 after authors rebuttal
limitations_and_societal_impact: did not see any negative societal impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a deep reinforcement learning method to solve the chip placement and routing problem. They first introduce an RL based method combined with a gradient-based method to approximate the wirelength. They then expand their RL agent to also support routing task as well.   
main_review: (I couldn't find any supplementary material nor appendix sections.)

Originality: 
- The method is an extension on recent work on deep RL for placement. It also combines a quick placement method (DREAMPlace) for calculating RL reward. They were also the first to propose an RL agent for combined task of placement and routing.  
- The paper provides a great literature review on the previous methods.

Quality:
- The authors provide some experimental results to support the claims, however, more experiments are required. 
- They provided a limited ablation study for their first method (on RND and GNN). 
- More experimental results are required, for example, the results in Figure 4 and 5, have to be an average of multiple runs with different seeds. 
- The authors didn't report any information about the runtime of the method and the impact of transfer learning.  
- The number of blocks in the evaluation are very limited. They could've used bigblue* blocks in the ISPD2005 as well.  
- The existing results do not show the scalability of the proposed methods for real production blocks which include more cells. 
- In Figure 4, Is the RND reward included in the y axis? If yes, then the comparison is meaningless, and if no, please explicitly mention what is y-axis.

Clarity:
- The paper is well written and organized. 
- In Figure 1a, the top left image, what is the unnamed block on the right? 
- Define acronyms before using them: SA, DRL.
- In Section 3.1, Action, It is not clear how macros that are larger than 1x1 are handled? How the grid size n is determined and why is it always square? 
- How RUDY is calculated, since it is not part of DREAMPlace? 
- A newer version of [1] can be find in: Mirhoseini, Azalia, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee et al. "A graph placement methodology for fast chip design." Nature 594, no. 7862 (2021): 207-212.
 

Significance:
- The proposed methods are interesting and important for the community. It provides new methods to address the complex problem of placement and routing and will be likely interesting for the researchers and practitioners. 
limitations_and_societal_impact: The authors explained the limitation of the work and it is possible societal impact.
- It would have been better if they could explain more in depth about the scalability limitation and the runtime of their method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a joint learning method for the placement of macros and standard cells, by the integration of reinforcement learning with a gradient based optimization. The paper also proposed to jointly solve placement and routing via reinforcement learning, which has been the first time attempted in literature. The designed policy network incorporates both CNN and GNN to provide a multi-view embedding with a global embedding and a node level embedding. The paper also proposed a random network distillation to encourage exploration. Empirically, the method surpasses the separate placement and routing pipeline by a notable margin.
main_review: Strengths:
1. The paper is clearly, well written. Explanations on placement and routing problems, graphical illustration are good. The paper provides a thoroughly written related work section. 

2. The proposed method obtains the full placement solution through end-to-end learning without clustering. Jointly solving placement and routing via reinforcement learning is new. The benefits are pretty clear: first, the placement solutions provides abundant training data for the routing agent, instead of relying on randomly generated data used in previous work. Routing, on the other hand, provides a direct objective for the placement agent to optimize, hence mitigating the reliance on an intermediate cost models and reducing bias in the reward. 

3. The placement involves the placement of macros (there are a few of them) and standard cells (there are many of them), which makes the solution more significant than solution [1]. The reviewer applauds the efforts making this possible. 

Weaknesses:
1. The advantage of combing CNN and GNN is small, as indicated in Figure 4. 

2. RND improves the performance significantly, but can be replaced with related methods such as entropy based methods or regularization based methods. The paper does not provide ablation study on that. 

3. There are not sufficient benchmarks. Only two circuits are provided. There is a lack of details on those circuits. 

4. There are not sufficient baselines to compare with. The reviewer understands that it could be a comparatively new domain, and not many related works are available. But the reviewer would be glad to see some comparisons with work [1] by reimplementing [1] in your setting. Also, additional results compared to human placement baseline is highly recommended. 


Detailed comments:
1. [1] approach is not a analytic solver or via gradient updates. It is still a reinforcement learning approach, using a transferable value network. 

2. There isn't a ablation study demonstrating the necessity of multi-view embedding. Comparisons of methods with and without global or node level information should be provided. 

3. How does the intrinsic reward based method via random network distillation compare to a entropy based method such as SAC [2]?

4. Why only two layers of GNN is used? It seems to small for a problem like placement and routing. 

Overall, the reviewer likes the paper about its novelty in combing RL and gradient-based methods, jointly optimizing placement of macros and standard cells end to end, and jointly optimizing placement and routing, the proposal of pretraining on a simpler proxy task. However, the downside is its insufficient comparisons with related work. The paper needs some stronger baselines. 


[1] Chip placement with deep reinforcement learning, https://arxiv.org/abs/2004.10746
[2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
, https://arxiv.org/abs/1801.01290
limitations_and_societal_impact: Both limitations and societal impact are sufficiently addressed.
review_title: null

Meta-review: There was a consensus among reviewers that this paper should be accepted. It is the first to propose an RL agent for the combined task of placement and routing in chip design. The main critique after initial reviews were insufficient experiments. However, this was addressed to the reviewers' satisfaction by extensive additional experiments in the rebuttal. Hence, I recommentd acceptance of this paper.
Decision: Accept (Poster)