On Joint Learning for Solving Placement and Routing in Chip Design
Authors: Ruoyu Cheng
Junchi Yan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Combinatorial Optimization

Abstract: For its advantage in GPU acceleration and less dependency on human experts, machine learning has been an emerging tool for solving the placement and routing problems, as two critical steps in modern chip design flow. Being still in its early stage, there are several fundamental issues unresolved: scalability, reward design, and end-to-end learning paradigm etc. To achieve end-to-end placement learning, we first propose a joint learning method for the placement of macros and standard cells, by the integration of reinforcement learning with a gradient based optimization scheme. To further bridge the placement with the subsequent routing task, we also develop a joint learning approach via reinforcement learning. One key design in our (reinforcement) learning paradigm involves a multi-view embedding model to encode both global graph level and local node level information of the input macros. Moreover, the random network distillation is devised to encourage exploration. Experiments on public chip design benchmarks show that our method can effectively learn from experience and also provide high-quality intermediate placement for the post standard cell placement, within few hours for training.

On Joint Learning for Solving Placement and Routing in Chip Design
Ruoyu Cheng Junchi Yan'
Department of Computer Science and Engineering MoË Key Lab of Artificial Intelligence, Al Institute Shanghai Jiao Tong University, Shanghai, China, 200240 (roy account, yan j@sjtu.edu. cn
Abstract
For its advantage in GPU acceleration and less dependency on human experts, machine learning has been an emerging tool for solving the placement and routing problems, as two critical steps in modern chip design flow. Being still in its early stage, there are fundamental issues: scalability, reward design, and end-to- end learning paradigm etc. To achieve end-to-end placement learning, we first propose joint learning method termed by DeepPlace for the placement of macros and standard cells, by the integration reinforcement learning with a gradient based optimization scheme. To further bridge the placement with the subsequent routing task, we also develop a joint learning approach via reinforcement learning fulfill both macro placement and routing, which is called DeepPR. One key design in our (reinforcement) learning paradigm involves a multi-view embedding model encode both global graph level and local node level information of the input macros. Moreover, the random network distillation is devised to encourage exploration. Experiments public chip design benchmarks show that method can effectively learn from experience and also provides intermediate placement for the post standard cell placement, within few hours for training.
Introduction
With the breakthrough of semiconductor technology, the scale of integrated circuit (IC) has surged exponentially, which challenges the scalability of cxisting Electronic Design Automation (EDA) algorithms and technologies. Placement is one of the most crucial but time-consuming steps of the chip design process. It maps the components of a netlist including macros and standard cells to locations the chip layout, where standard cells are basic logic cells e.g. logic gates and macros are functional blocks e.g. SRAMs. good placement leads to better chip area utilization, timing performance and routability. Based on the placement assignment, routing assigns wires to connect the components, which is strongly coupled with placement task. In addition, the placement solution also serves rough estimation of wirelength and congestion, which valuable in guiding the earlier stages of design flow. The objective of placement is to minimize metrics of power, performance, and area (PPA) without violating the constraints such placement density and routing congestion.
We provide a pipeline to the placement problem: the input of placement is a netlist represented by hypergraph H (V, E), where V denotes set of nodes (cells) and E denotes set of hyperedges (nets) that indicates the connectivity between circuit components. Marco placement firstly determines the locations of macros the chip canvas, followed by immense numbers of standard cells adjust their position based on adjacent macros and finally obtains the full placement solution, as shown in Fig. ((a) The routing problem, however, takes placement solution as input and tries to connect
*Corespondence author is Junchi Yan.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

those electronic components in a circuit coarsely. Without violating constraints on edges between neighboring tiles, the target of routing is to minimize the total wirelength, as shown in Fig. [(b]
In this work, we first propose an end-to-end learning approach DeepPlace for placement problem with two stages. The deep reinforcement learning (DRL.) agent places the macros sequentially, followed by a gradient-based optimization placer to arrange millions of standard cells. In addition, we develop a joint learning approach DeepPR (i.e. deep place and routing) for both placement and the subsequent routing task via reinforcement learning. The main contributions of this paper are:
1) For learning based placement, we propose end-to-end approach DeepPlace for both macros and standard cells, whereby the two kinds of components are sequentially arranged by reinforcement learning and neural network formed gradient optimization, respectively. To our best knowledge, this is the first learning approach for the joint placement solving of macro and standard cells. In the previous works, these two tasks are independently solved either for macro [T] standard cells 2
2) We also propose DeepPR to jointly solve placement and routing via (reinforcement) learning, which again to our best knowledge, has not been attempted in literature before.
3) To adapt reinforcement learning more effectively into our pipeline, we design a novel policy network that introduces both CNN and GNN to provide two views to the placement input, in contrast to previous works that use CNN or GNN [ alone to obtain the embedding. The hope that both global embedding and node level embedding information can be synthetically explored. We further adopt the random network distillation to encourage exploration in reinforcement learning.
4) We provide experimental evaluation for both joint macro/standard cell placement and joint place- ment & routing. Results show that our method surpasses the separate placement and routing pipeline by a notable margin. Code partly public available at: https//giiubb.cco///iaaaaaasssrt
2 Related Work
Classical methods for Placement. The history of global placement can trace back to the 1960s 5. A wide variety of methods have been proposed since then, most of which fall into three categories: partitioning-bast based methods, stochastic/hill-cimbbs
methods, and analytic solvers.
In early years, partition-basel methods adopted the idea of divide-and-conquee netlist and chip layout are partitioned recursively until the sublist can be solved by optimal solvers. This hierarchical structure makes them fast to execute and natural to extend for larger netlists, while scarifying the solution quality since each sub- sub-problem is solved independently. Some multi-level partitioning methods 77) were developed afterwards. Stochastic and hill-climbing, as the second category, are mainly based on annealing methods (8) which is inspired from annealing in metallurgy that involves heating and controlled cooling for optimal crystalline surfaces. In practice, simulated annealing (SA) optimizes a given placement solution by random perturbations with actions such as shifting, swapping and rotation of macros g9 [0]. Although SA is ficxible and able to find the global optimum, it is time-consuming and hard to deal with the ever-increasing scale of circuit.
For the analytic solvers, force-directed methods [T] and non-lineat optimizers 12 1 adopted. In comparison, the quadratic methods are computational efficient but showing relatively lower performance, while non-linear optimization approximates the cost functions more smoothly with the cost of higher complexity. Recently, however, modern analytical placers such as ePlace [14] and RePlAce [15] introduce electrostatics-base global-smooth density cost function and Nesterov's method nonlinear optimizer that achieve superior performance on public benchmarks. They formulate each node of the netlist as positively charged partiele. Nodes are adjusted by their repulsive force and the density function corresponds to system's 's potential energy. These analytical methods update positions of cells in gradient based optimization scheme and generally can handle millions of standard cells by parallelization on multi-threaded CPUs using partitioning reduce the running time.
are commonly
Nevertheless, all methods mentioned above perform heavy numerical computation for large-scale optimization problem on the CPUs, which lacks exploration of GPU's opportunity. DREAMPlace 2 inspired by the idea that the analytical placement problem is analogous to training a neural network. They both involve optimizing parameters and minimizing cost function. Based on the state-of-the-art analytical placement algorithm RePlAce, DREAMPlace implements hand-optimized key operators by deep learning toolkit PoTch and achieves over 30x speedup against CPU-based tools.

---

N2
N1
N3
macro placerent
standard cells
(N2:6, D)
(N3:C,
(N4:A, B, C)
Internaciate placesent
Input: netlist
(a) Introduction placement pipeline. Based the input netlist with node features, we are supposed determine positions macros as well as standard cells chip canvas.
(N2:8, C, (N2: IN3:C, (NA:A, B, netlist
routing
A
evaluate
N/3 N
total wirelength
A
B
result
placement solution
(b) Introduction to routing pipeline. Once the placement result has been obtained, routing assigns the wires to connect components in each net such as N1 and N2. Figure Illustration for concepts and the pipeline involved placement and routing problems.
Learning-base methods for Placement. Recently learning-bass methods for placement especially reinforcement learning (RL) have been proposed to obtain the generalization ability. Google [ proposes the first end-to-end learning method for placement macros that models chip placement as sequential decision making problem. In each step, the RL agent places one macro and target metrics are used reward until the last action. GNN is adopted the value network to encode the netlist information and deconvolution layers in the policy network output the mask of current macro position. Another line is to solve placement problem by combining RL and heuristics [16]. They propose a cyclic framework between the RL and SA, where RL module adjusts the relative spatial sequence between circuit components and SA further explores the solution space based on initialization. In comparison, the former one is learning based approach with the same objective function as analytic solvers, while the later essentially an annealing solver.
Classical & Learning-baid based methods for Routing, Global routing generally begins with decom- posing a multiple-pin net problem into set two-pin connection problems 17
After that, each pin-to-pin routing problem is solved by classical heuristic-based router such as rip-up and region-wise routing. Recently, machine learning techniques have been applied for routing information prediction, including routing congestion [22]. routability of a given placement (23) and circuit performance 244. Meanwhile, RL method has also been proposed to handle routing problems.
and reroute (20), force-directed routing
(25) uses DON model to decide actions of the routing direction, i.c. going north, south, etc at each step. [26) proposes an attention based REINFORCE algorithm to select routing orders and use pattern router to generate actual route once routing order is determined. While [27) applies genetic algorithm to create initial routing candidates and uses RL to fix the design rule violations incrementally.
Different from current learning beasenng-based placer such
that groups the standard cells into a few thousand clusters and uses force-directed methods for coarse placement solution, our work combines reinforcement learning with gradient based optimization scheme without clustering to obtain full placement solution through end-to-end learning scheme. While [5] considers local density function as penalty factor to address the requirement for routability, the proposed method, to our best knowledge, is the first attempt jointly solve placement and routing via reinforcement learning.

---

initial state
niddle state
final state
placement result
reward
(1) agent places one nacro in each step
(2) adopt gradient optinization for complete solution
(3) update the policy network via p2o algorithn
(a) Diagram of our learning-basee placer for arranging both macros and standard cells (DeepPlace)
initial state
middle state
final state
routing solution
wirelength
(1) agent places one stop
(2) classical router (or RL nodel) decides routing direction
optimize tasks the
function
(b) The joint learning approach to solve placement and routing via reinforcement learning (DeepPR)
Figure 2: Overview how our RL agent involves two subsequent tasks. We term our two methods by DeepPlace (for placement only) and DeepPR (i.e. deep placement and routing), respectively.
3 Methodology
3.1 Problem Formulation
First of all, we target the problem of macro placement, whose objective is to determine locations of macros on the chip canvas with no overlap and wirelength minimized. Our RL agent sequentially maps the macros to valid positions on the layout. Once all macros have been placed, we either fix their positions and adopt gradient-based placement optimization obtain a complete placement solution with corresponding evaluation metrics such wirelength and congestion, shown in Fig. 2(a) Or alternatively, we adopt classical router or another RL agent route the placement solution and regard the exact total wirelength rewards for both placement and routing task, which is shown in Fig. 2(b) The key elements of the Markov Decision Processes (MDPs) are defined as follows:
State s; the state representation consists of two parts, global image / portrayed the layout and netlist graph H/ which contains detailed position all macros that have been placed. Note that placement problem is analogous to board game (i.e., chess or GO), both of which are required to determine the position of pieces (macros). Therefore, we model the chess board (chip canvas) as binary image I where 1 denotes position that has been occupied. In addition, the rule of board game similar netlist graph H, which complementary the global image.
Action o the action space contains available positions the n x n canvas at time t, where n denotes the size of grid. Once spare position (æ, y) is selected by the current macro, set Lzy 1 and remove this position from the available list.
Reward rst the reward the end of episode is negative weighted sum of wirelength and routing congestion from the final solution. The weight is trade-off between main objective wirelength and routing congestion which indicates the routability for routing task. Different from other deep RI. placers that set the reward to for all previous actions, we adopt random network distillation (RND) inspired from 4 calculate intrinsic rewards at each time step.
The policy network learns to maximize the expected reward from placing prior chips and improves placement quality over episodes, which is updated by Proximal Policy Optimization (PPO) (28).

---

target f
flo
tuc
Canvas Smage
observation
PoLScy
=.X.
netlist grach
spodictor network /
(b) The random network distillation module Figure 3: Policy network architecture and random network distillation module for end-to-end placement (and routing). (a) The CNN network takes global image (as a representation for current placement) as input and generates the global embedding fe, while detailed node embedding f is obtained from GNN backbone. The policy network outputs the distribution over available positions after concatenation and fc layer. (b) Randomly initialized target network and predictor network take observation to generate embedding (o) and (o; 0) respectively. The expected MSE between two embeddings is used as intrinsic reward as well as loss function for predictor network in each step.
(a) Our policy network architecture
3.2 The Structure of Policy Network
Since both solving placement problem and playing board game such GO essentially to determine the position of macros (pieces) in a sequential manner, we model the current state as an image I of size n. We select from (32, 64) in the experiments. Tay when previous macro placed on position (x, y). This image representation gives an overview of the partial placement with loss of some detailed information. We further obtain the global embedding from convolutional neural network (CNN) which shows significant performance in GO game [29). Moreover, the netlist graph critical input information implies the rule of reward calculation, which is detailed guidance on the action prediction. In this case, we develop graph neural network (GNN) architecture that produces detailed node embedding for current macro consideration. The role of graph neural networks is to explore the physical meaning of netlist and distill information about connectivity of nodes into fow-dimensional vector representations which utilized in following calculations. After we obtain both global embedding from CNN and detailed node embedding from GNN, we fuse them by concatenation and pass the result to fully-con celle-cooneeted layer to generate a probability distribution over actions. We believe the multi-view embedding model is able to synthetically explore global and node level information. The whole structure policy network shown in Fig. 3(a)
3.3 Reward Design
3.3.1 Extrinsic Reward Design
Although the exact target of chip placement is minimize power, performance and area, it takes aedustry-standant EDA tools several hours to evaluate, which is unaffordable for RI. agent that needs thousands of examples to learn. It is natural to find approximate reward functions that positively correlated with true reward. We define cost functions with both wirelength and congestion, trying to optimize the performance and routability simultaneously:
Rg - Wwwlength((P... H) - A Congestion(P.) H)
(I)
where P denotes the placement solution, H denotes the netlist and À a hyperparameter that weights the relative importance according the settings of given chip.
Wirelength. As the true wirelength is actually decided by routing, we employ half half-perimeter wirelength (HPWI.) as the approximation for wirelength. The definition of HPWI. is half-perimeter of the bounding boxes for all nodes in the netlist, which can be roughly assumed as the length of its Steiner tree (the infimum bound of routing). For given net n, with the set of end points {z;) and )y.). the HPWL can be calculated by:
HPWL((.;) (MAX {a.) MIN {x)] 1) + (MAX (u:) MTN (v:) + +)
Congestion. The idea of congestion comes from the fact that each routed net will occupy a certain amount of available routing resources. We adopt Rectangular Uniform wire Density (RUDY) 30 to
5

---

approximate routing congestion as HPWL is an intermediate result during the calculation process, and accuracy of RUDY relatively high. We set the threshold for routing congestion as 0.1 in our experiments, otherwise it causes congestion check failure.
3.3.2 Intrinsic Reward Design
Note that no useful signal can be provided until the end of a trajectory. It makes the placement sparse reward task. Episodic reward is often prone to make algorithms get stuck during the training process and suffer from inferior performance and inefficient sample complexity. Inspired by the idea of random network distillation (RND) 4, we give an intrinsic bonus in each time step to encourage exploration shown in Fig. 3(b) There are two networks involved in RND; fixed and randomly initialized target network and a predictor network trained on global images collected by the agent. Given the current observation o (which also refers to the global image), the target network and predictor network generate embedding (o) and fo; 0) respectively. The intrinsic reward is:
Ry = [lf(o f(o)]l
After that, the predictor network is trained by SGD minimize this expected MSE which distills randomly initialized network into a trained one. This distillation error could be seen as quantification of prediction uncertainty. As novel state is expected to be dissimilar what the predictor has been trained on, the intrinsic reward also becomes higher to encourage visiting novel states.
3.4 Combination with Gradient- Placement Optimization
The placement for millions of standard cells is as important as placing the macros, considering its influence on the evaluation metrics for macro placement. To ensure the runtime of each iteration affordable for training, we apply sfate-of-the-ar gradient based optimization placer DREAMPlace [ arrange standard cells in the reward calculation step. On the hand, the position of large macros as fixed instances will influence the solution quality of gradient based optimization placer, which improve over time through training. On the other hand, better approximation as wirelength leads to better guidance for training the agent. As a result, the combination of RL agent with gradient based optimization placer will mutually enhance each other. Furthermore, the state-of-the-ar tool DREAMPlace implements key kernels in analytical placement, e.g., wirelength and density computation with deep learning toolkit, which fully explores the potential of GPU acceleration and reduces the runtime in less than minute.
the metrics such
3.5 Joint Learning of Placement and Routing
Routability one of the most critical factors to consider during placement, hence routing congestion is necessary component in the objective (reward) function in most previous methods. However, congestion as an implicit routability model is rough and not always accurate. Meanwhile, HPWL as proxy of wirelength also introduces bias towards the true target. This motivates learn placement and routing task, both of which try to minimize the wirelength in practice. We can adopt any routing method in DeepPR including RL model as well as classical router to determine the routing direction after decomposing the netlist obtained from the placement task to pin-to-pin routing problems. The overall wirelength is then used as episodic reward for both placement and routing agents to optimize two tasks respectively. The advantages of this joint learning paradigm are twofold. On one hand, placement solution provides abundant training data for the routing agent, instead of randomly generated data used in previous work which lacks of modeling the distribution of real domain data. On the other hand, routing provides a direct objective for the placement agent to optimize, hence relieving the need of intermediate cost models and reducing bias in the reward signal.
jointly
4 Experiments
4.1
Benchmarks and Settings
We perform experiments based on the well-studied academic benchmark suites ISPD-2005 [1]. which has not been investigated by learning-basee approaches before, to our best knowledge. To demonstrate the benefit from our end-to-end placement learning of both macros and standard cells, we embody the same designs as the ISPD-2005, except that most fixed macros are exchanged to movable ones for the agent to organize. Parameters of all the eight circuits in ISPD-2005 benchmark

---

Table 1: Statistics for our modified ISPD-2005 benchmark suites. RL agent only places those movable macros, and design density is the ratio of the area sum of total objects over area of placement region.
Circuits Total Cells
# Nets
Movable Macros
Fixed Macros
Design Density
adaptecl
211K
221K
514
29
75.7%
adaptec2
255K
266K
542
24
78.6%
adaptec3
451K
466K
710
74.5%
adaptec4
496K
516K
1309
20
62.7%
bigbluel
278K
284K
551
9
54.2%
bigblue2
558K
577K
948
22136
37.9%
bighlue3
1097K
1123K
1227
66
85.6%
bighlue4
2177K
2230K
659
7511
44.3%
after modification are summarized in Table ! This new benchmark is able to serve as an appropriate environment for evaluating algorithms for the macro placement. We also study two large-scale dataset bighlue3 and bighlue4 which include millions of cells to verify the scalability of our proposed method. It is worth noting that method can handle flexible number of movable/fixed macros well.
We use PPO in (28) for all the experiments implemented with Pytorch [32, and the GPU version of DREAMPlace 22 is adopted gradient based optimization placer for arranging standard cells. We adopt GCN 33 as the GNN backbone which consists two layers with 32 and 16 feature channels, respectively. The Adam optimizer [34] is used with 2.5 x 10-4 learning rate. All experiments run on NVIDIA GeForce RTX 2080Ti GPU, and parallelism is not considered in the experiment. Since reinforcement learning environment is deterministic, the error bars can be neglected.
4.2 Pretraining
Our goal is to train a reinforcement learning agent that can generate high-quality intermediate placement for the post standard cell placement task as well as subsequent routing task. However, training the end-to-end learning paradigms from scratch may suffer from longer time for convergence or even worse final performance, as the majority of the runtime taken by gradient based optimization placer in each episode. In addition, the initial policy network starts off with random weights and generates low-quality intermediate placement, which needs more steps for exploration in the immense state space. Our intuition is that an agent capable of minimizing the wirelength only containing those macros should be good enough to provide an intermediate placement. Therefore, we propose to train the policy network by a partial netlist which only consists of connectivity between macros, and then finetune this pre-trained policy for subsequent tasks. The idea is similar to curriculum learning that gradually introduces more difficult task to speed up training.
4.3 Results on Joint Macro/Standard cell Placement
As mentioned previously, our RL agent generates intermediate macro placement and then adopts gradient-based optimization placer to obtain complete placement solution, while utilizing the pre- trained policies training with partial netlist consisting of macros only. It is worth noting that minimizing the wirelength between macros (i.e., the pre-training task) will not directly lead global minimum of full placement, thus to what extent we pretrain the agent needs to consider attentively. Fig. 4 shows the convergence plots for agents training from scratch versus from pre-trained policy for different timesteps circuit adaplecl. The agent pre-trained for 300 iterations achieves the best result, while training from scratch tends to be less stable and performs the worst. This may due to the increasing complexity of reward function, so that it is hard for agent to predict performance of standard cell placement afterwards. Although the network pre-trained for 500 iterations starts with higher reward, it fails obtain a better placement, likely because excessive pre-training hurts exploration and is prone to overfitting in terms of true reward function. Hence we finetune the policy based on moderate pre-training in further experiments.
In Table 2 we compare the total wirelength of our end and-to-end learning approach DeepPlace with sequential learning placer that applies gradient based optimization after pre-training separately, as well as DREAMPlace which arranges movable macros by heuristie in the beginning of optimization. Both DeepPlace and sequential learning placer consistently generate viable placements, while DREAMPlace fails to meet congestion criteria in most cases for overlap between macros, which shows the advantage of our RL agent for macro placement. Our end-to-end approach DeepPlace
7

---

Table Comparison on macro/standard cell placement task on circuits from ISPD-2005. Per-
formance w.r.t wirelength for three methods: 1) our end-to-end DeepPlace, 2) sequential placer (improving [T] for maero and using DREAMPlace for standard cell placement, separately) and 3)
apply DREAMPlace directly on both macro and standard cell by treating them general cells.
Circutss
DeepPlace (ours)
Sequential Learning placer
Analytic Placer DREAMPlace
adaptecl
80117232
86675688
83749616
adaptec2
123265964
124125088
congestion check failure"
adaptec3
241072304
257956784
264789408
adaptec4
236391936
255770672
congestion check failure
bigblucl
140435296
168456768
congestion check failure
bighlue2
140465/188
142037456
congestion check failure
congestion check failure congestion check failure Note RePlace [S] supposed to always achieve the same wirelength as DREAMPlace (see also in because they only differ in the optimization schedule and DREAMPlace speeds up RePlace. The final macro/standard cell placement result fails meet congestion criteria as discussed in Sec. 3.3.1
bighlue3
150633360
460581.568
bigblue4
951984128
1039976064
which is probably due to overlap between macros. our analysis, the reason may be that DREAMPlace is not suited for macro placement as it directly searches for x-y position in continuous space while the other two solvers applies RI. for macro placement whose action is discrete by nature with some protection against overlap.
100
M man
w M
episodic
training scrotch
-100
pre-trained for 100 iterations pre
iterations pro trained for 500 ieerations
100
Timesteps (K)
Figure 4: Comparison of agents pre-trained on circuit adaplecl. Mean episodic return (i.e., the extrinsic reward with both wirelength and congestion) increases the iterations of pre-training increase in a certain range, while excessive pre-training hurts exploration.
also outperforms the sequential one in all test cases, demonstrating the advantage of joint learning paradigm. Fig. 5 we further compare the runtime of DeepPlace with classical placer RePlace on eight circuits from SSPD-2005 benchmark. We can see that the runtime of RL agent for macro placement grows much slower than gradient based optimization placer for standard cells, and our total runtime achieves approximately 4x speedup over RePlace on various circuits. Finally, the macro/standard cell placement results are visualized Fig.
4.4 Results on Joint Placement and Routing
We compare joint learning approach DeepPR to the separate pipeline of placement and routing. We evaluate our method on the same circuits as mentioned above, except that all standard cells are omitted for convenience. The strategy of pre-training is adopted in the experiments as well to speed up the training procedure and reach a higher reward, since training from scratch requires collect more experience. The evaluation of final wirelength after routing are shown in Table 3 It shows that the reinforcement learning agent dramatically improves the placement quality of macros with routing wirelength as the feedback in all test cases, compared with the separate pipeline. The significant difference between DeepPR and the separate pipeline indicates that HPWL as commonly used objective in placement is not accurate enough approximate the exact total wirelength after routing. Meanwhile, our joint learning scheme is able to provide unbiased reward signal by optimizing directly towards final metrics, which is potential to further increase the performance of current placers.

---

Our fi. Nacro Mlccament
Our graclent--uun lused slandard call plocemett
Classical Macer Rcfoce
Chesal
Figure 5: Runtime breakdown and comparison with classical placer on eight circuits from ISPD-2005. In our method DeepPlace, the RL module only occupies small percentage of runtime for larger circuit, and externally achieves approximately 4x speedup over the classic placer RePlace [15). Time cost of sequential learning placer the same DeepPlace, as their only difference lies in training protocol while model structure is the same. As shown in Table some results produced by RePlace faced with high congestion problem due to overlap between macros.
25000
10000
0000
4000
2000
JC-
5000
10000
15000
20000
25000
2000
4000
6000
8000
10000
12000
(a) circuit adaptec3
(b) circuit bighluel
Figure 6: Visualization of macro (in orange) /standard cell (in blue) placement by our DeepPlace on circuits adaptec3 and bigbluel from ISPD-2005. Our RL agent tends to place macros in each net close to each other, and searches the nearest valid position to replace two macros overlap, which perhaps contributes the final formation of diamond -shapdd placement.
Table 3: Final wirelength comparison for joint placement and routing problem on ISPD-2005 benchmark. Note time cost be regarded as the same for inference stage as the difference of the two compared methods are only for their training protocol while the model structure is the same.
Method
adaptec1
adaptec2
adaptec3
adaptco4
bighlue
bighlue?
bigblue3
highlue4
DeepPR (ours)
5298
22256
32839
63560
8602
16819
94083
17585
Separate Pipeline
6077
23725
37029
79573
9078
17849
98480
17829
time (s)
10.63
10.66
40.96
98.23
16.12
32.71
77.60
18.73
4.5 Ablation Study
To better verify the effectiveness of our multi-view policy network that introduces both CNN and GNN and the random network distillation module reward function, we conduct ablation studies on the pretraining task using modified environment (circuit) adaptecl. In Fig. 7 we compare

---

50
Mein epiodicc ratum
01
we RND and GNN
-100
-150
-200
50
100
150
200
250
300
Timestops
Figure 7: Mean episodic return comparison: the baseline global embedding from CNN, combined embedding from both CNN and GNN, and our agent with additional intrinsic reward from RND. Our agent significantly outperforms baselines circuit adaptecl (so for other circuits).
proposed agent with the agents without RND module as well as detailed node emhedding from GNN. is not surprising that the baseline method only shows modest improvement over the initial random policy, due to the sparse reward signal and immense possible placements. Note that the performance of using CNN GNN alone to obtain the embedding similar, hence we present one of them in the figure. Combination of global embedding and detailed node embedding makes it possible to learn rich representations of the state space, which achieves better performance than baseline structure. Nevertheless, our placer with RND to provide intrinsic reward for exploration in each step surpassed the baseline by large margin, which consistent with our discussion in Sec, 3.3.2
Conclusion and Outlook
We have presented an end-to-end learning paradigm for placement and routing, which are two of the most challenging and time-consuming tasks in chip design flow. For macro placement, we adopt the reinforeement learning while for the more tedious standard cell placement, we resort to the gradient based optimization technique which can more cost-effective, and successfully incorporate it into the end-to-end pipeline. One step further, we develop reinforcement learning for joint solving placement and routing, which has not been studied before. In particular, two-view based embedding model is developed to fuse both global and local information and distillation is devised to improve exploration. Experimental results on public benchmark show the effectiveness of our solver.
For future work, we plan to explore learning the placement of macros and standard cells together with routing for macros and standard cells in an end-to-end paradigm, since these two tasks strongly coupled in the chip design process. However, no existing work has fulfilled this goal to our knowledge, due to its large scale and complexity.
Our model currently is limited to a moderate number of macros for placement, due to the scalability issue of GNN and the large action space for reinforcement learning. Another improvement for future work is to take the size and shape macros into consideration to further reduce overlap between huge macros, as well as combine human heuristics and rules into the learning paradigm.
Limitation.
Potential negative socictal impacts. Our method facilitates the automated placement and routing, while in the EDA industry there are many staff working on chip design and our software may put some of them in a situation for being unemployed.
Acknowledgments and Disclosure of Funding
This work was partly supported by National Key Research and Development Program of China (2000AAA0107600), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102),, and Huawei Inc.
10

---

References
[1] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y.J. Lee, E. Johnson, O. Pathak, A. Nazi et al., "A graph placement methodology for fast chip design," Nature, vol. 594, no. 7862, pp. 207-212, 2021.
[2] Y. Lin, Z. Jiang, J. Gu, W. Li, Dhar, H. Ren, B. Khailany, and D. Z. Pan, "Dreamplace: Deep learning toolkit-enabled gpu acceleration for modern vlsi placement," IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems, 2020.
[3] T. Fösel, M. Y. Niu, F. Marquardt, and L. Li, "Quantum circuit optimization with deep rein- forcement learning," arXiv preprint arXiv:2103.07585, 2021.
[4] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, "Exploration by random network distillation," arXiv preprint urXiv::8100 12894, 2018.
[5] M. A. Breuer, "A class of min-cut placement algorithms." in Proceedings of the 14th Design Automation Conference, 1977, pp. 284-290.
[6] C. M. Fiduccia and R. M. Mattheyses, "A linear-time heuristic for improving network partitions,"
in 19th design automation conference. IEEE, 1982, pp. 175-181.
[7] A. R. Agnihotri, S. Ono, and P. Madden, "Recursive bisection placement: Feng shui 5.0 implementation details," in Proceedings of the 2005 international symposium on Physical design, 2005, pp. 230-232.
[8] Kirkpatrick,
Gelatt, and M. P. Vecchi, "Optimization by simulated annealing," science, vol. 220, no. 4598, pp. 671-680, 1983.
[9] S.-Y. Ho, S.-J. Ho, Y.-K. Lin, and W.-C Chu, "An orthogonal simulated annealing algorithm for large floorplanning problems," IEEE transactions on very large scale integration (VLSI) systems, vol. 12, no. 8, pp. 874-877. 200,
[10] M. Shunmugathammal, C. C. Columbus, and S. Anand, "A novel b* tree crossover-based simulated annealing algorithm for combinatorial optimization in vlsi fixed-outline floorplans," Circuits, Systems, and Signal Processing, vol. 39, no. 2, pp. 900-918, 2020.
[11] Spindler, U. Schlichtmann, and F. M. Johannes, "Kraftwerk2- fast force-directed quadratic placement approach using an accurate net model," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 27, no. pp. 1398-1411, 2008
[12] T.-C. Chen, Z.-W. Jiang, T.-C. Hsu, H.-C. Chen, and Y.-W, Chang, "Ntuplace3: An analytical placer for large-scale mixed-size designs with preplaced blocks and density constraints," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 27, no. 7, pp. 1228 1240, 2008.
[13] A. B. Kahng and Q. Wang, "Implementation and extensibility of an analytic placer," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 24, no. 5, pp. 734-747, 2005.
[14] Lu, P. Chen, C.-C. Chang, L. Sha, D. J. Huang, C.-C. Teng, and C.-K. Cheng, "eplace: Electrostatics-bute placement using fast fourier transform and nesterov's method," ACM Transactions on Design Automation of Electronic Systems (TODAES), vol. 20, no. 2, pp. 1-34, 2015.
[15] C.-K. Cheng, A. B. Kahng, I. Kang, and L. Wang, "Replace: Advancing solution quality and routability validation in global placement," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 38, no. 9, pp. 1717-1730, 2018.
[16] D. Vashisht, H. Rampal, H. Liao, Y. Lu, D. Shanbhag, E. Fallon, and L. B. Kara, "Placement in integrated circuits using cyclic reinforcement learning and simulated annealing," arXiv preprint arXiv:2011 1.07577. 2020.
[17] Hu and S. S. Sapatnekar, "A survey on multi-net global routing for integrated circuits." Integration, vol. 31, no. 1, pp. 1-49, 2001.
[18] J.-M. Ho, G. Vijayan, and C.-K. Wong, "New algorithms for the rectilinear steiner tree problem," IEEE transactions on computer-aidel design of integrated circuits and systems, vol. 9, no. pp. 185-193, 1990.
11

---

[19] C. Albrecht, "Global routing by new approximation algorithms for multicommodity flow," JEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 20, no. 5, pp. 622-632, 2001.
[20] M. Cho, K. Lu, K. Yuan, and D. Z. Pan, "Boxrouter 2.0: Architecture and implementation of a
hybrid and robust global router," in 2007 IEEF/ACM International Conference on Computer- Aided Design. IEEE, 2007, pp. 503-508.
[21] Mo, A. Tabbara, and R. K. Brayton, "A force-directed maze router," in IEEE/ACM Interna- tional Conference Computer Aided Design. ICCAD 2001. IEEE/ACM Digest of Technical Papers (Cat. No. 01CH37281). IEEE, 2001, pp- 404-007.
[22] R. Liang, H. Xiang, D. Pandey, L.. Reddy, S. Ramji, G.-J. Nam, and J. Hu, "Dre hotspot prediction at sub- 10nm process nodes using customized convolutional network," in Proceedings of the 2020 International Symposium on Physical Design, 2020, pp. 135-142.
[23] W.-T. Chan, Y. Du, A. B. Kahng. S. Nath, and K. Samadi, "Beol stack-aware routability prediction from placement using data mining techniques." in 2016 IEEE 34th International Conference on Computer Design (ICCD). IEEE, 2016, pp. 41-48.
[24] W.-K. Cheng, Y.-Y. Guo, and C.-S. Wu, "Evaluation of routability- driven macro placement with machine-learning technique," 2018 7th International Symposium on Next Generation Electronics (ISNE). IEEE, 2018, pp. 1-3.
[25] H. Liao, W. Zhang, X. Dong, B. Poczos, K. Shimada, and L. Burak Kara, "A deep reinforcement learning approach for global routing," Journal of Mechanical Design, vol. 142, no. 6, 2020.
[26] H. Liao, Q. Dong, X. Dong, W. Zhang, W. Zhang, W. Qi, E. Fallon, and L. B. Kara, "Attention routing: mask-assignment detailed routing using attention- based reinforcement learning," in ASME 2020 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers Digital Collection, 2020.
[27] H. Ren and M. Fojtik, "Standard cell routing with reinforcement learning and genetic algorithm in advanced technology nodes," in Proceedings of the 26th Asia and South Pacific Design Automation Conference, 2021, pp. 684-689.
[28] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal policy optimization algorithms," arXiv preprint arXiv: trXXiv:707.060 707.06637, 2017.
[29] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, Baker, M. Lai, A. Bolton et al., "Mastering the game of go without human knowledge," nature, vol. 550, no. 7676, pp. 354-359, 2017.
[30] Spindler and F. M. Johannes, "Fast and accurate routing demand estimation for efficient routability-driven placement," in 2007 Design, Automation & Test in Europe Conference & Exhibition. IEEE, 2007, pp- 1-6.
[31] G.-J. Nam, C. J. Alpert, P. Villarrubia, B. Winter, and M. Yildiz, "The ispd2005 placement contest and benchmark suite," in Proceedings of the 2005 international symposium on Physical
design, 2005, pp. 216-220.
[32] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, segghpeeformance deep learning library," arXiv preprint 19/2.01 2019.
[33] T. N. Kipf and M. Welling, "Spmi-superrised classification with graph convolutional networks." arXiv preprint arXiv: 600.002907 2016.
[34] D. P. Kingma and Ba. "Adam: A method for stochastic optimization." arXiv preprint 1rX.v.1412.6988, 2014.
12


Reviews:
[{"review_id": "viV-tx6nq9W", "reviewer": "Reviewer_5u46", "summary": "This paper proposes a joint learning method to solve the placement and routing problems together. Such problems used to be studied separately.  For the placement, this paper is also solving the placement of macro and standard cells together. Moreover, both CNN and GNN are adopted to provide different embedding model.  Finally, the experimental shows that the proposed jointly method has notable performance margin than traditional separate pipeline. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\tThis paper is targeting an important challenge in IC design. The idea that jointly solving placement and routing is promising solution since they are strongly coupled. And the placement’s quality will severely impact the routing performance. \n\tThe related work section is exhaustive which gives the reader a good understanding of the existing works about placement and routing. Not only the traditional methods but also the recently learning-based methods.\n\tIn the Methods part, the author clearly described the proposed method including the problem formulation, reward design, how to calculate wirelength and congestion,  how to utilize both CNN and GNN, and etc.\n\nCons:\n\tThe major issues are lack of details and insufficient experiment.\n\tIn the experiment, only 2 circuits are used to test the proposed method. The author shows the plot or table but failed to explain the reason behind the result. For example, the author mentioned pre-trained for 300 iterations achieves the best result. But why it achieves best and why only it has a big jump around 100K timesteps?  It is better if the author can re-organize the result, present them more clearly and analyze them in detail not only just show them.\n\n\tMinor issues:\n\tSome abbreviations are not explained such as DRL(Line 37, Page 2), SA(Line 66, Page 3). Although reader can guess their meaning, but it may cause unnecessary mis-leading. \n\n\tThe method section should be explained more than it is. Every sub-section only have one paragraph for how it is design. It is better if the author can explain why you design it like this? and how it works in the whole system?. And this will also help reader to follow your main story. \n\n\n---raised my score to 6 after authors rebuttal", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "did not see any negative societal impact", "review_text": "", "extra_scores": {"ethical_concerns": "did not notice any ethical concern", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "viOUlda16q1", "reviewer": "Reviewer_WfTv", "summary": "The paper introduces a joint placement and routing system based on deep reinforcement learning. They use two RL agents to do placement and routing and train them using a single reward function. This design and the reward structure enables them to reach closer to the global optimum. The results show that the learned policies are effective and beats the baselines they consider.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n* Joint optimisation of placement and routing. This is enabled through their careful design of the reward function that guides the optimization policies to the true target. More specifically, their use of wire length (HPWL) enables the authors to make placement decisions which are cognizant of routing output.\n* Use of both a CNN and GNN to model different aspects of the input. The netlist intrinsically has a graph structure, where as the die has a 2D structure. Their use of two different NN topologies to exploit these two different structures is commendable.\n* Usage of reward distillation to enable more exploration and to somewhat mitigate the sparse reward problem: authors correctly identifies that the unavailability of intermediate rewards makes the convergence of their policies slower and hence introduces random distillation to mitigate the problem somewhat.\n* Adoption of pre-training (tasks separately) and then fine-tuning \n\n# Weaknesses and questions for authors\n\n* Some of the terminologies of the paper is not that approachable by a non-domain expert. For instance, what is a macro and a standard cell. To improve the clarity of the paper, the authors should have a relatively small background section to ground these keywords that are used over and over again in the text.\n* Evaluation\n  * Authors mention about ablation studies to validate the usefulness of a multi-view policy network. However, I could not find some of those results in the paper. Specifically, I would like to know how much does having both a GNN and CNN help in the final performance of the policy?\n* The baselines are not explained properly. In order for a non-domain expert to appreciate the results, the baselines should be explained better. For instance, how is sequential learning placer different from DREAMPlace. A table with differences would be helpful.\n* I am not familiar with the scale of the two benchmarks. Does ISPD-2005 have only 2 benchmarks?  Are these problems of the same scale as usual place and route problems encountered in normal chip design?\n* Table 2 and 3 wire length statistics are of different magnitude orders. Is it because in Table 3 you only consider macros? Does that mean standard cell placement is more important than macros since it introduces more wires? Please explain.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors mention unemployment of engineers as a negative societal impact. I do not see any other major negative societal impact from this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "dD1w2kfRuix", "reviewer": "Reviewer_NVyt", "summary": "This paper proposed a joint learning method for the placement of macros and standard cells, by the integration of reinforcement learning with a gradient based optimization. The paper also proposed to jointly solve placement and routing via reinforcement learning, which has been the first time attempted in literature. The designed policy network incorporates both CNN and GNN to provide a multi-view embedding with a global embedding and a node level embedding. The paper also proposed a random network distillation to encourage exploration. Empirically, the method surpasses the separate placement and routing pipeline by a notable margin.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is clearly, well written. Explanations on placement and routing problems, graphical illustration are good. The paper provides a thoroughly written related work section. \n\n2. The proposed method obtains the full placement solution through end-to-end learning without clustering. Jointly solving placement and routing via reinforcement learning is new. The benefits are pretty clear: first, the placement solutions provides abundant training data for the routing agent, instead of relying on randomly generated data used in previous work. Routing, on the other hand, provides a direct objective for the placement agent to optimize, hence mitigating the reliance on an intermediate cost models and reducing bias in the reward. \n\n3. The placement involves the placement of macros (there are a few of them) and standard cells (there are many of them), which makes the solution more significant than solution [1]. The reviewer applauds the efforts making this possible. \n\nWeaknesses:\n1. The advantage of combing CNN and GNN is small, as indicated in Figure 4. \n\n2. RND improves the performance significantly, but can be replaced with related methods such as entropy based methods or regularization based methods. The paper does not provide ablation study on that. \n\n3. There are not sufficient benchmarks. Only two circuits are provided. There is a lack of details on those circuits. \n\n4. There are not sufficient baselines to compare with. The reviewer understands that it could be a comparatively new domain, and not many related works are available. But the reviewer would be glad to see some comparisons with work [1] by reimplementing [1] in your setting. Also, additional results compared to human placement baseline is highly recommended. \n\n\nDetailed comments:\n1. [1] approach is not a analytic solver or via gradient updates. It is still a reinforcement learning approach, using a transferable value network. \n\n2. There isn't a ablation study demonstrating the necessity of multi-view embedding. Comparisons of methods with and without global or node level information should be provided. \n\n3. How does the intrinsic reward based method via random network distillation compare to a entropy based method such as SAC [2]?\n\n4. Why only two layers of GNN is used? It seems to small for a problem like placement and routing. \n\nOverall, the reviewer likes the paper about its novelty in combing RL and gradient-based methods, jointly optimizing placement of macros and standard cells end to end, and jointly optimizing placement and routing, the proposal of pretraining on a simpler proxy task. However, the downside is its insufficient comparisons with related work. The paper needs some stronger baselines. \n\n\n[1] Chip placement with deep reinforcement learning, https://arxiv.org/abs/2004.10746\n[2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor\n, https://arxiv.org/abs/1801.01290", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both limitations and societal impact are sufficiently addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0dtIuh3Fnkl", "reviewer": "Reviewer_DTgM", "summary": "The paper proposes a deep reinforcement learning method to solve the chip placement and routing problem. They first introduce an RL based method combined with a gradient-based method to approximate the wirelength. They then expand their RL agent to also support routing task as well.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(I couldn't find any supplementary material nor appendix sections.)\n\nOriginality: \n- The method is an extension on recent work on deep RL for placement. It also combines a quick placement method (DREAMPlace) for calculating RL reward. They were also the first to propose an RL agent for combined task of placement and routing.  \n- The paper provides a great literature review on the previous methods.\n\nQuality:\n- The authors provide some experimental results to support the claims, however, more experiments are required. \n- They provided a limited ablation study for their first method (on RND and GNN). \n- More experimental results are required, for example, the results in Figure 4 and 5, have to be an average of multiple runs with different seeds. \n- The authors didn't report any information about the runtime of the method and the impact of transfer learning.  \n- The number of blocks in the evaluation are very limited. They could've used bigblue* blocks in the ISPD2005 as well.  \n- The existing results do not show the scalability of the proposed methods for real production blocks which include more cells. \n- In Figure 4, Is the RND reward included in the y axis? If yes, then the comparison is meaningless, and if no, please explicitly mention what is y-axis.\n\nClarity:\n- The paper is well written and organized. \n- In Figure 1a, the top left image, what is the unnamed block on the right? \n- Define acronyms before using them: SA, DRL.\n- In Section 3.1, Action, It is not clear how macros that are larger than 1x1 are handled? How the grid size n is determined and why is it always square? \n- How RUDY is calculated, since it is not part of DREAMPlace? \n- A newer version of [1] can be find in: Mirhoseini, Azalia, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee et al. \"A graph placement methodology for fast chip design.\" Nature 594, no. 7862 (2021): 207-212.\n \n\nSignificance:\n- The proposed methods are interesting and important for the community. It provides new methods to address the complex problem of placement and routing and will be likely interesting for the researchers and practitioners. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors explained the limitation of the work and it is possible societal impact.\n- It would have been better if they could explain more in depth about the scalability limitation and the runtime of their method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There was a consensus among reviewers that this paper should be accepted. It is the first to propose an RL agent for the combined task of placement and routing in chip design. The main critique after initial reviews were insufficient experiments. However, this was addressed to the reviewers' satisfaction by extensive additional experiments in the rebuttal. Hence, I recommentd acceptance of this paper.
Decision: Accept (Poster)