On Joint Learning for Solving Placement and Routing in Chip Design

1 Introduction . With the breakthrough of semiconductor technology , the scale of integrated circuit ( IC ) has surged exponentially , which challenges the scalability of existing Electronic Design Automation ( EDA ) algorithms and technologies . Placement is one of the most crucial but time-consuming steps of the chip design process . It maps the components of a netlist including macros and standard cells to locations on the chip layout , where standard cells are basic logic cells e.g . logic gates and macros are functional blocks e.g . SRAMs . A good placement leads to better chip area utilization , timing performance and routability . Based on the placement assignment , routing assigns wires to connect the components , which is strongly coupled with placement task . In addition , the placement solution also serves as a rough estimation of wirelength and congestion , which is valuable in guiding the earlier stages of design flow . The objective of placement is to minimize metrics of power , performance , and area ( PPA ) without violating the constraints such as placement density and routing congestion . We provide a pipeline to the placement problem : the input of placement is a netlist represented by hypergraph H = ( V , E ) , where V denotes set of nodes ( cells ) and E denotes set of hyperedges ( nets ) that indicates the connectivity between circuit components . Marco placement firstly determines the locations of macros on the chip canvas , followed by immense numbers of standard cells adjust their position based on adjacent macros and finally obtains the full placement solution , as shown in Fig . 1 ( a ) . The routing problem , however , takes placement solution as input and tries to connect ∗Correspondence author is Junchi Yan . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . those electronic components in a circuit coarsely . Without violating constraints on edges between neighboring tiles , the target of routing is to minimize the total wirelength , as shown in Fig . 1 ( b ) . In this work , we first propose an end-to-end learning approach DeepPlace for placement problem with two stages . The deep reinforcement learning ( DRL ) agent places the macros sequentially , followed by a gradient-based optimization placer to arrange millions of standard cells . In addition , we develop a joint learning approach DeepPR ( i.e . deep place and routing ) for both placement and the subsequent routing task via reinforcement learning . The main contributions of this paper are : 1 ) For learning based placement , we propose an end-to-end approach DeepPlace for both macros and standard cells , whereby the two kinds of components are sequentially arranged by reinforcement learning and neural network formed gradient optimization , respectively . To our best knowledge , this is the first learning approach for the joint placement solving of macro and standard cells . In the previous works , these two tasks are independently solved either for macro [ 1 ] or standard cells [ 2 ] . 2 ) We also propose DeepPR to jointly solve placement and routing via ( reinforcement ) learning , which again to our best knowledge , has not been attempted in literature before . 3 ) To adapt reinforcement learning more effectively into our pipeline , we design a novel policy network that introduces both CNN and GNN to provide two views to the placement input , in contrast to previous works that use CNN [ 3 ] or GNN [ 1 ] alone to obtain the embedding . The hope is that both global embedding and node level embedding information can be synthetically explored . We further adopt the random network distillation [ 4 ] to encourage exploration in reinforcement learning . 4 ) We provide experimental evaluation for both joint macro/standard cell placement and joint placement & routing . Results show that our method surpasses the separate placement and routing pipeline by a notable margin . Code partly public available at : https : //github.com/Thinklab-SJTU/EDA-AI . 2 Related Work . Classical methods for Placement . The history of global placement can trace back to the 1960s [ 5 , 6 ] . A wide variety of methods have been proposed since then , most of which fall into three categories : partitioning-based methods , stochastic/hill-climbing methods , and analytic solvers . In early years , partition-based methods adopted the idea of divide-and-conquer : netlist and chip layout are partitioned recursively until the sublist can be solved by optimal solvers . This hierarchical structure makes them fast to execute and natural to extend for larger netlists , while scarifying the solution quality since each sub-problem is solved independently . Some multi-level partitioning methods [ 7 ] were developed afterwards . Stochastic and hill-climbing , as the second category , are mainly based on annealing methods [ 8 ] which is inspired from annealing in metallurgy that involves heating and controlled cooling for optimal crystalline surfaces . In practice , simulated annealing ( SA ) optimizes a given placement solution by random perturbations with actions such as shifting , swapping and rotation of macros [ 9 , 10 ] . Although SA is flexible and able to find the global optimum , it is time-consuming and hard to deal with the ever-increasing scale of circuit . For the analytic solvers , force-directed methods [ 11 ] and non-linear optimizers [ 12 , 13 ] are commonly adopted . In comparison , the quadratic methods are computational efficient but showing relatively lower performance , while non-linear optimization approximates the cost functions more smoothly with the cost of higher complexity . Recently , however , modern analytical placers such as ePlace [ 14 ] and RePlAce [ 15 ] introduce electrostatics-based global-smooth density cost function and Nesterov ’ s method nonlinear optimizer that achieve superior performance on public benchmarks . They formulate each node of the netlist as positively charged particle . Nodes are adjusted by their repulsive force and the density function corresponds to system ’ s potential energy . These analytical methods update positions of cells in gradient based optimization scheme and generally can handle millions of standard cells by parallelization on multi-threaded CPUs using partitioning to reduce the running time . Nevertheless , all methods mentioned above perform heavy numerical computation for large-scale optimization problem on the CPUs , which lacks exploration of GPU ’ s opportunity . DREAMPlace [ 2 ] is inspired by the idea that the analytical placement problem is analogous to training a neural network . They both involve optimizing parameters and minimizing a cost function . Based on the state-of-the-art analytical placement algorithm RePlAce , DREAMPlace implements hand-optimized key operators by deep learning toolkit PyTorch and achieves over 30× speedup against CPU-based tools . Learning-based methods for Placement . Recently learning-based methods for placement especially reinforcement learning ( RL ) have been proposed to obtain the generalization ability . Google [ 1 ] proposes the first end-to-end learning method for placement of macros that models chip placement as a sequential decision making problem . In each step , the RL agent places one macro and target metrics are used as reward until the last action . GNN is adopted in the value network to encode the netlist information and deconvolution layers in the policy network output the mask of current macro position . Another line is to solve the placement problem by combining RL and heuristics [ 16 ] . They propose a cyclic framework between the RL and SA , where RL module adjusts the relative spatial sequence between circuit components and SA further explores the solution space based on RL initialization . In comparison , the former one is a learning based approach with the same objective function as analytic solvers , while the later is essentially an annealing solver . Classical & Learning-based methods for Routing . Global routing generally begins with decomposing a multiple-pin net problem into a set of two-pin connection problems [ 17 , 18 , 19 ] . After that , each pin-to-pin routing problem is solved by classical heuristic-based router such as rip-up and reroute [ 20 ] , force-directed routing [ 21 ] and region-wise routing . Recently , machine learning techniques have been applied for routing information prediction , including routing congestion [ 22 ] , the routability of a given placement [ 23 ] and circuit performance [ 24 ] . Meanwhile , RL method has also been proposed to handle routing problems . [ 25 ] uses a DQN model to decide actions of the routing direction , i.e . going north , south , etc at each step . [ 26 ] proposes an attention-based REINFORCE algorithm to select routing orders and use pattern router to generate actual route once routing order is determined . While [ 27 ] applies genetic algorithm to create initial routing candidates and uses RL to fix the design rule violations incrementally . Different from current learning-based placer such as [ 1 ] that groups the standard cells into a few thousand clusters and uses force-directed methods for coarse placement solution , our work combines reinforcement learning with a gradient based optimization scheme without clustering to obtain the full placement solution through end-to-end learning scheme . While [ 15 ] considers a local density function as penalty factor to address the requirement for routability , the proposed method , to our best knowledge , is the first attempt to jointly solve placement and routing via reinforcement learning . 3 Methodology . 3.1 Problem Formulation . First of all , we target the problem of macro placement , whose objective is to determine locations of macros on the chip canvas with no overlap and wirelength minimized . Our RL agent sequentially maps the macros to valid positions on the layout . Once all macros have been placed , we either fix their positions and adopt gradient-based placement optimization to obtain a complete placement solution with corresponding evaluation metrics such as wirelength and congestion , as shown in Fig . 2 ( a ) . Or alternatively , we adopt classical router or another RL agent to route the placement solution and regard the exact total wirelength as rewards for both placement and routing task , which is shown in Fig . 2 ( b ) . The key elements of the Markov Decision Processes ( MDPs ) are defined as follows : • State st : the state representation consists of two parts , global image I portrayed the layout and netlist graph H which contains detailed position of all macros that have been placed . Note that placement problem is analogous to a board game ( i.e. , chess or GO ) , both of which are required to determine the position of pieces ( macros ) . Therefore , we model the chess board ( chip canvas ) as a binary image I where 1 denotes position that has been occupied . In addition , the rule of board game is similar to netlist graph H , which is complementary to the global image . • Action at : the action space contains available positions in the n× n canvas at time t , where n denotes the size of grid . Once a spare position ( x , y ) is selected by the current macro , we set Ixy = 1 and remove this position from the available list . • Reward rt : the reward at the end of episode is a negative weighted sum of wirelength and routing congestion from the final solution . The weight is a trade-off between main objective wirelength and routing congestion which indicates the routability for routing task . Different from other deep RL placers that set the reward to 0 for all previous actions , we adopt random network distillation ( RND ) inspired from [ 4 ] to calculate intrinsic rewards at each time step . The policy network learns to maximize the expected reward from placing prior chips and improves placement quality over episodes , which is updated by Proximal Policy Optimization ( PPO ) [ 28 ] .

Summary: The paper proposes a deep reinforcement learning method to solve the chip placement and routing problem. They first introduce an RL based method combined with a gradient-based method to approximate the wirelength. They then expand their RL agent to also support routing task as well.