On Joint Learning for Solving Placement and Routing in Chip Design

1 Introduction . With the breakthrough of semiconductor technology , the scale of integrated circuit ( IC ) has surged exponentially , which challenges the scalability of existing Electronic Design Automation ( EDA ) algorithms and technologies . Placement is one of the most crucial but time-consuming steps of the chip design process . It maps the components of a netlist including macros and standard cells to locations on the chip layout , where standard cells are basic logic cells e.g . logic gates and macros are functional blocks e.g . SRAMs . A good placement leads to better chip area utilization , timing performance and routability . Based on the placement assignment , routing assigns wires to connect the components , which is strongly coupled with placement task . In addition , the placement solution also serves as a rough estimation of wirelength and congestion , which is valuable in guiding the earlier stages of design flow . The objective of placement is to minimize metrics of power , performance , and area ( PPA ) without violating the constraints such as placement density and routing congestion . We provide a pipeline to the placement problem : the input of placement is a netlist represented by hypergraph H = ( V , E ) , where V denotes set of nodes ( cells ) and E denotes set of hyperedges ( nets ) that indicates the connectivity between circuit components . Marco placement firstly determines the locations of macros on the chip canvas , followed by immense numbers of standard cells adjust their position based on adjacent macros and finally obtains the full placement solution , as shown in Fig . 1 ( a ) . The routing problem , however , takes placement solution as input and tries to connect ∗Correspondence author is Junchi Yan . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . those electronic components in a circuit coarsely . Without violating constraints on edges between neighboring tiles , the target of routing is to minimize the total wirelength , as shown in Fig . 1 ( b ) . In this work , we first propose an end-to-end learning approach DeepPlace for placement problem with two stages . The deep reinforcement learning ( DRL ) agent places the macros sequentially , followed by a gradient-based optimization placer to arrange millions of standard cells . In addition , we develop a joint learning approach DeepPR ( i.e . deep place and routing ) for both placement and the subsequent routing task via reinforcement learning . The main contributions of this paper are : 1 ) For learning based placement , we propose an end-to-end approach DeepPlace for both macros and standard cells , whereby the two kinds of components are sequentially arranged by reinforcement learning and neural network formed gradient optimization , respectively . To our best knowledge , this is the first learning approach for the joint placement solving of macro and standard cells . In the previous works , these two tasks are independently solved either for macro [ 1 ] or standard cells [ 2 ] . 2 ) We also propose DeepPR to jointly solve placement and routing via ( reinforcement ) learning , which again to our best knowledge , has not been attempted in literature before . 3 ) To adapt reinforcement learning more effectively into our pipeline , we design a novel policy network that introduces both CNN and GNN to provide two views to the placement input , in contrast to previous works that use CNN [ 3 ] or GNN [ 1 ] alone to obtain the embedding . The hope is that both global embedding and node level embedding information can be synthetically explored . We further adopt the random network distillation [ 4 ] to encourage exploration in reinforcement learning . 4 ) We provide experimental evaluation for both joint macro/standard cell placement and joint placement & routing . Results show that our method surpasses the separate placement and routing pipeline by a notable margin . Code partly public available at : https : //github.com/Thinklab-SJTU/EDA-AI . 2 Related Work . Classical methods for Placement . The history of global placement can trace back to the 1960s [ 5 , 6 ] . A wide variety of methods have been proposed since then , most of which fall into three categories : partitioning-based methods , stochastic/hill-climbing methods , and analytic solvers . In early years , partition-based methods adopted the idea of divide-and-conquer : netlist and chip layout are partitioned recursively until the sublist can be solved by optimal solvers . This hierarchical structure makes them fast to execute and natural to extend for larger netlists , while scarifying the solution quality since each sub-problem is solved independently . Some multi-level partitioning methods [ 7 ] were developed afterwards . Stochastic and hill-climbing , as the second category , are mainly based on annealing methods [ 8 ] which is inspired from annealing in metallurgy that involves heating and controlled cooling for optimal crystalline surfaces . In practice , simulated annealing ( SA ) optimizes a given placement solution by random perturbations with actions such as shifting , swapping and rotation of macros [ 9 , 10 ] . Although SA is flexible and able to find the global optimum , it is time-consuming and hard to deal with the ever-increasing scale of circuit . For the analytic solvers , force-directed methods [ 11 ] and non-linear optimizers [ 12 , 13 ] are commonly adopted . In comparison , the quadratic methods are computational efficient but showing relatively lower performance , while non-linear optimization approximates the cost functions more smoothly with the cost of higher complexity . Recently , however , modern analytical placers such as ePlace [ 14 ] and RePlAce [ 15 ] introduce electrostatics-based global-smooth density cost function and Nesterov ’ s method nonlinear optimizer that achieve superior performance on public benchmarks . They formulate each node of the netlist as positively charged particle . Nodes are adjusted by their repulsive force and the density function corresponds to system ’ s potential energy . These analytical methods update positions of cells in gradient based optimization scheme and generally can handle millions of standard cells by parallelization on multi-threaded CPUs using partitioning to reduce the running time . Nevertheless , all methods mentioned above perform heavy numerical computation for large-scale optimization problem on the CPUs , which lacks exploration of GPU ’ s opportunity . DREAMPlace [ 2 ] is inspired by the idea that the analytical placement problem is analogous to training a neural network . They both involve optimizing parameters and minimizing a cost function . Based on the state-of-the-art analytical placement algorithm RePlAce , DREAMPlace implements hand-optimized key operators by deep learning toolkit PyTorch and achieves over 30× speedup against CPU-based tools . Learning-based methods for Placement . Recently learning-based methods for placement especially reinforcement learning ( RL ) have been proposed to obtain the generalization ability . Google [ 1 ] proposes the first end-to-end learning method for placement of macros that models chip placement as a sequential decision making problem . In each step , the RL agent places one macro and target metrics are used as reward until the last action . GNN is adopted in the value network to encode the netlist information and deconvolution layers in the policy network output the mask of current macro position . Another line is to solve the placement problem by combining RL and heuristics [ 16 ] . They propose a cyclic framework between the RL and SA , where RL module adjusts the relative spatial sequence between circuit components and SA further explores the solution space based on RL initialization . In comparison , the former one is a learning based approach with the same objective function as analytic solvers , while the later is essentially an annealing solver . Classical & Learning-based methods for Routing . Global routing generally begins with decomposing a multiple-pin net problem into a set of two-pin connection problems [ 17 , 18 , 19 ] . After that , each pin-to-pin routing problem is solved by classical heuristic-based router such as rip-up and reroute [ 20 ] , force-directed routing [ 21 ] and region-wise routing . Recently , machine learning techniques have been applied for routing information prediction , including routing congestion [ 22 ] , the routability of a given placement [ 23 ] and circuit performance [ 24 ] . Meanwhile , RL method has also been proposed to handle routing problems . [ 25 ] uses a DQN model to decide actions of the routing direction , i.e . going north , south , etc at each step . [ 26 ] proposes an attention-based REINFORCE algorithm to select routing orders and use pattern router to generate actual route once routing order is determined . While [ 27 ] applies genetic algorithm to create initial routing candidates and uses RL to fix the design rule violations incrementally . Different from current learning-based placer such as [ 1 ] that groups the standard cells into a few thousand clusters and uses force-directed methods for coarse placement solution , our work combines reinforcement learning with a gradient based optimization scheme without clustering to obtain the full placement solution through end-to-end learning scheme . While [ 15 ] considers a local density function as penalty factor to address the requirement for routability , the proposed method , to our best knowledge , is the first attempt to jointly solve placement and routing via reinforcement learning . 3 Methodology . 3.1 Problem Formulation . First of all , we target the problem of macro placement , whose objective is to determine locations of macros on the chip canvas with no overlap and wirelength minimized . Our RL agent sequentially maps the macros to valid positions on the layout . Once all macros have been placed , we either fix their positions and adopt gradient-based placement optimization to obtain a complete placement solution with corresponding evaluation metrics such as wirelength and congestion , as shown in Fig . 2 ( a ) . Or alternatively , we adopt classical router or another RL agent to route the placement solution and regard the exact total wirelength as rewards for both placement and routing task , which is shown in Fig . 2 ( b ) . The key elements of the Markov Decision Processes ( MDPs ) are defined as follows : • State st : the state representation consists of two parts , global image I portrayed the layout and netlist graph H which contains detailed position of all macros that have been placed . Note that placement problem is analogous to a board game ( i.e. , chess or GO ) , both of which are required to determine the position of pieces ( macros ) . Therefore , we model the chess board ( chip canvas ) as a binary image I where 1 denotes position that has been occupied . In addition , the rule of board game is similar to netlist graph H , which is complementary to the global image . • Action at : the action space contains available positions in the n× n canvas at time t , where n denotes the size of grid . Once a spare position ( x , y ) is selected by the current macro , we set Ixy = 1 and remove this position from the available list . • Reward rt : the reward at the end of episode is a negative weighted sum of wirelength and routing congestion from the final solution . The weight is a trade-off between main objective wirelength and routing congestion which indicates the routability for routing task . Different from other deep RL placers that set the reward to 0 for all previous actions , we adopt random network distillation ( RND ) inspired from [ 4 ] to calculate intrinsic rewards at each time step . The policy network learns to maximize the expected reward from placing prior chips and improves placement quality over episodes , which is updated by Proximal Policy Optimization ( PPO ) [ 28 ] .

Summary: The paper introduces a joint placement and routing system based on deep reinforcement learning. They use two RL agents to do placement and routing and train them using a single reward function. This design and the reward structure enables them to reach closer to the global optimum. The results show that the learned policies are effective and beats the baselines they consider.