Reinforcement Learning Policy as Macro Regulator Rather than Macro Placer

Authors: ke  xue
ruo-tong  chen
xi  lin
yunqi  shi
shixiong  kai
siyuan  xu
chao  qian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:55:47

Abstract: In modern chip design, placement aims at placing millions of circuit modules, which is an essential step that significantly influences power, performance, and area (PPA) metrics. Recently, reinforcement learning (RL) has emerged as a promising technique for improving placement quality, especially macro placement. However, current RL-based placement methods suffer from long training times, low generalization ability, and inability to guarantee PPA results. A key issue lies in the problem formulation, i.e., using RL to place from scratch, which results in limits useful information and inaccurate rewards during the training process. In this work, we propose an approach that utilizes RL for the refinement stage, which allows the RL policy to learn how to adjust existing placement layouts, thereby receiving sufficient information for the policy to act and obtain relatively dense and precise rewards. Additionally, we introduce the concept of regularity during training, which is considered an important metric in the chip design industry but is often overlooked in current RL placement methods. We evaluate our approach on the ISPD 2005 and ICCAD 2015 benchmark, comparing the global half-perimeter wirelength and regularity of our proposed method against several competitive approaches. Besides, we test the PPA performance using commercial software, showing that RL as a regulator can achieve significant PPA improvements. Our RL regulator can fine-tune placements from any method and enhance their quality. Our work opens up new possibilities for the application of RL in placement, providing a more effective and efficient approach to optimizing chip design. Our code is available at \url{https://github.com/lamda-bbo/macro-regulator}.

Main paper: 
 introduction: 
 In the complex and evolving landscape of modern chip design, placement is a pivotal process that significantly influences the power, performance, and area (PPA) metrics of the final chip [21, 22] . A modern chip typically comprises thousands of macros (i.e., individual building blocks such as memories) and millions of standard cells (i.e., smaller basic components like logic gates). The macro placement result provides a fundamental solution for the subsequent processes (e.g., standard cells placement and routing), thus playing an important role [32] . For example, macro placement influences the placement of standard cells, and poor macro placement might make it challenging to place these cells optimally, leading to an unsatisfactory chip performance [33] . Moreover, an inappropriate macro placement can result in macro blockage in the core center, which harms the overall chip performance by causing unwanted effects such as routing congestion, inferior wirelength, and timing performance issues [26] . between MaskRegulate, DREAMPlace [19] , AutoDMP [1] , WireMask-EA [29] , and MaskPlace [16] , where lower values indicate better performance. These results are obtained using Cadence Innovus.Due to the lengthy and complex workflow of chip design, designers often rely on proxy metrics that can reflect the final results to guide the optimization process [2, 30, 20] . One important proxy metric is half-perimeter wirelength (HPWL), which provides an approximation for the routing wirelength and is widely used to measure the placement quality [2, 13, 28] . Traditional macro placement methods can be divided into two categories. Earlier approaches usually solve macro placement by black-box optimization (BBO) [24, 5, 12, 29] . They often suffer from the poor scalability due to the large-scale search space and high complexity of decoding a solution to a placement. Another type is analytical method [6, 7, 19] , which can solve the placement efficiently by approximating HPWL gradients. However, these methods are hard to guarantee the non-overlapping constrain between cells and are easy to be stuck in local optima [16, 34] .Reinforcement learning (RL) [31] has recently emerged as a promising technique to enhance the macro placement quality [23, 9, 8, 16, 15] . RL's ability to learn policies through interaction with a complex environment offers a novel pathway for addressing various challenges of macro placement. However, the application of RL is currently hindered by several limitations, including the long training time, an inability to guarantee PPA improvements, and the lack of generalization across different chip layouts. In this work, we highlight that a major contributing factor to these issues is the problem formulation, i.e., the conventional RL approach of placing macros from scratch often results in limited state information and inaccurate reward signal throughout the learning process.To address these challenges, we propose a novel RL approach called MaskRegulate that shifts the focus from initial placement to refining existing placement layouts. The RL policy acts as a regulator rather than a placer, which operates on pre-existing placements, thus allowing for access to comprehensive state information and enabling the acquisition of more precise rewards. This adjustment enhances the efficiency of the learning process and finally improves the final placement results. Furthermore, MaskRegulate introduces the concept of regularity [26] as a part of input information and a critical reward signal, which has been largely overlooked in previous research despite its significance in ensuring manufacturability and performance. Previous methods often only consider the HPWL metric, suffering from optimizing different metrics effectively. By integrating regularity into the RL framework, our approach aligns more closely with advanced chip requirements.The effectiveness of the proposed MaskRegulate is comprehensively evaluated on the ICCAD 2015 benchmark [14] , which is is currently one of the largest open-source benchmarks that allows us to evaluate PPA metrics such as congestion and timing slack. We first compare the global HPWL and regularity of our approach against several competitive methods. Additionally, we use the commercial electronic design automation (EDA) tool Cadence Innovus to evaluate the PPA performance, demonstrating that our proposed MaskRegulate can lead to significant PPA improvements, e.g., the placement layouts and two PPA metrics on superblue1, as shown in Figure 1 . Specifically, compared to MaskPlace (an advanced RL placer [16] ; MaskRegulate shares a similar architecture to it), MaskRegulate improves 17.08% on routing wirelength, 73.08% and 38.81 % on routed horizontal and vertical congestion overflow respectively, 18.35% on worst negative slack, 37.89% on total negative slack, and 46.17% on the number of violation points.This work provides a more effective approach for macro placement of modern chips, opening new possibilities for the application of RL in chip design. The contributions of this work are highlighted in three key points:• Novel problem formulation: Innovatively applying RL in the refinement stage of macro placement, which allows for more effective learning from structured state and accurate reward information, significantly enhancing the learning efficiency and effectiveness.• Integration of regularity: Introducing regularity, a critical yet previously overlooked metric in chip design, into the RL training framework, which not only aligns with industry practice but also enhances the chip PPA quality.• Impressive PPA improvement and comprehensive analysis: On the popular ICCAD 2015 benchmark, our proposed MaskRegulate demonstrates significant improvements in PPA metrics, showing the practical applicability and effectiveness of the RL regulator.2 Background 
placement: 
 The circuit in the placement stage is considered as a graph where vertices model gates. The main input information is the netlist N = (V, E), where V denotes the information (i.e., height and width) about all macros designated for placement on the chip, and E is a hyper-graph comprised of nets e i ∈ E, which encompasses multiple cells (including both macros and standard cells) and denotes their inter-connectivity in the routing stage. Given a netlist, a fixed canvas layout and a standard cell library, a placement method is expected to determine the appropriate physical locations of movable macros such that the total wirelength can be minimized. A macro placement solution s = {(x 1 , y 1 ), . . . , (x k , y k )} consists of the positions of all the macros {v i } k i=1 , where k denotes the total number of macros. One popular objective of macro placement is to minimize the total HPWL of all the nets while satisfying the cell density constraint, which is formulated as,EQUATIONwhere D denotes the density, ϵ is a threshold, and HP W L e is the HPWL of net e, which is defined as: HP W L e (s) = (max vi∈e x i -min vi∈e x i ) + (max vi∈e y i -min vi∈e y i ).There are three mainstream placement methods, i.e., analytical methods, black-box optimization methods, and learning-based methods. Analytical methods [4] place macros and standard cells simultaneously, which can be roughly categorized into quadratic placement and nonlinear placement.Quadratic placement [11, 18] iterates between an unconstrained quadratic programming phase to minimize wirelength and a heuristic spreading phase to remove overlaps. Nonlinear placement [6, 20, 7] formulates a nonlinear optimization problem and tries to directly solve it with gradient descent methods. Generally speaking, nonlinear placement can achieve better solution quality, while quadratic placement is more efficient. Recently, there has been extensive attention on GPU-accelerated nonlinear placement methods. For example, DREAMPlace [19, 17] transforms the non-linear placement problem in Eq. ( 1) into a neural network training problem, solves it by classical gradient descent and leverages GPU, enabling ultra-high parallelism and acceleration and producing state-of-the-art analytical placement quality.Black-box optimization methods for placement have a long history. Earlier methods such as SP [24] and B * -tree [5] have poor scalability due to the rectangular packing formulation. Recently, some blackbox optimization methods have made significant progress by changing the search space. AutoDMP [1] improves DREAMPlace by using Bayesian optimization to explore the configuration space and shows remarkable performance on multiple benchmarks. WireMask-BBO [29] adopts a wire-mask-guided greedy genotype-phenotype mapping and can be equipped with any BBO algorithm, demonstrating the superior performance over other types of methods.Researchers recently leverage RL-based methods for better placement quality to meet the demands of modern chip design. GraphPlace [23] first models macro placement as a RL problem. It divides the chip canvas into discrete grids, with each macro assigned discrete coordinates of grids, wherein the agent decides the placement of the current macro at each step. However, no reward is given until all the macros are placed, making the reward sparse and hard to learn. DeepPR [9] and PRNet [8] incorporate macro placement, standard cells placement, and routing to achieve better performance than GraphPlace, but may violate the non-overlap constraint. To address this issue, MaskPlace [16] introduces a dense reward and uses a pixel-level visual representation for circuit modules, which can comprehensively capture the configurations of thousands of pins, enabling fast placement in a full action space on a large canvas size. MaskPlace has many attractive benefits that previous methods do not have, e.g., 0% overlap, dense reward, and high training efficiency. ChiPFormer [15] incorporates an offline learning decision transformer and focuses on improving the generalizability of placer. EfficientPlace [10] integrates a global tree search algorithm to guide the optimization process, achieving remarkable placement quality within a short time.However, current RL methods exhibit several shortcomings: 1) Placing from scratch provides insufficient state information and inaccurate reward signals; 2) Most methods focus on minimizing wirelength, which may bring macro blockages and thus harm the final PPA metrics. In this work, we propose a novel RL approach for macro placement: an RL policy acts as a macro regulator rather than a macro placer. Specifically, our learned RL policy is designed to adjust macros based on an existing placement result, rather than placing all macros from scratch. This approach aims to refine and optimize pre-existing layouts, addressing the limitations of traditional RL-based placement methods.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See the last paragraph of the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 We present our proposed MaskRegulate here. Section 3.1 introduces our problem formulation and policy architecture, and Section 3.2 describes how to integrate regularity into the method.Problem formulation of RL regulator. In the Markov Decision Process (MDP) formulation of traditional RL placer, a macro is placed at each step [23, 9, 16, 15] . The placement order of macros is determined based on some pre-defined rules, such as the number of nets, the size of macros, and the number of connected modules that have been placed. An episode ends after all macros have been placed. Typically, the state representation includes information about the chip canvas, the macros that have already been placed, and the macro currently being placed. In GraphPlace [23] , the reward is determined only after all macros have been placed, resulting in a sparse reward signal that complicates the training process. Recent works have introduced various methods to densify the reward signal. For instance, WireMask [16] provides a more continuous reward based on the macros already placed. In contrast to RL placers, our RL regulator focuses on refining an existing placement by adjusting the location of one macro at each step. Unlike the placer, which initiates the placement process from scratch, the regulator benefits from additional information when adjusting each macro. Specifically, the regulator considers not only the macros that have already been placed but also the positions of all other macros. Furthermore, it enhances accuracy by taking into account all macros, even while employing a reward function similar to WireMask. (1,1) , 𝑃𝑃 (1,1) , 𝑃𝑃 (1,1) } 𝑃𝑃 (1,1) 𝑃𝑃 (2,1) 𝑃𝑃 (3,1) 𝑃𝑃 (2,2) 𝑃𝑃 (4,1) 𝑃𝑃 (3,2) Net 2={𝑃𝑃 (2,2) , 𝑃𝑃 (3,2) , 𝑃𝑃 (4,1) placing macros from scratch, as the incomplete state information of placer would be even worse in the case of unseen chips, resulting in poorer performance.} 𝑤𝑤1 𝑤𝑤2 ℎ1 ℎ2 feasible unfeasible 𝑀𝑀 1 𝑀𝑀 2 𝑀𝑀 3 𝑀𝑀 4 Position Mask 7 5 3 2 1 0 0 0 1 3 5 3 1 0 -1 -2 -2 -2 -1 1 3 1 -1 -2 -3 -4 -4 -4 -3 -1 2 0 -2 -3 -4 -5 -5 -5 -4 -2 1 -1 -3 -4 -5 -6 -6 -6 -5 -3 1 -1 -3 -4 -5 -6 -6 -6 -5 -3 2 0 -2 -3 -4 -5 -5 -5 -4 -2 3 1 -1 -2 -3 -4 -4 -4 -3 -1 5 3 1 0 -1 -2 -2 -2 -1 1 7 5 3 2 1 0 0 0 1 3 Wire Mask -4 -3 -2 -1 0 -1 -2 -3 -4 -4 -4 -3 -2 -1 0 -1 -2 -3 -4 -4 -3 -2 -1 0 1 0 -1 -2 -3 -3 -2 -1 0 1 2 1 0 -1 -2 -2 -1 0 1 2 3 2 1 0 -1 -1 0 1 2 3 4 3 2 1 0 0 -1 0 1 2 3 2 1 0 -1 -1 -2 -1 0 1 2 1 0 -1 -2 -2 -3 -2 -1 0 1 0 -1 -2 -3 -3 -4 -3 -2 -1 0 -1 -2 -3 -4 -4 Regularity Mask (a) Chip canvas (b) PositionMask (c) WireMask (d) RegularMaskPolicy architecture. Our policy architecture is illustrated in Figure 2 . The policy divides the chip canvas into several grids and utilizes visual information as inputs, converting chip information into pixel-level image masks. This approach has demonstrated superior efficiency and performance in RL placer policy learning [16, 15] . The inputs include an image of the current canvas, a PositionMask that identifies all valid positions for placing the current macro, a WireMask [16] that indicates the approximate wirelength change for placing the current macro at each valid position, and a RegularMask that indicates the change in regularity for placing the current macro at each valid position (which will be detailed in Section 3.2). An illustration of the PositionMask and WireMask is provided in Figure 3 . To facilitate broader adjustments, the PositionMask has been modified to consider only macros that have already been adjusted; thus, grids occupied by unadjusted macros are available for placement. In our MaskRegulate, the calculation of the WireMask is based on all macros, allowing its value to either increase or decrease. These values are normalized to the range [-1, 1], unlike the [0, 1] normalization used in [16] . Additionally, our framework introduces the RegularMask to quantify changes in regularity within the state and to encourage improvements in regularity through the reward function, as presented in Section 3.2.Why does regularity matters? Macro placement has significant impact on subsequent chip design processes, including standard cell placement and routing. If only focusing on minimizing wirelength (which is the case for most current RL placers), certain macros may end up positioned in the middle of the chip canvas, resulting in macro blockages [26] . This, in turn, leads to the division of available placement areas into separate and disconnected sub-regions. As a consequence, standard cells that are connected by the same net may be scattered across different placement sub-regions, resulting in increased overall wirelength and the potential routing challenges, which ultimately degrade the timing performance. Thus, a well-established practice among experienced engineers in macro placement is to place macros towards the peripheral regions of the chip to prevent macro blockage. In this work, we aim to integrate regularity in the learning-based placement approach to achieve placement preferences similar to those of experienced engineers.RegularMask. Intuitively, macros closer to the edges tend to have lower regularity. Therefore, we propose a simple and effective way to measure regularity. On a canvas, the regularity of a grid located at (x, y) is calculated as min{x, X max -x} + min{y, Y max -y}, where X max and Y max represent the real length of the horizontal and vertical axes, respectively. Given a macro to be placed, the RegularMask measures the value change in regularity for each valid placement position, as illustrated in Figure 3(d) .Reward and policy learning. The reward of MaskRegulate consists of two components: r wire and r reg , which represent the reduction of HPWL and the improvement in regularity, respectively, after refining the current macro. To mitigate the influence of scale differences on training caused by wirelength and regularity, both r wire and r reg are normalized to[0, 1]. The final reward is r = α • r wire + (1 -α) • r reg ,where α is a trade-off coefficient. We will analyze the influence of α in Section 4.4, showing that different α lead to different multi-objective preferences. The detailed information are presented in Appendix A.3. MaskRegulate treats the chip canvas as a grid and divides it into N × N cells, resulting in N 2 possible discrete actions. We use the popular proximal policy optimization (PPO) algorithm [27] to learn the regulator policy.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiment: 
 In this section, we first introduce the basic experimental settings, including the tasks and evaluation metrics in Section 4.1. Then, we try to answer the following three research questions (RQs) in Sections 4.2 to 4.4: 1) How does MaskRegulate perform compared to other methods? 2) How is the generalization ability of MaskRegulate? 3) How do the different parts of MaskRegulate affect the performance? Finally, we provide the visualization of placement results and congestion in Section 4.5.Tasks. We mainly use the ICCAD 2015 benchmark [14] as our test-bed, which includes sufficient advanced chip information and is currently one of the largest open-source benchmarks that allows us to evaluate congestion, timing and other PPA metrics. The benchmark statistics are listed in Table 3 in Appendix A.1. Although ICCAD 2015 is the benchmark we have found that closely reflects the current practices in the EDA industry, it still has some shortcomings. For example, it allows for a large placement area, resulting in loose placement results that do not adhere to the design principles of advanced modern chips. Note that the "A" in PPA denotes "Area", which is a core metric of chip design and should be minimized [3, 32] . Therefore, we scale down the chip's placement area, presenting further challenges for the compared methods. Besides, we also conduct experiments on ISPD 2005 benchmark [25] , which is also a popular benchmark in AI for chip design but does not have sufficient information for PPA evaluation. Detailed results can be found in Appendix B.Proxy evaluation metrics. We use the following two popular proxy metrics for a quick comparison of different algorithms: 1) Global HPWL. After determining the locations of all the macros, we use DREAMPlace [19] to place standard cells to obtain the global placement result, and then report the global HPWL (i.e., full HPWL involving both macros and standard cells). Compared to macro HPWL, global HPWL considers the total wirelength, typically on a scale that is two orders of magnitude larger, providing a better estimation of the final real performance of the chip. 2) Regularity: We compute the regularity values for all macros, which serve as a measurement of the overall regularity of the placement result. We run each algorithm for five times and report their mean and variance. We do not consider the rectangular uniform wire density (RUDY) metric [30] for congestion proxy, as this approximation is sometimes positively correlated with the HPWL metric and is not accurate [29] . Instead, we will evaluate congestion within our PPA evaluation.PPA evaluation metrics. The whole chip design process is lengthy and complex, and proxy metrics may not accurately capture the true performance of the chip. PPA metrics often require the use of commercial EDA tools to obtain precise results with expensive cost. In our experiments, we select the best placement result for PPA evaluation based on global HPWL from multiple runs. After obtaining the global placement results, we use commercial tool Cadence Innovus to proceed the subsequent stages and evaluate their PPA metrics, including routed wirelength, routed vertical and horizontal congestion overflow, worst negative slack, total negative slack, and the number of violation points. These metrics are extremely important measures of chip design and are typically considered to evaluate the quality of a chip comprehensively.4.2 RQ1: How does MaskRegulate perform compared to other methods?We consider the following methods to be compared: DREAMPlace [19] : A state-of-the-art analytical placer; AutoDMP [1] : A method that improves DREAMPlace by exploring its configuration space iteratively; WireMask-EA [29] MaskRegulate achieves the highest regularity on all the eight chips. We can observe a certain correlation between the proxy metric (global HPWL) and the real metric (rWL), but there still exists a gap, indicating the challenges involved in the placement task. Furthermore, we provide detailed visualizations of placement results in Figure 5 , where MaskRegulate shows significant improvements on congestion metrics. Besides, the final placement layouts of MaskRegulate are much regular than all the other methods.The generalization ability of RL policies is an important question to be investigated. In this section, we pre-train MaskRegulate and MaskPlace on the first four chips (i.e., superblue1, superblue3, superblue4, and superblue5) and test on the remaining four chips. To further validate the ability of MaskRegulate to adjust different initial placement results, we use it to adjust the results obtained by different initial placements on the test chips.The results are shown in Table 2 . On both the global HPWL and regularity metrics, MaskRegulate consistently outperforms MaskPlace, showcasing its stronger generalization capability. An interesting finding is that MaskRegulate performs better on unseen chips than on the chips it was trained on, specifically in terms of global HPWL, such as with superblue16. This may suggest that MaskRegulate has learned some general knowledge during the pre-training process, enabling it to overcome local optima that may arise from direct learning on the target chip.We investigate the influence of different parts and provide additional analysis in this section.Hyperparameters sensitivity analysis: different trade-off coefficient α leads to different multiobjective preferences. One hyperparameter of RegularMask is the coefficient α between HPWL reward r wire and regularity reward r reg , where a higher α indicates a preference for optimizing HPWL, and vice versa. In this section, we investigate the influence of the trade-off coefficient α. We train different MaskRegulate regulators with varying α values (ranging from 0.1 to 0.9) and report the proxy and PPA results in Ablation studies. We consider the following ablations of MaskRegulate. We provide the detail visualizations of placement results of all the methods on all the eight chips from ICCAD 2015. As shown in Figure 5 , our proposed MaskRegulate shows significant improvements on congestion metrics. Besides, the placement result of MaskRegulate is much regular than all the other methods.We conduct the following additional results to comprehensively show the effectiveness of our MaskRegulate. 1) To verify whether using a better model structure for the RL placer can compare to the regulator, we add comparison with recent proposed ChiPFormer [15] under a fair setting. 2) To further show the generalization ability of our methods, we conduct generalization experiments on the ISPD 2005 benchmark [25] . 3) To investigate whether MaskRegulate can be used to adjust any initial macro placement solution, we use the pre-trained model to fine-tune other placement results.  
final remarks: 
 Conclusion. In this paper, we present a novel RL problem formulation for macro placement, focusing on the development of a macro regulator rather than a placer. Our proposed method, MaskRegulate, demonstrates substantial improvements in chip placement quality by refining existing layouts instead of generating them from scratch. By integrating dense reward signals and emphasizing regularity, our approach effectively addresses the limitations of traditional RL-based placement methods, resulting in superior performance in PPA metrics across various chips. This advancement paves the way for more efficient and effective chip design through RL.Limitations and future work. This study has several primary limitations: it does not consider the impact of module aspect ratio and area factors on placement; it overlooks global wirelength and timing metrics during the training process; and it does not employ advanced transformer architectures [15] to enhance the generalization of the regulator. Chip design inherently involves different preferences, such as the need for compact size in mobile phone chips and larger sizes for computer chips. Therefore, future research should address these challenges and explore efficient methods to obtain a set of chip placements that accommodate different preferences using multi-objective optimization.The detailed statistics of our benchmarks are listed in Table 3 . Net 1={𝑃𝑃 (1,1) , 𝑃𝑃 (1,1) , 𝑃𝑃 (1,1) } 𝑃𝑃 (1,1) 𝑃𝑃 (2,1) 𝑃𝑃 (3,1) 𝑃𝑃 (2,2) 𝑃𝑃 (4,1) 𝑃𝑃 (3,2) Net 2={𝑃𝑃 (2,2) , 𝑃𝑃 (3,2) , 𝑃𝑃 (4,1) } HPWL (half perimeter wirelength) is an important metric which measures the placement quality before routing. Intuitively, Figure 6 illustrates a 2D chip canvas where M i and P (i,j) denote the i-th module to adjust and its j-th pin, respectively. Solid boxes in green and purple represent the bounding boxes for two distinct nets on the canvas. In concretely, "Net 1"(in green) connects modules M 1 , M 2 and M 3 using wires through pins P (1,1) , P (2,1) and P (3,1) , while "Net 2"(in purple) connects modules M2 , M3 and M4 using wires through pins P (2,2) , P (3,2) and P (4,1) . As shown in Figure 3 (a), HPWL can be computed as h 1 + w 1 + h 2 + w 2 .𝑤𝑤 1 𝑤𝑤 2 ℎ 1 ℎ 2We conclude some important settings of different methods. For the four compared methods (i.e., DREAMPlace 2 , AutoDMP 3 , WireMask-EA 4 , and MaskPlace5 ), we use their original implementations.Purely comparison between placer and regulator. Here, we only change the problem formulation for purely comparing placer and regulator. We implement Vanilla-MaskRegulate, where the only difference to MaskPlace is the problem formulation, and all the other components (e.g., state and reward) are the same. The results in Table 6 clearly demonstrates our motivation, highlighting the advantages of our regulator problem formulation. Training regularity-aware RL placer from scratch. Our proposed RegularMask and regularitybased reward function can also be used to train a RL placer from scratch. We implement MaskPlace+RegularMask and compare it with MaskPlace and MaskRegulate. The results show the advantages of the integration of regularity (between MaskPlace and MaskPlace + RegularMask) and our RL regular formulation (between MaskPlace + RegularMask and MaskRegulate).The above ablation results demonstrate the effectiveness of each component of MaskRegulate. Recently, ChiPFormer [15] incorporates an offline learning decision transformer to improve the generalizability. However, we find that even after fine-tuning for the same number of episodes as MaskRegulate, it is still challenging to achieve satisfactory results on ICCAD 2015. Due to the resource-intensive nature of fine-tuning it, we conducted only a partial set of experiments, which is why they were not included in the main paper. In the future, we plan to explore training a generalized Regulator based on the transformer and pre-train it on some chips from ICCAD 2015, and compare it with ChiPFormer that pre-train on the same training chips. We test the generalization on the ISPD 2005 benchmark [25] by directly using the pre-trained models on superblue 1, 3, 4, and 5 (i.e., the same models in Table 2 ) of MaskPlace and MaskRegulate to place and regulate the eight chips. As shown in Table 10 , MaskRegulate still outperforms MaskPlace in most cases, demonstrating our superior generalization ability and robustness. To investigate whether MaskRegulate can be used to adjust any initial macro placement solution, we conduct additional experiments to demonstrate this capability. We used the pre-trained model on superblue 1, 3, 4, and 5 (i.e., the same models in Tables 2 and 10 ) to adjust different placement results obtained by MaskPlace, AutoDMP, and WireMask-EA. The results are shown in Table 11 .MaskRegulate consistently improves regularity on all four unseen chips and enhances global HPWL on three chips. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The chip is the core productivity of modern society. Our method proposes a more efficient way of using reinforcement learning for macro placement of chips, which has the potential to enhance the quality of chip design.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper has no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper and used dataset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.https://github.com/limbo018/DREAMPlacehttps://github.com/NVlabs/AutoDMPhttps://github.com/lamda-bbo/WireMask-BBOhttps://github.com/laiyao1/maskplace 
 
 Appendix:  


Decision: This work proposes MaskRegulate that leverages PPO to find new chip designs, evaluates on standard benchmarks (ICCAD 2015) and shows better performance than existing baselines. The major contribution is to start from existing design and focus on refinement, and proposes novel intermediate metric like concept of regularity to improve the performance. Overall, all reviewers agree that the work is solid with thorough experiments.

Reviews: review: paper_summary: paper_summary: This paper presents an application of the RL algorithm in chip design optimization. They formulate the chip design process as an MDP process to make decisions based on the current state of the chip macro arrangement so that they can apply PPO to optimize the policies. Compared with starting from scratch, they focus on the fine-tuning stage of the macros for denser reward signals. Their experiments in a public benchmark, the ICCAD 2015 benchmark show a great improvement in the chip design metrics.
strength_weakness: strength_weakness: Compared with MaskPlace [1], the contributions of this paper lie in the expertise or experiences in the expert chip design area instead of the machine learning area, as MaskPlace has already used PPO. One thing I am concerned about is that NeurIPS may not be a good venue to discuss this contribution. 

[1] Lai, Y., Mu, Y., & Luo, P. (2022). Maskplace: Fast chip placement via reinforced visual representation learning. Advances in Neural Information Processing Systems, 35, 24019-24030.
questions: questions: [1] When you discretize the map, it becomes a combinatorial problem. Did you compare the sample complexity of using RL vs using some combinatorial solver?
[2] PPO can handle continuous action space, is there a possibility of improvement using continuous action space?
[3] What will be the future directions for this work or is this problem solved by RL?
limitations: limitations: The authors have discussed the limitations of this work.
paper_summary: paper_summary: This paper introduces a Macro Regulator that uses reinforcement learning (RL) to optimize existing macro placements. It reconsiders the application of RL in macro placement and incorporates a regularity metric into the reward function. The paper presents a comprehensive set of comparative experiments to ultimately demonstrate the superiority of the Macro Regulator. Ablation studies also confirm the effectiveness of each component in the proposed approach.

The contributions of the paper are specified as follows: Firstly, Macro Regulator shifts the focus of RL from placing macros from scratch to refining existing macro placements. This approach enables the RL policy to utilize more state information and achieve more accurate reward signals, enhancing learning efficiency and final PPA quality. Secondly, the paper introduces the concept of regularity into the RL framework, a crucial metric often overlooked in chip design. By incorporating regularity as part of the input information and reward signal, Macro Regulator ensures consistency in manufacturing and performance. Thirdly, the proposed method is evaluated on the ICCAD 2015 benchmark, demonstrating superior performance in terms of global placement half-perimeter wire length (HPWL) and regularity compared to various competing methods. Additionally, PPA metrics are tested using the commercial EDA tool Cadence Innovus, showing significant improvements.
strength_weakness: strength_weakness: While the paper presents a significant advancement in the field of macro placement using reinforcement learning, there are areas for improvement. Expanding the benchmark scope, including real-world case studies, demonstrating generalization capabilities, and providing more detailed algorithmic descriptions would enhance the overall impact and robustness of the work. Addressing these weaknesses would make the paper's contributions even more compelling and practical for real-world applications.
1. Limited Benchmark Scope
- Additional Datasets Needed: While the paper utilizes the ICCAD 2015 benchmark and Cadence Innovus for validation, the scope of the benchmarks is somewhat limited. Including additional datasets or real-world applications could strengthen the evaluation and demonstrate the generalizability of the proposed method. For example, benchmarks from other well-known contests or industrial examples with different characteristics could provide a more comprehensive assessment.
2. Generalization to Different Chip Designs
- Generalization Capability: While the paper mentions the generalization capabilities of Macro Regulator, more detailed experiments and analysis on different types of chip designs (e.g., various sizes, complexities, and design constraints) would strengthen the claims. Demonstrating the method's adaptability to a broader range of scenarios would enhance its perceived robustness and applicability.
3. Clarity of Method Description
- Detailed Algorithmic Steps: While the paper is generally clear, some parts of the methodology could benefit from more detailed descriptions. For example, providing pseudo-code or more granular steps of the Macro Regulator algorithm would help readers better understand the implementation details and reproduce the results. Additionally, regularity is an important metric, but the paper provides limited explanation of this metric. It would be beneficial to supplement more information about its definition and calculation. Furthermore, a more detailed explanation of essential factors such as the states and actions in reinforcement learning would enhance the paper.
questions: questions: In the abstract, the author states, "Our RL regulator can fine-tune placements from any method and enhance their quality." However, in the experimental section, it is mentioned, "For each chip, MaskRegulate uses DREAMPlace to obtain an initial macro placement result to be adjusted, which takes within a few minutes and has relatively low quality." There are two issues with this:
1. The claim of "from any method" is not entirely supported as DREAMPlace is essentially an analytical method, thus the experiments can only prove the ability to improve macro placement solutions from analytical methods, not from any other method.
2. The assertion of "relatively low quality" raises the question of whether the MaskRegulate method relies heavily on the quality of initial macro placement result. It would be worthwhile to explore the effect of optimizing from different quality macro placement solutions (such as higher but not the best macro placement result), rather than starting from relatively low-quality ones.

In summary, can the MaskRegulate method improve the quality of "any method" and "any quality of initial macro placement solution"?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper proposes a novel approach called MaskRegulate for the refinement stage of macro placement, using reinforcement learning (RL) methods. Specifically, it trains an RL policy to adjust the existing placement layouts, from which the policy can receive sufficient information, instead of placing from scratch. Regularity is also considered during the learning process, helping to enhance the placement quality. Experiments demonstrate that it outperforms previous competitive approaches, achieving significant half-perimeter wirelength (HPWL), power, performance, and area (PPA) improvements.
strength_weakness: strength_weakness: 1.	Regularity is also proposed in other works such as [1]. This paper does not discuss how it differs from them.
2.	The paper lacks some discussion on the necessity of using RL other than other methods as the regulator.

[1] A. Vidal-Obiols, J. Cortadella, J. Petit, M. Galceran-Oms, and F. Martorell. Rtl-aware dataflow-driven macro placement. In 2019 Design, Automation Test in Europe Conference Exhibition (DATE), 2019.
questions: questions: 1.	Can the regular mask be used in [2] WireMask-BBO?
2.	Why does GP HPWL improve since only MP HPWL is considered in the learning process? 

[2] Y. Shi, K. Xue, L. Song, and C. Qian. Macro placement by wire-mask-guided black-box optimization. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
limitations: limitations: Yes
paper_summary: paper_summary: This paper utilizes an online RL algorithm to adjust the existing placement layouts, instead of placing the blocks on the scratch. Additionally, the paper introduces one heuristic concept regularity, which better regularity results in higher PPA performance. Besides, this paper tests the PPA performance using commercial software, showing their methods can achieve significant PPA improvements. Experiments show the proposed method improves PPA performance and proxy in the ICCAD 2015 benchmark compared to the state-of-the-art placement works.
strength_weakness: strength_weakness: Several technical details need to be explained.

1.	The most concerning part is the new MDP. The action space is not well described. Compared to MaskPlace, MaskRegulate places the blocks based on the full placement results. So, if the current placing block is placed in an already occupied position, would the occupied block be switched by the current block, or the occupied block be plugged off the board? Compared to placing the blocks one by one, the termination function is clear that the episode would end once all blocks are placed. In this paper, the termination function is not clear since each step is the full placement with all blocks on the board. What is the termination function in this MDP? 
2.	WireMask feature map calculation is also ambiguous in this paper. Since the WireMask feature map is proposed in the MDP that places blocks one by one, it describes the amounts of gains in HPWL when placing the current block on each grid. But, in your MDP, the block seemed to be switched (related to the first question). How do you calculate the WireMask in your case?
3.	In the generalization part, for the unseen chips, is the model training part frozen to do forward inferences only?
questions: questions: Please see the pros and cons part.
limitations: limitations: The regularity is one heuristic knowledge summarized by experienced engineers. As the introduction in section 3.2 “Why does regularity matters?”, the engineer prefers to place macros toward the peripheral regions of the chip. 

Is it possible that there exists some other unknown heuristics, like rectangular shape, or triangle shape that also impact a good performance? 

It may be the open question that applying the existing human heuristic knowledge constraints the agent explores the whole state-action design space.
paper_summary: paper_summary: The paper proposes MaskRegulate, a method that utilizes reinforcement learning in the refinement stage for time-efficient training and accurate reward generation. Additionally, the authors introduce a regularity metric that can be used alongside HPWL in chip placement. To validate their proposed method, the authors evaluated the PPA performance, a real-world metric and compared with competitive baselines.
strength_weakness: strength_weakness: The overall presentation of the paper requires improvement, particularly in the clarity of labels and explanations for figures. Figure 3(a) is especially problematic, as the labels 'adjusting' and 'adjusted' are ambiguous and do not clearly indicate which blocks they refer to. To avoid potential misinterpretation, it is crucial to enhance both the figure and its accompanying explanation. Addressing these issues will significantly improve the paper's clarity and prevent possible misunderstandings.
questions: questions: 1. How does the model determine adjusted and unadjusted macros? Are the macros adjusted sequentially? If so, how is the order of sequence determined?
2. What happens if the model encounters a situation where no feasible space is available? Does it re-adjust previously adjusted macros?
3. In section 4.4, you mentioned using alpha=0.7 for experiments. Do these experiments refer to the results in Table 1 and Table 2? If so, please explicitly state this, as Figure 4 suggests that alpha is a very important hyperparameter.
limitations: limitations: - Apart from some presentation-related issues, I do not observe any significant limitations in this paper. 
- To further strengthen the quality of the work, the authors might consider including training convergence graphs for ChipFormer and MaskRegulate.

Comments: title: Dear reviewer, please read and respond to authors' rebuttal.
Dear reviewer, please read and respond to authors' rebuttal.
Dear reviewer, please read and respond to authors' rebuttal.
Official Comment by Reviewer NxAR
comment: Thank you for your feedback. We are pleased to hear that your concerns have been addressed and the score has been increased.

Regarding the follow-up Q2, we agree that our proposed formulation of RL can also be used for combinatorial methods. To the best of our knowledge, no existing work has used combinatorial optimization under this setting. One may use some suitable combinatorial methods under our proposed RL regulator formulation and the reduction of search space to investigate if they can be improved. We will leave this for future work. Thank you for your insightful suggestions.
Thank authors for answering my questions, I have adjusted my ratings.
Follow-up Q2: I guess your RL method benefits from the reduction in the state/action space, and similarly, combinatorial methods can also benefit from this, was there any baseline run under the same action space or problem setting?
This paper has diverse reviews and it would benefit a lot to start a discussion to clarify any confusing points.

Thanks!

Your AC.
This paper has diverse reviews and it would benefit a lot to start a discussion to clarify any confusing points.

Thanks!

Your AC.
This paper has diverse reviews and it would benefit a lot to start a discussion to clarify any confusing points.

Thanks!

Your AC.
Thank you very much for addressing my concerns. I will keep my score.
Thank you very much for addressing my concerns. I will keep my score.