Reinforcement Learning Policy as Macro Regulator Rather than Macro Placer
Authors: Ke Xue
Ruo-Tong Chen
Xi Lin
Yunqi Shi
Shixiong Kai
Siyuan Xu
Chao Qian
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement learning
EDA
Placement
Chip design
Primary area: machine_learning_for_physical_sciences

Abstract: In modern chip design, placement aims at placing millions of circuit modules, which is an essential step that significantly influences power, performance, and area (PPA) metrics. Recently, reinforcement learning (RL) has emerged as a promising technique for improving placement quality, especially macro placement. However, current RL-based placement methods suffer from long training times, low generalization ability, and inability to guarantee PPA results. A key issue lies in the problem formulation, i.e., using RL to place from scratch, which results in limits useful information and inaccurate rewards during the training process. In this work, we propose an approach that utilizes RL for the refinement stage, which allows the RL policy to learn how to adjust existing placement layouts, thereby receiving sufficient information for the policy to act and obtain relatively dense and precise rewards. Additionally, we introduce the concept of regularity during training, which is considered an important metric in the chip design industry but is often overlooked in current RL placement methods. We evaluate our approach on the ISPD 2005 and ICCAD 2015 benchmark, comparing the global half-perimeter wirelength and regularity of our proposed method against several competitive approaches. Besides, we test the PPA performance using commercial software, showing that RL as a regulator can achieve significant PPA improvements. Our RL regulator can fine-tune placements from any method and enhance their quality. Our work opens up new possibilities for the application of RL in placement, providing a more effective and efficient approach to optimizing chip design. Our code is available at \url{https://github.com/lamda-bbo/macro-regulator}.

Reinforcement Learning Policy as Macro Regulator Rather than Macro Placer
Ke Xue Ruo-Tong Chen Xi Lin Yunqi Shi Shixiong Kai Siyuan Xu², Chao Qian 1.2
National Key Laboratory for Novel Software Technology. Nanjing University School of Artificial Intelligence, Nanjing University Huawei Noah's Ark Lab
Abstract
In modern chip design, placement aims at placing millions of circuit modules, which is an essential step that significantly influences power, performance, and area (PPA) metrics. Recently, reinforcement learning (RL) has emerged promising technique for improving placement quality, especially macro placement. However, current RL-based placement methods suffer from long training times, low generalization ability, and inability to guarantee PPA results. A key issue lies in the problem formulation, i.e., using RL place from scratch, which results in limits useful information and inaccurate rewards during the training process. In this work, propose approach that utilizes RI. for the refinement stage, which allows the RL policy to learn how to adjust existing placement layouts, thereby receiving sufficient information for the policy act and obtain relatively dense and precise rewards. Additionally, we introduce the concept of regularity during training, which is considered an important metric the chip design industry but is often overlooked in current RL placement methods. We evaluate our approach on the ISPD 2005 and ICCAD 2015 benchmark, comparing the global half- nerf-eerrmeter wirelength and regularity of our proposed method against several competitive approaches. Besides, we test PPA performance using commercial software, showing that RI. as a regulator can achieve significant PPA improvements. Our RI. regulator can fine-tune placements from any method and enhance their quality. Our work opens up new possibilities for the application of RL in placement, providing a more effective and efficient approach to optimizing chip design. Our code is available httph::t//// //te::://tttttt
1 Introduction
In the complex and evolving landscape of modern chip design, placement is pivotal process that significantly influences the power, performance, and area (PPA) metrics of the final chip [21 22). modern chip typically comprises thousands of macros (i.e., individual building blocks such as memories) and millions of standard cells (i.e., smaller basic components like logic gates). The macro placement result provides fundamental solution for the subsequent processes (e.g., standard cells placement and routing), thus playing an important role [32]. For example, macro placement influences the placement of standard cells, and poor macro placement might make it challenging to place these cells optimally, leading to an unsatisfactory chip performance [33] Moreover, an inappropriate macro placement can result in macro blockage in the core center, which harms the overall chip performance by causing unwanted effects such as routing congestion, inferior wirelength, and timing performance issues [26]
"Corresoondence to Chao Qian <qianc @ nju.edu.en
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

14
DREAMPInce
12
AutoDMP
WireMask-BBBO
MaskPlace
10
MaskRegulate
Congestion (%)
8
(a) MaskPlace
2
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
-TNS (10$ μu))
(b) MaskRegulate (Ours)
(c) PPA results
Figure 1:
Placement layouts and congestions of (a) MaskPlace and (b) MaskRegulate the su-
perbluel from ICCAD 2015 benchmark [14), where the red points indicate the congestion critical regions. (c): Comparing two crucial PPA metrics, namely Congestion and total negative slack (TNS) between Regkleg,late, DREAMPlace [9], AutoDMP (. WireMask-EA [29], and MaskPlace (T6), where lower values indicate better performance. These results are obtained using Cadence Innovus.
Due to the lengthy and complex workflow of chip design, designers often rely proxy metrics that can reflect the final results to guide the optimization process 2 30 One important proxy metric is half-perimeter wirelength (HPWL), which provides approximation for the routing wirelength and is widely used to measure the placement quality [3] 28 Traditional macro placement methods can be divided into two categories. Earlier approaches usually solve macro placement by black-box
optimization (BBO) 24 12 22). They often suffer from the poor scalability due to the large-scale search space and high complexity of decoding solution placement. Another type is analytical method 6 19), which can solve the placement efficiently by approximating HPWL gradients. However, these methods are hard guarantee the oon-overla constrain between cells and are easy to be stuck in local optima [16 34].
Reinforcement learning (RL) 31 has recently emerged promising technique enhance the macro placement quality 23
15). RL's ability to learn policies through interaction with a complex environment offers novel pathway for addressing various challenges of macro placement. However, the application of RL is currently hindered by several limitations, including the long training time, inability guarantee PPA improvements, and the lack of generalization across different chip layouts. this work, we highlight that a major contributing factor to these issues the problem formulation, i.e., the conventional RL approach of placing macros from scratch often results in limited state information and inaccurate reward signal throughout the learning process.
To address these challenges, we propose novel RL approach called MaskRegulate that shifts focus from initial placement to refining existing placement layouts. The RL policy acts as a regulator rather than placer, which operates pre-existing placements, thus allowing for access to comprehensive state information and enabling the acquisition of more precise rewards. This adjustment enhances the efficiency of the learning process and finally improves the final placement results. Furthermore, MeskRegulate introduces the concept regularity [26] as a part of input information and critical reward signal, which has been largely overlooked in previous research despite its significance in ensuring manufacturability and performance. Previous methods often only consider the HPWL metric, suffering from optimizing different metrics effectively. By integrating regularity into the RL framework, our approach aligns more closely with advanced chip requirements.
The effectiveness of the proposed ReskRegulate is comprehensively evaluated on the ICCAD 2015 benchmark 114, which is is currently one of the largest open-source benchmarks that allows us to evaluate PPA metries such as congestion and timing slack. We first compare the global

---

HPWL and regularity of our approach against several competitive methods. Additionally, we use the commercial electronic design automation (EDA) tool Cadence Innovus to evaluate the PPA performance, demonstrating that our proposed MaskRegulate can lead to significant PPA improvements, e-g., the placement layouts and two PPA metrics on superbluel, as shown Figure [ Specifically, compared to MaskPlace (an advanced RL placer [66;; Regkleg shares a similar architecture to it), MeskRegulate improves 17.08% on routing wirelength, 73.08 73.08% and 38.81 % on routed horizontal and vertical congestion overflow respectively, 18.35% worst negative slack, 37.89% on total negative slack, and 46. 17% on the number violation points.
This work provides a more effective approach for macro placement of modern chips, opening new possibilities for the application of RL in chip design. The contributions of this work are highlighted in three key points:
Novel problem formulation: Innovatively applying RL in the refinement stage of macro placement, which allows for more effective learning from structured state and accurate reward information, significantly enhancing the learning efficiency and effectiveness.
Integration of regularity: Introducing regularity, a critical yet previously overlooked metric in chip design, into the RL training framework, which not only aligns with industry practice but also enhances the chip PPA quality.
Impressive PPA improvement and comprehensive analysis: On the popular ICCAD 2015 benchmark, our proposed MaskRegulate demonstrates significant improvements in PPA metrics, showing the practical applicability and effectiveness of the RL regulator.
2 Background
2.1 Placement
The cireuit in the placement stage is considered graph where vertices model gates. The main input information is the netlist N = (V, E), where V denotes the information (i.e., height and width) about all macros designated for placement on the chip, and E is hyper-graph comprised of nets E E, which encompasses multiple cells (including both macros and standard cells) and denotes their inter-comeettivity in the routing stage. Given netlist, a fixed canvas layout and standard cell library, a placement method is expected to determine the appropriate physical locations of movable macros such that the total wirelength can be minimized.
macro placement solution
[(11, y1),- ... (Ix, yk.)} consists of the positions of all the macros {e; {= where k denotes the total number of macros. One popular objective of macro placement is to minimize the total HPWL of all the nets while satisfying the cell density constraint, which is formulated as,
min HPW = min Σ HPW L.(8), s.t. D(s) Se EEE
where D denotes the density, is threshold, and HPW Le is the HPWL of net e, which is defined as: HPW (s) (max,,ee I3 min,,ee z;)) + (max..e y: min,, yi).
There are three mainstream placement methods, i.e., analytical methods, black-box optimization methods, and learning-base based methods. Analytical methods place macros and standard cells simultaneously, which can be roughly categorized into quadratic placement and nonlinear placement. Quadratic placement []] [8] iterates between an unconstrained quadratic programming phase to minimize wirelength and heuristic spreading phase to remove overlaps. Nonlinear placement [6] 20 77 formulates a nonlincar optimization problem and tries to directly solve with gradient descent methods. Generally speaking, nonlinear placement can achieve better solution quality, while quadratic placement is more efficient. Recently, there has been extensive attention on GPU-accelerated non- linear placement methods. For example, DREAMPlace [19 17 transforms the non-linear placement problem in Eq. (T) into a neural network training problem, solves it by classical gradient descent and leverages GPU, enabling ultra-high parallelism and acceleration and producing state-of sate-oof-hhe-ar analytical placement quality.
Black-box optimization methods for placement have a long history. Earlier methods such SP 24 and B -tree [5] have poor scalability due to the rectangular packing formulation. Recently, some black- box optimization methods have made significant progress by changing the search space. AutoDMP []]
3

---

improves DREAMPlace by using Bayesian optimization to explore the configuration space and shows
remarkable performance on multiple benchmarks. WireMask-BBBO (29 adopts wire-mask-guidde -guided greedy genotype- genotype-phenotype mapping and can be equipped with any BBO algorithm, demonstrating the superior performance over other types of methods.
2.2 RL for Macro Placement
Researchers recently leverage RL--based methods for better placement quality to meet the demands of modern chip design. GraphPlace [23) first models macro placement as RL problem. It divides the chip canvas into discrete grids, with each macro assigned discrete coordinates of grids, wherein the agent decides the placement of the current macro at each step. However, no reward is given until all the macros are placed, making the reward sparse and hard to learn. DeepPR 9 and PRNet ( incorporate macro placement, standard cells placement, and routing to achieve better performance than GraphPlace, but may violate the non-overlap constraint. To address this issue, MaskPlace [[6] introduces dense reward and uses pixel-Ievel visual representation for circuit modules, which can comprehensively capture the configurations of thousands of pins, enabling fast placement in full action space a large canvas size. MaskPlace has many attractive benefits that previous methods do not have, e.g.. 0% overlap, dense reward, and high training efficiency. ChiPFormer [15] incorporates an offline learning decision transformer and focuses on improving the generalizability of placer. EfficientPlace [10) integrates a global tree search algorithm to guide the optimization process, achieving remarkable placement quality within short time.
However, current RL methods exhibit several shortcomings: 1) Placing from scratch provides insufficient state information and inaccurate reward signals; 2) Most methods focus on minimizing wirelength, which may bring macro blockages and thus harm the final PPA metrics. In this work, we propose novel RI. approach for macro placement: a macro placer. Specifically, our learned RI. policy is designed to adjust macros based on an existing placement result, rather than placing all macros from seratch. This approach aims to refine and optimize pre-existing layouts, addressing the limitations of traditional RL-based placement methods.
RL policy acts as macro regulator rather than
3 Method
We present our proposed MaskRegulate here. Section 3.1 introduces our problem formulation and policy architecture, and Section 3.2 describes how to integrate regularity into the method.
3.1 Megklag Framework
Problem formulation of RL regulator. In the Markov Decision Process (MDP) formulation of traditional RL placer, a macro is placed at each step 23 16 [3]. The placement order of macros is determined based on some pre-defined rules, such as the number of nets, the size of macros, and the number of connected modules that have been placed. An episode ends after all macros have been placed. Typically, the state representation includes information about the chip canvas, the macros that have already been placed, and the macro currently being placed. In GraphPlace 23), the reward is determined only after all macros have been placed, resulting in sparse reward signal that complicates the training process. Recent works have introduced various methods densify the reward signal. For instance, WireMask [16] provides more continuous reward based on the macros already placed. In contrast to RL placers, our RL regulator focuses on refining an existing placement by adjusting the location of one macro at each step. Unlike the placer, which initiates the placement process from scratch, the regulator benefits from additional information when adjusting each macro. Specifically, the regulator considers not only the macros that have already been placed but also the positions of all other macros. Furthermore, it enhances accuracy by taking into account all macros, even while employing reward function similar to WireMask.
Due to the advantages mentioned above in the MDP problem formulation, even without considering additional factors (e.g., regularity), RL regulator is able to achieve better results compared to RL placer, as shown in our experiments in Appendix [.. Furthermore, our main experimental results demonstrate superior performance not only in proxy metrics but also in PPA metrics measured by commercial tools, as shown in Section 4.2 The regulator also exhibits better generalization abilities, as shown in Section 4.3 Intuitively, adjusting unseen chip is casier for the regulator compared to

---

fp
modules
M!
M
Position Mask hws fw
Mask unfeasible action
Local Mask
all modules
Fusion
Fr
Merge
Probability
Regularity Mask
Matrix actions
Far f(+l
(5,
Wire Mask
Policy Network
adjusted Mi---1
f
Global Mask Encoder/Decoder
Regularity
ppll fly
padjusted
(5.)
PC
Position embedding
Pixel-based State Representation
Value Network
Figure 2: Overview MeskRegu MeskRegulate shares a similar architecture to MaskPlace [6 except for the MDP formulation and the integration of regularity in the state and reward.
Nr
(a) Chip canvas
(b) PositionMask
(c) WireMask
(d) RegularMask
Figure 3: Illustration of chip canvas, PositionMask, WireMask and RegularMask. We use the left- bottom corner of the module to denotes location.
placing macros from scratch, as the incomplete state information of placer would be even worse in the case of unseen chips, resulting in poorer performance.
Policy architecture. Our policy architecture is illustrated in Figure 2 The policy divides the chip canvas into several grids and utilizes visual information as inputs, converting chip information into pixel-level image masks. This approach has demonstrated superior efficiency and performance in RL placer policy learning [16] [5). The inputs include an image of the current canvas, a PositionMask that identifies all valid positions for placing the current macro, WireMask (16) that indicates the approximate wirelength change for placing the current macro at each valid position, and RegularMask that indicates the change regularity for placing the current macro each valid position (which will be detailed in Section 3.2). An illustration of the PositionMask and WireMask provided in Figure consider only macros that have already been adjusted; thus, grids occupied by unadjusted macros are available for placement. In our MaskRegulate, the calculation of the WireMask is based on all macros, allowing its value to either increase or decrease. These values are normalized to the range [-1,1], 1). unlike the [0, 1] normalization used [16]. Additionally, our framework introduces the RegularMask to quantify changes in regularity within the state and to encourage improvements in regularity through the reward function, as presented in Section 3.2
To facilitate broader adjustments, the PositionMask has been modified to
3.2 Integration Regularity
Why does regularity matters? Macro placement has significant impact on subsequent chip design processes, including standard cell placement and routing. only focusing on minimizing wirelength (which is the case for most current RL placers), certain macros may end up positioned in the middle

---

of the chip canvas, resulting in macro blockages 26). This. in turn, leads to the division of available placement areas into separate and disconnected sub-regions. As a consequence, standard cells that are connected by the same net may be scattered across different placement sub-regions, resulting in increased overall wirelength and the potential routing challenges, which ultimately degrade the timing performance. Thus, a well-established practice among experienced engineers in macro placement is to place macros towards the peripheral regions of the chip to prevent macro blockage. In this work, we aim to integrate regularity in the learning-baset placement approach to achieve placement preferences similar to those of experienced engineers.
RegularMask. Intuitively, macros closer to the edges tend to have lower regularity. Therefore, we propose a simple and effective way to measure regularity. On a canvas, the regularity of a grid located at (z, y) is calculated as min X =} + min Ymmx yt. where Xmax and Ymax represent the real length of the horizontal and vertical axes, respectively. Given macro to be placed, the RegularMask measures the value change in regularity for each valid placement position, as illustrated in Figure dd)
Reward and policy learning. The reward of MaskRegulate consists of two components: Twire and Freg. which represent the reduction of HPWL and the improvement in regularity, respectively, after refining the current macro. To mitigate the influence of scale differences on training caused by wirelength and regularity, both Twirs and Trey are normalized to [0, 1]. The final reward is a Twire + (1 - a) -Treg. where a is a trade-off coefficient. We will analyze the influence of a in Section 4.4, showing that different a lead to different multi-objective preferences. The detailed information are presented in Appendix A.33 MaskRegulate treats the chip canvas as a grid and divides into N x N cells, resulting in N2 possible discrete actions. We use the popular proximal policy optimization (PPO) algorithm [7 to learn the regulator policy.
Experiment
In this section, we first introduce the basic experimental settings, including the tasks and evaluation
metrics in Section 4.1 Then, we try to answer the following three research questions (RQs) in
Sections 4.2 to 4.4 I) How does MaskRegulate perform compared to other methods? 2) How is the generalization ability of MaskRegulate? 3) How do the different parts of MaskRegulate affect the performance? Finally, we provide the visualization of placement results and congestion in Section 4.5
4.1 Experimental Settings
Tasks. We mainly use the ICCAD 2015 benchmark [1] our test-bed, which includes sufficient advanced chip information and is currently one of the largest open-source benchmarks that allows us to evaluate congestion, timing and other PPA metrics. The benchmark statistics are listed in Table in Appendix I. Although ICCAD 2015 is the benchmark we have found that closely reflects the current practices in the EDA industry, it still has some shortcomings. For example, it allows for large placement area, resulting in loose placement results that do not adhere to the design principles of advanced modern chips. Note that the "A" in PPA denotes "Area", which is a core metric of chip design and should be minimized
3 32]. Therefore, we scale down the chip's 's placement area, presenting further challenges for the compared methods. Besides, we also conduct experiments on ISPD 2005 benchmark (25), which is also a popular benchmark in AI for chip design but does not have sufficient information for PPA evaluation. Detailed results can be found Appendix B
Proxy evaluation metrics. We use the following two popular proxy metrics for a quick comparison of different algorithms: 1) Global HPWL. After determining the locations of all the macros, we use DREAMPlace [19] to place standard cells to obtain the global placement result, and then report the global HPWL (i.c.. full HPWL involving both macros and standard cells). Compared macro HPWL, global HPWL considers the total wirelength, typically on a scale that is two orders of magnitude larger, providing a better estimation of the final real performance of the chip. 2) Regularity: We compute the regularity values for all macros, which serve as a measurement of the overall regularity of the placement result. We run each algorithm for five times and report their mean and variance. We do not consider the rectangular uniform wire density (RUDY) metric 30 for congestion proxy, as this approximation is sometimes positively correlated with the HPWL metric and is not accurate 22 Instead, we will evaluate congestion within our PPA evaluation.

---

PPA evaluation metrics. The whole chip design process is lengthy and complex, and proxy metrics may not accurately capture the true performance of the chip. PPA metrics often require the use of commercial EDA tools to obtain precise results with expensive cost. In our experiments, we select the best placement result for PPA evaluation based on global HPWI. from multiple runs. After obtaining the global placement results, we use commercial tool Cadence Innovus to proceed the subsequent stages and evaluate their PPA metrics, including routed wirelength, routed vertical and horizontal congestion overflow, worst negative slack, total negative slack, and the number of violation points. These metrics are extremely important measures chip design and are typically considered to evaluate the quality of chip comprehensively.
4.2 RQ1: How does MaskRegulate perform compared to other methods?
We consider the following methods to be compared: DREAMPlace [19] A state-of-the-ar analytical placer; AutoDMP M: A method that improves DREAMPlace by exploring its configuration space iteratively; WireMask-EA [29] : A state-of-the-ar black-box macro placement method with EA as the optimizer; MaskPlace 16 A representative online RL methods, which shares similar policy
architecture, state, HPWL reward with our MaskRegulate.
For the same components, MaskPlace and MaskRegulate use the same settings, c.g., the number of grids, and the learning rate. Detailed information is provided in Appendix [A.3 [A.3] Additionally, in order to demonstrate that the regulator has higher training efficiency than the placer, MaskRegulate and MaskPlace are trained for 1000 and 2000 episodes, respectively. For each chip, Meskle uses
DREAMPlace obtain initial macro placement result to be adjusted, which takes within few minutes and has relatively low quality.
The overall evaluation results are shown in Table | MaskRegulate achieves the best average rank on both proxy and PPA metrics. DREAMPlace has the worst average ranking on wirelength, congestion, and timing. However, after adjustment by MaskRegulate, the obtained placements achieve the best average rank. Compared to MaskPlace, MaskRegulate leads to significant improvements in multiple PPA indicators: improves 17.08% on routing wirelength, 73.08% and 38.81 % routed horizontal and vertical congestion overflow respectively, 18.35% on worst negative slack, 37.89% on total negative slack, and 46. 17% on the number of violation points. By incorporating regularity, Reguleg achieves the highest regularity on all the eight chips. We can observe a certain correlation between the proxy metric (global HPWL) and the real metric (rWL), but there still exists a gap, indicating the challenges involved in the placement task. Furthermore, we provide detailed visualizations of placement results in Figure 5 where MaskRegulate shows significant improvements on congestion metrics. Besides, the final placement layouts of MaskRegulate are much regular than all the other methods.
4.3
RQ2: How the generalization ability of MaskRegulate?
The generalization ability of RL policies is an important question to be investigated. In this section, we pre-train MaskRegulate and MaskPlace on the first four chips (i.e., superbluel, superblue3, superblue4, and superblue5) and test the remaining four chips. To further validate the ability of MaskRegulate to adjust different initial placement results, we use it to adjust the results obtained by different initial placements the test chips.
The results are shown in Table 2 On both the global HPWL and regularity metries, Megulagulate consistently outperforms MaskPlace, showcasing its stronger generalization capability. An interesting finding is that MaskRegulate performs better on unseen chips than on the chips it was trained on, specifically in terms of global HPWL, such as with superblue16 16. This may suggest that MaskRegulate has learned some general knowledge during the pre-training process, enabling to overcome local optima that may arise from direct learning on the target chip.
4.4 RQ3: How do the different parts of Mask affect the performance?
We investigate the influence of different parts and provide additional analysis in this section.
Hyperparameters sensitivity analysis: different trade-off coefficient leads to different multi- objective preferences. One hyperparameter of RegularMask is the coefficient a between HPWI. reward Twire and regularity reward Treg. where a higher indicates preference for optimizing
7

---

Table Results of proxy metrics and PPA metrics on the ICCAD 2015 benchmarks. Global HPWL (1e8) and Regularity (1e6) are two proxy metrics. PPA metrics are evaluated by Cadence Innovus. The placement is performed by different methods, and the subsequent stages are performed by Cadence Innovus. rWL. (m) is the routed wirelength; rO-H (%) and rO-V (%) represent the routed horizontal and vertical congestion overflow, respectively; WNS (ns) the worst negative slack; TNS (1e5 ju) is the total negative slack; NVP (Ie4) is the number of violation points. WNS and TNS are the larger the better, while the other metrics are the smaller the better. The best result each metric
on each chip is bolded.
Proxy metries Global HPWL
metrics
Benchmark
Method
Regularity
rWL
rO-H
r0-V
WNS
TNS
NVP
DMP
8.96 0.84
4.15 0.04
154.23
17.15
4.48
-119.616
-2.91
3.35
AutoDMP
8.13 = 0.17
4.99 = 0.08
185.60
20.99
5.73
-124.572
-3.72
3.46
superbluel
WireMask-EA EA
8.07 = 0.38
4.41 0.15
149.49
7.62
0.38
-67.616
-3.57
2.94
MaskPlace
7.93 0.06
4.40 0.06
158.59
16.28
0.64
-72.070
-3.98
4.41
MaskRegulate
5.77 0.05
3.31
-60.532
DMP
12.87 +:
4.43
232.19
40.55
19.64
9969004
-2.36
AutoDMP
8.13
5.49
166.15
14.71
-76.566
-1.12
1.44
superbluc3
WireMask-EA EA
9.37 0.81
4.77 0.23
167.67
7.81
0.32
99.566
-1.57
2
MaskPlace
8.90 0.17
4.77 0.06
177.25
9.16
0.64
-111.041
-1.77
2.02
MaskRegulate
7.05 0.03
3.54 0.00
142.89
1.86
0.18
-83.635
-1.15
0.97
DMP
6.81 0.23
3.06 0.011
132.16
20.62
4.87
-73.192
-1.63
2.42
AutoDMP
4.57
3.41 0.06
82.94
48.1337
-0.64
superbluc4
WircMask-EA EA
0.07
3.25
110.20
8.29
0.61
-83,233
-1.85
MaskPlace
5.28 =
3.22 0.03
106.36
9.71
0.31
-67.995
-1.47
1.9
MaskRegulate
4.15 0.06
0.02 2.18
81.78
0.29
0.11
-49.771
-0.90
0.88
DMP
8.78 4 1.47
4.84
0.06
144.64
3.75
0.46
-58.907
-0.68
1.64
AutoDMP
12.67 1: 4.09
5.79 0.32
344.14
74.75
37.32
-197.175
-5.83
3.55
superblue5
WireMask-Ea EA
1..23:0..6
5.03 0.15
189.84
4.06
0.41
-75.115
-1.83
MaskPlace
9.81
4.86
4.79
0.37
-118.122
-2.98
2.62
MaskRegulate
6.94 + 0.00
0.01 4.23
137.79
0.02
0.02
-74.83
-0.73
1.32
DMP
22.70 0.91
4.24
0.02
427.71
100.32
73.18
-123.310
-6.55
7.02
AutoDMP
10.04 1.63
5.25 1 0.09
221.69
6.32
0.64
-52.556
-1.82
4.31
superblue7
WireMask-EAA EA
10.05 1 0.34
4.31 0.13
195.36
0.82
0.35
-73.070
-1.78
3.45
MaskPlace
4.36
204.32
0.33
-69.441
-2.18
MaskRegulate
7.90 0.03
0.00 3.03
162.72
0.59
0.64
-50.994
-1.48
2.27
4.33
261.35
5.16
-83.509
AutoDMP
11.48 1: 1.78
6.55
0.15
234.24
2.83
1.37
-1695540
-2.84
1.71
superblue
WireMask-FA RA
13.52 + 2.25
4.82 + 0.12
223.08
0.76
0.55
121,7885
-3.11
1.65
MuskPlace
10,94 0.21
4.75 1: 0.11
212.85
0.51
0.19
-81.916
-437
2.11
MuskRegulate
1
3,90
0.44
212.84
-77.980
-2.74
DMP
10.22
2.78 0.02
187.03
74.53
30.31
-38.3770
-4.14
4.95
AutoDMP
6.49
3.68
119.62
2.46
0.22
-41,292
-1.17
2.26
superblue16
WireMask-Ra RA
6.13
3.17 0.17
120.45
8.20
0.36
-89.395
-2.19
2.66
MuskPlace
5.53 0.04
0.04
3.08 +: 0.06
110.52
2.46
0.11
36.4488
-1.36
2.51
MaskRegulate
5.53
2.55 0.12
106.82
1.40
0.17
45.9622
-1.77
1.88
DMP
4
2.44 0.00
85.93
11.76
8.93
-73.429
-0.46
0.95
AutoDMP
3.02 002-000
3.01 0.10
61.57
1.02
0.02
-17.545
-0.32
superblue18
WireMask-Ei EA
3.46 0.07
2.62
-34.143
MuskPlace
3.49 + 0.06
2.59
0.08 1.55
70.24
1.08
0.40
-43.869
-0.76
1.31
MuskRegulate
2.96 0.01
0.00
60.64
0.04
0.03
28.285
-0.40
0.85
DMP
4.625
4.375
4.5
4.75
3.875
3.625
4.125
AutoDMP
3
3.375
3.5
3.375
2.75
2.25
2.5
Average Rank
WireMask-Fa
3.75
3.75
2.75
2.875
3,375
3.25
MaskPlace
2.375
3.25
3.125
3.125
2.375
3.125
3.875
MaskRegulate
1.25
1.125
1.125
1.625
1.875
1.875
1.5
8

---

Table Generalization results of proxy metrics on the four chips of ICCAD 2015 benchmarks. The best result of each metric on each chip is bolded.
Regulaee
MaskPlace
Benchmark
Global HPWL (Ie8)
Regularity (1e6)
Global HPWL (1e8)
Regularity (1e6)
superblue7
7.99 = 0.06
3.04 0.00
10.33 + 0.17
4.24 = 0.06
superblue 10
11.55 + 0.27
3.25 = 0.00
11.88 + 0.72
4.73 = 0.05
superblue16
5.16 0.05
1.87 0.00
5.97 0.20
3.11 + 0.03
superblue18
3.04 + 0.02
1.54 + 0.00
3.69 + 0.09
2.52 + 0.03
0,5
12.5
DREaMetee Viebla Meutter Studiopdee
Regularity (106)
0.6
Trude- ratão
Regulacity (106)
3.8
4.0
0.6
Trudc-oll rutso
Congestiant))
Meutthe
Masolopdaer
Voadea 680
Congestion
8.0
7.5
3.8
10
0.4
0.4
50
36
2.5
DRLAMPiee
DXLAAMPPace
6063 7.935
03 L. 18:20
0.5
1,0
2.0 2.5 3.0 3.5 4.0 4.5
1.0
2.0
3.0
6.0
70
Global HPWL (10%)
Global HPWL (10")
-TNS (10"
-TNS (10' (us
(a) Proxy metrics on superbluel
(b) Proxy metrics on superblue7
(c) PPA metries on superbluel
(d) PPA metrics on superblue7
Figure 4: Illustration of MaskRegulate regulators with varying a values (ranging from 0.1 to 0.9).
HPWL, and vice versa. In this section, we investigate the influence of the trade-off coefficient a. We train different MaskRegulate regulators with varying values (ranging from 0.1 to 0.9) and report the proxy and PPA results in Figure Due the expensive computational cost of PPA, we select four different trade-offss of MaskRegulate for evaluation. As expected, different values lead to different multi-objective preferences. In our experiments, we use a 0.7 for all the chips as it achieves relative balance between different objectives.
Ablation studies. We consider the following ablations of MaskRegulate. 1) Only changing the prob- lem formulation and purely comparing placer and regulator. We implement Manilla-MMsRRegulatee where the only difference to MaskPlace the problem formulation, and all the other components (e.g.. state and reward) are the same. The results show that Manill-MaakRegulate consistently outperforms MaskPlace in terms of Global HPWL. 2) MaskRegulate with without normalization. Since global HPWL has large scale than regularity, Mask w/o normalization does not prefer to consider regularity, which not what we expect. 3) Training regularity aware RL placer from scratch. We implement MaskPlace + RegularMask and compare it with MaskPlace and MaskRegulate, The results show the advantages of the integration of regularity (between MaskPlace and MaskPlace + RegularMask) and our RL regular formulation (between MaskPlace + RegularMask and MaskRegu- late). The above ablation results demonstrate the effectiveness of each component Mask Regulate Detailed results and discussions are provided in Appendix B.T due to space limitation.
4.5 Visualizations of placement results and congestion.
We provide the detail visualizations of placement results of all the methods on all the eight chips from ICCAD 2015. As shown in Figure 5 our proposed MaskRegulate shows significant improvements on congestion metrics. Besides, placement result MaskRegulate much regular than all the other methods.
4.6 Additional results.
We conduct the following additional results to comprehensively show the effectiveness of our MaskRegulate. 1) To verify whether using a better model structure for the RL placer can compare to the regulator, we add comparison with recent proposed ChiPFormer [15] under a fair setting. 2) To further show the generalization ability of our methods, we conduct generalization experiments on the ISPD 2005 benchmark 25) 3) To investigate whether MaskRegulate can be used to adjust any initial macro placement solution, use the pre-trained model to fine-tune other placement results.

---

(a) DREAMPlace
(b) AutoDMP
(c) WireMask-EA
(d) MaskPlace
(c) MaskRegulate
Figure 5: Placement layouts and congestions of different methods on the eight ICCAD 2015 bench- marks The congestion results are obtained by Cadence Innovus, where red points indicate the congestion critical regions.
These results further demonstrate the competitive results our proposed MaskRegulate. Detailed discussions are provided in Appendix B.2 B.3 andB.4 respectively.
Final Remarks
Conclusion. this paper, we present novel RL problem formulation for macro placement, focusing on the development of macro regulator rather than placer. Our proposed method, MaskRegulate, demonstrates substantial improvements chip placement quality by refining existing layouts instead of generating them from scratch. By integrating dense reward signals and emphasizing regularity, our approach effectively addresses the limitations of traditional RL-based placement methods, resulting in superior performance in PPA metrics across various chips. This advancement paves the way for more efficient and effective chip design through RL.
Limitations and future work. This study has several primary limitations: it does not consider the impact of module aspect ratio and area factors on placement; overlooks global wirelength and timing metrics during the training process; and it does not employ advanced transformer architectures [I5] to enhance the generalization of the regulator. Chip design inherently involves different preferences, such as the need for compact size in mobile phone chips and larger sizes for computer chips. Therefore, future research should address these challenges and explore efficient methods to obtain set of chip placements that accommodate different preferences using multi-objective optimization.
10

---

Acknowledgments and Disclosure of Funding
We thank the reviewers for their insightful and valuable comments. This work was supported by the National Science and Technology Major Project 11222D01166000 the National Science Foundation of China (62276124), and the Fundamental Research Funds for the Central Universities (14380020).
References
[1] A. Agnesina, Rajvanshi, T. Yang, G. Pradipta, A. Jiao, B. Keller, B. Khailany, and H. Ren. AutoDMP: Automated DREAMPlace-based macro placement. In Proceedings of the 27th International Symposium on Physical Design, pages 149-157, Virtual, 2023.
[2] A. E. Caldwell, A. B. Kahng, S. Mantik, I. L. Markov, and A. Zelikovsky. On wirelength esti- mations for row-based placement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, (995)1265-2278 1999.
[3] H. Chan, P. Mazumder, and K. Shahookar. Macro-cell and module placement by genetic adaptive search with bimmap-eepresented chromosome. Integration, 12(1):49-75 1991.
[4] Y. Chang, Z. Jiang, and T. Chen. Essential issues in analytical placement algorithms. IPSI Transactions on System LSI Design Methodology, 2: 2:145-166, 2009.
[5] Y.C. Chang, Y.W. Chang, G.-M. Wu, and S.-W. Wu. Bteere A new representation for non-slicing floorplans. In Proceedings of the 37th Annual Design Automation Conference, pages 458 463, Los Angeles, CA, 2000.
[6] T.-.. Chen, 7.-W. Jiang, T--C. Hsu, H.-C. Chen, and Y.-W. Chang. Ntuplace3: An analytical placer for large-scale mixed-size designs with preplaced blocks and density constraints. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2008.
[7] C.-K. Cheng, A. B. Kahng, I. Kang, and L. Wang. Replace: Advancing solution quality and routability validation in global placement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 3817):177 2018.
[8] R. Cheng, X. Lyu, Li,
J. Ye, J. Hao, and J. Yan. The policy-gradinn placement and generative routing neural networks for chip design. In Advances in Neural Information Processing Systems 35, New Orleans, LA, 2022.
[9] R. Cheng and J. Yan. On joint learning for solving placement and routing in chip design. In Advances in Neural Information Processing Systems 34, pages 16508-16519, Virtual, 2021.
[10] Z. Geng, J. Wang, Z. Liu, S. Xu, Z. Tang. M. Yuan, H. Jianye, Y. Zhang, and F. Wu. Rein- forcement learning within tree search for fast macro placement. In Proceedings of the 4 International Conference Machine Learning, 2024.
[11] X. He, Huang, L. Xiao, H. Tian, and E. F. Young. Ripple: robust and effective routability- driven placer. IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems, 32(10):1546-1533 2013.
[12] X. Hong, G. Huang, Y. Cai, J. Gu. S. Dong, C.-K. Cheng, and J. Gu. Comee block list: An effective and efficient topological representation of non-slicing floorplan. In Proceedings of the 13th IEEE/ACM International Conference on Computer Aided Design, pages 8-12, San Jose, CA, 2000.
[13] A. B. Kahng and S. Reda. A tale of two nets: Studies of wirelength progression in physical de- sign. In Proceedings of the 7th International Workshop System-level Interconnect Prediction, pages 17-24, Munich, Germany, 2006.
[14] M. Kim, J. Hu, J. Li, and N. Viswanathan. ICCAD-2015 CAD contest in incremental timing- driven placement and benchmark suite. In Proceedings of the IEEF/ACM International Confer- ence Computer-Aided Design, pages 921-92911222 Austin, TX, 2015.

---

[15] Y. Lai, J. Liu, Z. Tang, B. Wang. J. Hao, and P. Luo. Chipformer: Transferable chip placement via offline decision transformer. In Proceedings of the 40th International Conference on Machine Learning, pages 18346-18364, Honolulu, HA, 2023.
[16] Y. Lai, Y. Mu, and P. Luo. Maskplace: Fast chip placement via reinforced visual representation learning. In Advances in Neural Information Processing Systems 35, New Orleans, LA, 2022.
[17] Liao, Guo, Z. Guo, S. Liu, Lin, and B. Yu. Dreamplace 4.0: Timinn-driven placement with momentum-basee net weighting and lagrangian- refinement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 42(10):3374-3387, 2023.
[18] T. Lin, C. C. N. Chu, and G. Wu. POLAR 3.0: An ultrafast global placement engine. In Proceedings of the IEEE/ACM International Conference on Computer-Aided Design, pages 520-527, Austin, TX, 2015.
[19] Y. Lin, Z. Jiang, J. Gu, W. Li, S. Dhar, H. Ren, B. Khailany, and D. Z. Pan. DREAMPlace: Deep learning toolkit-enabled gpu acceleration for modern VLSI placement, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 40(4):748-761, 2020.
[20] J. Lu, P. Chen, C.-C. Chang, L. Sha, D. J.-H. Huang, C.-C. Teng, and C.-K. Cheng. ePlace: Electrostatics- based placement using fast Fourier transform and Nesterov's method. ACM Transactions Design Automation of Electronic Systems, 20(2):1-34 2015.
[21] D. MacMillen, R. Camposano, D. Hill, and T. W. Williams. An industrial view of clectronic design automation. IEÉE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 19(12):1428 1448, 2000.
[22] I. L. Markov, J. Hu, and M.-. Kim. Progress and challenges in VLSI placement research. In Proceedings of the 25th International Conference on Computer-Aided Design, pages 275-282, San Jose, CA, 2012.
[23] A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y-J. Lee, E. Johnson, O. Pathak, A. Nazi, et al. graph placement methodology for fast chip design. Nature, 594(7862):207-2122 2021.
[24] H. Murata, K. Fujiyoshi, S. Nakatake, and Y. Kajitani. VLSI module placement based on rectangle-packing by the sequence-pair. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems,
1996.
[25] G.-J. Nam, C. J. Alpert, P. Villarrubia, B. Winter, and M. Yildiz. The ISPD2005 placement contest and benchmark suite. Proceedings of the 9th International Symposium on Physical Design, pages 216-220, San Francisco, CA, 2005.
[26] Y. Pu, T. Chen, Z. He, C. Bai, H. Zheng, Y. Lin, and B. Yu. Incremacro: Incremental macro placement refinement. In Proceedings of the 2024 International Symposium on Physical Design, pages 169-176, 2024.
[27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: wrXiv:170..0633. 170.006477 2017.
[28] K. Shahookar and P. Mazumder. VLSI cell placement techniques. ACM Computing Surveys, 23(2):143-220, 1991.
[29] Y. Shi, K. Xue, L. Song, and C. Qian. Macro placement by wire-mask-guided black-box optimization. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.
[30] P. Spindler and F. M. Johannes. Fast and accurate routing demand estimation for efficient routability-driver placement. In Proceedings of the 14th Conference Design, Automation & Test in Europe, pages 1-6, Nice, France, 2007.
[31] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018
[32] M. Tang and X. Yao. A memetic algorithm for VLSI floorplanning. IEEE Transactions on Systems, Man, and Cybernetics, 37(1):22-69, 2007.
12

---

[33] D. Vashisht, H. Rampal, H. Liao, Y. Lu, D. Shanbhag, E. Fallon, and L. B. Kara. Placement in integrated circuits using cyclic reinforcement learning and simulated annealing, arXiv preprint arXiv:201 1.07577, 2020.
[34] K. Xue, X. Lin, Y. Shi, S. Kai, S. Xu, and C. Qian. Escaping local optima in global placement. arXiv preprint arXiv:2402.18311, 2024.
13

---

A Implementation Details
A.1 Benchmarks
The detailed statistics of our benchmarks are listed in Table 3
Table 3: Detailed statistics of the benchmarks.
Benchmark
#Cells
#Nets
#Pins
Idaptec1
210,904
221,142
944.053
adaptec2
254,457
266,009
1,069,482
adaptec3
450,927
466,758
1,875,039
adaptec4
494.716
515,951
1,912,420
bigblue
277,604
284,479
1,144.691
bighlue2
534,782
577,235
2,122,282
bigblue3
1,095,514
1,123,170
3,833,218
bigblue4
2,169,183
2,229,886
8,900,078
superbluel
1,209,716
1215710
3,767,494
superblue3
1,213,253
1,224.979
3,905,321
superblue4
795,645
802,513
2,497,940
superblue5
1,086,888
1,100,825
3,246,878
superblue7
1,931,639
1,933,945
6,372,094
superblue10
1,876,103
1,898,119
5,560,506
superblue16
981,559
999,902
3,013,268
superblue18
768,068
771,542
2,559,143
A.2 HPWL calculation
M
acjusted
P1.1)
3
P 3.22)
unadtudted
13
p (41))
((2.2)
M
2
143
1(2.1)
W;
adjusting
Figure 6: Illustration of chip canvas and calculation of HPWL.
HPWL (half perimeter wirelength) is important metric which measures the placement quality before routing. Intuitively, Figure 6 illustrates 2D chip canvas where M and p(ij) denote the i-th module to adjust and its j-th pin, respectively. Solid boxes in green and purple represent the bounding boxes for two distinct nets the canvas. In concretely, "Net I"(in green) connects modules M M² and 3 using wires through pins p(I.1), p(2.1) and p(3,1), while "Net 2"(in purple)
14

---

connects modules M2, M and M using wires through pins p(2.2), p(3.2) and p(4.)) As shown in Figure Ba). aa. HPWL can be computed as h wi h2 + W2-
A.3 Detailed settings of methods
We conclude some important settings of different methods. For the four compared methods (i.e.,
DREAMPlacd AutoDMI WireMask-EA EAA and MaskPlace we use their original implementa- tions.
The size of grids is 224, which is as the original implementation of MaskPlace [16]
Hyperparameters.
Table 4: Hyperparameters
Configuration
Value
Configuration
Value
Optimizer Total episode Batch size
Adam
Learning rate
2.5 x 10
1000
Epoch for update
10
64
Buffer capacity
5120
Clip e
0.2
Clip gradient norm
0.5
Reward discount 7
0.95
Mask soft coefficient
DREAMPlace evaluation number
3
Trade-off coefficient a
0.7
Grid soft coefficient
4
Neural Network architecture.
Table 5: Neural Network architecture
Block
Layer
Kernel Size
Output Shape
Conv
x 1
(224, 224, 12) (224, 224, 12) (224, 224, 1)
Local Mask Fusion
Conv
x 1
Conv
1x
ResNet-18
1000 784
Global Mask Encoder
FC
Deconv
3 x 3
(14, 14, 8)
Deconv
3 x 3
(28, 28, 4)
Global Mask Decoder
Deconv
3 3
(56, 56,
Deconv
3x3 3
(112, 112, 1)
Deconv
3 3
(224, 224,
Merge
Cony
1 1
(224, 224, 1)
Position Embedding
64
FC
64
FC for value
FC
64
FC
Device. CPU: Intel(R) Xeon(R) Gold 6430
GPU: 4 GeForee RTX 4090
http::///gihhbbb.cmmm//////aattt
httpp://ggiaaab.ccmmmmmmaabbbbb
BBD
15

---

B
Additional Experimental Results
B.1
Ablation studies
Purely comparison between placer and regulator. Here, we only change the problem formulation for purely comparing placer and regulator. We implement Vanilla-Maske Megkllaaa,,,,,,lttt, where the only difference to MaskPlace is the problem formulation, and all the other components (c.g., state and reward) are the same. The results in clearll demonstrates our motivation, highlighting the advantages of our regulator problem formulation.
on four chips of ICCAD 2015 benchmarks. The only one difference between these two methods is the problem formulation, where all the other components are the same. The best result on each chip is bolded.
Table 6: Results of MaskPlace and
MaskPlace |
Masila-MMskRegulltte Global HPWL (Ie8)
Benchmark
superbluel
7.93 0.06
5.58
0.05
superblue3
9.00 + 0.17
8.01 + 0.03
superblue4
5.28 0.03
4.31
0.01
superblue5
9.81 0.03
7.89 0.02
RegkRegu with without normalization. The results are shown in Table 7 Since global HPWL has large scale than regularity, MaskRegulate w/o normalization does not prefer to consider regularity, which is not what we expect.
Table 7: Results of MaskRegulate and MaskRegulate without normalization on the four chips of ICCAD 2015 benchmarks. The best result of each metric on each chip is bolded.
MeskRegulate
Regllegu w/o normalization Global HPWL (1e8) Regularity (1c6)
Benchmark
Global HPWL (Ie8)
Regularity (1e6)
superblue1
5.77 + 0.05
3.31 + 0.00
5.82 + 0.07
3.42 + 0.06
superblue3
7.05 + 0.03
3,54 + 0.00
6.71 + 0.03
3.59 + 0.01
superblue4
4.15 0.06
2.18 + 0.02
3.99 + 0.02
2.49 + 0.10
superblue5
6.94 + 0.00
4.23 + 0.01
7.03 + 0.04
4.26 + 0.01
Training regularity-aware RL placer from seratch. Our proposed RegularMask and regularity- based reward function can also be used to train a RI. placer from scratch. We implement MaskPlaco+RegulaMMaan and compare it with MaskPlace and MaskRegulate. The results show the advantages of the integration of regularity (between MaskPlace and MaskPlace + RegularMask) and our RL regular formulation (between MaskPlace RegularMask and MaskRegulate).
The above ablation results demonstrate the effectiveness of each component of MaskRegulate.
Table 8: Results of MaskPlace, MaskPlace RegularMask, MaskRegulate on the eight chips of ICCAD 2015 benchmarks. The best result of each metric each chip is bolded.
Benchmark
MaskPlace
MaskPlace RegularMusk
MaskRegulate
Global HPWL (1c8)
Rogularity (1e6)
Global HPWL (168)
Regularity (1c6)
Global HPWL (1c8)
Regularity (1c6)
superblue
7.93 + 0.06
4.40 0.06
7.44
0.08
3,87
0.06
5.77
0.05
3,31 = 0,00
superblue3
4.77
7,18
3,53
7,05
3.54
superblue4
5.28 1 0.03
3,22 1 0.03
4.49 0.03
2.21 0.03
4,15
0.06
2.18 0.,22
superblueS
9.81
7.52
superbluc?
9,99 - 0.05
4.36 2 0.03
8.74 = 0.09
3.07 0.01
7.90 = 0.03
3,03 = 0.00
superbluc10
10.94 0.21
4.75 +
10.90 0.30
4.46
11.23 0.38
3,90
0.44
superbluel6
5.53 0.04
3.08 4 0.06
5.72 0.03
2.94 0.06
5.35 0.05
1.85 0.02
superbluc18
0.09
2.59 0.08
2,89 0.03
1,56 0.00
0.01 2.96
1,55 0.00
16

---

B.2 Comparison with ChiPFormer
Recently, ChiPFormer incorporates an offline learning decision transformer to improve the generalizability. However, we find that even after fine-tuning for the same number of episodes as MaskRegulate, it is still challenging to achieve satisfactory results on ICCAD 2015. Due to the resource- intennivee nature of fine-tuning it, we conducted only partial set of experiments, which is why they were not included the main paper. In the future, we plan to explore training generalized Regulator based on the transformer and pre-train it on some chips from ICCAD 2015, and compare it with ChiPFormer that pre-train on the same training chips.
Table 9: Results of Regklegu and ChiPFormer on eight chips of ICCAD 2015 benchmarks. The best result of each metric on each chip is bolded.
Regklegulate
ChiPFormer
Benchmark
Global HPWL (le8)
Regularity (le6)
Global HPWL (1e8)
Regularity (1e6)
superblue1
5.77 + 0.05
3.31 0.00
8.09 + 0.00
4.84 + 0.00
superblue3
7.05 + 0.03
3.54 + 0.00
9.22 + 0.02
5.13 = 0.00
superblue4
4.15 + 0.06
2.18 + 0.02
5.11 + 0.00
3.66 + 0.00
superblue5
6.94 + 0.00
4.23 + 0.01
10.97 + 0.14
5.62 = 0.01
superblue7
7.90 + 0.03
3.03 = 0.00
16.81 0.09
+
superblue10
11.23 + 0.38
3.90 0.44
14.15
5.18 0.00
superblue16
5.35 + 0.05
1.85 - 0.02
5,88 0.00
3.67 1 0.00
superblue18
2.96 = 0.01
1.55 0.00
3.57 0.00
3.18 0.00
B.3 Experiments on ISPD 2005
We test the generalization the ISPD 2005 benchmark [25] by directly using the pre-trained models on superblue 1, 3, 4, and (i.e., the same models Table 2 of MaskPlace and MaskRegulate to place and regulate the eight chips. As shown in Table TO MeskRegulate still outperforms MaskPlace in most cases, demonstrating our superior generalization ability and robustness.
Table 10: Generalization results of proxy metrics on eight chips of ISPD 2005 benchmarks. The best result of each metric on each chip is bolded.
MaskPlace
MaskRegulate
Benchmark
Global HPWL (1e7)
Regularity (Ie3)
Global HPWL (1e7)
Regularity (1e3)
adaptecl
10.58 + 0.07
4.68 + 0.08
7.75 + 0.12
3.33 + 0.00
adaptec2
13.91 = 0.22
5.35 = 0.03
9.53 0.09
4.74 + 0.00
adaptec3
23.62 = 0.25
10.04 + 0.03
20.40 + 0.14
7.64 = 0.01
adaptec4
23.92 + 0.16
11.29 0.05
24.45 + 0.06
8.60 + 0.02
bigbluel
10.78 + 0.01
5.20 0.04
9.34 + 0.03
2.75 + 0.01
bighlue2
34.31 + 0.31
8.92 + 0.06
24.32 0.46
9.76 + 0.01
bigblue3
51.53 = 0.63
10.32 + 0.24
36.97 - 0.35
8.08 - 0.03
bigbluc4
134.97 + 2.45
17.32 + 0.20
93.96 = 0.35
12.38 - 0.03
B.4 Experiments on fine-tuning existing placement results
To investigate whether MaskRegulate can be used adjust any initial macro placement solution, we conduct additional experiments to demonstrate this capability. We used the pre-trained model superblue 4, and 5 (i.e., the same models in Tables and to adjust different placement results obtained by MaskPlace, AutoDMP, and WireMask-FAA The results are shown Table [T] MaskRegulate consistently improves regularity on all four unseen chips and enhances global HPWL on three chips.
17

---

Table 11: Results of proxy metrics four chips of ICCAD 2015 benchmarks. We policy trained on superbluel, superblue3, superblue4 and superblue5 to finetune the placements gained from MaskPlace, AutoDMP and WireMask-BBB on superblue7, superblue 10, superblue 16 and superblue18 88 The left column indicates the Global HPWL (Ie8) while the right column indicates the regularity (1c6). The best result of each metric on each chip bolded.
Method
superblue7
superblue 10
superblue16
superblue
MaskPlace
9.92
4.32
10.55
4.87
5.46
3,02
3.40
2.57
MaskRegulate + MaskPlace
8.53
3.05
11.20
3.25
5.27
1.89
2.88
1.56
AutoDMP ReskRegu + AutoDMP
10.68
3.96
11.57
3.73
5.87
2.56
3.02
2.29
8.28
3.02
11.39
3.25
5.11
1.89
2.94
1.56
WireMask-EA
9.53
4.33
10.99
4.86
5.91
3.10
3.37
2.60
MaskRegulate WireMask-EA
8.54
3.02
12.02
3.25
5.20
1.85
3.01
1.54
18

---

C
NeurIPS paper checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the last sentence of abstract and last paragraph of introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See the last paragraph of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
There is no theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided code in the supplemental file.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We have provided our code in the supplemental file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
httpi//gguidees/oeeubbbbbib cessinnnoiccyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guuii wwwiii/gguie///eeeeeeoecc CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided experimental details in Section 4.1 and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported the error bars in our experiments. Please see Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This information is provided in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///berrrps..c///bll///etta
Answer: [Yes]
Justification:
We have read the NeurIPS Code of Ethics and follow it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The chip is the core productivity of modern society. Our method proposes more efficient way of using reinforcement learning for macro placement of chips, which has the potential to enhance the quality of chip design.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper and used dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have provided code and models in supplymental file.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "KHk5kXcR49", "reviewer": "Reviewer_ChiY", "summary": "This paper presents an application of the RL algorithm in chip design optimization. They formulate the chip design process as an MDP process to make decisions based on the current state of the chip macro arrangement so that they can apply PPO to optimize the policies. Compared with starting from scratch, they focus on the fine-tuning stage of the macros for denser reward signals. Their experiments in a public benchmark, the ICCAD 2015 benchmark show a great improvement in the chip design metrics.", "questions": "[1] When you discretize the map, it becomes a combinatorial problem. Did you compare the sample complexity of using RL vs using some combinatorial solver?\n[2] PPO can handle continuous action space, is there a possibility of improvement using continuous action space?\n[3] What will be the future directions for this work or is this problem solved by RL?", "limitations": "The authors have discussed the limitations of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "According to the related work section, although there have been RL methods in this benchmark, this method improves previous methods by starting from macro placements instead of starting from scratch, which is a heuristic way of optimizing and simplifying the problem. This heuristic should be novel when combined with this class of methods.\n\nThe paper is technically solid in the problem formulation, experiments are sufficient with baselines, and reasonable ablation studies, and the results look good.\n\nI like the visualization part of the paper.", "weaknesses": "Compared with MaskPlace [1], the contributions of this paper lie in the expertise or experiences in the expert chip design area instead of the machine learning area, as MaskPlace has already used PPO. One thing I am concerned about is that NeurIPS may not be a good venue to discuss this contribution. \n\n[1] Lai, Y., Mu, Y., & Luo, P. (2022). Maskplace: Fast chip placement via reinforced visual representation learning. Advances in Neural Information Processing Systems, 35, 24019-24030.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below please find our response.\n\n### Q1 Compared with MaskPlace, the contributions of this paper lie in the expertise or experiences in the expert chip design area .... One thing I am concerned about is that NeurIPS may not be a good venue to discuss this contribution.\n\nThanks for your comments. However, we may not agree with you respectfully.  Compared with MaskPlace, the main contributions of our work are two folds:\n\n1. Problem formulation: MaskRegulate shifts the focus of RL from placing macros from scratch to refining existing macro placements, which allows for more effective learning from structured state and accurate reward information, significantly enhancing the learning efficiency.\n\n2. Integration of regularity: We introduce regularity, a critical yet overlooked metric in chip design, into the RL training framework, which not only aligns with industry practice but also enhances the chip PPA quality.\n\nWe have conducted comprehensive experiments to show the superior performance of MaskRegulate:\n\n- Our main experiments on the popular ICCAD 2015 benchmark have shown the significant improvements of MaskRegulate in PPA metrics, demonstrating the practical applicability and effectiveness of the RL regulator.\n\n- For the problem formulation, Table 6 in the paper has shown that Vanilla-MaskRegulate is better than MaskPlace. The only difference between them is the problem formulation, and all the other components (e.g., state and reward) are the same. Thus, the results in Table 6 clearly demonstrates our motivation, highlighting the advantages of our regulator problem formulation.\n\n- For the integration of regularity, Table 8 in the paper and Table R3 in the PDF file show that our proposed RegularMask can be integrated into other methods such as MaskPlace and WireMask-EA. \n\nNext, we will explain why NeurIPS is a good venue for our work. Recently, AI for chip design significantly expands the application domain of current AI technologies and enhance their impact [1]. In recent years, researchers have discovered that RL can assist chip design, as demonstrated in Google's Nature paper [2]. Since then, numerous top-tier venues (e.g., NeurIPS, ICML, and ICLR) have emerged with related research, aiming to further improve its effectiveness from different perspectives [3-9], just to name a few. Our work not only proposes a new placement paradigm but also introduces practical experience from the chip design field to guide the algorithm, which opens new possibilities for the application of RL in chip design. Therefore, we believe that a top-tier AI conference like NeurIPS is a proper venue for our work. Thank you.\n\n[1] Machine learning for electronic design automation: A survey. TODAES, 2021\n\n[2] A graph placement methodology for fast chip design. Nature, 2021.\n\n[3] On joint learning for solving placement and routing in chip design. NeurIPS, 2021.\n\n[4] MaskPlace: Fast chip placement via reinforced visual representation learning. NeurIPS, 2022.\n\n[5] Chipformer: Transferable chip placement via offline decision transformer. ICML, 2023.\n\n[6] Macro placement by wire-mask-guided black-box optimization. NeurIPS, 2023.\n\n[7] CircuitNet 2.0: An advanced dataset for promoting machine learning innovations in realistic chip design environment. ICLR, 2024.\n\n[8] Reinforcement learning within tree search for fast macro placement. ICML, 2024.\n\n[9] A hierarchical adaptive multi-task reinforcement learning framework for multiplier circuit design. ICML, 2024.\n\n### Q2 Discussion about combinatorial solver. \n\nGood points! Thanks for your insightful comments.\nPlacement is a very large-scale non-linear optimization problem. Using combinatorial solvers for placement has a long history [1-2], which belongs to analytical placement approach. They were initially used for placement of small-scale chips, but have gradually been replaced by more effective gradient-based approaches [3-4] in recent years. Currently, combinatorial solvers are typically used for small-scale and constrained problems, such as floorplanning [5].\n\nThe state-of-the-art analytical placement method is DREAMPlace [4], which is an important baseline in our work. Compared to DREAMPlace, our RL regulator has many advantages, as demonstrated in our paper. According to your comments, we will introduce the history of analytical placers in detail to provide a comprehensive understanding of related techniques for readers without the placement background. Thank you for bringing our attention to this point.\n\n[1] Analytical approaches to the combinatorial optimization in linear placement problems. TCAD, 1989.\n\n[2] BonnPlace: Placement of leading-edge chips by advanced combinatorial algorithms. TCAD, 2008.\n\n[3] Replace: Advancing solution quality and routability validation in global placement. TCAD, 2018.\n\n[4] DREAMPlace: Deep learning toolkit-enabled GPU acceleration for modern VLSI placement. TCAD, 2021.\n\n[5] Global floorplanning via semidefinite programming. DAC, 2023.\n\n### Q3 PPO can handle continuous action space, is there a possibility of improvement using continuous action space.\n\nThanks for your suggestions. Due to the large action space of placement, previous studies propose to discretize the layout into grids, thus reducing the action space and improving learning efficiency. We agree that learning in the continuous space can enhance decision flexibility, but this also brings challenges such as low convergence rate and under-exploration. We will consider it in the future. Thank you.\n\n### Q4 Future directions of RL\n\nOur main future work includes two aspects:\n1. Further improving the generalization of the RL policy by using better architectures and training methods.\n2. Utilizing some MO techniques (e.g., MORL) to obtain a set of Pareto-optimal placement results with different preferences across multiple objectives.\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cy1FFyatQ8", "reviewer": "Reviewer_NxAR", "summary": "This paper introduces a Macro Regulator that uses reinforcement learning (RL) to optimize existing macro placements. It reconsiders the application of RL in macro placement and incorporates a regularity metric into the reward function. The paper presents a comprehensive set of comparative experiments to ultimately demonstrate the superiority of the Macro Regulator. Ablation studies also confirm the effectiveness of each component in the proposed approach.\n\nThe contributions of the paper are specified as follows: Firstly, Macro Regulator shifts the focus of RL from placing macros from scratch to refining existing macro placements. This approach enables the RL policy to utilize more state information and achieve more accurate reward signals, enhancing learning efficiency and final PPA quality. Secondly, the paper introduces the concept of regularity into the RL framework, a crucial metric often overlooked in chip design. By incorporating regularity as part of the input information and reward signal, Macro Regulator ensures consistency in manufacturing and performance. Thirdly, the proposed method is evaluated on the ICCAD 2015 benchmark, demonstrating superior performance in terms of global placement half-perimeter wire length (HPWL) and regularity compared to various competing methods. Additionally, PPA metrics are tested using the commercial EDA tool Cadence Innovus, showing significant improvements.", "questions": "In the abstract, the author states, \"Our RL regulator can fine-tune placements from any method and enhance their quality.\" However, in the experimental section, it is mentioned, \"For each chip, MaskRegulate uses DREAMPlace to obtain an initial macro placement result to be adjusted, which takes within a few minutes and has relatively low quality.\" There are two issues with this:\n1. The claim of \"from any method\" is not entirely supported as DREAMPlace is essentially an analytical method, thus the experiments can only prove the ability to improve macro placement solutions from analytical methods, not from any other method.\n2. The assertion of \"relatively low quality\" raises the question of whether the MaskRegulate method relies heavily on the quality of initial macro placement result. It would be worthwhile to explore the effect of optimizing from different quality macro placement solutions (such as higher but not the best macro placement result), rather than starting from relatively low-quality ones.\n\nIn summary, can the MaskRegulate method improve the quality of \"any method\" and \"any quality of initial macro placement solution\"?", "limitations": "Yes.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "In summary, this paper makes a substantial contribution to the field of chip design through its novel problem formulation and high-quality experimental methodology. The originality of integrating regularity into the RL framework and the thoroughness of the experimental validation are particularly noteworthy. This work advances the state of the art in macro placement.\n1. Originality\n- Novel Problem Definition: The paper introduces a new problem formulation by shifting the focus from using RL to place from scratch to refining existing macro placements. This innovative approach allows for more efficient use of state information and more precise reward signals.\n- Integration of Regularity: The inclusion of regularity as a key metric in the RL framework is a novel contribution. Regularity is crucial for ensuring manufacturing consistency and performance but is often overlooked in existing methods.\n2. Quality\n- Robust Methodology: The paper employs a comprehensive experimental methodology, utilizing the ICCAD 2015 benchmark and commercial EDA tools (Cadence Innovus) for validation. The experiments are well-designed and include comparisons with multiple existing methods.\n- Detailed Analysis: The paper includes a comprehensive analysis of the proposed method, along with ablation studies that validate the effectiveness of different components. The detailed evaluation and the provided code help to enhance the credibility of the results.\n3. Significance\n- Impact on Chip Design: The proposed Macro Regulator addresses significant limitations of existing RL methods in macro placement, such as training time, reward sparsity, and generalization capability. By improving placement quality and PPA metrics, this work has the potential to significantly impact the efficiency and effectiveness of chip design processes.", "weaknesses": "While the paper presents a significant advancement in the field of macro placement using reinforcement learning, there are areas for improvement. Expanding the benchmark scope, including real-world case studies, demonstrating generalization capabilities, and providing more detailed algorithmic descriptions would enhance the overall impact and robustness of the work. Addressing these weaknesses would make the paper's contributions even more compelling and practical for real-world applications.\n1. Limited Benchmark Scope\n- Additional Datasets Needed: While the paper utilizes the ICCAD 2015 benchmark and Cadence Innovus for validation, the scope of the benchmarks is somewhat limited. Including additional datasets or real-world applications could strengthen the evaluation and demonstrate the generalizability of the proposed method. For example, benchmarks from other well-known contests or industrial examples with different characteristics could provide a more comprehensive assessment.\n2. Generalization to Different Chip Designs\n- Generalization Capability: While the paper mentions the generalization capabilities of Macro Regulator, more detailed experiments and analysis on different types of chip designs (e.g., various sizes, complexities, and design constraints) would strengthen the claims. Demonstrating the method's adaptability to a broader range of scenarios would enhance its perceived robustness and applicability.\n3. Clarity of Method Description\n- Detailed Algorithmic Steps: While the paper is generally clear, some parts of the methodology could benefit from more detailed descriptions. For example, providing pseudo-code or more granular steps of the Macro Regulator algorithm would help readers better understand the implementation details and reproduce the results. Additionally, regularity is an important metric, but the paper provides limited explanation of this metric. It would be beneficial to supplement more information about its definition and calculation. Furthermore, a more detailed explanation of essential factors such as the states and actions in reinforcement learning would enhance the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and valuable comments. Below please find our response. \n\n### Q1 Limited benchmark scope.\n\nThanks for your valuable comments. In our work, we use the ICCAD 2015 contest  as our benchmark, which is currently one of the largest open-source benchmarks that allows us to evaluate congestion, timing and other PPA metrics. To the best of our knowledge, ICCAD 2015 is the benchmark that closely reflects the current practices in the EDA industry. We agree with you that adding benchmark scopes would further strengthen our work. Thus, we add experiments on ISPD 2005 contest\nbenchmark, which is also a popular benchmark in AI for chip design but does not have sufficient information for PPA evaluation. The results can be found in Table R1 in our PDF file.  For detailed settings and discussions, please refer to the following Q2. Thank you.\n\n### Q2 Generalization ability\n\nThank you for your valuable comments. In our paper, we have tested MaskRegulate's generalization ability in Table 2. We have now further tested the generalization on the ISPD 2005 benchmark by directly using the pre-trained models on superblue 1, 3, 4, and 5 (i.e., the same models in Table 2) of MaskPlace and MaskRegulate to place and regulate the eight chips. As shown in Table R1, MaskRegulate still outperforms MaskPlace in most cases, demonstrating our superior generalization ability and robustness. We will include this experiment in our revised paper. Thank you very much.\n\n### Q3 Clarity of method description.\nThank you for your suggestions. However, we cannot update the paper now due to the rules of rebuttal phase. We will carefully revise our paper according to your suggestions, including adding pseudo-code and introducing regularity and MDP in detail. Thank you very much.\n\n### Q4 Can MaskRegulate improve the quality of any method and any quality of initial macro placement solution?\n\nThank you for your valuable comments. Yes, MaskRegulate can be used to adjust any initial macro placement solution. According to your suggestions, we have conducted additional experiments to demonstrate this capability. We used the pre-trained model on superblue 1, 3, 4, and 5 (i.e., the same models in Table 2 and R1) to adjust different placement results obtained by MaskPlace, AutoDMP, and WireMask-EA. The results are shown in Table R2. MaskRegulate consistently improves regularity on all four unseen chips and enhances global HPWL on three chips. We will include this experiment in our revised paper. Thank you very much.\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qtvLEr2j3v", "reviewer": "Reviewer_KS3m", "summary": "This paper proposes a novel approach called MaskRegulate for the refinement stage of macro placement, using reinforcement learning (RL) methods. Specifically, it trains an RL policy to adjust the existing placement layouts, from which the policy can receive sufficient information, instead of placing from scratch. Regularity is also considered during the learning process, helping to enhance the placement quality. Experiments demonstrate that it outperforms previous competitive approaches, achieving significant half-perimeter wirelength (HPWL), power, performance, and area (PPA) improvements.", "questions": "1.\tCan the regular mask be used in [2] WireMask-BBO?\n2.\tWhy does GP HPWL improve since only MP HPWL is considered in the learning process? \n\n[2] Y. Shi, K. Xue, L. Song, and C. Qian. Macro placement by wire-mask-guided black-box optimization. In Advances in Neural Information Processing Systems 36, New Orleans, LA, 2023.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper first explores RL method as a regulator instead of a placer, providing a more\neffective and efficient approach to optimizing chip design.\n2.\tRegular mask is employed to guide the learning process, which greatly improves the regularity of the chip layouts.\n3.\tPPA metrics are considered for a comprehensive analysis, showing the practical applicability and effectiveness of MaskRegulate.\n4.\tThe experiments are sufficient and solid, showing the great refinement and generalization ability of MaskRegulate.", "weaknesses": "1.\tRegularity is also proposed in other works such as [1]. This paper does not discuss how it differs from them.\n2.\tThe paper lacks some discussion on the necessity of using RL other than other methods as the regulator.\n\n[1] A. Vidal-Obiols, J. Cortadella, J. Petit, M. Galceran-Oms, and F. Martorell. Rtl-aware dataflow-driven macro placement. In 2019 Design, Automation Test in Europe Conference Exhibition (DATE), 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and positive comments. Below please find our response.\n\n### Q1 Discussion of regularity in other paper\n\nThanks for your valuable comments. \n\n[1] proposes a multi-level approach for macro placement that can leverage the hierarchy tree and effectively explore structural information at the RTL stage. The hierarchical partitions facilitate the integration of information such as dataflow, wirelength, and regularity. Then, [2] attempts to \"mimic\" the interaction between the RTL designer and the physical design engineer to produce human-quality macro placement results by exploiting the logical hierarchy as well as regularity and connectivity of macros.\n\nHowever, no studies have integrated regularity into RL. In our paper, we not only propose a new RL regulator framework but also integrate regularity into it. Additionally, our proposed RegularMask can be used to improve other methods, such as MaskPlace (as shown in Table 8 of the paper) and WireMask-BBO (as shown in the following answers to Q2). We will add this discussion to our revised paper. Thank you very much.\n\n[1] RTL-aware dataflow-driven macro placement. DATE, 2019\n\n[2] RTL-MP: Toward practical, human-quality chip planning and macro placement. ISPD, 2022.\n\n\n### Q2 Discussion of using RL as regulator. Can the regular mask be used in WireMask-BBO?\n\nThank you for your valuable comments. Compared to RL, BBO relies on iterative evaluation and cannot obtain a generalizable policy capable of rapid adjustment through forward inference alone. We believe other methods (such as BBO) can also utilize our proposed RegularMask. To investigate whether WireMask-BBO can benefit from incorporating RegularMask, we conducted additional experiments on the ICCAD 2015 benchmark to compare WireMask-EA and WireMask-EA + RegularMask. WireMask-EA + RegularMask comprehensively places and adjusts based on both WireMask and RegularMask. As shown in Table R3, WireMask-EA + RegularMask improves both global HPWL and regularity in all the cases, demonstrating the versatility of our proposed RegularMask. We will add this discussion and experiment to our revised paper. Thank you.\n\n\n### Q3  Why does GP HPWL improve since only MP HPWL is considered in the learning process?\n\nMacros (i.e., individual building blocks such as memories) have much larger sizes than standard cells (i.e., smaller basic components like logic gates), which significantly impacts the overall quality of the final chip. A common practice in placement is to first fix macro locations and then place standard cells [1-3]. After determining the positions of macros by optimizing MP HPWL, placing the remaining standard cells typically yields good final results, i.e., GP HPWL. However, to consistently improve GP HPWL, it's necessary to consider the sparse signal in the GP stage, which is one of our future areas of work as we stated in our paper. We will add this discussion to our revised paper. Thank you very much.\n\n[1] MaskPlace: Fast chip placement via reinforced visual representation learning. NeurIPS, 2022.\n\n[2] AutoDMP: Automated DREAMPlace-based macro placement. ISPD, 2023.\n\n[3] Macro placement by wire-mask-guided black-box optimization. NeurIPS, 2023.\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wV6adJhYQ7", "reviewer": "Reviewer_fJAy", "summary": "This paper utilizes an online RL algorithm to adjust the existing placement layouts, instead of placing the blocks on the scratch. Additionally, the paper introduces one heuristic concept regularity, which better regularity results in higher PPA performance. Besides, this paper tests the PPA performance using commercial software, showing their methods can achieve significant PPA improvements. Experiments show the proposed method improves PPA performance and proxy in the ICCAD 2015 benchmark compared to the state-of-the-art placement works.", "questions": "Please see the pros and cons part.", "limitations": "The regularity is one heuristic knowledge summarized by experienced engineers. As the introduction in section 3.2 “Why does regularity matters?”, the engineer prefers to place macros toward the peripheral regions of the chip. \n\nIs it possible that there exists some other unknown heuristics, like rectangular shape, or triangle shape that also impact a good performance? \n\nIt may be the open question that applying the existing human heuristic knowledge constraints the agent explores the whole state-action design space.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "1.\tThe paper introduces the new metric regularity, which is the heuristic knowledge by experienced engineers. Compared to the proxy metrics commonly used in previous work, it is more reflected in the PPA performance which is a more important metric considered by the industry.\n2.\tThe paper uses the commercial EDA tool to evaluate the efficiency of their methods. The results of the EDA tool analysis are convincing, and the performance is superior to the previous methods.", "weaknesses": "Several technical details need to be explained.\n\n1.\tThe most concerning part is the new MDP. The action space is not well described. Compared to MaskPlace, MaskRegulate places the blocks based on the full placement results. So, if the current placing block is placed in an already occupied position, would the occupied block be switched by the current block, or the occupied block be plugged off the board? Compared to placing the blocks one by one, the termination function is clear that the episode would end once all blocks are placed. In this paper, the termination function is not clear since each step is the full placement with all blocks on the board. What is the termination function in this MDP? \n2.\tWireMask feature map calculation is also ambiguous in this paper. Since the WireMask feature map is proposed in the MDP that places blocks one by one, it describes the amounts of gains in HPWL when placing the current block on each grid. But, in your MDP, the block seemed to be switched (related to the first question). How do you calculate the WireMask in your case?\n3.\tIn the generalization part, for the unseen chips, is the model training part frozen to do forward inferences only?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive comments. Below please find our response.\n\n### Q1 Technical details are not clear.\n\nThank you for carefully reading our paper and providing these detailed and valuable comments. We are sorry for the unclear presentations. \n\n**Q1-1 Termination function.**\n\nMaskRegulate sequentially adjusts all the macros in a certain order. All macros will be sorted according to certain rules and then adjusted one by one, as we stated in line 146 of the main paper. The MDP will terminate when all macros have been adjusted.\n\n**Q1-2 Would the occupied block be switched by the current block?**\n\nNo. When MaskRegulate is adjusting a macro, only the macros that have been adjusted (i.e., macros that appear earlier in the adjustment sequence) are considered as occupied, as illustrated in Figure 3(b) of the paper. The macro to be adjusted can temporarily occupy the position of unadjusted macros, and the HPWL is calculated in their respective nets.\n\nWe will revise these parts to make them clear. Thank you very much.\n\n### Q2 Calculation of WireMask\nThe WireMask is calculated by all the other macros' current locations, whether they are adjusted or not, as illustrated in Figure 5 of our paper.\n\n### Q3 Model inference in the generalization part.\nYes. We frozen the pre-trained model and only do forward inference in the generalization experiments. We will make this clear in our final revision. Thank you. \n\n### Q4 Integrate unknown heuristics\n\nThank you for your insightful comment and valuable suggestion. We would like to address your point as follows:\n\nIn the chip design domain, heuristic knowledge plays a crucial role, and state-of-the-art chips still heavily rely on expert knowledge. We acknowledge that there may exist other unknown heuristics that could potentially impact the performance of the placement process. Incorporating these additional heuristics is an important direction for future research. In our upcoming work, we plan to explore techniques like reward modeling to effectively capture and integrate a wider range of expert knowledge to guide the training of our RL regulator.\n\nThis work represents an initial attempt to incorporate advanced heuristic knowledge, specifically the idea that macros should be placed near the periphery of the chip, into the current practice of RL-based placement. Our results demonstrate that this approach yields significant improvements in placement quality. We believe that this work opens a door to enhance RL-based approaches by incorporating heuristic knowledge to make them more suitable for real-world placement tasks.\n\nWe greatly appreciate your suggestion and will include a discussion of these potential future directions in the revised version of our manuscript. Your feedback has been invaluable in helping us identify areas where we can clarify and expand upon our contributions.\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RxfoNc9ofB", "reviewer": "Reviewer_XyiM", "summary": "The paper proposes MaskRegulate, a method that utilizes reinforcement learning in the refinement stage for time-efficient training and accurate reward generation. Additionally, the authors introduce a regularity metric that can be used alongside HPWL in chip placement. To validate their proposed method, the authors evaluated the PPA performance, a real-world metric and compared with competitive baselines.", "questions": "1. How does the model determine adjusted and unadjusted macros? Are the macros adjusted sequentially? If so, how is the order of sequence determined?\n2. What happens if the model encounters a situation where no feasible space is available? Does it re-adjust previously adjusted macros?\n3. In section 4.4, you mentioned using alpha=0.7 for experiments. Do these experiments refer to the results in Table 1 and Table 2? If so, please explicitly state this, as Figure 4 suggests that alpha is a very important hyperparameter.", "limitations": "- Apart from some presentation-related issues, I do not observe any significant limitations in this paper. \n- To further strengthen the quality of the work, the authors might consider including training convergence graphs for ChipFormer and MaskRegulate.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- This paper highlights that a major challenge in applying RL to chip placement is the problem formulation.\n- The proposed MaskRegulate introduces the concept of regularity, which has been overlooked in previous works, and achieves state-of-the-art performance.\n- The authors validated their proposed method by evaluating it on the ICCAD 2015 benchmark and comparing it with competitive baselines. Furthermore, they assessed the real-world metric, PPA, using a commercial EDA tool to ensure the method's practicality in industry applications.\n- The visualization results in Figure 6 are impressive and effectively demonstrate the effectiveness of the proposed method. If possible, I recommend including some of these visualizations from Figure 6 in the main paper.", "weaknesses": "The overall presentation of the paper requires improvement, particularly in the clarity of labels and explanations for figures. Figure 3(a) is especially problematic, as the labels 'adjusting' and 'adjusted' are ambiguous and do not clearly indicate which blocks they refer to. To avoid potential misinterpretation, it is crucial to enhance both the figure and its accompanying explanation. Addressing these issues will significantly improve the paper's clarity and prevent possible misunderstandings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive comments. Below please find our response.\n\n### Q1  Figure 3(a) is not clear\nWe are sorry for the unclear figures. \nThe red macro $M^1$ denotes the macro that has been adjusted; the yellow macro $M^2$ is the current macro that is adjusting; the blue macros $M^3$ and $M^4$ are the macros that are unadjusted and will be adjusted in the future steps. We will revise this figure and accompany explanation in the caption to make it clear. Thank you for bringing this to our attention.\n\n### Q2 How does the model determine adjusted and unadjusted macros? Are the macros adjusted sequentially? \nYes, the macros are adjusted sequentially.\nAs we stated in lines 145-147 of the paper, the order of the macro sequence is determined by the corresponding net and size of a macro, and the number of connected modules that have been adjusted, which is the same as in previous studies [1-2]. We will revise to make this clear. Thank you.\n\n[1] MaskPlace: Fast Chip Placement via Reinforced Visual Representation Learning. NeurIPS, 2022.\n\n[2] Macro Placement by Wire-mask-guided Black-box Optimization. NeurIPS, 2023.\n\n\n### Q3 No feasible space is available? \n\nWe sincerely appreciate the reviewer's insightful question regarding the model's behavior in situations where no feasible space is available. In fact, throughout our entire training and test process, we did not encounter any instances where the final adjusted solution had overlaps. This can be due to the following three reasons:\n\n1. Prioritized adjustment sequence. We employ a specific adjustment sequence that prioritizes elements with greater impact on the layout, based on many aspects, as we answered in Q2. This will mitigate the risk of overlap.\n2. Position mask: MaskRegulate uses a position mask that effectively selects non-overlapping positions, fundamentally reducing the risk of overlapping occurrences. \n3. Integration of regularity. MaskRegulate is designed to encourage elements to align with edges, which not only enhances the overall layout but also further reduces the probability of overlapping.\n\nGiven these strategies, we have not encountered any instances of element overlap during our training and testing phases. However, we acknowledge that the possibility of overlap may exist in extreme scenarios. As you suggested, re-adjusting previously adjusted macros is a good idea for improving the model's robustness, which can lead to legal placement result by additional trials. Besides, we can introduce a negative reward to the overlapping scenario, which would penalize adjustments that lead to overlaps, thereby guiding the model towards better adjustments.\n\nWe are grateful to the reviewer for highlighting this potential issue. We will implement this mechanism in the future. This enhancement will further improve our model's robustness and adaptability, enabling it to handle a wider range of complex layout scenarios, including potential space constraint issues. Thank you very much!\n\n### Q4 Is $\\alpha=0.7$ in Table 1 and Table 2?\n\nYes, we set $\\alpha=0.7$ by default in our experiments, including those in Tables 1 and 2. We apologize for not explicitly stating this. We will revise to clarify this setting. Thank you for bringing this to our attention.\n\n### Further improvement 1: including some visualizations from Figure 6 in the main paper.\n\nThank you for your appreciation of our visualizations! We're glad that you found them valuable. In the light of your suggestions, we will adjust the layout and attempt to incorporate these visualizations into the revision of our paper. We believe this will enhance readability and improve the overall presentation of our research. Thanks for your constructive feedback.\n\n### Further improvement 2: adding training convergence graphs\n\nThank you for your valuable suggestion. We agree that including these figures would provide additional insights and further strengthen the quality of our work. However, due to space limitation of the response phase, we couldn't include these graphs currently. We will include them in the revision of our paper. Thank you.\n\n**We hope that our response has addressed your concerns, but if we missed anything please let us know.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are very grateful to the reviewers for carefully reviewing our paper and providing constructive comments and suggestions. We have revised the paper carefully according to the comments and suggestions, but we cannot upload the paper due to the NeurIPS rules. Our response to individual reviewers can be found in the personal replies, but we would also like to make a brief summary of revisions for your convenience.

Writing Enhancements:

- We correct typos and improve some method explanations to enhance overall readability.

Expanded Discussions:

- We incorporate a discussion on the combinatorial solver for macro placement.

- We explore the potential of using alternative approaches as macro regulators.

- We address the scenario where no feasible space is available and propose potential solutions.

Additional Experiments:

- We conduct generalization experiments using the ISPD 2005 benchmark to demonstrate the robustness of our method.

- We conduct generalization experiments to adjust various layouts obtained through different algorithms with different qualities.

- We implement and evaluate WireMask-BBO as a macro regulator using our proposed RegularMask.

- We include training convergence graphs to facilitate a comparative analysis of different RL-based approaches.

**We believe these revisions have significantly strengthened our paper and hope that your concerns have been addressed. We remain open to further feedback and are happy to provide additional clarification if needed.**

Meta-review: This work proposes MaskRegulate that leverages PPO to find new chip designs, evaluates on standard benchmarks (ICCAD 2015) and shows better performance than existing baselines. The major contribution is to start from existing design and focus on refinement, and proposes novel intermediate metric like concept of regularity to improve the performance. Overall, all reviewers agree that the work is solid with thorough experiments.
Decision: Accept (poster)