Benchmarking End-To-End Performance of AI-Based Chip Placement Algorithms

## Abstract

Abstract The increasing complexity of modern very-large-scale integration (VLSI) design highlights the significance of Electronic Design Automation (EDA) technologies.
Chip placement is a critical step in the EDA workflow, which positions chip modules on the canvas with the goal of optimizing performance, power, and area (PPA) metrics of final chip designs.
Recent advances have demonstrated the great potential of AI-based algorithms in enhancing chip placement.
However, due to the lengthy workflow of chip design, the evaluations of these algorithms often focus on intermediate surrogate metrics , which are easy to compute but frequently reveal a substantial misalignment with the end-to-end performance (i.e., the final design PPA).
To address this challenge, we introduce ChiPBench, which can effectively facilitate research in chip placement within the AI community. ChiPBench is a comprehensive benchmark specifically designed to evaluate the effectiveness of existing AI-based chip placement algorithms in improving final design PPA metrics. Specifically, we have gathered 20 20 20 20 circuits from various domains (e.g., CPU, GPU, and microcontrollers).
These designs are compiled by executing the workflow from the verilog source code, which preserves necessary physical implementation kits, enabling evaluations for the placement algorithms on their impacts on the final design PPA.
We executed six state-of-the-art AI-based chip placement algorithms on these designs and plugged the results of each single-point algorithm into the physical implementation workflow to obtain the final PPA results.
Experimental results show that even if intermediate metric of a single-point algorithm is dominant, while the final PPA results are unsatisfactory.
This suggests that the AI community should concentrate more on enhancing end-to-end performance rather than those intermediate surrogates.
We believe that our benchmark will serve as an effective evaluation framework to bridge the gap between academia and industry.

## 1 Introduction

The exponential growth in the scale of integrated circuits (ICs), in accordance with Moore’s law, has posed significant challenges to chip design (Huang et al., ; Lopera et al., ).
To handle the increasing complexity, many electronic design automation (EDA) tools have been developed to assist hardware engineers.
As shown in Figure 1, EDA tools automate various steps in the chip design workflow, including high-level synthesis, logic synthesis, physical design, testing and verification (Huang et al., ; Sánchez et al., ).

Chip placement is a critical step in the chip design workflow, which aims to position chip modules on the canvas, with the goal of optimizing the performance, power, and area (PPA) metrics of final chip designs (Cheng et al., ; Shi et al., ; Lin et al., ).
Traditionally, this is done manually by human expert designers, which costs much labor and necessitates much expert prior knowledge.
Therefore, a lot of design automation methods, especially those AI-based algorithms, have been developed to automate this process.
These methods mainly fall into two categories: optimization-based methods and reinforcement learning (RL)-based methods (Geng et al., ).
Optimization-based methods employ traditional optimization algorithms, such as simulated annealing (SA) (Vashisht et al., ) and evolutionary algorithms (EA) (Shi et al., ) to directly address the large-scale optimization problem, exploring the design space to identify near-optimal solutions.
In recent research, macro placement has been formulated as a Markov Decision Process (MDP), where the macro positions are determined sequentially (Mirhoseini et al., ).
Reinforcement learning (RL) has emerged as a promising technique for this task due to its ability to continuously improve performance based on feedback from the environment through trial and error (Cheng & Yan, ; Cheng et al., ; Lai et al., ; ).

Figure: Figure 1: Illustration of the modern chip design workflow.
Refer to caption: x1.png

However, due to the lengthy workflow of chip design, the evaluations of these algorithms often focus on intermediate surrogate metrics, which are easy to compute but frequently reveal a substantial misalignment with the end-to-end performance (i.e., the final design PPA).
On one hand, obtaining the end-to-end performance of a given chip placement solution requires a large amount of engineering efforts due to the lengthy workflow of chip design. In particular, we found that directly using the existing open-source EDA tools and chip placement datasets even usually fails to obtain the end-to-end performance. Thus, existing AI-based chip placement algorithms (Lai et al., ; Geng et al., ) train and evaluate learned models using intermediate surrogate metrics which are simple and easy to obtain.
On the other hand, as the PPA metrics are reflected by many aspects that have not been adequately considered in the previous stages, there exhibits a critical gap between the surrogate metrics and the final PPA objectives. Therefore, this gap significantly limits the use of existing AI-based placement algorithms in practical industrial scenarios.

To address this challenge, we propose ChiPBench, a comprehensive benchmark specifically designed to evaluate the effectiveness of existing AI-based chip placement algorithms in improving final design PPA metrics.
Appealing features of ChiPBench include its fully open source and reproducible characteristics, covering the entire EDA workflow from the source verilog code, and unifying the evaluation framework of AI-based chip placement methods using end-to-end performance.
Thus, ChiPBench can effectively facilitate research in chip placement within the AI community by taking the first step towards a fully reproducible unified evaluation framework using the end-to-end performance. In terms of the dataset, we have gathered $20$ circuits from various domains (e.g., CPU, GPU, and microcontrollers).
Then, these designs are compiled by executing the workflow from the verilog source code, which preserves sufficient physical implementation kits, enabling evaluations for the placement algorithms on their impacts on the final design PPA.
In terms of the evaluated algorithms, we executed six state-of-the-art AI-based chip placement algorithms on the aforementioned designs and *plugged* the results of each single-point algorithm into the physical implementation workflow to obtain the final PPA results.

Experimental results show that even if intermediate metric of a single-point algorithm is dominant, while the final PPA results are unsatisfactory. Moreover, visualization experiments demonstrate that intermediate metrics have weak correlation with the final design PPA, emphasizing the importance of developing algorithms towards optimizing the final design PPA rather than intermediate metrics. This suggests that the AI community should concentrate more on enhancing end-to-end performance rather than those intermediate metrics.
We believe that our benchmark will serve as an effective evaluation framework to bridge the gap between academia and industry.

We summarize our major contributions as follows.
(1) Our proposed ChiPBench is a reproducible and unified evaluation framework of existing AI-based chip placement algorithms using end-to-end performance with fully open source EDA tools. This can effectively facilitate research in chip placement within the AI community.
(2) We collected $20$ circuits from various domains, and construct a dataset by executing the EDA workflow from the verilog source code, preserving sufficient physical implementation kits for end-to-end performance evaluation.
(3) We evaluate six state-of-the-art AI-based chip placement algorithms, including most popular AI-based chip placement algorithms.
(4) Experiments demonstrate that intermediate metrics have weak correlation with the final design PPA, emphasizing the importance of developing algorithms towards optimizing the final design PPA rather than intermediate metrics.

## 2 Related Work

Datasets
Some well-known EDA conferences, such as ISPD and ICCAD, host contests addressing EDA challenges and offer benchmarks with processed data for researchers.
However, in the early years (e.g., ISPD2005 (Nam et al., ) and ICCAD2004 (Adya et al., )), the provided datasets used overly simplified Bookshelf formats, which are abstracted versions of the actual design kits.
Therefore, we cannot evaluate the final PPA of the placement results on those datasets.
Recently, ISPD2015 (Bustany et al., ) and ICCAD2015 (Kim et al., ) have offered benchmarks and datasets closer to real-world applications, including necessary netlist, library, and design exchange files, broadening their utility slightly.
Nevertheless, they still lack the essential information (e.g., necessary design kits) to run the open-source EDA tools such as OpenROAD (Kahng & Spyrou, ).
Beyond these conferences, some other datasets have been developed in various directions.
For example, the EPFL (Amarú et al., ) benchmarks and the larger OpenABC-D Chowdhury et al. () dataset concentrated on synthetic netlists, primarily for testing modern logic optimization tools with a focus on logic synthesis.
CircuitNet 2.0 Jiang et al. (), on the other hand, shifted the focus towards providing multi-modal data for prediction tasks, enhancing the capability for various prediction tasks through the use of diverse data modalities.
Compared with previous efforts, our proposed dataset focuses on the entire EDA workflow.
It provides complete files for each case and necessary design kits, such as timing constraints, library files, and LEF files, offering a comprehensive dataset that supports all stages of design and fosters a more integrated approach to chip design and evaluation.

Placement Algorithms
Recent advancements in AI technology within the EDA field have led to a variety of AI-based chip placement algorithms.
(1) Black-Box Optimization methods.
Simulated Annealing (Cheng et al., ) provides a probabilistic method for finding a good approximation of the global optimum.
Wire-Mask-Guided Black-Box Optimization (Shi et al., ) uses a wire-mask-guided greedy procedure to optimize macro placement efficiently.
(2) Analytical methods.
DREAMPlace (Lin et al., ) uses deep learning toolkits to achieve over a 30x speedup in placement tasks.
AutoDMP (Agnesina et al., ) leverages DREAMPlace for the concurrent placement of macros and standard cells, enhancing macro placement quality.
(3) Reinforcement Learning methods.
MaskPlace (Lai et al., ) treats chip placement as a visual representation learning problem, reducing wirelength and ensuring zero overlaps.
ChiPFormer (Lai et al., ) employs offline reinforcement learning, fine-tuning on unseen chips for better efficiency.
The evaluation of these algorithms mainly focuses on intermediate metrics.
In contrast, we utilized the end-to-end performance to evaluate six existing AI-based chip placement algorithms, encompassing a significant portion of mainstream AI-based placement algorithms.

## 3 Background on Electronic Design Automation

Electronic Design Automation (EDA) is a suite of software tools vital for designing and developing electronic systems, primarily integrated circuits (ICs).
These tools enable electrical engineers to efficiently transform innovative concepts into functional products, addressing the increasing complexity and demands of modern chip design.
EDA optimizes the entire design process from schematic capture to layout and fabrication, reducing time-to-market and enhancing design precision and sophistication.
In the chip design workflow, EDA tools support various functions: they perform simulations to verify circuit behavior, execute synthesis to convert high-level descriptions to gate-level implementations, and manage physical layouts to ensure designs can be realized in silicon.
As shown in Figure 1, the EDA design flow includes several key stages: logic synthesis  floorplanning, placement, Clock Tree Synthesis (CTS), and routing.
Below are concise descriptions of each stage, illustrating their importance in the integrated circuit development process.

Logic Synthesis transforms a high-level circuit description into an optimized gate-level netlist implementation (Berndt et al., ; Wang et al., ; ).
Floorplan involves deciding the layout of major components within an integrated circuit. During this stage, large blocks and core components are positioned in a way that balances signal integrity, power distribution, and area utilization.
Placement involves assigning specific locations to various circuit components within the core area of the chip, following the floorplanning stage. The primary objective of this stage is to strategically place the components to optimize performance metrics such as delay and power consumption while ensuring adherence to design rules (Geng et al., ).
Clock Tree Synthesis(CTS) creates a clock distribution network within an IC to minimize those clock effects, and ensure the correct timing synchronization for circuit operation.
Routing involves creating the physical paths for electrical connectivity between various components on the IC as per the netlist. This stage must handle multiple layers of the chip, avoid obstacles, manage signal integrity, and meet all electrical and timing constraints (Cheng et al., ).

Chip Placement The placement stage is crucially divided into two distinct phases: macro placement and cell placement.
(1) Macro placement is a critical very large-scale integration (VLSI) physical design problem that targets the arrangement of larger components, such as SRAMs and clock generators—often called macros. This phase significantly impacts the chip’s overall floorplan and essential design parameters like wirelength, power, and area.
(2) Following this, the standard cell placement phase addresses the arrangement of the more numerous and smaller standard cells, which serve as the fundamental building blocks of digital designs. This phase typically utilizes analytical solvers to secure an optimized configuration that not only minimizes wirelength but also enhances the electrical performance of the chip.

## 4 Dataset

### 4.1 Description of Designs

Due to the oversimplification of datasets in early years, there exists a significant gap between these datasets and real-world applications.
For instance, the usually used Bookshelf format (Nam et al., ; Adya et al., ) is overly simplified so that placement results given in such format are inapplicable for the subsequent stages to obtain a valid final design.
Some later datasets (Kim et al., ) provide the LEF/DEF and necessary files for running these stages, but the contained circuits are still limited and they still lack some information for open-source tools like OpenROAD to work. For instance, the library file lacks buffer definitions, which is necessary for the clock tree synthesis phase, and the LEF file has incomplete layer definitions, which hinders the routing phase. To address this issue, we construct a dataset with comprehensive physical implementation information across the entire flow.
Our dataset involves collecting a series of designs spanning various domains, including components such as CPUs, GPUs, network interfaces, image processing technologies, IoT devices, cryptographic units, and microcontrollers.
Additionally, the dataset features a diverse array of sizes, with cell ranges from thousands to nearly a million.
The statistics for each case is detailed in Table 1, and we defer more details to Table 4 in Appendix C.

**Table 1: Statistics of designs used in our benchmark.**
| Id | Design | #Cells | #Nets | #Macros | #Pins | #IOs | #Edges |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | CAN-Bus () | 815 | 935 | 0 | 2637 | 13 | 935 |
| 2 | iot shield () | 904 | 1006 | 0 | 2995 | 33 | 974 |
| 3 | FPGA-CAN () | 140848 | 178913 | 0 | 532024 | 4 | 176472 |
| 4 | subrisc () | 859382 | 1103295 | 0 | 3359066 | 34 | 1092653 |
| 5 | sha256 () | 10120 | 12283 | 0 | 38758 | 77 | 12176 |
| 6 | 8051 () | 13865 | 16424 | 0 | 50848 | 10 | 16174 |
| 7 | ariane133 () | 168551 | 184856 | 132 | 592261 | 495 | 183142 |
| 8 | bp () | 301030 | 333364 | 24 | 984093 | 1198 | 333364 |
| 9 | bp_be () | 50881 | 58428 | 10 | 182949 | 3029 | 58092 |
| 10 | camera interface () | 332 | 340 | 0 | 979 | 29 | 316 |
| 11 | image processing () | 2333 | 3270 | 0 | 7823 | 64 | 3170 |
| 12 | picorv () | 8851 | 10531 | 0 | 32195 | 409 | 10470 |
| 13 | serv () | 1291 | 1482 | 0 | 3915 | 306 | 1403 |
| 14 | ariane136 () | 175248 | 191081 | 136 | 609834 | 495 | 187911 |
| 15 | bp_fe () | 33206 | 36379 | 11 | 111510 | 2511 | 36203 |
| 16 | chameleon () | 61192 | 3396 | 4 | 10539 | 145 | 3285 |
| 17 | riscv32i () | 12740 | 10947 | 6 | 35370 | 135 | 9875 |
| 18 | swerv_wrapper () | 96435 | 105026 | 28 | 354652 | 1416 | 104565 |
| 19 | toygpu () | 368081 | 466513 | 0 | 1399167 | 11 | 461675 |
| 20 | mor1kx () | 104293 | 130743 | 0 | 374983 | 576 | 125979 |

### 4.2 Dataset Generation Pipeline

We use OpenROAD (Kahng & Spyrou, ), an open-source EDA tool, for generating our dataset.
OpenROAD integrates various tools, such as Yosys (Wolf, ) for logic synthesis, TritonMacroPlacer for macro placement, RePlAce (Cheng et al., ) for cell placement, TritonCTS for clock tree synthesis, and TritonRoute for detailed routing.
The choice of open-source tool allows for full reproducibility of our results and supports the promotion of the open-source community, ensuring that all generated data and methodologies are open-source.
The initial dataset generation starts with Verilog files as raw data.
OpenROAD performs logical synthesis to convert these high-level descriptions into a netlist, detailing the electrical connections among circuit components.
This netlist is then used by OpenROAD’s integrated floorplanning tool to configure the physical layout of the circuit on silicon.
The resulting design from the floorplanning stage is converted into LEF/DEF files by OpenROAD, facilitating the application of subsequent placement algorithms.
Simultaneously, we complete the EDA design flow through OpenROAD, generating data at subsequent stages, including placement, CTS, and routing.

## 5 Algorithms

AI-based chip placement algorithms can be roughly grouped into three categories: black-box optimization (BBO) methods, analytical methods (gradient-based methods), and reinforcement learning (RL) methods. Each category frames the placement task as an optimization problem but adopts distinct objectives and methodologies. We present details as follows.

### 5.1 Black-Box-Optimization (BBO) Methods

A straightforward intuition is to view the chip placement task as a black-box-optimization (BBO) problem, where the inner workings of the objective functions are inaccessible, and solutions are evaluated only based on the output metrics.

Simulated Annealing (SA) is a heuristic BBO optimization algorithm favored for its simplicity in implementation.
Specifically, the SA algorithm generates solutions by perturbing the solution space and then assessing the resulting representation.
Different methods have been developed to effectively map representations to placement solutions (Kirkpatrick et al., ; Sherwani, ; Ho et al., ; Shunmugathammal et al., ; Vashisht et al., ), such as sequence pair (Murata et al., ) and B^∗-tree (Chang et al., ).
Solutions are probabilistically accepted based on an annealing temperature to escape local optima in pursuit of a global optimum.
Due to its simplicity in implementation, the SA algorithm often serves as a strong baseline in previous studies.
In this work, we incorporate a specific SA implementation (Cheng et al., ) utilizing operations like swaps, shifts, and shuffles, and a cost function that balances wirelength, density, and congestion.

WireMask-EA (Shi et al., ) is a BBO framework recently appearing on the AI conference NeurIPS’2023.
It leverages wiremask to greedily guide the mapping from genotypes to phenotypes.
Here the concept of wiremask was first proposed by Lai et al. (), and it is a matrix representing the potential increases in HPWL when placing the next macro on the canvas.
Building on this, Shi et al. () implement different kinds of BBO algorithms, including random search (RS), evolutionary algorithm (EA), and Bayesian optimization (BO).

### 5.2 Analytical (Gradient-Based) Methods

Analytical methods formulate the optimization objective as an analytical function of module coordinates.
This formulation enables efficient solutions through techniques like quadratic programming (Kahng et al., ; Viswanathan et al., ; ; Spindler et al., ; Chen et al., ; Kim et al., ; Kim & Markov, ) and direct gradient descent (Lu et al., ; Cheng et al., ; Lin et al., ; ; Gu et al., ; Liao et al., ).
This work focuses on the gradient-based algorithms, which are by far the more mainstream algorithms.

DREAMPlace (Liao et al., ) is a GPU-accelerated framework that leverages differentiable proxies, such as approximate HPWL, as optimization objectives.
It was built upon the previous analytical placement algorithms, ePlace (Lu et al., ) and RePlAce (Cheng et al., ), yet significantly speeding up the placement process by using GPUs for acceleration.
The series of versions of DREAMPlace introduces diverse differentiable proxies to better align the PPA improvement.

AutoDMP (Agnesina et al., ) extends DREAMPlace by automating hyperparameter tuning through multi-objective Bayesian optimization.
It further speeds up the optimization process and reduces the heavy manual tuning efforts. At that time, this work showcased the promising potential of integrating GPU-accelerated algorithms with machine learning techniques for the automation of VLSI design.

### 5.3 Reinforcement Learning (RL) Methods

As VLSI systems grow in complexity, RL methods are being explored to enhance placement quality.
GraphPlace (Mirhoseini et al., ) first models macro placement as a RL problem.
Subsequently, DeepPR (Cheng & Yan, ) and PRNet (Cheng et al., ) establish a streamlined pipeline encompassing macro placement, cell placement, and routing.
However, they treat density as a soft constraint, which may violate non-overlap constraint during training.
Therefore, in this work, we mainly focus on MaskPlace and ChiPFormer, which are recent SOTA algorithms with hard non-overlapping constraints.

MaskPlace (Lai et al., ) represents the chip states as pixel-level visual inputs, including a wiremask (recording the HPWL increment for each grid), the viewmask (a global observation of the canvas), and the positionmask (to ensure non-overlapping constraint).
Furthermore, it uses dense reward to boost the sample efficiency.

ChiPFormer (Lai et al., ) represents the first offline RL method.
It is pretrained on various chips via offline RL and then fine-tuned on unseen chips for better efficiency.
As a result, the time for placement is significantly reduced.

## 6 Evaluation

### 6.1 Evaluation Metrics

#### 6.1.1 Final Design PPA Metrics

The primary goal of the entire Electronic Design Automation (EDA) workflow is to optimize the final PPA metrics.
PPA stands for performance, power, and area—three crucial dimensions used to evaluate the quality of a chip product.
These dimensions are assessed using several critical metrics, including worst negative slack (WNS), total negative slack (TNS), number of violating paths (NVP), power, and area.
Optimizing these PPA metrics has been a major focus in the industry, approached through expert-designed heuristics.
However, the challenge of PPA optimization has not been fully recognized within the AI community.
Bridging this gap and improving the incorporation of AI strategies into PPA optimization are key goals of this benchmark.

In terms of specific metrics, Worst Negative Slack (WNS) and Total Negative Slack (TNS) are essential for assessing the timing performance of a chip circuit. Slack is the discrepancy between the expected and required arrival times of a signal, with negative slack indicating a timing violation. WNS pinpoints the most severe negative slack within a circuit, thus identifying the most critical timing issue. Conversely, TNS aggregates all negative slacks, providing a comprehensive view of the circuit’s overall timing challenges. Moreover, the number of violating paths (NVP) counts the paths that fail to meet the timing constraints, further illustrating the timing performance issues.

#### 6.1.2 Intermediate Metrics

Commonly used intermediate surrogate metrics include Congestion, Wire Length (WL), Half Perimeter Wire Length (HPWL), and Macro HPWL (mHPWL).
Congestion evaluates the density of wires in different chip regions.
High congestion in certain areas can pose substantial challenges during the routing stage.
While not a direct component of the PPA metrics, managing congestion effectively is essential to ensure that the chip can be successfully manufactured.
Therefore, it is also considered as an evaluation metric in this paper.
Congestion is typically estimated after the Clock Tree Synthesis (CTS) stages but before the detailed routing stage, allowing for adjusting macro placement and routing strategies to mitigate potential issues.

Wire Length (WL) is the total length of all wires connecting all modules in a chip.
Half Perimeter Wire Length (HPWL) is the sum of half perimeters of bounding boxes that encompass all pins in each net.
It is widely used as an estimation of WL and is obtained after cell placement.
Macro HPWL (mHPWL) further simplifies HPWL by only considering the macros.
It is favored in recent studies as it can be immediately obtained after macro placement.
These metrics are thought to correlate with the final PPA, but they do not directly reflect the chip quality.

### 6.2 End-to-End Evaluation Workflow

We present an end-to-end evaluation workflow utilizing OpenROAD-flow-scripts (Kahng & Spyrou, ) for the various stages of the EDA design flow, as illustrated in Figure 2.
All tools used in this workflow are open-source, providing a significant advantage over other workflows that rely on commercial software. This workflow is designed to offer a comprehensive assessment of optimization algorithms at any stage of the design flow.

Figure: Figure 2: Illustration of our end-to-end evaluation workflow.
Refer to caption: x2.png

Our dataset comprises design kits needed for each stage of the physical design flow.
For evaluating any stage-specific algorithm, the output file from the preceding stage serves as the input for the algorithm under evaluation. The algorithm processes this input to generate its output, which is subsequently plugged into the OpenROAD design flow.
Ultimately, final performance metrics such as TNS, WNS, Area, and Power are reported, providing a comprehensive end-to-end performance assessment.This method offers a holistic set of metrics that can evaluate the optimization effects of any stage-specific algorithm on the final chip design, providing consistency of metrics and avoiding the limitations of overly simplified metrics confined to a single stage.It is particularly beneficial for the optimization and development of various algorithms, ensuring their improvements translate to practical enhancements in chip design and foster the development of more efficient and effective open-source EDA tools through a robust framework for testing and improvement.

### 6.3 Experimental Setup

We apply the aforementioned workflow to evaluate six macro placement algorithms: SA, WireMask-EA, MaskPlace, ChiPFormer, and the default algorithm in OpenROAD.
As most of these methods only support the circuit data in a BookShelf format, while the circuits in our used dataset are in a standard LEF/DEF, we start by converting the LEF/DEF files from the floorplan stage of our dataset to BookShelf format to serve as the input for the placement algorithms.
After finishing the macro placement stage, we use DREAMPlace to finish the placement of the standard cells.
In this stage it conduct two different operations, coarse placement and legalization, with the coarse placement splitting the cells simultaneously using gradient descent algorithms and the legalization stage correct the positions of cells subject to the non-overlapping constraint.
The resulting placement files in BookShelf format are then converted back to DEF format using Dreamplace and reintroduced into the original flow.
Finally, we report the final metrics, obtaining end-to-end evaluation results.
Additionally, we perform placement using OpenRoad’s native Place method and complete the entire flow to obtain the final metrics for comparison with other algorithms.

## 7 Results and Discussions

### 7.1 Main Results

We evaluate the AI-based chip placement algorithms, including SA, WireMask-EA, DREAMPlace, AutoDMP, MaskPlace, and ChiPFormer, using both intermediate metrics and end-to-end performance.
The results for macro placement are in Table 2.
ChiPFormer and WireMask-EA demonstrated a significant reduction in MacroHPWL compared to OpenROAD using TritonMacroPlacer.
WireMask-EA achieved the best performance in terms of MacroHPWL.
While these AI-based placement algorithms showed good performance on several intermediate metrics, they perform poorly in terms of the end-to-end metrics compared to OpenRoad, particularly in Power, TNS, and Area.
This outcome revealed a significant gap between the originally optimized MacroHPWL intermediate metrics and the final design PPA.

**Table 2: The evaluation results of AI-based macro placement algorithms. MacroHPWL, HPWL, Wirelength are intermediate metrics, and the other metrics evaluate the end-to-end performance.**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 0.698 | 1.115 | 1.165 | 1.170 | 1.058 | 1.092 | 1.096 | 0.865 | 1.007 |
| SA | 0.938 | 1.028 | 1.089 | 1.087 | 1.045 | 1.189 | 1.799 | 1.136 | 1.009 |
| DREAMPlace | 0.946 | 1.092 | 1.161 | 1.176 | 1.045 | 0.988 | 1.368 | 1.043 | 1.006 |
| AutoDMP | 0.762 | 0.950 | 0.981 | 0.981 | 1.021 | 0.986 | 1.089 | 0.973 | 1.005 |
| MaskPlace | 1.744 | 1.153 | 1.146 | 1.149 | 1.053 | 1.185 | 1.462 | 1.044 | 1.009 |
| ChiPFormer | 0.728 | 1.078 | 1.118 | 1.117 | 1.061 | 1.036 | 2.097 | 1.222 | 1.010 |
| OpenROAD | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |

**Table 3: The evaluation results of AI-based standard cell placement algorithms.**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |  |
| DREAMPlace | 0.990 | 1.002 | 1.013 | 0.983 | 1.227 | 4.948 | 6.135 | 1.002 |  |
| AutoDMP | 1.159 | 1.141 | 1.160 | 1.011 | 1.170 | 1.259 | 1.208 | 1.006 |  |
| OpenROAD | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |  |

Cell Placement As shown in Table 3, DREAMPlace achieved the best results in the intermediate metrics of HPWL, and performed well in terms of Power. However, OpenROAD achieved the best results in WNS, TNS, NVP, and area, further demonstrating the inconsistency between intermediate metrics and final PPA results.

### 7.2 Correlation Analysis

In this section, we conduct compute and discuss the correlation between the core optimization indicator MacroHPWL, used in existing placement algorithms, and the final chip performance metrics such as WNS, TNS, and wirelength, obtained through the OpenROAD process.

We use the Pearson correlation coefficient (Cohen et al., ) to evaluate the strength of linear correlation between pairs of metrics.
The formula for calculating the Pearson correlation coefficient takes the form of

$$ $r=\frac{\sum(X_{i}-\overline{X})(Y_{i}-\overline{Y})}{\sqrt{\sum(X_{i}- \overline{X})^{2}\sum(Y_{i}-\overline{Y})^{2}}},$ (1) $$

where $X_{i}$ and $Y_{i}$ are the observations, and $\overline{X}$ and $\overline{Y}$ are the respective means.

Figure: Figure 3: Correlation Between Metrics
Refer to caption: x3.png

The results are shown in Figure 3.
To calculate the correlation, the signs of all values are adjusted so that for all metrics the lower indicates the better.
The results show that MacroHPWL only has a weak correlation with the Wirelength, which indicates that existing algorithms that optimize MacroHPWL do not lead to an optimization on the Wirelength.
In contrast, HPWL shows a very strong positive correlation with actual Wirelength, indicating that HPWL works as an effective surrogate for approximating the Wirelength.

Figure: (a) WL-TNS/WNS based on Method
Refer to caption: x4.png

In addition to Wirelength, the final PPA metrics of the chip are associated with WNS/TNS, congestion, area, and power.
The analyses shows that the correlation between MacroHPWL and these metrics is weak, indicating that optimization of MacroHPWL has minimal impact on these performance indicators.
Moreover, the results in Figure 4 show that Wirelength exhibits weak correlations with WNS and TNS as well.
This implies that even if a single-point algorithm successfully optimizes metrics such as Wirelength, the ultimate physical implementation might only enhance one aspect of the PPA metrics and may not effectively optimize the other dimensions.
Therefore, more appropriate intermediate metrics are needed to better correlate with the actual PPA objectives.

### 7.3 Discussion

Our benchmark comprises design kits essential for each stage of the physical design flow, including netlists, libraries, rules, and constraints needed during the physical implementation stage.
This thorough inclusion allows for a convenient and detailed evaluation of algorithms at any specific stage of the physical design flow, enabling researchers and practitioners to test and compare the effectiveness of their solutions in a realistic, end-to-end environment.
We call on the AI researchers to pay more attention on the “shift-left” challenge from the real-world industrial scenarios, keeping towards the mission of bridging the huge gap between academic research and industrial applications.

## 8 Conclusion

This paper presents a comprehensive dataset that spans the entire spectrum of the EDA design process and an end-to-end evaluation method, which we used to assess several placement algorithms: DREAMPlace, AutoDMP, MaskPlace, ChipFormer, Wiremask-EA, and SA. Our evaluation revealed inconsistencies between the metrics currently emphasized by mainstream placement algorithms and the final performance outcomes. These findings highlight the need for a new perspective in the development of placement algorithms.

## Appendix A Technical details

### A.1 Experimental Details

In our workflow, it starts with OpenROAD. The file format for physical design in OpenROAD is OpenDB database (.odb), which contains LEF/DEF information. OpenROAD can be used to convert ODB to LEF/DEF, or LEF/DEF to ODB.
Initially, the ODB after macro placement in the OpenROAD flow is converted to LEF/DEF files, serving as inputs for other macro placement algorithms. Since most existing macro placement algorithms only support bookshelf format input, DREAMPlace’s code is used to convert LEF/DEF to bookshelf (.nodes, .pl, .nets) format. It is important to note that when DREAMPlace reads the DEF file, it treats the blockages in the BLOCKAGES part as virtual macros, so when using DREAMPlace’s PlaceDB in Python to read macro information, the virtual macros must be excluded.

For ChiPFormer, use the default model and perform 100 online iterations to fine-tune the macro layout.For MaskPlace, run 3000 epochs.Other settings use the defaults.

In the method of ChipFormer and MaskPlace, the experiments were run on an NVIDIA GeForce RTX 2080 Ti, taking one day for all cases.For the other algorithms, we used 32 CPUs (Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz), with a total time expenditure of two days.

Next, the macro layout (.pl) files are obtained after running ChiPFormer, MaskPlace, WireMask-EA, and SA, and then DREAMPlace is used to write the macro layout into DEF files. Since DREAMPlace does not modify the blockages in the DEF file, if blockages are defined in the DEF file, additional script modifications may be required.

After obtaining the macro-placed DEF files, they are converted to ODB using OpenROAD, followed by Tapcell and Welltie insertion, PDN generation, IO place, global place, detail place, and subsequent CTS and routing.

The performance of OpenROAD is limited; in the future, other global placement tools and detailed placement tools can be used to run the entire workflow.

### A.2 Encountered Errors

Due to certain parsing bugs in OpenROAD, when converting DEF files to ODB files, it is necessary to ensure that no object names in the DEF file contain ”/”, otherwise, there will be issues during timing tests (issues will occur when writing and reading SPEF file).

Since various algorithms are academic and cannot directly set the minimum spacing between macros (which can be set in OpenROAD), the optimization of macro positions by these algorithms may result in macros being placed too closely. When integrating back into OpenROAD, the following errors might occur:

OpenROAD’s limited capabilities in various stages also contribute to these errors. During PDN, ”Unable to repair all channels.” might be encountered.
During the global place stage, global placement might diverge.
During the detail place stage, detailed placement might fail (because the surrounding space of certain cells makes it impossible for OpenROAD to find space for adjustments).

The code of ChiPFormer has certain issues and needs modifications to be applied to other cases.
DREAMPlace directly uses the mix-size method with LEF/DEF for running macros.

For the ICCAD 2015 dataset, it includes the lib file, LEF/DEF files, netlist file, and sdc file. However, the lib file lacks buffer definitions, preventing CTS. The lef files have incomplete layer definitions, hindering routing.Additionaly, OpenRoad does not support the syntax of the ICCAD 2015 sdc file.

## Appendix B Limitation

Our dataset currently has limitations in terms of data volume. In the future, we aim to increase the dataset’s size and include more cases from various domains. This will enhance its generalizability and robustness, making it a more comprehensive and valuable resource for researchers . By expanding the dataset, we will improve its applicability across different areas of chip design and EDA, further refining and perfecting this important resource.

## Appendix C More Results

The detailed descriptions of each design are in Table 4. All the raw data from the experiment are in Tables 5-22. Methods or designs not shown in the table failed to complete the entire process due to the errors discussed earlier.

**Table 4: Detailed descriptions of our collected designs.**
| Id | Design | Description |
| --- | --- | --- |
| 1 | CAN-Bus | A CAN bus Controller |
| 2 | iot shield | IoT Shield for the Intel Galileo Development Board |
| 3 | FPGA-CAN | A lightweight CAN bus controller |
| 4 | subrisc | Simple Instruction-Set Computer for IoT edge devices |
| 5 | sha256 | Hardware implementation of the SHA-256 cryptographic hash function |
| 6 | 8051 | FPGA implementation of the 8051 Microcontroller |
| 7 | ariane133 | Ariane Core |
| 8 | black parrot | Full 64-bit RISC-V Core with Cache Coherence Directory |
| 9 | bp_be | Back-end of a 64-bit RISC-V Core with Cache Coherence Directory |
| 10 | camera interface | Camera interface |
| 11 | image processing | Image processing |
| 12 | picorv | A Size-Optimized RISC-V CPU |
| 13 | serv | An award-winning bit-serial RISC-V core |
| 14 | ariane136 | Ariane Core |
| 15 | bp_fe | Front-end of a 64-bit RISC-V Core with Cache Coherence Directory |
| 16 | chameleon | SoC core |
| 17 | riscv32i | Simple RISCV 32I core |
| 18 | swerv_wrapper | SweRV RISC-V Core 1.1 from Western Digital |
| 19 | toygpu | A simple GPU on a TinyFPGA BX |
| 20 | mor1kx | an OpenRISC processor IP core |

**Table 5: The results of ariane133**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 1124169.895 | 5027796.1 | 0.227 | 6609025 | 0.369 | -0.581 | -687.560 | 2662 | 770528 |
| SA | 1683330.105 | 5109178.9 | 0.227 | 6594317 | 0.365 | -0.812 | -1368.340 | 3253 | 764028 |
| DREAMPlace | 1111023.474 | 4835183.2 | 0.222 | 6460013 | 0.370 | -0.431 | -578.455 | 2181 | 768426 |
| AutoDMP | 828592.368 | 4198935.5 | 0.195 | 5662040 | 0.351 | -0.481 | -622.147 | 2280 | 760842 |
| MaskPlace | 4444289.000 | 6258454.3 | 0.260 | 7720348 | 0.373 | -0.847 | -1553.500 | 3277 | 769790 |
| ChiPFormer | 1253799.737 | 4996747.8 | 0.229 | 6650789 | 0.372 | -0.617 | -933.311 | 2707 | 772172 |
| OpenROAD | 2685856.053 | 5226018.8 | 0.235 | 6817628 | 0.358 | -0.492 | -609.207 | 2279 | 770622 |

**Table 6: The results of ariane136**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 1181404.421 | 5308713.2 | 0.243 | 7034760 | 0.578 | -1.688 | -4248.100 | 3546 | 791551 |
| SA | 1693986.737 | 5584824.9 | 0.252 | 7296870 | 0.574 | -1.711 | -4116.460 | 3662 | 792731 |
| DREAMPlace | 1067974.000 | 5157317.8 | 0.232 | 6885717 | 0.574 | -1.380 | -3293.090 | 3581 | 792608 |
| AutoDMP | 870515.895 | 4907336.9 | 0.229 | 6607637 | 0.569 | -1.412 | -3644.990 | 4071 | 791027 |
| MaskPlace | 4586233.421 | 6522052.1 | 0.290 | 8382804 | 0.586 | -1.513 | -3458.900 | 3608 | 796715 |
| ChiPFormer | 1269027.263 | 5078881.1 | 0.232 | 6705656 | 0.563 | -1.348 | -3266.360 | 3521 | 789667 |
| OpenROAD | 3067334.316 | 5540894.5 | 0.249 | 7176803 | 0.550 | -1.318 | -3204.870 | 3468 | 792446 |

**Table 7: The results of black parrot**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 112560.500 | 7310237.2 | 0.422 | 9637354 | 0.252 | -1.827 | -2.222 | 23 | 815781 |
| SA | 151696.447 | 6548499.6 | 0.385 | 8775996 | 0.248 | -1.896 | -165.720 | 925 | 813755 |
| AutoDMP | 135090.737 | 6338160.8 | 0.369 | 8427105 | 0.247 | -1.951 | -60.856 | 605 | 811220 |
| DREAMPlace | 161581.000 | 6798714.6 | 0.382 | 8914932 | 0.244 | -1.884 | -198.213 | 1398 | 813703 |
| MaskPlace | 324368.447 | 6808053.5 | 0.393 | 8966592 | 0.249 | -1.657 | -102.182 | 987 | 814013 |
| ChiPFormer | 138943.842 | 6893462.7 | 0.394 | 8993077 | 0.250 | -1.746 | -463.346 | 1687 | 816983 |
| OpenROAD | 119123.132 | 6242894.2 | 0.360 | 8209505 | 0.246 | -1.961 | -71.882 | 862 | 810360 |

**Table 8: The results of bpfe**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 45985.289 | 1566601.0 | 0.404 | 2206773 | 0.298 | -1.013 | -686.950 | 2263 | 234748 |
| SA | 59872.289 | 1289105.7 | 0.295 | 1633540 | 0.287 | -1.060 | -853.464 | 2459 | 237391 |
| DREAMPlace | 60280.000 | 1800046.2 | 0.498 | 2724715 | 0.301 | -1.171 | -808.973 | 2239 | 235832 |
| AutoDMP | 56107.000 | 1302271.3 | 0.293 | 1625619 | 0.292 | -1.158 | -765.161 | 2464 | 239894 |
| MaskPlace | 64024.000 | 1263243.8 | 0.285 | 1580587 | 0.285 | -1.294 | -641.347 | 1858 | 238130 |
| ChiPFormer | 45754.000 | 1282954.8 | 0.301 | 1670666 | 0.294 | -0.967 | -641.719 | 2457 | 235347 |
| OpenROAD | 53842.000 | 1167916.2 | 0.250 | 1389986 | 0.262 | -1.007 | -473.214 | 2441 | 231069 |

**Table 9: The results of bpbe**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 66729.237 | 1826669.2 | 0.331 | 2288717.000 | 0.397 | -1.829 | -2427.590 | 4816 | 262891.000 |
| SA | 85984.158 | 1939541.5 | 0.410 | 2816347.000 | 0.411 | -1.787 | -3412.900 | 5612 | 264921.000 |
| DREAMPlace | 87304.667 | 1777230.3 | 0.317 | 2219014.333 | 0.386 | -1.772 | -2294.093 | 4885 | 262545.667 |
| AutoDMP | 86263.842 | 1713273.4 | 0.307 | 2124910.000 | 0.384 | -1.819 | -2436.980 | 4909 | 261516.000 |
| MaskPlace | 114314.553 | 1909505.5 | 0.345 | 2386914.000 | 0.398 | -1.778 | -3458.640 | 4730 | 261446.000 |
| ChiPFormer | 62168.447 | 2175102.7 | 0.411 | 2830643.000 | 0.417 | -1.803 | -2611.130 | 5127 | 266110.000 |
| OpenROAD | 76561.237 | 1940820.8 | 0.337 | 2332089.000 | 0.389 | -1.827 | -2514.450 | 5034 | 259572.000 |

**Table 10: The results of swerv wrapper**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MacroHPWL $\downarrow$ | HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| WireMask-EA | 474447.237 | 4186894.0 | 0.446 | 5209646 | 0.682 | -1.127 | -1256.030 | 1668 | 679048 |
| SA | 630919.500 | 4125523.0 | 0.455 | 5313519 | 0.682 | -1.195 | -880.450 | 1789 | 678647 |
| DREAMPlace | 865544.000 | 3673896.3 | 0.375 | 4631643 | 0.666 | -0.889 | -640.107 | 1165 | 675207 |
| AutoDMP | 450662.000 | 3171618.7 | 0.352 | 4130837 | 0.632 | -0.711 | -709.121 | 1495 | 672435 |
| MaskPlace | 1270726.500 | 4414508.8 | 0.461 | 5391952 | 0.681 | -1.111 | -743.094 | 1443 | 679852 |
| ChiPFormer | 437638.237 | 4056268.8 | 0.440 | 5144218 | 0.684 | -1.079 | -893.142 | 1756 | 679744 |
| OpenROAD | 666704.000 | 3187567.0 | 0.342 | 4015115 | 0.627 | -0.978 | -751.619 | 1531 | 673169 |

**Table 11: The results of picorv32**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 88407.3 | 0.194 | 135544 | 0.048 | -0.147 | -0.521 | 29 | 18951.400 |
| AutoDMP | 132458.3 | 0.244 | 170435 | 0.050 | -0.142 | -0.525 | 37 | 19039.000 |
| OpenROAD | 87168.8 | 0.195 | 134211 | 0.048 | -0.107 | -0.661 | 38 | 18972.700 |

**Table 12: The results of FPGA-CAN**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 1472841.9 | 0.181 | 2437446 | 0.738 | -1.027 | -678.865 | 17688 | 378643 |
| AutoDMP | 1437916.7 | 0.178 | 2393640 | 0.745 | -0.892 | -108.201 | 3864 | 379675 |
| OpenROAD | 1483498.8 | 0.180 | 2394544 | 0.742 | -0.404 | -30.331 | 1843 | 379193 |

**Table 13: The results of CAN-Bus**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 3916.8 | 0.093 | 5683 | 0.005 | -0.074 | -0.074 | 1 | 1675.270 |
| AutoDMP | 4210.8 | 0.097 | 6024 | 0.005 | -0.074 | -0.074 | 1 | 1668.880 |
| OpenROAD | 3955.1 | 0.092 | 5623 | 0.005 | -0.079 | -0.079 | 1 | 1665.430 |

**Table 14: The results of serv**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 7194.0 | 0.165 | 11128 | 0.007 | -0.563 | -10.333 | 81 | 2125.870 |
| AutoDMP | 10361.9 | 0.261 | 17455 | 0.008 | -0.593 | -11.893 | 120 | 2193.170 |
| OpenROAD | 7165.0 | 0.163 | 10638 | 0.007 | -0.565 | -10.210 | 73 | 2097.940 |

**Table 15: The results of sha256**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 114376.4 | 0.203 | 170230 | 0.161 | -0.542 | -9.367 | 40 | 23476.400 |
| AutoDMP | 120832.7 | 0.206 | 173822 | 0.172 | -0.573 | -9.891 | 39 | 23398.700 |
| OpenROAD | 113841.8 | 0.202 | 168199 | 0.162 | -0.528 | -8.667 | 39 | 23426.900 |

**Table 16: The results of iot_shield**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 5014.3 | 0.118 | 7346 | 0.006 | -0.167 | -2.457 | 27 | 1712.770 |
| AutoDMP | 5976.2 | 0.137 | 8653 | 0.006 | -0.152 | -2.367 | 29 | 1730.600 |
| OpenROAD | 5067.3 | 0.119 | 7349 | 0.006 | -0.152 | -2.186 | 20 | 1719.420 |

**Table 17: The results of FPGA_image_processing**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 16556.5 | 0.123 | 23453 | 0.201 | -0.924 | -184.592 | 736 | 5797.470 |
| AutoDMP | 22203.0 | 0.160 | 30745 | 0.211 | -0.930 | -200.125 | 710 | 5812.630 |
| OpenROAD | 16689.0 | 0.120 | 22746 | 0.199 | -0.919 | -165.009 | 609 | 5758.100 |

**Table 18: The results of 8051-master**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 142162.6 | 0.201 | 210127 | 0.075 | -0.634 | -16.460 | 35 | 29484.200 |
| AutoDMP | 131174.7 | 0.186 | 195536 | 0.074 | -0.633 | -15.643 | 35 | 29365.600 |
| OpenROAD | 141245.3 | 0.201 | 207737 | 0.073 | -0.603 | -16.664 | 41 | 29461.400 |

**Table 19: The results of mor1kx**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 1723145.8 | 0.366 | 3421791 | 0.703 | -1.863 | -19564.300 | 26568 | 289570 |
| AutoDMP | 2258941.0 | 0.524 | 4893199 | 0.735 | -3.879 | -44135.900 | 26600 | 285455 |
| OpenROAD | 2006747.0 | 0.393 | 3661548 | 0.703 | -0.796 | -4928.920 | 25509 | 313065 |

**Table 20: The results of toygpu**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 4347733.8 | 0.198 | 6784639 | 1.101 | -1.503 | -131.661 | 101 | 979293 |
| OpenROAD | 4657703.6 | 0.212 | 7185236 | 1.108 | -1.294 | -103.668 | 101 | 981911 |

**Table 21: The results of subrisc**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 13920769.0 | 0.264 | 21051063 | 2.774 | -0.847 | -5742.680 | 29283 | 2341490 |
| AutoDMP | 13446106.8 | 0.238 | 19114559.0 | 2.697 | -0.67 | -150.178 | 466 | 2319150.0 |
| OpenROAD | 14386152.9 | 0.253 | 20108941 | 3.425 | -0.782 | -305.819 | 674 | 2334330.0 |

**Table 22: The results of Camera_Interface**
| Method | Intermediate Metrics | PPA Metrics |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| HPWL $\downarrow$ | Congestion $\downarrow$ | Wirelength $\downarrow$ | Power $\downarrow$ | WNS $\downarrow$ | TNS $\downarrow$ | NVP $\downarrow$ | Area $\downarrow$ |  |
| DREAMPlace | 13920769.0 | 0.264 | 21051063 | 2.774 | -0.847 | -5742.680 | 29283 | 2341490 |
| AutoDMP | 13446106.8 | 0.238 | 19114559 | 2.697 | -0.673 | -150.178 | 466 | 2319150 |
| OpenROAD | 14386152.9 | 0.253 | 20108941 | 3.425 | -0.782 | -305.819 | 674 | 2334330 |

## Appendix D License

The code and propose dataset will be publicly accessible. We include the following licenses for the raw data we used in this paper.

- •
CAN-Bus : [MIT](https://github.com/Tommydag/CAN-Bus-Controller/blob/master/LICENSE)
- •
FPGA-CAN :[GPL-3.0](https://github.com/WangXuan95/FPGA-CAN/blob/main/LICENSE)
- •
sha256:[BSD-2-Clause](https://github.com/secworks/sha256/blob/master/LICENSE)
- •
camera interface:[MIT](https://github.com/AngeloJacobo/FPGA_OV7670_Camera_Interface/blob/main/LICENSE)
- •
picorv:[ISC](https://github.com/YosysHQ/picorv32/blob/main/COPYING)
- •
serv:[ISC](https://github.com/olofk/serv/blob/main/LICENSE)
- •
mor1kx:[CERN-OHL-W](https://github.com/openrisc/mor1kx?tab=License-1-ov-file#readme)
- •
ariane133:[SOLDERPAD HARDWARE](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/ariane136/LICENSE)
- •
ariane136:[SOLDERPAD HARDWARE](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/ariane136/LICENSE)
- •
bp:[BSD-3-Clause](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/black_parrot/LICENSE)
- •
bp_be:[BSD-3-Clause](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/bp_be_top/LICENSE)
- •
bp_fe:[BSD-3-Clause](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/bp_fe_top/LICENSE)
- •
swerv_wrapper:[Apache](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/swerv/LICENSE)
- •
riscv32i:[Apache](https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/src/riscv32i/LICENSE)

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: I appreciate the authors focusing on the end-to-end performance evaluation of the placement algorithms, which is more concerned by the industry before. From the evaluation results, the weak correlation between the intermediated metrics and the final PPA results are meaningful to researchers.

However, I still have some concerns and suggestions about this work.

1. This framework heavily depends on the OpenROAD framework. I think OpenROAD has provided an easy-to-use EDA toolchain, including core algorithms, framework implementing, tcl scripts, PDK, and sample designs. This work collects more designs from different sources, including OpenROAD. Especially, the authors also cited a project called *OpenROAD flow scripts*, which includes some circuits, support synthesis, and physical design flow, and is more comprehensive than some compared datasets in Table 1. Therefore, more comparison with *OpenROAD flow scripts* could further highlight this work’s unique contributions.
2. Section 5 spends a substantial portion to introduce some placement algorithms, which are just objects to be evaluated not the essentials for the framework. The authors could re-evaluate the content of Section 5, considering integrating it into other relevant sections or removing it entirely.
3. The type of PDK is missing in the paper and this is one of the most basic information for EDA evaluation.
4. In the analysis, the authors provide some observations from the performance of different algorithms, such as how buffer insertion will affect the final TNS and WNS. However, considering the three major types of placement algorithms,  e.g., BBO, gradient-based, and RL, this paper lacks an analysis of the weak correlation and the inner behavior of placement algorithms. Maybe some new evaluation methods should be applied to this task.
soundness: 3
strengths: 1. The comprehensive summary of current dataset, placement algorithms, and EDA flow.
2. Supported by the open source tool that makes all data accessible to users.
3. The meaningful insights about the weak correlation between the intermediate metrics and the final design PPA.
summary: The authors propose a new framework, ChipBench, to evaluate the end-to-end performance of AI-based placement algorithms. They also collect a dataset including 20 circuits from various domains and implement EDA flow on these circuits. Finally, they evaluate six placement algorithms in this framework and reveal the weak correlation between the intermediate metrics and the final design PPA.
weaknesses: 1. This framework heavily depends on the OpenROAD framework but the authors don’t compare with it.
2. Section 5 is not essential for the evaluation and the technical novelty is low.
3. The type of PDK is missing in the paper.
4. The analysis merely focuses on the flow but not the inner behavior of placement algorithms.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: The pitfall of selecting poor intermediate metric is well demonstrated. But in the grand scheme of things, it is unclear how much real-world difference does it make. Take area from Table 5 as an example of the most concrete end metric. Among all the alternatives the difference between the best and worst is 3.8%. Why couldn't we declare the placement problem is largely a solved one?
soundness: 3
strengths: The framework seems complete and readily available for others to use.
The analysis using the framework is helpful for those working on new AI-based placement algorithms.
summary: This paper presents two things: a complete framework (ChiPBench) for chip placement with some benchmark circuits and a comparison of different placement algorithms using the framework. The framework is available on GitHub, allowing others to use it as well. The comparison provide some insights on how to use different metrics in AI-based placement algorithms. For instance, MacroHPWL is a commonly used metric in recent works as it is available sooner in the placement process than some other metrics. It turns out, the metric has a rather low correlation with other metric more reflecting of final performance of the placement.
weaknesses: It is unclear what exact circuit is being used for the "main" results presented in Table 3 and 4. While a reader can obtain details in the appendix for individual circuit results, Table 3 and 4's results should be clearly identified. The "standard cell" is unhelpful (and actually quite confusing: are you placing a single cell?).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * Why the surrogate metrics failed to correlate strongly with final PPA results? Given the weak correlation with final PPA metrics, it would be helpful to understand whether alternatives might better reflect the design objectives.
* What were the guideline in developing the benchmark suite? Additional insight into the design process for the benchmark circuits could offer helpful context for other researchers interested in contributing to or building upon this work.
soundness: 3
strengths: * The benchmark provides a well-rounded set of diverse circuit designs, covering various application domains, sizes, and characteristics, making it highly useful for generalizable evaluation of AI-driven placement algorithms.
* The open-source nature encourage wider adoption and enabling researchers to build on this foundation for further improvements in AI-based chip design.
* The evaluation spans multiple AI-based algorithmic approaches, making the suite a valuable comparative tool.
* By revealing the limited correlation between intermediate metrics and final PPA results, that may guide future research toward more effective and reliable evaluation criteria.
summary: This paper introduces a benchmark suite designed to evaluate AI-based chip placement algorithms, incorporating 20 circuit designs compiled from Verilog. The authors evaluate six algorithms—covering black-box optimization (BBO), gradient-based, and reinforcement learning (RL) approaches—against this benchmark for end-to-end performance. A key finding from the evaluation is the weak correlation between intermediate surrogate metrics and final design metrics (such as power, performance, and area, or PPA), highlighting the need for more reliable metrics. The benchmark suite, made openly available, represents a valuable resource for both academic and industry researchers, featuring diverse design characteristics across application domains, sizes, and configurations.
weaknesses: * The paper could benefit from more detailed analysis comparing each algorithm’s strengths and weaknesses on various design types. 
* The benchmark’s 20-circuit scale, while diverse, may not fully capture the vast complexity of real-world designs. Discussion of potential extensions or adaptability to larger, more complex circuits would be useful for extending its practical impact.

---

confidence: 1
contribution: 1
initial_rating: 1
presentation: 2
questions: None
soundness: 2
strengths: The paper presents a new benchmark suite to evaluate existing AI-based chip placement algorithms, which is probably essential for the chip-design community.
summary: The paper presents ChipBench, a benchmark specifically designed to evaluate the effectiveness
of existing AI-based chip placement algorithms.
weaknesses: - The paper isn’t relevant to the ICLR community, it is mostly relevant to the chip-design community.
- The paper violates ICLR’s double-blind requirement. The project is open-sourced in GitHub (https://github.com/MIRALab-USTC/ChiPBench), and the repo isn’t anonymous.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 4
questions: - Will the benchmark data be open-sourced?
- Can the authors provide the platform configuration (such as GPU, CPU, etc.) in the experiment section, as well as the total runtime of the evaluation scripts and the runtime of each placer?
soundness: 2
strengths: - The writing is clear and helps in understanding the motivation and solution.
- The process of constructing the benchmark is detailed.
- The orientation is clear. It is important for the AI community to obtain PPA and other metrics in chip design tasks.
summary: This article describes a process that starts with Verilog files and goes through logic synthesis, floorplanning, placement, CTS, and routing stages to ultimately generate routed metrics, which are used as a dataset. The authors use the open-source ASIC chip design framework OpenROAD and replace the algorithm of the placement stage in their experiments to evaluate the differences in intermediate surrogate metrics and final PPA results among different placement algorithms.
weaknesses: - The contribution of the paper is moderate. In the chip design field, the main issue for the AI community is not the inability to obtain PPA metrics, but the need to obtain them **quickly**. In fact, the placement results on datasets like CircuitNet 2.0 can be fully evaluated using either open-source tools like OpenROAD or commercial tools. This workflow is already streamlined (in essence, this part of the contribution comes from OpenROAD, not from this work itself). This work has facilitated the format exchange process between LEF/DEF and Bookshelf, but its contribution is limited.
- The conclusion of the paper is trivial. The mismatch between intermediate surrogate metrics and subsequent stages' intermediate surrogate metrics is a trivial phenomenon. In the EDA flow, intermediate surrogate metrics are used to **quickly** estimate the indicators of later stages in earlier stages. For example, HPWL used in the placement stage is a first-order approximate estimate for subsequent delay. Generally speaking, the closer the metric is to the subsequent stage, the more accurate it is; however, the evaluation cost is also higher.
- The choice of data is unremarkable. The authors need to explain why researchers from the AI or EDA community should choose the benchmark proposed by the authors. For example, the superblue series from ICCAD2015 is characterized by its large scale, the ISPD2015 dataset features fence regions, and CircuitNet 2.0 provides multimodal data that can help AI researchers reduce the time running commercial tools.