Title: FlexPlanner: Flexible 3D Floorplanning via Deep Reinforcement Learning in Hybrid Action Space with Multi-Modality Representation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Floorplanning

3D Floorplanning

EDA

Reinforcement Learning
Abstract: In the Integrated Circuit (IC) design flow, floorplanning (FP) determines the position and shape of each block. Serving as a prototype for downstream tasks, it is critical and establishes the upper bound of the final PPA (Power, Performance, Area). However, with the emergence of 3D IC with stacked layers, existing methods are not flexible enough to handle the versatile constraints. Besides, they typically face difficulties in aligning the cross-die modules in 3D ICs due to their heuristic representations, which could potentially result in severe data transfer failures. To address these issues, we propose FlexPlanner, a flexible learning-based method in hybrid action space with multi-modality representation to simultaneously handle position, aspect ratio, and alignment of blocks. To our best knowledge, FlexPlanner is the first learning-based approach to discard heuristic-based search in the 3D FP task. Thus, the solution space is not limited by the heuristic floorplanning representation, allowing for significant improvements in both wirelength and alignment scores. Specifically, FlexPlanner models 3D FP based on multi-modalities, including vision, graph, and sequence. To address the non-trivial heuristic-dependent issue, we design a sophisticated policy network with hybrid action space and asynchronous layer decision mechanism that allow for determining the versatile properties of each block. Experiments on public benchmarks MCNC and GSRC show the effectiveness. We significantly improve the alignment score from 0.474 to 0.940 and achieve an average reduction of 16% in wirelength. Moreover, our method also demonstrates zero-shot transferability on unseen circuits.

FlexPlanner: Flexible 3D Floorplanning via Deep Reinforcement Learning in Hybrid Action Space with Multi-Modality Representation

Abstract

In the Integrated Circuit (IC) design flow, floorplanning (FP) determines the position and shape of each block. Serving as a prototype for downstream tasks, it is critical and establishes the upper bound of the final PPA (Power, Performance, Area). However, with the emergence of 3D IC with stacked layers, existing methods are not flexible enough to handle the versatile constraints. Besides, they typically face difficulties in aligning the cross-die modules in 3D ICs due to their heuristic representations, which could potentially result in severe data transfer failures. To address these issues, we propose FlexPlanner, a flexible learning-based method in hybrid action space with multi-modality representation to simultaneously handle position, aspect ratio, and alignment of blocks. To our best knowledge, FlexPlanner is the first learning-based approach to discard heuristic-based search in the 3D FP task. Thus, the solution space is not limited by the heuristic floorplanning representation, allowing for significant improvements in both wirelength and alignment scores. Specifically, FlexPlanner models 3D FP based on multi-modalities, including vision, graph, and sequence. To address the non-trivial heuristic-dependent issue, we design a sophisticated policy network with hybrid action space and asynchronous layer decision mechanism that allow for determining the versatile properties of each block. Experiments on public benchmarks MCNC and GSRC show the effectiveness. We significantly improve the alignment score from 0.474 to 0.940 and achieve an average reduction of 16% in wirelength. Moreover, our method also demonstrates zero-shot transferability on unseen circuits. Code is publicly available at: https://github.com/Thinklab-SJTU/EDA-AI .

1 Introduction

In the very beginning stage of physical design in Electronic Design Automation (EDA), floorplanning (FP) plays a critical role. As a subsequent stage of hardware design [1] and logic synthesis [2], floorplanning provides a prototype for downstream tasks [3], ranging from power delivery network (PDN) design [4] to placement [5, 6] & routing [7, 8] (P&R), hence determining the upper bound of final PPA (Power, Performance, Area). Recognized as an NP-hard problem [9], FP establishes

the chip's physical layout by optimizing the position and shape of the major blocks to minimize interconnect lengths and ensure efficient silicon area utilization. As floorplanning technologies evolve, 3D FP with stacked layers emerges with more challenges. In particular, the cross-die module alignment becomes another pivotal factor in 3D FP [10, 11, 12]. For instance, vertical buses [11] for cross-die communication connect the aligned blocks spread among multiple dies [11]. Another example is the Memory-on-Logic technology [12, 13, 14], partitioning the processor [15] into two tiers: memory tier and logic tier. The memory tier consists of memory blocks and customized intellectual property cores (IP cores) [16], while the logic tier contains other components, such as logic blocks. Blocks on different dies should be aligned together, enabling the communication established by bonding bumps or pads [14].

Existing works can be categorized into heuristic-based methods, analytical approaches, and learningbased methods. Heuristics-based methods [17, 18, 19, 20, 21] model the FP with a certain heuristic representation. To refine the current FP, they modify the heuristic representation and convert it to the corresponding FP through a decoding scheme. However, this implementation limits the flexibility to directly adjust the position of blocks. After a single modification, the entire FP result needs to be regenerated, incapable of making fine-grained adjustments. Moreover, the alignment constraints cannot be satisfied by simply incorporating alignment metrics into its heuristics. Analytical approaches [9, 22] compute the gradient of objectives w.r.t. block position and utilize gradient descent technique to optimize FP. However, the calculation of alignment is non-differentiable, making them inapplicable in 3D FP. Recently, with the emergence of Reinforcement Learning (RL), learning-based methods are promising in FP [23, 24, 25, 26, 27, 28, 29]. Among these approaches, [23, 24, 25] still retain traditional FP representation, thereby leading to the same issues encountered by heuristic methods. Conversely, [26, 27, 28, 29] focus on determining block positions in 2D scenarios. However, when directly implemented in 3D scenarios, these methods could result in 1) overlooking alignment requirements and 2) multi-die property. Specifically, in 2D FP, blocks are arranged exclusively on a single die and organized in a queue that represents the placing order of all blocks. However, in 3D scenarios, multiple queues exist due to the multi-die property, necessitating a layer decision mechanism to merge these queues and determine the comprehensive placing order. This is a critical issue, yet it remains insufficiently explored in current research. Additionally, current learning-based methods are 3) incapable of addressing the variable aspect ratio of modules. Characteristics of typical approaches are summarized in Table 1.

To address the aforementioned challenges, we propose

, a flexible deep-learning- based

FlexPlanner approach in

hybrid action

space with

multi-modality representation for 3D FP. Flex-

Planner directly outputs the final FP result, with- out the reliance on any heuristic representation.

Under the Actor-Critic framework, the policy network consists of three sub-modules, respon-

sible for determining the position, layer, and

AR: aspect ratio of blocks. Aln: cross-die block alignment. Mod: Modality. *H = Heuristics, V = Vision, G = Graph, S = Sequence.

aspect ratio of blocks, spanning a hybrid action space. Empirical results demonstrate the effectiveness and significance of FlexPlanner. The main contributions are highlighted as follows:

First learning-based method to discard heuristic-based search in the 3D FP task. We propose a novel learning-based method with flexible hybrid action space for 3D FP, simultaneously handling the position, aspect ratio, and cross-die alignment of blocks. Without relying on the heuristic-based search, FlexPlanner allows the position and aspect ratio of each block to be explored across a comprehensive spectrum, rather than be limited by the constraints of heuristic FP representation, thereby breaking through the upper bound of performance. And we propose an innovative strategy for more effectively addressing the alignment issue.

Tackle the non-trivial issue of dependency on heuristics by incorporating hybrid action space and multi-modality representation. It is non-trivial to avoid the dependency on heuristics-based search in 3D FP due to the difficulty of modeling the complex solution space. Heuristics can only represent a subset of the entire solution space, resulting in limitations on the upper bound performance. To address this issue, we initially introduce three modalities, including vision, graph, and sequence, to comprehensively represent the state space. Additionally, we design a sophisticated policy network with hybrid action space and asynchronous layer decision mechanisms, enabling learning versatile properties such as position, aspect ratio, layer for each block in a 3D FP setting.

Zero-shot transferability. Leveraging the advantage of the learning scheme and multi-modalities, FlexPlanner demonstrates the ability to exhibit zero-shot transferability on previously unseen circuits. This capability shows strengths in its efficiency, as it conserves substantial training resources when confronted with new IC cases.

SOTA experimental results with significant alignment improvement. Within the learning framework, FlexPlanner achieves state-of-the-art performance on wirelength and alignment in 3D FP. Specifically, the average reduction in wirelength arrives at 16%, compared to previous works. Moreover, by effectively incorporating the alignment constraint, we achieve 0.940 on the alignment score, significantly surpassing the previous SOTA score of 0.474.

2 Preliminary and Formulation

Floorplan. The 2D floorplan task aims to determine the position and shape of each block given the block list, I/O port list, and netlist. Based on this, the 3D floorplan task is further required to place all blocks across multiple dies/layers. Each die/layer d ∈ D is a rectangular region with width W and height H , and all dies are of the same shape. Specifically, the block list is denoted as B = { b 1 , b 2 , . . . , b n } with n blocks, where each block b i is a rectangle with width w i , height h i , and area a i = w i · h i . The bottom-left coordinate of block b i is denoted as ( x i , y i ) , and the layer where the b i is located is denoted as z i . There are two types of blocks: hard block and soft block . For a hard block , its aspect ratio AR i = w i h i is fixed. For a soft block , AR i can vary between [AR min , AR max ] , while the area must always satisfy a i = w i · h i . Additionally, we denote the I/O port list as T = { t 1 , t 2 , . . . , t m } with m ports 2 . Each port t j is viewed as a point with a predetermined position ( x j , y j , z j ) , where ( x j , y j ) is the coordinate, and z j is the layer index. Netlist is defined as a set consisting of all nets, where each net is a set of blocks and ports, representing the interconnections. Since the layer z i of block b i is pre-assigned, we only need to determine the coordinate ( x i , y i ) and the aspect ratio AR i for all blocks to optimize the following objectives:

Alignment. Given two blocks b i , b j on different layers, alignment evaluates the overlap/intersection area between them on the common projected 2D plane. We define the alignment score aln( i, j ) :

where aln m ( i, j ) is the required minimum alignment area between block b i and b j . The total alignment score should be maximized to satisfy the alignment requirement. ( b i , b j ) forms an alignment pair if aln m ( i, j ) > 0 , and b i , b j are mutual alignment partners .

Cross-die block alignment is common in 3D FP [11, 13, 14]. Taking two blocks in different dies as an example, vertical buses [11] or bonding bumps/pads [14] are employed for communication, which requires capabilities for cross-die block alignment. That is, considering their projection onto a 2D plane, the related blocks must exhibit some minimum intersecting region, denoted as aln m ( i, j ) .

2) HPWL and 3) Overlap. Half Perimeter Wire Length (HPWL) is an approximate metric of wirelength. It can be computed much more efficiently, as accurate wirelength can be accessed only after the time-consuming routing stage. The summation of HPWL should be minimized :

where m i is either a block or port in net and x c i is the center x-coordinate. For a block, x c i = x i + w i 2 , and for a port, x c i = x i . Given two blocks b i , b j on the same die, the overlap area between them should be minimized (detailed calculation is given in Alg. 4 in Appendix G.1). Besides, all blocks should be placed within the fixed outline.

Critic

Position

Decider

Layer

Decider

Decider

Vision

Graph

Sequence

layer prob

AR prob

aln mask

pos mask

pos prob

masked

pos prob

Multi-Modal State

Policy Network

Hybrid

Action Space

constraints

Shared Encoder

Canvas

Alignment X

Alignment Y

Alignment

Valid Position

0

1

0

10

20

0

10

20

30

0

250

500

750

0

1

(a)

(b)

(c)

(d)

(e)

Axis Y

3 Methodology

Overview. The 3D floorplanning task can be formulated as an episodic Markov Decision Process. As shown in Fig. 1, pipeline of our approach mainly includes state, hybrid action space, policy network and critic network. State s t consists of three modalities, including vision, graph, and sequence. Action a t is represented as ( x, y, z, AR) , where position ( x, y, z ) are discrete variables and aspect ratio AR is a continuous variable. These properties form a hybrid action space. Under the Actor-Critic framework, critic V ϕ ( s t ) evaluates the current state. Policy network π θ ( a t | s t ) determines the 2D coordinate ( x t , y t ) of current block b t , the layer z t +1 to access next block b t +1 , and the aspect ratio AR t +1 of b t +1 . To explicitly impose constraints on the action space, masks are applied to the probability matrix of block positions. This process filters out invalid coordinates that violate constraints, such as non-alignment, overlap, or out-of-bounds locations. We respectively introduce the multi-modalities, layer decision, and reward function design in Sec. 3.1, Sec. 3.2 and Sec. 3.3.

3.1 Multi-Modality Representation of 3D Floorplanning

The state space contains three modalities: FP vision, netlist graph, and block placing sequence.

3.1.1 Vision Modality

It represents current floorplanning through images. We utilize four vision masks to depict chip layout.

Alignment Mask. f a ∈ N W × H ( N is the field of natural number) is a matrix to evaluate the alignment area between blocks b i and its alignment partner b j . If b j has already been placed, f a [ x, y ] is the intersection area on a 2D projected plane if b i is placed at ( x, y ) . If b j has not been placed yet, the

alignment mask of b i will be set to a matrix full-filled with aln m ( i, j ) . The native approach has the complexity O ( WH ) . However, when W or H is large, it has a low efficiency. We design an efficient alignment mask generation algorithm with meshgrid operation, which only iterates in regions causing projection overlap between these blocks and harnessing the power of parallel computing, thus reducing the complexity to O ( wh ) namely O (1) . Demonstration of alignment and details of the algorithm are shown in Fig. 2 and Alg. 1. Alignment mask is also utilized to filter out the positions that do not satisfy the alignment constraint.

Canvas Mask. Canvas mask f c ∈ N |D|× W × H is a global observation of current chip layout. The canvas mask is initialized to all zeros. After placing a block b at each step, we modify the canvas mask with f c [ z, x : x + w,y : y + h ] += 1 . Based on the canvas mask, we can implement a fast calculation of the total overlap of chip layout, shown in Alg. 4 in Appendix G.1.

Wire Mask & Position Mask [28]. Wire mask f w ∈ N W × H is a matrix for how HPWL will increase if a block is placed at the position. It records the increase of HPWL by placing the current block to all candidate positions. Position mask f p ∈ { 0 , 1 } W × H indicates available positions for current block to place without overlap

Algorithm 1: Alignment mask generation.

Input: Current block b i , alignment partner b j , chip width and height W,H Output: Alignment mask f ( i ) a for block b i x s = max(0 , x j - w i ) , x e = min( x j + w j , W ) y s = max(0 , y j - h i ) , y e = min( y j + h j , H ) x i = arange ( x s , x e ) , y i = arange ( y s , y e ) X i , Y i = meshgrid ( x i , y i ) U i = X i , V i = X i + w i U j = x j , V j = x j + w j // broadcast to a matrix U = where ( U i > U j , U i , U j ) V = where ( V i < V j , V i , V j ) f ( i ) ax = max(0 , V - U ) // we omit y-dimension due to page limitation f ( i ) a = zeros ( W,H ) , f ( i ) a [ X i , Y i ] = f ( i ) ax ⊙ f ( i ) ay

or out-of-boundary. 1 implies this position is feasible to place the block.

Overall, in step t , given current block b t , we concatenate f ( t ) a , f ( t ) c , f ( t ) w , f ( t ) p . We also incorporate masks f ( t +1) w , f ( t +1) p of header block in FIFO queue q (introduced in sequence modality in Sec. 3.1.2) in each die, which represent all possible choices for next block in step t +1 , providing policy with future horizon. All of these input masks constitute the floorplanning vision modality.

3.1.2 Graph and Sequence Modality

Graph Modality. Given a netlist, we convert it to a graph G ( V, E ) , where V is the set of vertices and E is the set of edges. For two blocks b i , b j within a net, we add edges e ij , e ji between vertex pair ( v i , v j ) . For the vertex feature, We select ( bid, x, y, z, w, h, a, p ) , where bid is the block index and p indicates whether this block has already been placed or not. Considering each block has a placing order showing the property as a sequence, it is natural to utilize positional encoding [31] to model this feature. Furthermore, we employ a Graph Attention Network [32] to produce embeddings of graph and nodes, representing the logical connection among blocks.

Sequence Modality. Given a die d i , blocks in d i are sorted by their area in descending order, forming a FIFO (first in, first out) queue q i . Thus, the block placing order for each die is pre-determined, and each queue can be viewed as a sequence. Combined with block features, we enhance and re-organize the multi-die sequence S ∈ R |D|× L × C , where L is the maximum number of blocks in a single die, C is the number of features. We select ( bid, x, y, z, w, h, a, p ) as sequence features. The sequence modality provides the model with global observation of entire block placing order, and is employed in the asynchronous layer decision discussed in Sec. 3.2.

3.2 Asynchronous Layer Decision

In 2D FP, blocks are arranged on a single die and are organized in a FIFO queue representing the placing order. However, in 3D scenarios, multiple queues exist due to the multi-die property, necessitating a layer decision mechanism to merge these queues and determine the comprehensive placing order. A native approach is synchronous block placing, in which we initially place all blocks on the first die, followed by blocks on the second die, and continue in this manner until the last die. In [26, 27, 28, 30] with synchronous placing, the entire placing order keeps fixed. However, this synchronous die-by-die placing order neglects the cross-die connections and alignment requirements, leading to a relatively poor FP layout. To address this issue, we propose an asynchronous layer decision mechanism, which determines the layer for accessing next block. If the policy selects die d i for next step, the header block of the FIFO queue q i will be popped as the next block to place.

During the training process under asynchronous block placing, on one hand, the entire block placing order could vary hugely and become unstable due to the sparsity of the reward, making it non-trivial for position decider to determine the positions of blocks. On the other hand, the layer decision module could converge too fast to mode collapse (a fixed placing order leading to poor quality), empirically degenerating to synchronous die-by-die block placing. In our analysis, it is the short forward horizon leading to this problem. Since policy is only able to sense current and next states, it lacks the global receptive field of entire placing order.

To address these issues, we enhance the representation with the sequence modality, and employ Transformer [31] to extract global placing order feature. Given the multi-die sequence feature S ∈ R |D|× L × C , self-attention is computed. S is viewed as the source sequence input with length L . The memory M ∈ R |D|× L × L , output of self-attention of S is given by [31]:

where MSA is the multi-head self-attention, LN is layer normalization, and FFN is the feed-forward network. Next, cross-attention is applied on M and current block feature b k . The single block feature b k ∈ R 1 × C is treated as the target sequence with length 1 , serving as the query vector. The output of cross-attention O ∈ R |D|× 1 × C is [31]:

where MHA( Q , K , V ) denotes multi-head attention. Finally, the next layer decision probability vector can be further computed through linear projection and softmax operation.

3.3 Reward Function with Local Advantage and Global Baseline

In 3D floorplanning task, final wirelength, overlap and alignment can only be accessed at the end of each episode, leading to a sparse reward. GraphPlace [26] uses this sparse reward design, where rewards at intermediate steps are all zeros. DeepPlace [27] adopts it with additional intrinsic reward via Random Network Distillation [33]. MaskPlace [28] introduces a dense reward scheme based on partial HPWL, which is calculated only on the currently placed blocks at each step. However, in two former methods [26, 27], they fail to accurately sense the intermediate quality through reward, and in MaskPlace [28], only difference between local

Algorithm 2: Reward function with local advantage and global baseline.

adjacent steps is involved, lacking the global view of the whole episode. As a result, all these reward designs demonstrate relatively poor performances, especially with the complicated hybrid action space consisting of position, layer and aspect ratio.

To alleviate this problem, we design a novel reward function with local advantage and global baseline. We define local advantage as the difference of metric between two adjacent steps, and global baseline as the overall metric at the end of an episode. With local advantage, our model has the ability to acquire current state is whether better or worse than the previous. Global baseline depicts the overall quality of the final floorplanning result. It is also essential for asynchronous layer decision module to avoid early convergence and degeneration to poor die-by-die synchronous block placing order, shown in Sec. 4.4 and Fig. 6b. Details of reward design is shown in Alg. 2.

3.4 Flexible RL with Hybrid Action Space for 3D Floorplanning

We employ RL with a hybrid action space to address the 3D FP task. The state space consists of three modalities, including floorplanning vision, netlist graph and sequence of block placing order. The hybrid action space is formulated as X × Y × Z × R , with X , Y , Z as discrete sets and R as a continuous set. In each step t , the policy outputs three distributions: 1) a discrete probability distribution for 2D position ( x t , y t ) of current block b t , 2) a discrete probability distribution to determine the layer z t +1 for accessing next block b t +1 , and finally 3) the mean value and standard deviation of a contiguous Gaussian distribution to determine the aspect ratio AR t +1 for b t +1 . The

reason for action a t containing the layer and aspect ratio for next step t +1 instead of t is that: after the execution of a t , next state s t +1 can be generated only after block b t +1 to place at step t +1 and its shape have already been determined, since b t +1 and its shape ( w t +1 , h t +1 ) are involved in the calculation of alignment mask and wire mask. To guarantee adherence to the specified non-overlap and alignment constraints, position mask f p and alignment mask f a are incorporated in 2D position decision. Only ( x, y ) satisfying f p [ x, y ] = 1 and f a [ x, y ] ≥ aln m are considered as valid positions, where aln m is the required minimum alignment area between current block and its alignment partner. Finally, without the reliance on conventional heuristic FP representation, our approach exhibits more flexibility to directly solve position, aspect ratio and cross-die alignment for blocks.

We select the Actor-Critic [34] framework and Hybrid Proximal Policy Optimization [35, 36] algorithm. The objective function of our hybrid policy π θ ( a t | s t ) can be formulated as:

where k = 1 , 2 , 3 represents position, layer and aspect ratio decision. λ k is the weight for each clip loss. r ( k ) t ( θ ) is the probability ratio π θ ( a ( k ) t | s t ) π θ old ( a ( k ) t | s t ) . ˆ A t denotes the generalized advantage estimation

(GAE) [37], and G t = ˆ A t + V t is the cumulative discounted reward [37, 38]. V t is the estimated state value from critic network V ϕ ( s t ) , and critic network is updated with minimizing Mean Squared Error (MSE) L ( ϕ ) = λ ϕ · ˆ E t [ ( G t -V ϕ ( s t )) 2 ] . Entropy of each action distribution is also added as a regularization term for exploration encouragement. Training algorithm is shown in Appendix G.2, and details of model architecture are shown in Appendix D.

4 Experiment and Analysis

4.1 Evaluation Protocol and Benchmark

We evaluate the performance of FlexPlanner and other typical methods on public benchmark MCNC 3 and GSRC 4 shown in Table 7 in Appendix B. The floorplanning region is set to square region. I/O ports are projected to the fixed outline, and their positions are kept unchanged. Aspect ratio of each block can vary in range [ 1 2 , 2 ] . Each experiment is run for five times with different seeds. More implementation details and hyper-parameter settings can be found in Appendix C.

Die 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Die 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

w/o aln partner

(a) Floorplan of circuit n50

Die 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Die 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

w/o aln partner

(b) Floorplan of circuit n100

4.2 Comparison with Baselines

Methods including heuristic-based and learning-based approaches are selected as baselines, and the implementation details are shown in Appendix E. Average alignment score and total HPWL are employed as evaluation metrics. Results are shown in Table 2 and 3. For alignment score, our approach achieves 0.940, significantly surpassing the second-best method which scores 0.474. For HPWL, our approach achieves an average reduction of 16%. Our method reduces HPWL to 249,053, and the second-best method (Wiremask-BBO) reduces it to 296,791. However, it only achieves 0.222 for alignment score, failing to effectively tackle the alignment constraint, empirically showing that our approach is capable of address alignment and HPWL simultaneously. Demonstration of 3D floorplanning results by our method are also shown in Fig. 3 and Fig. 4. We also compare performances on overlap, out-of-bound and runtime, shown in Appendix F. Capability and flexibility of our approach to address pre-placed modules (PPMs) are also demonstrated in Appendix F.5.

Die 0

1

2

3

4

5

6

7

8

9

10

Die 1

1

2

3

4

5

6

7

8

9

10

Aligned

Not Aligned

without aln partner

(a) GraphPlace, alignment score = 0.40

Die 0

1

2

3

4

5

6

7

8

9

10

Die 1

1

2

3

4

5

6

7

8

9

10

Not Aligned

Aligned

without aln partner

(b) DeepPlace, alignment score = 0.31

Die 0

1

2

3

4

5

6

7

8

9

10

Die 1

1

2

3

4

5

6

7

8

9

10

Aligned

Not Aligned

without aln partner

(c) Wiremask-BBO, alignment score = 0.35

Die 0

1

2

3

4

5

6

7

8

9

10

Die 1

1

2

3

4

5

6

7

8

9

10

Aligned

without aln partner

(d) Ours, alignment score = 0.98

0

500

1000

Epoch

0.5

0.6

0.7

0.8

0.9

Alignment

Finetune

From scratch

0

500

1000

Epoch

3.2

3.4

3.6

3.8

HPWL

1e5

Finetune

From scratch

0

200

400

600

Epoch

0.5

0.6

0.7

0.8

0.9

Alignment

Finetune

From scratch

0

200

400

600

Epoch

4.75

5.00

5.25

5.50

5.75

HPWL

1e5

Finetune

From scratch

4.3 Transferable 3D Floorplanning

The zero-shot transferability of our method is also evaluated. We firstly train the model on circuit n100, and directly test the performance without any fine-tuning. In Table 4, ratio is calculated between running inference and training on corresponding circuit. It demonstrates that our model exhibits a good zero-shot transferability, on either smaller or larger cases. The transferability is also demonstrated through the training curves between fine-tuning (based on pre-trained weights of circuit n100) and training from scratch. In Fig. 5, through fine-tune technique, better or similar performance can be achieved, conserving substantial training resources.

4.4 Ablation Studies

We ablate effectiveness of each component in our approach, including asynchronous layer decision, multi-modality representation and reward function. Experiments on circuit n100 are shown in Table 5. Removing the alignment mask leads to a huge drop of alignment score, from 0.961 to 0.349. We also evaluate the effectiveness of incorporating the alignment mask as an input feature and a constraint. In Fig. 6a, 'input' refers to the scenario where we feed the alignment mask as input to the model, while 'constr.' indicates that the alignment mask is utilized to filter out invalid positions that do not satisfy the alignment constraint. As the input, alignment mask plays a critical role to effectively capture the alignment information. As the constraint, it reduces the action space and accelerates training process.

Shown in the column 'sync' in Table 5, fixed synchronous die-by-die block placing order performs worse than asynchronous layer decision, since in the latter, more flexibility is provided to plan the entire placing order. Our reward design, incorporating the local advantage and global baseline, is crucial for fully leveraging the asynchronous layer decision mechanism. In Fig. 6b, with the guidance of our reward, the layer decision module keeps active to learn an optimal placing order. However, with other reward design schemes, it is faced with either rapid convergence to the degeneration of synchronous die-by-die decision-making, or unstable oscillation. Under reward function 'diff' in MaskPlace [28], only the difference between two consecutive steps is focused, while the global reward is disregarded, leading to rapid convergence to the degeneration of synchronous die-by-die layer decision. Under reward function 'sparse' in GraphPlace [26], rewards for intermediate steps are all zeros, and only the reward of the final step is non-zero. The policy lacks the modeling of local information, resulting in its inability to accurately assess the influence of current action on the entire decision-making process.

0

200

400

600

800

1000

Epoch

0.0

0.2

0.4

0.6

0.8

1.0

Alignment

input + constr.

input only

constr. only

both N/A

0

250

500

750 10001250150017502000

Epoch

10

20

30

40

50

Sum of first half seq (

Z

)

diff

sparse

ours

uniform

Figure 6: (a) Effectiveness of the alignment mask. As the input feature, it plays a critical role in capturing the alignment information. As the constraint (constr.), it reduces the action space and accelerates the training process. (b) Effectiveness of rewards on layer decision, shown in circuit n100 with episode length L = 100 and |D| = 2 dies. z t is the determined layer index at step t , and we note Z = ∑ L/ 2 t =1 z t . Z → 0 or Z → L/ 2 means degeneration to die-by-die synchronous layer decision (almost all die 0 or 1 in the first half episode).

5 Related Works

Classical methods in FP can be roughly categorized into heuristics and analytical approaches. The former typically models FP with a certain representation, such as B*-tree [17], Corner Block List [18] and Sequence Pair [39], based on which heuristic algorithm (especially Simulated Annealing [40]) is utilized to search for an optimal solution. Finally, the representation is converted to corresponding FP result via a decoding scheme. Apart from the heuristics, analytical approaches [9, 22, 41] regard the FP problem as an electrostatic system [42, 43]. They compute the gradient of objective functions w.r.t. block coordinates, and utilize gradient descent-based algorithm to optimize the solution.

RL-based floorplanning approaches. Recently, floorplanning also attracted attention from the reinforcement learning communities. For instance, [23, 24, 25] incorporate traditional FP representation and RL. Among these methods, at each step, the policy network either 1) decides whether to accept the randomly perturbed state or not [23, 25] or 2) determines how to perturb the current state [24]. [26, 27, 28] primarily focus on making decisions about block position in 2D scenarios with RL. GraphPlace [26] and DeepPlace [27] incorporate graph and vision as representation. MaskPlace [28] employs visual representation, and designs a dense reward function based on partial HPWL.

RL with hybrid action space. Common RL can be categorized into discrete and continuous action spaces. However, in certain scenarios, besides the discrete action, we need to make decisions regarding its parameters, typically residing in continuous spaces [36]. Consequently, it gives rise to a hybrid action space. [44] proposes to discretize the continuous portion, which creates a large discrete set and sacrifices fine-grained control. Alternatively, [45, 46, 47, 48] convert the discrete action selection into a continuous space with an actor network and employ a DQN-based [49, 50] algorithm for training, but found to be unstable and inefficient. [36] suggests to use multiple policy heads consisting of one for discrete actions and the others for corresponding continuous parameters separately. [51] proposes to encode the hybrid action space to a continuous space with GAE [37].

6 Conclusion

In this paper, we propose FlexPlanner, a flexible learning-based method in hybrid action space with multi-modality representation for 3D floorplanning task. It involves no reliance on heuristicbased search, thus achieves better flexibility to tackle position, aspect ratio and cross-die alignment for blocks under complex constraints. FlexPlanner outperforms baselines in alignment score and wirelength, and it also demonstrates zero-shot transferability on unseen circuits. This paper also has some limitations for future work: further optimization in 3D FP could be involved such as thermal optimization. Our method has no potential harm to the public society at the moment.

References

Yao Lai, Sungyoung Lee, Guojin Chen, Souradip Poddar, Mengkang Hu, David Z Pan, and Ping Luo. Analogcoder: Analog circuit design via training-free code generation. arXiv preprint arXiv:2405.14918 , 2024.

Yao Lai, Jinxin Liu, David Z Pan, and Ping Luo. Scalable and effective arithmetic tree generation for adder and multiplier designs. arXiv preprint arXiv:2405.06758 , 2024.

Lei Chen, Yiqi Chen, Zhufei Chu, Wenji Fang, Tsung-Yi Ho, Ru Huang, Yu Huang, Sadaf Khan, Min Li, Xingquan Li, et al. Large circuit models: opportunities and challenges. Science China Information Sciences , 2024.

Chen Wang, Jingkun Mao, Giuseppe Selli, Shaofeng Luan, Lin Zhang, Jun Fan, David J Pommerenke, Richard E DuBroff, and James L Drewniak. An efficient approach for power delivery network design with closed-form expressions for parasitic interconnect inductances. IEEE Transactions on Advanced Packaging , 2006.

Chung-Kuan Cheng, Andrew B Kahng, Ilgweon Kang, and Lutong Wang. Replace: Advancing solution quality and routability validation in global placement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2018.

Ruizhe Zhong, Junjie Ye, Zhentao Tang, Shixiong Kai, Mingxuan Yuan, Jianye Hao, and Junchi Yan. Preroutgnn for timing prediction with order preserving partition: Global circuit pre-training, local delay learning and attentional cell modeling. In Proceedings of the AAAI Conference on Artificial Intelligence , 2024.

Jinwei Liu, Chak-Wa Pui, Fangzhou Wang, and Evangeline FY Young. Cugr: Detailed-routability-driven 3d global routing with probabilistic resource model. In 2020 57th ACM/IEEE Design Automation Conference (DAC) , 2020.

Xingbo Du, Chonghua Wang, Ruizhe Zhong, and Junchi Yan. Hubrouter: Learning global routing via hub generation and pin-hub connection. Advances in Neural Information Processing Systems , 36, 2023.

Ximeng Li, Keyu Peng, Fuxing Huang, and Wenxing Zhu. Pef: Poisson's equation based large-scale fixed-outline floorplanning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2022.

Jill HY Law, Evangeline FY Young, and Royce LS Ching. Block alignment in 3d floorplan using layered tcg. In Proceedings of the 16th ACM Great Lakes symposium on VLSI , 2006.

Johann Knechtel, Evangeline FY Young, and Jens Lienig. Planning massive interconnects in 3-d chips. TCAD , 2015.

Anthony Agnesina, Moritz Brunion, Jinwoo Kim, Alberto Garcia-Ortiz, Dragomir Milojevic, Francky Catthoor, Gioele Mirabelli, Manu Komalan, and Sung Kyu Lim. Power, performance, area, and cost analysis of face-to-face bonded 3d ics. IEEE Transactions on Components, Packaging and Manufacturing Technology , 2023.

Lennart Bamberg, Alberto García-Ortiz, Lingjun Zhu, Sai Pentapati, Sung Kyu Lim, et al. Macro-3d: A physical design methodology for face-to-face-stacked heterogeneous 3d ics. In 2020 Design, Automation &Test in Europe Conference & Exhibition (DATE) , 2020.

Sai Pentapati, Anthony Agnesina, Moritz Brunion, Yen-Hsiang Huang, and Sung Kyu Lim. On legalization of die bonding bumps and pads for 3d ics. In Proceedings of the 2023 International Symposium on Physical Design , 2023.

Sankatali Venkateswarlu, Subrat Mishra, Herman Oprins, Bjorn Vermeersch, Moritz Brunion, Jun-Han Han, Mircea R Stan, Dwaipayan Biswas, Pieter Weckx, and Francky Catthoor. Impact of 3-d integration on thermal performance of risc-v mempool multicore soc. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2023.

David Koblah, Rabin Acharya, Daniel Capecci, Olivia Dizon-Paradis, Shahin Tajik, Fatemeh Ganji, Damon Woodard, and Domenic Forte. A survey and perspective on artificial intelligence for security-aware electronic design automation. ACM Transactions on Design Automation of Electronic Systems , 2023.

Yun-Chih Chang, Yao-Wen Chang, Guang-Ming Wu, and Shu-Wei Wu. B*-trees: A new representation for non-slicing floorplans. In Design Automation Conference (DAC) , 2000.

Jai-Ming Lin, Yao-Wen Chang, and Shih-Ping Lin. Corner sequence-a p-admissible floorplan representation with a worst case linear-time packing scheme. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2003.

Xianlong Hong, Sheqin Dong, Gang Huang, Yici Cai, Chung-Kuan Cheng, and Jun Gu. Corner block list representation and its application to floorplan optimization. IEEE Transactions on Circuits and Systems II: Express Briefs , 2004.

Hai Zhou and Jia Wang. Acg-adjacent constraint graph for general floorplans. In IEEE International Conference on Computer Design: VLSI in Computers and Processors , 2004.

Paul Falkenstern, Yuan Xie, Yao-Wen Chang, and Yu Wang. Three-dimensional integrated circuits (3d ic) floorplan and power/ground network co-synthesis. In Asia and South Pacific Design Automation Conference (ASP-DAC) , 2010.

Fuxing Huang, Duanxiang Liu, Xingquan Li, Bei Yu, and Wenxing Zhu. Handling orientation and aspect ratio of modules in electrostatics-based large scale fixed-outline floorplanning. In IEEE/ACM International Conference on Computer Aided Design (ICCAD) , 2023.

Qi Xu, Hao Geng, Song Chen, Bo Yuan, Cheng Zhuo, Yi Kang, and Xiaoqing Wen. Goodfloorplan: Graph convolutional network and reinforcement learning-based floorplanning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 2021.

Mohammad Amini, Zhanguang Zhang, Surya Penmetsa, Yingxue Zhang, Jianye Hao, and Wulong Liu. Generalizable floorplanner through corner block list representation and hypergraph embedding. In SIGKDD , 2022.

Wenbo Guan, Xiaoyan Tang, Hongliang Lu, Yuming Zhang, and Yimen Zhang. Thermal-aware fixedoutline 3-d ic floorplanning: An end-to-end learning-based approach. IEEE Transactions on Very Large Scale Integration (VLSI) Systems , 2023.

Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodology for fast chip design. Nature , 2021.

Ruoyu Cheng and Junchi Yan. On joint learning for solving placement and routing in chip design. NeurIPS , 2021.

Yao Lai, Yao Mu, and Ping Luo. Maskplace: Fast chip placement via reinforced visual representation learning. NeurIPS , 2022.

Yao Lai, Jinxin Liu, Zhentao Tang, Bin Wang, Jianye Hao, and Ping Luo. Chipformer: Transferable chip placement via offline decision transformer. In ICML , 2023.

Yunqi Shi, Ke Xue, Song Lei, and Chao Qian. Macro placement by wire-mask-guided black-box optimization. NeurIPS , 2023.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR , 2018.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894 , 2018.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. NeurIPS , 1999.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu. Hybrid actor-critic reinforcement learning in parameterized action space. In IJCAI , 2019.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Jiayi Weng, Huayu Chen, Dong Yan, Kaichao You, Alexis Duburcq, Minghao Zhang, Yi Su, Hang Su, and Jun Zhu. Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research , 2022.

Hiroshi Murata, Kunihiro Fujiyoshi, Shigetoshi Nakatake, and Yoji Kajitani. Vlsi module placement based on rectangle-packing by the sequence-pair. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems , 1996.

Dimitris Bertsimas and John Tsitsiklis. Simulated annealing. Statistical science , 1993.

Xingbo Du, Ruizhe Zhong, Shixiong Kai, Zhentao Tang, Siyuan Xu, Jianye Hao, Mingxuan Yuan, and Junchi Yan. Jigsawplanner: Jigsaw-like floorplanner for eliminating whitespace and overlap among complex rectilinear modules. In 2024 IEEE/ACM International Conference on Computer Aided Design (ICCAD) , pages 1-9. ACM, 2024.

Jingwei Lu, Pengwen Chen, Chin-Chih Chang, Lu Sha, Dennis Jen-Hsin Huang, Chin-Chi Teng, and Chung-Kuan Cheng. eplace: Electrostatics-based placement using fast fourier transform and nesterov's method. ACM Transactions on Design Automation of Electronic Systems (TODAES) , 2015.

Yibo Lin, Shounak Dhar, Wuxi Li, Haoxing Ren, Brucek Khailany, and David Z Pan. Dreamplace: Deep learning toolkit-enabled gpu acceleration for modern vlsi placement. In Proceedings of the 56th Annual Design Automation Conference 2019 , 2019.

Alexander A Sherstov and Peter Stone. Function approximation via tile coding: Automating parameter choice. In International symposium on abstraction, reformulation, and approximation , 2005.

Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143 , 2015.

Warwick Masson, Pravesh Ranchod, and George Konidaris. Reinforcement learning with parameterized actions. In Proceedings of the AAAI conference on artificial intelligence , 2016.

Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo Fu, Tong Zhang, Ji Liu, and Han Liu. Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space. arXiv preprint arXiv:1810.06394 , 2018.

Craig J Bester, Steven D James, and George D Konidaris. Multi-pass q-networks for deep reinforcement learning with parameterised action spaces. arXiv preprint arXiv:1905.04388 , 2019.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR , 2016.

Boyan Li, Hongyao Tang, YAN ZHENG, HAO Jianye, Pengyi Li, Zhen Wang, Zhaopeng Meng, and LI Wang. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. In ICLR , 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS , 2019.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. NeurIPS , 2016.

Bing Xu. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 , 2015.

A Notation

All notations in this paper are shown in Table 6.

B Statistics of Benchmark

We evaluate the performance of FlexPlanner and other typical methods on public benchmark MCNC 5 and GSRC 6 shown in Table 7. The 'alignment' means the number of blocks with alignment partner.

C Implementation Details

C.1 Computational Resources

We use PyTorch [52] deep learning framework and tianshou [38] Reinforcement Learning framework. We select Adam [53] optimizer with learning rate 0 . 0001 . We train and test our model on a Linux server with one NVIDIA GeForce RTX 3090 GPU with 24 GB CUDA memory, two AMD Ryzen Threadripper 3970X 32-Core Processors at 3.70 GHz and 128 GB RAM.

C.2 Minimum Alignment Requirement Configuration

For the alignment setting, given an alignment pair ( b i , b j ) , the minimum alignment requirement aln m ( i, j ) is set to aln m ( i, j ) = α ij · min { a i , a j } , where a i , a j are the area of block b i , b j . The coefficient α ij controls the minimum alignment requirement between block b i , b j , and can be adjusted according to the specific circumstances for each alignment pair. In our experiments, we set all α ij = 1 . 0 , which is the most challenging scenario to evaluate the effectiveness of our approach.

C.3 Hyper-Parameters

Other hyper-parameters are shown in Table 8.

D Model Architecture

We introduce the networks in our pipeline shown in Fig. 1 as follows:

Shared Encoder. The shared encoder E ϕ mainly consists of two parts: 1) a CNN-based backbone for floorplanning vision modality input, and 2) a GNN (Graph Neural Network)-based backbone Graph Attention Network [32] with two layers for graph modality.

In step t , given current block b t , we concatenate the alignment mask f ( t ) a , canvas mask f ( t ) c , wire mask f ( t ) w , and position mask f ( t ) p together. We also incorporate f ( t +1) w , f ( t +1) p of header block in FIFO queue q of each die. These masks of queue header blocks represent all possible choices in step t +1 , providing policy with future horizon. All of these input masks constitute the floorplanning vision modality.

Taking the netlist graph as input, the GNN backbone outputs nodes embeddings, representing the local receptive field information. Besides, a global average pooling layer is applied on these node embeddings to calculate the graph embedding, which captures the global view of the whole netlist graph.

Critic Network. The critic network V ϕ mainly consists of three parts: 1) the shared encoder E ϕ , 2) a sequence Transformer [31] with two encoder layers and two decoder layers for block placing order sequence, and 3) a die/layer embedding model to represent each die/layer with a learnable feature vector. The shared encoder E ϕ is responsible for processing the floorplanning vision input, and the Transformer is for the input sequence modality. Serving as the source sequence in Transformer, the multi-die sequence feature S ∈ R |D|× L × C is employed in the calculation of memory, which is the output of self-attention. And the single block feature

b k ∈ R 1 × C is treated as the target sequence with length 1 , serving as the query in cross-attention. To Stabilize training, the shared encoder E ϕ is only updated during the updating process of critic network. As a result, the critic network is responsible to take input as the state s t and output corresponding state value v t .

Policy Network.

Position Decider. The position decider network π (1) θ mainly consists of a generator network. It takes input as the output of the shared encoder E ϕ , and outputs a 2D feature map as the probability matrix to determine the 2D position for current block. We borrow the generator architecture from InfoGAN [54], which utilizes up-sampling layer instead of transposed convolutional layer to realize scaling-up. It contains three blocks and each block is with convolutional layer, batch normalization, leaky ReLU [55] activation and up-sampling layer. Finally, it outputs a probability matrix M ( t ) p ∈ R W × H to determine the position ( x t , y t ) for current block b t .

Layer Decider. The layer decider network π (2) θ mainly consists of three parts: 1) a CNN-based model for FP vision modality, 2) a die embedding model to represent current input die, and 3) a Transformer network to process the input block placing order sequence for each die. The CNN model consists of three blocks, and each block has one convolutional layer, one max pooling layer, with ReLU as activation function. The Transformer model consists of two encoder layers and two decoder layers, and is designed to process the input multi-die block placing order sequence. The layer decider network takes input as the state s t and outputs a probability vector p ( t ) z ∈ R |D| to determine the layer z t +1 for accessing next block b t +1 .

Aspect Ratio Decider. The aspect ratio decider network π (3) θ mainly has a CNN model, consisting of three 2D convolutional layers with ReLU activation function and max pooling layer. It takes input as the floorplanning vision modality in state s t and the feature of next block b t +1 to determine the aspect ratio AR t +1 for b t +1 . Since the aspect ratio is an action in contiguous space, we select Gaussian distribution to depict it, and the network outputs corresponding mean value and standard deviation.

E Baselines

The baselines referred in Sec. 4 are introduced as follows:

3D-B*-SA [21] is a heuristic-based approach, which represents a 3D floorplanning by B*-tree [17]. Simulated Annealing (SA) [40] is selected to search an optimal result. At each iteration, current B*-tree is randomly perturbed to a new state, which will be accepted based on a certain probability. Finally, the B*-tree is converted to corresponding floorplannning result via a decoding scheme. In previous scenarios, only HPWL and out-of-bound penalty are incorporated into heuristics (cost/energy function). To optimize the alignment in 3D FP, we also combine the alignment score into its heuristics to guide the solution searching.

RL-CBL [24] combines heuristic-based search Corner Block List (CBL) [18, 19] and Reinforcement Learning together, utilizing policy network to determine how to perturb current FP at each step. It utilizes intermediate out-of-bound penalty and final HPWL as reward function. In 3D FP with multiple stacked layers, we extend CBL to 3D scenario. In order to realize the optimization of cross-die block alignment, we also incorporate alignment score into reward function. However, it is incapable of addressing the variable aspect ratio of soft blocks.

Wiremask-BBO [30] is a black-box optimization (BBO) framework, by using a wire-mask-guided [28] greedy procedure for objective evaluation. At each step, it utilize the position mask to filter out invalid positions causing overlap or out-of-bound. Within the valid positions, the location with the minimum increment of HPWL is selected to place current block. Considering the alignment requirement in 3D FP, we incorporate the alignment mask to further filter out invalid positions which do not satisfy the alignment constraint. However, it is incapable of addressing the variable aspect ratio of soft blocks.

GraphPlace [26], DeepPlace [27], MaskPlace [28] are three learning-based approaches, utilizing Reinforcement Learning to decide the position for each block on 2D chip layout. Variable aspect ratios of soft blocks are not taken into consideration. In this task, we extend them to 3D scenarios, and incorporate the alignment mask to filter out invalid positions where alignment constraint is not satisfied. Besides, alignment score is also involved into the calculation of reward function, with the purpose of assisting them to realize the optimization of cross-die alignment.

F Addtional Experiments

F.1 Runtime Comparison

Comparison of runtime among our approach and other baselines are shown in Table 9. The runtime of our method is either lower or comparable to other learn-based methods [26, 27, 28], and is better than the heuristic-based method [21], where tens of thousands of iterations are required. Pipeline of Wiremask-BBO [30] is the same as

MaskPlace [28], while neural network is not involved in it, contributing to the shortest inference time. For the approach RL-CBL [24], it is faster than other learning-based methods since its environment is relatively simple, only responsible for decoding the heuristic representation Corner Block List (CBL) [18, 19] into corresponding FP result. However, the performance is also limited by CBL, and the final FP result is further worse than other methods without the reliance of heuristic representation.

F.2 Out-of-Bound

For heuristics-based methods, such as 3D-B*-SA [21] and RL-CBL [24], the heuristic representation will be converted to corresponding floorplan result through a decoding scheme. Although non-overlap can be guaranteed, blocks in the final FP could be out of the fixed outline, leading to the low area utilization of chip die and out-of-bound. In our method FlexPlanner, each block can be naturally placed within the boundary due to the alignment mask and position mask. Consequently, the occurrence of out-of-bound floorplan is effectively prevented. We compare the out-of-bound in Table 10, and the outbound is calculated as follows:

F.3 Overlap

We also compare performance on overlap among our method and baselines, shown in Table 11. Our method reduces the overlap to 0.001, which comparable with 3D-B*-SA [21], RL-CBL [24], and better than other methods. In 3D-B*-SA [21] and RL-CBL [24], although non-overlap can be guaranteed, blocks in corresponding FP could be out of the fixed outline, leading to the low area utilization of chip die and out-of-bound as discussed in Appendix F.2.

F.4 Influence of Grid Size

Due to the different size of circuits, we discretize the sizes into uniform region. Given a chip die with original shape W o × H o , the region is projected into W × H . For each block b with shape w × h , its shape is also projected into max { 1 , round( w · W W o ) } × max { 1 , round( h · H H o ) } . We also empirically investigate the

influence of different grid size W,H , shown in Table 12. Metric 'Error' evaluates the area error between the projected region and the original region as follows:

where B is the set of all blocks. Similar performances in terms of alignment, HPWL, and overlap are achieved across different grid number settings. Lower grid numbers offer faster execution speeds but may result in relatively higher area errors that may not meet precision requirements. Higher grid numbers produce more fine-grained floorplanning results but lead to a larger action space and require additional computational resources. As a trade-off between runtime and precision, a grid size of 128 is selected.

F.5 Floorplanning with Pre-Placed Modules

Our approach is also capable of addressing the circuits with pre-placed modules (PPMs). PPMs are blocks whose position and aspect ratio are pre-determined and fixed during the FP process. Heuristics-based methods [21, 24] are incapable of address the existence of PPMs. In these heuristics approaches, after the perturbation is applied on current FP representation, the entire FP layout result may undergo changes. During this process, it cannot be guaranteed that the positions and shapes of pre-placed modules will remain unchanged. Consequently, they lack the flexibility to address circuits with PPMs. Our method FlexPlanner is capable to solve this issue. At the beginning of FP process (the beginning of an episode), all PPMs are placed on their pre-determined positions, and their aspect ratios keep unchanged. After that, we modify the canvas mask and position mask, indicating that these positions have been occupied by PPMs. In each step t to place a movable block b t , the overlap among b t and other blocks (PPMs and other placed blocks) can be avoided. The floorplanning result with PPMs is shown in Fig. 7.

G Algorithms

G.1 Algorithm for Calculation of Alignment Rate and Overlap

The calculation of alignment score aln t and overlap o t at step t are shown in Alg. 3 and Alg. 4. These values are involved in the calculation of reward function with local advantage and global baseline shown in Alg. 2.

G.2 Training Algorithm

The overall training pipeline is shown in Alg. 5, based on PPO [35] algorithm with RL framework tianshou [38]. During a training epoch, the policy network collects training data into replay buffer by interacting with the environment. To accelerate the training process, the policy network simultaneously interacts with n e environments

Die 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Die 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Pre-Placed Module (PPM)

w/o aln partner

(a) Circuit n50 with 6 PPMs.

Die 0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Die 1

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

Pre-Placed Module (PPM)

w/o aln partner

Algorithm 3: Alignment score calculation.

Input: Current block b t , alignment score aln t - 1 at step t - 1 , number of alignment pairs n a Output: Alignment score aln t at step t

aln t = aln t - 1

if b i has alignment parnter b j and both b i , b j have already been placed then

aln

t

+=

aln(

i,j

)

n

a

// according to Eq. 1

end

Algorithm 4: Overlap calculation.

Input:

Current block b t , canvas mask f c , chip die width W and height H

Output: Overlap o t at step t

f c [ z i , x i : x i + w i , y i : y i + h i ] += 1 calculate overlap o t as follows:

in parallel. We set the buffer size to satisfy

ensuring that each episode in the replay buffer is completed and terminated with its final step, where L buf is the size of replay buffer. After data collection, we further compute the reward, cumulative discounted reward and advantage. Since each episode is completed in the replay buffer, each step has the corresponding terminated step (end step) within the same episode to calculate the reward with local advantage and global baseline. Finally, both critic network and policy network are updated.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We make clear claims about contributions and scope of our paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the liminations in Sec. 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided detailed hyper-parameters settings in Appendix C and the full training pipeline in Alg. 5 in Appendix G.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

We have provided detailed hyper-parameters settings in Appendix C and the full training pipeline in Alg. 5 in Appendix G.1. Code and data will be public available once our paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided detailed hyper-parameters settings in Appendix C and the full training pipeline in Alg. 5 in Appendix G.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Experiments in Sec. 4 are run five times with different seeds. Both mean value and standard deviation are provided.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information about compute resources in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, is with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Broader impacts including both positive and negative societal impacts are discussed in Sec. 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The license for both datasets GSRC and MCNC is MIT which can be found in http: //vlsicad.eecs.umich.edu/BK/copyright.html

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors have proposed FlexPlanner, a flexible 3D floorplanning method with deep reinforcement learning. Existing learning methods mainly focus on the 2D scenarios. However, it suffers from overlooking alignment requirements and multi-die property. To address these, FlexPlanner learns a hybrid action space with multi-modality representation. It contains three modules to estimate position, layer and aspect ratio of blocks. Experiments demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The problem it addresses is clearly stated. While most existing learning methods target at 2D FP task, this paper address 3D FP task, and also show the difficulties when directly applying 2D methods to 3D.\n\n2. This work introduces three modalities to represent the state space, which show better representative ability than heuristics-based methods.\n\n3. The writing of the paper is clear.\n\n4. Significant improvements on alignment scores according to Table 2.\n\nWEAKNESSES:\n1. It would be better to include a teaser that better shows the problems when directly applying 2D FP method to this 3D scenario.\n\n2. The novelty of the paper is a concern. From my point of view, the idea of using three modalities to address the issue is quite straightforward, which lacks of novelty. In the rebuttal, the authors should further highlight the novelty of the technical designs of this work.\n\n3. Please further explain the significant improvements in Table 2, which is very impressive. It would be better to include some visualization comparisons as well as more explanations on why existing methods obtain such low alignment scores.\n\n4. I have some concerns about the baselines in the table. It should include more recent baselines within 1 year. Also, I notice that [14] was published in 2010 -- why do this method still achieves comparable results with other baselines?\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper proposes the FlexPlanner, a reinforcement learning-based method utilizing multi-modality representation, including vision, graph, and sequence, to handle different challenging scenarios.
Additionally, the design of the action space to uniformly handle constraints represents a new and innovative method. 
This approach, being largely free from heuristics, is more aligned with machine learning principles and advances the field of learning for floorplanning, particularly in 3D scenarios. 
The performance improvement from 0.474 to 0.940 on public benchmarks is significant, and the method also demonstrates strong transfer learning capabilities.

STRENGTHS:
1. This paper is well written, with a clear background and motivation, and Table 1 is informative and categorizes the features of existing methods, which helps me understand the approach. 

2. The use of vision, graph, and sequence modalities provides a rich representation of the planning problem, expanding the action space and enabling the learning of multiple properties, such as position, aspect ratio, and layer for each block.

3. The experimental results are much better than previous methods on benchmarks in terms of wirelength and alignment. 

4. The appendix part is also informative and useful for better understanding the approach and the results

WEAKNESSES:
There is no weakness in this paper, to be honest.

QUESTIONS:
As RL could be a controversial approach for placement and routing, how do the authors defend their RL-based methodology to the floorplanning problem?

LIMITATIONS:
No specific limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new learning-based method for IC design that simultaneously handles the position, aspect ratios, and alignment of blocks.  The method achieves significant improvements compared to baselines by leveraging reinforcement learning with a hybrid action space and multi-modality representation to optimize block positions, aspect ratios, and cross-die alignments.\n\nSTRENGTHS:\n* The application of RL with a novel reward function in a hybrid action space to 3D IC FP is very interesting and novel, moving away from traditional heuristic-based methods. \n* The paper demonstrates notable improvements compared to existing baselines on the evaluated benchmarks.\n* The paper clearly defines the used metrics.\n\nWEAKNESSES:\nClarity and Writing:\n* While the paper is overall well-organized, some sections could benefit from additional explanations or simplifications to make the content more accessible to readers without an IC design background.\n* Important design details  (e.g., the critic network) are deferred to the appendix and not introduced in the main text.\n* Although the benchmarks and baselines used are relevant, discussing them in more detail could significantly enhance the presentation.\n\n\nEvaluation Pipeline:\n* Providing more details on the exact setup of training and test data could improve transparency.\n* RL methods are known to suffer from stability issues and are often sensitive to hyperparams. How sensitive is the method to the hyperparams used? This should be evaluated and discussed to comment on the reproducibility of the results.\n* How well does the method generalize to larger circuits beyond those in the benchmarks? A discussion in this direction would be highly appreciated.\n\nDesign Choices:\n* There are many design choices in the paper. For instance, the significance of the multi-modality input is not entirely clear. The model takes in \"vision,\" \"graph,\" and \"sequence\" modalities, but it is not specified how much each of these contributes to the final performance. While the ablation studies show that the alignment mask is a good design choice, they do not provide much intuition about the importance of other design choices, such as the canvas mask or input modalities (e.g., sequence and graph). Are these necessary?\n\nQUESTIONS:\nsee above.\n\nLIMITATIONS:\nYes, the limitations are briefly addressed in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper proposes a learning-based method called FlexPlanner  in hybrid action space with multi-modality representation to simultaneously handle position, aspect ratio, and alignment of blocks.  FlexPlanner models 3D FP based on multi-modalities, includ15 ing vision, graph, and sequence.  The work designs a policy network with hybrid action space and asynchronous layer decision mechanisms that allow for determining the versatile properties of each block.  Experiments on public benchmarks MCNC and GSRC  show the effectiveness.

STRENGTHS:
1. The paper introduces a new learning-based method in hybrid action space with multi-modality representation for 3D floorplanning task.  
2. The paper explains the rationale behind each proposed module/component of the model with experimental verifications of the module's effectiveness.

WEAKNESSES:
Lack of Clarity: Certain sentences in the paper are unclear and difficult to understand, hindering the comprehension of the proposed methodology.  For example, the elaborated form of ablation experiment section is similar to an experiment report.

Insufficient Experimental: The paper may lack some ablation experiments about the important hyperparameters.

QUESTIONS:
Unfair comparisons: There are unfair comparisons with other work. For instance, the methods  GraphPlace [35] and DeepPlace [13] originally incorporated graph and vision as representations.  MaskPlace [25] only employed visual representation, whereas the authors’ method used graph, vision and sequence. Three multimodalities provide more information.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-15