Collaborative Symmetricity Exploitation for Offline Learning of Hardware Design Solver
Authors: Haeyeon Kim
Minsu Kim
Joungho Kim
Jinkyoo Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: Symmetricity
Offline learning
Hardware design solver

Abstract: This paper proposes \textit{collaborative symmetricity exploitation} (CSE) framework to train a solver for the decoupling capacitor placement problem (DPP) benchmark, one of the significant hardware design problems. Due to the sequentially coupled multi-level property of the hardware design process, the design condition of DPP changes depending on the design of higher-level problems. Also, the online evaluation of real-world electrical performance through simulation is extremely costly. Thus, a data-efficient offline learning method to train a solver (i.e., contextualized policy) with high generalization capability over changing task conditions is necessary. In this paper, we apply the CSE framework to train a DPP solver using a limited number of offline expert data. Leveraging the symmetricity for offline learning of hardware design solver has two major advantages: it increases data-efficiency by reducing the solution space and improves generalization capability by capturing the invariant nature present regardless of changing conditions. The proposed CSE is composed of two learning schemes: expert exploitation and self-exploitation. Expert exploitation induces symmetricity during the imitation learning process with offline expert data and self-exploitation induces symmetricity during the consistency learning process with self-generated data. Extensive experiments verified that CSE with zero-shot inference outperforms the neural baselines and iterative conventional design methods on the DPP benchmark. Furthermore, CSE showed promising extrapolation capability as it greatly outperforms the expert method used to generate the offline data for training. Scalability and flexibility of the proposed method were also verified for practical use of CSE in industry.

Under review as a conference paper at ICLR 2023
COLLABORATIVE S SYMMETRICITY EXPLOITATION FOR OFFLINE LEARNING OF HARDWARE DESIGN SOLVER
Anonymous authors Paper under double-blind review
ABSTRACT
This paper proposes collaborative symmetricity exploitation (CSE), novel symmet- ric learning scheme of contextual policy for offline black-box placement problems. Leveraging the symmetricity increases data- cuta-efficieney by reducing the solution space, and improves generalization capability by capturing the invariant nature present regardless of changing context. To this end, we design a learning scheme that reduces the order bias (ex., neural network recognizes {1, 2, 3) and {2, 1,3] as difference placement design) inherited from sequential pecision-mmainn scheme of neural policy by imposing action-permuten permutation (AP)-sym mAp)-yymmerr (i.e, the per- muted sequences
of the original sequence) of placement problems. We first defined the order bias and proved that AP-symmetricity is im- posed when the order bias of neural policy becomes zero. Then, we designed two collaborative losses for learning neural policy with reduced order bias: expert ex- ploitation and self-exploitation. The expert exploitation loss is designed clone the behavior of the expert solutions considering order bias. The self self-exploitation loss is designed be a special form of order bias where it measures AP-symmetricity from a self- eeff-generated solution. CSE is applied to the decoupling capacitor place- ment problem (DPP) benchmark, a significant offline black-box placement design problem in hardware domain that requires contextual policy. Experiments show that CSE outperforms state-of-the-e solver for the DPP benchmark.
commettic--lacemeenn
INTRODUCTION
With the CMOS technology shrinking and increasing data rate, the design complexity of very large- scale integrated (VL.I) has increased. Human experts are no longer able to design hardware without the help of electrical design automation (EDA) tools, and EDA tools now suffer from long simulation time and insufficient computing power, making machine learning (ML) application to hardware design inevitable. Many studies have already shown that deep reinforcement learning (DRL), one of the representative ML methods for sequential decision making, is promising in various tasks in modern chip design; chip placement (Mirhoseini et al. 2021: Agnesina et al. (020), routing (Liao al. 2019 2020), circuit design ZZhao & Zhang 2020), logic synthesis (Hosny al. 2020
Haaswijk et al.) 2018) and bi-level hardware optimization (Cheng & Yan 2021).
However, most previous DRL-based hardware design methods do not take the following into consid- cration. (a) Online simulators for hardware are usually time intensive and inaccurate; thus, learning with existing offline data by experts is more reliable. Since there exists a limited number of offline hardware data, data-efficient learning scheme is necessary. (b) Hardware design is composed of electrically coupled multi-level tasks where task conditions are determined by the design of higher- level tasks; thus, a solver (i.e., contextualized policy conditioned by higher-level tasks) with high generalization capability to adapt to varying task conditions is necessary.
In this paper, we leverage the solution symmetricity of placement problem for data efficiency and generalization capability. Conventional sequential decision-makinn schemes for placement problems (Park et al., 2020; Mirhoseini et al. 202 1 Cheng & Yan (021 auto-regressively generate solutions without considering the solution symmetricity, thus having the order bias; the neural network identifies the action permuttatonnn (AP) symmetric solutions (i.e. identical placement designs), for instance, {1, 2, 3} and {2, 1,3), as different solutions. Our proposed method overcomes the order bias limitation of the previous sequential decision-making schemes with novel regularization

---

Under review as a conference paper at ICLR 2023
technique. Tackling the order bias (i.e. inducing AP- (--mmmetrriitt)) improves the data efficiency of training and generalization capability of the trained policy due to the two following reasons. First, data efficiency in training can be improved learning the AP -yyymmettiyy reduces the exploration space (see Fig. Tp: neural network can automatically learn not only from the explored trajectories but also from their symmetric solution trajectories without additional exploration and simulation. Second, generalization capability on task variation can be improved as AP-symmetricity is the task-agnostic nature of placement problems.
To this end, we devised collaborative symmetricity exploitation (CSE) framework, simple but effec- tive method to induce A--symmetriyity with two collaborative learning schemes: expert exploitation and self-exploitation. The expert exploitation sim- ply augments the offline expert data (sequential data) with random permutation operator and uses it for imitation learning. The self-exploitation generates pseudo-labeled solutions from the current training policy, transforms the pseudo-labeled solution with a random permutation operator, and forces the solver have an identical probability to generate the original pseudo-labeled solution and the transformed solution.
Explored Trajestory Symmetric Trajentories
AF-transformation
O
[13,11,3]
[13, 23, 11]
[23, 13, 11]
Learning with Simulation Learning with AP-symmetrictty
Figure 1: Conventional sequential decision- making method's heterogeneous trajectories from AP-Symmetric solution group.
To verify the effectiveness of CSE, we applied CSE to the decoupling capacitance (decap) placement problem (DPP), one of the significant hardware design benchmarks. The objective of DPP is to place given number of decaps on power distribution network (PDN) with two varying conditions: keep-out regions and probing port location, determined by higher-level problems such as chip placement and routing. The goal of CSE is to train solver (i.e., contextualized policy) that has high generalization capability to any given task condition.
Contribution 1: A novel symmetric learning scheme for contextualized policy. There exists
several works (Cohen & Welling, [016)] TToomas et al.] 201 8. [Fuchs et al.) [220 Satorras et al. [021]
that learn various symmetricities of input data in the domain space for regression and classification
tasks. However, learning symmetricity in solution space is less studied as learning the symmertrcities
in solution space of sequential policy (generative decision) is challenging. [Bengio al. (2021)
tackled solution symmetricity of sequential policy by turning the Markov decision process (MDP) tree model into the directed acyclic graph (DAG) flow model. However, they target single-task optimization where the optimal solution set is unchanged. On the other hand, our CSE is an effective solution symmetric learning scheme for the contextualized policy capable of adapting to newly given lask-condition.
DPP benchmark release. DPP is widely studied task in hardware domain without public release of the simulation models and source codes for the methods. Also, DPP can be seen as a contextual offline black box optimization benchmark with extended properties compared the design-bench (Trabucco et al.) 2022). representative non-contextual offline black-box optimization benchmark. In this work, by releasing the DPP benchmark with open-source simulation models and our reproduced baselines, DRL-based methods, meta-heuristic methods, behavior cloning- based methods, and our state-of-ther ofethe-rrtttttt CSE method, we expect huge industrial impacts on the hardware and the ML communities.
Contribution 2:
2 DECAP PLACEMENT PROBLEM (DPP) FORMULATION
This paper seeks to solve the decoupling capacitor placement problem (DPP), one of the essential hardware design problems. Decoupling capacitor (decap) is a hardware component that reduces power noise along the power distribution network (PDN) of hardware devices and improves the power integrity (PI). With transistor scaling and continuously decreasing supply voltage margin (Hwang) et al. 202I. ). power noise has become a huge technical bottleneck in high-speed computing systems. Generally, the more decaps are placed, the more reliable the power supply is. However, adding more decaps requires more space and is costly. Thus, finding an optimal placement of decaps is essential in terms of hardware performance and cost/space-saving.
2

---

Under review as a conference paper at ICLR 2023
Prodinng Port
Recurrent Contect Network (RCN)
Korp- Out Region
Probing Port Contect Network
Derag
(PCN)
((+0) a Di
Cprobe
prabe
Decader
Danaker
AM Encoder
Temporal Factorization
DMarkov Decision
fttline Expert Dota
Sc+1
[xx
a
Sg
Policy Imiitation Learning with Contexts (PCN, RCN)
((*"))
((a
Inference
patodln)
Cellon-Pemmutaiin (A (AP)-Sy SAP-Syymetriiitty
Unseen DPPs
(See Figure 4)
DPP Design Set
Figure 2: Overall pipeline of DPP contextualized policy parameterization with offline learning.
The goal of DPP benchmark is to optimally place a pre-defined number of decaps on a target PDN, given two conditions determined by higher-level tasks. First, keep-out regions are action-restricted areas where decaps cannot be placed as a design constraint. Second, probing port the target chip/logic block location where the objective, power integrity (PI), is evaluated. The ports PDN are represented as Neww x Neot grids, and the number of decaps is denoted by K. See Appeeddxx A.2 for details of PDN and decap modeling for the benchmark.
Remark that DPP cannot be formulated as a conventional mixed- inttger linear programming (MILP)- based combinatorial optimization because PI performance not be formulated a closed analytical form but can only be measured or simulated. This study aims to learn an effective DPP solver that be used in practice.
2.1 CONTEXTUAL, MARKOV DECISION PROCESS (MDP) OF DPP
As shown in Fig. 2] the procedure for solving DPP is modeled contextual Markov decision process (CMDP). CMDP is an augmented MDP proposed by (Hallak et al. 2015). The parameters of CMDP, transition and reward function, change based on the context variable. DPP can be formulated as CMDP, where the objective function J (: az) is determined by the task-condition a X (i.e. context); the X' is task-condition space.
Specifically, our objective function [[ ( is determined by PDN which is contextualized by æ. The contextualized PDN is represented where each grid (i.c., port) on PDN is represented as æ, = (I,, V/s cj), in which x., y/4 indicate 2D coordinates of location, indicates the condition of port whether it belongs probing port Lprobe (c: = 2), keep-out regions Ikeepout (c, = 1), or decap allowed ports Tallowed (c; = 0). Note that Ikeepout and Jallowed represent index sets consisting of indices corresponding keep-out regions and decap allowed ports, respectively. Iprobe refers to an index of probing port. See Appendix A.3
set of three-dimensional feature vectors x = ) ) /.rrss X.Nest
The design process sequentially places decaps the available PDN ports until planning all the designated decaps. We model this CMDP with state, action, and policy as follows:
State contains task-condition and previous selected actions:
(x, 01:-1)-
Action a, E {l,..., Nrow x Neot s;-1 is defined as an allocation of a decap to one the available ports PDN. The available ports are the ports PDN except for the probing port, keep-out ports, and the previously selected ports. The concatenation of sequentially selected actions a = d1.K indicates the final decap placement solution.
Policy Te ()) is the probability of producing specific solution = a1:K. given task-condition æ, and is factorized as:
K
Φ = II
(e())))))
1=1
where po(ar) (8)) is the segmented one-step action policy parameterized by the neural network.
3

---

Under review as a conference paper at ICLR 2023
The objective of DPP is to find the optimal parameter 0* of the policy Too as:
arg max (x) (a) [{);
where px (a) is the probability distribution for varying task-condition æ and J is objective function. Finding the optimal policy for various DPPs is task contextual learning problem, in which each DPP has distinct task condition. Once the task æ specified by Px(x), the state-action space with complexity of (Nrow 1 is determined. Thus, an efficient policy no (a(a)) should K capture the contextual features among varying task conditions æ.
Note that CSE is based on imitation learning, by which objective function J is implicitly induced through offline expert data. The objective function DPP described equation
2.2 OBJECTIVE FUNCTION OF DPP
PPP
Target PDN modeling
Nrow
Rigt
Neal
(a) Unit-cell representation of target PDN of hardware device
(b) 7-paramettr of target PDN.
Figure 3: Unit-cell and Z- Z-parameter representations of real- world target PDN.
Performance Evaluation Metric. The performance of DPP is evaluated by power integrity (PI) simulation that computes the level of impedance suppression over a specified frequency domain and quantified as:
J = (Ziinitia (ffina(()) IGHz f
FEF
where Zimitial and fiiat are the initial and final impedance at the frequency f before and after placing decaps, respectively. F is the set of specified frequency points. As shown in F the PI simulation for (Nrow x Neot) PDN requires a Nyow-Neoll x NNow X fifreq (number of frequency points) sized Z-parameter matrix calculation because each port is electrically coupled the rest of the ports and Z (i.e., impedance) is Z. parameeter matrix calculation is costly. The more impedance is suppressed, the better the power integrity and the higher the performance score. Note that this performance metric was also used for collecting the offline expert data using a genetic algorithm (GA).
Thus, performance evaluation with large
3 METHODOLOGY
This section provides technical details of the proposed collaborative symmetricity exploitation (CSE) framework and the modified attention model (AM) neural architecture with two domain- speccfiise devised context neural networks for training DPP solver.
3.1 ACTION-PERMUTT PERMUTTPERM SYMMETRICITY AND ORDER BIAS OF PLACEMENT TASK
The symmetricity found in placement problems is the action
AAP)-yymmtrricity. the order of placement does not affect the design performance. Let us denote I, a permutation of an action sequence {1,.., K) 3. where K is the length of the action sequence. We then define the AP- transformation TAP = {4) as a set of all possible permutations. The AP-symmetricity of DPP is induced the learned solver through the traasfffoornnno TAP.
K!
Definition 3.1 (AP-symmetriciit). For any A. X.t TAp where A is solution space and X' is task-condition space,
Scala-valued Function f: Ax is AP-symmetric if, f(a, x) (o))=f((((e f(t(a), Conditional probability π is AP-symmetric if, :T x) T(((a))=).
4

---

Under review as a conference paper at ICLR 2023
The objective function J A + DPP is an AP-symmetric function because and t(a) have identical placement design. The main role of CSE is to induce AP-symmetricity to the policy π (conditional probability) to reflect the AP-ymmmetrictty of objective function J.
Moreover, we define an order bias metric, b(T; p). to measure AP- A--yymmetricity.
Definition 3.2 (Order bias on distributions [pxx PPA: PA:: D. For a conditional probability T (a)æ)) where X (lask-condition space) and A (solution space), the order bias ((T;; p) is defined as:
b(T: p) Paapp (t)l - x (((a)]=)]11)
Intuitively, the order bias bi;;; p) is general property of sequential solution generation scheme. It measures how much the solver x ((a()) has different probabilities to generate AP-symmetric solutions. The order bias metric holds for the following theorem:
Theorem 3.1. Task-conditioned policy T(aaa)) is AP-symmetric if and only if order bias is zero
(b(TT;p) p) = =0) while the distributions are non-zero, px((x) > 0, O,Pa(a) > O, Pr (t) > O, for any X, E A and & TAP. Proof. See Appendix G
3.2 COLLABORATIVE SYMMETRICITY EXPLOITATION (CSB) FRAMEWORK
Deep [(x10) "
Dang
[3,712216]]
Tar
[212211,3]
Expert Explnitation (EE)
[16.12.7,3]]
Offinee Expert Data
Expert Data
Angmented Data
Pollcy
5,2,5,
Self- Exploitation
(SE)
(,4,
Self Gemerated Data
Transformed Data
Copy Policy
Figure 4: Illustration of collaborative symmetricity exploitation (CSE) process.
The CSE framework was designed to induce the AP-symmetricity to the trained model to improve the generalization capability and allow data-efficiency training.
To train a contextualized policy with a limited number of expert data, we designed the CSE loss term L. consisting of expert exploitation loss LExpert and self-exploitation loss LSstf- Each loss function is mainly designed to reduce bias order (Definition 3.2
L = Expert + ALself
(4)
Lxspert = - Ea Drig RogTo a [x]] Lsaf = Exwilt E a Bentlr [Il=g(((())) =t(t(a))(=)ll))
(5)
(6)
Expert Exploitation. The major role of expert exploitation is to train high-quality symmet- ric contextualized policy for various task-conditions by leveraging the offline expert data a" with TAP. The TAP transforms the existing offline expert data a for P times to augment the offline expert dataset Deep = ((x(1, alo) N to reflect the AP-symmetric nature of the placement task. Specifically, we randomly choose (t1,..., Lr, TAr generate Daug ((x()) (())), (z(1), t (a()))) (z(1), (a())))) )- Then, Expert is expressed as teacher- forcing imitation learning scheme with the augmented expert dataset Daug. Note that expert exploita- tion is expected reduce order bias defined with the three uniform distributions; æ of Up (æ), of UDD..,, (a), and t. of (TTA(())
Celf-Exploitation. While Dang only contains expert quality, self-exploitation involves self eelf-generated data, whose quality is poor at the beginning but improves over the phase of training. Thus, the self-exploitation scheme is designed to induce the AP-symmetricity a wider action space to achieve greater generalization capability. Formally, self-exploitation loss is a special form of order bias defined based on the distributions, æ ~Ux, a ~Mg (current policy) and ~UZaps where is a uniform distribution; Lseff = b( p {Ux, Tg. (rrr)).
5

---

Under review as a conference paper at ICLR 2023
3.3 CONTEXTUAL ATTENTION MODEL
To further improve the generalization capability of the trained DPP solver, we modified the attention model (AM) (Kool et al. 2019) and termed contextual attention model. As described in Fig. 2 the decision-maiee making procedure consists of two newly devised context neural networks; (1) encoder capturing initial design conditions while contextualizing the probing port through the probing port context network (PCN), and (2) decoder sequentially allocating decaps on PDN while contextualizing the stages of the partial solution through the recurrent context network (RCN). See Appendix for detailed implementation.
Encoder. Our encoder consists of multi-head attention (MHA) and feedforward (FF), similar to the transformer network (Vaswani al. 2017). The encoder takes the task-conditioned PDN feature vector æ as an input and outputs all node embedding h. Encoding is processed once at the initial state, t O. The node embedding h is time-invarian (i.e., fixed after the encoding process) and is used in the decoding process. We proposed a novel probing context network (PCN) in the encoding process so that the learned solver can adapt well to a new task. PCN is a simple but effective two-layer perceptron model with a ReLU activation layer that takes the hidden embedding of the probing port node prooe and outputs the probing port contextual vector Cprobe = MLPP (h
Decoder. With the node embedding h generated by the encoder, decoder sequentially selects an action Gr until placing all K decaps. At each step I, the decoder takes (1) the node embedding (static information), (2) the current state st, (3) the probing port contextual vector Cprobe, and (4) the previous action context vector Ca._1 = MLP MLPRON (ha,1)) generated by the recurrent context network (RCN) inputs and outputs new action a₁. To leverage sequential state transitions in decoder, we devised the recurrent context network (RCN). RCN is a two-layer perceptron model with ReLU activation layer that embeds the previously selected node's ' embedding h at step t into Ca.- 1 Then, overall context vector is updated as e = Cprobe + Ca._1 and is used as query for attention decoder that eventually infers the next action 0g by the attention mechanism, where the key and value comes from h. See (ppenddix C.2 for detailed process.
RELATED WORKS
Machine Learning-bas bassdd Methods for DPP. Deep reinforcement learning (DRL) has been widely used to solve DPP. Park et al. (2018) employed Q-learning and Park et al. (2020); Zhang et al. (0020) applied convolutional neural network (CNN)-based Q approximators to solve a target DPP. However, their methods require large iterations involving costly reward calculations and their trained policies were non-reusable; if the DPP condition changes, they must be re-trained. In an effort to overcome the reusability limitation by training solver, Park et al. (2022); Kim et al. (2021) implemented promising neural combinatorial optimization (NCO) models, the attention model (Kool et al.) 2019)
and pointer network (Vinyals al. 2015), construct contextualized policy without iterative exploration and domain knowledge. However, their methods still showed poor data-efficiency in training and unsatisfactory generalization performance.
Symmetricity Learning in Solution Space. There exists several studies to leverage the symmetricity in solution space. [won et al.] (2020) suggested a new reinforcement learning scheme. policy optimization for multiple optima (POMO) to leverage the traveling salesman problem (TSP)'s solution symmetricity, the eyelic property that identical solution can be expressed as N heterogeneous trajectories by permuting initially visited node. Kim et al. (2022) proposed the symmetric neural combinatorial optimization (Sym-NCO) method, which is an extension of POMO to general-purpose symmetric learning for various combinatorial optimization tasks. [Bengio et al. (2021) proposed generative flow net (GFlowNet) to train policy distribution proportional to reward distribution π xc R considering solution symmetricity. The GFlownet suggests a sequential decision-makin a directed acyclic graph (DAG), instead of classical tree structure, to induce solution symmetricity. The GelowNet is applied solve molecule optimization and bio-sequential design (Jain et al. 2022).
scheme with
While POMO (woon et al. 2020) and Sym-NCO Kim et al. (2022) leverage DRL, CSE focuses on offline imitation learning. Though GFlowNet (Bengio et al. 2021 can be trained in fully offline manner, it is not yet designed for training a contextualized policy. Thus, CSE is positioned between POMO and Gflownet as an offline symmetricity learning method to train contextualized policy.

---

Under review as a conference paper at ICLR 2023
Table 1: Performance evaluation with the average score of 100 PDN cases (the higher the better).
Method
Method Type
Pi Simulation (M)
Avg. Score
Random Search
Online Scarch
10,000
12.70
Genetic Algorithm (expert policy)
Online Search
100
12.56
Genetic Algorithm
Online Search
500
12.79
AM-RL (Park et al.
2022
Pretrained
1
11.71
Arb-RL (Kim et al.
2021
Pretrained
1
9,60
AM (Park al 2022
IL
Pretrained
1
12.06
Arb (Kim er 2021
IL
Pretrained
1
10.80
CSE (ours)
Pretrained
12.88
EXPERIMENTAL RESULTS
5.1
SETUP
PDN and Decap Specifications. The PDN used for verification is hhip-ppakagee hierarchical PDN, modeled by the segmentation method (Kim et al. 2010 Cho al., (201)). The PDN model
is represented as (Nrow x Noot) (10 x 10) grids over 201 frequency points linearly distributed between 200MHz and 20GHz, which gives 100 X 100 x 201 R 2M impedances to be evaluated per each task; the PDN scale is reasonable to reflect the simulation intensiveness of DPP. The RLGC electrical parameters of PDN unit-cell are shown in Appendix A.2 Out of the Nrow x Neol ports on PDN, one is assigned as a probing port and 0 to 15 ports are assigned as keep-out ports (see Appendix A.4). Decap is modeled as a unit-cell with single port that is attached to a specific port on PDN when action is made. The RLGC electrical parameters of decap unit-cell are also shown in in[pppenixx A.2 100 PDN cases for test and another 100 PDN cases for validation were generated for performance evaluation. We made sure test data, validation data and training data did not overlap. We used the number of decap K = 20 for every training, but K can be changed during the inference.
Offline Expert Data Collection. Since expert data for the DPP benchmark was not available, we synthetically generated offline expert data using genetic algorithm (GA) for this study. The number of iterations done for collecting single data represented as M. Note that the higher the M, the better the quality of data, but the higher the simulation cost. We used GA (M 100) to collect the offline expert data. In addition, we denote N as the number of offline expert data used for training CSE. Note that the lower the N the more data-efficient the training is.
Hyperparameters. For training, we generate three transformed data per expert data. We denote P(= 3) as the number of AP-transformed data per offline data. Thus, the total number of guiding data becomes N x (P + 1). For instance, P 3, N 50 makes total 50 x + 50 = 200 guiding data. We set the distribution p. described in Section 2.1
uniform distribution for training.
We used N = 2000 offline expert data for training CSE and IL-based baselines. For the learning algorithm, we used ADAM (Kingma & Ba 2015) with a learning rate of 10-5, We trained model with batch size 100 for < 200 and batch size 1, 0000 for N 1, 000 and 2, 000. We trained for a maximum of 200 epochs for each model; used the model with the best validation score for CSE and other ML baselines to evaluate performance. See Appendix D for detailed setup.
Baselines for Comparison. For search heuristic baseline methods, implemented random search and genetic algorithm. Since these methods are iterative solvers, they require a large number of simulations (i.e., .M > 100. See Table 1p for each problem. For non-iterative baaning-based solvers, we reported two RL baselines, AM-RL (Park et al. 2022) and Arb-RL (Kim et al. (02T)) and two IL baselines, AM-IL IL and Arb-IL, which are modified RL and Arb-RL with imitation learning instead of reinforcement learning, to investigate the effectiveness of CSE components compared to the same imitation learning approaches with different learning strategies/neural architecture. Implementation details of the baselines are provided in Appendix D.2 and Appendix D.3
5.2 GENERALIZATION CAPABILITY EVALUATION
To verify the generalization capability of the trained solver, each method is given the same unseen 100 DPPs and the average performance score was measured, after allocating K = 20 decaps on each.
7

---

Under review as a conference paper at ICLR 2023
As shown in Table 1 our CSE significantly outperformed all baselines in terms of average performance score. Online search methods generally find solutions that give a high average performance. This is due to large number of searching iterations M, which incurs the same number of costly simulations. On the other hand, the learning-base -aaddd baselines and CSE do not require simulations to generate solutions; once trained, they only require single simulation measure the performance. Though learning-based methods can easily find such a solution, CSE is the only method capable of finding a solution that outperforms the highly iterative online search methods by a zero-shot inference.
When the number of costly simulations was limited, RL-based methods (AM-RL, Arb-RL) showed poorer generalization capability than their IL versions (AM-IL, Arb-IL) due inefficiency in exploring over extremely large combinatorial action space of DPP. We believe that imitation learning approach, fitting the policy with offline expert data, has greater exploration capability with the help of expert policy thus able to achieve higher performance with a limited simulation budget (see Appendix D.2). Note that we have an infinite budget for reward simulation (which never happens in real-world hardware setting), DRL could achieve greater performance and generalization capability with a sufficient learning loop.
Among the IL approaches trained with the same number of offline expert data (N = =2 000), CSE showed the highest performance. We believe that such higher generalization capability comes from both symmetricity exploitation schemes and the newly devised neural architecture: (1) expert exploitation and self-exploitation with symmetric label transformation amplify the number of data to train with and induce solution symmetricity to improve generalization capability. (2) the neural architecture with PCN and RCN makes the policy easily adapt to new task conditions.
Extrapolation over Expert Method. The CSE policy trained with offline expert data generated by the expert policy, GA (100), outperformed GA [500), with zero-shot inference. That is, the CSE policy trained with low-quality offline expert data produced higher-quality designs. We believe this was possible because we trained a factorized form of policy that does not predict labels in a single step but produced a solution through a serial iterative roll-out process, during which good strategy for placing decaps can be identified. In addition, the CSE with symmetric label transformation has further guided the policy to learn such an effective decap placement design scheme.
25.0
Valisattnn Score
valvoeeoom Score
Vilreaation Searee
11.0
Originel ACI
Origina AM
Contestual AM
Contectual AM
Original AM
Contestual AM EE
Originel AM EE SE
Originad AM
Contestioal 10
Contediool
(Dmm)
105
30
15
Traming
Looch
(a) CSE in Original AM
(b) CSE in Contextual AM
(c) Original vs. Contextual AM
Figure 5: Ablation study on CSE components
Ablation Study. We conducted ablation studies to validate the effectiveness of CSE components and context neural networks with sparse offline data (N 100). We ablated the effectiveness of expert exploitation (EE) and self-exploitation (SE) in two policy networks: original AM and contextual AM (ours). The original AM refers to the AM-IL baseline. Each component of CSE supported increasing generalization capability in both policy networks and the contextual AM with newly devised context neural networks was verified to outperform the original AM. Therefore, we verified that both CSE components and the modifications of AM successfully contributed to the promising performance.
Order Bias Measurement. We empirically show that our CSE successfully induces AP-symmetricity by reducing the order bias (see Appendix H].
5.3 OFFLINE DATA EFFICIENCY EVALUATION
We investigated how the number N of offline data generated by the expert method, GA(M = 100), affects the design perfor- mance of CSE and the two baselines, AM-IL and Arb-IL. As shown in Fig. 6 CSE outperformed the baselines in all N vari- ation; CSE trained with N 100 even performed better than the baselines trained with N = 2000. Moreover, CSE monotonically improved with N while the others saturated when N > 500.
123
CSE (Duno
AN--L
Arb.L.
Figure 6: Offline data efficiefyy evaluation (P = 3, 8)
8

---

Under review as a conference paper at ICLR 2023
Table 2: Scalability evaluations on larger PDN scale and varying number of decap K.
Scale Variables
Methods
PDN Scale
Number of Decap. K
GA [100)
AM (Park et al. 2022 +IL
CSE (ours)
12
11.77
10.22
12.23
16
12.25
11.13
12.60
10x 10
20
12.53
11.71
12.81
24
12.79
12.20
12.95
30
13.02
12.62
13.11
20
7.61
6.23
8.47
15x15
40
7.69
7.75
8.54
5.4 SCALABILITY EVALUATION
For scalability verification, learning-bast based DPP methods were
pre-trained for fixed scale PDN, (10 x 10), and fixed number of decaps. K 20. Then, the pre-trained models were asked to place decaps of varying K E (12, 16, 20, 20330 } on (10 10) PDN and varying K E {20, 40) for a larger (15 15) PDN without additional training (i.e, zero-shot). We chose two baseline methods for comparison: GA {100) and AM-IL. As shown in Table 2| our CSE outperformed GA {100} and AM-IL for all scales. Furthermore, CSE achieved greater performance with fewer decaps. Reducing the number of decaps has significant industrial impact; as hardware devices are mass-produced, reducing single decap saves enormous fabrication cost.
5.5 FLEXIBILITY VERIFICATION FOR PRACTICAL APPLICATION
To verify the practical applicability, we applied the proposed CSE a real-world hardware application, high bandwidth memory (HBM), which is an interposer-based 2.5D IC. As shown in Fig. 7 the hierar- chical PDN model of HBM is composed of (40 x 40) package PDN, (40 60) interposer PDN and (15 20) on-chip PDN, each layers connected by TSV + C4 bumps and microbumps.
On-chip
Manelump
TSV
Interpowe
PKG
Sideview
For performance evaluation, we compared the CSE(N 1000} to GA[M = 100), AM-IL(N = 1000} and Arb-IL([N = 1000) on placing varying number of decaps, K, for 100 test cases. The pre-trained solvers with K = 20 were used for CSE, AM-IL and Arb-IL without additional training.
Ammm
PKG
Interpoer
12mm
De-dkip
0.225mm
20mm
Fig. verifies that CSE achieved higher performance with signifi- cantly fewer decaps. For instance, the performance score of 26.68, attained with 40 decaps by M = 100) and 34 decaps by AM-IL. was achieved with only 26 decaps by zero-shot CSE. Arb-IL was unable to achieve 26.68 even with 80 decaps. Power noise analysis in a test case was carried out (see AAppendix E.3). where the ini- tial power noise before placing decap was V0.546mV. CSE able reduce the power noise to 0.610mV (-94.2%) with 26 decaps while GA M = 100] reduced to 0.682mV (-93.5%) with 40 decaps. Reducing the number of decaps a huge contribution to the industry as hardware devices are mass- producd,, reducing single decap can greatly reduce production costs.
20mm
(b) Topview
Figure 7: Structure of HBM
PDN model
6 CONCLUSION
This paper proposed the collaborative symmetricity exploitation (CSE) framework for training contextualized policy (i.e., solver) of placement tasks in an offline manner. The CSE was applied to decap placement problem (DPP) and achieved the most promising perfor- mance among all baseline methods. The CSE was also validated on several scales and on multiple hardware devices. The CSE a general purpose offline learning scheme for placement tasks that be further applied to other hardware placement tasks including chip placement, ball grid array (BGA) placement, and via placement.
Bir1 2o13
Somber Sonie
Figure 8: Performance com- parison with number of decap variation on HBM

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anthony Agnesina, Kyungwook Chang, and Sung Kyu Lim, Vlsi placement parameter optimization using deep reinforcement learning. In Proceedings of the 39th International Conference on Computer-Aided Design, ICCAD '20, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781 450380263. doi: 00.1145/3400302.44156666 Utthtts:://dol org/
10. 19.114 11411112220 /0445/34400 000002 34156990
Irwan Bello al. Neural combinatorial optimization with reinforcement learning. CoRR, 1.09940 2016. /tR//ttt://arxww.or//////
09940
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. Advances in Neural Information Processing Systems, 34:27381-27394, 2021.
Ruoyu Cheng and Junchi Yan. On joint learning for solving placement and routing in chip design. Advances in Neural Information Processing Systems, 34, 2021.
Kyungjun Cho, Youngwoo Kim, Subin Kim, Hyunwook Park, Junyong Park, Seongsoo Lee, Daeyong Shim, Kangseol Lee, Sangmook Oh, and Joungho Kim. Fast and accurate power distribution network modeling of silicon interposer for 2.5 ics with multiarray tsvs. IEEE Transactions on Components, Packaging and Manufacturing Technology, 9 2019. doi: 10.1109/
TCPMT.2019 28950033.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2999. PMLR, 2016
Francesco de Paulis, Riccardo Cecchetti, Carlo Olivieri, and Markus Buecker. Genetic algorithm pdn optimization based on minimum number of decoupling capacitors applied to arbitrary tar- get impedance. In 2020 IEEE International Symposium on Electromagnetic Compatibility Sig- nal/Power Integrity (EMCSI), pp. 428-433, 2020. doi: 10. L0L1100//S118923222000020 1911008.
Ihsan Erdin and Ramachandra Achar. Multi-objective optimization of decoupling capacitors for place- ment and component value. IEEE Transactions on Components, Packaging and Manufacturing Technology, 910) 2019. doi: 10. 101109/CCPMT.2199993056
Jun Fan, J.L. Knighten, A. Orlandi, N.W. Smith, and J.L. Drewniak. Quantifying decoupling capacitor location. In IEEE International Symposium on Electromagnetic Compatibility. Symposiun Record (Cat. No.00CH370166, volume 2, pp- 761-766 vol.2, 2000. doi: 10. 109/1SEMM..2000.444711
S. Farrahi and Ethan Koether. Effect of power plane inductance on power delivery networks. In DesignCon, 2019.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se trastrafffosmmrs: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
Winston Haaswijk, Edo Collins, Benoit Seguin, Mathias Soeken, Frédéric Kaplan, Sabine Süsstrunk, and Giovanni De Micheli. Deep learning for logic optimization algorithms. In 2018 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-4, 2018. doi: 10. 109A/SASS. 2018.8351885.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv:1502.0221 2015.
Abdelrahman Hosny, Soheil Hashemi, Mohamed Shalan, and Sherief Reda. Drills: Deep rein- forcement learning for logic synthesis. In 2020 25th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 581-586. 2020. doi: 10. 1AL.1///S--AAA755.222222 201.6..5..2.0000555, 9045559.
Jisoo Hwang, Jun So Pak, Dooseok Yoon, Heeseok Lee, James Jeong, Yun Heo, and IIryong Kim. Enhancing on-die pdn for optimal use of package pdn with decoupling capacitor. In 2021 IEEE 71st Electronic Components and Technology Conference (ECTC), pp. 1825-1830, 2021. doi: 10.1 1111FFCCTC2229622200002222
10

---

Under review as a conference paper at ICLR 2023
Moksh Jain, Emmanuel Bengio, Alex Semaandez-Garicee Jarrid Rector- Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang. Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference Machine Learning, pp. 9786-9801. PMLR, 2022.
Jack Juang, Ling Zhang, Zurab Kiguradze, Bo Pu, Shuai Jin, and Chulsoon Hwang. mod- ified genetic algorithm for the selection of decoupling capacitors in pdn design. In 2021 IEEE International Joint EMC/SI/PI and EMC Europe Symposium, pp. 712-717, 2021. doi: 10.1 1.L1IPECSSI/VEAACCUgg229922.0..1
Haeyeon Kim, Hyunwook Park, Minsu Kim, Seonguk Choi, Jihun Kim, Joonsang Park, Seongguk Kim, Subin Kim, and Joungho Kim. Deep reinforcement learning framework for optimal de- coupling capacitor placement on general pdn with an arbitrary probing port. In 2021 IEEE 30th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS), pp. 1-3, 2021. doi: 10. D0.1109/PEESS1141...00 99991194.
Jaemin Kim et al. Chip-package hierarchical power distribution network modeling and analysis based on a segmentation method. IEEE Transactions on Advanced Puckaging, 33(3))647-659, 2010. doi: 10.1 (100///AADDV.210200333676
Minsu Kim, Junyoung Park, and Jinkyoo Park. Sym-nco: Leveraging symmetricity for neural combinatorial optimization. arXiv preprint arXiv:2205.13209, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. http
w/wpxiv.orggabs///
6980
Wouter Kool, Herke Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2019, URL /attps://opereeiiee
Yeong-Dae Kwon, Jinho Choo, Byoungjip Kim, Iljoo Yoon, Youngjune Gwon, and Seungjai Min. Pomo: Policy optimization with multiple optima for reinforcement learning. Advances Neural Information Processing Systems, 33, 2020.
Haiguang Liao, Wentai Zhang, Xuliang Dong, Barnabas Poczos, Kenji Shimada, and Levent Burak Kara. A Deep Reinforcement Learning Approach for Global Routing. Journal of Mechanical Design, 142(6), 11 2019. ISSN 1050-0472. doi: 00.1115/1.4045044 URL
1115/1 4045044 061701.
Haiguang Liao, Qingyi Dong, Weiyi Qi, Elias Fallon, and Levent Burak Kara. Track-assignment detailed routing using attention-based policy model with supervision. In Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD, pp. 105-110, 2020.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young- Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, William Hang, Emre Tuncer, Quoe Le, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean. A graph placement methodology for fast chip design. Nature, 594:207-212, 06 2021. doi: 1111038/441586-021--33333
Hyunwook Park, Junyong Park, Subin Kim, Dachwan Lho, Shinyoung Park, Gapycol Park, Kyungjun Cho, and Joungho Kim. Reinforcement learning- based optimal on-board decoupling capacitor design method. In 2018 IEEE 27th Conference on Electrical Performance of Electronic Packaging and Systems (EPEPS), pp. 213-215, 2018. doi: 10. 109//09/PPPS.... 2018 8534195.
Hyunwook Park, Junyong Park, Subin Kim, Kyungjun Cho, Dachwan Lho, Seungtaek Jeong, Shiny- oung Park, Gapyeol Park. Boogyo Sim, Seongguk Kim, Youngwoo Kim, and Joungho Kim. Deep reinforcement learning-bar based optimal decoupling capacitor design method for silicon interposer- based d ics. IEEE Transactions on Components. Packaging and Manufacturing Technology. 1063):467478, 2020. doi: 10. 00.1099TTCMMT.20022277000

---

Under review as a conference paper at ICLR 2023
Hyunwook Park, Minsu Kim, Seongguk Kim, Keunwoo Kim, Haeyeon Kim, Tacin Shin, Keeyoung Son, Boogyo Sim, Subin Kim, Seungtaek Jeong, et al. Transformer network- based reinforcement learning method for power distribution network (pdn) optimization of high bandwidth memory (hbm). IEEE Transactions on Microwave Theory and Techniques, 70(1 70(11):47724786 2022.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pp. 9323-9332. PMLR, 2021.
Madhavan Swaminathan and Engin. Power Integrity Modeling and Design for Semiconductors and Systems. Prentice Hall Press, USA, first edition, 2007. ISBN 97801361
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equiiariann neural networks for 3d point clouds. arXiv preprint arXiv: 1102.08802.0822 000000...22....... 2018.
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. arXiv preprint arXiv:2202.08450 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz. Kaiser, and Illia Polosukhin. Attention is you need. In I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30, pp. 5998-6008 Curran Asso- ciates, Inc., 2017. URL htttpp::/pprceeeiiiioo neurips wttpp//pppp/////////e/ee / pdf
SE55e243357deee110 FFT622255776e51100000ccccaaaaaa2
Oriol Vinyals. Meire Fortunato, and Navdeep Jaitly. Pointer networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL
neur ips. cc/
Zhifei Xu, Zihao Wang. Yin Sun, Chulsoon Hwang. Hervé Delingette, and Jun Fan. Jitter-aware economic pán optimization with genetic algorithm. IEEE Transactions Microwave Theory
and Techniques, 69(8):3715-3722 2021. doi: 10.1 10.10099MM1120011 13087188.
Ling Zhang, Wenchang Huang, Jack Juang, Hank Lin, Bin-Chyi Tseng, and Chulsoon Hwang. An enhanced deep reinforcement learning algorithm for decoupling capacitor selection in power distri- bution network design. In 2020 IEEE International Symposium on Electromagnetic Compatibility Signal/Power Integrity (EMCSI), pp. 245-250, 2020. doi: 10. 100/EMCSS33922 212.
Zhenxin Zhao and Lihong Zhang. Deep reinforcement learning for analog circuit sizing. 2020 IEEE International Symposium Circuits and Systems (ISCAS), pp. 1-5, 2020. doi: 10. 109/ 1.EAS44731.2020.099
12

---

Under review as a conference paper at ICLR 2023
A DPP ELECTRICAL MODELING AND PROBLEM DEFINITION
This section provides electrical modeling details of PDN and decap models used for verification of CSE in DPP. Note that these electrical models can be substituted by those of your interest. There are three methods to extract PDN and decap models that are also used for objective evaluation; 3D EM simulation tool, ADS circuit simulation tool, and unit-cell segmentation method. For each method, there exists trade-off between time complexity and accuracy. See Table 3] Out of the three methods, we used the unit-cell segmentation method for a benchmark. Simulation time was evaluated using the same PDN model on Intel 17. Note that simulation time depends the size and complexity of the PDN model.
Table 3: Time Taken for an Objective Evaluation of a PDN model described in Appendix A.2
Simulation Method
Time Taken
EM Simulation Tool
10 hours
ADS Circuit Simulation Tool
23.58 sec
A.1 DOMAIN PERSPECTIVE DECAP PLACEMENT PROBLEM
Chip
Voltage Sounce (VRM)
PKC
(a) An example of hicrarchical power distribution net- work (PDN).
VRM
PCB
PKG
Chip
/VV/
AM mYn
Cra
Cwo
(b) Electrical circuit model of the hierarchical PDN in
Ai
Source of Water
City
Apartment
Household
Water storage tanks
(c) Water supply chain from the source to houschold.
Figure 9: Illustration of Hierarchical Power Distribution Network (PDN) analogous to Water Supply Chain.
The development of AI has led to an increased demand for high- peerfrrrfoomeece computing systems. nigh-perffrmance computing systems not only require precise design of hardware chips such as CPU, GPU and DRAM. but also require stable delivery power to the operating integrated circuits. Power delivery has become huge technical bottleneck of hardware devices due to the continuously decreasing supply voltage margin along with the technology shrink of CMOS transistors (Hwang
et al. 2021)
Fig. (a) shows the power distribution network (PDN) consisting of all the power/ground planes from the voltage source operating chips. Power is generated in VRM and delivered through
13

---

Under review as a conference paper at ICLR 2023
electrical interconnections of PCB, package and chip. Finding ways meet the desired voltage and current from the power source to destinations along the PDN detrimental because failure in achieving power integrity (PI) leads to various reliability problems such incorrect switching of transistors, crosstalk from neighboring signals, and timing margin errors (Swaminathan & Engin 2007). Decoupling capacitors (decaps) placed on the PDN allows the reliable power supply to the operating chips, thus improving the power integrity of hardware. As shown in Fig. 9 (b)-(c), the role of decap is analogous to that of water storage tanks, placed along the city, apartment, and household, that can provide water uninterruptedly and reliably. As if placing more water tanks can make the water supply more stable, placing more decaps can make power supply more reliable. However, because adding more decaps requires more space and is costly, optimally placement of decaps is important in terms PI and cost/space-saving.
A.2 PDN AND DECAP MODELS FOR VERIFICATION
Unit-Cell Segmentation Method, The segmentation method (Kim et al. 2010 is simple and fast way to generate approximated electrical models. Because the analysis of the full electrical model using EM simulation is very time-consuming, we divided the full PDN model into smaller unit-cells and constructed the full PDN model using the unit-cell segmentation method. For fast simulation, we used equation-based python implemented segmentation method, illustrated in Fig. 10
Segmentation method was used for generation of PDN model consisting of a chip layer and package layer for verification as illustrated in Fig. 10 (a). The segmentation method was also used for objective evaluation of DPP. When solution for DPP is made, decaps are placed on the corresponding ports on PDN using the segmentation method as illustrated in Fig. 10 (b).
(a) Generation of Chip PDN.
Decap unit ce
(b) Decap Placement.
Boundary Condition::
Vi
Zun
Zup
Pal
V
Zoo
Zng [b]
Zpa
Zpp
(p)
V2
Zyp
Zoa
[ig
[V
Zan
Twded 17
Aup (.mm)) )-m))~10) 7o
[1
Zou (pp
Zun) 7pn
Zed Zmgllpp + Zanu Zagh
Zo
(c) Segmentation Method.
Figure 10: Segmentation Method Implemented for PDN Generation and Decap Placement PDN.
The PDN model we used for verification has a two-layer structure; package layer at the bottom and a chip layer on top of it as illustrated Fig. The PDN was modeled through the unit-cell segmentation method. Package layer was composed of 40 x 40 package unit-cells and chip layer was composed of 10 x 10 (i.e, Nrow x Neot) chip unit-cells. Because the DPP benchmark places MOS type decaps, which are placed chip, ports are only available on chip. Thus, extracted 10 x 10 ports information from the chip layer. See Fig. 14 (a), illustrating the chip PDN divided into 10 x 10 units and each unit-cell numbered.
14

---

Under review as a conference paper at ICLR 2023
Top Layer: Chip
Boltom Layer Package
Package: 20mm x 2tmm (10x 10 cell
(a) Top-View of PDN model.
(b) Side-View of PDN model.
Figure 11: Top-view and Side-view of PDN Model used for Verification
The electrical models of package and chip unit-cells that are used to build the PDN model for verification are described in Fig. 12] The chip layer is composed of 10 x 10 unit-cells, and the package layer is composed of 40 x 40 unit-cells using the segmentation method. The corresponding values of electrical parameters are listed in Table
Wax
Kos Soy hare
Wae
Power Layer Ground Layer
Pay
Cort
10miss
log
Siazheric
10unis
Balanced Transmission Line Model of Chip Unit-Cell.
Was
Asa
Power Layer
Rees
Ground Layer
feee
40 unite
(b) Balanced Transmission Line Model of Package Unit-Cell.
Figure 12: Electrical Modeling of Chip and Package Unit-Cells for PDN Model generation.
Table 4: Width and Electrical Parameters for Chip and Package Unit-Cells used for Verification
Unit-Cell Model
W
R
L
G
C
Chip Package
300
0.26 Ω
22pH 0.25nH
1.2mS
0.77pF' 0.045pF
0.5mm
0.093
5.4pS
We implemented MOS type decap for verification. Decap model and its electrical parameters shown in Fig. 13] As mentioned in appendix Fig. (b), the solution to DPP evaluated using the segmentation method.
Note that these electrical parameters and PDN structures were used as a benchmark. For practical use of CSE, these PDN and decap models can be substiruted by those of your interests.
15

---

Under review as a conference paper at ICLR 2023
Cuoap O.1nF
Raccas 1 43.6m/2
Figure 13: Decap Unit-Cell with the Electrical Parameters used for Verification.
A.3 INPUT PROBLEM PDN AND OUTPUT DECAP PLACEMENT DATA STRUCTURE
31
36
41
47
Rgy Riy
B1
22
85
66
S7
30
Kax Reg
Kay Ked Feg
S1
33
95
Probing part
Keqp-Ou Regien
Deog
((0,0)) 1.0002 (37.) (RROL D.... (38.0)
Soutton::
33,33,77 59.61.6.22 62.68.92.977
(a) Input Problem PDN.
(b) Output Decap Placement Solution.
Figure 14: Illustration of how the DPP problem with specific condition is given an input and decap placement solution is generated as an output.
Each unit-cell (i.e, port) of the PDN model described in Appenddi A.2 is represented as set of 3D feature vectors composed of x-coordinate, y-coordinate and port condition; representing keep- out region, 2 representing probing port and O for the decap allowed ports. Total 10 x 10 (i.e, Nrow X (ool) 3D vectors represent the problem PDN. The solution to DPP is the placement decaps. As illustrated in Fig. 14 (b), the solution given a set of port numbers corresponding each decap location.
A.4 RANDOM PROBLEM GENERATION OF DPP
To randomly generate decap placement problems (DPPs) with distinct conditions for training, test and validation, a probing index probe is selected randomly from a uniform distribution of (1,..., Nrow x Neot). Then keep-out region indices keeepott randomly selected through the following two stages: the number of keep-out regions |[ccepout| is randomly selected from a uniform distribution of 0 ~ 15. Then, set of indices of keep-out ports Ikeepout is generated by random selection from the uniform distribution of (1...., Nrow x Neot ). We generated 100 test problems and 100 validation problems for 10 x 10 PDN and 50 test problems and 50 validation problems for 15 x 15 PDN. We made sure the training, test, and validation problems do not overlap.
B EXPERT LABEL COLLECTION
We used a genetic algorithm (GA) as the expert policy to collect expert guiding labels for imitation learning. GA is the most widely used search hcuristic method for DPP (Erdin & Achar 2019 de Paulis et al. 2020 Xu et al.. 2021 Juang et al., 2021). We devised our own GA for DPP, the objective of which is find the placement of given number (K) of decaps PDN with probing port and 0-15 keep-out regions that best suppresses the impedance of the probing port.
M is the number of samples to undergo objective evaluation to give the best solution. The value of M is defined by the size of population Po times the number generation G. K refers to the number of decaps be placed. Pattte is the number of elite population.
Notations.
16

---

Under review as a conference paper at ICLR 2023
Guiding Dataset. To generate expert labels, guiding problems generated in the same way test dataset was generated. We made sure the guiding data problems do not overlap with the test dataset problems. Also, made each guiding problem does not overlap with each other. Each guiding data problem goes through the following process described in Fig. 15 to collect the corresponding expert label.
Initial Population (Po) Generation (x = 1)
Best Half Selection
Elitism
Crossover
x=x+1
Mutation
No
Is G?
Yes
Select Best
Figure 15: Process Flow of Genetic Algorithm for DPP.
comopending
20%
12 D9 25
Sacol
(a) Initial Population Generation.
Elitism.
Numberr Disogeatt
Poxtation UNO
Crosovel
00
Hal Catee Zundia Fostane
Munaee Ovcrlaginn Accken
Maliidd Posdatin
11 98
12 81
(NV)
XP
(c) Crossover.
(d) Mutation.
Figure 16: Illustration of each GA Operators used for DPP Guiding Data Generation.
Population and Generation. For GA {M- = 100} (expert policy), we fixed the size of population as Po 20 and the number of generation as G = 5, which makes up total number of samples to be = P. x G = 100. Each solution in the initial population is generated randomly. As described in Fig. 14 (b). each solution consists of K numbers, each representing a decap location PDN. Note that each solution consists random numbers from 0 to 99 except numbers corresponding to probing port and keep-out region locations.
17

---

Under review as a conference paper at ICLR 2023
Once the initial population is generated randomly, population is generated through elitism, crossover, and mutation. This whole process of generating a new population makes one generation; the Generation process is iterated for G times.
Elitism. Once initial population formulated, the entire population undergoes objective evaluation and gets sorted in order of objective value. The size of elite population is pre-defined as Palite 4 for GA {M = 100) (expert policy). That means the top 4 solutions in the population become the elite population and are kept for the next generation.
Crossover. Crossover is a process by which new population candidates are generated. Each solution of the current population including the elites is divided in half. Then, described in Fig. 16 (c), half the solutions on the left and the other half on the right go through random crossover for Po times to generate a new population. If the elite population available, Po Pelite random crossover takes place so that the total population size becomes Po. including the elite population.
Mutation. According Fig. (d), there may exist solutions with overlapping numbers after the random crossover. We replace the overlapping number with randomly generated number, and we call this mutation.
Select Best. When G is reached, the final population is evaluated by the performance metric. Then, solution with the highest objective value becomes the final guiding solution for the given DPP.
The guiding problems and corresponding solutions generated as a result of GA are saved and used as guiding expert labels for imitation learning.
C DETAILS OF NEURAL ARCHITECTURE DESIGN
Our neural architecture has the AM (Kool et al. 2019) with context modification. The AM is transformer( Vaswani et 2017)-based -ecoder model designed to solve combinatorial optimization problems. We used conventional notations from transformer (Vaswani et al. (017)) and AM (Kool et al. 2019), including multi-head attention (MHA), feed forward (FF), query, key and value (Q, K, V). Because their terminologies are well organized, we tried to keep every notation as possible. In this paper, we focused on presenting the main differences between AM and our architecture. See Kool et al. (2019) for detailed mechanism of AM.
C.1 CHANGE NOTATIONS.
There are small revisions made from Kool et al. (001). In AM, TSP nodes are presented as Ii, i E {1.... N). where N refers to the number of TSP nodes. This paper uses Iprobe for the node of the probing port, Ikeepout for nodes of the keep-out regions and Lallowed for nodes of the decap-allowed ports.
Kool al. 2019 denotes action π (for representing permutation action), but we denoted action as a.
In. Kool et al. 2019), the notation, (M), refers to N times MHA in encoder, denoted this notation as h just for readability.
There two additional notations: Cprobe is the probing context embedding from the probing port context network (PCN in section 3.2 and Ca,-1 is the recurrent context embedding from the recurrent
context network (RCN in section 3.2 for step
C.2 HIGHLIGHT OF MODIFICATIONS: CONTEXT EMBEDDING.
The main difference between the AM and ours is the context embedding and is illustrated in Fig.
AM's Kool et al. 2019) context embedding is presented as follows:
h(e) = M A A(h()) ha.-1, ha
Context embedding of AM. Since the AM was originally designed for TSP and its invariant problems, AM's context embedding is implemented for capturing the entire graph by taking the average of all
18

---

Under review as a conference paper at ICLR 2023
Original AM Encoder
Modified AM Encoder (ours)
pprobe
Agrobe
prebee
Hprobe
Ahcepout
Preepoot
Reeppont
MHA
Rgeepaur
Lattowed
Mallowed
Ancrage
Tattowedd
Matloune
Probing
Gantest Nutwork
high Graph
Sarake Probing Coatiaa
Embaiding
Emthedding
Original AM Decuder (t-Z)
Modified AM Decoder ((-z) (ours)
Ryrobe
pershe
Ikeepputt
Componiiayy
Pala.15,) Pribuiiity
Mkeepend
Compathiny
Pelauis) Poobability
MHA
Mastago Pasiog
MHA
Naltowed
Ratloved
Prrting
Phayy
Phay
Pay N 1--
Attantion qaary
Bey-
[Fa, Cgnahee
Altetiin qarr
Contest
hi
Figure 17: Overview of main difference between AM and modified version of AM.
node embedding. (g)). tratsstraniition with ha--1 and final destination with hay Note that TSP is a routing problem, where it must return to the first node (i.e, destination node is first visited node).
Context embedding of AM for DPP (AM-RL (Park et al., 22022). Park al. (2022) also used the AM for decap placement with modification of context embedding. Park et al. (2022) tried to add hprobe. to capture the location of probing port follows:
h(c) M AH(h()) ha.-1. tep,
Context embedding of Ours. We observed that (g)) degrades the performance of the model for DPP. DPP is different from TSP: need new DPP-specific context embedding strategy. Therefore. we tried to focus on the probing port more than others by proposing the PCN. We removed h(()) and ha, from the context embedding and replaced them with our newly designed context embedding. Our context embedding is described as follows:
h(c) MH A (Cprobe + h)
(9)
Cprobe = MIP PCP (hprooo)
(10)
Ca MLP RCN (hai.))
(11)
Note that both MLP and MLP two-layer perceptron models with ReLU activation, where input and output dimensions are identical (d 128 in experiments).
C.3 CALCULATION OF PROBABILITY.
Probability calculations using context hidden embedding h (c)- and PDN hidden embedding h, i E (1,.., Nrow X Neat) in (11-14) are exactly identical to (5-8) in Kool et al. (2019) except the masking mechanism in equation 13 and equation 14, Because Kool et al. (2019) solves TSP, so they mask the previously selected actions by forcing
compatibility M(c)-- For DPP, mask not only the previously selected actions d1 but also the probing port index prote and the keep-out
region indices Keepoutt is forbidden choose the Iprabe. Keepout and previously selected actions
a1a-1
Query, key and value computed by:
g. W k; WKh,
h;, v;
W
(12)
19

---

Under review as a conference paper at ICLR 2023
Note that W W and WV are 128-to-128 linear projections.
After that, compatibility U[e) is computed by the dot product of query and key, with masking mechanism (setting not to select actions in 8t-1).
4 kg V128
j $ probee keeeoout a1:f-1
(6)
(13)
00
otherwise
The tanh clipping is done following Bello et al. (016)) and Kool et al. (2019).
10 tanh
if j & Iprobe Ikeeport 01:1-1
(c)
v128
(14)
00
otherwise.
Finally, probability can be computed using softmax function as follows:
e ((o)
Po (a+ = il 8t) = Σ, e4
(15)
D DETAILED EXPERIMENTAL SETTINGS
This section provides detailed experimental settings for main experiments and ablation studies.
D.1 TRAINING HYPERPARAMETERS.
There are several hyperparameters for training: we tried to fix the hyperparameters as Kool al. (2019) did for showing their frameworks' practicality. We then provided several ablation studies on each hyperparameter to analyze how each component contributes performance improvement.
Training hyperparameters are set to be identical to those presented in AM for TSP (Kool al. 2019) except learning rate, unsupervised regularization rate X, the number of expert data N, number of action permutation transformed data per expert data P and batch size B.
Table 5: Hyperparameter setting for training model.
Hyperparameter
Value
learning
0.00001
8x1032
1000
P
3
B
1000
D.2 IMPLEMENTATION OF ML BASELINES.
There are two main ML baselines, Arb-RL (Kim et al. 2021 and AM-RL Park et al. 2022).
Arb-RL. Arb-RL is PointerNet-based DPP solver proposed Kim al. 2001 However, reproducible source code was not available. Therefore, we implemented the Arb-RL following the implementation of Bello with batchsize B 100 that makes total 160, 000 PI simulation.
(2016) and paper of Kim al. (2021). We set the training step 1, 600
Arb-IL. Arb- IL is an imitation learning version of Arb-RL trained by our training data. We set N 2000, B 1000 for training Arb-IL.
20

---

Under review as a conference paper at ICLR 2023
AM-RL. AM-RL is AM-based DPP solver proposed by Park al. (2022) We reproduced AM-RL
by following implementation of Kool et al. 2019 and paper of Park et al. (2022). We set the training step 2, 000 with batchsize B 100 that makes total 200, 000 PI simulation.
AM-IL. AM-IL is an imitation learning version of AM-RL trained by our training data. For experiments in Table 1, we set N 2000 and B 1000 for training. For ablation study, we mainly ablate N, when = 100 we set B = 100. Here is the training sample complexity (the number PI simulations during training) of each ML baselines and CSE:
Table 6: Training sample complexity of ML baselines and CSE.
Methods
The Number of Pl simulations for Training
Arb-RL
200,000
AM-RL
200,000
Arb-II. {
2000)
200,000 (N 2000, M
100 from GA expert)
AM-IL IN 2000]
200,000 (.V 2000, M 100 from GA expert)
CSE (N 100 (ours)
10,000 (NV 100, M
100 from GA expert)
CSE 1000) (ours)
100,000 1000, 100 from expert)
CSE
2000) (ours)
200,000 (NV 2000,
100 from expert)
During the inference phase, each learned model produces greedy solution from their policies (i.e., M = 1) following (Kool et al. 2019).
21

---

Under review as a conference paper at ICLR 2023
D.3 IMPLEMENTATION META- META-HEURISTIC BASELINES.
Genetic Algorithm (GA), GA {M = 100 and GA {M = } are implemented as baselines. For detailed procedures and operators used for GA, see Appendix.B B. GA (M 100) the expert policy used to generate expert data for imitation learning in CSE. For GA {M 100), the size of population, Po. 20, number of generation, G, is 5 and elite population, Pelite. is For GA {M = 500}, Po is 50, G' is 10 and Petite is 10.
Random Search (RS). The random search method generates M random samples for given problem and selects the best sample with the highest objective value.
12.8
Perforance
12.4
12.2
12.0
Genetic Algorithm
Random Search
11.8
CSE(ourS) M-1
10a
Number of Iterations (M)
Figure 18: Performance of GA and RS with varying number of iterations (M) comparison CSE at M
Fig. 18 shows the performance of GA and RS depending the number of iterations (M). The performance was measured by taking the average of 100 test data solved by each method at each M. GA outperformed RS at every M, and the performance increased with increasing M for both methods. However, the gradient of performance increment decreased with increasing M. On the other hand, our CSE showed higher performance than GA [ = 100} and RS (M = 10, 000} with a single inference
E EXPERIMENTAL RESULTS IN TERMS OF POWER INTEGRITY
The objective of DPP is to suppress impedance of the probing port as much as possible over specified frequency range and is measured by the objective metric, Obj = (ZieiiZintial (S)
(fimat IGHz Performance CSE was evaluated comparison to GA {M 100} (expert policy), GA {M 500}, RS {M = 10, 000}. AM-RL and AM-IL on unseen 100 PDN cases. Each method was asked to place 20 decaps (K 20) on each test.
E. IMPEDANCE SUPPRESSION PLOTS
22

---

Under review as a conference paper at ICLR 2023
Intial Impedarion,
Intaal Impotiane,
WARL (Mb=2)
AMMBL (Pb-1)
AML (10m1)
[M-1)
(Me-200)
(Mo-220)
(M (Mi-302)
No (me- (OQUs)
Ho Ow 100002
Lociouri (kp-1]
107
uuf Trequrrcy (OP
1 Prequency [CRc]]
Test Case 1.
Test Case 2.
Inttal Impedario,
Intial Impostance, (P0w1)
AVALL (Pw22)
AML, [M-11
(N-1)
(R-200)
(M62002)
impoctnne
30F
32"
(c) Test Case 3.
(d) Test Case 4.
Iniiaal impedtaion
Impaetanee
AMMI (M--33
AMA (M-13
X004. (M-13
(M--200)
CA (MM-200)
(M4-200)
(9e220)
(M-10000)
tuhs
RS (H-20000))
imposinns
CSblona (Mo2)
CSelorral (M
107
10:1
mequency (OPG]
Proqucncy [oha]
Test Case
Test Case
Figure 19: Impedance suppressed by each method, GA (M = 100} (expert policy). GA (M -500). RS {M = 10, 0000. AM-RL AM-IL and CSE (Ours) for 6 example PDN cases out of 100 test dataset. (The lower the better.)
23

---

Under review as a conference paper at ICLR 2023
E.2 DECAP PLACEMENT TENDENCY ANALYSIS
(a) GA {M 100
Text Case 1,3coe--11. 11.79
Cose 2. 3..0
Text Cass Soore-13.42
Test 4, 12.2e-12.25
Text Casc Seore 12.13
Text Case 6, Score 12.97
(b) GA (M 500).
Seare
Score-1133
S..00e-1333
Save- 12.12
12.97
(c) {M
10, 000).
Cast 1o00=10.0
Tost Somm-6.3) 61
Tod Scoee=12
Tedt Coc 4, Score
Casc Serno110.7
Tost Scato-6
(d) AM-RL {M 11.
Tell Casc 1..0o--1.72
Toe Casc 2, Scorc Scorp-10.58 10.58
Tox Case 3, Scnne- 12.70
Toxt Case 4, Scorp11190 11.00
Tol Casc Score-10.3 10.51
Tex Clac Scoro-13.1
(c) AM-II (M 1}.
11
CSE(ours) M 1}.
Figure 20: Corresponding decap placement solutions to Fig. 19 by each method. Red represents probing port, black represents keep-out ports and blue represents decap locations.
Fig. 21 shows the decap placement solutions of 6 PDN cases plotted in Fig. 19 The solutions by the heurchtie methods, GA and RS, tend to be scattered while the solutions by learning based methods based
methods, AM-RL,, AM-II. and CSE, clustered. Since
24

---

Under review as a conference paper at ICLR 2023
random generations, they do not show clear tendency. On the other hand, learning based methods based on a policy so that they have distinct tendency in placing decaps.
The role of placing decaps in hardware design is to decouple loop inductance of PDN. In terms of PI, analysis of loop inductance is critical, but at the same time, is complex (Farrahi & Koether 2019). The loop inductance distribution of PDN highly depends various design parameters such the location of probing port, spacing between power/ground, size of PDN, and hierarchical layout of PDN (Fan al., 2000). When human experts place decaps on PDN, there too many domain rules to consider. On the other hand, CSE understands the PDN structure and its electrical properties by data-driven learning. According to tig. 21] CSE tends to place decaps near the probing port, which is a well-known expert rule in the PI domain.
E.3 POWER NOISE ANALYSIS ON HBM PDN
Innal Lmpesaase (K 0)
Imperlinee [33
GA (M-100, K-0)
10"10
Frequency (GlL))
(a) Impedance suppression.
5.278mV
Initial Trgeedncee (K of GA (M6-100, X-40)
(aaV)
CSE [M-1,
Vno 10.548mV
-5.27mmV
Time (as)
(b) Initial power noise hefore decap placement.
0.341mV 0.305mVV
0.2
4n 0.810m/
K. -0.62mmV
0.2
100,
CSE (M=1, K-20
(16)
-0.300mV -441 mV
(c) Power noise after decap placement by our CSE = 1, K 26} and GA(M = 100, K = 403.
Figure 21:
Power noise analysis in terms of simultaneous switching noise (SSN) on HBM PDN before and after decap placed by our CSE M = 1, K = 26} and GA(M = 100, K = 40}.
Appendix E.3 analyzes the performance of CSE in comparison to GA[M = 100) in terms of power noise. Out of 100 test cases on HBM PDN, we randomly chose test case and carried out peak-to- peak power noise analysis for circuit block, phase locked loop (PLL), operating at 5GHz. Note that CSE placed 26 decaps and GA M 100) placed 40 decaps. CSE reduced power noise more than GA 100) with 14 less decaps. The impedances of the probing port power distribution network (PDN) before and after decap placed by CSE and GA| = 100 are presented in Fig. 21a The time-domain power noise before and after decap placement by each method is shown in Fig. 21b For performance comparison, Fig. 21c shows the time-domain power noise after decap placement by CSE and GA ( [M 100).
25

---

Under review as a conference paper at ICLR 2023
F FURTHER ABLATION STUDY
This section reports ablation studies on action permutation invariance and hyperparameters N (number of guiding samples), À (weight of self-exploitation loss term), and (number of permutation transformed labels).
F.1 ABLATION STUDY ON N
N the number of expert labels generated by the expert policy, GA {M = 100}. We ablate N {100, 500, 1000, 2000} with fixed / 3 and = and compare AM-IL baseline for all N. As shown in Table 7 CSE with N 2000 gives the best performance and CSE outperforms AM-IL for all N variations. Performance of AM-IL is saturated N 500 while the performance of CSE continuously increases with the increase N.
Table 7: Ablation study on N for CSE (P = 3, À = 8) and AM-II..
Validation Score
AM-IL [N
100)
11.60 12.98
CSE (ours) NN 100)
AM-IL IN 500)
12.37
CSE (ours) (VV 500)
12.99
AM-IL IN 1000)
12.23
CSE (ours) [ = 1000)
13.09
AM-IL 2000
12.32
CSE (ours) (N 2000)
13.13
13
Validation Score
12
AM-IL (N=100)
(N=100) (ours)
AM-IL (N=500)
10
CSE (N-500) (ours)
AM -I. (N=1000)
9
CSE (N=1000) (ours)
AM-IL (N=2000)
CSE (N-2000) (ours)
200
600
800
1000
Training Step
Figure 22: Validation graph CSE in comparison to AM-II for varying number of offline expert data N {1000 500, 1000, 2000}.
26

---

Under review as a conference paper at ICLR 2023
F.2 ABLATION STUDY ON X
refers to the weight of self-exploitation loss term Lsaff, in the collaborative learning loss L == Expert + ACsety. To set À x Ly be 0.1 ~ 1, we first multiplied 10 to À because the probability of a specific solution extremely small. Then, we ablated for X E {1, 7, 8, 9, 10) (1032 is omitted) with fixed N 100 and P 3. For every X, it prevents overfitting of the model in comparison to the baselines trained only with L Expert (see Fig. 23). According to the Table = gives the best validation scores.
Table 8: Ablation study of À on fixed P and N 100.
(x10#3)
Validation Score
12.96
12.96
12.94
12.96
12.98
12.98
12.97
10
12.96
Only IL,
12.97
13.0
12.5
12.0
Only
lambda:1
Validation
lambda:2
lambda:4
11.0
lambda:6
lambda:7
10.5
lambde:8
10.0
lambda:10
50
100
125
150
175
200
Training Epoch
Figure 23: Validation graph of X 4, 7,8, 10) fixed P 3 3 and N 100.
27

---

Under review as a conference paper at ICLR 2023
F.3 ABLATION STUDY ON P
P is the number of permutation transformed labels per each expert label used for imitation learning- based expert exploitation. We ablate P E {3, 5, 7} with fixed N = 100 and À 8 and compared collaborative symmetricity exploitation (i.e., both expert and self-exploitation) to only expert ex- ploitation training case. As shown in Table give best performances. For every P. (Expert exploitation Self-cxploitation / gives the better performances, indicating self-exploitation scheme well prevents overfitting of training process for sparse dataset.
with (Expert exploitation + Self-exploitation)
Table 9: Ablation study P with and without unsupervised loss term.
Validation Score
Expert exploitation (P 3)
12,97
+ Self- exploitation (A 8]
12.98
Expert exploitation (P = 5)
12.95
+ Self- exploitation (Á 8]
12.95
Expert exploitation (P
12.93
+ Self- exploitation
(À
12.95
13.0
12.5
Validation Score
12.0
11.5
P:3
P:3-lambda:8
11.0
P:5
P:5-lambda:8
10.5
P:7
P.7-lambda:8
0
25
50
75
100
125
150
175
200
Training Epoch
Figure 24: Validation score of P ablation with and without self-exploitation loss term.
28

---

Under review as a conference paper at ICLR 2023
G PROOF OF THEOREM 3.
(+) Suppose that policy Talaa) is AP-symmetric. Then, by the Definition 3.1 =(a/2) = > <{((a))=) for any A, E
Therefore,
b(T:p) = (x) Pa -ttp [tt(a||=)]1] =
(+) Suppose that ((T;p) = O, where (x( > 0,pA(a) > O,PTAP (1) > 0.
Assume that there exist a" E x* X. and f" TAP, such that <(a"(=") =(t(a")<=") Then,
b(T:p) = 2 px(x ) A (1" [[[ - ([(((())=="))) ||| > 0,
Bo
x+(a)|=))111
which results in contradiction. Therefore, x(a/2) <(((a)(=) for any Fa E A, Le, policy π ((aaz) (x) is AP-smmmetric.
H ORDER BIAS MEASUREMENT
This section reports the order bias measurements of AM-II and our CSE. We measured l(x,, p {Uxx, n, NTUlAAP } for sample width 100 and took the average value. As shown in Appenddix H our CSE significantly reduced the order bias, verifying that CSE successfully induced the AP-symmetricity.
Table 10: Evaluation of Order Bias
b( p (1xx n. N.4rAP }
AM-II.
8.70x 10 10
CSE (ours)
1.25 10 28
29


Reviews:
[{"review_id": "uNVtoYVlq8p", "reviewer": "Reviewer_1PDU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for decoupling capacitor placement problem including.", "summary_of_the_review": "An interesting idea for a specific application, but I am not sure if it brings enough novelty and broader applicability.", "strength_and_weaknesses": "Strength:\n- This paper is well organised and easy to follow. The proposed design is thoroughly introduced.\n- Related works are well covered and discussed. \n- Experimental results have backed the claims and suggests good performance in real-world applications\n\nWeaknesses:\n- Compare to \"Collaborative Distillation Meta Learning for Simulation Intensive Hardware Design\" https://arxiv.org/abs/2205.13225, which also based on some collaborative approach and attention model, what extra novelty does this work has brought? \n- Experimental results could be more convincing, Is there any benchmark on the practical application (5.5) comparing against other methods like AM or ArB? On top of the previous comment, why does the proposed work have the same stability test result as the referenced work?\n- A minor note,  figure one has the same branches placement subfigure. It could be my misunderstanding, but if so this can still benefit from more explanation. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oZgoMr_66m", "reviewer": "Reviewer_XtSs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a collaborative symmetricity exploitation framework to train a solver for the decoupling capacity placement problem benchmark. Since hardware design is multi-level and sequential and performance simulation is costly, data-efficient offline learning with good generalization becomes critical. This paper applies CSE to train a DPP solver using a limited number of offline expert data. Expoert exploitation induces symmetricity during the limited learning process and self-exploitation induces symmetricicty during the consistency learning process with self-generated data. ", "summary_of_the_review": "The application in this paper is something not many people are aware of, which may penalize the paper. But the reviewer still feels that a machine learning paper should aim for general benchmarks that many of us are more familiar with. The reviewer understands that there are prior works getting traction by applying old techniques to a new domain (computer systems, circuit design, etc.), but those works did a great job promoting a new field of applications so that more and more researchers can build work on top of theirs. \n\nin order to improve this work, consider:\n- Promoting the application in this paper and make sure more people in this community are aware of it and would love to work on methods to improve the state-of-the-arts. \n\n- Be clear which parts are novel and which parts are compilation of existing work. If the proposed method is novel, be very specific in the introduction and abstract. \n\n- Create more thorough evaluation with more benchmarks and stronger baselines. Explain in details why the baselines are not good enough. ", "strength_and_weaknesses": "Strengths:\n- The leverage of symmetricity in placement problems can effectively improve data-efficiency of training and generalization over changing task conditions. \n\n\nWeaknesses:\n- The task of decoupling capacitance placement problem is not something the general community are familiar with. The proposed method can be very useful for that particular problem, but the generalization is questionable unless it can be evaluated on other tasks or domains. \n\n- The proposed method is not novel as policy gradient based reinforce and self-supervised learning are not new. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Su6vh2mnVB", "reviewer": "Reviewer_PJWA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper has proposed the idea of collaborative symmetricity exploitation (CSE) that merges all AP-symmetric solution trajectories of placement problems into a single merged trajectory. This idea can reduce the search space and is applied to tackle the decoupling capacitance placement problem. Specifically, this paper has devised a neural network model to generate the sequential placement action based on the existing placement state. The proposed approach is empirically compared against reinforcement learning RL or imitation learning (IL) baselines. The proposed approach shows better efficiency than baselines.", "summary_of_the_review": "The idea of utilizing the solution symmetricity for certain placement problems is interesting. The proposed method is sound and has potential to be applied in industrial scenarios. However, more empirical evaluations are required to fully demonstrate the advantages of the proposed method over other baselines. Also, it's good to show some perturbation analysis of hyper-parameters.", "strength_and_weaknesses": "Strength:\n1. The major challenge for solving such placement problems is the combinatorial explosion.  The idea of applying solution symmetricity is interesting and can potentially reduce the problem difficulty significantly.\n2. The writing of this paper is good and the presentation is clear.\n3. The considered problem is theoretically formulated as an MDP so that the similar idea might be easily applicable to other scenarios.\n\nWeaknesses: \n1. In the decap placement problem considered here, the order of placement of each decap is not important, which is assumed in this paper. Whether the CSE idea is still applicable somehow if the order of placement of each element matters?\n2. The evaluation of the efficiency of the proposed method is somewhat quite sparse, only comparison with the AM-IL baseline. With these inadequate evaluations, it's not very convincing that how the proposed approach is more efficient than other baselines.\n3. The novelty of this paper mainly lies in the symmetricity exploitation. The design choices of the recurrent neural architecture are quite similar to typical architectures for tackling such placement problems.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2YIycEnFRVf", "reviewer": "Reviewer_1Sai", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a collaborative symmetricity exploitation framework to solve the decoupling capacitor placement problem. The framework leverages action-permutation symmetry in offline imitation learning to augment data and to improve generalization capability. It can outperform iterative online search methods like GA in DDP with zero-shot inference.", "summary_of_the_review": "The paper provides a novel framework for DDP. The framework is impressive considering its performance and data-efficiency. Some parts can be improved.", "strength_and_weaknesses": "Strength:\n1. A new imitation learning framework for hardware tasks, featuring zero-shot, low requirements for training data.\n2. Experiment results seem good.\n3. Sufficient supplementary materials.\n\nWeakness:\n1. Figure 2&4 are not clear enough. It is recommended to explain the complete process of the entire framework in the main text.\n2. Further explanations should be given on the reason why the CSE policy trained with low-quality offline expert data can produce higher-quality designs if possible.\n3. Typo in Figure 1.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject