Computing Circuits Optimization via Model-Based Circuit Genetic Evolution
Authors: Zhihai Wang
Jie Wang
Xilin Xia
Dongsheng Zuo
Lei Chen
Yuzhe Ma
Jianye HAO
Mingxuan Yuan
Feng Wu
Venue: ICLR 2025 Poster (2025)
Keywords: AI Chips Design
Computing Circuits Optimization
Evolutionary Algorithm
Reinforcement Learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Optimizing computing circuits such as multipliers and adders is a fundamental challenge in modern integrated circuit design. Recent efforts propose formulating this optimization problem as a reinforcement learning (RL) proxy task, offering a promising approach to search high-speed and area-efficient circuit design solutions. However, we show that the RL-based formulation (proxy task) converges to a local optimal design solution (original task) due to the deceptive reward signals and incrementally localized actions in the RL-based formulation. To address this challenge, we propose a novel model-based circuit genetic evolution (MUTE) framework, which reformulates the problem as a genetic evolution process by proposing a grid-based genetic representation of design solutions. This novel formulation avoids misleading rewards by evaluating and improving generated solutions using the true objective value rather than proxy rewards. To promote globally diverse exploration, MUTE proposes a multi-granularity genetic crossover operator that recombines design substructures at varying column ranges between two grid-based genetic solutions. To the best of our knowledge, MUTE is the first to reformulate the problem as a circuit genetic evolution process, which enables effectively searching for global optimal design solutions. We evaluate MUTE on several fundamental computing circuits, including multipliers, adders, and multiply-accumulate circuits. Experiments on these circuits demonstrate that MUTE significantly Pareto-dominates state-of-the-art approaches in terms of both area and delay. Moreover, experiments demonstrate that circuits designed by MUTE well generalize to large-scale computation-intensive circuits as well.

Published as a conference paper at ICLR 2025
COMPUTING CIRCUITS OPTIMIZATION VIA MODEL- BASED CIRCUIT GENETIC EVOLUTION
Zhihai Wang'; Jie Wang", Xilin Xia', Dongsheng Zuo?, Lei Chen', Yuzhe Ma²,
JianYe Hao Mingxuan Yuan?, Feng Wu'
MoEE Key Laboratory of Brain-inspired Intelligent Perception and Cognition,
University of Science and Technology of China
2 Noah's 's Ark Lab, Huawei Technologies
3 Microelectronics Thrust, Hong Kong University of Science and Technology (Guangzhou)
4 College of Intelligence and Computing, Tianjin University
ABSTRACT
Optimizing computing circuits such as multipliers and adders is a fundamental challenge in modern integrated circuit design. Recent efforts propose formulating this optimization problem as a reinforcement learning (RL) proxy task, offering promising approach search high-speed and area-efficient circuit design solu- tions. However, we show that the RL-bbased formulation (proxy task) converges to local optimal design solution (original task) due to the deceptive reward sig- nals and incrementally localized actions in the RL-based formulation. To ad- dress this challenge, we propose a novel model-based ciregit genefic evolution (MUTE) framework, which reformulates the problem as genetic evolution pro- cess by proposing grid grid-based genetic representation of design solutions. This novel formulation avoids misleading rewards by evaluating and improving gener- ated solutions using the true objective value rather than proxy rewards. To pro- mote globally diverse exploration, MUTE proposes multi- granularity genetic crossover operator that recombines design substructures at varying column ranges between two grid-based genetic solutions. To the best of our knowledge, MUTE is the first to reformulate the problem as a circuit genetic evolution process, which enables effectively searching for global optimal design solutions. We evaluate MUTE on several fundamental computing circuits, including multipliers, adders, and multiply-accumuule MUTE significantly Parcto-dominates state-of-the-ar approaches in terms of both area and delay. Morcover, experiments demonstrate that circuits designed by MUTE well generalize to large-scale computation somputation-intensiee circuits as well.
circuits. Experiments on these circuits demonstrate that
INTRODUCTION
Computing circuits such as multipliers and adders serve as the fundamental building blocks in nu- merous real-world circuits, particularly in central processing units, graphics processing units, and artificial intelligence (AI) chips (Holdsworth] 1987; Das et al. 2019: Sze et al.) 2020). The mul- tiplication and addition operations stand out as the most fundamental and frequently utilized arith- metic operations across various
compputttioonintesiiii applications. including deep neural networks (DNNs), digital signal processors, and microprocessors (Hashemian) 2002; Elguibaly 2000 Zuo et al. 2023). Notably, in many popular DNN architectures such as ResNet (He et al. 2016). VIT (Dosovitskiy et al., 2021). Transformer (Vaswani et al.) 2017), and BERT (Devlin et al. 2019), the multiplication and addition operations constitute over 99% of all operations. Therefore, the design of high-speed and area-efficient computing circuits plays a pivotal role in enhancing the performance of computation-intensiv applications, especially in Al chips.
However, computing circuit optimization is challenging combinatorial optimization problem due to its -hrd nature (Hillar & Lim 2013 Song et al. 2022). On one hand, the combinatorial
'This work was done when Zhihai Wang was an intern at Huawei Noah's Ark Lab.
1 Corresponning author. Email: jiewangx @ustc.edu.cn.

---

Published as a conference paper at ICLR 2025
design space grows exponentially with the input bit widths of the computing circuits (Roy et al. 2021]. On the other hand, evaluating the pyst-synthesis performance of a circuit design (i.e., design performance) with circuit synthesis tools is highly consuminn,, leading to high sampling costs. Therefore, searching high-speed and area-efficient circuits in the vast design space using limited samples emerges significant challenge.
To search high-speed and area-efficient circuits, recent efforts (Roy et al. 2021 Zuo et al. 2023 Song et al. (022) propose formulating the computing circuits optimization problem as a reinforce- ment learning (RL) proxy task, offering a promising avenue for optimizing circuit designs using limited samples. Specifically, they start from an initial design solution, learn policies to incremen- tally modify the local design structure, and utilize design performance gains between two consec- utive designs as reward signals. Intuitively, the manually designed rewards can guide RI. agents to explore directions that progressively improve design performance at each step.
However, we show that the RL-based formulation (proxy task) converges to a local optimal de- sign solution (original task) due to the deceptive reward signals and incrementally localized actions. First, the reward signals based on performance gains between two consecutive designs are deceptive, as maximizing the cumulative discounted rewards misaligns with the true objective. More specif- ically, the proxy RL formulation indeed optimizes the cumulative discounted performance of all encountered design solutions across a trajectory, while the true objective is to find the single best- performing designs. Second, the actions based on the incrementally local modifications of design structure suffer from poor exploration capability, and thus struggle to escape local optima.
To address these challenges, we propose a novel model-based circuit genetic evolution (MUTE) framework, which proposes grid-based genetic representation of solutions and reformulates the problem as a circuit genetic evolution process. The evolution formulation is an iterative pro- cess between circuit genetic variation and model-based selection, where each iteration evaluates and improves solutions using the true objective value, thus gradually converging toward the best- performing solution (i.e., the original task). To promote globally diverse exploration for escaping local optima, MUTE proposes a multi-granulan granularity crossover operator that recombines design sub- structures at varying column ranges between two grid-based genetic solutions. Moreover, to tackle the problem of high sampling costs, MUTE introduces model-based selection method, which learns a model for rapid evaluation a large number of solutions.
We evaluate MUTE on several fundamental computing circuits, including multipliers, adders, and multiply-accumulate circuits. Experiments on these circuits, spanning a wide range of input widths, demonstrate that MUTE discovers state-of-the-an designs that significantly Pareto-dominate those produced by manual design, mathematical optimization, and learning- based approaches, improving the hypervolume by up to 38%. Moreover, we deploy circuits optimized by MUTE and the base- lines into large-scale computation-intensivy circuits, and experiments show that MUTE significantly outperforms the baselines in terms of both area and delay. Our results highlight the superior abil- ity of MUTE to discover high-speed and area-efficient circuits for real-world important computing applications, especially for high-performance AI chips.
We summarize our major contributions as follows. (1) We show that the RL-based formulation for computing circuits optimization converges local optimal design solution, indicating a signif- icant objective gap between the RL-based formulation and the true objective.
(2) To the best of our knowledge, our MUTE the first to reformulate the optimization problem as a novel circuit genetic evolution process, which enables effectively searching for the global optimal circuit design solutions. (3) MUTE proposes a multi-granularity genetic crossover operator to promote globally diverse exploration of the design space. (4) Experiments show that MUTE significantly outperforms state-of-the thett-rrtttt approaches in terms of both area and delay.
BACKGROUND
2.1
COMPUTING CIRCUITS ARCHITECTURE
Most computing circuits such as prefix adders, vector adders, subtracters, multipliers, and multiply- accumulate circuits rely on two fundamental circuit structures, i.e., the Compressor Tree and Prefix Tree WWeste& & Harris 2015) Roy et al.) 2021 Zuo et al. 2023 Wang et al. 2024g). Note that the Compressor Tree and Prefix Tree both share similar tree structures that can both be represented by grid-based design solutions. We take multiplier circuit with four input bits as an example in-
2

---

Published as a conference paper at ICLR 2025
multalicand nuspper
Compresee Crtd
Pietie Tricket Gesenior
Cary Sex. Ahal
Stages
Stage 01110 012000000
Stape 2 310 11310/00
Comydds
Campusor Grid
Skge 0 0 0000i00
Compressor Tree
Cem 22 cnpprso Si Aall
Skag2
0
Compressed
The combore of 32 pxperssccs
Products
Oocpresn Faceds
wi ruggle
Casy Poosayaa Adke
Grid-Based Design
Products
Pizal
10000900
Basie E component
Compressor Tree Structure
Solution Representatio
(a) Multiplier Architecture
(b) Compressor Tree
Figure 1: An illustration of the multiplication process and multiplier architecture.
troduce the Compressor Tree structure as shown in Figure In binary multiplication, two unsigned binary numbers-tth multiplicand and the multiplier- are combined to yield their product. Contem- porary multiplier designs typically comprise three primary components: a partial product generator (PPG), a Compressor Tree, and carry propagation adder (CPA). Initially, the PPG generates bit matrix based on the multiplicand and multiplier, with each element representing a partial product. Subsequently, the Compressor Tree compresses each column of the bit matrix one or two bits by concurrently summing up the partial products within each column. Finally, the CPA aggregates the resultant bit matrix from the Compressor Tree to derive the final product.
In constructing a Compressor Tree, a large number of full and half adders are typically employed to execute the summation of generated partial products concurrently. A full adder, i.e., 3:2 compressor, accepts three inputs-two single-bit values and a carry-in bit-and produces two out- puts: sum bit and a carry-out bit. half adder, i.e., 2:2 compressor, takes two single-bit values as inputs and yields two outputs: sum bit and a carry-out bit. Notably, when 3:2 (2:2) compressor is applied to the i-th column, reduces two (one) bits in column while increasing one bit in column (i+ 1). Thus, Compressor Tree employs numerous compressors (i.e., full and half adders) across multiple stages to compress the partial products matrix into only two rows in parallel, significantly dominating the final performance of multiplier circuit. Moreover, modifying the arrangement of 3:2 and 2:2 compressors within a Compressor Tree can result in significantly different tree structure designs, leading to variable design performance.
2.2 RL FOR COMPUTING CIRCUITS OPTIMIZATION
As the Compressor Tree and/or Prefix Tree usually dominates the final performance of a comput- ing circuit (Zuo et al. 2023 Xiao et al. 2021), recent efforts have focused on optimizing the tree structure by formulating the optimization problem
reinforcement learning (RL) proxy task (Zuo eet al. 2023 Roy et al. (02T). We take the existing RL-based Compressor Tree optimization method as example. RL-MUL starts from an initial Compressor Tree design solution, learns policies to sequentially modify the design structure locally, and utilizes design performance gains as reward signals. We specify the state space, action space, and reward function follows. (1) State Space S. RL-MUL formulates each legal design solution as a state, where each state is represented by grid-based image. (2) Action Space A. RL-MUL designs four types of local modifications Compressor Tree solution at certain column. These local modifications include adding 2:2 com- pressor, removing a 2:2 compressor, replacing 3:2 compressor with a 2:2 compressor, and replacing 2:2 compressor with 3:2 compressor. The action space is discrete set composed of 4 x Nc dis- crete actions, where Nc denotes the number of columns. Each action i [0. I...., (4 x Nc I)] is represented by executing the j-th modification type the k-th column, where j i (mod 4) and k = L. (3) Reward Function r. RL-MUL uses a circuit synthesis tool to obtain the performance of the designed solution at each step. The reward r, is defined as the difference between the area (delay) of the design at step /-1 and that at step 1. That is, r(St, 42 Nt+1) f(xx) f(),+),, where f denotes the design evaluation function. Finally, RL-MUL. leverages the deep Q-network algorithm
Mnih et al., 2015) to train Q-networks. We defer details to Appendix D
3 LIMITATIONS OF EXISTING RL FORMULATION
3.1 DECEPTIVE REWARD SIGNALS
Existing methods formulate the optimization problem as an infinit-hhorizon Markov decision pro- cess (MDP) denoted by tuple (S, po), where S denotes the state space, A denotes the
3

---

Published as a conference paper at ICLR 2025
denotes the reward function, 7 denotes the deterministic transition function, y denotes the discount factor, and oo denotes the given initial design solution. Based on the MDP, the return of deterministic policy is defined as R E yrr(,,, a, St+1), where so = fo. de = (8)), and Sp+1 = at). Note that the reward is defined by the performance gain between the states St and S1+1, i.e.,
action space,
a, 5141) = f(s,) - (()+1).
(I)
Here f: - R denotes the underlying evaluation function of design solutions given by circuit synthesis tools. Note that multiple evaluation functions are employed, such as area and delay evalu- ation functions. For ease of analysis and consistent with previous work (Roy et al. 2021 Zuo et al.) 2023), we assume that the evaluation function f is a linear weighted average of these evaluation functions. Intuitively, the manually designed proxy rewards based on performance gains are able to guide RL agents toward directions that progressively improve design performance, as the RL agent receives positive rewards for improving design performance. Thus, desired question is: Does the optimal policy in the RL formulation converge to the global optimal design solution?
To investigate this question, we first theoretically show the RL-based optimal policy converges to a local optimal design solution. Then we empirically show that the underlying evaluation function is highly oscillatory, resulting in the local optimal design solutions found by the optimal policy diverging significantly from the global optimal solution.
Theoretical Analysis We assume that the state space S is finite. For simplicity, we assume a ter- minal action for each state that can terminate the episode at this state. We define state E S as a local optimum of the function f for all action E A we have f(T(s, 2 f()).
Theorem 3.1. The optimal RL policy x² arg max, RR terminates at a state, and the state is local optimal design solution of the evaluation fincctin f.
This theorem demonstrates the superior capability of RL methods in achieving local optimal solu- tions. However, this raises a further question: Is the converged local optimal point also the global optimum? Given the lack of detailed information about the optimization objective function f. rigorous analysis of this problem is currently infeasible. Therefore, we present an intuitive and em- pirical analysis to demonstrate that the converged local optimal solution can significantly diverge from the global optimal solution as follows.
Illustrative Example We revisit the optimization objective in the RL formulation, i.c.,
8
RT Σ 1=0
-(())
Σ (1--Y) ((SH)) 1=0
(2)
This implies that standard RL. methods in the existing formulation aim to minimize the cumulative discounted performance of all visited solutions across a trajectory, except the initial state, when the discount factor y < 1. This is practical discount factor setting in standard RL and previous methods (Roy et al. 202T Zuo et al., 2023). In contrast, the circuit optimization task is a best-case-seeking task, Le., the final performance is measured by the single or few best-performing design solutions found during training. Consequently, the RL-based optimization objective is inconsistent with the original optimization objective, possibly leading significant optimization objective gap.
To illustrate the optimization objec- tive gap problem, we provide mo- tivating example as shown in Fig- ure 2 (Left). Specifically, we il- lustrate two distinct trajectories in the circuit optimization environment from a given starting solution fol- lowing two deterministic policies æ and πT.. We denote the two trajec- tories by (s "a s .t 21
Trejstories Twn Policiss
Training Curve on 64-bit Booth Nuttppier
ream--d3 mi-d.
430
Waightd
Weighted
$20
$10
400
5000
2000
3000
4000
5000
Stops
Steps
aa .7.....77 a.aa.. spectively. Eaah point in Figure 2
s s
Figure 2: (Left) A motivating example of two distinct tra- jectories with confficting returns and found best solutions. (Right) A practical training curve of the EA method.
and (s "0 "sm"a" S7....5))
re-
(Left) corresponds to the performance of a state across the trajectory. As shown in Figure 2 (Left),
the return of the policy πa is larger than that of the policy æi, while the best solution found by
4

---

Published as a conference paper at ICLR 2025
Population Initialization
Actin
ligett
Pree
Pepubaton? ?
Initialio
Populations
Seitenn
Salitee
Update
Sequential Mutation Operator
Hopplay Mimary
Geai Crosawer
Greec
Toage Stae
Trals
Synthenis Ezv
Pize Select
Coung
Evalustion Model
Colazn
Gouog
Bub Staxtox
Gri--lowwd The Stradtro Eepreoerattonn
Geneth
Offigring Solations
Coun)
Multi-Crambrity Crosover Operator
Genetic Formolation
Madal-RRose Cascada Ranking
Clicaitt Variation Operaters
Figure 3: An illustration of the MUTE framework with a) Genetic Formulation, b) Population Ini- tialization, c) Circuit Genetic Variation Operators, and d) Model-Based Cascade Ranking.
is better than that found by æ²2 This illustrates a significant optimization objective gap between the RL-based formulation and the original true objective.
3.2 INCREMENTALLY LOCALIZED ACTIONS
Non-smooth Objective Functions We further empirically investigate the properties of the objective function f. As the domain of f is the high-dimensional space S, directly visualizing the landscape of the objective function is challenging. Instead, sample a large number of diverse state points from the state space to approximate the function's 's behavior. Specifically, we visualize the training curve produced by a simple evolutionary algorithm (EA) that uses random actions from the RL action space a E A to repeatedly perturb the current best solution locally (see Appendix [E] for details.) As shown in Figure (Right), the sampled function values exhibit significant oscillations, indicating that the underlying objective function is highly oscillatory as well. The major reason for the oscillations of the evaluation function stems from the complex optimization mechanisms employed by circuit synthesis tools. Even minor modifications to the circuit structure can result in substantial performance variations when evaluated by these tools. Consequently, the oscillatory nature of the optimization objective results in numerous local optimal solutions.
Limited Exploration Ability of Localized Actions The oscillatory nature of the objective functions results in numerous local optimal solutions, thus requiring globally diverse exploration of the design space to escape local optima. However, the actions in the existing RI. formulation are limited to local modifications of design structure at a certain column, severely constraining the agent's ability to explore diverse or distant regions of the search space. As a result, the search process may become confined to suboptimal regions, limiting the chances of discovering global optima.
4 A MODEL.-BASHE CIRCUIT GENETIC EVOLUTION FRAMEWORK
We start with an overview of our proposed MUTE in Section 4.1 Next, we outline the formal procedure at the core of MUTE, specifying the circuit genetic evolution formulation and population initialization in Section 4.2 our proposed efficient and effective circuit genetic variation operators in Section 4.3 and model--ased cascade ranking for selection in Section 4.4
4.1 OVERVIEW OF OUR FRAMEWORK
We provide an illustration of our MUTE in Figure 3 To bridge the gap between the RL-based formulation and the original task, we design grid grid-based genetic representation of solutions, and reformulate the computing circuits optimization problem as circuit genetic evolution process. The evolution formulation an iterative process between circuit genetic variation and model-based se- lection, where each iteration evaluates and improves solutions using the true objective value, thus bridging the objective gap. First, we propose learning-based population initialization method to accelerate the evolution process by leveraging the existing RL methods to generate population of nigh-performing design solutions. Then, we propose efficient and effective genetic variation oper- ators to avoid redundant exploration and promote globally diverse exploration. Finally, to further improve sample efficiency, we propose a model-basee -asee cascade ranking method for efficient selection from large number of generated offspring design solutions.
5

---

Published as a conference paper at ICLR 2025
4.2 GENETIC EVOLUTION FORMULATION AND POPULATION INITIALIZATION
Circuit Genetic Evolution Formulation For computing circuits such adders and multipliers, we can represent each circuit design solution by a grid of numbers. We take a compressor tree design solution as an example. As the total number of different full and/or half adders at each column implicitly encodes the structure of a compressor tree, it significantly impacts the post-synthesis performance of multipliers (Xiao al. 2021: Zuo et al. 2023). Specifically, we employ a 2 x Nc grid to represent the compressor tree design solution, where two rows denote the total number of full and half adders across Ne columns, respectively.
To enable circuit genetic evolution, we propose to formulate each element in the grid- design solution representation a circuit gene, and thus the grid as genetic individual. Note that the existing RL-based method can formulate the grid-based design solution as the state representation as well. Thus, the grid-based genetic formulation allows us seamlessly incorporate learning into our genetic evolution framework, which can significantly improve sample efficiency.
To directly optimize the original circuit design performance, we formulate the fitness function as the underlying design evaluation function f given by circuit synthesis tools. Thus, the optimization objective of our circuit genetic formulation takes the form of arg min f(s), where S denotes the set of all possible legal design solutions.
Learning-Basee Population Initialization Although the performance of existing RL methods is limited by deceptive reward signals, they can efficiently converge to a local optimal design solution. Thus, to speed up the evolution process, we incorporate the learning method into our population ini- tialization to generate set of serh-perfrrmin circuit design solutions. During the learning process, we maintain an elite pool of N best-performing design solutions, and progressively update the pool at each training episode. Finally, we use the elite pool as initial population.
Following previous work (Song et al. 2022 Zuo et al. 2023 Wang et al. 2024g). we use a scalarized version of the Deep Q-network (DQN) algorithm (Mnih et al.] 2015) to learn solution modification policies. We maintain a progressively updated elite pool, and each episode starts the environment with a design solution so sampled from the elite pool. Every action a, from the Q-network Qo modifies the design solution S; to another design solution S4+1, and returns a weighted reward that indicates the decrease in the normalized circuit area and delay. That is,
r, a (area( s,) (s))(s))))))>))))))))))))))))( wa (delay (s,) delay(s)+1)).
After each episode, we insert each design solution at this episode into the elite pool when its design performance is better than the worst solution in the current elite pool.
4.3 EFFICIENT AND EFFECTIVE GENETIC VARIATION OPERATORS
Designing efficient and effective genetic variation operators is important for the success of genetic algorithms (Zhu et al. 2023a Bai et al., 2023b Li et al. 2024a). Thus, we propose a sequential mu- tation operator for efficient long- term exploration via sequential modifications of design solutions based on the Q-network learned during initialization. We propose a multi-granularity crossover op- erator for globally diverse exploration by recombining two genetic solutions across diverse granu- larities. By using these genetic variation operators, we can generate a large set of offspring solutions from a population of design solutions at each iteration. We present details as follows.
A Sequential Mutation Operator for Efficient Exploration To explore the design solution space, common idea is to perform random local modifications on design solutions. However, the random modification strategy can lead to redundant and myopic exploration, thus resulting in low sample efficiency and sub-optimal solutions. To promote efficient long- terg-term exploration, we propose a se- quential mutation operator to perform sequential local modifications on a given solution based on the learned Q-network. Specifically, for design solution so sampled from the current generation of populations, we leverage the learned Q-network Qe to guide episodic modifications sign solution by sampling T local modification actions for generating mutated solutions. Thus, we generate 7' mutated offspring solutions by sampling sequence of modification actions from the Q- network, i.e., a, = arg max,, a (s a). = O....,T. T. To prevent premature convergence, we follow the e-greedy strategy (Sutton & Barto 1998) to sample actions.
the de-
This sequential mutation operator offers two key advantages. (1) Guided by the learned Q-function, can prioritize modifications that enhance design performance, rather than relying on random

---

Published as a conference paper at ICLR 2025
modifications. (2) By leveraging the long-term predictive capabilities of the learned Q-function, can identify strong combinations of local modifications, leading to more effective design solutions.
A Muati-Grrnularity Crossover Operator for Diverse Exploration As shown in Figure the objective function f is highly oscillatory, resulting in many local optimal design solutions. However, relying solely existing local modifications severely restricts the exploration capability, as it only incrementally adjusts the number of full or half adders at a single column within a specific design solution. To enable global exploration for escaping local optima, it is crucial develop design variation operators capable of making globally diverse exploration of the design space.
To this end, propose a multi- multi-granularity genetic crossover operator that recombines design sub- structures at varying column ranges between two grid-based genetic solutions. The key ad advantage of this approach its ability expand design variation from the single-gene level the cross-column level. This allows for transitions from small changes involving single column to more signifi- cant modifications that span multiple columns. This significantly enhances our global exploration capability, improving the ability to escape local optima.
Single Gene/Column Crossover Given two high-performing design solutions 5 and st, we view the solutions as a sequence of genes, i.e., 5 -²= = {18]2 (0), (0),, (8922 (N), (N)) and s (0), (g)))).....))))
To recombine the two sish-performing solutions to obtain diverse offspring solutions, the single column crossover randomly selects a column and recombines the genes at that column. Specifically, the column granularity crossover operator generates two children in the following form:
= (.... (g) (i). (1), .-. } and s = f..... (g) (i), (i).
(4)
where i the selected column. Meanwhile, the single gene crossover only recombines the genes representing a certain adder type at the selected column.
Cross Columns Crossover Given two high-peeformiig design solutions 5 and s the cross columns crossover randomly selects two columns and recombines the two grid-based genetic solutions within the selected two columns. Specifically, the cross columns crossover generates two offspring solu- tions in the following form:
(... (g (i). (2).).)))))))))
(1) (2) and
= (.... (1) ()) ................................ ()) (i), ..)....
(5)
where and are the selected two columns. Depending the values of i and j. this crossover recombine substructures across any different columns, thereby promoting global diverse exploration.
Note that the crossover operator could lead to illegal solutions. Thus, we design simple legalization rules following previous work (Zuo et al. 2023) to ensure that these generated children are legal design solutions. Please refer to Appendix F,5 for details.
4.4 MODEL-BASED CASCADE RANKING FOR EFFICIENT SELECTION
The success of genetic algorithms often relies on sampling large number of solutions (Zhu et al., 2023a Bai et al., 2023b Li al. 2024a). However, in circuit optimization tasks, evaluating each design solution using circuit synthesis tools is highly time-consuming, which significantly limits the number of samples for searching. To significantly improve sample efficiency, we propose learning a design evaluation model fe using the collected populations during evolution.
Model Training Inspired by standard model-based RL (Janner et al. 2019 Wang al. 2023c we first train a prediction model using the collected replay buffer during the population initialization process, and then adaptively update the model using few populations with true evaluations during the evolution process. In terms of the model architecture, we employ the ResNet- 18 as the state encoder and a multi-head decoder to predict the area and delay of the input state. The multi-head decoder comprises two multi-layer perceptrons (MLPs), each with two hidden layers with 256 units the
and ReLU activations. In terms of the training details, the model parameters. We use the Adam optimizer with a learning rate of le-3.
squared error loss to update
Model Usage Previous madel-based RL methods Janner et al.) 2019 Wang et al. 2023c) have shown that directly using the learned model to replace the real environment suffers from model ex- ploitation, i.e., overfitting model errors. To address this challenge, we propose model-based
7

---

Published as a conference paper at ICLR 2025
8-bit (And)
16-bit (And)
32-bir (And)
64-bit (And)
0,75
(80)
Delay (n))
Delay (00
Delay (m)
0.70
220--
1.5
0.65
0.60
400
150
300
330
1750
2000
2250
2500
7000
8000
9000
10000
30500
35000
40000
Area (umm²) ")
Area (um"")
Area (umm
Area (um²)
8-bil (Booth)
16-bii (Booth)
32-bil (Booth)
64-bil (Booth)
Delny (03)
0.5-
Delny (es)
Delay (03)
Delny (8))
5.0
2.00
0.7
300 Area (um²) )
60c
1500
1750
2000
2250
6000
7000
5000
20000
22500
25000
27500
Area (um²)
Arrea (um
Area (um²)
Wallace
GOMIL
RL-MUL
AdaResstt
HAVE
(Ours)
Figure 4: The results demonstrate that the multipliers optimized by MUTE consistently and signif- icantly pareto-dominate the designs optimized by all five baselines across eight multiplier design problems, improving the hypervolume by up to 38%.
cascade ranking method to efficiently and accurately select high-performing solutions from large set of generated solutions. The key idea is to progressively select solutions through two-stage rank- ing models, where the first stage model our learned model and the second stage model is the real circuit synthesis environment. Specifically, we primarily use the model to rapidly evaluate the children solutions generated by the designed genetic variation operators. We generate at least 100 children solutions at each iteration, and use the model to pre-rank these solutions. The top-5 solu- tions are then selected for evaluation in the true environment. This approach allows us to generate a substantial number of children solutions, promoting diverse global exploration.
EXPERIMENTS
We first introduce the experimental setup, baselines, and evaluation metrics in Section 5.1 Then, our experiments are designed with four primary objectives. 1) We evaluate the performance of MUTE in optimizing computing circuits, including multipliers, adders, and MACs, across broad range of input widths Section 2) We investigate the generalization performance of multipliers optimized by MUTE to large-scale macros widely-used in real-world AI chips in Section 3) We perform carefully designed ablation studies to demonstrate the importance of our genetic evolution formulation and provide insights into each component in MUTE in Section 5.4 4) We conduct thorough trade-off evaluation of the runtime and performance gains of MUTE in Section 3.5
5.1 EXPERIMENTAL SETTINGS
Experimental Setup Throughout our experiments, we utilize the OpenROAD flow (Ajayi & Blaauw] 2019) alongside the NanGate 45mm open-cell library (Nangate Inc. 2008) for circuit syn- thesis, coupled with OpenSTA (Parallax Software Inc.) for timing analysis. The setting follows previous work (Zuo et al. 2023). These tools represent the state-of-the-art open-source EDA tools, and are widely used in research of EDA (Kahng 2021: Tan et al. 2021 Pilipović al. 2021 Our training procedure employs the Adam optimizer [Ruder] within the PyTorch framework (Paszke et al. 2019). For fair comparison, we controlled the training time of our method to be comparable with the bascline methods (see Appendix H.2). We apply our method to eight distinct multiplier design tasks, encompassing 8-bit, 16-bit, 32-bit, and 64-bit multipliers based on both AND based and Booth encoding- based PPG techniques. Moreover, we apply our method to six distinct adder and MAC design tasks as well. We defer more results to Appendix H
Baselines Our baselines encompass five competitive approaches, ranging from traditional human- designed heuristics to state-of-the-arr (SOTA) learning- methods. 1) Wallace Tree Walllcc 1964) represents classical -dsiinde compression algorithm. 2) GOMIL (Xiao et al.] 2021 is an expert-designed algorithmic method based on integer programming. 3) RL-MUL (Zuo et al,
2023 4) AdaReset (Song et al., 2022), and 5) HAVE (Wang et al. 2024g) three recent SOTA RL- based multiplier and adder optimization methods.
Evaluation Metrics The computing circuits optimization problem is optimization task with multiple confficting objectives, such as area and delay. Thus, we employ two widely-used multi- objective optimization evaluation metrics (Basaklar et al. 2022 Hung al. 2023) compare the
8

---

Published as a conference paper at ICLR 2025
Table 1: The results demonstrate that MUTE significantly outperforms previous SOTA approaches in terms of the hypervolume on both adder and MAC design tasks.
Adder
16-hit
32-bit
64-hit
Methods
Hyper
Improvement
HyperVolume
Improvement(e)
HyperValume
Improvementt
RE-MUL.
88.03
211.76
AdaReset
92.66
5 26
213.71
0.92
$13.77
1.91
HAVE
113.92
29.41
254.09
19.99
566.26
12.48
MUTE (Ours)
116.94
32.84
281.66
33,01
582.73
15.75
MAC
16-hit
32-bit
64-hit
Methods
Hyper Volume
Improvement(S) 1
HyperVolumee
Improvement(4)
HyperValume
Improvemeent))))
RE-MUL
371.10
NA
4114.00
NA
6184.50 50
NA
AdaReser
369.60
-0.40
5123.00
24.53
10212
65.13
HAVE
10601.57
MUTE (Ours)
414.80
11.78
5843.00
42,03
11487.37
K5.74
(And)
32-bit (And)
(Booth)
32-bit (Booth)
2.5
Delay (0s)
Delay (6z)
Delay
12-
2.0
1-
Deloy
2.0
1.5
G000000
700000
2.00
2.25
2,30
2.75
V0000
3000000
330000
600000
1:30
1.55
2.00
Arca (pam²)
Aren (um)")
Area (um²)
Airca (umm))
Wallane
GOMIL
RL-MUL
AdaReset
HAVE
MUTE (Ours)
Figure 5: The results illustrate that PE arrays designed by MUTE consistently and significantly outperform the designs discovered by all five baselines in terms of Pareto-dominance across four multiplier design problems, i.e., 16-bit (And), 32-bit (And), 16-bit (Booth), 32-bit (Booth).
performance of our method with the baselines. First, we visualize the found Pareto points in terms of the area and delay for eircuits designed by both our method and the baselines. Second, we utilize the hypervolume (HV) of the found Pareto points, which is defined by the volume of the region between a reference point and these found Pareto points, We defer more details to Appendix F.7
5.2 MAIN EVALUATION OF OPTIMIZING COMPUTING CIRCUITS
Multiplier Design We highlight the superiority of MUTE through a comparative analysis with five competitive baselines on eight multiplier design problems across wide range of input sizes. The results in Figure 4 demonstrate that multipliers optimized by MUTE consistently and significantly outperform designs produced by all baselines across all eight multiplier design tasks, Moreover, we present the hypervolume of the Pareto points discovered by MUTE in Tables 6 and 7 in Appendix H.I The results demonstrate that MUTE achieves a substantial improvement over the previous SOTA, improving the hypervolume by up to 38%. Overall, these results demonstrate the strong ability of MUTE to optimize multipliers, leading to significant reductions in both area and delay.
Broad Applicability to Adders and MACs To demonstrate that our approach is able to optimize a broad class of computing circuits, we apply our MUTE to optimizing two more fundamental com- puting circuits, i.e., adders and MACs. Specifically, we compare our MUTE with the three SOTA RL-based computing circuits optimization methods, i.e., RL-MUL (Zuo et al. 2023). AdaReset (Song et al., 2022). and HAVE (Wang et al. 2024g). adders and MACs. As shown in Table D the results demonstrate that MUTE significantly outperforms previous SOTA approaches, improving the hypervolume by up 42% compared to RL-MUL. The results not only highlight the superiority of our MUTE over previous SOTA approaches on optimizing computing circuits, but also demonstrate the broad applicability of our MUTE to wide range of fundamental computing circuits.
5.3 GENERALIZATION TO LARGE-SCALE CIRCUITS
To evaluate the generalization ability of our designed computing units to large-scall scale real-world computing circuits with numerous circuit units, we integrate these units optimized by MUTE and baselines into Processing Flement (PF) arrays (Park & Chung. 2020 [Snn et al. (023). which follows previous work (Zuo et al. 2023 Wang et al. 2024g). PE arrays are widely used in parallel computing tasks and large-scale data processing like Deep Neural Network (DNN) accelerators.
The results in Figure 5 show that PE arrays incorporating multipliers optimized by MUTE consis- tently and significantly Pareto-dominate those utilizing multipliers obtained from baselines. Fur- thermore, we present the hypervolume of the Pareto frontiers discovered by MUTE in Table II in Appendix H.4, The results demonstrate a significant improvement in hypervolume achieved by MUTE, outperformnng previous SOTA by up 48. 48%. We provide detailed results in Appendix

---

Published as a conference paper at ICLR 2025
Table 3: Analysis on the trade-off between performance gains and runtime of each module in MUTE.
16-bit And
32-bit And
Methods Wallace HAVE SA
HyperVolume
Improvementt (%)
Time (h)
HyperVolume
Improvementt St)
RunTime (h)
352.91
-25 96
1685,66 61
-70.09
505.83
12,50
20.33
5822.03
3,29
37.27
449.63
NA
9.33
5636.51
NA
17.40
CGE
32.14
6285.00
11.51
18.35
CGE-Leaming
608.25
35,28
15.17
6336.50
12,42
26.25
CGE+Laaming-Moll (MUTE)
622.55
38,46
17.33
6461.665
14.64
37,11
H.4
Overall, the results highlight the strong capability of MUTE to well generalize large-scale crpputatiionintensst circuits, thereby substantially improving real-world Al chips.
5.4 ABLATION STUDY
We present carefully designed ablation studies on multiplier design tasks as follows.
Contribution of Each Component To demon- strate the effectiveness of each component within MUTE, we conduct a thorough ablation study on multiplier design tasks.
Table 2: The results demonstrate that each com- ponent within MUTE is significant.
16:30AAn
32 bz And
Methods
HygerShiane
Improwement 93
HyperVotoee T
Engowemeen
Wallixe
332.91
NA 87.60
1655.51
NAA
MUTE (Okea)
621.58
6461.65
283.4
In terms of the efficient and effective genetic variation module, we have designed two meth- ods, called MUTE without Crossover (w/o Crossover) and MUTE without Mutation (w/o Mutation). MUTE without Crossover and MUTE without Mutation removes the designed genetic crossover and mutation operators, respectively. The results in Table show that our designed genetic mutation and crossover operators are both critical for optimizing computing circuits, demon- strating the strong ability of the designed operators for promoting efficient and diverse exploration.
Gesele Vasintio Module
75.61
255:72
Monation
10
81.26
6385,00
Meakd-Haaal Mislale
Lesning Midile
wis Lousieg
578.10
73.65
$766.00
242:07
In terms of the model module, we have designed MUTE without Model (w/o Model) by removing the model-basee module. The results in Table show that learning model can further improve the found designs in terms of the hypervolume. In terms of the learning module, we have designed MUTE without Learning (w/o Learning) by removing the learning module. The results demonstrate the significance of introducing learning into our genetic evolution for improving sampling efficiency.
5.5 PERFORMANCE GAIN AND RUNTIME TRA EVALUATION
To further investigate the cost-effectiveeess of each module of MUTE, conducted ablation study evaluating the trade-off between runtime and performance gains. MUTE consists of three modules: (1) the Cireuit Genetic Evolution (CGE) module, (2) the Learning module, and (3) the Model-hased module. The CGE module reformulates the multiplier design as a genetic evolution problem, using grid-based genetic representation and a random mutation policy. The Learning mod- ule introduces policy-guided population initialization and mutation. The Model-based module in- corporates learned model and a cascade ranking selection procedure.
Experimental results, summarized in Table 3] show that the CGE module sigmificantly improves performance, achieving up to 32. 14% gain with minimal runtime increase. The Learning module, while increasing runtime due to its computational overhead, enhances exploration efficiency and improves hypervolume. The Model-based module further improves hypervolume by 3. 38%, though it increases runtime due to model training and additional sample collection. Despite these increases, the total runtime of MUTE remains comparable to the recent state-of-the-arr approach, HAVE, high- lighting the balance between performance gains and computational costs.
CONCLUSION
In this paper, we theoretically and empirically show significant objective gap between the existing RL-based formulation and the original task due to the deceptive reward signals and incrementally localized actions in the RL-based formulation. To address this challenge, we propose a novel model- based circuit genetic evolution (MUTE) framework, which reformulates the problem as a genetic evolution process by proposing grid-based genetic representation of design solutions. Experiments on these circuits demonstrate that MUTE significantly dorenttd state-of of-te-of-the-art the-art approaches in terms of both area and delay, improving the hypervolume by up 38%.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
The authors would like to thank all the anonymous reviewers for their insightful comments and valu- able suggestions. This work was supported by the National Key R&D Program of China under con- tract 2022ZD01 19801 and the National Nature Science Foundations of China grants U23A20388, 62021001 and 624B1011.
REPRODUCIBILITY STATEMENT
In this study, to ensure the reproducibility of our approach, we provide key information from the main text and Appendix follows.
1. Algorithm. We provide the architecture and illustration of our MUTE in Figure and
Section We also provide the detailed implementation of MUTE Appendix See Appendix F.6 for the hyperparameters of MUTE.
2. Source Code. To facilitate the evaluation process and support thor- ough review, we have released our source code at the following link:
https:/annoyymous-4ppe...amm
3. Experimental Details. We provide detailed experiment settings in Section 5.1
4. Theoretical Proofs. We provide all proofs in Appendix A
REFERENCES
Hervé Abdi and Lynne Williams. Principal component analysis. Wiley interdisciplinary reviews:
computational statistics, 2(4):433 459, 2010.
Tutu Ajayi and David Blaauw. Openroad: Toward a self-driving, open-source digital layout im- plementation tool chain. In Proceedings of Government Microcircuit Applications and Critical Technology Conference, 2019.
Bushra Alhijawi and Arafat Awajan. Genetic algorithms: Theory, genetic operators, solutions, and applications. Evolutionary Intelligence, 17(3):1245-123 1256, 2024.
Nina Amenta, Marshall Bern, and Manolis Kamvysselis. new voronoi-based surface reconstruc- tion algorithm. In Proceedings of the 25th annual conference Computer graphics and interac- tive techniques, 415-421, 1998.
Nina Amenta, Sunghee Choi, Tamal K Dey, and Naveen Leckha. A simple algorithm for homcomor- phic surface reconstruction. Proceedings of the sixteenth annual symposium on Computational geometry, pp. 213-222 2000.
Chen Bai, Qi Sun, Jianwang Zhai, Yuzhe Ma, Bei Yu, and Martin D.F. Wong. Boom-explorer. Risc- boom microarchitecture design space exploration framework. In 2021 IEEF/ACM International Conference Computer Aided Design (ICCAD), pp. 1-9, 2021. doi: 10.1 10.109//CCAD5199...
2021 9643455.
Chen Bai, Jiayi Huang, Xuechao Wei, Yuzhe Ma, Sicheng Li, Hongzhong Zheng, Bei Yu, and Yuan Xie. Archexplorer: Microarchitecture exploration via bottleneck analysis. In 2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 268-282, 2023a.
Hui Bai, Ran Cheng, and Yaochu Jin. Evolutionary reinforcement learning: A survey. Intelligent
Computing, 2:0025, 2023b.
Yingi Bai, Jie Wang. Lei Chen, Zhihai Wang, Yufei Kuang. Mingxuan Yuan, Jianye Hao, and Feng Wu. A graph enhanced symbolic discovery framework for efficient circuit synthesis. In The Thirteenth International Conference on Learning Representations, 2025.
Toygun Basaklar, Suat Gumussoy, and Umit Y Ogras. Pd-morl: Preference- driven multi objectv reinforcement learning algorithm. arXiv preprint arXiv:2208.07914 2022.

---

Published as a conference paper at ICLR 2025
Andrew Beaumont- and C-C Lim. Parallel prefix adder design. In Proceedings 15th IEEE Symposium on Computer Arithmetic. ARITH-15 2001, pp. 118--255. IEEE, 2001.
Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial opti-
mization: a methodological tour d d'horizon. European Journal of Operational Research, 290(2): 405-421, 2021.
Brent and Kung. regular layout for parallel adders. IEEE Iransactions on Computers, 100(3): 260-264, 1982.
Frédéric Cazals and Joachim Giesen. Delaunay triangulation based surface reconstruction. Effec- rive computational geometry for curves and surfaces, pp. 231-276. Springer, 2006.
Lei Chen, Xing Li, Tsaras Dimitrios, Zhihai Wang. Yinqi Bai, and Mingxuan Yuan. general framework for efficient logic synthesis. In 2024 2nd International Symposium of Electronics Design Automation (ISEDA), pp. 361-362. IEEE, 2024a.
Ruifeng Chen, Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Feng Xu, and Yang Yu. Foresight distri- bution adjustment for off-policy reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 317-325, 2024b.
Shixin Chen, Shanyi Li, Zhen Zhuang, Su Zheng, Zheng Liang, Tsung-Y Yi Ho, Bei Yu, and Al- berto I. Langiovanni-Vincenelle Floorplet: Performance-aware floorplan framework for chiplet integration. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023.
Shixin Chen, Su Zheng, Chen Bai, Wenqian Zhao, Shuo Yin, Yang Bai, and Bei Yu. Soc-tuner: An importance-guided exploration framework for dnn-targeting design. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 207-212. IEEE, 2024c.
Luigi Dadda. Some schemes for fast serial input multipliers. In 1983 IEEE 6th Symposium Computer Arithmetic (ARITH), pp. 52-59. IEEE, 1983.
Biplab Das, Avijit Kumar Paul, and Debashis De. An unconventional arithmetic logic unit design and computing in actin quantum cellular automata. Microsystem Technologies, pp. 1-14, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/V1/N19-11 https://doi.org/ wtt.rr//.....11/111 wet.e11661wwwwll
URL
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: 7/ /ooeezeeveww net httt///oummiiido wit.uumiidxiiccccNXTiiee
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pp. 1329-1338. PMLR, 2016.
Fayez Elguibaly. A fast parallel multiplier-accamulltor using the modified booth algorithm. IEEE Transactions on Circuits and Systems II: Analog and Digital Signal Processing, 47(9):902-908, 2000.
Gautam Garai. Application of genetic algorithm in numerous scientific fields. In Genetic G algorithms. IntechOpen, 2022.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
12

---

Published as a conference paper at ICLR 2025
Maxime Gasse, Didier Chetelat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combi- natorial optimization with graph convolutional neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Zijie Geng, Xijun Li, Jie Wang, Xiao Li, Yongdong Zhang, and Feng Wu. A deep instance generative framework for milp solvers under limited data availability. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 16025-26043 Curran Associates, Inc., 2023.
Zijie Geng, Jie Wang, Ziyan Liu, Siyuan Xu, Zhentao Tang, Mingxuan Yuan, Jianye Hao, Yongdong Zhang, and Feng Wu. Reinforcement learning within tree search for fast macro placement. In International Conference Machine Learning. PMLR, 2024.
Zijie Geng, Jie Wang, Xijun Li, Fangzhou Zhu, Jianye HAO, Bin Li, and Feng Wu. Differentiable integer linear programming. In The Thirteenth International Conference on Learning Represen- tations, 2025a. URL URL
net wttpt//ooommd/aP/PPTTTT
Zijie Geng, Jie Wang, Ziyan Liu, Siyuan Xu, Zhentao Tang, Shixiong Kai, Mingxuan Yuan, Jianye HAO, and Feng Wu. LaMPlace: Learning to optimize cross-stage metrics in macro placement. In The Thirteenth International Conference on Learning Representations, 2025b. URL https:
//openzeview http://wwweettooooooeee/e/liia/.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learming. pp. 1861-1870. PMLR, 2018.
Reza Hashemian. A new multiplier using wallace structure and carry select adder with pipelining. In ISCAS '02 Conference Proceedings, 2002.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10. 10.1109/CVPR.2016.999
Christopher Hillar and Lek-Heng Lim. Most tensor problems are np-hard. Journal of the ACM (JACM), 60(6):1-39, 2013.
B. Holdsworth. Digital logic design, 2nd ed. 1987. ISBN 0408015667.
Yi-Qi Hu, Xu-Hui Liu, Shu-Qiao Li, and Yang Yu. Cascaded algorithm selection with extreme- region ucb bandit. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10);
6782-6794, 2021.
Guyue Huang, Jingbo Hu, Yifan He, Jialong Liu, Mingyuan Ma, Zhaoyang Shen, Juejian Wu, Yuan- fan Xu, Hengrui Zhang, Kai Zhong, et al. Machine learning for electronic design automation: A survey, ACM Transactions Design Automation of Electronic Systems (TODAES), 26(5):1-46, 2021.
Wei Hung, Bo Kai Huang, Ping-Chun Hsieh, and Xi Liu. Q-pensieve: Boosting sample efficiency of multi-objective RL through memory sharing of q-snapshots. In The Eleventh International Conference on Learning Representations, 2023.
wwwp:ewww.........e.eeeeeeeeeeoe www.:ewwwwweeeetmmemmmm forum?
id=AwMaBXLIJE
Nichh Itoh, Yasumasa Tsukamoto, Takeshi Shibagaki, Koji Nii, Hidehiro Takata, and Hiroshi Makino. A 32/spl times/24-bit multiplier-accumulaa with advanced rectangular styled wallace- tree structure. In 2005 IEEE International Symposium on Circuits and Systems (ISCAS), pp- 73-76. IEEE, 2005.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Chengxing Jia, Fuxiang Zhang, Yi-Chen Li, Chen-Xiao Gao, Xu-Hui Liu, Lei Yuan, Zongzhang Zhang, and Yang Yu. Disentangling policy from offline task representation learning via adversar- ial data augmentation. arXiv preprint 2003v2240.007611 2024.
13

---

Published as a conference paper at ICLR 2025
Andrew B Kahng. Advancing placement. In Proceedings of the 2021 International Symposium Physical Design, pp. 15-22, 2021.
Eukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020.
Yao Lai, Yao Mu, and Ping Luo. Maskplace: Fast chip placement via reinforced visual representa- tion learning. Advances Neural Information Processing Systems, 35:24019-24030, 2022.
Yao Lai, Jinxin Liu, Zhentao Tang, Bin Wang, Jianye Hao, and Ping Luo. Chipformer: Transfer- able chip placement via offline decision transformer. In International Conference on Machine Learming, pp. 18346-18364, PMLR, 2023.
Yao Lai, Jinxin Liu, David Z. Pan, and Ping Luo. Scalable and effective arithmetic tree generation for adder and multiplier designs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL httos:
net//
Pengyi Li, Jianye Hao, Hongyao Tang, Xian Fu, Yan Zhen, and Ke Tang. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey on hybrid algorithms. IEEE Transactions on Evolutionary Computation, 2024a.
Pengyi Li, HAO Jianye, Hongyao Tang, Yan Zheng, and Fazl Barez. Value-evolutionary-basee reinforcement learning. In Forty-first International Conference on Machine Learning, 2024b.
Xihan Li, Xing Li, Lei Chen, Xing Zhang, Mingxuan Yuan, and Jun Wang. Circuit transformer: End-to-end circuit design by predicting the next gate, 2024c.
Xijun Li, Fangzhou Zhu, Hui-Ling Zhen, Weilin Luo, Meng Lu, Yimin Huang, Zhenan Fan, Zirui Zhou, Yufei Kuang, Zhihai Wang, Zijie Geng, Yang Li, Haoyang Liu, Zhiwu An, Muming Yang, Jianshu Li, Jie Wang, Junchi Yan, Defeng Sun, Tao Zhong, Yong Zhang, Jia Zeng, Mingxuan Yuan, Jianye Hao, Jun Yao, and Kun Mao. Machine learning insides optverse ai solver: Design principles and applications, 2024d.
Xing Li, Lei Chen, Jiantang Zhang, Shuang Wen, Weihua Sheng, Yu Huang, and Mingxuan Yuan. Effisyn: Efficient logic synthesis with dynamic scoring and pruning. In 2023 IEEE/ACM In- ternational Conference on Computer Aided Design (ICCAD), pp. 1-9, 2023. doi: 10.1
ICCAD57390.2023.1003 10323902.
Yixuan Li, Can Chen, Jiajun Li, Jiahui Duan, Xiongwei Han, Tao Zhong, Vincent Chau, Weiwei Wu, and Wanyuan Wang. Fast and interpretable mixed-integer linear program solving by learning model reduction. arXiv preprint arXiv:2501.003m1 2024e.
Yixuan Li, Wanyuan Wang, Weiyi Xu, Yanchen Deng, and Weiwei Wu. Factor graph neural network meets max-sum: A real-time route planning algorithm for massive-scale trips. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1165- 1173, 2024f.
Haotian Ling, Zhihai Wang, and Jie Wang. Learning to stop cut generation for efficient mixed- integer linear programming. Proceedings of the AAAI Conference on Artificial Intelligence, 38 (18):20759-20767, Mar. 2024. doi: 10. wa.11669/ai..33388 org/index. pww..pppppAAAAa... AAAI http:///AAaattttteeeeeeeeee
URL htttts://////sst aaai.
Haoyang Liu, Yufei Kuang, Jie Wang, Xijun Li, Yongdong Zhang, and Feng Wu. Promoting generalization for exact solvers via adversarial instance augmentation, 2023a. URL https:
//arxiv. hbrgg//bb//e /2310 1..10.1411 14161
Haoyang Liu, Jie Wang. Wanbo Zhang, Zijie Geng, Yufei Kuang, Xijun Li, Bin Li, Yongdong Zhang, and Feng Wu. MILP-studin MILP instance generation via block structure decomposition. In The ehihty- Annual Conference on Neural Information Processing Systems, 2024a. URL
https://cpenrevies net http://www....eooooooooooaaaaaa
14

---

Published as a conference paper at ICLR 2025
Haoyang Liu, Jie Wang, Zijie Geng, Xijun Li, Yuxuan Zong, Fangzhou Zhu, Jianye HAO, and Feng Wu. Apollo-MILP: An alternating prediction-corrcciion correction neural solving framework for mixed- integer linear programming. In The Thirteenth International Conference Learning Represen- tations, 2025. httltttts:// /ttp::///eeeeeeeeeeeeeeeeeeeeeee net. net/oouummm (oorummmddmFPPOPPPooo
Jianhua Liu, Shuo Zhou, Haikun Zhu, and Chung-Kuan Cheng, An algorithmic approach for generic parallel adders. In /CCAD-200 2003. International Conference on Computer Aided Design (IEEE Cat. No. 03CH37486). pp. 734 740. IEEE, 2003.
Jiantao Liu, Carmine Rizzi, and Lana Josipović. Load-store queue sizing for efficient dataflow circuits. In 2022 International Conference on Field-Progee Peeld-Programmbbe Technology (ICFPT), pp. 1- 9, 2022. doi: 10. 1109/CFPT665622222999777777
Junfeng Liu, Liwei Ni, Xingquan Li, Min Zhou, Lei Chen, Xing Li, Qinghua Zhao, and Shuai Ma. Aimap: Learning improve technology mapping for asics via delay prediction. In 2023 IEEE s1 International Conference on Computer Design (ICCD), pp. 344-347. IEEE, 2023b.
Junfeng Liu, Min Zhou, Shuai Ma, and Lujia Pan. Mata*: Combining learnable node matching with a* algorithm for approximate graph edit distance computation. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 1503-1512, 2023c.
Junfeng Liu, Liwei Ni, Lei Chen, Xing Li, Qinghua Zhao, Xingquan Li, and Shuai Ma. A delay- driven iterative technology mapping framework. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2024b. doi: 10. A.199TCCAD..222.5224446.
Qiyuan Liu, Qi Zhou, Rui Yang, and Jie Wang. Robust representation learning by clustering with bisimulation metrics for visual reinforcement learning with distractions. Phirty-Seventl AAAI Conference on Artificial Intelligence, pp. 8843-8851. AAAI Press, 2023d.
Xu-Hui Liu, Zhenghai Xue, Jingcheng Pang, Shengyi Jiang, Feng Xu, and Yang Yu. Regret mini- mization experience replay in off-policy reinforcement learning. Advances in Neural Information 2021.
Processing Systems,
Xu-Hui Liu, Tian-Shuo Liu, Shengyi Jiang, Ruifeng Chen, Zhilong Zhang, Xinwei Chen, and Yang Yu. Energy-guided diffusion sampling for offline-to-online reinforcement learning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, 2024c.
Xuhui Liu, Feng Xu, Xinyu Zhang, Tianyuan Liu, Shengyi Jiang, Ruifeng Chen, Zongzhang Zhang, and Yang Yu. How to guide your learner: Imitation learning with active adaptive expert volvement. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagemt Systems, pp. 1276-1284, London, UK, 2023e.
Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an ad- versarial regularizer. In The Thihth- Annual Conference on Neural Information Processing Systems.
Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled framework for autonomous 1lm agents with provable sample efficiency. arXiv preprint arXiv:2309. 17382, 2023f.
Zhihan Liu, Miao Lu, Wei Xiong, Han Zhong, Hao Hu, Shenao Zhang, Sirui Zheng, Zhuoran Yang, and Zhaoran Wang. Maximize to explore: One objective function fusing estimation, planning, and exploration. Advances in Neural Information Processing Systems, 36, 2024d.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, et al. A graph placement methodol- ogy for fast chip design. Nature, 594(7862):07-2212. 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529 533, February 2015. ISSN 00280836. URL http://dx. wwwp:odoo..ooo11... 0tt//10.1033 1038 naturel 4236
15

---

Published as a conference paper at ICLR 2025
Nangate Inc. "Open Cell Library v2008 10 SPI". openlibrary/ 2008.
http://ww..nnggatt.
Walter Lau Neto, Matheus Trevisan Moreira, Luca Amaru, Cunxi Yu, and Emmrnuel Gaillar- don. Read your circuit: leveraging word embedding to guide logie optimization. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp. 530-535, 2021. 2021.
Vojin G. Oklobdzija, David Villeger, and Simon S. Liu. A method for speed optimized partial product reduction and generation of fast parallel multipliers using an algorithmic approach. IEEE Transactions computers, 45(3):294-306, 1996.
Parallax Software Inc.
OpenSTA.
OpenSTA
Sang-Soo Park and Ki-Seok Chung. 911::334, 2020.
Cenna: cost-effective neural network accelerator. Electronics,
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buu Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- Associates, Inc., 2019. URL htt httpt:ssss//pooeeeddeeeeo neur ps. wwwporrips.ccppppppeeepee wc/paper__ppppeeeeepeeeeeeeee fi Fises/
Ratko Pilipović, Patricio Bulić, and Uroš Lotrič. two-stage operand trimming approximate log- arithmic multiplier. IEEE Transactions on Circuits and Systems I: Regular Papers, 68(6):2535- 2545, 2021.
Haoxing Ren and Jiang Hu. Machine Learning Applications in Electronic Design Automation. Springer Nature, 2023.
Rajarshi Roy, Jonathan Raiman, Neel Kant, Ilyas Elkin, Robert Kirby, Michael Siu, Stuart Oberman, Saad Godil, and Bryan Catanzaro. Prefixrl: Optimization of parallel prefix circuits using deep reinforcement learning. In 2021 58th ACM/IÉEE Design Automation Conference (DAC), pp-
853-858. IEEE, 2021.
Subhendu Roy, Mihir Choudhury, Ruchir Puri, and David Pan. Towards optimal performance area trade-off in adders by synthesis of parallel prefix structures. In Proceedings of the 50th Annual Design Automation Conference, pp. 1-8, 2013.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747, 2016.
Daniela Sánchez, Lorenzo Servadei, Gamze Naz Kiprit, Robert Wille, and Wolfgang Feker. A comprehensive survey on electronic design automation and graph neural networks: Theory and applications. ACM Trans. Des. Autom. Electron. Syst., 28(2), feb 2023. ISSN 1084-4309. doi: 10.1145/3543853, URL /ei. 1ttp:////....r1111111111100/////
Yunqi Shi, Ke Xue, Lei Song, and Chao Qian. Macro placement by wire-mask-guided black-box optimization. In Advances Neural Information Processing Systems, volume 36, pp. 6825-6843, New Orleans, LA, 2023a.
Yunqi Shi, Siyuan Xu, Shixiong Kai, Xi Lin, Ke Xue, Mingxuan Yuan, and Chao Qian. Timing- driven global placement by efficient critical path extraction. In Proceedings of Design Automation Test in Europe Conference Exhibition (DATE), 2025a.
Zhihao Shi, Xize Liang, and Jie Wang. LMC: Fast training of GNNs via subgraph sampling with provable convergence. In The Eleventh International Conference on Learning Representations, 2023b. URL htt.:/tttt//ppeeeeeeeeeee htwpa//wwweeeeee.foooooooo/ssa/a
16

---

Published as a conference paper at ICLR 2025
Zhihao Shi, Jie Wang, Fanghua Lu, Hanzhu Chen, Defu Lian, Zheng Wang, Jieping Ye, and Feng Wu. Label deconvolution for node representation learning on large-scale attributed graphs against learning bias. IEEE Transactions Pattern Analysis and Machine Intelligence, 46(12):11273 11286, 2024, doi: 10. 1009/T/AAMM.22244444490000
Zhihao Shi, Jie Wang, Zhiwei Zhuang, Xize Liang, Bin Li, and Feng Wu. Accurate and scal- able graph neural networks via message invariance. In The Thirteenth International Confer- ence on Learning Representations, 2025b. URL URthttps: httptt:://ppereeeeeeee http///ppp///veieeeee ew. net meetfforum?io
UqrFPhcmFp p
Jack Sklansky. Conditional-sum addition logic. IRE Transactions on Electronic computers, (2): 226-231, 1960.
Adam Slowik and Halina Kwasnicka. Evolutionary algorithms and their applications to engineering problems. Neural Computing and Applications, 32:1 92:12363-12377 2020.
Hyun-Wook
Wook Son, Ali Al-Hamid, Yong-Seok Na, Dong- Yeong Lee, and Hyung-Wor Won Kim. Cnn accelerator using proposed diagonal cyclic array for minimizing memory accesses. Computers, Materials & Continua, 76(2), 2023.
Jialin Song, Rajarshi Roy. Jonathan Raiman, Robert Kirby, Neel Kant, Saad Godil, and Bryan Catan- zaro. Multi-objective reinforcement learning with adaptive pareto reset for prefix adder design. In Workshop on for Systems at NeurIPS, 2022.
Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 15):1054-, 1998. doi: 199../....998.11222
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural nehworks. Springer, 2020.
Cheng Tan, Chenhao Xic, Ang Li, Kevin J Barker, and Antonino Tumeo. Aurora: Automated refinement of coarse- grained reconfigurable accelerators. In 202/ Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1388-1393. IEEE, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033. IEEE, 2012.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning, In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need, In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpes::/pproceeeanggg..c www.eeiss.....eeeeeeeeeeeeeeeeee Papp:cccccssaaaaaappa.ppp.
files/paper/20177
Christopher S Wallace. A suggestion for fast multiplier. IEEE Transactions on electronic Com- puters, (1):14-17, 1964.
Chenguang Wang and Tianshu Yu. Efficient training of multi-task neural solver with multi-armed bandits. arXiv preprint arXhv:2305.0623 2023.
Chenguang Wang, Yaodong Yang, Congying Han, Tiande Guo, Haifeng Zhang, and Jun Wang. A game-the approach for improving generalization ability of tsp solvers. 2021.
Chenguang Wang, Congying Han, Tiande Guo, and Man Ding. Solving uncapacitated p-median problem with reinforcement learning assisted by graph attention networks. Applied Intelligence, 53(2):201--0255, 2023a.
Chenguang Wang, Zhouliang Yu, Stephen McAleer, Tianshu Yu, and Yaodong Yang. Asp: Learn a universal neural solver! IEEE Transactions Pattern Analysis and Machine Intelligence, 2024a.
17

---

Published as a conference paper at ICLR 2025
Jie Wang, Rui Yang. Zijie Geng, Zhihao Shi, Mingxuan Ye, Qi Zhou, Shuiwang Ji, Bin Li, Yongdong Zhang, and Feng Wu. Generalization in visual reinforcement learning with the reward sequence distribution. arXiv preprint arXi:: 2023b.
Jie Wang. Zhihai Wang, Xijun Li, Yufei Kuang, Zhihao Shi, Fangzhou Zhu, Mingxuan Yuan, Jía Zeng. Yongdong Zhang, and Feng Wu. Learning to cut via hierarchical sequence/set model for efficient mixed-integer programming, 2024b.
Jie Wang, Zhihai Wang, Xijun Li, Yufei Kuang, Zhihao Shi, Fangzhou Zhu, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning to cut via hierarchical sequence/set model for efficient -ixeeenteeger programming. IEEE Transactions on Pattern Analysis and Machine Intel- doi: 10. 11100//AAM..202.4332222
ligence, pp. 1-17, 2024c.
Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, and Furong Huang. Coplanner: Plan to roll out conservatively but to explore optimistically for model-based rl. In The Twelfth International Conference on Learning Representations, 2023c.
Zhihai Wang. Jie Wang. Qi Zhou, Bin Li, and Houqiang Li. Sample-efficient reinforcement learning via conservative model-based actor-critic. Proceedings of the AAAI Conference on doi: 10.1609/aaai, www.a/a.i..ii........ w36... 20839. URL
Artificial Intelligence, 36(8):8612-8620, Jun. 2022.
https:// aaai arg. index php/ AAAI wwt.aa/aicleeeeeee////aa
Zhihai Wang. Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and
Feng Wu. Learning cut selection for -exedeinteger linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations, 2023d. URL
https: https://openreviie wpw.revieeeeeeeeeeoeee net. www./neeueennneeeeeeeer
9bRNcK
Zhihai Wang,
Taoxing Pan, Qi Zhou, and Jie Wang. Efficient exploration in resource-restricted reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8); 10279-10287, Jun. 2023e. doi: 10. 16.1/69/aai v37.8.22222 URL htttht:///osss www.. Index.php/Bae www.aa//arrttteeeeeeeeeeeeeeeeee
Zhihai Wang, Lei Chen, Jie Wang, Yinqi Bai, Xing Li, Xijun Li, Mingxuan Yuan, Jianye Hao, Yongdong Zhang, and Feng Wu. A circuit domain generalization framework for efficient logic synthesis in chip design. In Forty-first International Conference on Machine Learning. PMLR, 2024d.
Zhihai Wang, Zijie Geng, Zhaojie Tu, Jie Wang. Yuxi Qian, Zhexuan Xu, Ziyan Liu, Siyuan Xu, Zhentao Tang. Shixiong Kai, et al. Benchmarking end-to-end performance of ai-based chip place- ment algorithms. arXiv preprint arXiv:2407. 1506,, 2024e.
Zhihai Wang, Jie Wang, Qingyue Yang, Yinqi Bai, Xing Li, Lei Chen, Jianye HAO, Mingxuan Yuan, Bin Li, Yongdong Zhang, and Feng Wu. Towards next-generation logic synthesis: A scalable neural circuit generation framework. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024f. URL https://openreet net//orumm
ZYNYhh3ocW
Zhihai Wang, Jie Wang, Dongsheng Zuo, Ji Yunjie, Xilin Xia, Yuzhe Ma, Jianye HAO, Mingxuan Yuan, Yongdong Zhang, and Feng Wu. A hierarchical adaptive multi-task reinforcement learn- ing framework for multiplier circuit design. In Forty-first International Conference on Machine Learning, 2024g. URL http::::///pp/eeeee/e ntt/forumm fiteerumiii---e - L 7GaUSEB
Neil HE Weste and David Harris. CMOS VLSI design: a circuits and systems perspective. Pearson Education India, 2015.
Weihua Xiao, Weikang Qian, and Weiqiang Liu. Gomil: Global optimization of multiplier by integer linear programming. In 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 374-379. IEEE, 2021.
Rui Yang, Jie Wang, Zijie Geng, Mingxuan Ye, Shuiwang Ji, Bin Li, and Feng Wu. Learning resk-relevant representations for generalization via characteristic functions of reward sequence distributions. In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2242-2252. ACM, 2022a.
18

---

Published as a conference paper at ICLR 2025
Rui Yang, Jie Wang. Zijie Geng, Mingxuan Ye, Shuiwang Ji, Bin Li, and Feng Wu. Learning task-relevant representations for generalization via characteristic functions of reward sequence distributions. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2242-2252, 2022b.
Rui Yang, Jie Wang, Guoping Wu, and Bin Li. Uncertainty-based offline variational bayesian rein- forcement learning for robustness under diverse data corruptions. In Advances in Neural Infor- mation Processing Systems 38, 2024.
Xufeng Yao, Haoyang Li, Tsz Ho Chan, Wenyi Xiao, Mingxuan Yuan, Yu Huang, Lei Chen, and Bei Yu. Hdldebugger: Streamlining hdl debugging with large language models. arXiv preprint arXiv:2403.116771 2024a.
Xufeng Yao, Yiwen Wang, Xing Li, Yingzhao Lian, Ran Chen, Lei Chen, Mingxuan Yuan, Hong Xu, and Bei Yu. Rtlrewriter: Methodologies for large models aided rtl code optimization. arXiv preprint arXiv:2409. 1441, 2024b.
Jiaxi Zhang, Qiuyang Gao, Yijiang Guo, Bizhao Shi, and Guojie Luo. Easymac: design exploration- enabled multiplier-accumil 27th Asia and South Pucific Design Automation Conference (ASP-DAC), pp. 647-653, IEEE, 2022.
generator using canonical architectural representation. In 2022
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprini Stti::2405.193322. 2024.
Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuan- dong Tian. Noveld: A simple yet effective exploration criterion. Advances in Neural Information Processing Systems, 34:25217-25230, 2021.
Qingling Zhu, Xiaoqiang Wu, Qiuzhen Lin, Lijia Ma, Jianqiang Li, Zhong Ming, and Jianyong Chen. A survey evolutionary reinforcement learning algorithms. Neurocomputing, 556: 126628, 2023a.
Xuliang Zhu, Ruofei Tang, Lei Chen, Xing Li, Xin Huang, Mingxuan Yuan, Weihua Sheng, and Jianliang Xu. A database dependent framework for k-input maximum fanout-free window rewrit- ing. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2023b. doi: AO.1/499D 2023999922222177777
Dongsheng Zuo, Yikang Ouyang, and Yuzhe Ma. RI-mul: Multiplier design optimization with deep reinforcement learning, In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6, 2023. doi: 10. N0.1109/DAC59922.2000 210.
Dongsheng Zuo, Jiadong Zhu, Chenglin Li, and Yuzhe Ma. Ufo-mac: A unified framework for optimization of high-performance multipliers and multipyy-cccmmultton arXiv preprint arXiv:2408.06935, 2024a.
Dongsheng Zuo, Jiadong Zhu, Yikang Ouyang, and Yuzhe Ma. RI-mul 2.0: Multiplier design optimization with parallel deep reinforcement learning and space reduction. ACM Transactions on Design Automation of Electronic Systems, 2024b.
19

---

Published as a conference paper at ICLR 2025
A THEORETICAL ANALYSIS
This section presents a theoretical analysis of the significant misalignment between the result of the optimal policy obtained through RL and the original optimal circuit design. First, the proof of Theorem 3.1 is provided subsection A. Second, subsection offers further theoretical insights into the deceptive nature of the existing reward formulation.
Throughout the following theoretical analysis, we assume that the state space S is finite, and there is terminal action for each state that can terminate the episode at this state. The assumptions usually hold in practical multiplier optimization problems. We focus on deterministic policies.
A.1 PROOF OF THEOREM 3.1
Lemma A.1. For any policy æ in the MDP, there exists a policy n' that terminates at a certain state such that R= 2 Ra
Proof. Since state space S is finite, the states in trajectory generated by policy π are finite as well. Therefore, there exists state ST = arg min,er, f(s), where T₃ the of states in the trajectory of π and 7' is finite number. We denote the trajectory Tx by (50, 81, S7. .T. }.
Then we construct new policy n' that generates a trajectory Tx' ......., s7., where sy S1, Vs S T. Note that the trajectory Tx² terminates at the state Then we have
R Σ (f(s)) f((+)))
(6)
T-1
T-1
Σ f(s,)
Σ ((
1=0
T-2
f(())
Σ - -f))) -
(8)
1=0
f((o) (1 (1- Σ
y f((+))) - -1 -(1-))))(111>))) Σ
f(r)
(9)
1=0
1=T-1
T-2
89
2 f((0) - (1 y1 Σ
y' f((++1 - (1-x)
Σ
y' f(mm)
(10)
1=0
1=1-1
R"
(11)
Then prove Theorem 3.1 as follows.
Proof. Recall that Theorem 3.1 states that "The optimal RL policy x¹ = arg max, R terminates at a state, and the state is local optimal state of the evaluation function f." We prove this Theorem in the following two steps.
(1) π² has a terminal state If the optimal policy π¹ doesn't terminate, then according to Lemma A.1 there exists distinct policy with higher return, which contradicts with the definition of the optimal policy .." Thus, x² terminates at a certain state.
(2) Local Minimality Denote the terminate state of π² as ST. By contradiction, suppose Eao A. ag # terminate such that f ((T(ST, ao)) < f(s)). Then consider a new policy Λí whose trajectory is identical to before the 7'thh step but execute action ag rather than terminating 7-tth step, and
20

---

Published as a conference paper at ICLR 2025
A Sub-Trajectory from the Traning Curve
402
Traning Curve
mnd
Trough Paints
Park Points
400
Weighted PPA
398
396
394
4305
4310
4315
4320
4325
4330
Steps
Figure 6: Sampled points from the training curve illustrating the oscillatory properties of the evalu- ation function.
execute terminate at step 7 T+1. 1. Then the return of policy x' is
R= Σ ((((s) ((++1))
(12)
R +y ((T(s)))((((s
(13)
Rx
(14)
which contradicts the definition of the optimal x².
A.2 MULTIPLE HIGH HILLS CONDITION FOR GAP EXISTENCE
Theorem 3.J indicates that the existing reward formulation guides the RL agent to evolve the circuit from an initial design to a locally optimal solution. However, the conditions under which this local optimum is the same as the global optimum remain unclear. In this subsection, we provide heuristic conclusions based on the oscillatory behavior of the objective function.
First, we sample trajectory from the sigh- dimeemiioinnnl solution space using basic EA algorithm for the purpose of visualization and simplified analysis, denoted as Tsim. as shown in Figure Our analysis focuses on the impact of the proxy reward function on the optimization objective along the sampled one-dimensional function curve, avoiding the complexities of the high-dimensional state space. Using the sampled trajectory Tsim, we define a simplified MDP (Sim-MDP) with the tuple (Ssim. Asim Tsm. y, Mo). Here, Ssim TSim- The action space is simplified into Asim = (Go, Terminate). The transition function is defined as ( Go) = S1+1, and the episode terminates upon executing the terminate action. The reward function, discount factor, and initial state are consistent with the original MDP. Any policy in this Sim-MDP corresponds to a trajectory terminating certain state. The Sim-MDP simplifies state and action spaces while keeping the reward function unchanged, facilitating the analysis of the optimization objective gap.
Based on the MDm-MDP, we calculate the cumulative discounted rewards of policies starting from the initial state and terminating at various points along the curve. We then compare the returns of these policies with the return of the policy π.- that converges to the global minimum point s" in the Sim- MDP. For discount factors of 0.8, 0.9, and 0.99, the proportions of policies with returns higher than Ty accounts for approximately 54%,,% 54.07% and 45.71% respectively. Moreover, for discount factors of 0.8, 0.9, and 0.99, the relative objective gap between the optimal RL policy and the global optimal point are roughly 35.52% 34.75%, and 26.19%. 19%, respectively. This demonstrates a significant gap between the RL proxy optimization objective and the original optimization objective, attributed to the highly oscillatory nature of the optimization objective function.
Second, to further understand the reasons for this gap, we theoretically show conditions for the existence of optimization objective gap in the Sim-MDP, based on the observation that the evaluation function is highly oscillatory, leading to many peak and trough points. We first rigorously formulate the evaluation function f across the sampled trajectory Tsam by using many repeated hills, which is inspired by the properties of oscillatory functions. A state s, Ssin is Peak Point if f(s,)
21

---

Published as a conference paper at ICLR 2025
11) and f(s)) f(++1). In contrast, state s E Ssim is Trough Point if f(,) < f()--1) and f(s)) < f(5++1). Without loss of generality, we assume the initial state of Sim-MDP is a through point. We denote p; as the step index of the i-th peak point, and the step index of the i-th through point. The i-th Hill is defined by a set of states between the i-th trough point and the (i+ 1)-th trough point, i.e., Pi == {sylt; I S f;+1). We denote the number of ascending steps Pi -i-4ib by mi and the number of descending steps 1,+1 Pi by ni Inspired by the fact that the action will always affine the structure of the multiplier which changes the evaluation function, for the i- th Hill we denote the lower bound of the variation of f as 6; == minje(u.pp) {11() - f(s,))l )); on the other hand the evaluation function f is bounded, thus we denote the upper bound of the variation of as := maxje (p) {If(x)) - f(x The formulation is illustrated in Figure 6 With the multiple hills formulation, we provide a Multiple High Hills Condition for the existence of optimization objective gap.
Theorem A.2 (Multiple High Hills Condition). Denote the number of hills in the Sim-MDP before s' N, where s" arg min f(s) is the global optimal state in Sim-MDP. If there exists the i-th trough point St, such that
N Σ
yd) y"1 yd
1 y-y
y Ej 1-y 8j
< 0,
(15)
then the optimization objective gap exists, i.e., the optimal policy in Sim-MDP T'sm converges to a sub-optimal solution rather than the global optimal state s".
Proof. Suppose s* is achieved at step T, and denote the policy that terminates at s as To- For any through point St,, consider a new policy xt' that terminates at St, and its return is
4-1 Σ y' (f(ss))((sm)
R
(16)
Then we have
R
R
Σ (
(17)
F4
N
fja-1
((((s)) - f((mm))
(18)
N
1y+my-1
1jtmytn
((((s)) - f(S))))
Σ
((((s)) - -((mm)))
(19)
fm7y+my
my
ny
Σ
( + y/)
y' -((())))((((1)))
(20)
m
ni-1
$
Σ
y' (-6i) + + Σ y Ei
(21)
/
Σ
N
1 ym
(-6;) +y my 1- nd
1 - El
(22)
1 y
(23)
Then Nyy y the optimal policy in Sim-MDP sim converges to sub-optimal solution rather than the global optimal state s".
1-y" T-y (-5i) +
my l-y"| ns
< 0 = R < R² j.e. x' has higher return, and further
T-y
This Theorem implies that if there are multiple high hills, ie.. unexpected high peak points (large oj), on the trajectory between trough point (i.e., local optimum) and the global optimum, then
22

---

Published as a conference paper at ICLR 2025
Landscape of RL-MUL 16-Bit And
Landscape of HAVE 16-BIt And
Teactay of RurSs
Gobe Nonu
Gode Money
Figure 7: Optimization objective landscape of 16-bit and gate-based multiplier. The global minimum marked as the red point. The grey lines mark RL-MUL and HAVE trajectories, with points indicating the optimal solutions found every five episodes. The optimum found by RL-MUL and HAVE is marked as the yellow point, with the objective value annotated nearby.
the RL optimization objective converges to the local optimum rather than the global optimum, due to the noisy information the cumulative discounted performance objective from those unexpected high peak points.
B VISUALIZATION OF OPTIMIZATION OBJECTIVE LANDSCAPE
In this section, visualize the optimization objective landscape using the 16-bit AND gate-based multiplier optimization task.
Visualizing optimization objective landscape
(1) Data Collection To ensure comprehensive sampling and maximize coverage of the solution space, we first employ the RL-MUL (Zuo al. 2023) and HAVE (Wang et al.) 2024g) algorithms to generate initial populations, each comprising 2,500 solutions. Building on this intiall population, we extensively apply our genetic variation operators, ultimately producing a total of 50,000 solutions.
(2) Data Visualization To visualize the high-dimensional solution space, we use Principal Compo- nent Analysis (PCA) (Abdi Williams 2010) to reduce its dimensionality two dimensions. This enables the creation of 3D surface plot depicting the relationship between solutions and their fit- ness values, i.e., our optimization objective landscape. We reconstruct the objective function surface from the data points using Delaunay triangulation, which is widely used in surface reconstruction from a set of points (Amenta al. 1998 2000 Cazals & Giesen 2006).
(3) Results As shown in Figure 7 the results reveal that the optimization surface is highly oscillatory and characterized by numerous local optima. Using the visualization method described above, we transformed the points collected during the training processes of RL-MUL and HAVE into curves. The results reveal that both RL-MUL and HAVE converged to specific local optima.
The convergence into local optima of RL-MUL and HAVE
Using the visualization method described above, we transformed the points collected during the training processes of RL-MUL and HAVE into curves. The results reveal that both RL-MUL and HAVE converged to specific local optima.
C RELATED WORK
C.1 COMPUTING CIRCUITS OPTIMIZATION
Computing circuits like adders and multipliers are widely employed in practical applications, lever- aging Compressor Tree and Prefix Tree for efficient parallel operations. Generally, optimization methods for these circuits can be categorized into three main approaches. (1) Manual designs in- volve leveraging human expertise to craft architectures derived from regular designs, which require
23

---

Published as a conference paper at ICLR 2025
substantial engineering effort. Various compressor Trees have been devised to reduce partial prod- ucts, as shown in (Wallace 1964; Dadda 1983; Itoh et al. 2005| Oklobdzija et al. 1996), while
Prefix Trees are optimized for more efficient parallel addition, as demonstrated in (Beaumont-Smith
& Lim 2001; Sklansky 1 1960, Brent & Kung
1982). (2) Conventional algorithmic methods (Xiao al. [021 Zuo al. 2024a Liu al. 2003 Roy al. 2013) generate circuit architectures using
specific strategies such as mathematical programming and heuristic search. However, they often optimize circuits using proxy metries such as size and depth, which may result in significant dis- crepancy from actual performance in the real design flow. (3) Recent methods (Zuo et al.) [023]
2024b, Roy et al., 202T; Song et al.] 2022] Wang et al.] 2024g] Lai et al.) (024) propose using reinforeement learning to optimize circuits based on the post-synthesis metries, incorporating syn- thesis into the optimization loop, This offers promising approaches to bridge the gap between proxy
metrics and actual performance. In this paper, we focus on optimizing computing circuits using post-synthesis metrics as well.
C.2 REINFORCEMENT LEARNING
Reinforcement Learning (RL) has achieved great success in sequential decision-making problems.
encompassing applications from video game playing to robotic control(Mnih al. 2015 Kaiser
eet al., 2020) Duan et al., 2016); IZhang et al.
2024; Liu et al.). RL approaches can be generally
divided into model-free Haarnoja et al. (2018):
: Wang et al. (2023c); Yang al. (2022b); : Liu et
(2024d); Wang et al. (2023b): Liu et al. (201); Yang et al. (2022a); Liu et al. (2023d). model-basee
Janner et al. (201 9) Liu al. (023t): : Wang et al. (2022) : and offine RL Hu et al. (202I): Chen
et al. (2024b); Jia et al. (2024); Liu et al. (2023e): Yang et al. (2024): Liu et al. (2024c) approaches.
In this paper, our MUTE falls into the model-based category.
C.3 GENETIC EVOLUTION ALGORITHMS
Genetic evolutionary algorithms (GA) are one of the most established and famous optimization methods, encompassing diverse range of variants that find extensive applications across various fields (Garai 2022 Alhijawi & Awajan 2024). Inspired by Darwinian theories of species evolution in nature, genetic algorithms utilize selection, crossover, and mutation operators to evolve solutions, ultimately achieving global optimization (Slowik & Kwasnicka 2020). Recently, Evolutionary Re- inforcement Learning algorithms (ERLs) have emerged a promising solution, effectively integrat- ing the strengths of both reinforcement learning and evolutionary algorithms (Zhu et al. 2023a Bai baaning-based based population initialization, sequential mutation operator, multi quutti-raauulaityy crossover operator, and based cascade ranking within a genetic algorithm framework for optimizing computing circuits.
al.) 2023b [i et al., 2024a). In this paper, we propose a learning-
C.4
MACHINE LEARNING FOR CHIP DESIGN
With the exponential growth in chip complexity driven by advances in semiconductor technology, the application of machine learning (ML) to assist in the automated chip design workflow has gar- nered significant attention in recent years (Mirhoseini et al.] 2021 Huang et al., 2021 Sánchez et al. 2023] Neto al. [022 Lai et al. [022] (023). The chip design process encompasses several
stages (Huang et al.| 2021 Ren & Hu 2023). including high-level synthesis ( (Yao et al. [024 Liu
al. 2022: Yao et 2024b), b,bb.....).. logic synthesis (Li et al.) 2023
Zhu et al. 2023b Li al. 2024c
Liu et al. 2023bb2220 e 2024b Bai et al. 2025 Wang et al. 2024f d
Chen et al. (222a)) placement (Lai
al. [2022 2023 Geng al. 2024 Chen et al. 2023 Shi et al. 2023a [025a] Wang et 2024e
Geng et al. 2025b), and design space exploration (Chen et al.
2024c
Bai et al. 2023a
202
among others.
C.5 MACHINE LEARNING FOR COMBINATORIAL OPTIMIZATION
Optimizing multiplier circuit designs is also essentially a combinatorial optimization problem. The
use of machine learning to tackle combinatorial optimization problems has been active topic of
significant interest in recent years (Bengio et al. 2021 Gasse et al. 2019 Shi et al. 20256] 20236
2024 Geng al. 2023 Wang et al. 2023d 2024b Ling al. 2024
2024d Wang al.
24

---

Published as a conference paper at ICLR 2025
0
0 1 2
0
2
1
0
matrix representation
Stage
0
TO
0
Stage 2
0
1
0
0
T
tensor representation T
Compressor Tree
Figure 8: State representation in RL-MUL.
2021
2024a
2023a
Wang & Yu
2023
Wang et al. 2024c
Geng et al. 2025a
Liu et al. 2023a
2025
2024a
Liet al.
2024f e).
D IMPLEMENTATION DETAILS OF RL-MUL
State Representation We use the total number of 3:2 and 2:2 compressors in each column to present the multiplier structure. As illustrated in Figure 8 4-bit multiplier structure and its matrix represen- tation M are shown. Given a matrix M R (xx)) where mij indicates the quantity of the i-th compressor used in column j. RL-MUL follows fixed scheme extend the matrix M to tensor to obtain unique representation for the assignment of compressors in multiplier stages. As shown in Figure & RL-MUL utilizes T RKx(M,+N))x57 total types of compressors, ST is the number of compression stages, and No and M, is the input width. For any element 1 within T, it signifies the utilization of the i-th type of compressor at column and stage i. The assignment method is to assign the compressors from the least significant bit (LSB) columns to the most significant bit (MSB) columns and assign the 3:2 compressors first as many as possible. After assigning the 3:2 compressors, if column j there are still more than two PPs, it assigns the 2:2 compressors. Repeat this progress until all compressors are assigned. For example at column in Figure 8 we first assign a 3:2 compressor compressor in the second stage.
represent multi-stage state, where K the
first stage, then assign a 2:2
Legalization Rules When selecting actions, RL-MUL exclusively considers whether the action re- duces the final production products to either or 2. RL-MUL has four actions, including adding 3:2 compressor, removing a 2:2 compressor, replacing a 3:2 compressor, and replacing a 2:2 com- pressor. Furthermore, action performed at column j will have impact column j+1 and cause column j+1 illegal due to the propagation of the carry bit. RL-MUL employs a legalization strategy that refines the state from column j+1 1 to the most significant bit, ensuring that the PPs in every line are reduced to or following the actions. The strategy adds a 3:2 or replaces a 2:2 compressor there an over of PPs. and deletes a compressor if there is lack of PPs.
E IMPLEMENTATION DETAILS THE BASELINES
GOMIL Xiao et al. 2021 is global optimization method that simultancously considers the CT and CPA. The author provides the open-source C++ code. We can extract the required structure from its solution files.
E.1 RL-BASED BASELINES
RL-MUL Zuo et al. 2023) encodes the state into a tensor T described Appendix D using ResNet- 18 the network backbone and training based the DQN algorithm. Different from the Random method, RL-MUL only chooses the action randomly in warm-up steps. future steps, chooses the action that maximize the masked Q-value of the network.
25

---

Published as a conference paper at ICLR 2025
MBPO Janner et al. (2019) is state-of-the-an model-based RL method, which can significantly improve sample efficieccy by learning an environment model. We implement the algorithm in our multiplier optimization environment by setting the Update- To- fo-Data ratio as five.
E.2 EVOLUTIONARY ALGORITHMS
MFEA Slowik & Kwasnicka (2020) is population based global optimization method inspired by biological evolution. We maintain a population of candidate solutions, iteratively evolving them through random mutations, crossovers, and selections. The population is then updated based on the fitness values of the individuals.
MBBO Garnett (2023) is global optimization method. We model the solution space high- dimensional vector space. We first sample a trajectory using a random walk. Then during each iteration, fit RBF kernel Gaussian Process model and use UCB (Upper Confidence Bound) as the acquisition function to determine the next sampling point.
F IMPLEMENTATION DETAILS OF OUR MUTE
F.1 HARDWARF SPECIFICATION
Our experiments were executed on a Linux-based system equipped with a 3.60 GHz Intel Xeon Gold 6246R CPU and NVIDIA RTX 3090 GPU.
F.2 SYNTHESIS TOOL. SETUP
45 is widely used standard cell library in the semiconductor industry. It is open source and free, and we can obtain at httptttsss//sssaaeeesese wwwplllwwwccooocooaaooooooooooot wtt.://w./eeecceeeeeeeeeeeeeeeee httm:///eeieeee/irrrrr-eeeaeeeee
Readers can refer OpenROAD fwwwwwsrripeeeee scripts distribution.
cww. The OpenROAD- Project/ seeking the artifact of OpenROAD flow matched with the
https://gitan:
In terms of the verilog generation, previous work uses EasyMAC Zhang et al. (2022) to imple- ment it. We encode our CT following EasyMAC Zhang al. (2022) rules which use a sequence Sei PoP1 ... Pr to represent a CT. Each compressor. Considering that generating Verilog HDL codes by EasyMAC and running the logical synthesis are still time-consuming, we directly generate multiplier Verilog codes using our designed template. Compared to EAsyMAC, our method can generate verilog code faster. To ensure fairness in comparison, we have employed a uniform default adder implement CPA for all methods.
(index,, type, signifies the index and type of
F.3 DETAILS ON LEARNING-BASER POPULATION INITIALIZATION
F.3.1 BEST-CASE LEARNING
In terms of population Initialization, propose best-case learning module, which maintains elite pool with 20 currently found best design solutions for enhanced diversity. We restart the initial state by sampling a state from the elite pool the beginning of each episode. In terms of the RL algorithms, we follow previous work (Zuo et al. 2023: Wang et al. 2024g) to use DQN agent learn Q-functions for selecting modification actions. In terms of the Q-network architecture, we use the ResNet-18 18 as the tensor state encoder, and use a multi-layer perceptron (MLP) to predict Q-values for each candidate action. The MI.P contains two hidden layers with 256 units and the ReLU activation function. To train the Q-network, we use an Adam optimizer, and set the learning rate as le-4. For a fair comparison, we set hyperparameters to align with previous work (Zuo et al, 2023 Wang et al. 2024g).
F.3.2
EVALUATION MODEL LEARNING AND CONSERVATIVE MODEL USAGE
In terms of the model architecture, we employ the ResNet- 18 the tensor state encoder and multi- head decoder to predict the area and delay of the imput state. The multi-hen head decoder comprises two multi-layer perceptrons (MLPs), each with two hidden layers with 256 units and ReI.U activation.
26

---

Published as a conference paper at ICLR 2025
In terms of the training details, we use the mean squared error loss update the model parameters. We use the Adam optimizer with a learning rate of 1e-3.
In terms of model usage, primarily use the model to fast evaluate the children solutions generated by the designed genetic variation operators. Specifically, we generate at least 100 children solutions each iteration, and use the model to pre-rank these solutions. The top-5 solutions are then selected for evaluation in the true environment. This approach allows us to generate a substantial number of children solutions, promoting extensive globally diverse exploration. Note that we do not use the model in the RL learning, as it will suffer from the cumulative multi-step model errors due to the sequential characteristics of RL methods. In contrast, using the model in our evolution process only suffers from single-step model errors.
RunTTime of Design Evaluation As demonstrated in Ta- ble 9 our learned model can significantly reduce the de- sign evaluation time compared to calling synthesis tools. RL-MUL (Zuo et al.) (023) employs EasyMAC and OpenRoad, while Vgen refers to a Verilog generation method we implemented, detailed in Appendix F22 Al-
Table 4: Runtime comparison
RunTime 100 samples)
Method/Circuit
And
MangMAC+OpenRood
29300
10930
Vgen-OpenRenn Model (Ouns)
303
973
though Vgen considerably accelerates evaluation compared to EasyMAC, it remains inefficient when evaluating large volumes of design solutions.
F.4 DETAILS ON THE RL-GUIDED MUTATION OPERATOR
The action space The actions consist of four types of local modifications to a Compressor Tree solution at specific column. These modifications include adding 2:2 compressor, removing 2:2 compressor, replacing a 3:2 compressor with a 2:2 compressor, and replacing a 2:2 compressor with 3:2 compressor.
Q-network model Our Q-network comprises a ResNet-18 (He et al. 2016) as an encoder to repre- sent the input state, and a multi-layer perceptron as a decoder to predict Q-values. The input state a grid-based genetic representation of the design solution. The output comprises the state-action values for each action.
The learning process We employ the Deep Q-network (DQN) algorithm (Mnih et al. 2015) to train the Q-network. During the population initialization phase, train the Q-network using collected interactions with the circuit synthesis environment. We then periodically update the Q-network by sampling some interactions with the circuit synthesis environment throughout the evolution process.
Managing invalid designs We indeed apply a legalization rule to transforming any invalid design solution into a valid solution, which is designed by RL-MUL [Zuo et al., 2023). Specifically, valid design requires each column to have exactly one or two remaining partial products after com- pression. Invalid designs-resulting from actions that impact subsequent columns- occur when a column has either zero or three remaining partial products. To resolve this, we implement a le- galization process that starts from the affected column and progresses toward the most significant column. For columns with three remaining partial products, 2:2 compressor is either replaced with or augmented by a 3:2 compressor. For columns with zero remaining partial products, a 2:2 or 3:2 compressor is removed, appropriate. This process ensures that all columns maintain valid number of remaining partial products (cither one or two).
F.5 DETAILS ON THE MULTI-GRANULARITY CROSSOVER OPERATOR
Legalization Mechanism Note that the crossover operators may lead to illegal solutions. Thus, we need to design a reasonable and simple legalization mechanism. Fortunately, we can follow the legalization rule proposed in RL-MUL (Zuo et al. 2023) to legalize these illegal solutions. Specifically, given a state that is modified from column i to any column, we can refine the state from column to the most significant bit, ensuring that the final partial products (PPs) in every line reduced to or 2 following the actions. The legalization rule adds a 3:2 or replaces a 2:2 compressor if there is an over of PPs, and deletes compressor if there is lack of PPs.
27

---

Published as a conference paper at ICLR 2025
objective
Parelo points
objective 2
Reference point
Non-Pareto points
P(1) E 2 p())
P(L)
Hypervolume
P(2)
P(3)
P(S)
P(4)
objective
objective
(a) Pareto Points
(b) Hypervolume
Figure 9: (a) An example for Pareto optimal set with objectives and Pareto optimal solutions (Pareto points). (b) An example for hypervolume with a selected reference point Integrated area H(P. ra) the union of the rectangular areas where the reference point ro and the Pareto point P(i) are diagonally opposite corners.
F.6 ADDITIONAL COMMON HYPERPARAMETERS
In the above sections, we have provided implementation details and hyperparameters. Here, we list the common parameters used in the comparative evaluation and ablation study in Table[5| Note that we use the same hyperparameter as that of previous work (Zuo et al. 2023 Wang et al. 2024g) if possible for fair comparison.
Table 5: Common parameters used in the comparative evaluation and ablation study.
Parameter
Value
Learning-Based Population Initialization Module
environment steps per learning episode
25
policy updates per environment step optimizer
Adam
discount (y) total learning episodes for initialization
0.8
40
Genetic Variation Module
samples generated by sequential mutation operator each iteration
100
samples generated by genetic crossover operator at each iteration total iterations for evolution
200
400
Model Bodel-Based Module
samples for circuit synthesis evaluation at each iteration
F.7 EVALUATION METRICS
Indeed, the multiplier optimization problem is a multi-objective optimization task with multiple conflicting objectives, such method with baselines. First, we visualize the approximated Pareto front in terms of the area and delay for multipliers designed by method and baselines. Second, we use the hypervolume of the approximated Pareto front. We present details on the two metrics as follows.
and delay. Thus,
two evaluation metrics to compare
Multi-Objective Optimization Metrics
28

---

Published as a conference paper at ICLR 2025
Without loss of generality, considering maximization optimization problem in objectives, we aim to find the set of optimal solutions known as the Pareto optimal set. For an n-objective optimization problem, solution Pareto dominates another solution y if is not worse than y in all objectives and has at least one strictly better value, i.e., Vi e [1, nI. fi(x) 2 f(() A 3i n1. fi(x) > fi(y)- A Pareto optimal solution is not dominated by any solution, and the set composed of all Pareto optimal solutions is referred to as the Pareto optimal solution set. One metric to evaluate the quality of Pareto optimal solution set is hypervolume, which illustrated in Figure 9 The hypervolume of a set is the volume of the space that is dominated by the solution in the set. When calculating the hypervolume of set, we need choose reference point. When reference points are fixed, Pareto solution set with larger hypervolume is considered superior.
Definition F.1 (Hypervolume metric). Let P be Pareto front approximation in an n-dimensional objective space and contain N solutions. Let ro R" be the reference point. Then, the hypervolume metric is defined as:
ro) = f (((P.m)(2)zz
where II(P,)) m) (a E 2 IPI ro 3 z P())..()) is the i-th solution in P. N is the relation operator of objective dominance, and (P.)) i a Dirac delta function that equals if Z E H(),ro) and 0 otherwise.
F.8 CONTRIBUTION OF OUR WORK TO AI COMMUNITY
Advancing AI Chips Our work directly contributes to the advancement of Al chips, such as NVIDIA's GPUs and Google's TPUs, by introducing an innovative optimization framework for the design of high-speed, area-efficient, and energy-effiie ceffrgy-ffiicien computing circuits. Note that NVIDIA's AI researchers have integrated Al-designed adders into their H100 chip [5], demonstrating the potential of our Al-based approach for advancing AI chips. The ability optimize Al chips is crucial for ad- dressing the ever-grov wver-growing computational demands of modern Al systems, ensuring their scalability, efficiency, and sustainability.
A Novel and Broadly Applicable Genetic Evolution Algorithm Our work introduces a sequen- tial mutation operator and multi-granularity crossover operator that leverages grid-based genetic solution representation to facilitate efficient and diverse exploration of large search spaces. This approach presents broadly applicable framework suitable for addressing a wide range of search problems. Moreover, we propose a model-based cascade ranking approach, which efficiently and accurately selects high-performing solutions from large pool of generated candidates. These con- tributions offer versatile and robust methodology for tackling complex optimization problems.
Identifying the Limitations of a Commonly-Used RL Formulation for Combinatorial Opti- mization The existing RL formulation for computing circuit optimization adheres to widely adopted paradigm neural combinatorial optimization [6, 8], commonly referred to as the "learn- to-improve" framework. In this paradigm, the state is defined represents a local modification to the solution, and the reward is based on the performance improve- ment achieved. This paper theoretically and empirically demonstrates that the RL-based formulation tends to converge to local optima, primarily due to deceptive reward signals and incrementally lo- calized actions. These findings provide valuable insights for developing more robust and effective methods applicable to a broad class of neural combinatorial optimization problems.
candidate solution, the action
G LICENSES
We credit the following open-source code and data used in this paper. We will also open-source our code once the paper is accepted.
Environment
1. OpeneoadFFowwScripts
BSD 3-Clause License
2. OpenRoad BSD 3-Clause License
3. Yosys ISC License
4. EasyMAC No License
29

---

Published as a conference paper at ICLR 2025
Algorithms
1. GOMIL No License
2. MBPO MIT License
3. NovelD Creative Commons Public Licenses
H MORE RESULTS
H.1 MORE RESULTS OF MAIN EVALUATION
The details about the hypervolume of MUTE and other baselines can be found Table and Table
7 Table [ records the hypervolumes of the method on four multipliers with different bit-widths based the And-Gate, showing that method has the greatest improvement. Table shows improvements on the multipliers based on Booth-encode.
Table 6: We record the hypervolume of multipliers based And-Gate. The results demonstrate that MUTE has the maximum hypervolume on each circuit design task.
8-hi And
16-bi And
328ii And
64-1it And
Martoads
Hyperbblume
Impromement:C (5))
Hyporkelume
Imperement (15)
Hyperkolmme
Impersonene (3))
HyperNatume:
Imprmvemntt
Wullace
L4P
332.91
NA
1685.61
No.
13871180
NA
GOMIL
155
2,05%
394,
18.45%
3304.07 67
95/051
16015.51
RLAMUL
622.55
87.00%
36419.85
Table 7: We record the hypervolume of multipliers based Booth-encode. The results demonstrate that MUTE has the maximum hypervolume on each circuit design task.
S-hi Booth
16-bi Booth
32-hi Busth
64--hii Boxths
Methods Wallace
HoperVolume
Imspeveneett (5)
Improvement (%6
HyperVolame
Imprevenent (15)
HyyesViiuute 13184.57
Imperveneent (5)
504
NA.
625.70
NS
2003 401
N/A
NA.
HAVE
339.72
11,43%
975.94
55.981)
7452.70
64.13%%
25910.38
366.33
20.36%
1060.86
69.55%
8057.07
99.17%
29441.15
Table 8: The runtime of MUTE is comparable to or shorter than that of the recent state-of-the-ar HAVE, while MUTE significantly improves hypervolume.
163x Ard
3a Ara
MUTH (tken)
11:35
631.55
-333
31.11
0481 A/
-131
15.37
425.3
5948
-131
HAVE
20.33
309.53
433
27-22
5522.0
-381
NEBTO
254
491.53
4276.1
401
MFEA
10.
433.30
400
183
$471.00
400
MIRBO
123
093.9
$449
H.2 RUNTIME COMPARISON OF MUTE WITH BASELINES
The results in Table S indicate that the runtime of our method is comparable or shorter than that of the recent state-of-the-ar HAVE (Wang et al. 2024g). while our method significantly improves the hypervolume of found Pareto points.
H.3 MORE RESULTS OF COMPARISON WITH RL METHOOS
Figure 10 illustrates the Pareto frontier of our MUTE and all RL-based methods. Moreover, we provide the results of hypervolume on 16-bit Booth and bi- Booth in Table Through the table and figure, we can observe that MUTE outperforms other RL methods comprchensively, achieving the smallest area and delay.
30

---

Published as a conference paper at ICLR 2025
Table 9: Results of comparison with RL methods on 16-bit Booth and 32-bit Booth.
1i- Booth
32-bit Booth
Methods
HyperVolume
Improvement (%)
HlyperVolume
Improvement (%)
Wallace
625.70
NA
4045.40
NA
RL-MUL
897.02
43.36
6090.67
50.56
NovelD
873.31
39.57
6436.80
59.11
PD-MORL
871.02
39.21
6639.44
64.12
DDQN
894.74
43.00
6504.40
60.79
MBPO
942.20
50.58
6056.30
49.71
RL-EA
910.00
45.44
6970.98
72.32
MBPO-HA
932.86
49.09
6772.67
67.42
MUTE (Ours)
1060.86
69.55
8057.07
99.17
(And)
32-bit (And)
(Boohh)
(Booth)
Delay (me)
1.3
Delay (ms)
2,5
Dalay (nt)
2.00
2.0
1.1-
1.50
1.5
1.0
1750
2000
22b0
2500
7000
8000
9000
10000
1500
1750
2000
8000
7000
8000
Area (um²)
Area (um²)
Area (umm")
Area (Wm))
MUTE (Ours)
NovelD
MBPO
DDON
PD-MRL
RL-MUL
MBPO-EA
Figure 10:
The results demonstrate that multipliers optimized by MUTE consistently and signifi- cantly outperform designs produced by all RL-based methods in terms of Pareto-dominance across four multiplier design problems.
H.4 MORE RESULTS OF GENERALIZATION
Table shows the hypervolume of PE arrays designed by MUTE and other baselines. MUTE achieves the highest hypervolume across all circuit designs.
H.5 MORE ABLATION STUDY
The Importance of Our Genetic Evolution Formulation Although have compared our
Table 10: MUTE significantly outperforms ad- vanced (evolutionary) RI. methods.
MUTE with three specifically designed RL methods for computing circuits optimization, i.e., RL-MUL, AdaReset, and HAVE, the three methods are all based on the deep Q-network (DQN) algorithm (Mnih et al.) 2015). which is a classical RL method. To further demonstrate the superiority of our formulation over the ex- isting RL. formulation, we further apply five ad- vanced RL methods to multiplier design tasks. Specifically, we compare MUTE with four ad- vanced RL methods, including NovelD (Zhang ctt al. (021). PD-MORL (Basaklar et al.) (022). DDQN (Van Hasselt al. (016), and МBPО
1sbb.As
22 Bi And
Hyperollame
Inperveentt
Hyper/otune
Ingroeeenn:
Wallice
332.91
NA
2685.5
Ni
Specilicaly Desgged RL. Metrods
5120.11
AdaRRea HAVE
471.20
42.14
5768.20 20
242.21
504.4
51,67
5823,00
245.40
Adhmnced Standerd Mahvods
NevelD
20
42.14
4953-57
PO-MORL
465.03
45.69
4663.43
175.78
DIOO MILO
473.30
41.14
4773.5) 11
181 19
47
4976.11
195
VEB RL
485.00
45,69
$402.00
235.48
Our
Formulaioss
MUTE (Ous))
87,00
6461.65
Janner et al. 2019), and an evolutionary RI. method, i.e., VEB-RL (Li et al. 20246).
The results in Table 10 suggest the following key conclusions. (1) MUTE significantly outperforms these advanced (evolutionary) RL methods, demonstrating the superiority of our proposed circuit genetic evolution formulation. (2) Advanced RL methods do not consistently and significantly out- perform DQN-based circuit optimization methods, i.e., RL-MUL, AdaReset, and HAVE This im- plies that the multiplier optimization task diverges significantly from standard RL benchmarks. such as Mujoco control (Todorov et al. 2012), due to its unique challenges.
31

---

Published as a conference paper at ICLR 2025
Table 11: We record the hypervolume of PE arrays across four PE array design problems. The
results demonstrate that MUTE has the maximum hypervolume on each circuit design task.
And
Booth
Methods
HyperNalume
Imprevement (5)
HygerVolame
Improvement (%)
HyperVolum:
Insesvement/()) (5)
HyperVoluee
ImgrovenenttS
Wallace
731735.2 23
NA
329263.02
N/y
156874.00
NAA
692705 30
N/.
GOMIL
$4R20107
16.00%
627041 10
00:445
124168.7 70
11:02/8
628020.70 700
-0 344%
RL-MUL
108507.00 50
41.65%
944995.00
187.00%
219779 20
40. 1110%%
1049519.00 00
$1.51%
Adulfeset
100073.20
42.42%
1009747700
A75
23081699
47.14%
1237266
78.01%
HAVE!
108227
1019152.00
9275
(Cuus)
258221.60
1385264.80
32


Reviews:
[{"review_id": "lWEzy6eHuv", "reviewer": "Reviewer_UJgx", "summary": "Paper presents a genetic algorithm based optimization framework (MUTE) to finding more hardware efficient multipliers and adders with respect to area and latency. The main degrees of freedom are the placement of the compressor blocks in the compressor trees. The results clearly show that their framework finds more optimal pareto fronts compared to existing RL based optimization approaches and standard synthesis techniques.", "questions": "1)  Have you considered hybrid RL / genetic optimization approaches?  What about other optimization algorithms (simulated annealing w/ predictive heuristics)?  Can you compare MUTE's performance with a simple simulated annealing + heuristics?  \n2)  Can you consider adding some context and tie-in to the larger benefits of this work to the AI/ML community? Either outside of circuit design, or even w/in circuit design, how it helps to advance the AI/ML field? \n3)  From Table 3, there is a relatively small difference in hypervolume when removing individual components in the ablation study.  Can you provide a more detailed analysis of the trade-offs between performance gains and computational costs for each component? Can you also do the same for a simplified version of MUTE since it seems like this could be an interesting approach? \n4)  Can you consider adding a discussion of the complexity and runtime tradeoffs in a new sub-section in the results section rather than the Appendix?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1)  clear problem statement and objectives, with a lot of discourse on RL related work\n2)  good discussion of the theory regarding multi-variable optimization and pareto fronts /hyper volume metrics\n3)  nice coverage of the limitations of pure RL approaches\n4)  sound results including lots of experiments and ablation studies \n5)  novel application of genetic algorithms for HW optimization", "weaknesses": "1) limited discussion/exploration for hybrid RL/discrete optimization algorithms \n2) lack of discussion about the larger contributions to AI/ML community (how this helps) \n3) section 4.4 on the cascade ranking and sampling are important to the paper, but many details are deferred to the appendix.   \n4) Table 3 with the contribution for each component is a bit dubious - it suggests to me that the components may not be as important as stated in the paper since they all come close to the all inclusive framework  results\n5) not sharing framework / code during evaluation phase (only conditionally based acceptance)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29JOKbOYXI", "reviewer": "Reviewer_cbXg", "summary": "This paper tackles a fundamental challenge in circuit design by proposing an innovative approach called MUTE, a model-based circuit genetic evolution framework. The authors effectively identify limitations in current reinforcement learning (RL)-based methods, such as deceptive reward signals and limited action scope, which hinder achieving optimal circuit designs. MUTE addresses these gaps by introducing a grid-based genetic representation, allowing for a global search of design solutions that enhance performance across various circuits, including multipliers, adders, and multiply-accumulate circuits. Experimental results are impressive, with MUTE consistently outperforming state-of-the-art methods in area and delay optimization, achieving up to a 38% improvement in hypervolume metrics. Additionally, MUTE shows strong scalability to larger, more complex circuits, making it a highly promising direction for future research in circuit optimization.", "questions": "- The authors assume that deceptive reward signals arise from inconsistencies between the RL and original optimization objectives, which may contribute to the oscillations observed in Figures 2 and 6. Could you elaborate on how this assumption impacts the results and whether analyzing the search space and optimization landscape in related works like RL-MUL, AdaReset, and HAVE might clarify these oscillations?\n- The paper mentions an RL-guided mutation strategy for circuit design, yet details about the Q-network, including its structure, learning process, and action space, are limited. Could the authors provide more specifics on how the Q-network decides on mutations and manages invalid designs resulting from these mutations? Further explanation could clarify how the framework ensures robustness in the mutation process, especially in Section 4.3, where details on this aspect are limited.\n- Why do the authors choose a two-stage cascade ranking model instead of a single, adaptive ranking model that could be continuously updated with real-world evaluations? Could you justify this choice by explaining the specific benefits of two models over one in the context of fitness evaluation and performance estimation? An ablation study is recommended to compare the proposed two-stage ranking model with a single ranking model, an adaptive ranking model, and real-world evaluation-based ranking. How would each strategy affect performance estimation accuracy, Pareto ranking preservation, and evaluation time?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors effectively identify and address significant shortcomings in existing RL-based methods, specifically the issues of deceptive reward signals and localized actions, which often lead to suboptimal solutions in circuit design. \n- The proposed framework, MUTE, is a fresh perspective on circuit design optimization. Specifically, MUTE's grid-based genetic representation allows for a more comprehensive and global search of optimal design solutions, providing a significant advantage over RL methods that may focus too narrowly on localized actions.\n- The paper backs its claims with extensive empirical evidence, showing that MUTE outperforms state-of-the-art methods on fundamental computing circuits like multipliers, adders, and multiply-accumulate circuits.\n- The evaluation section is comprehensive and detailed; the results are also promising. They demonstrate the effectiveness of the MUTE framework, with improvements in hypervolume metrics by up to 38% and clear and measurable advancements in area and delay optimization.\n- Overall, the paper is very well written and flows nicely. Also, by introducing a framework that can Pareto-dominate existing and recent methods in terms of area efficiency and speed, the paper makes a notable contribution to the field of circuit optimization.", "weaknesses": "- The authors posit an assumption on deceptive reward signals—through the inconsistency between the RL and original optimization objectives. While a detailed theoretical proof is provided in the appendix, the oscillations shown in Figure 2 and Figure 6 are highly related to the search space and the scope of the underlying optimization problem. The authors are recommended to study the optimization objective landscape and the convergence into local optima in previous RL-based works, notably RL-MUL, AdaReset, and HAVE.\n- The idea behind RL-guided mutation to apply strategic modifications to the circuit design is compelling. However, details about the learned Q-network and how it guides the mutation are missing in the paper, particularly in section 4.3. The authors should add more details about the Q-network model, learning, action space (i.e., what mutations are to be applied), and more importantly, how the evolutionary framework handles invalid circuit designs after mutations.\n- The rationale behind leveraging two cascade models to rank the explored circuit designs needs to be further justified–specifically, why use two models instead of one ranking model that can be updated with real-world evaluation during the evolution process? Furthermore, details about the learned model to estimate the fitness and performance of the circuit design are missing. The authors are recommended to emphasize the fitness evaluation stage and running methods proposed in this paper by conducting an ablation study when using a single ranking model, a single adaptive ranking model (whose learning parameters can be updated by some real-world evaluations during the optimization process), the proposed two-stage ranking model, and ranking based on real-world evaluations only. The four strategies should be compared for performance estimation accuracy, Pareto ranking preserving, and evaluation time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dbfnrBAGSG", "reviewer": "Reviewer_JnCS", "summary": "This paper proposed a novel model-based circuit genetic evolution (MUTE) framework for optimizing computing circuits such as multipliers and adders by proposing a grid-based genetic representation of design solutions, which is true objective value rather than proxy rewards used in reinforcement learning (RL) frameworks. This work also proposes a multi-granularity genetic crossover operator to promote globally diverse exploration. Experiment results demonstrate MUTE’s superior ability to optimize the circuit to achieve excellent performance in terms of area and latency and is able to generalize to large-scale computation-intensive circuits.", "questions": "Genetic Evolution Algorithms (GA) are established algorithms developed years ago. Is MUTE the first GA approach applied to optimize computing circuits? Are there any other global optimization algorithms, such as simulated annealing or other types of EA algorithms, used in this problem? The majority of the paper seems to narrow the related work to only EA and RL, which might not be the whole picture of this problem. I am curious about the advantages and novelty MUTE has over other traditional global optimization algorithms.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "• The paper is well-written and has both theoretical and experimental details to support its argument about RL's focus on local optimization.\n• MUTE achieves relatively better local optimum compared to previous methods in empirical results.", "weaknesses": "• The paper argues that RL’s objective function is a proxy since it optimizes cumulative reward rather than the best reward achieved in the trajectory. This might be a significant problem in greedy RL algorithms such as DQN. There are other RL algorithms, such as PPO, that use entropy to promote exploration to escape local optimum. How does MUTE compare to this type of RL algorithm that promote exploration during optimization?\n• The runtime for MUTE is longer than that of previous EA methods and some RL methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)