Learning Semantic Representations to Verify Hardware Designs
Authors: Shobha Vasudevan
Wenjie Jiang
David Bieber
Rishabh Singh
HAMID SHOJAEI
C. Richard Ho
Charles Sutton
Venue: NeurIPS 2021 Poster (2021)
Keywords: Hardware Design
Verification
Graph Convolutional Networks
Test generation

Abstract: Verification is a serious bottleneck in the industrial hardware design cycle, routinely requiring person-years of effort. Practical verification relies on a "best effort" process that simulates the design on test inputs. This suggests a new research question: Can this simulation data be exploited to learn a continuous representation of a hardware design that allows us to predict its functionality? As a first approach to this new problem, we introduce Design2Vec, a deep architecture that learns semantic abstractions of hardware designs. The key idea is to work at a higher level of abstraction than the gate or the bit level, namely the Register Transfer Level (RTL), which is somewhat analogous to software source code, and can be represented by a graph that incorporates control and data flow. This allows us to learn representations of RTL syntax and semantics using a graph neural network. We apply these representations to several tasks within verification, including predicting what cover points of the design will be exercised by a test, and generating new tests that will exercise desired cover points. We evaluate Design2Vec on three real-world hardware designs, including an industrial chip used in commercial data centers. Our results demonstrate that Design2Vec dramatically outperforms baseline approaches that do not incorporate the RTL semantics, scales to industrial designs, and can generate tests that exercise design points that are currently hard to cover with manually written tests by design verification experts.


Learning Semantic Representations to Verify Hardware Designs
Shobha Vasudevan Google Research, Brain Team shovasu@google. com
Wenjie Jiang
David Bieber
Google Research, Brain Team
Google Research, Brain Team
@google.eogl...ooooo com
dbieber@google.cm com
Rishabh Singh
Hamid Shojaei
Richard Ho Google
Google X
Google hamids@google..om com
risingggoogle.com com com
riho@google.com
Charles Sutton Google Research, Brain Team charleessutton@google.... com
Abstract
Verification a serious bottleneck in the industrial hardware design cycle, routinely requiring person-years of effort. Practical verification relies "best effort" process that simulates the design on test inputs. This suggests a new research question: Can this simulation data be exploited to learn continuous representation of a hardware design that allows us to predict its functionality? As a first approach introduce Design2Vec, deep architecture that learns
this new problem, semantic abstractions of hardware designs. The key idea is to work at higher level abstraction than the gate the bit level, namely the Register Transfer Level (RTL), which is similar to software source code, and can be represented by a graph that incorporates control and data flow. This allows us to learn representations of RTL syntax and semantics using graph neural network. We apply these representations to several tasks within verification, including predicting what cover points of the design will be covered (simulated) by test, and generating new tests to cover desired cover points. We evaluate Design2Vec on three real-world hardware designs, including the TPU, Google's 's industrial chip used in commercial data centers. Our results demonstrate that Design2 dramatically outperforms baseline approaches that do not incorporate the RTL semantics and scales to industrial designs. It generates tests that cover design points that are considered hard to cover with manually written tests by design verification experts in fraction the time.
Introduction
Hardware designs are verified to check if design implements the architectural specification correctly. Verification is widely considered the most serious bottleneck in the contemporary industrial hardware design cycle [2 [5] It requires 60-75% of the time, compute, and human resources during the design phase, routinely taking multiple person-yeary of effort. Verification is complex problem because modern hardware has billions of inputs and flops, and the number of states is exponential in the number of flops and inputs. Checking every state the system infeasible due to combinatorial
Equal contributors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

state space explosion. While there is excellent research in automatic hardware verification techniques involving formal and static analysis, their applicability in practice is limited by scale. Instead, practical design verification sacrifices automation and completeness for a "best effort, risk reducing" process based on simulating the design on test inputs P
Traditionally, formal verification methods analyze synchronous hardware designs through reachability analysis of gate level state transition graph. Each node in this graph corresponds to single value of bit-level state of all registers, and edges correspond to the legal changes in state that the design can make in a single clock cycle. It is well known in the formal methods community that most questions about hardware functionality can be posed as reachability questions in the state transition graph. This suggests a fundamental new research question in representation learning: Can we learn a continuous representation of a hardware design that allows us to predict its functionality?
This paper presents first approach this problem. Since designs are practically verified by simulations using millions of test inputs, this gives us a ready source of training data. To avoid the combinatorial explosion at the gate level, we approach the problem at higher level of abstraction, the Register Transfer Level (RTL) that describes hardware at the bit-vector or integer level. RTL is described in a Hardware Description Language (Verilog RTL (3I) that is syntactically similar to the source code of software, while modeling the concurrent, non-deterministic, non-terminating, and reactive semantics of hardware. Despite the higher level of abstraction, RTL static analysis approaches for reachability analysis [4] and test input generation [31] do not scale to even reasonably sized practical designs. So our research question can be restated as: Can we use simulation data to learn a tractable continuous representation that can predict the answers to the state reachability queries in hardware?
In this paper, we introduce Vesi an architecture for learning continuous representations that capture the semantics of hardware design. We choose to represent the design as a control data flow graph (CDFG) at the RTL level. Based on the CDFG, we use graph neural networks (GNN) to learn representations that predict states reached by the design when simulated on test inputs. While standard GNN architectures do work well, we achieve improved performance by introducing new propagation layer that specifically incorporates the concurrent and con-terminating semantics of RTL. Design2 Vec is trained to predict if, given a simulation test input, a particular branch (like a case statement in software) will be covered. The CDFG is an abstraction of the gate-level state space since one edge in the CDFG maps to many edges in the gate-level transition graph. We can therefore interpret Design2Vec as learning an abstraction of the intractable gate-level design state space.
We apply Vesign2Vec to two practical problems in hardware verification: coverage prediction and test generation. The coverage prediction model can act as proxy simulator, and engineer can query it to estimate coverage in seconds, instead of waiting for night of simulations. Our test generation method uses a gradient-bai based search over the trained Design2 Vec model to generate new tests. is desirable to find tests for hard to cover points, or points human experts find difficult to cover after reaching around 80% (or more) cumulative coverage using random testing. This coverage plateau can take months to close, effectively taking multiple expert months of productivity and effort.
We demonstrate Vec is able to successfully learn representations of multiple designs: (i) IBEX v1 M. a RISC-V V design (ii) IBEX v2, IBEX enhanced with security features, and (iii) Tensor Processing Unit (TPU) [23], Google's 's industrial scale commercial infrastructure ML accelerator chip, and performs dramatically better than black-box baselines that are uninformed by knowledge of the design (up 50% better, and on average 20% better for real designs). Our results show that Design2Vec achieves over 90% accuracy in coverage prediction on the industrial TPU design, making it ideal to serve as a proxy simulator that can evaluate if test can cover a given point within seconds. Our results on test generation show that Design2 Vec is able to successfully find tests for hard to cover points in real designs in a fraction of the time (up to 88 fewer simulator calls for TPU and 40 fewer for IBEX) as compared to random testing and black-box optimizer. This order of magnitude improvement can potentially lead to huge savings. Our key contributions follow.
We introduce the problem of learning continuous representations (abstractions) of hardware seman- ties that can be used for various tasks in hardware verification. We present Design2 Vec, model that encodes the syntactic and semantic structure of an RTL design using a GNN based architecture.
2 All future references to verification will imply simulation based design verification as the majority practice in industry, not formal verification.
2

---

We propose the RTL IPA-GNN, which extends recent architecture for learning to execute software [6] to model the concurrent and tonminating semanties of RTL.
We introduce a test generation algorithm that uses Design2 to generate focused tests for a subset of cover points. The tests generated by Design2Vec are able to successfully cover hard to cover points using significantly fewer simulations than state of practice human expert guided random testing and state of the art black-box optimization tool.
We demonstrate the scalability of the approach large industrial scale designs. Our approach is practical in industrial settings.
2
Design2Vec: Representation learning of design abstractions
Verilog RTL source code is powerful tool for writing non-terminating, continuous, reactive "pro- grams" R to design hardware. A program in RTL is structured as a modular hierarchy, with multiple concurrent blocks executing within each module. We consider the subset of behavioral RTL that can be synthesized into gate level designs (3]
Figure I shows an example snippet of Verilog RTL source code. A "module" corresponds to hardware construct (e.g. a decoder or ALU). A module can instantiate other modules (e.g. cache module may instantiate prefetch module). Each "variable" in the RTL. corresponds to either an input signal (input), an output signal (output), a register that stores values (reg) or a temporary variable (wire). The bit width of each such variable is declared. In the example, a, b, and state are two-bit registers, C, d, p. and q are two-bit input signals.
This design has three concurrent (always) blocks denoted by three colors. For synchronous hardware, an external clock signal triggers execution of each concurrent block per clock cycle. On each clock cycle, one statement in each always block is executed. The statements in an always block are thereby executed in sequence over multiple consecutive clock cycles. When executing a statement, the input values of variables in the current cycle come from the output values of variables from the previous cycle. There an implicit loop from the end of each block back to start, simulating the non-terminating nature of hardware. Within a cycle, the order in which the always blocks are exccuted is non-determiiistic. In practice, it is as determined by an RTL simulator. Figure 4 in Appendix A shows the execution (simulation) over three cycle window of this example.
RTL designs are typically simulated using an RTL design simulator, which does not suffice for the design goals of our analysis. For use with machine learning, we choose to represent RTL as control data flow graph (CDFG). The CDFG we construct for each Verilog RTL program encodes the program's 's simulation semantics, so that our models may make inferences about the their behavior.
The CDFG has nodes and heterogeneous edges. Nodes correspond statements in code. Edges corresponds to either data flow or control flow. Dotted lines show data dependencies between concurrently executing blocks. The root node corresponds the begin node of the top module. Branches denote localized node sequences. The designs that we consider contain branches, but not loops, which are atypical in synthesizable RTL.
test is set of high level parameters, each of which can be Boolean, integer or categorical. A test defines distribution inputs from this distribution, and simulates the design on that sequence of inputs. The output of the testbench is Boolean imput vectors applied to the RTL design under test. These input vectors execute different design paths. All the branches executed along different design paths are said to be covered.
input signals to the design. When test is run, the testbench samples many
A test is written to achieve different types of design code coverage. Branch coverage tends to be the most important metric. Branch coverage of a test refers to branches executed by that test. Each branch is referred to cover point. A cover point over the RTL CDFG is identified as a sequence of nodes starting from the branch control node and ending at the destination of the branch. Two local branch cover points lie along the same global path if they have amcestor/descendaat relationship. Figure 5 in Appendix A. shows the executed branches the input arrives at a certain clock cycle.
2.1 Architecture
Our Design2 Vec architecture, shown in Figure 2 is trained on the supervised task of coverage prediction. The network takes as input a cover point C, represented as a sequence of CDFG notes, and test parameter vector I, and outputs the probability is s_ihitC, I) that the test I covers C.
3

---

module Alc. d.p.C. stote)
parameter active - 2'001:
c>d
parameter idie se-=2 220100
Branches (local)
input [=0] c.
n2 anpe1
n1-n2-n4
Input (s:0]
n1-n3-n4
[1:0]
M
n5-n6-n8
n5-n7-n8
always@(")
ced
n9-n10-n12
ifce
n9-n1t-n12
aise
e
always@(*)
07: biq+1
Paths (global)
nB
n1-n2-n4-n9-n10-m12
always
(Nc-8) b 9:1
n5-n7-n8-n9-n10-m12
alsa
b ge1
nio: state=active
state-idle
endmodule
Figure 1: An example snidDet of Verilog RTL source code module and the corresponding CDFG. is hit f(cover point, test)
FeedForward
hit (o/1)
Concat
Node fusion (LSTM)
Feed Forwert
Cover point
CDFG
Modes
Inputs (varete))
andia
rlat
Storage
GCN
Mendity
Rode vor butss
Totersnmbed
Branch
(conotarti)
Telkrss
trouts/coomtetto
Figure 2: The Vesi architecture that takes as input cover point and an input test vector, and predicts the corresponding coverage. CDFG: Control and data flow graph. TP: test parameter. CP: cover point.
Cover point embedding: We use graph convolution network (GCN) over the Verilog CDFG, Each node N has attributesffattues denoted by f such as node identifier, node type, fan in and fan out. In addition, the RTL token sequence s" =< s/ : . considered as an additional attribute. We first perform an embedding of the token sequence ((s") using an LSTM (or alternatively using pre-trained word2vec module):
% > in the program statement is also
((s") = LSTM(< sf, ................................ >).
The initial embedding node & ((0) (m) is computed as concatenation of the node attribute values and the token sequence embedding:
(0) (n) Concat(< ((s"), ff >).
Let the embedding of a graph G a step denoted by (G) ( (n) N). We use GCN (with learnable parameters 0 to perform sequence of graph convolution steps to update the node embeddings:
(1(++)))) fo (G))
VE (0.1.....77)
cover point C' =< n1, n2. . 7m > is a sequence of nodes. The cover point identity is used to select the corresponding sequence of node embeddings of the cover point nodes. Since these sequences are of varying length, an LSTM layer is used to produce the cover point embeddings
(C) = LSTM( (m1),, (m2), (T()
(T) (nm) >)
(4)
Test parameter embedding: For the input test parameters I, each of which can integers or categorical, we learn an embedding for the parameters by passing them through feed forward MLP layer as
(()) = MLP(Concat(i, (..)).
(5)
MLPP layer: Finally, the test parameter ((I) and cover point embeddings (T) are concatenated and provided to a feed forward layer with sigmoid activation to predict for each cover point
is hit(C,( MLP, L.Po(CCoaat(( (T) (C), o(1)))
(6)

---

Given a supervised dataset of test inputs and the corresponding coverage information of different cover points the RTL design, the network is trained using binary cross entropy loss. We find that our model learns more from local paths than global paths (root to cover point). A reason for this is that GNNs are not proficient at learning long path information. the case of a control flow, tracing and learning global paths is relevant. We use shortcut edges, where we add edges along every kth node along a path, to reinforce the relationship of long paths (results Appendix D). We use variation of the GNN, called the GNN-MLP 7 to propagate edge information. We also used gating layers and residual layers in the GGNN.
2.2 RTL IPA-GNN architecture
We enhance the recently proposed Instruction Pointer Attention Graph Neural Network (IPA-GNN) architecture [6] that explicitly models control flow programs. This section motivates and precisely describes each of the architectural enhancements novel RTL IPA-GNN architecture makes over the original IPA-GNN in order to model the concurrent, non-terminating semantics of RTL.
First, since RTL hardware is highly parallel, the RTL IPA-GNN maintains a separate instruction pointer for each always block in the hardware design. The original includes only a single instruction pointer. Making this change allows modeling the concurrent execution of all modules in an RTL specification. Accordingly, the RTL IPA-GNN instantiates its soft instruction pointer Pt.n as
Po,x 1 is always block start mode
and our RTL IPA-GNN implementation computes its hidden state proposals as
= Denne(hh.1..)).
Second, the domain of RTL requires the model support switch conditions, not just binary conditions. So, we modify the soft branch decision mechanism of the IPA-GNN. In the RTL IPA-GNN, the soft branch decisions at timestep given
by mm softmax (Dense(ho-1m)) Embed(er,m)).
where m E (Z,) is control node child of Zn.
Embed(en,,n) is embedding for the control edge from I, to Zm. It includes embedding of whether the condition is positive or negative, the first variable referenced by the condition, and the form of the condition. It is computed three properties of the condition represented by the edge.
concatenation of learned embedding of each of these
Third, data flow in an RTL design is also more complex than in single-threaded program. We model data flow in the RTL IPA-GNN architecture by propagating messages between nodes along data flow edges at each step of the model. This entails aggregating hidden state proposals both from control node state proposals (these are within a node's ' always block) and from the proposals of other parent nodes (which may be from different always block), giving
Σ
Pi-1.- be, n
"nn' +
Σ
On
12 (
Here Newl S N denotes the set control nodes in the CDFG.
Finally, we introduce control edge from the end of each always block to the start of the always block to model non-termination, a critical RTL property. With this update to the CDFG, the instruc- tion pointer probability mass in the RTL IPA-GNN returns to the start of each always block after reaching its conclusion. Incorporating RTL semantics into our modeling decisions improves coverage prediction performance in some settings compared with domain independent GNNs.
2.3 Gradient-based search for test generation
We now present an algorithm to generate tests to cover different cover points in an RTL design using a trained Ves model. S the overall flow. The key idea of Algorithm I ii to perform a gradient based search to maximize the Design2Vec predicted probability of covering desired cover points. The algorithm takes input set C of uncovered points for which would like
5

---

Algorithm Test Input Generation
C
Pick uncovered
Compute gradient of
Gradde
Param!
1: Input: A set uncovered points C (C1, tar i C
NN(C) with
Parmm22
respect to
Parum
4:05
3: while C * do PiclBandonCorsePPonn((((
Parm?
Adjust pammeter value based grodiert informationn
5:
6:
for = 1...K do
C= C
Parm3
hordback/
2
Tuw
(Optimize prob. parameters é- randon()
Design2Vec
Panam!
8:
Parum
while has converged
10:
I -- (itt___hh(CCC))
coverage
11:
12:
I.append( end for
13:
Figure 3: The overall workflow to Design2 in gener- ative loop generate input test parameters.
end while
18:
return
to generate tests. For each uncovered point E C, the algorithm first selects random test a vector of test parameters. The algorithm then computes the objective function as the predicted probability of covering the input cover point, and computes gradients of the objective function with respect each parameter of I. The test parameters in I are updated using gradient ascent method until / converges (Line T0 Algorithm []. This process is repeated from different random initializations list of K test parameters. Finally, the tests with highest predicted coverage are run through an RTL simulator to get the actual coverage.
to get a
Impact of Design2 Vec solutions in practical verification
In this section, detail the verification flow, and highlight the value proposition brought by the Design2Vec solutions to this problem. Figure 6 the Appendix A.2 shows the industrial verification flow. The current approach for verifying RTL designs constrained random verification [9] This involves a complex program called the testbench. The input to the testbench is set of test parameters written by verification engineers that define the distribution over inputs that will be applied to the RTL design. Some inputs need to be constrained, while others will be randomized by the testbench to widely sample the input space. The testbench output Boolean level stimulus that is applied to the primary inputs of the RTL design. Each such high level "test" corresponds to millions of cycles of Boolean input stimuli that run through overnight regression. Bug reports and coverage assessed at the end of a nightly regression. This process is iterated upon until there are 0 bugs and 100% of the design cover points are covered. Simulator calls are costly and need to be minimized. Effectively, this entire process costs multiple engineer years of productivity and resources.
Figure 7 iillustrates how Design2 would integrate in the loop. We select two tasks for Design2 Vec to provide practical value constrained random verification: (i) Coverage prediction, or predicting which cover points in the design would be covered by given test. Such predictive model can serve as proxy simulator, and an engineer can query it to estimate coverage instantaneously, instead of waiting for night of simulation. (ii) Test parameter generation, or automating tests towards faster coverage closure, especially for covering hard-to-cover points. in the verification cycle.
Coverage prediction experiments
We first evaluate the Design2 architecture on the task of coverage prediction. We evaluate two key research questions: (i) whether Design2Vec is able to exploit information from the RTL hardware design improve predictions and (ii) whether coverage prediction good enough for Design2Vec to serve as a proxy simulator.
We evaluate our methods the three real-worls designs from Table 6. For each design, obtain training data by generating random tests and sampling each test parameter uniformly. For each test, use the testbench to randomly sample input test stimulus, and Verilog RTL simulator obtain ground-truth labels of whether a cover point is covered by that test or not, in the form of a coverage
In this paper we use tool that is better than human guided random testing as baseline for hard to cover points.

---

vector. We sample 1696 tests for IBEX v1, 1938 for IBEX v2, and 1995 for TPU. Some cover points are very easy to predict. e.g., they are covered by almost 0% or nearly 100% of the random tests. To avoid evaluating our models on these trivial cases, we only include cover points in the data if they are covered by between 10% and 90% of the random tests. After this filtering, there are 160 cover points in the data set for the IBEX v1 design, 177 for IBEX v2, and 3781 for TPU.
Furthermore, there a particular type of generalization that is vital to applying coverage prediction in practice. Recall that when a design engineer uses a coverage prediction model, it is because they are proposing a new test that should exercise a cover point that is not exercised by the existing test. Therefore, in practice the primary concern is how well the model predicts whether a test that it has never seen during training will exercise a cover point that the model has never seen how to cover during training. For this reason, we divide the data into training and test point by cover point, such that no cover point and no test occurs in both the training and validation set for the learned models All results are the median over three random train-validation splits.
We ask two key questions in our experiments: 1) does representation learning allow Design2 to generalize from cover points in the training set to related cover points (e.g. neighbors in CDFG) and 2) do the graph neural networks provide deeper learning of CDFG graph structure and cover point relationships than more shallow representations?
Table compares Design2 Vec to three baselines across all three designs, varying the size of the training set. One is the naive statistical frequency bascline of guessing the most common value; is the average positive rate over all cover points in the validation data (or (1 (1-average positive rate), whichever is larger). This baseline does not take into account any correlations between cover points or test inputs. The second baseline is multilayer perceptron (MLP) that treats the design as a black box, and does not take into account any information from the RTL. The MLP takes as input the test parameters, represented the same in Design2 Vecc and the numeric index of the cover point, represented as a one-hot vector. While it cannot generalize across cover points, the MLP can still learn to generalize across test parameters (e.g.. some test parameters activate many cover points, some few). In that sense it is stronger than statistical frequency.
The third baseline is node sequence embedding, stronger baseline that enhances MLP by allowing it to generalize across cover points. Recall that every cover point is defined as a sequence of nodes down control-flow path from the root of always block to a particular node, e.g., m1-n2-n4 in Figure 1. We use node sequences over all the training cover points in a Word2Vec model to learn embeddings for each control flow node, which is then concatenated and padded into a cover point embedding. This representation of the cover point is used instead of the one-hot representation in the MLP. Cover points that have structural proximity in the CDFG graph, e.g., m1-22-n4 and n1-n3-n4 would have similar embeddings. This baseline can generalize to new cover points if there is a nearby coverpoint in training. While it takes graph structure into account, does not have the full flexibility of GNN-style message propagation. First two baselines answer question I while node sequence embedding helps answer 3.
In these results, Design2 Vec uses the RTL IPA-GNN Section 2.2p as the GNN layer. Hyperparameters including number of layers, learning rates, and embedding dimensions are reported in Appendix [B]
Notably, the MLP performs catastrophically poorly on the test data, is slightly better than statistical frequency. Indeed, on the largest design, the industrial TPU design, Design2 has an accuracy of 47% higher in absolute terms than the MLP. We observe that the training accuracy of MLP is high (above 95%), the MLP overfits and fails to generalize to cover points outside the training set. This is expected, since it has no information about which of the training cover points are most close to the cover points in validation set. The MLP prediction is based only on test parameter features. Node sequence embedding performs much better than the two blackbox models in every case, indicating that even shallow representation learning using RTL CDFG structure helps generalization.
The Design2 Vec model wins by a substantial margin in every case, indicating that the GNN based architecture is able to learn deep relationships in the design and generalize effectively to majority of the cover points unseen during training.
WWe avoid using the term "test set" to refer to the data on which we evaluate our machine learning methods. avoid confusion with tests of the RTL design.
7

---

Table 1: Accuracy at coverage prediction for Design2Vec (with the RTL IPA-GNN layer) compared
to black-box multi-layer perceptron (MLP), which does not have information about the design.
IBEX vl
IBEX v2
TPU
Train cover points
50%
80%
90%
50%
80%
90%
50%
80%
90%
Design2Vec
74.2
77.3
77.8
73.4
78.0
80.3
90.5
90.6
91.1
Node seq embedding
59.7
59.1
63.2
58.5
57.3
59.0
87.9
88.4
88.6
MLP Statistical frequency
57.5
56.8
56.8
58.7
58.0
58.2
42.8
42.5
34.7
50.5
51.6
50.8
54.1
54.5
54.7
68.5
68.6
68.6
Table 2: Coverage prediction accuracies of different GNN architectures within Design2Vec.
IBEX v1 80%
IBEX v2
TPU
Node fusion
50%
90%
50%
80%
90%
50%
80%
90%
GCN
Istm
74.0
73.0
73.2
70.4
74.5
73.9
90.9
90.5
91.1
mean
74.1
75.8
74.0
69.0
73.9
72.5
GGNN
Istm
75.0
75.9
74.5
71.4
77.0
76.0
91.1
90.4
91.2
mean
73.8
75.6
76.4
70.0
75.1
72.3
GNN-MLP
Istm
73.8
76.8
78.5
70.7
74.5
71.2
91.0
90.1
91.1
mean
73.0
76.4
74.4
70.1
75.1
72.0
RTL IPA-GNN
Istm
74.2
77.3
77.8
73.4
78.0
80.3
90.5
90.6
91.1
mcan
73.6
77.2
76.3
72.2
75.8
78.9
This pattern holds across all three designs. On IBEX v1 and v2, the difference between the two models is smaller (although still around 15% absolute). This is due to the irregular structure of the lbex designs, with fewer repeated hardware modules, causing the control flow path to be harder to predict. On the TPU industrial design is the largest and most complex of the three, Design2 has over 90% accuracy. The node sequence embedding also has high accuracy on this design. This can be attributed the highly regular structure of TPU with a relatively easier to predict control flow.
For Design2Vec, we also compare several variants of graph neural networks: graph convolution networks 25),, gated graph neural networks 28 GNN-MLP (7), and the RTL IPA-GNN (Section 2.2). These results are shown in Table 2 Additionally, we vary the method by which representations of CDFG nodes aggregated to represent cover points. Recall from (Section 2.1 that to represent cover point, we take the final node embeddings from the GNN, apply a mask to obtain only a few relevant CDFG, and then aggregate them. We compare using an LSTM, as in 4 simply taking the mean. We compare this for different GNN architectures, and note that aggregation methods performed similarly on IBEX, so we evaluate only LSTM aggregation on TPU. Overall, all GNN variants performed similarly. RTL IPA-GNN performs similarly to the other GNN architectures on IBEX v1 and TPU, but is significantly better on IBEX v2. Since the RTL IPA-GNN performs similarly or better than other GNNs across the three designs, we use this as the main Design2 Vec model Table We also tuned several other features of the architecture, including presence of the residual connections, embedding size, and label smoothing (details in Appendix B and C).
Given that TPU is a typical ML accelerator chip whose architecture will have similar proper- ties across generations, the high coverage prediction accuracy (>90% shows the potential of Design2Vee Vec to serve as a proxy simulator taking seconds instead of night of simulations.
5 Test generation using Design2Ves Vec
In this section, we evaluate Design2Vec the test generation task, especially for hard to cover points during coverage closure. In general, there are two RTL testing approaches, namely directed testing and random testing. Directed testing refers to targeted testing of specific functionality in RTL, whereas random testing refers to undirected, random perturbation of inputs with the goal of covering the design space maximally. Design2 Vec based test generation is directed and should ideally be compared with a directed testing baseline. However, we did not find a comparable directed testing tool in the open source (or commercially)
At this time, there is no practical tool for automated directed testing in industry. Some directed testing tools have been proposed in literature but are not available in the public domain for comparison. Random testing tools in open source use different tool flows and RTL language, making comparison infeasible.

---

We present two comparative studies between Design2 and Vizier SRP, a random testing tool [8 that uses black-box optimization to maximize total coverage all cover points with every test. This comparison is inherently unequal due to the difference in their objectives of the two tools. We compare both tools with respect to coverage achieved and number of simulator calls made by each tool to achieve that coverage. Given that simulator calls are expensive, this is a relevant metric to compare.
Vizier SRP functions as follows. In each trial, it generates a test for the total set of cover points in the design, and calls the simulator. uses actual coverage feedback in an active learning loop along with Bayesian optimization to generate tests with progressively higher coverage.
In the first study, we find points that are to hard to cover for Vizier SRP and challenge Design2 Vec to generate tests for those points. To find hard to cover points, we run Vizier SRP until around 80% is seen that the remaining uncovered points indeed rare from cover probabilities on a randomly sampled dataset (not used for training).
cumulative coverage is reached. From Table 3
We intercept Vizier SRP after running for number of trials (200 for IBEX vl, 243 for TPU). We provide the same sample tests collected by Vizier SRP as training data for Design2 Vec. For each point uncovered until that point by Vizier SRP, we provide them as (unseen) target points for the Design2 Vee test generation algorithm described in T
Design2 Vec generates multiple test recommendations, of which we select the top ranking recommen- dation and call the simulator for evaluating actual coverage. If the target cover point uncovered, we generate a different test using Design2 Vec and repeat the same procedure until it is covered. For this study, we run a small number of Design2Ves Vec tests (25). Table Appendix demonstrates some example parameterized tests generated by Design2
We continue to run Vizier SRP further (upto 400 trials) and record the coverage at every trial (between 201-400) If an uncovered point gets covered, we measure the number of simulator calls (trials) taken by Vizier SRP cover for the first time and compare with the number of simulator calls (tests) used by Design2Vec to cover it. Note that Vizier SRP uses active learning for the remaining trials until 400, while Design2 Vec uses zero shot learning in this configuration.
After 200 trials of Vizier, there were 22 uncovered points in IBEX and 23 uncovered points in the TPU. Table shows the results of this comparison. Both Vizier SRP and Vec cover 3 cover points. Vizier SRP covers 2 cover points that Design2 does not. Design2 Vec covers cover point within 12 tests that Vizier SRP does not upto 400. Similarly, for 882, due its low probability, Vizier SRP needs 24 tests to cover it, while Design2 Vec uses only 3 tests to cover. Neither Design2Vec nor Vizier SRP cover 16 cover points. We expect Design2 Vec to cover more if it is run beyond 25 tests. Table similar results on the hard to cover points in the TPU design.
In the second study, we compare Design2Vec with Vizier as test generation tool for overall coverage. We train the two tools independently with different datasets, closer to the practical use case. We evaluate the total coverage of Vesign2 as compared to Vizier for sample of randomly selected points with varying cover probabilities (in the spectrum of always covered to rarely/never covered). We randomly select 10 cover points from each of the three buckets of cover probabilities and hid these 30 cover points from Design2Vec. We generate tests for each of the 30 cover points using individual cover points as the search objective and report the number of tests required for Design2 Vec and Vizier to hit the cover point for the first time. Table 12 in Appendix shshwws the summarized observations over the 30 randomly selected points. Design2 Vec is clearly valuable when generating tests hard to cover points.
In practice, the intended use case of Vesign2 is to complement Vizier SRP. While Vizier SRP can maximize for cumulative coverage, Design2Vec can generate tests for hard to cover points. Notably, hard to cover points take lot of time and resources to cover in practice. These results show the power of representation learning in Design2Vec. Despite the lack of examples covering hard to cover points, Vec is able to generate a test to cover these cover points with orders of magnitude fewer simulator calls than a black-box optimizer and random testing.
6 Related work
Automated test generation for RTL: Most previous techniques for RTL test generation generate Boolean level stimuli at the inputs of the RTL design [36]] 31
32
27
g
By virtue of operating at

---

Table 3: Comparison of Ves and black-box optimizer tests for covering hard to cover points: IBEX v1. Number of tests are RTL simulations.
Table 4: Comparison of Design2 and black-box optimization to hit target cover points: TPU. Number of tests are RTL simulations.
Cover point
Number of used tests
ID
Prob.
Vizier
Design2 Vec
401
0.0012
Not covered
29
526
0.0059
Not covered
12
Cover point
Number of used tests
528
0.0035
Not covered
10
ID
Prob.
Vizier
Design2 Vec
879
0.0071
42
14
35793
0.0
22
17
882
0.0059
24
36996
0.0
2
886
0.0018
93
25
36372
0.0
78
17
664
0.01
55
Not covered
881
0.0053
108
Not covered
14 cover points
Not covered
Not covered
Table 5: Comparing test generation of easy, medium and hard to cover points. Design2Vec is very efficient generating tests for hard cover points. Summarized result of table in Appendix Table
Cover probabilities
Difficulty
Summary of comparison
[0.5, 1.0)
Easy
Both Vizier and Design2 cover all points with a single test.
[0.2,
Medium
Vizier and Design2 Vec cover 9 out of 10. Design2 Vec takes 3 fewer tests on average.
[0.05, 0.2)
Hard
Design2Vec takes 20 fewer tests than Vizier on average. Upto 40 fewer in cases.
higher abstraction level of parameterized tests, it is orders of magnitude more scalable than the Boolean input stimulus generation techniques. Static analysis based approaches for test generation are based on traversing the state transition graph at a logic gate level, or the RTL design have inherent scalability limitations. Approaches that combine static and dynamic analysis like
34
19
5
HYBRO 29 and concolic testing [36, 31 32] in RTL rely SMT/SAT solvers, which are also limited by scale. Other approaches random forests and decision trees with static analysis and formal verification 30 14] require manual feature engineering and handcrafted algorithms.
Neural program testing: While our approach learns the semantics of a design, learning based fuzzing 17 does not take program semantics into account. In contrast our approach that models semantics of an RTL design, Neuzz
355 approximates a program using only the input-output behavior of the program, which is similar our black-box MLP baseline. GMETAEXP (iI] models test generation as a reinforcement learning problem with the objective of maximizing total program coverage, where programs are represented using a GGNN.
Learning to execute: Among methods that model program semantics, 38 present an approach to use LSTM to embed program as input and generate the output as the output sequence. IPA-GNN 6 represents the program execution using an RNN augmented with a differentiable mechanism to represent next instruction after a statement execution. Unlike these works that learn to generate program output, we tackle the problem of reachability, or learning to reach a state/node in graph. Abstractions for verification: Abstraction techniques have been used to scale design verification for
many decades. Some techniques are property specific [10 226. design specific (20), data abstractions (word level abstractions) 124 37), language based (abstract interpretation) 112 (22), execution or
structure based 13. These abstractions are typically defined manually and are challenging to create in precise way to both scale the verification as well as maintain desired precision. In contrast, GNN based abstractions are task-specifie, in the form of continuous representations of the CDFG nodes, that generalizes both across tasks, as well as designs.
7 Conclusion
We present an approach that is able to learn representations of hardware designs that can predict their functionality. With Vesi we demonstrate, for the first time, the ability of deep learning in creating abstractions of the hardware design state space. These abstractions outperform black- box baselines and human baselines (state of practice) in verification by orders of magnitude in performance as well as scale. They can also generalize across different designs. Since Design2 Vec learns reachability over the RTL design graph, can also potentially generalize to other verification tasks like debugging, property generation and root causing. More broadly, this work shows the power and potential of deep learning to make quantum leap in progress in the area of verification, which considered a longstanding practical and scientific challenge in computing.
10

---

Acknowledgements
We thank Milad Hashemi and Sat Chatterjee for insightful discussions. We thank Dexter Aronstam and Gil Tabak for developing the key components of Design2 Vec infrastructure that enabled us to perform experiments.
References
[1] IBEX RTL source. httpss///ithuo..com////////////
[2] Functional Verification
study.
https://blogs.aww www.://swww....mee......ee.e siemens com/ the- www--too--wss-no---eeeee- researcch group-
vor1fi/ctiommorroooo//////////0/
[3] leee standard for systemverilog-unifee hardware design, specification, and verification lan- guage. IEEE Std /800-2017 (Revision of IEEE Sid 1800-2012), pages 1-1315, 2018. doi: 10.1 10.1109/EEFSTTD 201.2222989..99
[4] Viraj Athavale, Sai Ma, Samuel Hertz, and Shobha Vasudevan. Code coverage of assertions using rtl source code analysis. In 2014 51st ACM/EDAC/IEEE Design Automation Conference (DAC), pages 1-6, 2014. doi: 10.1145/2593069 259
[5] K. Bansal and M. S. Hsiao. Optimization of mutant space for rtl test generation. In 2018 IEEE 36th International Conference on Computer Design (ICCD), pages 472-475, 2018.
[6] David Bieber, Charles Sutton, Hugo Larochelle, and Daniel Tarlow. Learning to execute programs with instruction pointer attention graph neural networks. CoRR, 122220,0.12621, 2020. URL httpt: htttp::/aarxvv..orr/bb/////
12621.
[7] Mare Brockschmidt. GNN-FiLM: Graph neural networks with feature-wise linear modulation. In International Conference on Machine Learning (ICML), 2020.
[8] Eris Furkan Taylor Michael Bedford Egele Manuel Canakci Sadullah, Delshadtehrani Leila and Joshi Ajay. Directfuzz: Automated test generation for designs using directed graybox fuzzing. In 58th JEEE/ACM Design Automation Conference Proceedings, 2021, 2021.
[9] Supratik Chakraborty, Kuldeep S. Meel, and Moshe Y. Vardi. Balancing scalability and uniformity in SAT witness generator. CoRR, abs/1 0bs/1403.6222 403.6246, 2014. http:///t//a org/ hbs/1403.6244
[10] Edmund Clarke, Orna Grumberg, Somesh Jha, Yuan Lu, and Helmut Veith. Counterexample- guided abstraction refinement for symbolic model checking. J. ACM, 50(5):752-79) -744444 September 2003.
[11] Hanjun Dai, Yujia Li, Chenglong Wang, Rishabh Singh, Po-Sen Huang. and Pushmeet Kohli. Learning transferable graph exploration. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, NeurIPS, pages 2514-2525, 2019.
[12] Dennis Dams, Rob Gerth, and Orna Grumberg. Abstract interpretation of reactive systems. ACM Trans. Program. Lang. Syst., 19(2):253-22) 2911 1997.
[13] Sistla A.P. Emerson, F.A. Symmetry and model checking. In Formal Methods in System Design, pages 105-131, 1996.
[14] Monica Farkash. Bryan Hickerson, and Michael Behm. Coverage learned targeted validation for incremental hw changes. In Proceedings of the 51st Annual Design Automation Conference, page 1-6, 2014.
[15] Harry Foster. Why the design productivity gap never happened. In Jörg Henkel, editor, The IEEF/ACM International Conference Computer-Aidest Design, ICCAD' 13, San Jose, CA. USA, November 18-21, 2013, pages 581 1884 IEFE, 2013. doi: 10. 10.1109/ICAA.2001 201 3.6661166117 URL
https://doii TRRLhtts::/
org/ 10. 1109/ ICCAD. 2013 6691175

---

[16] Harry D. Foster. Trends in functional verification: a 2014 industry study. In Proceedings of the 52nd Annual Design Automation Conference. San Francisco, CA, USA, June 7-11, 2015, pages 48:1 18:66 ACM, 2015. doi: 10. (0.1144522744777774499) URL https:t:s://doi org/ 10. 1145/ 7744769.274499
[17] Patrice Godefroid, Hila Peleg, and Rishabh Singh. Learn&fuzz: machine learning for input fuzzing. In Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen, editors, ASE, pages 50-59. IEEE Computer Society, 2017.
[18] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, and D. Sculley, editors. Google Vizier: Service for Black-Box Optimization, 2017. URL http:://wwz http//wwwwdd... wog.doogg/..oooooo black-box- alack-box-optimmzaaiioo
[19] Yang Guo, Wanxia Qu, Tun Li, and Sikun Li. Coverage driven test generation framework for rtl functional verification. In 2007 10th IEEE International Conference on Computer-Aided Design and Computer Graphics, pages 321-326, 2007.
[20] Rajiv Gupta, Mary Jean, Harrold Mary, and Lou Soffa. Program slicing-based regression testing techniques+. Journal of Software Testing, Verification, and Reliability, 6:83-112, 1996.
[21] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016. doi: 10.1 10.1109/CVPRR2011
[22] Charles Hymans. Design and implementation of an abstract interpreter for vhdl. In Daniel Geist and Enrico Tronci, editors, Correct Hardware Design and Verification Methods, pages 263-269, 2003.
[23] Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, AI Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami. Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary. Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami,, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps. Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit 2017.
[24] Pnucli A. Kesten, Y. Control and data abstraction: the cornerstones of practical formal verifica- tion. International Journal of Software Tools for Technology Transfer, 2:328-30 342, 2000.
[25] Thomas N Kipf and Max Welling, Semi-Supervised classification with graph convolutional networks. International Conference on Learning Representations (ICLR), 2017.
[26] Daniel Kroening, Alex Groce, and Edmund Clarke. Counterexample guided abstraction refine- ment via program execution. In International Conference Formal Engineering Methods, volume 3308, 08 2004.
[27] Kevin Laeufer, Jack Koenig, Donggyu Kim, Jonathan Bachrach, and Koushik Sen. Rfuzz: Coverage-directed fuzz testing of rtl on fpgas. In 2018 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pages 1-8, 2018.
[28] Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow, Gated graph sequence neural networks. International Conference on Learning Representations (ICLR), 2016.
[29] Lingyi Liu and Shobha Vasudevan. Efficient validation input generation in rtl by hybridized source code analysis. In 2011 Design, Automation Test in Europe, pages 1-6, 2011.
12

---

[30] Lingyi Liu, David Sheridan, William Tuohy, and Shobha Vasudevan. A technique for test coverage closure using goldmine. IEEE Transactions on Computer-Aided Design of Integrated Circuils and Systems, 31(5):790-800) 2012.
[31] Yangdi Lyu and Prabhat Mishra. Automated test generation for activation of assertions in RTL models. In Asia and South Pacific Design Automation Conference (ASP-DAC), page 223-228, 2020.
[32] Yangdi Lyu, Xiaoke Qin, Mingsong Chen, and Prabhat Mishra. Directed test generation for validation of cache coherence protocols. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 38(1):163-176, 2019.
[33] Rafacl Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pages 4696-4705, 2019.
[34] Jen-Chieh Ou, Daniel G. Saab, Qiang Qiang, and Jacob A. Abraham. Reducing verification overhead with rtl slicing. In Proceedings of the 17th ACM Great Lakes Symposium on VLSI, page 399-404, 2007.
[35] Dongdong She, Kexin Pei, Dave Epstein. Junfeng Yang, Baishakhi Ray, and Suman Jana. NEUZZ: efficient fuzzing with neural program smoothing. In 2019 IEEE Symposium on Security and Privacy, SP 2019,, pages 803-81 IEEE, 2019.
[36] Hasini Witharana, Yangdi Lyu, and Prabhat Mishra. Directed test generation for activation of security assertions rtl models. ACM Transactions on Design Automation of Electronic Systems, 26(4), 2021.
[37] Cunxi Yu and Maciej Ciesielski. Automatic word-level abstraction of datapath, In 2016 IEEE International Symposium Circuits and Systems (ISCAS), pages 1718-1721, 2016.
[38] Wojciech Zaremba and Ilya Sutskever. Learning to execute. CoRR, abs/ 1414.4615 2014.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss overall challenges
and limitations of verification in the introduction; although our work aims improve
current processes, it is not immune to these overall limitations of the field.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] This paper considers a technological problem that is several layers removed from direct social impact, so it is difficult to identify specific direct negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Although the code is proprietary, reproduce the main experimental results. Furthermore, we evaluate on the open source IBEX v1 and IBEX v2 designs, that others can reproduce work.
provide details about the model and architectures required to
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section B
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [No] We run experiments multiple times and report the median over random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix B
4. If you are using existing assets (e.g., code, data, models) or curating/reelasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use existing real-world hardware designs, which we cite.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "mPTkr6Z6nm", "reviewer": "Reviewer_z6UE", "summary": "This paper introduces Design2Vec, which models the semantics of RTL programs using a graph neural network. The representation is used for 2 tasks: coverage prediction and test generation. The experiments on different designs show that Design2Vec can achieve high accuracy on the coverage prediction task and can also generate tests that cover hard-to-cover points. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written. The use of learning for the particular task is well-motivated. The architecture of the network appears reasonable and the performance of the model on the two tasks shows a clear benefit of the system. Using a gradient-based method on the learned model to generate tests for some targeted cover points is a clever idea. And it is encouraging to see that this approach can actually generate meaningful tests. There seems to be real potential to deploy Design2Vec in real-world hardware testing scenarios.\n\nThe experimental evaluation is extensive and shows the benefit of the proposed architectures. The only reason that I reserve my score to 6, for now, is that I have two concerns for the second experiment (section 5). First, why do the authors choose Vizier as the baseline instead of existing RTL test generation techniques as mentioned in the first paragraph of the related work? The paper mentions that they are less scalable than the presented technique. Concrete evaluation results of some of those tools on the same designs would strengthen the claim and also provide more validation to the presented learning-based technique. Second, it is not explicit from the paper whether Table 3 and Table 4 contain all the cover points in C_uncov. If this is not the case, are there cover points whether Vizier can cover faster than Design2Vec? If so how many? Are there points that Design2Vec cannot generate tests to cover?\n\nMinor comments:\nline 99: extra 0 at the end of the line.\nline 335: blacklbox -> blackbox\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While the authors discuss the general challenges in hardware verification, they apply mostly to formal verification. It would be beneficial if the authors could discuss challenges specific to test-based methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "ZwGRV83Gv41", "reviewer": "Reviewer_N6sS", "summary": "This paper introduces a graph based deep architecture that can learn and understand hardware design. Experiment results suggest good performance in generating test for hardware verification.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a novel model Design2Vec to embed RTL design with a graph based architecture. This paper is well organised and have enough information for readers to follow. The authors provided illustrative figures to go through both RTL architecture examples and network architecture. Experimental results do show much better performance in Design2Vec compare to vanilla network in  verification coverage, and good performance for proposed RTL IPA_GNN. Overall, this is an interesting paper tacking an important problem in hardware verification.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not discuss its limitation. Here are some of my questions and suggestions:\n\t1. Does the proposed method perform better in pure combinational logic (without register), it seems it may be much easier to model without state related registers, it would be interesting to see a comparison between sequential  design and combinational design.\n\t2. How does it scale? What would be the sweet spot in terms of  design complexity to train it on? \n\t3. Is possible to extend this to analog model, probably at least system-verilog.\n\t4. For related work, it would be great to separate them into non-ML based and ML-based, and have a section to state the novelty of the proposed method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2YYJxqptwnW", "reviewer": "Reviewer_D7qc", "summary": "The paper proposes an approach for predicting whether a given set of inputs will induce a given branch to be taken in an RTL program, along with an algorithm for synthesizing inputs to the program that induce a given branch to be taken. The objective is to aid verification of proposed hardware designs by reducing the number of actual simulations required to find test cases that fully cover the control flow graph of the RTL program. The approach is to use a graph neural network, enhanced with some task-specific features, along with a gradient-descent algorithm for finding inputs that induce a given branch to be taken. The paper evaluates the approach, showing that it leads to relatively high branch-taken predictive accuracy on three hardware architectures, and showing that the approach requires fewer evaluations than Bayesian Optimization driven search to find inputs that induce a given branch.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As context on my expertise, I am familiar with the semantics of RTL though not with all of the terminology in the paper nor deeply with the goal of design verification process of hardware. I am more of an expert in using neural networks to learn semantics of programs.\n\n## Originality\n\nTo my knowledge, the paper is original in its approach and domain of application. The most similar approach in the literature is NEUZZ [34], which uses a similar approach for fuzz testing of programs; however, the approach and domain of application in this paper are substantially novel.\n\n## Quality\n\n- Section 2:\n  - This comment does not affect my score, but I'm curious about the rationale behind the choice to include the test inputs at the final MLP rather than earlier in the GNN. Intuitively, I would expect it to be much easier to predict which branches are taken using a GNN that considers the program input along with the program structure, as opposed to a MLP that considers the program inputs and an embedding of the program structure.\n  - The RTL IPA-GNN is claimed as a contribution, but a formal description is not in the main body of the paper. There is also no ablation to compare it to the original IPA-GNN.\n  - There are not enough details provided on Algorithm 1 / Figure 3 to reproduce this work. Specifically, Figure 3 shows a step that says to \"Adjust parameter value based on gradient information\", but doesn't say how to actually do that. Figure 3 also says to \"Pick an uncovered coverpoint\", but this does not correspond to anything in Algorithm 1.\n- Section 4:\n  - Does not evaluate research question (ii), \"whether the performance of Design2Vec can add practical value to industrial-scale designs\", other than stating that 90% accuracy is relatively high (lines 251-254). It is not obvious that 90% accuracy is sufficient, or even that this specific task (predicting whether branches are taken) adds \"practical value\".\n  - MLP baseline: maybe I misunderstand something about this, but if the input is a one-hot cover point that has never been observed in training, shouldn't the MLP essentially be no better than guessing based on frequency of whether arbitrary cover points are taken or not (by symmetry)? Given that this is the case, including the cover points into the MLP model is essentially guaranteed to lower the test accuracy, as it is noise that can be overfit on during the training phase. It is therefore hard to reason about Design2Vec's accuracy in comparison to this unfairly bad baseline.\n- Section 5:\n  - Despite the qualification on line 288 that the \"comparison is inherently unequal\", this section repeatedly compares Vizier and Design2Vec, stating that Design2Vec is superior (by bolding Table 3 and 4, comparing the number of tests required in line 310, etc.). Overall, this evaluation seems backwards to me, since it is not possible to reason about Design2Vec's performance over a flawed baseline. Instead, it might make sense to try to use Design2Vec to perform the same task as Vizier, to evaluate the hypothesis that the GNN-based surrogate model which is then optimized against using the gradients outperforms the Gaussian Process underlying Vizier which is optimized with Bayesian Optimziation; the fact that Design2Vec is able to target specific cover points seems like an additional benefit which cannot be directly evaluated against Vizier.\n  - How were these 3-4 specific hard to cover points chosen? Were these the entire set of points that were identified to take more than 5 night of simulation?\n\n## Clarity\n\nThe mix of background and definitions in Section 2.1 is very hard to understand as someone who is familiar with the semantics but not so much the terminology of RTL. Specifically, the following points are not clear:\n- The inclusion of the phi nodes (n4, n8, n12) or generally the parsing of RTL into the graph is not discussed in the text.\n- Figure 4, as referenced on Line 107, is in the appendix and not the main body of the paper, which the reference doesn't state.\n- Line 99: do the semantics of the execution depend on this nondeterminism? Based on this description, they don't seem to.\n- Line 110: \"localized\" isn't defined here, and I'm not entirely sure what it means. I would have expected a \"branch\" to be a single edge in the graph, not a triplet of nodes (as it seems to be in Figure 1).\n- Line 111: \"root\" is also not well defined, though it's a bit more possible to infer from context.\n- Line 114: After reading this definition of \"cover point\" I'm still not entirely clear on what it is. Is a \"cover point\" exactly the same as a branch?\n- Line 117: I'm not sure what a \"parameter\" is -- do these correspond to the parameters in Verilog? Or are these \"input\"s?\n- Figure 1: what is the meaning of the colors? E.g., why are n5-n8 colored blue in most of the figure but green in the \"Paths (global)\" segment?\n\nOverall, this section would be helped by much more formal definitions, as opposed to the informal English definitions currently used.\n\nOther comments:\n- It would be helpful to include some description of the base IPA-GNN architecture in Section 2.3. Also, if the RTL IPA-GNN architecture is claimed as a contribution, it should be described in the main body of the paper.\n- Line 212: It would be helpful to mention that Table 7 is in the Appendix. Also, Table 7 does not show 3 real-world designs; it seems to only show \"IBEX v1\" and \"FastML block\", and not \"IBEX v2\".\n- For all tables, it would be nice to show some notion of error bars (such as the min and max accuracy across the trials)\n- Line 99: there is an extraneous \"0\" at the end of the line.\n- Line 298: This reference to Table 1 should probably be Algorithm 1.\n- In general, the paper mixes between content in the main body of the paper and content in the appendix too much, such that it is hard to follow what is being proposed and where to see various tables.\n\n\n## Significance\n\nI see two primary directions in which this paper could be significant:\n- The immediate claims of accurately reducing hardware simulation burden, if borne out, could significantly help hardware design\n- The RTL IPA-GNN is an interesting contribution, in that it aims to model the semantics of a language unlike many of the other languages commonly modeled by GNNs (in that it is parallel and nondeterministic). Concretely, RTL IPA-GNN concretely could help future work trying to model RTL, and more abstractly it provides an interesting point of comparison against many other architectures for semantic understanding of code.\n\n## Score\n\nReject.\n\nWhile I find the overall motivation and approach compelling, the paper's evaluation and clarity are not sufficient. My highest order concerns are that:\n- the paper does not satisfactorily evaluate the claim that \"Design2Vec can add practical value to industrial-scale designs\"\n- the baselines used in Sections 4 and 5 (the MLP and Vizier) are inappropriate for comparing against Design2Vec (in that they neither correspond to reasonable and easily understandable baselines, nor are they representative examples of any prior work on the specific tasks under consideration)\n- Section 2 is not clear enough to understand or reproduce the results in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are no significant limitations or potential for negative societal impact that are immediately apparent that the authors don't address.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no significant ethical concerns raised by the paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea of predictive coverage/testing for RTL designs using modern representation learning methods is interesting and potentially useful. Most of the reviewer discussion surrounded the best ways to evaluate such a method; eventually reviewers agreed that the authors did as well as could be expected given the state of open tools for verification -- this broader community issue should be mentioned in the revision. Some reviewers also requested clarifications and details that should be incorporated in the revision. Overall a good step in this application space.
Decision: Accept (Poster)