Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach for predicting whether a given set of inputs will induce a given branch to be taken in an RTL program, along with an algorithm for synthesizing inputs to the program that induce a given branch to be taken. The objective is to aid verification of proposed hardware designs by reducing the number of actual simulations required to find test cases that fully cover the control flow graph of the RTL program. The approach is to use a graph neural network, enhanced with some task-specific features, along with a gradient-descent algorithm for finding inputs that induce a given branch to be taken. The paper evaluates the approach, showing that it leads to relatively high branch-taken predictive accuracy on three hardware architectures, and showing that the approach requires fewer evaluations than Bayesian Optimization driven search to find inputs that induce a given branch.
main_review: As context on my expertise, I am familiar with the semantics of RTL though not with all of the terminology in the paper nor deeply with the goal of design verification process of hardware. I am more of an expert in using neural networks to learn semantics of programs.

## Originality

To my knowledge, the paper is original in its approach and domain of application. The most similar approach in the literature is NEUZZ [34], which uses a similar approach for fuzz testing of programs; however, the approach and domain of application in this paper are substantially novel.

## Quality

- Section 2:
  - This comment does not affect my score, but I'm curious about the rationale behind the choice to include the test inputs at the final MLP rather than earlier in the GNN. Intuitively, I would expect it to be much easier to predict which branches are taken using a GNN that considers the program input along with the program structure, as opposed to a MLP that considers the program inputs and an embedding of the program structure.
  - The RTL IPA-GNN is claimed as a contribution, but a formal description is not in the main body of the paper. There is also no ablation to compare it to the original IPA-GNN.
  - There are not enough details provided on Algorithm 1 / Figure 3 to reproduce this work. Specifically, Figure 3 shows a step that says to "Adjust parameter value based on gradient information", but doesn't say how to actually do that. Figure 3 also says to "Pick an uncovered coverpoint", but this does not correspond to anything in Algorithm 1.
- Section 4:
  - Does not evaluate research question (ii), "whether the performance of Design2Vec can add practical value to industrial-scale designs", other than stating that 90% accuracy is relatively high (lines 251-254). It is not obvious that 90% accuracy is sufficient, or even that this specific task (predicting whether branches are taken) adds "practical value".
  - MLP baseline: maybe I misunderstand something about this, but if the input is a one-hot cover point that has never been observed in training, shouldn't the MLP essentially be no better than guessing based on frequency of whether arbitrary cover points are taken or not (by symmetry)? Given that this is the case, including the cover points into the MLP model is essentially guaranteed to lower the test accuracy, as it is noise that can be overfit on during the training phase. It is therefore hard to reason about Design2Vec's accuracy in comparison to this unfairly bad baseline.
- Section 5:
  - Despite the qualification on line 288 that the "comparison is inherently unequal", this section repeatedly compares Vizier and Design2Vec, stating that Design2Vec is superior (by bolding Table 3 and 4, comparing the number of tests required in line 310, etc.). Overall, this evaluation seems backwards to me, since it is not possible to reason about Design2Vec's performance over a flawed baseline. Instead, it might make sense to try to use Design2Vec to perform the same task as Vizier, to evaluate the hypothesis that the GNN-based surrogate model which is then optimized against using the gradients outperforms the Gaussian Process underlying Vizier which is optimized with Bayesian Optimziation; the fact that Design2Vec is able to target specific cover points seems like an additional benefit which cannot be directly evaluated against Vizier.
  - How were these 3-4 specific hard to cover points chosen? Were these the entire set of points that were identified to take more than 5 night of simulation?

## Clarity

The mix of background and definitions in Section 2.1 is very hard to understand as someone who is familiar with the semantics but not so much the terminology of RTL. Specifically, the following points are not clear:
- The inclusion of the phi nodes (n4, n8, n12) or generally the parsing of RTL into the graph is not discussed in the text.
- Figure 4, as referenced on Line 107, is in the appendix and not the main body of the paper, which the reference doesn't state.
- Line 99: do the semantics of the execution depend on this nondeterminism? Based on this description, they don't seem to.
- Line 110: "localized" isn't defined here, and I'm not entirely sure what it means. I would have expected a "branch" to be a single edge in the graph, not a triplet of nodes (as it seems to be in Figure 1).
- Line 111: "root" is also not well defined, though it's a bit more possible to infer from context.
- Line 114: After reading this definition of "cover point" I'm still not entirely clear on what it is. Is a "cover point" exactly the same as a branch?
- Line 117: I'm not sure what a "parameter" is -- do these correspond to the parameters in Verilog? Or are these "input"s?
- Figure 1: what is the meaning of the colors? E.g., why are n5-n8 colored blue in most of the figure but green in the "Paths (global)" segment?

Overall, this section would be helped by much more formal definitions, as opposed to the informal English definitions currently used.

Other comments:
- It would be helpful to include some description of the base IPA-GNN architecture in Section 2.3. Also, if the RTL IPA-GNN architecture is claimed as a contribution, it should be described in the main body of the paper.
- Line 212: It would be helpful to mention that Table 7 is in the Appendix. Also, Table 7 does not show 3 real-world designs; it seems to only show "IBEX v1" and "FastML block", and not "IBEX v2".
- For all tables, it would be nice to show some notion of error bars (such as the min and max accuracy across the trials)
- Line 99: there is an extraneous "0" at the end of the line.
- Line 298: This reference to Table 1 should probably be Algorithm 1.
- In general, the paper mixes between content in the main body of the paper and content in the appendix too much, such that it is hard to follow what is being proposed and where to see various tables.


## Significance

I see two primary directions in which this paper could be significant:
- The immediate claims of accurately reducing hardware simulation burden, if borne out, could significantly help hardware design
- The RTL IPA-GNN is an interesting contribution, in that it aims to model the semantics of a language unlike many of the other languages commonly modeled by GNNs (in that it is parallel and nondeterministic). Concretely, RTL IPA-GNN concretely could help future work trying to model RTL, and more abstractly it provides an interesting point of comparison against many other architectures for semantic understanding of code.

## Score

Reject.

While I find the overall motivation and approach compelling, the paper's evaluation and clarity are not sufficient. My highest order concerns are that:
- the paper does not satisfactorily evaluate the claim that "Design2Vec can add practical value to industrial-scale designs"
- the baselines used in Sections 4 and 5 (the MLP and Vizier) are inappropriate for comparing against Design2Vec (in that they neither correspond to reasonable and easily understandable baselines, nor are they representative examples of any prior work on the specific tasks under consideration)
- Section 2 is not clear enough to understand or reproduce the results in the paper.

limitations_and_societal_impact: There are no significant limitations or potential for negative societal impact that are immediately apparent that the authors don't address.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a graph based deep architecture that can learn and understand hardware design. Experiment results suggest good performance in generating test for hardware verification.

main_review: This paper presents a novel model Design2Vec to embed RTL design with a graph based architecture. This paper is well organised and have enough information for readers to follow. The authors provided illustrative figures to go through both RTL architecture examples and network architecture. Experimental results do show much better performance in Design2Vec compare to vanilla network in  verification coverage, and good performance for proposed RTL IPA_GNN. Overall, this is an interesting paper tacking an important problem in hardware verification.

limitations_and_societal_impact: This paper does not discuss its limitation. Here are some of my questions and suggestions:
	1. Does the proposed method perform better in pure combinational logic (without register), it seems it may be much easier to model without state related registers, it would be interesting to see a comparison between sequential  design and combinational design.
	2. How does it scale? What would be the sweet spot in terms of  design complexity to train it on? 
	3. Is possible to extend this to analog model, probably at least system-verilog.
	4. For related work, it would be great to separate them into non-ML based and ML-based, and have a section to state the novelty of the proposed method.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces Design2Vec, which models the semantics of RTL programs using a graph neural network. The representation is used for 2 tasks: coverage prediction and test generation. The experiments on different designs show that Design2Vec can achieve high accuracy on the coverage prediction task and can also generate tests that cover hard-to-cover points. 
main_review: The paper is well-written. The use of learning for the particular task is well-motivated. The architecture of the network appears reasonable and the performance of the model on the two tasks shows a clear benefit of the system. Using a gradient-based method on the learned model to generate tests for some targeted cover points is a clever idea. And it is encouraging to see that this approach can actually generate meaningful tests. There seems to be real potential to deploy Design2Vec in real-world hardware testing scenarios.

The experimental evaluation is extensive and shows the benefit of the proposed architectures. The only reason that I reserve my score to 6, for now, is that I have two concerns for the second experiment (section 5). First, why do the authors choose Vizier as the baseline instead of existing RTL test generation techniques as mentioned in the first paragraph of the related work? The paper mentions that they are less scalable than the presented technique. Concrete evaluation results of some of those tools on the same designs would strengthen the claim and also provide more validation to the presented learning-based technique. Second, it is not explicit from the paper whether Table 3 and Table 4 contain all the cover points in C_uncov. If this is not the case, are there cover points whether Vizier can cover faster than Design2Vec? If so how many? Are there points that Design2Vec cannot generate tests to cover?

Minor comments:
line 99: extra 0 at the end of the line.
line 335: blacklbox -> blackbox

limitations_and_societal_impact: While the authors discuss the general challenges in hardware verification, they apply mostly to formal verification. It would be beneficial if the authors could discuss challenges specific to test-based methods.
review_title: null

Meta-review: The idea of predictive coverage/testing for RTL designs using modern representation learning methods is interesting and potentially useful. Most of the reviewer discussion surrounded the best ways to evaluate such a method; eventually reviewers agreed that the authors did as well as could be expected given the state of open tools for verification -- this broader community issue should be mentioned in the revision. Some reviewers also requested clarifications and details that should be incorporated in the revision. Overall a good step in this application space.
Decision: Accept (Poster)