LLM4DV: Using Large Language Models for Hardware Test Stimuli Generation

## Abstract

Abstract Test stimuli generation has been a crucial but labor-intensive task in hardware design verification. In this paper, we revolutionize this process by harnessing the power of large language models (LLMs) and present a novel benchmarking framework, LLM4DV. This framework introduces a prompt template for interactively eliciting test stimuli from the LLM, along with four innovative prompting improvements to support the pipeline execution and further enhance its performance. We compare LLM4DV to traditional constrained-random testing (CRT), using three self-designed design-under-test (DUT) modules. Experiments demonstrate that LLM4DV excels in efficiently handling straightforward DUT scenarios, leveraging its ability to employ basic mathematical reasoning and pre-trained knowledge. While it exhibits reduced efficiency in complex task settings, it still outperforms CRT in relative terms. The proposed framework and the DUT modules used in our experiments will be open-sourced upon publication.

## 1 Introduction

Design verification (DV) checks the correctness of hardware designs. The verification process takes in inputs, or test stimuli, passes them into the hardware design-under-test (DUT), and compares the result to expected outputs from a software golden model [Witharana et al., ]. The scope of the verification, also known as a coverage plan, is defined and agreed upon in advance, ensuring all functional properties of the design are tested.

A coverage plan in DV is defined to specify a list of coverage points to be tested, which are particular outputs and machine states that the verification process need to cover. Each cover point is associated with a coverage bin that counts how many times the cover point has been exercised. The goal of the verification process is to achieve $100\%$ functional coverage rate based on the coverage plan, meaning that all coverage bins would have a non-zero value. [Figure 0(a)](#S1.F0.sf1) illustrates an example coverage plan where different coverage bins can together form their cross coverage bins.

Effective test stimuli generation has been a major challenge in meeting $100\%$ coverage [Witharana et al., ]. For a simple design, verification can be done with individual directed tests, in which test stimuli (inputs for the DUT) are manually generated. For more complex designs, a large number of stimuli is required for exercising as much of the design’s functionality as possible. Traditionally, constrained-random testing (CRT) has been used for generating vast random but valid test stimuli and attempting to “hit” the bins. However, CRT is inefficient to hit bins with complicated conditions. Often, this necessitates extensive human engineering involvement in the test stimuli design process.

Figure: (a) Example coverage plan
Refer to caption: x1.png

Recent large language models (LLMs), trained on immense amounts of text and human feedback data, are capable of solving tasks of distinct form that require multi-step reasoning and generating high-quality samples [Brown et al., , Lightman et al., , Touvron et al., ]. For example, Codex [Chen et al., ] can produce functionally correct code bodies from natural language docstring descriptions. Open-source model Llama 2 [Touvron et al., ], using instruction tuning and Reinforcement Learning with Human Feedback (RLHF) [Christiano et al., , Stiennon et al., ] for fine-tuning, emerges impressive generalization and tool usage ability. This brings up an important question: can LLMs effectively minimise the amount of human effort involved in hardware Design Verification (DV)? This study specifically focuses on one arguably most labor-intensive aspect in the DV process - generating test stimuli - and explores the potential of using LLMs to assist with this task.

In this work, we establish a benchmarking framework named LLM4DV (Large Language Model for Design Verification). This is a benchmark framework that utilizes LLMs for test stimuli generation. In this paper, we make the following contributions:

- •
We design and construct three DUT modules: a Primitive Data Prefetcher Core, an Ibex Instruction Decoder, and an Ibex CPU.
- •
We propose and develop a framework that uses prompted LLMs to generate test stimuli for hardware designs. Also, we propose four improvements on the prompting strategies to improve the effectiveness of the generated stimuli.
- •
We evaluate the framework with the three DUT modules. We show that our framework with optimal improvement measures achieves $98.94\%$, $86.19\%$, and $5.61\%$ coverage rates on these modules respectively in a realistic experimental setup. We will open-source our framework with these three modules to allow both the machine learning and hardware design communities to experiment their ideas.

## 2 Related Work

While the application of LLMs on hardware design verification has been brought to focus only recently, test code generation for software engineering with LLMs has been well-studied and made remarkable performance [Chen et al., , Schäfer et al., , Lukasczyk and Fraser, ]. Chen et al. [] utilized LLM to generate potential implementations and test cases for a function by providing the LLM with the signature and a description of the function and select the best solution based on functionality agreement. Schäfer et al. [] proposed a pipeline to generate unit tests for existing code, which iteratively refine the prompt to generate better tests. These studies show promising results on software code, while we shift focus to using LLM on Hardware Description Language (HDL) and hardware design specification, which involves a different setting and more sophisticated procedure.

In hardware design verification, assertion-based verification (ABV) is also widely adopted together with code coverage, functional coverage, and validation using generated test patterns [Witharana et al., ]. ABV uses assertions in HDL to detect violation of predefined properties. Kande et al. [] proposed a pipeline for automatically generating SystemVerilog assertions (SVA) with LLM to ensure hardware security. However, ABV requires test patterns (i.e. input test stimuli) to activate given assertions and therefore reveal vulnerabilities.
For simulation-based ABV approaches, traditional test generation using random or constrained-random tests cannot guarantee to activate assertions with complex conditions in a reasonable time. To speed up assertion coverage, Pal et al. [] proposed bias random test generation, which considers the DUT as a black box and restricts test generation to only input/output signals. Ferro et al. [] used combinatorial testing, which provides a set of combinations of user-selected values, to select test stimuli most suitable to cover corner cases. Tong et al. [] proposed a method that searches the compact assertion-based automata for failure and acceptance nodes before test generation. Simulation-based test generation has also been incorporated with formal method-based test generation. Lyu and Mishra [] utilized concolic testing to activate assertions, which regards assertions as branches, searches through the branch statement tree with heuristics to efficiently obtain a path, and generates test stimuli to cover the obtained branch targets. These methods, however, are subjected to complexity explosion problems [Witharana et al., ] and fail to effectively make use of user’s knowledge about the design. We overcome this issue by utilizing LLM’s pre-trained knowledge to reason about the given coverage plan and guide the test stimuli generation.

## 3 Method

Our experiments use an LLM in the test stimuli generation process, together with a testbench with a DUT to form the whole LLM4DV framework, as shown in [Figure 0(b)](#S1.F0.sf2). The following subsections describes the basic DV framework, the prompt templates for the LLM, and four prompting improvements. [Figure 2](#S3.F2) gives a general picture of the prompt templates and prompting improvements.

Figure: Figure 2: Coverage-feedback prompt template and four prompting improvements. This framework corresponds to the stimulus generation agent component in [Figure 0(b)](#S1.F0.sf2). We highlight the different design options possible for the four prompting improvements (Best-iterative-message sampling, Dialogue Restarting Plans, Missed-Bin Sampling and Best-Iterative-Message Buffer Resetting Plan).
Refer to caption: x3.png

### 3.1 Basic framework

The testbench has three major components: a stimulus generation agent (the prompted LLM), a design-under-test (DUT), and a coverage monitor. In each timestep, the stimulus generation agent provides a test stimulus and inputs it to the DUT. The coverage monitor inspects the DUT’s inputs, outputs, and internal states; determines whether there are hits of bins; and updates the current coverage and returns it to the stimulus generation agent for generating the next stimulus.

Our basic framework introduce a prompt generator and an LLM as the stimulus generation agent. In each generation cycle, the prompt generator generates a prompt based on a template (see [Section 3.2](#S3.SS2)) and the current coverage feedback from the coverage monitor. The LLM takes in the prompt and generates a natural language response, from which the test stimulus values are extracted by a stimulus extractor, and then pushed into a stimulus buffer.

In every timestep when the stimulus generation agent is requested for a test stimulus, it takes out the oldest value in its stimulus buffer; if the buffer is empty, the agent starts a new generation cycle, in which the LLM takes in a new request and a list of new stimuli will added to the buffer.

The pipeline, or a trial, stops in two situations. First, if all cover points have been reached. Second, if the stimulus generation agent doesn’t hit any bins within 25 responses or hit less than three bins within 40 responses, we suppose the agent is “exhausted” i.e. becoming ineffective or inefficient to cover any new bins, and the pipeline stops. See [Algorithm 1](#alg1) in Appendix.

### 3.2 Prompting templates

We provide a Coverage-Feedback Template to generate prompts for the LLM. When constructing it, we utilize prompt engineering techniques including 1) System message: it is included at the beginning of every prompt, and is used to prime the model with context, instructions, or other information relevant to the use case; 2) Start with clear instructions; 3) Repeat instructions at the end; 4) Add clear syntax: punctuation, headings, and section markers; 5) Specifying the output structure.

#### Coverage-Feedback Template

The Coverage-feedback prompt template contains templates for the system message, initial query, and iterative queries.

- •
The system message clarifies the expected response format and specify other requirements.
- •
The initial query is the first user message in a dialogue. It contains three parts:
1.
Task introduction: a description of what is included in this prompt and what the LLM will be asked to do.
2.
Coverage plan summary: a description of cover points of the coverage plan.
3.
Initial question: a one-line instruction.
- •
The iterative queries are the user messages following the first assistant (LLM) response. Each contains three parts:
1.
Result summary: a general feedback which:
–
if the previous assistant response was gibberish (i.e. contains mostly nonsense words) or didn’t follow the output format, the result summary repeats the output format requirement; otherwise,
–
if the previous assistant response failed to hit any new bins, the result summary points that out and ask for a new list of stimuli;
–
if the previous assistant responses hit some bins, the result summary points that out and ask for a new list of stimuli.
2.
Differences: a list of uncovered bins.
3.
Iterative question: a one-line instruction, repeating the output format requirement if previous response was gibberish or didn’t follow the output format.

### 3.3 Four prompting improvements

In our experiments, we develop two improvements necessary for making the framework executable and two improvements to further improve its performance. We also evaluate different design choices for these improvements. A full ablation study with all these improvements and their design choices are in [Appendix F](#A6) and [Appendix B](#A2), in this section, we mainly discuss the best attempts.

Missed-bin sampling
In most generation cycles in a trial, there would be hundreds to thousands bins uncovered. The iterative queries can’t include all of them because the prompt’s length would exceed the LLM’s input token number limit. Meanwhile, exposing too many uncovered bins to the LLM confuses the LLM on which mistakes should it resolve first.

We propose missed-bin sampling, which samples a number of bins from all uncovered bins to be included in the differences part of iterative queries. Our experiment finds that more random sampling methods encourage the agent to cover bins with stricter hitting conditions, and more stable sampling methods make the agent more efficient in hitting the easier bins.

We define three sampling methods, see green boxes in [Figure 2](#S3.F2). (1) Pure Random Sampling, which randomly samples seven bins from all uncovered bins. (2) Coverpoint Type-based Sampling, which samples from “easier bins” and “harder bins” respectively. (3) Mixed Coverpoint Type-based and Pure Random Sampling, which switches between previous two strategies whenever the agent becomes inefficient with current strategy.

Best-iterative-message sampling
The LLM needs previous messages in the dialogue to learn about what has happened. However, as the dialogue grows, the length of input may exceed the LLM’s input limit. One solution is summarizing previous message, which helps generalizing concepts in the dialogue but loses details, which is crucial in our task. On the other hand, sampling from previous messages acceptably loses some generality meanwhile preserves key details, including the bin description and positive examples (i.e responses that successfully hit many bins) useful for covering edge cover points.

We propose four sampling methods, see blue boxes in [Figure 2](#S3.F2). (I) Recent Responses, where we keep the initial query (and its response), and three most recent iterative queries (and their responses). (II) Successful Responses, where we keep random three iterative queries that hit most number of bins. (III) Mixed Recent and Successful Responses, where we keep two most successful and one most recent iterative query. (IV) Successful Difficult Responses, which is similar to Successful Responses but each “harder bin” counts as 2.5 bins.

Dialogue restarting
LLM sometimes behaves stubborn, repeating mistakes it made previously. We introduce a dialogue restarting mechanism to resolve this problem. When the LLM hits less than three new bins within $t$ responses, we clear the dialogue record and restart from the system message and initial query.

We define four restarting plans, see purple boxes in [Figure 2](#S3.F2). (a) Normal Tolerance, where $t=7$. (b) Low Tolerance, where $t=4$. (c) High Tolerance, where $t=10$. (d) Coverage Rate-based Tolerance, where $t=4$ in the beginning and $t=7$ after reaching certain coverage rate threshold.

Best-iterative-message buffer resetting
When the dialogue record is reset, the buffer for best iterative messages in Best-iterative-message sampling can also be cleared or kept. These two strategies display a trade-off between “effectively forgetting past mistakes” and “learn about the task faster after restart”.

We define three resetting plans for the best-iterative-message buffer, see cyan boxes in [Figure 2](#S3.F2). (i) Clearing best-messages. (ii) Keeping best-messages. (iii) Stable-restart Keeping best-messages: keeping the buffer on restarts, but not using it for the first four responses after restarts.

## 4 Evaluation

We evaluate our LLM4DV framework on three DUT modules: Primitive Data Prefetcher Core, Ibex Instruction Decoder, and Ibex CPU. These modules and their testbenches are constructed by ourselves. With Verilator and cocotb, we could efficiently simulate and test the modules in Python. We use OpenAI’s GPT-3.5-turbo-0613 model [OpenAI, ] in our “fixed-budget” experiments. See [Appendix D](#A4) for DUT details, and [Appendix C](#A3) for experimental setup details.

**Table 1: Results of fixed-budget experiments: A1, A2, A3 contrast Successful Responses (II) + Stable-restart Keeping best-messages (iii) vs. Successful Difficult Responses (IV) + Stable-restart Keeping best-messages (iii) vs. Successful Difficult Responses (IV) + Keeping best-messages (ii). B1, B2 contrast Normal Tolerance (a) vs. Coverage Rate-based Tolerance (d). C1, C2 contrast Normal Tolerance (a) vs. Low Tolerance (b). For details of the configurations, see [Table 2](#A5.T2) in Appendix.**
| Config | Max coverage | Max coverage rate | Avg msg/trial | Stdev msg/trial |
| --- | --- | --- | --- | --- |
| Primitive Data Prefetcher Core |  |  |  |  |
| Random | 4 | 0.39% | - | - |
| A1 | 987 | 95.45% | 641 | 104.24 |
| A2 | 1023 | 98.94% | 617.5 | 165.06 |
| A3 | 1007 | 97.39% | 459.33 | 287.21 |
| Ibex CPU Instruction Decoder |  |  |  |  |
| Random | 1136 | 53.92% | - | - |
| B1 | 1695 | 80.45% | 864 | 147.92 |
| B2 | 1816 | 86.19% | 844.25 | 127.94 |
| Ibex CPU |  |  |  |  |
| Random | 3 | 1.53% | - | - |
| C1 | 10 | 5.10% | 42.49 | 11.74 |
| C2 | 11 | 5.61% | 45.72 | 16.17 |

### 4.1 Primitive Data Prefetcher Core

The Primitive Data Prefetcher Core module requires test stimuli in specific stride patterns. We run the CRT baseline and the fixed-budget experiments with GPT 3.5 and three prominent configurations. [Table 1](#S4.T1) shows the results of the experiments. We see that the best configuration, which uses Coverpoint Type-based Sampling, Successful Difficult Responses, Normal Tolerance, and Stable-restart Keeping best-messages, achieved 98.94% coverage rate. This demonstrates that with our prompting template and improvement methods, the stimulus generation agent is capable of making simple mathematical reasoning to hit the bins efficiently.

### 4.2 Ibex Instruction Decoder

The Ibex Instruction Decoder module takes in RISC-V instruction codes. [Table 1](#S4.T1) shows the results of the experiments. We see that the best configuration, which uses Mixed Coverpoint Type-based and Pure Random Sampling, Successful Difficult Responses, Coverage Rate-based Tolerance, and Stable-restart Keeping best-messages, achieved 86.19% coverage rate. This demonstrates that the stimulus generation agent can make use of its pre-trained knowledge to hit the bins efficiently.

### 4.3 Ibex CPU

The Ibex CPU module takes in a list of updates to its instruction memory in each cycle, and execute the instruction memory accordingly. This module is more complex than the previous two modules. [Table 1](#S4.T1) shows the results of the experiments. We see that the best configuration, which uses Coverpoint Type-based Sampling, Successful Responses, Fast Tolerance, and Stable-restart Keeping best-messages, only achieved $5.61\%$ coverage rate. Although it’s better than the CRT, this shows that on complex task settings, the framework are deficient in producing effective updates and may require further prompt engineering and prompting improvements to enhance its performance.

## 5 Conclusion and Discussion

This study proposes LLM4DV, a benchmarking framework for test stimuli generation in design verification. We build three DUT modules and construct a stimulus generation agent, together with a prompt template and four prompting improvements. Through experimentation on the modules, we show that LLM-assisted test stimuli generation could achieve impressive coverage rates on several design units, and its performance is significantly better than CRT for relatively complex coverage plans. However, when the task setting becomes more complex, for instance, a full CPU design, LLM fails to achieve a fairly good coverage rate.

To our understanding, our study represents the first systematic exploration of utilizing LLMs to assist in the area of hardware design verification. More specifically, we focus on the generation of test stimuli, which is a task known for its high reliance on human intervention.
Our designs and findings will be open-sourced upon publication. Our framework is designed to accommodate arbitrary hardware designs and offers users the flexibility to switch between different LLMs or customize their own prompting strategy.
We also provided three high-quality hardware designs with different difficulties for the users to experiment their new LLMs or prompts.
We anticipate that our developed framework will provide researchers with a user-friendly interface to further explore the potential of LLMs in aiding hardware design verification.

## Appendix A Basic framework algorithm

Figure: Algorithm 1 Basic DV pipeline

[Algorithm 1](#alg1) shows the workflow of the basic framework as in [Section 3.1](#S3.SS1).

## Appendix B Details of prompting improvements

This section describes the design choices of our four prompting improvements. We also compare these design choices with ablation experiments, see [Appendix F](#A6).

### B.1 Missed-bin sampling

We define three sampling methods:

- •
(1) Pure Random Sampling: randomly samples seven bins from all uncovered bins.
- •
(2) Coverpoint Type-based Sampling: we categorize all bins into “easier bins” and “harder bins” based on their difficulties to be covered, and order them based on their names; when sampling, we always take the first two uncovered bins, then either randomly sample five bins from all uncovered bins if there are no “easier bins” left, or sample three “easier bins” and two “harder bins”.
- •
(3) Mixed Coverpoint Type-based and Pure Random Sampling: when the coverage ratio is below $15\%$, it keeps using Coverpoint Type-based Sampling; when the coverage ratio is larger than $15\%$, it switches between Coverpoint Type-based Sampling and Pure Random Sampling whenever the current sampling method hits less than three new bins within four responses. The number of $15\%$ is obtained empirically.

### B.2 Best-iterative-message sampling

We propose four sampling methods:

- •
(I) Recent Responses: keeps the initial query (and its response), and three most recent iterative queries (and their responses).
- •
(II) Successful Responses: keeps the initial query and three iterative queries with responses that hit maximum number of bins; if multiple maximums exist, uniformly samples three of them.
- •
(III) Mixed Recent and Successful Responses: keeps the initial query, two most successful iterative queries, and one most recent iterative query.
- •
(IV) Successful Difficult Responses: similar to successful responses, but each “harder bin” as in [Section B.1](#A2.SS1) counts as $2.5$ bins.

### B.3 Dialogue restarting

We define four restarting plans:

- •
(a) Normal Tolerance: restarts the dialogue if the LLM hits less than three bins within $7$ responses.
- •
(b) Low Tolerance: restarts the dialogue if the LLM hits less than three bins within $4$ responses.
- •
(c) High Tolerance: restarts the dialogue if the LLM hits less than three bins within $10$ responses.
- •
(d) Coverage Rate-based Tolerance: uses Low Tolerance when the coverage ratio is below $15\%$, and Normal Tolerance afterwards. The number of $15\%$ is intentionally set as the same value as in Mixed Coverpoint Type-based and Pure Random Sampling in [Section B.1](#A2.SS1).

### B.4 Best-iterative-message buffer resetting

We define three resetting plans for the best-iterative-message buffer:

- •
(i) Clearing best-messages: clears the buffer on dialogue restarts.
- •
(ii) Keeping best-messages: keeps the buffer on dialogue restarts.
- •
(iii) Stable-restart Keeping best-messages: keeps the buffer on dialogue restarts, but not using it for the first four responses after restarts.

## Appendix C Fixed-budget experiment setup

#### Large language models

We use OpenAI’s GPT-3.5-turbo-0613 model [OpenAI, ] in our experiments. For OpenAI’s API calls, we set parameters as $\texttt{temperature}=0.4$, $\texttt{top_p}=1.0$, and $\texttt{max_tokens}=600$. These parameters are decided empirically. We also use Llama 2 7B model [Touvron et al., ] in ablation experiments, see [Appendix F](#A6).

#### Experimental setup and measurement

We run “fixed-budget experiments” on the three DUT modules, each with certain configurations of prompt template and methods of improvement measures. The configurations are selected as the ones with the most significant results in ablation experiments (see [Appendix F](#A6)).

In each fixed-budget experiment, the experiment routine would have a budget of 10,000,000 tokens (input and output tokens are counted in the same way). Here’s how it works:

- 1.
The experiment routine starts with a trial as described in [Section 3.1](#S3.SS1).
- 2.
The trial either ends as described in [Section 3.1](#S3.SS1) or when it runs out of the token budget.
- 3.
If there is budget left, the experiment routine starts a new trial, in which the stimulus generation agent, the testbench, and the coverage record are reset.
- 4.
The experiment routine ends when the budget is used up.

Running a fixed-budget experiment takes $6$ to $20$ hours, depending on the DUT module and the LLM service status.

The fixed-budget setup reflects the realistic use case where the financial budget for DV is reflected on the token budget. The multi-trial setup allows configurations to trade off “steady slow increase”, where a trial takes many messages but stably achieves a high coverage, and “random fast increase”, where a trial takes less messages but may either reach a high coverage by chance or end up with a low coverage.

We measure the maximum coverage each configuration achieves in trials in an experiment routine as its performance. We use the mean and the standard deviation of message count per trial as a reference for capability and stability of configurations. We don’t measure the size of the union set of covered bins across trials because we intend to test our framework’s capacity across different DUT modules rather than to reach $100\%$ coverage rate on a specific DUT.

We compare our framework to a constrained-random test (CRT) generation agent baseline, which generates 1,000,000 integers within the input range. Particularly, for the Ibex CPU module, the CRT generates an instruction to update the current PC in each cycle.

## Appendix D Details of DUTs

This section explains our three DUT modules and their coverage plans respectively.

### D.1 Primitive Data Prefetcher Core

The Primitive Data Prefetcher Core takes in 32-bit integers and detects whether there’s a stride pattern in it. This module requires relatively high mathematical reasoning capability for the LLM.

Our coverage plan contains 1034 bins of the following types:

- •
Single-stride bins: counts when 16 consecutive integers $a_{0},a_{1},...,a_{15}$ satisfy $a_{i+1}-a_{i}=c$ for some constant $-16\leq c\leq 15$.
- •
Double-stride bins: counts when 16 consecutive integers satisfy an alternative stride width pattern, formally $a_{2i+2}-a_{2i+1}=c_{1}$ and $a_{2i+1}-a_{2i}=c_{2}$ for some $-16\leq c_{1},c_{2}\leq 15$ and $c_{1}\neq c_{2}$.
- •
Misc bins: including
–
Single-stride positive/negative overflow bins: a single stride pattern with $c<-16$ (negative overflow) or $c>15$ (positive overflow).
–
Double-stride pp/pn/np/nn overflow bins: a double stride pattern with $c_{1}$ and $c_{2}$ positively / negatively overflow respectively.
–
No-stride-to-single/double: counts when 16 integers satisfying no stride pattern are followed by 16 integers with single / double stride pattern.
–
Single/double-to-double/single: counts when 16 integers satisfying single / double stride pattern are followed by 16 integers with double / single stride pattern.

### D.2 Ibex Instruction Decoder

The Ibex Instruction Decoder is an instruction decoder for 32-bit RISC-V instruction codes. This module involves almost no mathematical reasoning but requires knowledge about RISC-V knowledge. Through preliminary tests, we find that GPT 3.5 and Llama 2 are pre-trained with RISC-V knowledge, and thus our bin descriptions don’t need to explain the instruction formats and operations in detail.

Our coverage plan contains 2107 bins of the following types:

- •
ALU operation bins: counts when an instruction represents one of 26 pre-defined ALU operations such as ADD, ADDI, XOR, LW, etc.
- •
Register port bins: counts when an instruction uses the port of the specific register. There are 32 registers, and each has two read ports and one write port, which are used when the register file is taken as the first source, second source, and the destination register, respectively.
- •
Cross coverage bins: the Cartesian product of ALU operation bins and register port bins. Counts when an instruction satisfies both bins simultaneously (some of the product, such as ADDI and read_port_A of any register, are invalid and not included in the coverage plan).

### D.3 Ibex CPU

The Ibex CPU is a full RISC-V CPU with instruction and data memory. Different from the previous two modules, in every cycle the agent provides a stimulus of a list of address-instruction pairs $(a,i$, each presents a dynamic update that changes the instruction memory content at address $a$ into value $i$. The CPU performs an instructions in the instruction memory in each timestep and wait for another list of updates (which can be empty). Apart from the current coverage, the testbench also returns the current program counter (PC) and the last executed instruction to the stimulus generation agent. Therefore, the task requires knowledge of both RISC-V and working principle of CPU and memory, and demands managing the complicated task setting.

Our coverage plan contains 196 bins of the following types:

- •
Operation bins: for each of pre-defined ten R-type operations, three S-type instruction, and one J-type instruction (JAL), we consider the following four bins:
–
seen: counts when an instruction performs the operation;
–
zero_dst: if available, counts when the instruction performs the operation, with the destination register (rd) as zero (reg #0);
–
zero_src: if available, counts when the instruction performs the operation, with one of the source registers (rs) as zero (reg #0);
–
same_src: if available, counts when the instruction performs the operation, taking the same register as source registers (rs).
- •
Jump bins: for the JAL operation, we consider forward and backward jumps respectively.
- •
Hazard bins: for each pair of the pre-defined operations, we consider a simplified read-after-write (RaW) hazard, which counts when the later instruction reads from a register that the previous instruction is writing to.

## Appendix E Configurations in fixed-budget experiments

**Table 2: Configurations of fixed-budget experiments, strategies been contrasted in each module are marked in bold fonts.**
| Conf | Improvement | Method |
| --- | --- | --- |
| Primitive Data Prefetcher Core |  |  |
| A1 | Prompt template | Original prompt template |
| Missed-bin sampling | Coverpoint Type-based Sampling (2) |  |
| Best-iterative-message sampling | Successful Responses (II) |  |
| Dialogue restarting | Normal Tolerance (a) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages (iii) |  |
| A2 | Prompt template | Original prompt template |
| Missed-bin sampling | Coverpoint Type-based Sampling (2) |  |
| Best-iterative-message sampling | Successful Difficult Responses (IV) |  |
| Dialogue restarting | Normal Tolerance (a) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages (iii) |  |
| A3 | Prompt template | Original prompt template |
| Missed-bin sampling | Coverpoint Type-based Sampling (2) |  |
| Best-iterative-message sampling | Successful Difficult Responses (IV) |  |
| Dialogue restarting | Normal Tolerance (a) |  |
| Best-iterative-message buffer resetting | Keeping best-messages (ii) |  |
| Ibex Instruction Decoder |  |  |
| B1 | Prompt template | Prompt template with one-line task introduction |
| Missed-bin sampling | Mixed Coverpoint Type-based and Pure Random Sampling (3) |  |
| Best-iterative-message sampling | Successful Difficult Responses (IV) |  |
| Dialogue restarting | Normal Tolerance (a) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages (iii) |  |
| B2 | Prompt template | Prompt template with one-line task introduction |
| Missed-bin sampling | Mixed Coverpoint Type-based and Pure Random Sampling (3) |  |
| Best-iterative-message sampling | Successful Difficult Responses (IV) |  |
| Dialogue restarting | Coverage Rate-based Tolerance (d) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages (iii) |  |
| Ibex CPU |  |  |
| C1 | Prompt template | Original prompt template |
| Missed-bin sampling | Coverpoint Type-based Sampling (2) |  |
| Best-iterative-message sampling | Successful Responses (IV) |  |
| Dialogue restarting | Normal Tolerance (a) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages |  |
| C2 | Prompt template | Original prompt template |
| Missed-bin sampling | Coverpoint Type-based Sampling (2) |  |
| Best-iterative-message sampling | Successful Responses (IV) |  |
| Dialogue restarting | Low Tolerance (b) |  |
| Best-iterative-message buffer resetting | Stable-restart Keeping best-messages |  |

[Table 2](#A5.T2) shows the details of LLM4DV configurations in our fixed-budget experiment results ([Table 1](#S4.T1)), including methods used for prompt templates and strategies of prompting improvements.

## Appendix F Ablation experiments

Due to the cost of money and time for OpenAI’s API requests and experiment running, we compare configurations of the stimulus generation agent by their performances on simpler experiment setups.

### F.1 Ablation experiment setups

We run “limited-message experiments” and “infinite-message experiments” on the Primitive Data Prefetcher Core module and the Ibex Instruction Decoder module. In a limited-message experiment, the experiment routine runs a trial with an upper bound of the number of responses that is smaller relative to the average response number for the trial to “exhaust” (as in [Section 3.1](#S3.SS1)). In a infinite-message experiment, the upper bound of response number is much larger than the average response number for the trial to “exhaust”, allowing the agent to fully exert its capacity.

We use two LLMs in our experiment: OpenAI’s GPT-3.5-turbo-0613 model, and Meta AI’s Llama 2 7B model. For GPT 3.5, we use OpenAI’s API calls, with parameters as $\texttt{temperature}=0.4$, $\texttt{top_p}=1.0$, and $\texttt{max_tokens}=600$. For Llama 2 7B, we put the downloaded Llama 2 directory in the same level as our project directory. We use one A100 GPU with 80GB memory to run the language model. We call the model with parameters as $\texttt{temperature}=0.4$, $\texttt{top_p}=0.9$, $\texttt{max_gen_tokens}=800$, $\texttt{max_seq_len}=10000$, and $\texttt{max_batch_size}=4$. These parameters are decided empirically.

Other experimental setup follows [Appendix C](#A3).

**Table 3: Results of infinite-message experiments on the Primitive Data Prefetcher Core module**
| Config | Max cov (/1034) | Avg cov | Stdev cov | Avg cov/msg | Stdev cov/msg |
| --- | --- | --- | --- | --- | --- |
| (1) | 890 | 664.33 | 195.62 | 2.59 | 0.18 |
| (2) | 953 | 681 | 276.61 | 1.59 | 0.41 |

### F.2 On Primitive Data Prefetcher Core

Figure: Figure 3: Infinite-message experiments on the Primitive Data Prefetcher Core module. Each line represents the trial reaching the maximum coverage on a configuration, and dots on it shows dialogue restarting points. All trials use Original prompt template except Prefetcher 7, which uses Prompt template with negative feedback.
Refer to caption: x4.png

We run infinite-message experiments with several configurations of prompt templates and improvement measures on the Primitive Data Prefetcher Core module.

The results are displayed in [Table 3](#A6.T3) and [Figure 3](#A6.F3). As we can see, on this module:

- •
Coverpoint Type-based Sampling for stride detector bins achieves better coverage but poorer coverage-per-message than Pure Random Sampling.
- •
Successful Responses and Successful Difficult Responses are generally better than Recent Responses.
- •
While Keeping best-messages and Stable-restart Keeping best-messages are generally better than Clearing best-messages, Keeping best-messages is least stable.
- •
Prompt template with negative feedback, which include a one-sentence “what not to do” for each bin in iterative queries, makes no significant improvement than Original prompt template.

Based on these conclusions, we run fixed-budget experiments on three most prominent combinations of strategies, as in [Section 4.1](#S4.SS1).

**Table 4: Results of limited-message experiments on the Ibex Instruction Decoder module**
| Config | Max cov (/2107) | Avg cov | Stdev cov | Avg cov/msg | Stdev cov/msg |
| --- | --- | --- | --- | --- | --- |
| (1) | 1251 | 872.5 | 505.96 | 3.66 | 0.5 |
| (2) | 1163 | 1052 | 102.47 | 4.78 | 0.61 |

**Table 5: Results of infinite-message experiments on the Ibex Instruction Decoder module: all configurations use Prompting template with one-line task introduction and Successful Difficult Responses. (1)-(4) use Stable-restart Keeping best-messages; (1)(2) use Coverage Rate-based Tolerance for Ibex decoder bins, (3)(4) use Normal Tolerance; (1)(3) use Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins, (2)(4) use Coverpoint Type-based Sampling for Ibex decoder bins. (5) uses Keeping best-messages, Coverage Rate-based Tolerance for Ibex decoder bins, and Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins.**
| Config | Max cov (/2107) | Avg cov | Stdev cov | Avg cov/msg | Stdev cov/msg |
| --- | --- | --- | --- | --- | --- |
| (1) | 1807 | 1568 | 338 | 2.55 | 0.62 |
| (2) | 1513 | 1302.5 | 297.69 | 2.83 | 0.41 |
| (3) | 1861 | 1684 | 250.32 | 3.85 | 0.07 |
| (4) | 1642 | 1322.33 | 472.17 | 2.71 | 0.56 |
| (5) | 721 | 369.5 | 497.1 | 0.85 | 0.18 |

### F.3 On Ibex Instruction Decoder

First, we run limited-message experiments (at most 300 responses per trial) with several configurations of prompt templates (see [Section 3.2](#S3.SS2)) and improvement measures (see [Section 3.3](#S3.SS3)) on the Ibex Instruction Decoder module.

The results are displayed in [Table 4](#A6.T4). As we can see, on this module:

- •
Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins is better than Coverpoint Type-based Sampling for Ibex decoder bins in terms of average coverage, coverage stability, and average coverage-per-message.
- •
Successful Difficult Responses is generally better than Successful Responses and Mixed Recent and Successful Responses.
- •
While Low Tolerance has poorer coverage than Normal Tolerance, its coverage-per-message is significantly better, which means it covers easier bins faster; the Coverage Rate-based Tolerance for Ibex decoder bins that combines Low Tolerance and Normal Tolerance generally performs the best.
- •
The combination of High Tolerance and Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins makes no significant improvement than Normal Tolerance and Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins, whereas its standard deviations are both larger.
- •
Prompt template with one-line task introduction is generally better than Original prompt template.

Figure: Figure 4: Infinite-message experiments on the Ibex Instruction Decoder module. Each line represents the trial reaching the maximum coverage on a configuration, and dots on it shows dialogue restarting points. All trials use Prompt template with one-line task introduction.
Refer to caption: x5.png

Second, we run infinite-message experiments with five configurations on this module. The results are displayed in [Table 5](#A6.T5) and [Figure 4](#A6.F4). As we can see, on this module:

- •
Mixed Coverpoint Type-based and Pure Random Sampling for Ibex decoder bins performs better than Coverpoint Type-based Sampling for Ibex decoder bins with longer trials.
- •
Normal Tolerance and Coverage Rate-based Tolerance performs similarly.
- •
Keeping best-messages performs much worse than Stable-restart Keeping best-messages on this module.

Based on these conclusions, we run fixed-budget experiments on two most prominent combinations of strategies, as in [Section 4.2](#S4.SS2).

**Table 6: Results of LLM comparison experiments on the Ibex Instruction Decoder module. Both stimulus generation agents use Prompting template with one-line task introduction, Coverpoint Type-based Sampling for Ibex decoder bins, Successful Responses, Normal Tolerance, and Stable-restart Keeping best-messages.**
| Config | Max cov (/2107) | Avg cov | Stdev cov | Avg cov/msg | Stdev cov/msg |
| --- | --- | --- | --- | --- | --- |
| Llama 2 7B | 17 | 15.67 | 2.31 | 0.43 | 0.08 |
| GPT 3.5 | 1462 | 1203.67 | 323.38 | 4.01 | 1.08 |

### F.4 LLM comparison

We compare the two choices of LLM by running limited-message experiments on the Ibex Instruction Decoder module. The results are displayed in [Table 6](#A6.T6). As we can see, GPT 3.5 is significantly better than Llama 2 7B. In other experiments with Llama 2 7B, we also observe frequent gibbering response that has never happened with GPT 3.5.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1) Given the fixed thresholds in dialog restarting, how does LLM4DV adapt to variability in coverage rates across designs with differing complexity? Could dynamic thresholds improve restart efficiency?
2)	For designs with detailed CPU architectures, like those with complex instruction sets (e.g., RISC-V), how does LLM4DV ensure that test stimuli generation adequately captures inter-instruction dependencies to achieve comprehensive coverage?
3)	Does LLM4DV include mechanisms to help identify or diagnose the root causes of missed coverage bins? How could such a feature enhance the utility of the framework for iterative debugging workflows?
soundness: 2
strengths: The paper tries to address a relevant problem and is well written, and well organized.
summary: The paper presents LLM4DV, an open-source benchmarking framework designed to automate hardware test stimuli generation using large language models (LLMs) for hardware design verification (DV). LLM4DV leverages six LLMs and introduces six prompt engineering improvements to tackle the labor-intensive DV process, aiming to enhance coverage by generating stimuli for various hardware devices under test (DUTs). The framework evaluates these LLMs across eight hardware designs, comparing their coverage rates and effectiveness against conventional constrained-random testing (CRT) and formal verification methods. While LLM4DV demonstrates notable performance improvements on simpler DUTs, its efficacy declines with complex hardware architectures due to limitations like state-space explosion, token window constraints, and a high dependency on prompt engineering. The study concludes that LLMs show promise for automating aspects of DV but require further adaptation to handle more complex verification scenarios effectively.
weaknesses: 1) The LLM4DV framework relies on missed-bin sampling to manage the large number of uncovered coverage bins, but this approach introduces randomness that may hinder consistent exploration of harder-to-hit bins. This sampling method may lead to poor prioritization, with the LLM potentially focusing on simpler bins at the expense of complex corner cases that are critical for comprehensive hardware verification.
2)	The dialog restarting mechanism, designed to mitigate the issue of LLMs "getting stuck" by clearing dialog history, lacks adaptability. The fixed thresholds (like hitting fewer than three bins within 40 messages) are rigid and may not accommodate variability across hardware designs with different complexity levels. This static approach may cause premature restarts, losing valuable accumulated context and decreasing efficiency in achieving coverage.
3)	Including DUT source code in the prompt often exceeds the token limit for many LLMs, especially in the case of designs with complex logic or multiple functional modules. This limitation results in truncating prompts or excluding critical code portions, reducing the model’s ability to effectively map test stimuli to specific coverage bins.
4)	The framework’s effectiveness heavily depends on manually engineered prompts and specific configurations, such as coverage-feedback templates and best-iterative-message sampling. This approach introduces overhead and suggests a lack of automated prompt optimization, which could otherwise make LLM4DV more adaptable and less reliant on human intervention.
5)	For designs with large state spaces, such as components with storage elements or multiple queues (e.g., AMPLE Prefetcher Weight Bank), the LLM4DV framework struggles to maintain efficient coverage. This is because the LLM’s state exploration strategy does not scale well with increasing design complexity, leading to a "state-space explosion" where the LLM cannot feasibly cover all unique states or transitions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See Weakness.
soundness: 3
strengths: LLM4DV reduces the need for manual testing inputs, saving time and resources. It also Includes multiple evaluation metrics that allow comprehensive assessment beyond just coverage, such as message count efficiency.
summary: The article introduces LLM4DV, an open-source benchmark framework leveraging Large Language Models to automate hardware test stimuli generation, specifically for hardware Design Verification. Hardware DV is crucial for ensuring that hardware designs function correctly, typically involving extensive human input to design and execute test stimuli that cover a range of scenarios. LLM4DV streamlines this process by generating stimuli with LLMs, reducing human effort and improving test coverage.
weaknesses: 1. I have observed several works with approaches similar to this submission in applying prompting methods to hardware verification and testing. Notably:

*AssertLLM: Generating and Evaluating Hardware Verification Assertions from Design Specifications via Multi-LLMs*

*Towards LLM-Powered Verilog RTL Assistant: Self-Verification and Self-Correction*

*LLM-assisted Generation of Hardware Assertions*

I suggest the author to compare LLM4DV with these works.

2. Novelty Concerns and Venue Suitability:

While the LLM4DV framework in this submission introduces noteworthy adaptations of prompting methods tailored to Hardware Testing, many of these underlying techniques are not novel. Prompting-based methods have been extensively applied in other domains since at least 2022. Specifically:

The Best-iterative-message Sampling described in Section 4.4 resembles a method for summarizing conversational memory, as discussed in Generative Agents: Interactive Simulacra of Human Behavior.

The Few-shot Prompting approach in Section 5 has parallels with work on few-shot learning found in PPT: Pre-trained Prompt Tuning for Few-shot Learning.

3.  Reproducibility and Open-Source Framework Availability:

The authors claim to provide an open-source benchmarking framework, but I was unable to find any code, supplementary materials, or anonymous links for verification. Given my experience with works claiming to be open-source yet only offering minimal or incomplete repositories, I recommend the authors to provide anonymous links during the rebuttal stage would allow me to verify the authenticity and reproducibility of this contribution, which is crucial for enabling follow-up work by future researchers.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: This has already been addressed in the Weaknesses section.
soundness: 3
strengths: This research work is highly interesting, offering a promising solution to a critical challenge in hardware design and verification. The manuscript is well-organized and well-written. The LLM4DV framework is thoroughly described, detailing each of its component modules. The evaluation setup is clearly explained, and the results and analysis effectively support the authors' claims. The appendix section provides extensive scientific details that clarify the motivation for benchmarking and the working principles of the framework.
summary: This work introduces LLM4DV, a framework that uses prompted LLMs to generate test stimuli for hardware Design Verification (DV). The workflow supports a plug-and-play setup, allowing users to experiment with various LLMs, hardware designs, and test coverage plans. The authors develop six prompt enhancements for effective automated DV, establishing strong baselines within LLM4DV as a testbed for studying LLM agentic behavior.

Key contributions include:

A. The creation of three DUT (Design-Under-Test) modules: (1) a Primitive Data Prefetcher Core, (2) an Ibex Instruction Decoder, and (3) an Ibex CPU, and the integration of 5 additional open-source designs, providing a diverse range of DUTs for users.

B. Evaluation of LLM4DV on 8 DUT modules with custom metrics, demonstrating coverage rates from 71.76% to 100% in realistic scenarios with optimized prompts.

Additionally, the authors open-sourced LLM4DV, encouraging both the machine learning and hardware design communities to experiment with it.
weaknesses: I have one minor suggestion: Algorithm A.2 should be included in the main text rather than in the appendix, as it would aid in understanding the background more effectively.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you clarify what is the goal of the paper? Is the main goal to say that LLMs do better than non-LLM state of the art in the task of DV? Or is the goal to evaluate LLMs on this task (and propose a framework to do so)?

Can you please add a more detailed discussion of existing non-LLM based methods that address the problem of DV? In particular, you can discuss aspects such as their test coverage rates and scalability and compare them to LLM-based methods.
soundness: 3
strengths: -	The problem that the authors address is important and challenging.
-	The authors provide an interesting evaluation of SOTA LLMs on the task of hardware design verification.
-	The paper is clear and easy to read.
summary: The paper proposes LLM4DV, a framework that uses LLMs to generate test stimuli for hardware design verification (DV). The authors then use this framework to evaluate state-of-the-art LLMs on this task. The authors also propose LLM prompting strategies to maximize success in this task. They show that these LLMs achieve test coverage rates ranging from 71.76% to 100%.
weaknesses: While the proposed work is important, the paper has a limited discussion of related work and a limited comparison with existing work (that does not use LLMs). The problem of hardware design verification is well-studied. Examples of papers worth discussing:

-	HIVE: Scalable Hardware-Firmware Co-Verification using Scenario-based Decomposition and Automated Hint Extraction.

-	Survey of Machine Learning for Software-assisted Hardware Design Verification: Past, Present, and Prospect.

As the paper is written now, it is not clear what is its main contribution. Is the main goal of the paper to say that LLMs do better than non-LLM state of the art in the task of DV? Or is the goal to evaluate LLMs on this task (and propose a framework to do so). The authors should clarify this. The paper shows indeed that LLMs can do the task of DV when evaluated using the benchmark proposed by the authors, but it is not written to convince the reader that using LLMs to address this task is better than using other state-of-the-art methods. If your goal is to propose LLMs as the new state-of-the-art in addressing the problem of hardware design verification, you should have a detailed analysis of existing work and should compare with state-of-the-art methods. You also need to use the same benchmarks used by state-of-the-art methods (instead of creating a new benchmark).

The current paper has a high value. It clearly shows that LLMs can perform DV with high coverage rates, but as it is written now it is not convincing enough to show that LLMs are better than existing work in solving the problem.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Please compare with recent works in using LLMs for hardware verification.
soundness: 2
strengths: The authors effectively design the prompting strategies to guide LLMs in generating tests for hardware designs. Additionally, the framework has been tested on various LLM models. The experiments are performed on some industry-level designs such as CPUs, highlighting the impressive capabilities of current LLM technology.
summary: This paper proposes a framework using LLM to generate tests for hardware designs. The framework employs a coverage-directed strategy and incorporates well-designed prompting techniques. The authors evaluated the framework on various hardware designs including CPUs. The experimental results show that the framework achieves performance that is comparable to, or better than, traditional approaches such as CRT and formal verification.
weaknesses: As a submission to the ICLR conference, the paper lacks novelty in its application of AI concepts, appearing more as an engineering effort. Additionally, there are several similar works on LLMs for hardware test generation that the authors do not mention in this paper, including:

Wang X, Wan G W, Wong S Z, et al. "ChatCPU: An Agile CPU Design & Verification Platform with LLM" (61st Design Automation Conference, 2024).
Qiu R, Zhang G L, Drechsler R, et al. "AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Design" (2024 International Symposium on Machine Learning for CAD, 2024).
Ma R, Yang Y, Liu Z, et al. "VerilogReader: LLM-Aided Hardware Test Generation" (2024 LLM Aided Design Workshop, 2024).
M. Hassan, S. Ahmadi-Pour, K. Qayyum, C. K. Jha, and R. Drechsler. "LLM-Guided Formal Verification Coupled with Mutation Testing" (2024 Design, Automation & Test in Europe Conference & Exhibition, 2024).

While this submission provides valuable insights into prompting strategies for hardware verification, its contributions appear limited. I recommend that the authors discuss and compare LLM4DV with these related studies and consider submitting the paper to a conference more focused on hardware domains.