CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

## Abstract

Abstract Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making “minor” mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.

## 1 Introduction

Large Language Models (LLMs) have achieved significant success across various natural language processing tasks and have extended their capabilities to code generation, leading to the development of specialized models targeting code generation. The effectiveness of these models is largely influenced by the size and quality of their training datasets, as highlighted by scaling laws (Achiam et al., ; Zhang et al., ). Prominent code LLMs have set new benchmarks records by utilizing extensive, synthetically generated datasets through methods like Self-Instruct (Wang et al., ; Chaudhary, ), Evol-Instruct (Xu et al., ), and OSS-Instruct (Wei et al., ). These synthetic data generation techniques allow code LLMs to generate a wide range of complex code examples, enhancing their training and performance in real-world coding scenarios.

While most code LLMs concentrate on software programming languages, there is increasing interest in developing models for hardware description languages (HDLs), which are essential for chip design and hardware verification. Despite efforts to collect and synthesize more diverse Verilog code to enhance specialized code LLMs (Liu et al., ; Pei et al., ; Cui et al., ; Zhao et al., ), HDLs still face challenges akin to those encountered in low-resource languages (Cassano et al., ). These challenges are mainly due to the limited availability of high-quality instruction-following data and the constrained capability of existing LLMs to generate RTL code, which affects the models’ performance and their ability to generalize across programming languages.

Developing high-quality synthetic Verilog code for training code large language models (LLMs) faces significant challenges due to two primary factors. Firstly, Verilog is considered a low-resource language (Cassano et al., ), meaning there is a scarcity of available training data compared to high-resource software programming languages like Python. This limited data availability restricts the models’ ability to learn diverse and complex coding patterns effectively. Secondly, verifying the correctness of hardware description language (HDL) code, such as Verilog, is inherently more complex than verifying software code. While software code correctness can often be assessed using random test cases and automated unit tests (Chen et al., ), hardware code requires comprehensive testbenches and rigorous verification planning and methodologies. This additional complexity makes it challenging to ensure that synthetic Verilog code is functionally accurate (Bhandari et al., ; Qiu et al., ), posing a barrier to improving model performance.

In this paper, we start with a thorough analysis of fine-tuned large language models (LLMs) applied to Verilog code, using synthetic data techniques from previous works. Our analysis reveals two key issues: (1) models have difficulty handling non-textual elements in problem statements, indicating challenges in interpreting complex or unconventional inputs; and (2) there is notable variability in the models’ pass rates across different benchmark problems and training checkpoints, exposing inconsistencies in learning outcomes, often due to the models making “minor” programming mistakes.

Given the limitations identified in our analysis of relying solely on LLMs for generating synthetic data, we shift our focus to improving data curation to address these issues. Current LLMs frequently struggle with interpreting and processing non-textual representations and are insufficient in generating effective testbenches for evaluating solution quality. Therefore, instead of depending exclusively on LLMs to address data quality concerns, we develop targeted fine-tuning data to better mitigate these problems. Experimental results demonstrate that our models achieve state-of-the-art (SOTA) results on VerilogEval (Liu et al., ) and RTLLM v1.1 (Lu et al., ) benchmarks, outperforming prior works by large margins on problems with human-level description. The major contributions of this paper are as follows:

- •
We perform a thorough analysis of fine-tuned LLMs on Verilog code using previously established synthetic data generation methods, uncovering challenges with non-textual elements and notable variability in performance across benchmark problems during training.
- •
We create correct-by-construction data to ensure solution correctness, incorporating Karnaugh Maps, state-transition diagrams, and waveforms, which significantly enhance the model’s ability to handle non-textual representations.
- •
We develop an automated framework that utilizes LLMs to generate error reports from benchmark problems at various checkpoints, which are then injected into open-source code to create a fine-tuning dataset targeted at correcting the model’s specific “minor” mistakes.
- •
We rigorously evaluate the latest foundational and frontier code models. We note that recent advanced models like GPT-4o already reached competitive performance compared to previous efforts targeting Verilog code generation.
- •
Experimental results demonstrate that models fine-tuned with our data achieve state-of-the-art performance on Verilog coding. Specifically, our fine-tuned model based on Starcoder2-15B (Lozhkov et al., ) outperforms prior SOTA results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.

## 2 Examining fine-tuned LLMs Using Synthetic Generated Data on Verilog Coding

In this section, we start with a thorough analysis of fine-tuned large language models (LLMs) applied to Verilog code. We adapt previous approaches for generating synthetic data for general coding to focus on Verilog code. For our pilot study, we only present results based on fine-tuning StarCoder2-15B (Lozhkov et al., ). Details on experimental settings are the same as in Section 4. We assess model performance in Verilog code completion and identify two main issues. First, the models demonstrate notably poor performance when dealing with non-textual elements in the problem statements. Second, the variability in the models’ pass rates across different benchmark problems and training checkpoints suggests inconsistencies in learning outcomes and model variability.

### 2.1 Synthetic Data Generation for Verilog Coding

We build on previous methods for synthetic data generation by applying Self-Instruct (Wang et al., ) and OSS-Instruct (Wei et al., ) with custom prompt templates tailored for Verilog coding. To enhance data coverage and diversity, we supplement these techniques with additional context from Wikipedia and textbooks. We also prompt models to generate problem descriptions to include non-textual representations.

**Table 1: Data quantity SDG.**
| Method | Quantity |
| --- | --- |
| Self-Instruct | 24.7k |
| OSS-Instruct | 28.4k |
| Docu-Instruct | 12.0k |
| Non-textual | 15.0k |
| SDG Total | 80.1k |

We use nemotron-4-340b-instruct (Nvidia et al., ) selected for its open license that allows commercial use.
Our process includes deduplication and a decontamination procedure akin to that outlined by Li et al. (). Additionally, we conduct syntax checks to eliminate coding problems containing docstrings or solutions from Verilog benchmarks. To ensure further data quality, we discard code solutions that fail these syntax checks and apply self-verification (Weng et al., ) to remove entries where the LLM identifies errors in the solution. Table 1 shows the quantity of our synthetic data generation (denoted as SDG) after deduplication and filtering, yielding a total of 80.1k fine-tuning examples.

##### Self-Instruct

We follow the approach outlined in Wang et al. () to generate synthetic Verilog coding problems. Initially, we randomly generate from the LLM and curate 50 questions that request Verilog coding problems without any in-context examples. From these, we then randomly choose 1 to 5 seed questions to use as in-context examples.

##### OSS-Instruct

We begin by processing pretraining code data to extract our seed code from The Stack v2 (Lozhkov et al., ), focusing on Verilog and SystemVerilog. Following the approach in Liu et al. (), we post-process this data by selecting self-contained Verilog code that passes syntax checks using Pyverilog (Takamaeda-Yamazaki, ). With the refined seed code data, we then prompt large language models (LLMs) to use this code as inspiration for generating Verilog coding problems similar to Wei et al. ().

##### Docu-Instruct

Drawing inspiration from Nvidia et al. () and Sudalairaj et al. (), we utilize document sources from Wikipedia and textbooks for instruction generation. We begin by filtering Wikipedia entries, prompting the LLM to classify whether the content pertains to hardware design or Verilog coding concepts. Additionally, we manually selected approximately relevant 100 textbooks. These textbooks are then segmented into chunks of paragraphs or sentences, ensuring each chunk contains fewer than 2k tokens.

##### Non-textual Representations

VerilogEval-Human (Liu et al., ) includes benchmark problems involving non-textual representations. For example, Boolean logic tables and Karnaugh maps are presented in tabular formats, state-transition diagrams for finite state machines are depicted as edge lists and sequential waveforms are described in tables with signal outputs recorded at various time steps. To incorporate such representations, we encouraged LLMs to generate problems from open-source code, with instructions to utilize these tabular data structures.

### 2.2 Challenges with Non-Textual Representations

**Table 2: pass@1 results on VerilogEval sampled with temperature of 0.8.**
| Model | Machine | Human | NonText |
| --- | --- | --- | --- |
| GPT-4o | 63.7 | 55.4 | 27.0 |
| Starcoder2 | 57.7 | 29.1 | 10.3 |
| Starcoder2-SDG | 73.7 | 47.4 | 22.2 |

We observe that models underperform on benchmark problems involving non-textual input formats, such as Karnaugh Maps, state-transition diagrams, and waveforms. Table 2 shows the pass@1 results for the VerilogEval (Liu et al., ). Additionally, we have identified a subset of 45 questions within VerilogEval-Human that include non-textual representations, termed VerilogEval-NonText. It appears that models like GPT-4o and Starcoder2 struggle with these non-textual formats, likely due to insufficient representation of such data during both pretraining and fine-tuning. Despite our efforts to generate such questions during synthetic data creation, our fine-tuned models still lag in these areas.
This outcome is not entirely surprising, given that the LLMs used were also ineffective at generating problems with these representations, complicating the validation of fine-tuning data.
These results suggest that merely including non-textual data is insufficient; ensuring the quality and correctness of the data, particularly that the code solutions accurately align with these representations, is crucial.

### 2.3 Variability on Pass Rates During Training

Figure: Figure 1: Agreement by pass rate between two subsequent checkpoints during training on SDG data for problems in VerilogEval-Human.
Refer to caption: extracted/5865737/Figs/starcoder_scatter.png

During our training, we observed significant variability in the model’s pass rate on specific benchmark problems across different checkpoints. We note such variance is different from training instability (Wortsman et al., ) as we observe a stable decrease in the training loss. This variability persists even in the later stages of training, despite using a low learning rate. We illustrate this variability in Figure 1. The scatter plot tracks the pass rate for each problem in VerilogEval-Human, with each point representing the pass rate for the same problem across two checkpoints. The size of each point indicates the number of problems with the same pass rates for the two model checkpoints. We further categorize the region into areas where the checkpoints agree on problem difficulty and areas where they do not.

Alarmingly, we find that nearly 15% of the problems show significant discrepancies between these two checkpoints, with an equal number of problems demonstrating improvement and degradation. Our detailed analysis of the sampled code completions for such problems when pass rate degrades suggests that the model is generally on the right track but makes “minor” errors that are small, detailed, and seemingly trivial. While it is possible that LLMs experience catastrophic forgetting during fine-tuning (Luo et al., ), we do not anticipate this being a major factor due to the low learning rate and the small number of gradient updates (64 steps with 16k data samples). Instead, we believe the primary issue is our inability to ensure the quality of our data, particularly in verifying whether the sampled code solutions correctly solve the code problems.

## 3 Improving Verilog Coding with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

Based on our detailed analysis of the limitations of relying solely on LLMs for generating synthetic data, we focus our data curation efforts to address these shortcomings. Our goal is to enhance data quality and ensure the correctness of solutions for the generated problems. We have found that current LLMs often lack the capability to understand and process non-textual representations effectively and are unable to generate satisfactory testbenches for assessing solution quality. Consequently, rather than depending entirely on LLMs to resolve data quality issues, we instead create targeted fine-tuning data to mitigate these problems.

Figure: Figure 2: Generating non-textual problems and correct-by-construction solutions.
Refer to caption: x1.png

### 3.1 Ensuring Quality Through Correct-by-Construction

**Table 3: Data quantity CC.**
| Method | Quantity |
| --- | --- |
| KMap | 12.5k |
| FSM | 8.0k |
| Waveforms | 8.0k |
| CC Total | 28.5k |

We generate Verilog code problems and solutions that are correct-by-construction. Our focus is on creating problems and solutions for non-textual representations. The overview of the process for generating non-textual problems and correct-by-construction solutions is illustrated in Figure 2.
Table 3 shows the quantity of our correct-by-construction generation data (referred to as CC). To prevent data contamination, we exclude entries that duplicate the data representations of benchmark problems.

##### Karnaugh Maps and Truth Tables (KMap)

We start by sampling random configurations, which include selecting the number of variables and their names. After determining the number of variables, we randomly choose valid minterms and don’t-cares. For $n$ variables, there are $2^{n}$ possible states, and each state can be assigned one of three values (0, 1, or x), leading to $3^{2^{n}}$ possible combinations of minterms and don’t-cares. From these minterms, we derive the sum-of-products (SOP) form to represent the Boolean logic. We then create Truth Tables and Karnaugh Maps based on the chosen minterms and don’t-cares. In the KMap, Gray encoding is used as default for the row and column sequences to ensure that only a single bit changes between adjacent cells. Additionally, we apply modifications by transposing the map and randomly swapping adjacent rows or columns. We randomly sample from $n=\{3,4\}$ variables.

##### State Transition Graphs and Tables (FSM)

We construct problems for finite-state machines (FSMs) with state-transition representations with a similar approach to KMaps. We begin by sampling random configurations, including the number of states (e.g., 4, 6, or 10) and the bit width of the input (e.g., 1 or 2). We then create the transition graph, ensuring that it is both meaningful and legally defined. We generate state-transition graphs for both Moore and Mealy state machines. From these graphs, we produce edge-list and transition table representations. Finally, we construct the Verilog code to implement the logic for state transitions and output assignments.

Figure: Algorithm 1 Generate transition graph for Moore FSM.

Algorithm 1 outlines the process for generating a Moore FSM with random transitions. State reachability is ensured by first constructing a tree. Legality for state transition is ensured by ensuring each node has an out-degree of $2^{w}$ with the input bit width of $w$. The result is an FSM where the transitions between states are randomly assigned but conform to the specified input bit width. The algorithm can be easily modified for a Mealy FSM by assigning the output to the edges rather than nodes.

Figure: Figure 3: State transition logic.
Refer to caption: x2.png

Figure 3 illustrates our approach for generating state transition logic in Verilog from a state-transition graph. Our method predominantly employs an out-edge focused strategy for state transitions. Additionally, we incorporate in-edge focused transition logic to address specific challenges encountered in benchmark problems. These benchmarks often involve states represented using one-hot encoding and require rigorous testing of non-default states.

##### Waveforms

We utilize correct-by-construction code solutions for both KMaps and FSMs. Because these codes are generated using similar templates, designing corresponding testbenches is straightforward. We simulate the generated code to produce waveform Value Change Dump (VCD) files. These VCD files are then parsed and converted into waveform representations. Our approach covers KMaps as combinational circuits and FSMs as sequential circuit waveforms.

### 3.2 Mitigating “Minor” Errors with targeted code repair

Figure: Figure 4: An overview of our approach in generating targeted code repair data.
Refer to caption: x3.png

Our analysis revealed that the models were generally on the right track to correct solutions but were making minor errors—small, detailed, and seemingly trivial. Unlike complex, unsolvable problems, these minor errors could be easily corrected by language models. This insight led us to develop a new strategy centered on targeted error code repair. Our approach includes creating detailed error reports on benchmark problems, re-creating these errors on correct open-source code, and conducting rigorous validation to ensure quality. We use nemotron-4-340b-instruct as the LLM to construct our targeted Repair data. We generated 847 error reports across the three benchmarks and produced 2,736 data samples. After filtering, this resulted in a final set of 1,406 targeted code repair data points.

##### Error Report Construction

To systematically address the issue, we first created a comprehensive Error Report for benchmark problems using LLMs, targeting those with significant pass rate fluctuations across training checkpoints for models on SDG data. We prompt the LLM to examine the nature of the mistakes by comparing correct and erroneous code completions for each problem, categorizing the errors into functional types (such as logic, multiplexing, and arithmetic operations). Additionally, we prompt LLMs to identify mistakes in common error patterns, for example, frequent mistakes in arithmetic shifts, bit manipulation, and unintended latch creation. This detailed report not only categorizes the errors but also highlights areas where the model consistently underperforms.

##### Targeted Code Repair Dataset

Building on the error report, we further develop a targeted code repair dataset to address these common errors. This dataset is constructed using two main sources: the errors identified in the Error Report and correct code snippets gathered from open-source repositories. We introduced the identified errors into correct code snippets to create repair problems, which include a problem description, erroneous code implementation, and hints about the nature of the error and how to fix it. This targeted strategy enables the model to learn how to avoid common errors and generate improved code completions, thereby enhancing model accuracy.

##### Quality Assurance with LLM Validation

To ensure the reliability of the error report and the code repair dataset, we implemented a two-phase validation process with LLMs. In the first phase, we conducted a self-consistency check of the Error Report by having the language model attempt to fix the error code based on the report’s hints. This step verifies the accuracy of the report by confirming that the model can resolve the errors using the provided guidance. In the second phase, during the generation of the code repair dataset, we apply self-verification, including deduplication, syntax filtering, and benchmark decontamination. These measures ensure the dataset’s quality and uniqueness, preventing overlap with evaluation benchmarks.

## 4 Experiment

### 4.1 Implementation Details

##### Training Data

Our fine-tuning training data is comprised of 80.1k LLM synthetic generated data using various prompting methods as described in Section 2.1, 28.5k data samples generated correct-by-construction aimed at non-textual representations detailed in Section 3.1, and 1.4k carefully filtered data for targeted code repair as outlined in Section 3.2. We refer to each data set as SDG, CC, and Repair, respectively.

##### Pretrained Models

Following prior work, we use CodeLlama-7b-Instruct (Roziere et al., ) and Deepseek-Coder-6.7b-Instruct (Guo et al., ) as the base model, formatting our data according to their default chat prompt templates. Additionally, we explore the Starcoder2-15B (Lozhkov et al., ) model in our experiments.

##### Model Training

Training is conducted with 32 NVIDIA A100-80GB GPUs through the Distributed Data Parallel (DDP) module from PyTorch. We set the learning rate at 5e-5 for CodeLlama and DeepSeek-Coder, and 1e-5 for Starcoder2. We use Adam (Kingma & Ba, ) as our optimizer with full parameter updates and truncate sequence lengths longer than 4096 tokens. We used a batch size of 256 samples. We fine-tune models for 1 epoch using a standard cross entropy loss on the response tokens (while masking loss on prompt tokens).

##### Model Inference

We use vLLM (Kwon et al., ) where the inference engine is set up with bf16 dtype, tensor parallel size of 8, and a maximum token limit of 4096. We sample each problem 20 times. We report the best results from two different temperatures 0.2 and 0.8, as consistent with prior work (Liu et al., ; Zhao et al., ).

**Table 4: We compare our models with various baseline models on VerilogEval (Liu et al., ). We update the results from Zhao et al. () with the latest foundational and frontier code models. The best results for each model type are highlighted in bold. We use red to denote the best model overall and blue to indicate the best results among models other than ours.**
| Type | Model | Size | VerilogEval (Liu et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Machine (%) | Human (%) |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |
| Foundational<br>Models | Llama-3.1 | 8B | 48.7 | 67.3 | 74.1 | 26.9 | 37.8 | 44.2 |
| Llama-3.1 | 405B | 67.3 | 75.1 | 76.9 | 53.8 | 61.0 | 62.8 |  |
| Nemotron-4 | 340B | 53.0 | 60.3 | 62.2 | 43.1 | 48.3 | 50.0 |  |
| GPT-3.5-turbo | - | 58.0 | 74.0 | 77.6 | 31.2 | 44.1 | 47.4 |  |
| GPT-4o | - | 65.9 | 71.4 | 72.7 | 57.1 | 63.9 | 66.7 |  |
| Code<br>Models | CodeLlama | 7B | 43.1 | 47.1 | 47.7 | 18.2 | 22.7 | 24.3 |
| CodeQwen | 7B | 46.5 | 54.9 | 56.4 | 22.5 | 26.1 | 28.0 |  |
| Starcoder2 | 15B | 68.7 | 82.3 | 88.5 | 37.7 | 50.6 | 57.2 |  |
| DeepSeek-Coder | 6.7B | 52.2 | 55.4 | 56.8 | 30.2 | 33.9 | 34.9 |  |
| DeepSeek-Coder-V2 | 16B | 67.4 | 78.3 | 81.8 | 46.9 | 55.9 | 58.9 |  |
| DeepSeek-Coder-V2 | 236B | 68.2 | 74.1 | 76.2 | 56.4 | 62.2 | 66.0 |  |
| RTLCoder<br>(Liu et al., ) | Mistral | 7B | 62.5 | 72.2 | 76.6 | 36.7 | 45.5 | 49.2 |
| DeepSeek-Coder | 7B | 61.2 | 76.5 | 81.8 | 41.6 | 50.1 | 53.4 |  |
| BetterV<br>(Pei et al., ) | CodeLlama | 7B | 64.2 | 75.4 | 79.1 | 40.9 | 50.0 | 53.3 |
| DeepSeek-Coder | 6.7B | 67.8 | 79.1 | 84.0 | 45.9 | 53.3 | 57.6 |  |
| CodeQwen | 7B | 68.1 | 79.4 | 84.5 | 46.1 | 53.7 | 58.2 |  |
| CodeV<br>(Zhao et al., ) | CodeLlama | 7B | 78.1 | 86.0 | 88.5 | 45.2 | 59.5 | 63.8 |
| DeepSeek-Coder | 6.7B | 77.9 | 88.6 | 90.7 | 52.7 | 62.5 | 67.3 |  |
| CodeQwen | 7B | 77.6 | 88.2 | 90.7 | 53.2 | 65.1 | 68.5 |  |
| OriGen (Cui et al., ) | DeepSeek-Coder | 6.7B | 74.1 | 82.4 | 85.7 | 54.4 | 60.1 | 64.2 |
| Ours<br>SDG-CC-Repair | CodeLlama | 7B | 78.1 | 85.5 | 87.8 | 63.1 | 67.8 | 69.7 |
| DeepSeek-Coder | 6.7B | 77.8 | 85.5 | 88.1 | 65.4 | 70.0 | 72.1 |  |
| Starcoder2 | 15B | 81.9 | 86.9 | 88.1 | 68.0 | 72.4 | 74.6 |  |

### 4.2 Evaluation Metric and Benchmark

Evaluation Metric Following prior work (Chen et al., ; Liu et al., ), for each experiment we use the unbiased pass@k metric to measure the Verilog generation accuracy. The pass@k metric estimates the proportion of problems that can be solved at least once in k attempts:

$$ $pass@k:=\mathbb{E}_{\text{Problems}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k} }\right],$ (1) $$

where $n\geq k$ represents the total number of trials for each problem, and $c$ represents the number of trials that pass the functional check.

VerilogEval (Liu et al., ) contains two subsets of problems, where VerilogEval-Human contains manually converted problem descriptions from the original HDLBits website, and VerilogEval-Machine with GPT-3.5 generated problem descriptions.

RTLLM (Lu et al., ) is an open-source benchmark designed for generating Register Transfer Level (RTL) code from natural language instructions. It evaluates models on syntax correctness, functional correctness, and design quality, offering a thorough analysis of model outputs.

### 4.3 Results

**Table 5: Evaluations on RTLLM v1.1 (Lu et al., ) using unbiased pass@k metrics. The best results for each model type are highlighted in bold. We use red to denote the best model overall and blue to indicate the best results among models other than ours. We re-evaluate all models (see Appendix A for details).**
| Type | Model | Size | RTLLM v1.1 (Lu et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Syntax (%) | Func. (%) |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |
| Foundational<br>Models | Llama-3.1 | 8B | 40.7 | 60.6 | 65.5 | 19.3 | 34.7 | 37.9 |
| Llama-3.1 | 405B | 56.5 | 64.4 | 72.4 | 38.9 | 45.8 | 51.7 |  |
| Nemotron-4 | 340B | 41.7 | 47.2 | 48.3 | 18.9 | 20.7 | 20.7 |  |
| GPT-3.5-turbo | - | 50.3 | 61.2 | 65.5 | 28.3 | 36.9 | 41.4 |  |
| GPT-4o | - | 50.3 | 59.9 | 62.1 | 33.8 | 44.4 | 48.3 |  |
| Code<br>Models | CodeLlama | 7B | 46.6 | 62.6 | 68.9 | 17.9 | 29.9 | 34.5 |
| CodeQwen | 7B | 45.8 | 65.8 | 72.4 | 24.1 | 34.0 | 37.9 |  |
| Starcoder2 | 15B | 38.3 | 81.0 | 94.7 | 15.5 | 37.6 | 45.7 |  |
| DeepSeek-Coder | 6.7B | 51.4 | 64.4 | 68.9 | 23.1 | 29.3 | 34.5 |  |
| DeepSeek-Coder-V2 | 16B | 51.4 | 57.8 | 58.6 | 33.1 | 37.1 | 37.9 |  |
| DeepSeek-Coder-V2 | 236B | 63.4 | 78.1 | 79.3 | 34.5 | 50.2 | 55.1 |  |
| RTLCoder<br>(Liu et al., ) | Mistral | 7B | 64.6 | 73.7 | 78.3 | 24.5 | 37.3 | 42.3 |
| DeepSeek-Coder | 6.7B | 73.4 | 83.9 | 86.2 | 35.8 | 40.3 | 43.1 |  |
| CodeV<br>(Zhao et al., ) | CodeLlama | 7B | 79.0 | 89.2 | 89.9 | 39.4 | 50.3 | 53.1 |
| DeepSeek-Coder | 6.7B | 78.3 | 87.4 | 89.1 | 42.4 | 51.5 | 53.2 |  |
| CodeQwen | 7B | 78.8 | 89.5 | 92.4 | 36.6 | 53.3 | 61.3 |  |
| OriGen (Cui et al., ) | DeepSeek-Coder | 6.7B | - | - | - | - | 65.5 | - |
| Ours<br>SDG-CC-Repair | CodeLlama | 7B | 85.7 | 93.9 | 94.8 | 42.6 | 52.9 | 58.2 |
| DeepSeek-Coder | 6.7B | 84.3 | 92.9 | 95.4 | 53.1 | 58.8 | 62.6 |  |
| Starcoder2 | 15B | 79.8 | 93.9 | 96.2 | 49.0 | 65.8 | 74.5 |  |

##### Main Results

Table 4 and Table 5 compare our models with baselines on VerilogEval and RTLLM.
We mainly source baseline results from Zhao et al. (). For RTLLM we found a large variance with biased pass@5, thus we re-evalaute all models and report unbiased pass@k metric. We further rigorously evaluate the latest foundational and frontier code models, including Llama-3.1 (Dubey et al., ), DeepSeek-Coder-V2 (DeepSeek-AI et al., ), and GPT-4o. Recent foundational and frontier code models already reached competitive performance compared to previous efforts targeting Verilog code generation.

Compared to previous approaches like CodeV (Zhao et al., ), our models achieve comparable performance on VerilogEval-Machine and show significant improvements on benchmarks with human-like descriptions. Machine descriptions often provide detailed, line-by-line coding instructions, whereas human descriptions are high-level, integrating problem-solving skills and a deeper understanding of the hardware module’s functionality. Enhancing the model’s ability to handle human-like descriptions is crucial, as these more accurately reflect how designers interact with the models and set expectations for Verilog generation. Our fine-tuned Starcoder2-15B surpasses previous state-of-the-art results by 3.8%, 10.9%, and 6.6% in pass@1 metrics on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.

**Table 6: Ablation study on training data.**
| Model | VerilogEval | RTLLM v1.1 |  |
| --- | --- | --- | --- |
| Machine | Human | Func |  |
| pass@1 (%) | pass@5 (%) |  |  |
| Starcoder2-15B | 68.7 | 37.7 | 37.6 |
| SDG | 75.2 | 54.7 | 62.1 |
| SDG-CC | 73.9 | 62.0 | 62.8 |
| SDG-CC-Repair | 81.9 | 68.0 | 65.8 |

Table 6 highlights the effectiveness of our generated data fine-tuned on Starcoder2-15B. Our CC data enhances the model’s ability to handle non-textual representations, leading to improved scores on VerilogEval-Human. Our targeted code Repair data boosts performance across all benchmarks, suggesting that the model has learned to generalize from code repair tasks and reduce similar errors during code completion.

##### Improved Variability During Training

Figure: Figure 5: Agreement by pass rate between two subsequent checkpoints during training on SDG-CC-Repair for problems in VerilogEval-Human.
Refer to caption: extracted/5865737/Figs/starcoder_post_scatter.png

Figure 5 displays the pass rates for two consecutive checkpoints of Starcoder2-SDG-CC-Repair on VerilogEval-Human problems, sampled with a temperature of 0.8. Compared to Figure 1, the updated model shows significant improvements by (1) moving previously unsolved problems into the solved category, including those with non-textual representations addressed by our correct-by-construction CC data, and (2) reducing the number of problems with large pass rate discrepancies, particularly where performance had degraded. The targeted repair data has effectively mitigated the model’s tendency to repeat common mistakes found in our Repair dataset, despite the noise inherent in synthetically generated SDG data.

##### Scaling Data for Non-textual Representations

Figure: Figure 6: pass@1 on non-textual problems with total number of CC data with temperature 0.8.
Refer to caption: extracted/5865737/Figs/target_scale.png

Figure 6 illustrates the scaling of correct-by-construction (CC) data and the fine-tuned Starcoder2-15B pass rate on problems involving non-textual representations. We expanded our testing to include strictly in-distribution test set, with each category containing around 50 problems. The results show that the model can quickly learn and comprehend these non-textual representations with as few as 4k training data samples, with the pass rate steadily improving as more data is provided. Additionally, the model demonstrates the ability to generalize to VerilogEval-NonText benchmark problems. While our models achieve near-perfect scores on KMap and FSM problems, they perform less effectively on Waveforms, suggesting that reverse engineering circuits from waveforms pose a greater challenge.

##### Ensuring Quality for Targeted Code Repair

**Table 7: Ablation study on Repair data quality with Starcoder2-15B.**
| Model | VerilogEval | RTLLM v1.1 |  |
| --- | --- | --- | --- |
| Machine | Human | Func |  |
| pass@1 (%) | pass@5 (%) |  |  |
| SDG-CC | 73.9 | 62.0 | 62.8 |
| SDG-CC-Repair | 81.9 | 68.0 | 65.8 |
| w/o self-consistency | 75.3 | 63.3 | 63.7 |
| w/o error report | 76.9 | 59.6 | 59.4 |

The quality control mechanisms integrated into the data generation pipeline are crucial for improving model performance, particularly in correcting minor errors through targeted code repair. To evaluate the impact of these quality controls, we conducted an ablation study in Table 7), where we systematically removed each component of the targeted code repair generation pipeline and assessed the resulting model performance. Specifically, we eliminated the self-consistency checks that validate whether the generated error report effectively guides the LLMs in correcting mistakes. Additionally, we tested the removal of the error report entirely, substituting it with random errors injected into the open-source code by the LLMs. The benchmark results indicate a significant performance drop when these validation processes are excluded. These findings highlight the essential role of both the self-consistency checks and the targeted error report in improving the model’s ability to correct errors.

## 5 Related Work

##### Synthetic Data Generation for Model Fine-tuning.

The performance of large language models (LLMs) hinge on the quality and diversity of their training data. To address the limitations of manual datasets, synthetic data generation methods (Wang et al., ; Xu et al., ) have been developed to automatically create instruction-following examples from LLMs, reducing reliance on human annotations. Various techniques enhance data quality:  Wang et al. () generates multiple reasoning traces and selects the most frequent output to improve robustness, while other approaches (Lightman et al., ; Zhang et al., ) assess response quality based on these traces. Self-training methods utilize synthetic data for iterative fine-tuning, boosting reasoning capabilities (Singh et al., ; Feng et al., ). These advancements show how synthetic data can effectively scale and optimize models through iterative feedback.

##### Large Language Models for Code Generation.

Recent breakthroughs in large language models (LLMs) have greatly enhanced their capability to tackle complex code generation tasks. Much of the research focuses on developing LLMs specialized for code by continuing their pretraining on code data (Guo et al., ; Bai et al., ; Roziere et al., ; DeepSeek-AI et al., ) from open-source repositories like GitHub (Kocetkov et al., ; Lozhkov et al., ) and commit histories (Muennighoff et al., ). Further improvements to these models come from reinforcement learning (Le et al., ) and more often instruction fine-tuning, which involves techniques to address more complex coding problems (Luo et al., ), increasing diversity with unlabeled open-source code (Wei et al., ; Yu et al., ; Wu et al., ), ensuring solution correctness through self-written tests (Chen et al., ), and validating and debugging code execution through interactions with LLM agents (Lei et al., ).

##### Large Language Models for Verilog Coding.

While most code LLMs target software languages, there is increasing interest in models for hardware description languages like Verilog, essential for chip design and verification (Liu et al., ). Previous work has addressed the challenge of limited data through various methods, including synthetic data generation (Liu et al., ), multi-level summarization of open-source Verilog code (Zhao et al., ), and enhanced code augmentation with self-reflection based on compiler feedback (Tsai et al., ; Cui et al., ). Other approaches focus on improving functional correctness and circuit performance through Monte Carlo Tree Search (DeLorenzo et al., ) and discriminator-guided sampling (Pei et al., ).

## 6 Discussions

In this work, we refer to synthetic data generation as methods of using large language models (LLMs) in data generation. While our approach—ensuring correctness through correct-by-construction—could also be considered “synthetic” and resembles methods explored in works like AlphaGeometry (Trinh et al., ), our problems are much simpler and on a smaller scale. Our observations about the variability of models on specific problems align with the findings of  Meta AI (), where “the model knows how to produce the right answer, but it does not know how to select it.” Instead of striving for absolute data correctness, preference learning (Rafailov et al., ; Ethayarajh et al., ) or reinforcement learning (Bai et al., ; Le et al., ), we generate targeted repair data by analyzing errors and re-create such scenarios by injecting similar errors into open-source code, somewhat analogous to how humans consolidate memories during sleep by integrating new information with past experiences (Walker & Stickgold, ; Stickgold, ).

## 7 Conclusion

This paper addresses key challenges in Verilog code generation with correct-by-construction data generation and targeted code repair data strategies. We identified significant issues with synthetic data generation, including difficulties with non-textual representations and variability in performance during training across benchmarks. To address these challenges, we generated data that is correct-by-construction and create targeted repair data by injecting errors to open-source code. Our approach led to substantial improvements, with models fine-tuned using our methods achieving state-of-the-art results on VerilogEval and RTLLM benchmarks. These advancements highlight the effectiveness of our strategies in enhancing model performance in Verilog code generation.

## Appendix A Detailed Results

### A.1 Our Models

We present our models’ results on Verilog benchmarks tested with temperatures 0.2 and 0.8. We ablate across different data blends, with SDG indicating using LLM synthetic generated data in Section 2.1, CC indicating correct-by-construction data targeting non-textual representations in Section 3.1, and Repair representing our targed code repair dataset in Section 3.2.

Our results for RTLLM use the open-source Icarus Verilog simulator(^1^11[https://github.com/steveicarus/iverilog](https://github.com/steveicarus/iverilog)) to check syntax and functional pass rates. This might lead to lower pass rate scores compared to previous work that used Synopsys VCS, as Icarus Verilog does not support all syntax.

**Table 8: Results for our models, across different dataset and temperature on VerilogEval.**
| Model | Dataset | Temperature | VerilogEval (Liu et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Machine (%) | Human (%) |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |
| Starcoder2-15b | SDG | 0.2 | 75.2 | 79.2 | 80.1 | 54.7 | 60.1 | 61.2 |
| 0.8 | 73.7 | 84.0 | 86.1 | 47.4 | 61.9 | 64.8 |  |  |
| SDG-CC | 0.2 | 73.9 | 78.1 | 79.5 | 62.0 | 65.6 | 67.0 |  |
| 0.8 | 72.9 | 84.1 | 87.1 | 58.5 | 70.3 | 73.7 |  |  |
| SDG-CC-Repair | 0.2 | 81.9 | 84.2 | 85.0 | 68.0 | 71.7 | 72.0 |  |
| 0.8 | 78.1 | 86.9 | 88.1 | 64.1 | 72.4 | 74.6 |  |  |
| DeepSeek-6.7b-Instruct | SDG | 0.2 | 73.4 | 77.8 | 78.9 | 48.3 | 53.2 | 54.5 |
| 0.8 | 71.4 | 82.5 | 85.4 | 44.0 | 58.1 | 62.3 |  |  |
| SDG-CC | 0.2 | 72.6 | 78.2 | 79.3 | 58.5 | 62.6 | 63.5 |  |
| 0.8 | 70.2 | 83.1 | 85.4 | 56.3 | 67.0 | 70.7 |  |  |
| SDG-CC-Repair | 0.2 | 77.8 | 82.7 | 83.4 | 65.4 | 67.7 | 68.2 |  |
| 0.8 | 75.2 | 85.5 | 88.1 | 61.6 | 70.0 | 72.1 |  |  |
| CodeLlama-7b-Instruct | SDG | 0.2 | 74.5 | 77.9 | 78.8 | 45.3 | 50.3 | 51.5 |
| 0.8 | 71.2 | 82.6 | 85.1 | 42.6 | 55.6 | 59.0 |  |  |
| SDG-CC | 0.2 | 74.2 | 77.4 | 78.1 | 55.1 | 61.0 | 62.4 |  |
| 0.8 | 70.0 | 81.2 | 83.7 | 51.6 | 64.4 | 67.7 |  |  |
| SDG-CC-Repair | 0.2 | 78.1 | 81.5 | 81.7 | 63.1 | 66.2 | 66.8 |  |
| 0.8 | 73.7 | 85.5 | 87.8 | 58.1 | 67.8 | 69.7 |  |  |

**Table 9: Results for our models, across different dataset and temperature on RTLLM.**
| Model | Dataset | Temperature | RTLLM v1.1 Lu et al. () |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Syntax (%) | Func. (%) |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |
| Starcoder2-15b | SDG | 0.2 | 78.1 | 86.5 | 90.1 | 49.0 | 60.4 | 66.3 |
| 0.8 | 77.1 | 89.0 | 94.1 | 43.8 | 62.1 | 68.0 |  |  |
| SDG-CC | 0.2 | 78.3 | 89.3 | 92.7 | 45.5 | 58.3 | 62.0 |  |
| 0.8 | 76.9 | 92.6 | 95.5 | 38.4 | 62.8 | 70.4 |  |  |
| SDG-CC-Repair | 0.2 | 79.8 | 87.9 | 90.5 | 49.0 | 59.1 | 62.6 |  |
| 0.8 | 79.3 | 93.9 | 96.2 | 45.3 | 65.8 | 74.5 |  |  |
| DeepSeek-6.7b-Instruct | SDG | 0.2 | 79.3 | 86.8 | 90.5 | 40.3 | 45.9 | 49.6 |
| 0.8 | 76.6 | 92.5 | 96.2 | 40.0 | 53.8 | 63.6 |  |  |
| SDG-CC | 0.2 | 73.6 | 84.5 | 86.0 | 44.3 | 52.2 | 54.3 |  |
| 0.8 | 76.7 | 90.5 | 93.8 | 39.5 | 56.4 | 63.1 |  |  |
| SDG-CC-Repair | 0.2 | 84.3 | 92.2 | 93.0 | 53.1 | 58.8 | 60.3 |  |
| 0.8 | 80.0 | 92.9 | 95.4 | 45.5 | 57.9 | 62.6 |  |  |
| CodeLlama-7b-Instruct | SDG | 0.2 | 74.0 | 82.5 | 86.8 | 30.0 | 33.9 | 35.8 |
| 0.8 | 70.9 | 89.1 | 94.5 | 34.0 | 47.2 | 52.8 |  |  |
| SDG-CC | 0.2 | 75.0 | 90.2 | 94.6 | 39.7 | 44.4 | 47.2 |  |
| 0.8 | 76.4 | 93.9 | 96.3 | 35.5 | 47.6 | 52.7 |  |  |
| SDG-CC-Repair | 0.2 | 85.7 | 93.9 | 94.8 | 42.6 | 49.4 | 51.2 |  |
| 0.8 | 80.3 | 93.9 | 94.8 | 36.9 | 52.9 | 58.2 |  |  |

### A.2 Foundational and Frontier Code Models

We present detailed results on recent foundational and frontier code models. We also re-evaluate all models on RTLLM using unbiased pass@k metric.

**Table 10: Results on foundational and code models on VerilogEval.**
| Type | Model | Size | Temp | VerilogEval (Liu et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Machine (%) | Human (%) |  |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |  |
| Foundational<br>Models | Llama-3.1 | 8B | 0.2 | 48.7 | 66.2 | 70.6 | 26.9 | 36.9 | 40.4 |
| 0.8 | 42.1 | 67.3 | 74.1 | 23.0 | 37.8 | 44.2 |  |  |  |
| Llama-3.1 | 70B | 0.2 | 66.7 | 73.8 | 76.9 | 48.7 | 53.6 | 55.1 |  |
| 0.8 | 64.5 | 77.7 | 80.4 | 48.0 | 57.0 | 60.9 |  |  |  |
| Llama-3.1 | 405B | 0.2 | 67.3 | 72.8 | 74.1 | 51.9 | 57.0 | 58.9 |  |
| 0.8 | 66.4 | 75.1 | 76.9 | 53.8 | 61.0 | 62.8 |  |  |  |
| Nemotron-4 | 340B | 0.2 | 53.0 | 59.1 | 61.5 | 43.1 | 43.9 | 44.9 |  |
| 0.8 | 50.8 | 60.3 | 62.2 | 40.8 | 48.3 | 50.0 |  |  |  |
| GPT-3.5-turbo | - | 0.2 | 58.0 | 66.4 | 68.5 | 31.2 | 39.4 | 41.7 |  |
|  | 0.8 | 56.6 | 74.0 | 77.6 | 28.9 | 44.1 | 47.4 |  |  |
| GPT-4 | - | 0.2 | 53.2 | 63.7 | 66.4 | 36.1 | 43.5 | 46.2 |  |
|  | 0.8 | 35.3 | 53.4 | 58.9 | 35.2 | 53.4 | 58.9 |  |  |
| GPT-4-turbo | - | 0.2 | 57.8 | 66.7 | 70.6 | 54.1 | 61.2 | 62.8 |  |
|  | 0.8 | 56.9 | 69.5 | 73.4 | 53.6 | 63.6 | 66.7 |  |  |
|  | GPT-4o | - | 0.2 | 65.9 | 68.9 | 69.2 | 57.1 | 61.3 | 62.2 |
|  |  | 0.8 | 62.9 | 71.4 | 72.7 | 55.4 | 63.9 | 66.7 |  |
| Code<br>Models | Starcoder2 | 15B | 0.2 | 68.7 | 76.7 | 78.6 | 37.7 | 48.3 | 51.1 |
| 0.8 | 57.7 | 82.3 | 88.5 | 29.1 | 50.6 | 57.2 |  |  |  |
| DeepSeek-Coder-V2 | 16B | 0.2 | 67.4 | 74.6 | 76.2 | 46.9 | 53.3 | 54.5 |  |
| 0.8 | 65.6 | 78.3 | 81.8 | 46.3 | 55.9 | 58.9 |  |  |  |
| DeepSeek-Coder-V2 | 236B | 0.2 | 68.2 | 72.7 | 75.0 | 56.4 | 60.7 | 64.3 |  |
| 0.8 | 66.5 | 74.1 | 76.2 | 54.8 | 62.2 | 66.0 |  |  |  |

**Table 11: Results on foundational and code models on RTLLM.**
| Type | Model | Size | Temp | RTLLM v1.1 (Lu et al., ) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Syntax (%) | Func. (%) |  |  |  |  |  |  |  |  |
| pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 |  |  |  |  |
| Foundational<br>Models | Llama-3.1 | 8B | 0.2 | 39.7 | 53.1 | 55.2 | 19.3 | 25.8 | 27.6 |
| 0.8 | 40.7 | 60.6 | 65.5 | 17.6 | 34.7 | 37.9 |  |  |  |
| Llama-3.1 | 70B | 0.2 | 47.9 | 51.7 | 55.2 | 34.1 | 34.5 | 34.5 |  |
| 0.8 | 48.9 | 57.6 | 58.6 | 29.6 | 31.0 | 31.0 |  |  |  |
| Llama-3.1 | 405B | 0.2 | 56.5 | 63.9 | 65.5 | 38.9 | 45.0 | 48.3 |  |
| 0.8 | 52.1 | 64.4 | 72.4 | 35.8 | 45.8 | 51.7 |  |  |  |
| Nemotron-4 | 340B | 0.2 | 41.7 | 47.2 | 48.3 | 14.1 | 15.5 | 17.2 |  |
| 0.8 | 41.7 | 46.3 | 48.3 | 18.9 | 20.7 | 20.7 |  |  |  |
| GPT-3.5-turbo | - | 0.2 | 50.3 | 58.2 | 58.6 | 28.3 | 36.9 | 41.4 |  |
| 0.8 | 48.2 | 61.2 | 65.5 | 24.1 | 36.9 | 41.4 |  |  |  |
| GPT-4 | - | 0.2 | 49.3 | 65.9 | 68.9 | 30.0 | 44.4 | 48.3 |  |
| 0.8 | 42.8 | 61.2 | 65.5 | 25.9 | 40.0 | 44.8 |  |  |  |
| GPT-4-turbo | - | 0.2 | 38.9 | 44.8 | 48.3 | 27.2 | 35.1 | 37.9 |  |
| 0.8 | 40.3 | 48.8 | 51.7 | 27.5 | 40.2 | 44.8 |  |  |  |
| GPT-4o | - | 0.2 | 50.3 | 59.9 | 62.1 | 33.8 | 44.4 | 48.3 |  |
| 0.8 | 47.5 | 63.2 | 66.7 | 31.3 | 44.1 | 48.3 |  |  |  |
| Code<br>Models | CodeLlama | 7B | 0.2 | 46.6 | 62.6 | 68.9 | 17.9 | 29.9 | 34.5 |
| 0.8 | 34.8 | 59.7 | 68.9 | 13.4 | 25.9 | 31.0 |  |  |  |
| CodeQwen | 7B | 0.2 | 45.8 | 55.8 | 58.6 | 24.1 | 33.1 | 37.9 |  |
| 0.8 | 45.5 | 65.7 | 72.4 | 22.4 | 34.0 | 37.9 |  |  |  |
| Starcoder2 | 15B | 0.2 | 38.3 | 77.5 | 86.3 | 15.5 | 37.6 | 44.6 |  |
| 0.8 | 31.6 | 81.0 | 94.7 | 11.0 | 34.2 | 45.7 |  |  |  |
| DeepSeek-Coder | 6.7B | 0.2 | 51.4 | 62.6 | 65.5 | 23.1 | 26.8 | 27.6 |  |
| 0.8 | 49.7 | 64.4 | 68.9 | 21.0 | 29.3 | 34.5 |  |  |  |
| DeepSeek-Coder-V2 | 16B | 0.2 | 51.4 | 51.7 | 51.7 | 33.1 | 34.5 | 34.5 |  |
| 0.8 | 51.4 | 57.8 | 58.6 | 30.0 | 37.1 | 37.9 |  |  |  |
|  | DeepSeek-Coder-V2 | 236B | 0.2 | 63.4 | 73.0 | 79.3 | 34.5 | 44.9 | 52.9 |
|  | 0.8 | 61.8 | 78.1 | 79.3 | 32.9 | 50.2 | 55.1 |  |  |

### A.3 Details on Evaluations

We format the prompt input as follows for VerilogEval, where the detail_description is the problem description (Machine or Human) and prompt field is the problem module header. We include module headers to avoid confusion on the signals naming.

An example of mux2to1 in VerilogEval-Human:

We use similar templates for RTLLM v1.1, where we extract the top module header from the reference solution and provide it as input. Below is an example of adder_8bit:

We use default chat templates and default system prompts for open-source models tested. For GPT models from OpenAI, we use the following system prompt:

We post-process model responses to extract code. We extract content enclosed by triple backticks and remove the language identifier (Verilog). We then extract code enclosed in module and endmodule keywords with response.find(’module’) and response.rfind(’endmodule’). If the extracted code does not include a module header, the reference solution’s module header will be prepended. The code is then tested with the provided testbenches with the Icarus Verilog (iverilog) simulator to evaluate for syntax and functional correctness. This might lead to lower pass rate scores for RTLLM compared to previous work that used Synopsys VCS, as Icarus Verilog does not support all syntax.

### A.4 VerilogEval-NonText

We select the following 45 problems from VerilogEval-Human that consists of non-textual representations in their problem descriptions:

2012_q1g, 2012_q2b, 2012_q2fsm, 2013_q2afsm, 2014_q3bfsm, 2014_q3c, always_nolatches, circuit1, circuit10, circuit2, circuit3, circuit4, circuit5, circuit6, circuit7, circuit8, circuit9, ece241_2013_q7, ece241_2014_q3, ece241_2014_q5b, fsm1, fsm1s, fsm2, fsm2s, fsm3, fsm3comb, fsm3onehot, fsm3s, fsm_onehot, fsm_ps2data, kmap1, kmap2, kmap3, kmap4, m2014_q3, m2014_q6, m2014_q6b, m2014_q6c, mt2015_q4, mt2015_q4a, mt2015_q4b, review2015_fsmonehot, rule110, rule90, truthtable1

### A.5 Template Problems for Correct-by-Construction Data

When generating correct-by-construction CC data, we select 11 problems from VerilogEval-NonText to use as representative templates for constructing our prompts. To prevent contamination, we ensure that benchmark problems are excluded from our data. While our prompts will resemble those of the selected problems, the non-textual representations and solutions will differ. Additionally, to prevent overfitting to specific prompt templates, we use LLMs to rewrite the problem instructions for 20% of our data. Furthermore, we create validation test problems that are strictly in-distribution, based on the chosen problems.

Karnaugh Maps and Truth Tables: kmap1, m2014_q3, truthtable1.

State Transition Graphs and Tables: 2012_q2b, 2014_q3c, ece241_2014_q5b, fsm3comb, fsm3onehot, fsm_onehot, m2014_q6b, m2014_q6c.

Waveforms: We do not base our data on any benchmark problems specifically.

**Table 12: Scaling Repair data.**
| Model | VerilogEval | RTLLM v1.1 |  |
| --- | --- | --- | --- |
| Machine | Human | Func |  |
| pass@1 (%) | pass@5 (%) |  |  |
| SDG-CC | 73.9 | 62.0 | 62.8 |
| SDG-CC-Repair 1k | 81.9 | 68.0 | 65.8 |
| SDG-CC-Repair 7k | 82.2 | 67.4 | 64.5 |

### A.6 Scaling Repair Data

As shown in Table 12, a carefully filtered dataset of 1.4k samples achieves comparable performance to a 7.8k dataset. This suggests that merely increasing the dataset size by injecting the same types of errors does not contribute meaningfully to improving model performance.

**Table 13: Iterative code repair.**
| Model | VerilogEval | RTLLM v1.1 |  |
| --- | --- | --- | --- |
| Machine | Human | Func |  |
| pass@1 (%) | pass@5 (%) |  |  |
| SDG-CC | 73.9 | 62.0 | 62.8 |
| SDG-CC-Repair Iter 1 | 81.9 | 68.0 | 65.8 |
| SDG-CC-Repair Iter 2 | 81.3 | 68.1 | 65.6 |

### A.7 Iterative Code Repair

We conduct a second iteration by generating 2.7k repair data for the model based on the Repair data from the first iteration. As shown in  Table 13, performance mostly saturates after this initial iteration. We suspect that the remaining issues are likely due to significant errors that are challenging to correct.

### A.8 Diversity of Generated Code

We assess the diversity of the code generated by our models. We measure this diversity using BLEU score, Jaccard similarity, and abstract tree edit distance (TSED) in Song et al. (). The VerilogEval-Human problems are categorized into NonText and Text, as described in Section A.4. For each problem, we compute the average code diversity score across sampled codes for the same problem and report the mean score for all problems. For TSED, we use PyVerilog (Takamaeda-Yamazaki, ) to extract the abstract syntax tree, and codes that fail syntax checks are excluded from the analysis.

Table 14 presents the results on code diversity. We sample 20 solutions with temperature of 0.8 for each model. We observe that fine-tuned models generally show a decrease in code diversity for both Text and NonText problems. This reduction is expected, as BLEU and Jaccard metrics account for both correct and incorrect code solutions, and there are often multiple ways to implement a correct solution. When comparing our fine-tuned models with GPT-4o, code diversity is similar for Text problems, but our models exhibit poor diversity for NonText problems. This is anticipated, given that the CC training dataset for NonText problems is generated using correct-by-construction methods and follows similar templates for Verilog code. However, our models demonstrate comparable diversity to GPT-4o for Text problems, particularly in TSED metric.

**Table 14: Diversity of generated code solutions on VerilogEval-Human sampled with temperature of 0.8. Lower scores indicate higher diversity.**
| Type | Models | Text | NonText |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Jaccard | BLEU | TSED | Jaccard | BLEU | TSED |  |  |
| Pretrained<br>Models | CodeLlama | 0.5330 | 0.3808 | 0.4255 | 0.4707 | 0.2507 | 0.3521 |
| DeepSeek-Coder | 0.6606 | 0.5454 | 0.5956 | 0.6548 | 0.3797 | 0.3847 |  |
| Starcoder2 | 0.7724 | 0.5084 | 0.5520 | 0.7212 | 0.3607 | 0.4020 |  |
| GPT-4o | 0.6798 | 0.6633 | 0.6906 | 0.7390 | 0.6376 | 0.6137 |  |
| Ours<br>SDG-CC-Repair | CodeLlama | 0.6848 | 0.5992 | 0.6354 | 0.8583 | 0.7242 | 0.7158 |
| DeepSeek-Coder | 0.6828 | 0.6040 | 0.6319 | 0.8308 | 0.6866 | 0.6598 |  |
| Starcoder2 | 0.7018 | 0.6381 | 0.6721 | 0.8799 | 0.7750 | 0.7740 |  |

## Appendix B Examples of Targeted Code Repair Data

Figure: Listing 1: Correct Code

Figure: Listing 2: Error Code

Figure: Figure 7: An example demonstrating the process for targeted code repair. In this example, training checkpoints during training would have the model sometimes generated correct and error solutions. We use LLMs to first summarize the errors for a detailed Error Report and then inject the errors to open-source code to construct Repair data.

Figure: Listing 3: Correct Code

Figure: Listing 4: Error Code

Figure: Figure 8: Another example based on shift4 from VerilogEval-Human.

Figure: Listing 5: Correct Code

Figure: Listing 6: Error Code

Figure: Figure 9: Another example based on multi_booth_8bit from RTLLM.

## Appendix C Examples of Correct-by-Construction Data for Non-Textual Representations

### C.1 Karnaugh Maps and Truth Tables

Figure: Figure 10: Example of a generated problem with Karnaugh Map representation in the problem description. We include detailed step-by-step analysis for reaching the correct solution.

### C.2 State Transition Graphs and Tables

Figure: Figure 11: Example of problem with state transition tables.

Figure: Figure 12: Example of problem with Moore FSM.

Figure: Figure 13: Example of problem with Mealy FSM.

Figure: Figure 14: Example of problem with in-edge focused solution shown in Figure 3.

### C.3 Waveforms

Figure: Figure 15: Example of a waveform problem for combinational circuits.

Figure: Figure 16: Example of a waveform problem for sequential circuits.

## Appendix D Prompt Templates

### D.1 Synthetic Data Generation

#### D.1.1 Self-Instruct

Figure: Figure 17: Prompt used to generate initial 50 seed problems for Self-Instruct.

Figure: Figure 18: Prompt used for Self-Instruct.

#### D.1.2 OSS-Instruct

Figure: Figure 19: Prompt used for OSS-Instruct. We also include prompts inspired from Evol-Instruct (Luo et al., ) to increase problem difficulty.

#### D.1.3 Docu-Instruct

Figure: Figure 20: Prompt used for Docu-Instruct with Wikipedia and textbooks.

Figure: Figure 21: Prompt used to filter Verilog related Wikipedia pages.

#### D.1.4 Non-textual Representations

Figure: Figure 22: An prompt example to encourage LLMs to generate questions with Karnaugh Maps.

#### D.1.5 Prompts for Sampling Solutions with LLM Generated Problems

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: * Fig 1 is kind of confusing. Why choose checkpoints 1 and 2? Would we hope for the pass@k for checkpoint 2 to be higher than for chkpt 1?  This scatter-plot resembles a confusion matrix-- why choose the scatter plot representation over a different option? The value of figure 1 is made more apparent once we see figure 5. Maybe the two could be presented closer to one another in a camera-ready. How were the "solvable" and "unsolvable" regions chosen? 
* L319: how do we know that the ability to self-correct (validating via self-consistency) is due to a good error report, and not the model's ability to correct independent of the error report? Especially since the examples from which error reports are generated did yield both correct and incorrect generations, to start with.
* is the amount of training data consistent between all rows of Table 6?
soundness: 4
strengths: * This paper presents a clear discussion of an important and under-explored topic. Low-level programming languages are an appealing area in which to automate code reasoning, and programs in HDLs are notoriously difficult to verify. 
* thorough evaluation in terms of comparison to other SDG methods and other baselines. Appropriate ablations further convey the value of all components of their approach.
* the code repair generation process is compelling, and validation well-grounded in existing literature. I anticipate that it's highly transferrable to other domains of data as well. 
* the combination of using hand-crafted methods for highly-underrepresented or challenging concepts ("non-textual elements") and automated self-consistency-based methods for intermediate concepts (generating the repair data) paints a cohesive picture for SDG, especially in this domain.
summary: This paper does a thorough evaluation of LLMs for verilog code generation. They first analyze existing model performance on Verilog code generation tasks, identify that "non-textual representations" are commonly mis-reasoned about, use this to motivate two new methods for improving SDG for verilog code gen tasks, and test their approach against other SDG approaches. They find that their method outperforms baselines.
weaknesses: * the data generation processes for Karnaugh maps, state-transition diagrams, and waveforms are pretty hand-crafted. This makes this method difficult to transfer to other identified model weakness categories, and requires human-tuning to identify the best data gen method per category. This approach also may not work as well, if at all, on some categories. (For example, the findings of L461 that indicate the Waveforms problems do not improve as much as the other approaches.) An automated method for designing the data construction may scale better. (out of scope for this paper though, and I would not consider this a reason for rejection)

---

confidence: 1
contribution: 2
initial_rating: 5
presentation: 1
questions: 1. Could the authors discuss how this method for Verilog-specific elements might be adapted for other HDLs or general programming languages?

2. Figures 2, 4, 5, and 6, along with Tables 4 and 5, could benefit from clearer formatting and structure. Could the authors enhance these visuals to improve readability and clarify how the best results are highlighted across different model types?
soundness: 3
strengths: The primary contributions of this paper include the introduction of correct-by-construction data generation, which focuses on non-textual data representations that are essential for Verilog code and often challenging for LLMs. By incorporating Karnaugh maps, state-transition diagrams, and waveforms, the model’s capacity to interpret and generate these complex data formats improves. The experimental results demonstrate notable improvements over previous approaches on multiple benchmarks​.
summary: The paper introduces CraftRTL, a novel approach to Verilog code generation by leveraging a combination of synthetic data generation and targeted code repair to improve accuracy and robustness.
weaknesses: The methods presented, particularly the correct-by-construction data targeting non-textual representations, are tailored heavily to Verilog-specific constructs such as Karnaugh maps, state-transition diagrams, and waveforms. While this adaptation effectively improves performance for Verilog code generation, the approach may have limited applicability to other hardware description languages or general programming languages that do not rely on these specific data formats. A broader discussion on how these techniques could be generalized would strengthen the paper's impact.

Several figures and tables in the paper, notably Figures 2, 4, 5, and 6, as well as Tables 4 and 5, suffer from presentation clarity issues. Figures lack a cohesive and clear structure, making it difficult for readers to follow the exact steps. In Tables 4 and 5, the inconsistent formatting of model types and unclear emphasis on the best-performing results within each category lead to potential confusion in understanding the experimental results.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Why focus solely on karnaugh maps, state-transition diagrams, and waveforms? They do not represent all types of non-textual representations.
2. It is essential to ensure that the generated error report can effectively guide the model in correcting errors. How do the authors validate its effectiveness?
3. In the "Targeted Code Repair Dataset" section, I suggest the author to  provide classification and proportion of the "minor" errors. Additionally, were any additional data augmentation measures taken for high-frequency errors during dataset construction?
soundness: 3
strengths: 1. The paper conducts a detailed empirical analysis of the two main issues in Verilog code.
2. The paper provides a thorough comparison with existing methods and shows good performance.
summary: This paper discusses two main issues when LLMs  handling Verilog code: models have difficulty handling non-textual elements in problem statements and models making "minor" programming mistakes. To address these issues, the authors specifically created a transformed non-textual dataset and code repair dataset to fine-tune the model. The results demonstrate that the fine-tuned Starcoder2-15B surpasses the prior state-of-the-art results in Pass@1 performance, achieving improvements of 3.8\%, 10.9\%, and 6.6\% on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.
weaknesses: 1. The main contribution of this work lies in constructing a fine-tuning dataset to address non-textual data and minor error issues. The technical contribution of the paper is limited.
2. What is the specific definition of "minor" errors, and what common characteristics do they share?
3. The font size of Figures 2 and 4 is too small to read.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - Checkpoint Selection: what is the selection criteria of your checkpoints in Figure 1 and Figure 5? You mentions "two consecutive checkpoints" in Line 434. Additionally, you only fine-tune your model for one epoch (Line 364), so at least one checkpoint might not see all training data. Whether such difference affects your results?
- Application Scenario: This paper addresses an important problem in Verilog code generation utilizing domain knowledge of Verilog. So I am curious how such approach is applied to similar tasks, e.g., code generation without sufficient training data?
soundness: 4
strengths: - This paper is clearly written and easy to comprehend
- This paper is well-motivated and address an important downstream task, automated Verilog code generation
- This paper includes a comprehensive and reliable data construction pipeline
- This paper conduct a comprehensive evaluation on three LLMs with SOTA baselines
summary: - This paper performs a thorough analysis of fine-tuned LLMs on Verilog code, and revealing two main challenges of automated Verilog code generation
- This paper creates a large number of correct-by-construction data to ensure solution correctness, incorporating Karnaugh Maps, state-transition diagrams, and waveforms
- This paper develops an automated framework that utilizes LLMs to generate error reports from benchmark problems
- Its evaluation results demonstrate that models fine-tuned with our data achieve state-of-the-art performance on Verilog coding, outperforming prior SOTA results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively
weaknesses: Actually I like this paper, especially the data construction section; however, there are still some minor concerns:

- "Quality Assurance with LLM Validation" (Line 317): Please provide more evidence about the choice of LLM validation. What is the rationale (or its limitations) of not using deterministic validation approaches, e.g., model checking?
- In Section 2.3, you mention "significant variability in the model’s pass rate on specific benchmark problems across different checkpoints" (Line 164), while the results in Figure 1 indicates a highly positive correlation. Also, 15% discrepancies is also acceptable between two checkpoints. Can you provide me with a stronger evidence to support this claim, e.g., Pearson Correlation Coefficient, or explain why such difference is significant in this task?
- Application scenario: This paper mainly utilizes domain specific patterns of various types of Verilog while it might be difficult when applied to similar tasks, e.g., code generation without sufficient training data.
- Reproducibility Statement: this subsection exceeds the 10 pages limit. I think it should be placed within the first 10 pages or directly moved to appendix
- Availability: this paper does not provide an available artifact

Meta-review: The paper introduces methods for training code LLMs on an important hardware description language, doing a deep dive on the specific kinds of errors made on those problems and addressing them through correct-by-construction synthetic data generation. The primary strength is that it addresses an important problem and has a robust empirical evaluation, while the primary weakness is that the method is not particularly conceptually creative and relatively narrow in its practical implications. I recommend accepting this paper due to it's practical importance, and the fact that researchers in the ML and HDL communities could likely build on these models and methods.

Decision: Accept (Poster)