CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair
Authors: Mingjie Liu
Yun-Da Tsai
Wenfei Zhou
Haoxing Ren
Venue: ICLR 2025 Poster (2025)
Keywords: Verilog Code Generation
Synthetic Data Generation
Large Language Models
Primary area: generative models
TL;DR: We produce high-quality Verilog finetuning data that is correct-by-construction for non-textual representations and develop targeted code repair data by injecting errors into open-source code.

Abstract: Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making ''minor'' mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8\%, 10.9\%, 6.6\% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.

Published as a conference paper at ICLR 2025
CRAFTRTL: HIGH-QUALITY SYNTHETIC DATA GENERATION FOR VERILOG CODE MODELS WITH CORRECT-EY-CONSSTUUCIIO RESENTATIONS AND TARGETED CODE REPAIR
REP-
Mingjie Liu", Yun-Da Tsai', Wenfei Zhou, Haoxing Ren
NVIDIA Corporation
(mingjiel, yundat, wenfeiz,
ABSTRACT
Despite the significant progress made in code generation with large language mod- els, challenges persist, especially with hardware description languages such as Verilog. This paper first presents analysis of fine-tuned LLMs Verilog cod- ing, with synthetic data from prior methods. We identify two main issues: dif- ficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making "minor" mistakes. To address these limitations, enhance data curation by creating poreec-by-conssruecee resentations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open- source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-arr results by 3.8%, 10.9%, 6.6% for pass on VeriahnEval-Machhnne,, VerilogEval-Human and RTLLM.
data targeting non-textual rep-
INTRODUCTION
Large Language Models (LLMs) have achieved significant success across various natural language processing tasks and have extended their capabilities to code generation. leading to the development of specialized models targeting code generation. The effectiveness of these models is largely in- fluenced by the size and quality of their training datasets, as highlighted by scaling laws (Achiam et al. |023] /Zhan et al., (024a)). Prominent code L.MMs have set new henchmarks records by uti- lizing extensive, synthetically generated datasets through methods like Self-Instruct (Wang et al.)
2022 Chaudhary (023), Evol-Instruct (Xu et al. 2023). and OSS-Instruct (Wei et al. (023). These synthetic data generation techniques allow code LLMs to generate a wide range of complex code examples, enhancing their training and performance in real-world coding scenarios.
While most code LLMs concentrate on software programming languages, there is increasing interest in developing models for hardware description languages (HDLss), which essential for chip de- sign and hardware verification. Despite efforts to collect and synthesize more diverse Verilog code enhance specialized code LLMs (Liu et al.) 2023c Pei al. 2024 Cui al. 2024 Zhao al.) 2024), HDLS still face challenges akin to those encountered in low-resource languages (Cassano) al. 2022). These challenges mainly due the limited availability of high-quality instruction- following data and the constrained capability of existing LLMs to generate RTL code, which affects the models' performance and their ability to generalize across programming languages.
Developing high-quality synthetic Verilog code for training code large language models (LLMs) faces significant challenges due to two primary factors. Firstly, Verilog is considered a low-resource language (Cassano et al. (022),, meaning there is scarcity of available training data compared to high-resource software programming languages like Python. This limited data availability restricts the models' ability to learn diverse and complex coding patterns effectively. Secondly, verifying the correctness of hardware description language (HDL) code, such as Verilog, is inherently more
"qqual Contributions.

---

Published as a conference paper at ICLR 2025
complex than verifying software code. While software code correctness often be assessed using random test cases and automated unit tests (Chen et al.] 2022), hardware code requires comprehen- sive testbenches and rigorous verification planning and methodologies. This additional complexity makes challenging to ensure that synthetic Verilog code functionally accurate (Bhandari et al., 2024
Qiu al. 2024), posing barrier improving model performance.
In this paper, we start with a thorough analysis of fine-tuned large language models (LLMs) applied Verilog code, using synthetic data techniques from previous works. Our analysis reveals key issues: (1) models have difficulty handling non-textual elements in problem statements, indicating challenges in interpreting complex or unconventional inputs; and (2) there is notable variability in the models' pass rates across different benchmark problems and training checkpoints, exposing in- consistencies in learning outcomes, often due to the models making "minor" programming mistakes.
Given the limitations identified in our analysis of relying solely on LLMs for generating synthetic data, we shift our focus to improving data curation address these issues. Current LLMs fre- quently struggle with interpreting and processing non-textual representations and are insufficient in generating effective testbenches for evaluating solution quality. Therefore, instead of depending ex- clusively LLMs to address data quality concerns, we develop targeted fine-tuning data to better mitigate these problems. Experimental results demonstrate that our models achieve ooce--of-heaatt (SOTA) results VerilogEval (Liu et al. 2023b) and RTLLM v1.1 (Lu al. 2024) benchmarks, outperforming prior works by large margins on problems with human-level description. The major contributions of this paper are as follows:
We perform a thorough analysis of fine-tuned LLMs on Verilog code using previously established synthetic data generation methods, uncovering challenges with non-textual el- ements and notable variability performance across benchmark problems during training.
We create borrect-by-consttuccio data to ensure solution correctness, incorporating Kar- naugh Maps, state-transition diagrams, and waveforms, which significantly enhance the model's ability to handle non-textual representations.
We develop an automated framework that utilizes LLMs to generate error reports from benchmark problems at various checkpoints, which then injected into open-source code to create a fine-tuning dataset targeted at correcting the model's 's specific "minor" mistakes.
We rigorously evaluate the latest foundational and frontier code models. We note that recent advanced models like GPT-4o already reached competitive performance compared to previous efforts targeting Verilog code generation.
Experimental results demonstrate that models fine-tuned with our data achieve state-of-the- art performance on Verilog coding. Specifically, our fine-tuned model based on Starcoder2- 15B (Lozhkov et al. (024) outperforms prior SOTA results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogExvl--Machng VerilogEval- Humaa and RTLLM, respectively.
2 EXAMINING FINE-TUNED LLMs USING SYNTHETIC GENERATED DATA ON VERILOG CODING
In this section, we start with thorough analysis of fine-tuned large language models (LLMs) applied to Verilog code. We adapt previous approaches for generating synthetic data for general coding to focus on Verilog code. For our pilot study, we only present results based on fine-tuning StarCoder2- 15B (Lozhkov et al.) 2024). Details experimental settings are the same as in Section 4 We assess model performance in Verilog code completion and identify two main issues. First, the mod- els demonstrate notably poor performance when dealing with non-textual elements in the problem statements. Second, the variability in the models' pass rates across different benchmark problems and training checkpoints suggests inconsistencies in learning outcomes and model variability.
2.1 SYNTHETIC DATA GENERATION FOR VERILOG CODING
We build previous methods for synthetic data generation by applying
(Wang al.)
2022] and Instruct (Wei et al. (023) with custom prompt templates tailored for Verilog coding. To enhance data coverage and diversity, supplement these techniques with additional context
2

---

Published as a conference paper at ICLR 2025
from Wikipedia and textbooks. We also prompt models to generate problem descriptions to include
non-textual representations.
We use nemotron-4-333 -ettttroon--40b--ime -4400-instruct (Nvidia et al. 2024) selected for its open license that allows commercial use. Our process includes deduplication and decontamination procedure akin to that out- lined by Li et al. (2023). Additionally, we conduct syntax checks to eliminate coding problems containing docstrings from Verilog benchmarks. To ensure further data quality, we dis- card code solutions that fail these syntax checks and apply self- verification (Weng et al., 2023) to remove entries where the LLM identifies errors in the solution. Table [ shows the quantity of our synthetic data generation (denoted as SDG) after deduplication and filtering, yielding total of 80. fine-tuning examples.
Table 1: Data quantity SDG.
Method
Quantity
solutions
Sclf-Instruct
24.7k
OSS-Instruct
28.4k
Docu-Instruct
12.0k
Non-textual
15.0k
SDG Total 80.1k 1k
Self-Instruct We follow the approach outlined in Wang et al. (2022) to generate synthetic Verilog coding problems. Initially, we randomly generate from the LLM and curate 50 questions that request Verilog coding problems without any in-context exam- ples. From these, we then randomly choose 1 to 5 seed questions to use as in-context examples.
OSS-Instruct We begin by processing pretraining code data extract our seed code from The Stack v2 (Lozhkov et al.] 2024). focusing on Verilog and System Following the approach in Liu al. (2023b), we post-process this data by selecting self-contained Verilog code that passes syntax checks using Pyverilog (Takamaeda- Yamazaki| (015). With the refined seed code data, we then prompt large language models (LL.Ms) to use this code as inspiration for generating Verilog coding problems similar to
Wei et al. (2023).
Docu-Instruct Drawing inspiration from Nvidia et al. (2024) and Sudalairaj et al. (2024), we
utilize document sources from Wikipedia and textbooks for instruction generation. We begin by filtering Wikipedia entries, prompting the LLM to classify whether the content pertains to hardware design or Verilog coding concepts. Additionally, textbooks. These textbooks are then segmented into chunks of paragraphs or sentences, ensuring each chunk contains fewer than 2k tokens.
manually selected approximately relevant 100
Non-textual Representations VerilogExallHuman (Liu et al. 2023b) includes benchmark prob- lems involving non-textual representations. For example, Boolean logic tables and Karnaugh maps are presented in tabular formats, state-transition diagrams for finite state machines are depicted as edge lists and sequential waveforms are described in tables with signal outputs recorded at various time steps. To incorporate such representations, we encouraged LLMs to generate problems from open-source code, with instructions to utilize these tabular data structures.
2.2 CHALLENGES WITH NON- TEXTUAL REPRESENTATION
We observe that models underperform on bench- mark problems involving non-textual input formats, such as Karnaugh Maps, state-transition diagrams, and waveforms. Table 2 shows the pass@1 results for the VerilogEval (Liu et al. 2023b). Addition- ally, we have identified a subset of 45 questions within BaaloogEva--ummaae that include non-textual representations, termed VerilogEval-NonTer pears that models like GPT-4o and Starcoder2 strug- gle with these non-textual formats, likely due to insufficient representation of such data during both pretraining and fine-tuning. Despite our efforts to generate such questions during synthetic data cre- ation, our fine-tuned models still lag in these areas. This outcome is not entirely surprising, given that the LLMs used were also ineffective at generating problems with these representations, compli- cating the validation of fine-tuning data. These results suggest that merely including non-textual data insufficient; ensuring the quality and correctness of the data, particularly that the code solutions accurately align with these representations, is crucial.
Table 2: pass @ results on VerilogEval sam- pled with temperature of 0.8.
Model
Machine
Human
NonText
GPT-4o
63.7
55.4
27.0
Starcoder2
57.7
29.1
10.3
Starcoder2-SDG
73.7
47.4
22.2
It ap-
3

---

Published as a conference paper at ICLR 2025
12
25
02-
03
chadlpain)
chacperri
(a) Starcoder2-15B on SDG.
(b) Starcoder2-155 on SDG-CC-Repair.
Figure I: Our methods reduce pass rate variability during training: SDG (left) shows high volatility with significant degradation on many problems, while SDG-CC-Repai (right) stabilizes learning outcomes on solvable problems (details in Appendix(A.10 A.
2.3 VARIABILITY ON PASS RATES DURING TRAINING
During our training, we observed significant variability in the model's pass rate on specific bench- mark problems across different checkpoints. We note such variance different from training insta- bility (Wortsman et al. 2023) as we observe a stable decrease in the training loss. This variability persists even in the later stages of training, despite using low learning rate. We illustrate this vari- ability in Figure Ta The scatter plot tracks the pass rate for each problem in VerilogEval-Humaan with each point representing the pass rate for the same problem across two checkpoints. The size of each point indicates the number of problems with the same pass rates for the two model checkpoints. We further categorize the region into areas where the checkpoints agree on problem difficulty and areas where they do not.
Alarmingly, we find that nearly 15% of the problems show significant discrepancies between these two checkpoints, with equal number of problems demonstrating improvement and degradation. Our detailed analysis of the sampled code completions for such problems when pass rate degrades suggests that the model is generally on the right track but makes "minor" errors that small, detailed, and seemingly trivial. While is possible that LLMs experience catastrophic forgetting during fine-tuning (Luo et al.) 2024a). we do not anticipate this being a major factor due to the low learning rate and the small number of gradient updates (64 steps with 16k data samples). Instead, we believe the primary issue is our inability to ensure the quality of our data, partieularly in verifying whether the sampled code solutions correctly solve the code problems.
3 IMPROVING VERILOG CODING WITH DOBRECT-BY-CONSTRUCII NON-TEXTUAL REPRESENTATIONS AND TARGETED CODE REPAIR
Based on our detailed analysis of the limitations of relying solely on LLMs for generating syn- thetic data, we focus our data curation efforts address these shorteomings. Our goal is to enhance data quality and ensure the correctness of solutions for the generated problems. We have found that current LLMs often lack the capability understand and process non-textual representations effectively and are unable to generate satisfactory testbenches for assessing solution quality. Con- sequently, rather than depending entirely LLMS resolve data quality issues, we instead create targeted fine-tuning data to mitigate these problems.
3.1 ENSURING QUALITY THROUGH
We generate Verilog code problems and solutions that are correct-by boeet--oyyooototioonnn Our focus is creating problems and solutions for non-textual representations. Table shows the quantity of our correct-by-construccte generation data (referred to as CC). To prevent data contamination, we exclude entries that duplicate the data representations of benchmark problems.
4

---

Published as a conference paper at ICLR 2025
Karnaugh Maps and Truth Tables (KMap) We start by sam- pling random configurations, which include selecting the number of variables and their names. After determining the number of vari- ables, we randomly choose valid minterms and don't t-cares. For variables, there are 2n possible states, and each state be signed one of three values (0, I, or x), leading to 3 possible com- binations of minterms and don't-care t-cares. From these minterms, we derive the sum-of- produucssss (SOP) form to represent the Boolean logic. We then create Truth Tables and Karnaugh Maps based on the chosen minterms and don' tonrrss In the KMap, Gray encoding is used default for the row and column sequences to ensure that only single bit changes between adjacent cells. Additionally, we apply modifications by transposing the map and randomly swapping adjacent rows columns. We randomly sample from n = {3, 4} variables.
Table 3: Data quantity CC.
Method
Quantity
KMup
12.5k
FSM
8.0k
Waveforms
8.0k
CC Total
28.5k
State Transition Graphs and Tables (FSM) We construct problems for finite-state machines (FSMs) with state-transition representations with similar approach to K.aass. We begin by sam- pling random configurations, including the number of states (e.g.. 4, 6, or 10) and the bit width of the input (e.g., I or 2). We then create the transition graph, ensuring that it is both meaningful and legally defined. We generate state-transition graphs for both Moore and Mealy state machines. From these graphs, we produce edge-list and transition table representations. Finally, we construct the Verilog code to implement the logic for state transitions and output assignments.
Algorithm D outlines the process for generating Moore FSM with random transitions. State reacha- bility is ensured by first construct- ing a tree. Legality for state tran- sition ensured by ensuring each node has an out-degree of 2d with the input bit width of w. The re- sult is an FSM where transitions between states are randomly as- signed but conform to the speci- fied input bit width. The algorithm can be easily modified for Mealy FSM by assigning the output to the edges rather than nodes.
Algorithm Generate transition graph for Moore FSM.
Input: Number of states n, bit width of input Output: FSM graph with transitions and states Initialize the number of states n and bit width of input w Randomly generate tree with nodes Define the root the tree as the reset state for each node in the tree do
Assign a unique state to the node Assign output to the node end for for each node in the tree do
Add additional transition edges to form graph Ensure that each node has an out-degree of 2 end for
Figure 2 illustrates our approach for generating state transition logic in Verilog from state-transition graph. Our method predominantly employs an out- edge focused strategy for state transitions. Addi- tionally, we incorporate in-edge focused transition logic address specific challenges encountered in benchmark problems. These benchmarks often in- volve states represented using one-hot encoding and require rigorous testing of non-default states.
out-edge
DLD
=in
A9
endoaase
G11
state|]]
inj [state(C]
Figure 2: State transition logic.
Waveforms We utilize correct-by-cone currectby-coonstuuctio code solutions for both KMaps and FSMs. Be- cause these codes are generated using similar templates, designing corresponding testbenches is straightforward. We simulate the generated code to produce waveform Value Change Dump (VCD) files. These VCD files are then parsed and converted into waveform representations. Our approach covers KMaps as combinational circuits and FSMs as sequential circuit waveforms.
3.2 MITIGATING "MINOR" ERRORS WITH TARGETED CODE REPAIR
Our analysis revealed that the models generally the right track to correct solutions but were making minor errors-sman detailed, and seemingly trivial. Unlike complex, unsolvable problems, these minor errors could be easily corrected by language models. This insight led us
5

---

Published as a conference paper at ICLR 2025
LLL Generated Error Report
Pronge Broor Analysis
Crapert: Conniraatii wires
Valarta Training Trajectery
Mes dus le de LSB dicer pridiies nur in
tays in Bry de Preeeee tauh eeter
New gevmera
D. mita rosken brem ruserioe curter mune
Targsted Cude Repair Training Dats
Prompt for Error Injection
Vidkkanel Error Report
Re-gosenate Farve Repeet
You grana Vertog vocule cemcononsheiie in the of
Teur god proeutt conndia Veriiee
Code Snippet
le and crur al cppat priblits
Clous atdlin Soviding actuss rightar corncty
Self-Censitency
Connect Cede Salution
enrerr roport
Figure 3: Overview of our approach for generating targeted code repair data: (1) prompting the LLM to generate detailed error reports from correct and erroneous code, (2) validating error report quality by ensuring the LLM can debug the errors based on the report, and (3) leveraging the LLM inject similar errors into open-source code, creating diverse training dataset.
to develop detailed error reports on benchmark problems, re-creating these errors on correct open-source code, and conducting rigorous validation ensure quality. We remootron443300-insttme to construct our targeted Repair data. We generated 847 error reports across the three benchmarks and produced 2,736 data samples. After filtering, this resulted final set of 1,406 targeted code repair data points.
strategy centered targeted error code repair. Our approach includes creating
as the LLM
Error Report Construction To systematically address the issue, we first created a comprehen- sive Error Report for benchmark problems using LLMs. targeting those with significant pass rate fluctuations across training checkpoints for models on SDG data. We prompt the LLM to examine the nature of the mistakes by comparing correct and erroneous code completions for each problem, categorizing the errors into common error types (details in Appendix A.9). This detailed report not only categorizes the eπrors but also highlights areas where the model consistently underperforms.
Targeted Code Repair Dataset Building on the error report, we further develop a targeted code repair dataset to address these common errors. This dataset is constructed using two main sources: the errors identified in the Error Report and correct code snippets gathered from open-source repos- itories. We introduced the identified errors into correct code snippets create repair problems, which include a problem description, erroneous code implementation, and hints about the nature of the error and how fix it. This targeted strategy enables the model to learn how to avoid common errors and generate improved code completions, thereby enhancing model accuracy.
Quality Assurance with LLM Validation To ensure the reliability of the error report and the code repair dataset, we implemented a two-phase validation process with LLMss. In the first phase, we conducted a self-consistency check of the Error Report by having the language model attempt the fix error code based on the report's hints. This step verifies the accuracy of the report by confirming that the model can resolve the errors using the provided guidance, whereas directly prompting the LLM without detailed егror reports could resolve only 13% of the errors. In the second phase, during the generation of the code repair dataset, we apply serffieriiication, including deduplication, syntax filtering, and benchmark decontamination. These measures ensure the dataset's quality and uniqueness, preventing overlap with evaluation benchmarks.
4 EXPERIMENT
4.1 IMPLEMENTATION DETAILS
Training Data Our fine-tuning training data is comprised of 80.1k LLM synthetic generated data using various prompting methods as described in Section 2.J] 28.5k data samples generated correct- by-construction aimed non-textual representations detailed in Section 3.1 and 1.4k carefully fil-

---

Published as a conference paper at ICLR 2025
Table 4: We compare our models with various baseline models on VerilogEval (Liu al. 2023b
We update the results from Zhao et al. (2024) with the latest foundational and frontier code models. The best results are highlighted in bold.
VerilogEival
Type
Model
Size
Machine (%6
Fruman ((9)
pass
pass
Llama-3.1
8B
48.7
67.3
74.1
26.9
37,8
44.2
Foundational Models
Llama-3.1
4058
67.3
75.1
76.9
62.8
Nemotron-4
340B
53.0
60.3
62.2
43.1
48.3
50.0
GPT-3,5-turb
58.0
74.0
77.6
31.2
44,1
47.4
GPT-40
71.4
72.7
Cadel
43.1
47.1
47.7
22.7
CudeQwen
7B
46.5
54.9
56.4
22.5
26.1
28.0
Code Models
Starcoder2
DespSeck-Coder
6.7B
52.2
55.4
56.8
30.2
33.9
34,9
DeepScek-Codore V2
67.4
78.3
DeepSeck- CeepSeek-CCder-... V2
236B
68.2
74.1
76.2
56.4
622.2
66.0
RTLCoder
Mistral
7B
62.5
72.2
76.6
36.7
45.5
49.2
Liu 2025c
DoepSeck-Cder
7B
61.2
76.5
81.8
41.6
50.1
53.4
BetterV Pei 2024
CadeLlama
64.2
75.4
79.1
40.9
30.0
53.3
DaspScck-Coder
6.7B
67.8
79.1
84.0
45.9
53.3
57.6
CodeQwen
68.1
79.4
84.5
46.1
53.7
CodeV Zhao al., 2024
DeepSeek Codeer
Codel
7B
77.9
78.1
86.0
888.5
45.2
59.5
63.8
CodeQwen
7B
77.6
88.2
90.7
53.2
65.1
68.5
OnCinn (Cui al. 2024
DeepSeek
74.1
87.8
Ours SDG-CC-Repair
DeepSeck-Coder
6,7B
77.8
85.5
88.1
65.4
70.0
72.1
Starcoder2
81,9
86.9
88.1
tered data for targeted code repair as outlined in Section 3.2 and Repair, respectively.
We refer to each data set as SDG, CC,
Following prior work, we use CodeLlama- 7 (Roziere et al. 2023) (Guo et al. 2024) as the base model, formatting our data accord- ing to their default chat prompt templates. Additionally, we explore the Starcoder2-15B (Lozhkov et al. 2024) model in our experiments.
Pretrained Models
and Coreeee-Coodr--.77---ee
Model Training Training is conducted with 32 NVIDIA A100-80GB GPUs through the Dis- tributed Data Parallel (DDP) module from Py Porcc.. We set the learning rate at 5e-5 for Codel and DeepSeek-Codee mizer with full parameter updates and truncate sequence lengths longer than 4096 tokens. We used a batch size of 256 samples. We fine-tune models for epoch using standard cross entropy loss on the response tokens (while masking loss prompt tokens).
and lc-5 for Starcoder2. We use Adam (Kingma & Bal 2017) our opti-
Model Inference We use vLLM (Kwon et al. 2025 where the inference engine is up with 6 dtype, tensor parallel size of 8, and a maximum token limit 4096. We sample each problem 20 times. We report the best results from different temperatures 0.2 and 0.8, consistent with prior work (Liu et al. 2023c Zhao el al. 2024
4.2 EVALUATION METRIC AND BENCHMARK
Evaluation Metric Following prior work (Chen al. 2021 Liu al. 2023a), for each experiment we use the unbiased pass @k metric to measure the Verilog generation accuracy. The pass@k metric estimates the proportion of problems that can be solved least once in attempts:
pass@k = Prroblems
where 2 i represents the total number of trials for each problem, and represents the number of trials that pass the functional check.
VerilogEval (Liu et al., 2023b) contains two subsets of problems, where KarlogEval-Humane tains manually converted problem descriptions from the original HDLBits website, and VerilogEval- Machine with GPT-3.5 generated problem descriptions.

---

Published as a conference paper at ICLR 2025
Table 5: Evaluations on RTLLM v1.1 (L et al. 2024) using unbiased pass@k metries. The best
results are highlighted in bold. We re- re-evaluate all models (see Appendix A for details).
RILLM vI.I [Lu al 2024
Type
Model
Syntax
(%5)
p46s48
pass@
passia [0
pass
813
pass @ TO
Llama-3.1
60.6
37.9
Foundational
Llama-3.1
405B
56.5
64.4
72.4
38 9
45.8
51.7
Models
Nemotron-4
340R
41.7
47.2
48.3
18.9
20.7
20.7
G.rro.5.turoo
50.3
61.2
65.5
28.3
36.9
41.4
GPT-4o
50.3
59.9
62.1
33.8
44.4
48.3
Cade
7B
46.6
62.6
68.9
17.9
29.9
34.5
CodeQwean
45.8
65.8
72.4
24.1
34,0
37.9
Code
Stareoder2
158
38.3
81.0
94,7
15.5
37.6
45.7
DeepSeek-Coth
23.1
29.3
DeepSeek Coder V2
16B
$1.4
57.8
58.6
33.1
37.1
37.9
DeepSokkCooud Co
78.1
34.5
50.2
RTLCoder [[.u et al. 2023c
Mistral
73.7
24.5
42.3
DeepSeek Coder
6.7B
73.4
83.9
86.2
35.8
40.3
43.1
CodeV Zhan al. 2024
DeepSeek Coder
Codel
6.7B
75
78.3
87.4
89.2
89.1
42.4
39.4
50.3
$1.5
53.1
53.2
CodeQwen
7B
78.8
89.5
92.4
36.6
53.3
61.3
Oniiie (Cui et M044
DeepSeek Coder Coder
6.78
65.3
Codee lamn
85.7
93.9
94.8
42.6
52.9
58.2
SDG-CC- Repair
Ours
DeepSeek Coder
6.7B
84.3
92.9
95.4
53.1
58.8
62.6
Starvoder2
15B
79.8
93.9
96.2
49.0
65.8
74.5
RTLLM (Lu et al., 2024) is an open-source benchmark designed for generating Register Transfer Level (RTL) code from natural language instructions. It evaluates models on syntax correctness, functional correctness, and design quality, offering a thorough analysis of model outputs.
4.3 RESULTS
Main Results Table and Table 5 compare our models with baselines on VerilogEval and RTLLM. We mainly source baseline results from Zhao al. (2024). For RTLLM we found large variance with biased pass@5, thus we re-evalaute all models and report unbiased pass@k @k metric. We further rigorously evaluate the latest foundational and frontier code models, including Llama-3.1 (Dubey et al., 2024). DeepSeek-Coder- cent foundational and frontier code models already reached competitive performance compared to previous efforts targeting Verilog code generation.
(DeepSeek-A) et al. 2024), and GPT-4o. Re-
Compared previous approaches like CodeV (Zhao al. 2024), our models achieve compara- ble performance on VerilogEval-Maching and show significant improvements on benchmarks with human-like descriptions. Machine descriptions often provide detailed, line-by-line coding instruc- tions, whereas human descriptions are high-level, integrating problem-solving skills and a deeper understanding the hardware module's functionality. Enhancing the model's ability to handle human-like descriptions is crucial, these more accurately reflect how designers interact with the models and set expectations for Verilog generation. Our fine-tuned Starcoder2-1 15B surpasses previ- tfte-offthee-rr results by 3.8%, 10.9%, and 6.6% in pass @ metrics KarlooEval-MMaching HaalooEval-Humaaee and RTLLM, respectively.
Table 6 highlights the effectiveness of our generated data fine-tuned on Starcoder2-155. Our CC data en- hances the model's ability to handle non-textual repre- sentations, leading to improved scores on VerilogEval- Human. Our targeted code Repair data boosts per- formance across all benchmarks, suggesting that the model has learned to generalize from code repair tasks and reduce similar errors during code completion.
Table 6: Ablation study on training data. Data quantity indicated in parentheses.
Verilog
RTLLM
Model
Michine Human
Fan:
pass & (%)
passGi5 (%)
Stumedder2.11
37.6
SIDGTRO.1E
75.2
31.7
62.1
SDG-CC (108 6k)
73.9
62.0
67.8
SDC--CC-R (1)
81.9
68.9
65.8
Improved Variability During Training Figure Ib displays the pass rates for two consecutive on VerilagEEval-umman problems, sampled with tem- perature of 0.8. Compared to Figure Ia the updated model shows significant improvements by (1) moving previously unsolved problems into the solved category, including those with non-textual representations addressed by our correct-b--construcce CC data, and (2) reducing the number of problems with large pass rate discrepancies, particularly where performance had degraded. The tar-
checkpoints of Starcoder2-SDD--CC-eRe
8

---

Published as a conference paper at ICLR 2025
geted repair data has effectively mitigated the model's tendency repeat common mistakes found in our Repair dataset, despite the noise inherent in synthetically generated SDG data.
Scaling Data for Non-textual Representa- tions Figure 4 illustrates the scaling of correct- by-construction (CC) data and the fine-tuned Starcoder2-15B pass rate on problems involving non-textual representations. We expanded our test- ing to include strictly in-distribution test set, with each category containing around 50 problems. The results show that the model can quickly learn and comprehend these non-textual representations with as few as 4k training data samples, with the pass rate steadily improving as more data is provided. Additionally, the model demonstrates the ability to generalize to VerilogEvallNon Text benchmark problems. While our models achieve near-perfect scores on KMap and FSM problems, they perform less effectively on Waveforms, suggesting that reverse engineering circuits from waveforms pose a greater challenge.
0.955
0.665
0.986
1.00
0.846
(193)
0.001
0.799
02510261
0.68
0.210
0771
pass
0.6
Avi
0.57%
0,393
10639
0,68
0.77g
0.4
KMtp FSM
0.2
0,050
VerilogEsall Noatest
00000
30
Total Number of Targeted Data Samples (k)
Figure 4: pass@1 on non-textual problems with total number of CC data with tempera- ture 0.8.
Ensuring Quality for Targeted Code Repair The quality control mechanisms integrated into the data generation pipeline are crucial for improving model performance, particularly in correcting minor errors through targeted code repair. To evaluate the impact of these quality controls, we conducted an ablation study in Table 7 where we systematically removed each component of the targeted code repair generation pipeline and assessed the resulting model performance. Specifically, we eliminated the self- self-consistency checks that validate whether the gen- erated error report effectively guides the I.LMS in correcting mistakes. Additionally, we tested the removal of the error report entirely, substituting it with random errors injected into the open-source code by the LLMs. The benchmark results indicate a significant performance drop when these vali- dation processes are excluded. These findings highlight the essential role of both the self-consistency checks and the targeted error report in improving the model's ability to correct errors.
Table 7: Ablation study on Repair data qual- ity with 255.
VerilogEval
RILLM v1.1
Model
Machine
Human
Func
pass GP (%)
pas (%)
SDG-CC
73.9
62.8
SDG-CC-Repair
81.9
68.0
65.8
w/o self coo-sistteeccy
75.3
63.7
w/o error report
76.9
59.6
59.4
RELATED WORK
Synthetic Data Generation for Model Fine-tuning. The performance of large language models (LLMs) hinge on the quality and diversity of their training data. To address the limitations of manual datasets, synthetic data generation methods (Wang et al. [022 Xu et al., (023) have been developed to automatically create cnntruction-folowiinn examples from LLMs, reducing reliance human an- notations. Various techniques enhance data quality: Wang et al. (2022) generates multiple reasoning traces and selects the most frequent output to improve robustness, while other approaches (Light- man et al. 2003) Zhang et al.) 2024b) assess response quality based on these traces. Self-training methods utilize synthetic data for iterative fine-tuning, boosting reasoning capabilities (Singh al. 2023 Feng et al. (023).. These advancements show how synthetic data can effectively scale and optimize models through iterative feedback.
Large Language Models for Code Generation. Recent breakthroughs in large language models (LLMs) have greatly enhanced their capability to tackle complex code generation tasks. Much of the research focuses developing LLMs specialized for code by continuing their pretraining on
code data (Guo et al. 2024 Bai et al. 2023; Roziere et al.) 2023 DeepSeek-AA et al. 2024) from
open-source repositories like GitHub (Kocetkov al. [022 Lozhkov al.) 2024) and commit
histories (Mucnnighoff et al. 2023). Further improvements to these models come from reinforce-
ment learning (Le et al. 2022) and more often instruction fine-tuning, which involves techniques

---

Published as a conference paper at ICLR 2025
to address more complex coding problems (Luo et al. 2024b), increasing diversity with unlabeled open-source code (Wei et al.] 2023; Yu et al. 2024; Wu et al. 2024). ensuring solution correctness
through self- written tests (Chen et al. (2022). and validating and debugging code execution through
interactions with LLM agents (Lei et al. 2024).
Large Language Models for Verilog Coding. While most code LLMs target software languages, there is increasing interest in models for hardware description languages like Verilog, essential for chip design and verification (Liu et al.) [024). Previous work has addressed the challenge of limited data through various methods, including synthetic data generation (Liu et al. 2023c), multi-level summarization of open-source Verilog code (Zhao et al. 2024), and enhanced code augmentation with self-reflection based on compiler feedback (Tsai et al.) 2023 Cui al. 2024). Other ap-
proaches focus on improving functional correctness and circuit performance through Monte Carlo
Tree Search (DeLorenzo et al. 2024) and discriminator-guidded sampling (Pei et 2024
DISCUSSIONS
In this work, we refer to synthetic data generation as methods of using large language mod- els (LLMs) in data generation. While our appreach-ensuring correctness through correct-by- construction-cout also be considered "synthetic" and resembles methods explored in works like AlphaGeometry (Trinh al.] 2024), our problems are much simpler and on a smaller scale. Our observations about the variability of models on specific problems align with the findings of MMea Al 2024). where "the model knows how to produce the right answer, but does not know how se- lect ir." Instead of striving for absolute data correctness, preference learning (Rafailov et al.) [2024] Ethayarajh al. 2024) or reinforcement learning (Bai al., 2022 Le et al. (202)). we generate targeted repair data by analyzing errors and re-create such scenarios by injecting similar errors into open-source code, somewhat analogous to how humans consolidate memories during sleep by inte- grating new information with past experiences (Walker & Stickgold [2004] (Stickoold] 2005). Further discussions on the generalizability and broader impact our work are provided in Appendix B
7 CONCLUSION
This paper addresses key challenges in Verilog code generation with correct-by bynttboryctotttttionnn data generation and targeted code repair data strategies. We identified significant issues with synthetic data generation, including difficulties with non-textual representations and variability in perfor- mance during training across benchmarks. To address these challenges, we generated data that is correct- correct-by-constre bocoosstuuttion and create targeted repair data by injecting errors to open-source code. Our approach led to substantial improvements, with models fine-tuned using our methods achieving hate-offthe-ar results VerilogEval and RTLLM benchmarks. These advancements highlight the effectiveness of our strategies in enhancing model performance in Verilog code generation.
Reproducibility Statement We provide the following details: evaluation benchmarks in Ap- pendix A.3 examples of the process for generating targeted code repair data in Appendix C and data examples from forrect-by-constrcctt Additionally, we include prompt templates used for data generation in Appendix E Our data gener- ation pipeline is available: https: //ttthb//bttttt/// wtwpe//ww..ooo///aaaa//ttatttttt
targeting non-textual representations in Appendix D
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint vrXiv:2303.08774 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless
10

---

Published as a conference paper at ICLR 2025
assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Christopher Batten, Nathaniel Pinckney, Mingjie Liu, Haoxing Ren, and Brucek Khailany. Pyhdl- eval: An IIm evaluation framework for hardware design using python-embedded embedded dsls. In Pro-
ceedings of the 2024 ACM/IEEE International Symposium
Machine Learning for CAD,
MLCAD '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400706998. doi: 10.1145/567044443363594 3670474 3685948
URL
1145/
Jitendra Bhandari, Johann Knechtel, Ramesh Narayanaswamy, Siddharth Garg, and Ramesh Karri. Llm-aided testbench generation and bug detection for finite-state machines, 2024. URL https:
// wwtpw///.gabb///....... 17132
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to bench- marking neural code generation, 2022. URL
Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Anders Freeman, Car- olyn Jane Anderson, Molly Q Feldman, Michael Greenberg, Abhinav Jangda, and Arjun Guha. Knowledge transfer from high-resource low-resource programming languages for code Ilms, 2024.
Kaiyan Chang, Zhirong Chen, Yunhao Zhou, Wenlong Zhu, kun wang. Haobo Xu, Cangyuan Li, Mengdi Wang. Shengwen Liang, Huawei Li, Yinhe Han, and Ying Wang. Natural language not enough: Benchmarking multi-modal generative ai for verilog generation, 2024. URL https:
//6b1.org/1001111///775333
3676679
Sahil Chaudhary, Code alpaca: An instructiinn-ollooinng in Ilama model for code generation. 2023.
https:
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin. Jian Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests, 2022. RtLLhhtsss///rxiiiii abs/
2207. 10397
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint arXiv:2107.033744 2021.
Fan Cui, Chenyang Yin, Kexing Zhou, Youwei Xiao, Guangyu Sun, Qiang Xu, Qipeng Guo, Demin Song, Dahua Lin, Xingcheng Zhang, et al. Origen: Enhancing rtl code generation with code-to- code augmentation and self-reflection. arXiv preprint arXiv:2407. 16237,, 2024.
DeepSeek-AL, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng. Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseek-coder-vv. Breaking the barrier of closed-source models in code intelligence, 2024. BtLhttts:://aae .rg. abs 240 11931
Matthew DeLorenzo, Animesh Basak Chowdhury, Vasudev Gohil, Shailja Thakur, Ramesh Karri, Siddharth Garg, and Jeyavijayan Rajendran. Make every move count: based high- hugh-quality rtl code generation using mets, 2024. URL http://ss//aaaav..oo///////////0 03289
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong.
11

---

Published as a conference paper at ICLR 2025
Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruy Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, llyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden AI- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid EI-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Vantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng. Prajiwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He. Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennahasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho. Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani. Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute. Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- -aptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg. Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Vecraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia. Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manay Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson,
12

---

Published as a conference paper at ICLR 2025
Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang, Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin. Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero. Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li. Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL httphtt:ss //arxiv org/ abs /4007 21783
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Xidong Feng, Ziyu Wan, Muning Wen, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search guide large language model decoding and training. arXiv preprint arXiv:2309.17179. 2023.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401. 14/96, 2024.
René Just, Darioush Jalali, and Michael D. Ernst. Defects4j: a database of existing faults to enable controlled testing studies for java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437-440, New York, NY, USA, 2014. Asso-
ciation for Computing Machinery. ISBN 9781450326452. doi: 10.1145/2610384.222 26228855.... URL
https: /lg 211.111111111122288888808
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL
https://arxiv. / abs 1412. 6980
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jemite,, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code, 2022. URL 15533
https:://axxvv org. www.r/////////////////////eeee//
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoic.. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Hung Lc. Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022. URL https: wttp:xiwvvvgg.r///o//eoeeo
http://xxvvvvrrro hbtp:/bs/222000 22002000700
Bin Lei, Yuchen Li, and Qiuwu Chen. Autocoder: Enhancing code large language model with 14906
AIEV-INSTRUCT, 2024. URL https://arxxv..rr////aaa
13

---

Published as a conference paper at ICLR 2025
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang. Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zehaze, Ming-Ho Yee, Lo- gesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luc- cioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you!, 2023. httph:ttt::/aateeeee
org/ abs absaaaa//../ 2305. 06161
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305..20500 2023.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a. URL https:
//openzeview wttp:/ooouu...-1111111.....
Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code generation. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pp. 1-8. IEEE, 2023b.
Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongiann Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Ankit Jindal, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Ghasem Pasandi, Sreedhar Pratty, Jonathan Raiman, Ambar Sarkar, Zhengjiang Shao, Hanfci Sun, Pratik P Suthar, Varun Tej. Walker Turner, Kaizhe Xu, and Haoxing Ren. Chipnemo: Nomain-adapter 1lms for chip design, 2024. URL https::/ arxiiv org/ abs/ 231 00176
/aapss///axxvv..
Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. Rtlcoder: Outper- forming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution. arXiv preprint 3rXiv:2312.08611 2023c.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poiries Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, al. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173 2024.
Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rıllm: An open-source benchmark for design rtl generation with large language model. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 722-727. JEEE, 2024.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, 2024a. URL. https: org/ abs 2308 ..000 08747
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024b. URL
net
Meta AI. Introducing meta llama 3: The most capable openly available 1lm to date, 2024. URL /et.///eaaaa 11ama-3 Accessed: 2024-09-10.
https: //ai .etta
14

---

Published as a conference paper at ICLR 2025
Niklas Muennighoff. Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruc- tion tuning code large language models. arXiv preprint 2:Xiv:2308.07124. 2023.
Daniel Nichols, Joshua H. Davis, Zhaojun Xie, Arjun Rajaram, and Abhinav Bhatele. Can large language models write parallel code? In Proceedings of the 33rd International Symposium on Prrf-Peeformance Parallel and Distributed Computing, HPDC '24, pp. 181-,94, New York, NY. USA, 2024. Association for Computing Machinery. ISBN 9798400704130. 3625549.3658689 URL
doi: 10.1145/ ww..z///...1111/111 1111116335555555 3658689
Nvidia, :. Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Alek- sander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grze- gorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long. Ameya Sumil Mahabaleshwarkar, Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupin- der Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping. Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Se- wall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shub- ham Toshniwal, Hao Wang, Zhilin Wang, Jiaxuan You. Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https://erxxie http:///rxxvvv.r//eeee. httgg//bo/240555 11704
Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. Betterv: Controlled verilog generation with discriminative guidance. arXiv preprint arXiv:2402.03333 03375, 2024.
Ruidi Qiu, Grace Li Zhang, Rolf Drechsler, Ulf Schlichtmann, and Bing Li. Autobench: Automatic testbench generation and evaluation using 11ms for hdl design, 2024. URL httlhtttss//aaa
httg/bb//4000 03891
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. 1183308..99000 12950, 2023.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jachoon Lee, Kelvin Xu, Aaron Parisi, al. Beyond human data: Scaling self-training for soovvlmm-oovvigggg with language models. arXiv preprint arXiv:2312.06588 2023.
Yewei Song, Cedric Lothritz, Daniel Tang, Tegawendé F, Bissyandé, and Jacques Klein. Revisiting code similarity evaluation with abstract syntax tree edit distance, 2024. URL https:t/:///r
org/abs /00ps/2404.m 08817
Robert Stickgold. Sleep- deeeddeeendent memory consolidation. Nature, 4372277633 2005.
Shivehander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David Cox, and Akash Srívastava. Lab: Large-scale alignment for chatbots, 2024. URL htttps://arxvv.ot
abs/ abs/2403. 220.20103...1111
Shinya Takamaeda- Yamazaki. Pyverilog: A pathon-based hardware design processing toolkit for verilog hdl. In Applied Reconfigurable Computing, volume 9040 of Lec- nure Notes in Computer Science, pp. 451-460. Springer International Publishing, Apr
2015. doi: 10.1007/978-3-319-1622 42. URL http://dx doi. ott..o.00////11000..77 378-3-319-1622
15

---

Published as a conference paper at ICLR 2025
Ali TehraniJamsaz, Arijit Bhattacharjee, Le Chen, Nesreen K. Ahmed, Amir Yazdanbakhsh, and Ali Jannesari. Coderosetta: Pushing the boundaries of unsupervised code translation for parallel programming, 2024.
hbtp://b//2// 20527
Trieu H Trinh, Yuhuai Wu, Quoe V Le, He He, and Thang Luong. Solving olympiad geometry
without human demonstrations. Nature, 625(7995):476-482, 2024.
YunDa Tsai, Mingjie Liu, and Haoxing Ren. Rtlfixer: Automatically fixing rtl syntax errors with large language models. arXiv preprint arXiv:2311.1654; 2023.
Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Trans. Softw. Eng. Methodol,, 28(4), September 2019. ISSN 1049-331X. doi:
10.1145/3340544, URL httthtttp///o: /ttp://////ol.or00001111/////// 11444441114/334044444
Matthew P.
Walker and Robert Stickgold. Sleep-dependent learning and memory consolida- tion. Neuron, 44(1):121-133, 2004. ISSN 0896-6273. doi: http:///bo.org//0.0006//...uu
2004.08.031. URL https:: /www www.sciienccic rect. com/ wwtm///ccienc///araicle//////// 5089662730400544
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generate instructions. arXiv preprint urXiv:2212. 10000000,, 2022.
Yuxiang Wei, Zhe Wang. Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Source code is all you need. arXiv preprint arXiv:2312.02120, 2023.
Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification, 2023. URL https: 09561
sarxiv.org/abs.
Mitchell Wortsman, Peter J. Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John D. Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, Jeffrey Pennington, Jascha Sohl-dicks,ein, Kelvin Xu, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Small-scale proxies for large-scale transformer training instabilities, 2023. URL httlthttt://arttti .rg. htt.:/bb.///3/////// 14322
Yutong Wu, Di Huang, Wenxuan Shi, Wei Wang, Lingzhe Gao, Shihao Liu, Ziyuan Nan, Kaizhao Yuan, Rui Zhang. Xishan Zhang, Zidong Du, Qi Guo, Yewen Pu, Dawei Yin, Xing Hu, and Yunji Chen. Inversecoder: Unleashing the power of instruction-tune code Ilms with inverse-instruct, 2024. URL/https://aret www.aaawwwwrr.aaaaaaaaaaaarrrere abs
Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). pp. 1482-1494, 2023. doi: 10. 10.109//SE448812222..00000
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.1 /2244, 2023.
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning, 2024. URL ltttt:::::///at //axxiv wttg/ass///33
14187
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets Ilm finetuning: The effect of data, model and finetuning method. arXiv preprint arXiv:2402.17193 2024a.
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*: Llm self- training via process reward guided tree search. arXiv preprint arXiv:2406.03816, 2024b.
Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, and Zhi Jin. Codedpo: Aligning code models with self generated and verified source code, 2024c. URL
2410.
Yang Zhao, Di Huang, Chongxiao Li, pengwei Jin, Ziyuan Nan, Tianyun Ma, Lei Qi, Yansong Pan, Zhenxing Zhang, Rui Zhang, et al. Codev: Empowering Ilms for verilog generation through multi-level summarization. arXiv preprint arXiv:2407. 10424, 2024.
16

---

Published as a conference paper at ICLR 2025
A DETAILED RESULTS
A.1 OUR MODELS
We present our models' results on Verilog benchmarks tested with temperatures 0.2 and 0.8. We ablate across different data blends, with SDG indicating using LLM synthetic generated data in Sec-
tion 2.1 CC indicating correct-y--oonsrrcctt data targeting non-textual representations in Sec- tion and Repair representing targeted code repair dataset in Section 3.2
Our results for RTLLM use the open-source learus Verilog simulator to check syntax and functional pass rates. This might lead to lower pass rate scores compared to previous work that used Synopsys VCS, as Icarus Verilog does not support all syntax.
Table 8: Results for our models, across different dataset and temperature on VerilogEval.
VerilogEval
Liu
20236
Model
Temperature
(%)
pase
pass
pass
0.2
SDG
0.8
73.7
84.0
86.1
47.4
61.9
64.8
0.2
73.9
78.
79.5
62.0
65.6
67.0
Stareoder2-15h
SDG-CC
0.8
72.9
84.1
87.1
58.5
70.3
73.7
SDG-CC-Repair
0.2
81.9
84.2
85.0
68.0
71.7
72.0
0.8
78.1
86.9
88.1
64.1
72.4
74.6
0.2
73.4
77.8
78.9
48.3
53.2
54.5
SDG
0.8
71.4
44.0
58.1
DeepScol-67b-Insttuuc 7b-Instruct
SDG-CC
0.8
70.2
83.1
85.4
56.3
20.7
SDG-CC-Repair
0.2
77.8
82.7
83.4
65.4
67.7
68.2
0.8
75.2
85.5
88.1
61.6
70.0
72.1
0.2
74.5
77.9
78.8
45.3
50.3
51.5
0.8
71.2
82.6
85.1
42.6
55.6
59.0
0.2
74.2
77.4
78:1
55.1
61.0
62.4
Codel Jamu-7b-maa Instract
SDG-CC
0.8
70.0
81.2
83.7
51.6
64.4
67.7
SDG-CC-Repai
Table 9: Results for our models, across different dataset and temperature on RTLLM.
RILLM vl.
Lu
2024
Model
Dataset
Temperature
(%)
Func: (%)
pa
passea
6ass
pass
pass
78.1
86.5
90.1
49.0
60.4
66.3
SDG
77.1
89.0
94.1
43.8
62.1
68.0
Starender2-15h
SDG-CC
0.2
78.3
89.3
92.7
45.5
58.3
62.0
0.8
76.9
92.6
95.5
38.4
62.8
70.4
SDG-CC-Repair
0.2
79.8
87.9
90.5
49.0
59.1
62.6
0.8
79.3
93.9
96.2
45.3
65.8
74.5
0.2
79.3
86.8
90.5
40.3
45.9
49.6
SDG
0.8
76.6
92.5
96.2
40.0
53.8
63.6
DeepSeek-6. Th-Instruct
SDG-CC
73.6
84.55
86.0
44.3
52.2
54.3
76.7
90.5
93.8
39.5
56.4
63.1
92.2
93.0
53.1
58.8
60.3
SDG-CC Repair
33.9
SDG
CodeL 7b Instruct
SDG-CC
93.9
47..6
52.7
85.7
93.9
94.8
42.6
49.4
51.2
SDG-CC- Re-Rair
80.3
93.9
94.8
36.9
52.9
58.2
17

---

Published as a conference paper at ICLR 2025
A.2 FOUNDATIONAL AND FRONTIER CODE MODELS
We present detailed results on recent foundational and frontier code models. We also re-evaluate all models RTLLM using unbiased pass @k metric.
Table 10: Results on foundational and code models on VerilogEval.
VerilogEval
(Liu al. 2023b
Type
Model
Size
Temp
Machine (%)
Iuman (S)
pase
pass pass GR5
pass
pass
pass00.5
pass
0.2
48.7
66.2
70.6
26.9
36.9
Llama-3.1
8B
0.8
42.1
67.3
74.1
23.0
37.8
44,2
0.2
66.7
73.8
76.9
48.7
53.6
55.1
Llama-3.1
70B
0.8
64.5
77.7
80.4
48.0
57.0
60.9
0.2
67.3
72.8
74.1
51.9
57.0
58.9
Llama-3.1
405B
66.4
75.1
76.9
53.8
61.0
62.8
Foundational
0.2
53.0
59.1
61.5
43.1
43.9
44.9
Nemotron-4
340B
Models
0.8
50.8
60.3
62.2
40.8
48.3
50.0
0.2
58.0
66.4
31.2
39.4
41.7
GPT.3.5-turbo
56.6
74.0
77.6
28.9
44.1
47.4
53.2
43.5
GPT-4
53.4
35.2
53.4
57.8
66.7
54.1
61.2
tur---turbo
GPT-4o
62.9
71.4
55.4
63.9
68.7
76.7
37.7
48.3
Starcoder2
15B
57.7
82.3
29.1
50.6
Code
53.3
DespSeck-Code-V2
16B
Model
72.7
56.4
DeepSeek-Coders
236B
Table 11: Results on foundational and code models on RTLLM.
RTLLM v1.1
(Luet al.
2024
Typo
Model
Size
Temp
Syntax (%)
Func. (%)
pass
@5
pass@
pass@5
pass@
0.2
39.7
53.1
55.2
19.3
25.8
27.6
Llama-3.1
0.8
40.7
60.6
65.5
17.6
34.7
37.9
0.2
47.9
51.7
55.2
34.1
34.5
34.5
Llama-3.1
70B
0.8
48.9
57.6
58.6
29.6
31.0
31.0
Ilama-3.1
0.2
56.5
63.9
65.5
38.9
45.0
48.3
0.8
52.1
64.4
72.4
35.8
45.8
0.2
41.7
47.2
48.3
14.1
15.5
17.2
Nemotron-4
Foundational
0.8
41.7
46.3
48.3
18.9
20.7
20.7
Models
GPT.3.5-tubo
0.2
50.3
58.2
58.6
28.3
36.9
41.4
0.8
48.2
61.2
65.5
24.1
36.9
41
0.2
49.3
65.9
68.9
30.0
44.4
48.3
GPT-4
0.8
42.8
61.2
65.5
25.9
40.0
44.8
0.2
38.9
44.8
48.3
27.2
35.1
37.9
GPT-4-turoo
0.8
40.3
48.8
51.7
27.5
40.2
44.8
GPT-44
0.2
50.3
59.9
62.1
33.8
44.4
48.3
0.8
47.5
63.2
66.7
31.3
44.1
48.3
17.9
CodeLlama
7B
CodoQwen
7B
34.0
Code Models
Starcoder2
15B
DeepSeek Coder
6.7B
64.4
21.0
29.3
51.4
51.7
34.5
DeepSeel-Coder.V>
16R
51.4
57.8
58.6
30.0
37.1
37.9
63.4
73.0
79.3
34.5
44.9
52.9
DeepSeek-Cooe Coder V2
236B
61.8
78.1
79.3
32.9
50.2
18

---

Published as a conference paper at ICLR 2025
A.3 DETAILS ON EVALUATIONS
We format the prompt input as follows for VerilogEval, where the detail deecridddsciittion is the problem description (Machine Human) and prompi field the problem module header. We include module headers to avoid confusion on the signals naming.
prompt ["taakk[ 'detail description' h.teriioonnssiippppee sttrisst)nnnnnttttt
prompt'] .strip())" "
An example of mux2tol in VerilonEEvllHHuman:
Create 2-1 multiplexer. When sel=0, choose a. When sel=1, choose b.
module dop_module (
input a, input input sel, output out
We use similar templates for RTLLM v1.1, where we extract the top module header from the refer- ence solution and provide it as input. Below is an example of celder_sbit
Please act professional verilog designer.
Implement a module of an 8-bit adder with multiple bit-level adders in combinational logic.
Module name:
adder Sbit
Input ports:
a[7:0]: : 8 bit input operand A.
b[7:0]: 8 bit input operand B.
cin: Carry-in input.
Output ports:
sum[7:0]: 8-bit output representing the sum of and B.
cout: Carry-out output.
Implementation
The module utilizes a series of bit level adders (full adders) to perform the addition operation.
Give me complete code.
module adder 8bit( input [7:0] b, input cin, output [7:0] sum,
output cout);
We use default chat templates and default system prompts for open-source models tested. For GPT models from OpenAl, we use the following system prompt:
Please act as a professional verilog designer.
We post-process model responses extract code. We extract content enclosed by triple backticks and remove the language identifier (Verilog). We then extract code enclosed module and endmodule keywords with response. find 'mmouule) and response. rfind 'ennmodule" If the extracted code does not include module header, the reference solution's 's module header will be prepended. The code is then tested with the provided testbenches with the Icarus Verilog (iverilog) simulator to evaluate for syntax and functional correctness. This might lead to lower pass rate scores for RTLLM compared previous work that used Synopsys VCS, as Icarus Verilog does not support all syntax.
19

---

Published as a conference paper at ICLR 2025
A.4 VERILOGEVAL-NONTEXT
We select the following 45 problems from VerilogEval-Human that consists of non-textual represen- tations in their problem descriptions:
nolatches, circuitl, circuit10. circuit2, circuit3, circuit4, circuit5, circuit6, circuit7, circuit8, cir- cuit9, ece241_200 2001__203_gi_ cce241_2014_00 ece241_2014
20,2_q1g, 2012_q2b, 2012_q2fsm, 2013_ 2013_q2afsm, 2014_q3bfsn 2014_qIe
fsml, fsmls, fsm2, fsm2s, fsm3, fsm3comb, fsm3onehot, fsm3s, fsmm_neehot, fsm_ps2data, kmapl, kmap2, kmap3, kmap4, m2014_q3. m2014_ _6. m2014_qói m2014_ga _600_ mr2015_gt m12015 _4a, mt2015_q4h _4b, re- view2015 fsmonehot, few20i5_fsmonchor, rulel10, rule90, truthtablel
A.5 TEMPLATE PROBLEMS FOR
DATA
When generating surect-b--constructe CC data, we select 11 problems from VerilogEval- NonText to use as representative templates for constructing our prompts. To prevent contamination, ensure that benchmark problems are excluded from our data. While our prompts will resem- ble those of the selected problems, the non-textual representations and solutions will differ. Addi- tionally, to prevent overfitting to specific prompt templates, we use LLMs to rewrite the problem instructions for 20% of our data. Furthermore, we create validation test problems that are strictly in-distribution, based on the chosen problems.
Karnaugh Maps and Truth Tables: kmapl, m2014_g.. truthtablel.
State Transition Graphs and Tables: 2012_q2b, 2014 2014_q3c, ece241 cce241_2014__1 2014__50bb fsm3comb, fsm3onehot, onehot, m2014 46bbb m2014_g0p 96c. f
Waveforms: We do not base our data on any benchmark problems specifically.
A.6 SCALING REPAIR DATA
Table 12: Scaling Repair data.
carefully filtered dataset of 1.4k samples achieves comparable performance to a 7.8k dataset. This suggests that merely increasing the dataset size by injecting the same types of er- rors does contribute meaningfully to improving model performance.
As shown in Table 12
VerilogEval Human
RTLLM
Model
Machine
Func
pass @ (%)
pass (%)
SDG-CC
73.9
62.0
62.8
LDG--CC-Rppal 1K
81.9
68.0
65.8
Repair
A.7 ITERATIVE CODE REPAIR
Table 13: Iterative code repair.
We conduct second iteration by generating 2.7k repair data for the model based on the Repair data from the first iteration. As shown in Table 13 per- formance mostly saturates after this initial iteration. We suspect that the remaining issues are likely due to significant emrors that are challenging to correct.
VerilogEval
RTLLM v1.1
Model
Machine
Human
Fune
pass (%)
pass (%)
SDG-CC
13.9
62.0
62,8
SDGGCC- feer
81.9
68.0
63.8
SDG-CC CC.Repair lter
81.3
68.1
65.6
A.8 DIVERSITY OF GENERATED CODE
We assess the diversity of the code generated by our models. We measure this diversity using BLEU score, Jaccard similarity, and abstract tree edit distance (TSED) in [Song et al.) (2024). The VerilogEvall problems are categorized into NonText and Text, as described in Appendix AA.4 For each problem, we compute the average code diversity score across sampled codes for the same problem and report the mean score for all problems. For TSED, Yamazaki 2015) to extract the abstract syntax tree, and codes that fail syntax checks are excluded from the analysis.
PyVerilo (Takamaeda-
Table 14 presents the results on code diversity. We sample 20 solutions with temperature of 0.8 for each model. We observe that finettuned models generally show a decrease in code diversity for both Text and NonText problems. This reduction is expected, as BLEU and Jaccard metrics account for both correct and incorrect code solutions, and there are often multiple ways implement correct
20

---

Published as a conference paper at ICLR 2025
solution. When comparing our fine-tuned models with GPT-4o, code diversity is similar for Text problems, but our models exhibit poor diversity for NonText problems. This is anticipated, given that the CC training dataset for NonText problems is generated using correct-by correct-y--oossruuee cy-costtrnntion meth- ods and follows similar templates for Verilog code. However, our models demonstrate comparable diversity GPT-4o for Text problems, particularly in TSED metric.
Table 14: Diversity of generated code solutions on VerilogEval-Human sampled with temperature of 0.8. Lower scores indicate higher diversity.
Text
NonText
Type
Models
Jaccard
BLEU
TSED
Jaccard
BLEU
TSED
CodeLlama
0.5330
0.3808
0.4255
0.4707
0.2507
0.3521
Pretrained
DeepSeck-
0.6606
0.5454
0,5956
0.6548
0.3797
0.3847
Models
Starcoder2
0.7724
0.5084
0.5520
0.7212
0.3607
0.4020
GPTT-oo 0.6798
0.6633
0.6906
0.7390
0.6376
(.6137
CodeI.lama
0.6848
0.5992
0.6354
0.8583
0.7242
0.7158
Ours
DeepSeek-Coder
0.6828
0.6040
0.6319
0.8308
0.6866
0.6598
SIDG-CC-Repair
Starcoder2
0.7018
0.6381
0.6721
0.8799
0.7750
0.7740
VerilogEval Human (Overall)
Type
Models
Jaccard
BLEU
TSED
CodeI.lama
0.5155
0.3441
0.4156
Pretrained
DeepSeck-Coder
0.6590
0.4987
0.5505
Models
Starcoder2
0.7580
0.4667
0.5198
GPT-4o 0.6965
0.6561
0.6802
CodeLlama
0.7333
0.6345
0.6515
Ours SDG-CC-Repaic Repair
DeepSeck-Coled
0.7246
0.6273
0.6379
Starcoder2
0.7512
0.6767
0.6942
A.9
ERROR TYPES OF LLM GENERATED ERROR REPORTS
Table 15: Error types of LLM generated error reports.
Error Type
#Errors
One-line Description
Vector Concatenation
15.3%
Errors during vector concatenation bit slicing.
Incorrect Initialization
13.1%
Missing or faulty initialization of registers or signals.
Boolean Logic Flaws
12.4%
Logical inconsistencies errrr in combinational logic expressions.
Shift Operation Faults
10.2%
Misaligned unintended behavior during shift operations.
Timing Violations
10.2%
Errors where signal propagation violates timing requirements.
KMap Misinterpretation
8.8%
Incorrect derivation of Boolean expressions from Karnaugh maps.
Latch Hazards
6.5%
Unintended latches caused by missing faulty conditions.
Bit Manipulation Bugs
7.3%
Errors operations like masking, flipping, or extracting specific bits.
Casez: Priority Conflicts
4.4%
Ambiguities or conflicts in casez or case statements.
Nested Loop Design Flaws
3.7%
Incorrect or inefficient nested loop designs.
Others
8. 11
Miscellancous errors not covered above.
Table 15 shows the distribution of common types in LLM-generated error reports, along with brief one-line descriptions. Most of these "minor" emrors occur solvable problems and stem from hardware-specific concepts (e.g., shift operations, timing violations) and Verilog related issues
21

---

Published as a conference paper at ICLR 2025
common in software languages (e.g., latch hazards, casez priority conflicts). When generating tar- geted repair training data, we randomly sample detailed error reports and open-source code snippets, ensuring the error type distribution in training aligns with their natural occurrences.
A.10 DETAILS ON FIGURE
discussed our findings on training variability in learning outcomes for specific bench- mark problems. To analyze this, we saved check- points every 64 gradient steps during training and tracked the pass rates of specific benchmarks. Our training process is limited to single epoch, as fur- ther training was found to be not helpful. We classify problems with pass rates exceeding 67% as solvable, and those below 33% as unsolvable. For the visualizations in Figure [ we selected the final two saved checkpoints, detailed ity but also less degradations and improved accuracy: specifically, most problems in checkpoint2 should show higher pass rates than checkpointl, assuming that training on additional data enhances model performance. However, shown in Figure Ta training SDG data results in significant degradation of pass rates for many problems between checkpoint and checkpoint2. In contrast Fig- ure Ib demonstrates reduced degradation and improvement in more problems. We further elaborate such findings in Table 17 where we display pass rates for selected benchmark problems with high volatility from Hummann Humma throughout the training progression.
In Section 2.3
Table 16: Checkpoints of Figure
checkpointi
check checkpoin12
Model
Steps
Epoch
Steps
Epoch
SDG
256
0.82
320
1.0
SDG-CC-Repair
386
0.86
448
1.0
Table[[6] The ideal outcome is not merely reduced variabil-
Table 17:
We displays pass rates for selected benchmark problems from VerilogEval- throughout the training progression. Each entry shows the pass rate for SDG-CC-Repair (SDG), with SDG results in parentheses.
Problem
Step 64
Step 128
Step 256
Step 320
Step 386
Step 448
m2014_ep __444hhhhhh__
(1.0
(0.9)
(0.967))
1.0 (0.875)
(-0
(-)
always nolatches
1.0 (0.867)
1.0 (0.9)
1.0 (0.6)
1.0 (0.833)
1-0
1-0
vectorT
(0.633)
1.0 (0.925)
1.0 (0.467)
0.95 (0.925)
(-0
(-0
fsm2s
(0.8)
1.0 (0.8334)
0.8 (0.775)
1.0 (0.967)
(-0
L-
fsm3comb
1.0 (0.0)
0.95 (1.0)
(0.533)
1.0 (0.233)
(-0
1.0
We believe such volatility primarily is due to noise in SDG data where we not verify solution correctness. Because of the difficulties of verifying coding solutions in hardware descriptive lan- guages, we instead generate targeted repair data for L.MMs to learn to mitigate common errors which have shown to generalize to writing correct code during completion. To the best of our knowledge, we are the first work to describe such findings and provide an effective solution.
B FURTHER DISCUSSIONS AND BROADER IMPACTS
In this section, we provide further discussions to address concerns regarding the novelty, generaliz- ability, and significance of proposed methods. We offer clarifications to highlight the relevance and broader impact of our work, underscoring its value to the broad research community.
B.1 GENERALIZABILITY OF CORRECT- COOSSSRRUTTOOOTIIOOOON DATA GENERATION
Our approach to curating correct-y--construcce data is largely inspired by Trinh et al. 2024)) who introduced a mathematically rigorous method utilizing symbolic deduction engines to con- struct synthetic training data, significantly improving LLM capabilities in solving Olympiad geom- ctry problems. Similarly, our method ensures the correctness problems and solutions through a custom-designed data generation pipeline, leveraging custom-designed solvers to generate accurate solutions to their corresponding problems. In contrast to methods distilling LLM responses like Self- Instruct (Wang et al.) 2022), our lution accuracy without relying on strong LLM performance on downstream tasks. We hope that mathematically rigorous approach generating synthetic data further inspire future work
approach ensures data quality and
22

---

Published as a conference paper at ICLR 2025
on improving LLMs general capabilities in areas such math, coding, and symbolic reasoning. Moreover, we recognize that adapting these methods to other domains may require human tuning to identify the best data generation method, and note that automating this process for scalability could be a promising future research direction.
B.2 NOVELTY AND GENERALIZABILITY OF TARGETED CODE REPAIR
Our analysis show that LLMs frequently make "minor" errors in Verilog coding, often correctable within few lines of code. We attribute this primarily to the LLMs' insufficient training in com- prehending problem descriptions and instructions alongside their correct solutions. Prior research has tackled this challenge by improving data quality. For instance, Chen et al. (2022) filters incor- rect code using tests generated by LLMs, while Zhang et al. (2024c) creates preference learning datasets by ranking code through self-validation. Lei et al. (2024) focus generating fine-tuning data through code completion, test validation, and debugging with LLM agents, while Le et al. (2022) trained reward models based on compilation and unit test outcomes enhance LLM perfor- mance via reinforcement learning. However, low-resource languages face additional obstacles due to limited data availability, making it particularly difficult to synthesize unit tests directly in these languages. To address this issue, [assano et al. (2024) introduced lightweight compilers to translate test cases from source to target languages.
Chenn
Verilog coding encounters challenges typical of low-resource languages, compounded additional domain-specific challenges hardware description language rather than conventional program- ming language. Its unique characteristics pose significant barriers to knowledge transfer from high-resource languages, as highlighted in studies on execution performance parallel program- ming (Nichols et al. 2024) and high-performance computing extensions (TehraniJamsaz et al.) 2024]. To address these challenges, we propose novel pipeline for generating targeted code re- pair data. While automatic code repair has been extensively studied, most existing methods focus on widely-used programming languages (Xia et al. 2023), relying on data of buggy code and fixes from open-source repositories (Tufano al.] 2019 Just al. 2014). In contrast, our pipeline utilizes a small set of well-curated benchmarks and testbench to automate the generation of error reports, quality assurance, and augmentation of training datasets by injecting similar errors into open-source code. Our results highlight the effectiveness of this approach, which is language agnostic and can be adapted other low-resource and -emmii--ppecffc programming languages.
B.3 SIGNIFICANCE OF NON-TEXTUAL DATA REPRESENTATIONS IN HARDWARE DESIGN
In this work, emphasize the significance of non-textual data representations, specifically Kar- naugh maps, state-transition diagrams, and waveforms, for accurately capturing hardware function- ality. These representations and verbosity inherent in natural language descriptions. While they may be specific to hardware design, they not limited to Verilog and can be applied to various domain-specific languages (DSLs) for hardware design. This is supported by |Batten et al. (2024), who leveraged similar non- textual representations from VerilogEval- to evaluate the performance of LLMs on several Python-embeddeed hardware design DSLs.
widely utilized by hardware designers to mitigate the ambiguity
In this study, we focus exclusively on limited representations, which constitute significant portion (45 problems, approximately 30%) of all problems in the Earilogvall-Human benchmark (details in Appendix A.4). We exclude other types of non-textual representations due the lack of suitable benchmark for evaluating LLMs in Verilog coding. Chang et al. (2024) emphasize the importance of non-textual representations, particularly visual representations, in describing hardware designs. While their work targets lanual-l models and is therefore beyond the scope of this study, we recognize that similar methodologies of our work such as sereec-by-connstuue
methods could be employed to generate training data for visual representations, such as circuit schematics, data flow diagrams, and state transition graphs.
23

---

Published as a conference paper at ICLR 2025
C EXAMPLES OF TARGETED CODE REPAIR DATA
Problem description: vector3 from VerilogEvll-Man
The top module contains 6 ports that are fed into concatenation which outputs 4 ports. The first input "a" fed into the first port in the concalenation, the second input "b" is fed into the second port the concatenation,
into third port of the concatenation, fourth input "d" is fed into the fourth port of the concatenation, the fifth input "e" is fed into the fifth port of the concatenation, the sixth input "I" is fed into the sixth port of the concatenation. The last bit of the concatenation is 21b11. This value fed in to the last port of the concatenation.
third input "c"
module top module input [4:0] a,
input [4:0] b,
input [4:0]
input [4:0]
input [4:0] e,
input [4:0] f,
output [7:0] W,
output [7:0] X,
output [7:0] y,
output [7:0]
)
Concatenate inputs with the last bit as 2'b11 assign {w, x, 2} (2 (2'b11, a, b, d, e, f);
assign (w, x, y, 2} (a. b, c, e, 2'b11};
endmodule
endmodule
Correct Code
Error Code
LLM generated Error Report
Error Type:
Incorrect vector concatenation and splitting
Category: Combinatorial: wiring
Description: Steps to repair the erroncous implementation:
Concatenating the two bits at the beginning concatenated vector.
Assign the output vectors from the concatenated vector in the correct order and bit ranges
example:
assign (w, X, y. 2) b11, a, c, e, f);
To correct implementation. you should concatenate the last '2'bl'' on the right, shown the correct implementation:
assign [w, x, z} [a, b, c, d, f, 2 2'b11);
LLM constructed Repair data from open-source code
Input:
#*## 1. Problem Description
You are given a Verilog module that demonstrates the use of bit slicing and concatenation in loop. The goal is to correctly display specific portions of 32-bit register using for-loop and explicit slic- ing. However, there commonly made in the concatenation and bit slicing order that causes incorrect results displayed. Your task to identify and fix the error in the code.
24

---

Published as a conference paper at ICLR 2025
##NA
Erroneous Implementation
module block;
reg [31:0] data;
int
initial begin
data 32 CAFE;
// Loop display 8-bit segments of the data register
for (i 0; i < 4; i++) begin
// Incorrectly reversing the bit order in the concatenation $display ("data(8-90d +: 8] 0x 1, data(8+(3 1) 8));
end
// Incorrect slicing order, placing higher bits in lower indexes $display ("data[7:0] 0xt0h", data(31:241);
sdisplay ("data[15:8] 0xt0h", data[23:161]]
sdisplay ('data[23:16] 0xk0h", data(15:8)):
$display ("data[31:24] 0xh0h", data[7:0]);
end endmodule
In this erroneous implementation:
The loop erroneously reverses the order of the 8-bit slices. The explicit bit-slicing outside loop has the slices ordered incorrectly, assigning the MSB the least significant part and vice versa.
#### Hints for Fixing
1. **Correct the loop:**
The loop currently reverses the order of the 8-bit slices. Instead of accessing 'data(8*(3i)) +: 8]', you should access "data[8*i 8]' to display the correct slices of the register. **Fix the slicing outside
loop:**
Ensure that the LSB ('data[7:0J") corresponds to the lower bits of the 'data' register, and the MSB ('data[31:28) corresponds to the higher bits. Correct the slicing order for accurate display.
The corrected implementation should ensure that both the loop and explicit bit-slicing access the reg- ister correctly. without reversing the bit order.
Output:
module block;
reg [31:0] data;
int
1;
initial begin
data 32'hFACE_ CAFE; for (i = 0; i < 4; i++) begin
$display ("data[8+80d +: 8] 0x*0h", i, data[8*i +: 8));
end $display ("data[7:0]
0x%0h", data[7:8]);
sdisplay ("data[15:8] 0x*th" data(11:8]]
sdisplay ('data[23:16) 0x40h", data(23:16));
$display ("data[31:24]
0xx0h" data(31:24)):
end endmodule
Figure 5: An example demonstrating the process for targeted code repair. In this example, training checkpoints during training would have the model sometimes generated correct and error solutions. We use LLMs to first summarize the errors for detailed Error Report and then inject the emorr to open-source code to construct Repair data.
25

---

Published as a conference paper at ICLR 2025
Problem deseription: shift4 from VerilogEval-Hunng
Build 4-bit shift register (right shift), with asynchronous positive edge triggered areset, synchronous
active high signals load, and enable.
(1) areset: Resets shift register
(2) load: Loads shift register with data[3:0] instead shifting.
// (3) ena: Shift right (q[3) becomes zero, q/0] is shifted out and disappears).
// (4) q: The contents of the shift register. If both the load and ena inputs are asserted (I), the load
imput has higher priority.
module top_ module( input clk, input areset, input load, input ena, input [3:0] data, output reg [3:0] q);
always @(posedge clk or posedge areset) begin if (areset) begin
always @(posedge clk or posedge areset) begin
if (areset) begin
// Asynchronous positive edge triggered areset
// Asynchronous positive edge triggered areset
q <= 4' A'b000,;
<= 4'b0;
end else if (load) begin
end else if (load) begin
// Synchronous active high
// Synchronous active high signals
signals load
load
q data;
<= data;
end else if (ena) begin
end else if (ena) begin
// Shift right
// Shift right
q (1'b0, q[::1]);
(q[2:0), 1'b9);
end
end
end
endmodule
endmodule
Correct Code
Error Code
LLM generated Error Report
Eror Type: shifting operation
Category: Sequential: shift registers
Description:
The error in the erroneous implementation related to the shifting operation the shift register.
<= {q[2:0), 1 1'b0};
This line is intended shift the bits in the register the right, but it incorrectly connects the least significant bit (q(O)) to the most significant bit (q[3)) instead of connecting zero the most signifi- cant bit. This results in circular shift instead of a right shift.
To correct this error, the line should be changed to:
bo, q[3:1]};
Here the steps repair the erroneous implementation
Identify line of code that performs shifting operation
Notice that the least significant bit (q(0) is connected to the most significant bit (q(3)) instead of
zero.
the most significant bit and shift the remaining bits Test the implementation to ensure that behaves as expected.
Change the line of code connect
the right
26

---

Published as a conference paper at ICLR 2025
LLM constructed Repair data from open- open-source code
Input:
You are provided with a Verilog module that implements a shift register. However, the implementa- tion contains common enror related to shifting operations. Your lask is to identify the error, which involves mistake in the way bits shifted within the register. Specifically, the problem results in an incorrect circular shift rather than intended right shift operation. You need to fix the errrr ensure that the register shifts correctly.
Erroneous Implementation:
module shift_ registers (clk, clken, SI, 50);
parameter WIDTH = 32;
input clk, clken, SI;
output SO; reg [WIDTH 1:0] shreg; awwas @(posedge clk) begin
if (clken)
// Erroneous shifting operation: circular shift instead right shift shreg (WIDTH- 2:00, (WIDT -1];
end assign 50 shreg (WIDTH-1]] endmodule
In this erroneous implementation:
The shift register incorrectly shifts the most significant bit (MSB) into the least significant bit (LSB), creating circular shift. The correct behavior should be shifting in the new input bit ('SI') while moving existing bits to the right.
Hints for Fixing:
1. **Ieentff the shifting operation error:**
The line 'shreg [shreg| MinrggWIDTH-2200 shreg/WIDTH-II): causes circular shift because the MSB is incorrectly fed back into the LSB position.
**Fix the shifting logie:**
Replace the erroneous MSB assignment with the correct new input bit ('SI'. The correct line should be "shreg (shreg[WIDTTH2200), SI);' to shift in the new input bit from the 'SI' input and ensure the bits shift the right.
By fixing the shifting operation, the register will now shift correctly, with the new input filling the LSB and the existing bits shifting to the right.
Output:
module shift_ registers_ (clk, clken, SI, 50); parameter WIDTH = 32; input clk, clken, SI;
output SO;
reg [WIDTH 1:0] shreg; always @(posedge clk)
begin
if (clken)
shreg [Whreg (WIDTH-2:1), SI);
end
assign SO = shreg [WIDTH-1]
endmodule
Figure 6: Another example based on shift4 from VerilogEval- Human..
27

---

Published as a conference paper at ICLR 2025
Problem description: multi booth 8bit from RTLLM
Please act as professional verilog designer.
Implement 8-bit Radix-4 booth multiplier that performs multiplication two 8-bir inputs b) using Booth algorithm. utilizes clock signal (clk), and signal (reset), and provides the product output and ready signal (riy). The ready signal inly) is set to to indicale the completion of multiphication process.
Module
multi booth_8Bic
Input ports:
clk: Clock signal used for synchronous operation.
reset: Reset signal used to initialize multiplier module.
a: 8-bit input representing the multiplicand.
b: 8-bit impul representing the multiplier.
Outpul
p: 16 output representing the product of the multiplication.
rdy: Ready signal indicating completion the multiplication operation.
Implementation:
positive edge
signal (clk) positive edge
signal (reset), module performs
multiplication process.
If the reset signal (reset) is high, two 16-bit registers multiplier 8a[7), multiplicand <= 8b[7], (reset) is low, indicating normal operation, the module checks if the counter (Shit ctr) less than 16.
If the If the counter (ctr) is less than 16, the multiplicand register (multiplicand) is left-shifted by to simulate the Booth algorithm's shifting operation.
If the current bit of the multiplier register (mullpplint((ur)) is 1, indicating positive Booth encoding, the product register (p) accumulates value
multiplicand register (multiphicand), The counter (ctr) is incremented by I. Once the counter (ctr) reaches 16, indicating the completion of the multiplication process, the ready signal (rdy) set to 1. Give the complete code.
nodule multi_bbothh booth, (p. rdy, clk, reset, a, b);
multi. boothi.bbbb.Bii (p, rdy, clk, reset, a, b);
input clk, reset:
input reset:
input 17:0] a.
input [7:0] a.
output reg [15:0] p:
output reg [15:0] p:
output reg rdy;
reg [15:0] multiplicand, multiplier;
output reg rdy;
reg [15:0] multiplier, multiplicand;
reg [4:0] ctr:
reg [4:8] ctr;
always @lposedge clk or posedge reset) begin
a always @(posedge clk or posedge reset) begin
if (reset) begin
if (reset) begin
multiplicand (18(b17))), b);
multiplier ((8(a(7))),
miltiplier ((8(a(7))) 2
miltiplicand < ((8(b)71)), ctr 0;
<= 8;
end else begin
else begin ictr 16) begin
16) begin
multiplicand multiplicand 1;
multiplicand multiplicand 1; (mmlttiglir(ccrr)
(nultiplier (ctr]) begin
<= p multialicand:
p <= p multiplicand: ctr ctr end else begin rdy <= 1; end end end
end ctr ctr 1;
end else begin rdy < 1;
end end
end endnodule
endnodule
Correct Code
Error Code
LLM generated Error Report
Emor Type:
Incorrect initialization
Category: Soquential: counters
Description:
In the erronoous implementation, the register is not initialized to zero during reset. This can lead to incorrect multiplicttion values. correct implementation initializes the
unknown and may contain rogister zero during reset, ensuring that the multiplication result is accurate.
results, as initial value
To repuir the erroneous implementation, follow these steps:
Locate the always block that handles the and clock
this it's block that starts with always
@(posedge elk or posedge reset).
Within (reset) block, line initialize the p register p
28

---

Published as a conference paper at ICLR 2025
LLM constructed Repair data from open- spen-source code
Input:
Problem Description:
You given Verilog module that implements 4-bit counter. The counter should reset to zero when the reset signal is active and increment with each positive clock edge. However, there is common eror in this implementation: the counter up correctly initialized during which may unpredictable behavior reset. Your identify and fix this initialization ertor. Eironeous Inplementation:
register
nodule up_ counter(
input Clk, input reset.
output (3:0] counter
):
[3:0] counter-up;
@(posedge elk posedge reset)
begin
(reset)
counter up 4'd3;
initialization
counter_up counter_up 4'd1;
end
assign counter counter up:
endrodule
this implementation:
The counter_ up register ster is incorrently initialized to 4'd3 during reset, instend of heing reset to zera (4'd0), which can cause counter behavior when activated.
Hints lor Fixing:
Identify initialization issue: counter reset zero, currently, heing set during causes incorret counting behavior after reset.
reset logic: Modify the (react) block the always block to properly initialize the counter to zero. Change counter_up 4'd3; to counter_up
4'd0;
Ensure proper resct behavior: After fixing the inirialization, the counter will correctly reset to zoro when the reset signal is increment correctly
clock eycle thereafter.
Output:
nodulee up. counter(
input clk,
input reset.
output[3:8] counter
[3:0] counter.up;
@(posedge clk posedge reset)
begin
if(rest
counter up 4'de;
counter-up counter_up 4'd1;
end
assign counter counter up:
endrodule
Figure 7: Another example based on multi_booth 8bit from RTLLM.
29

---

Published as a conference paper at ICLR 2025
D EXAMPLES OF CORRECT.BY-CONSTTICCTON DATA FOR NON-TEXTUAL REPRESENTATIONS
D.1 KARNAUGH MAPS AND TRUTH TABLES
Problem
Implement the circuit described by the Karnaugh map below.
//
//
ab
0
1
//
//
0
//
11
0
1/
10
module top_module(
input a,
input b,
input
output out
);
Solution
The input variables are: ['a', 'b', 'c'].
Based on the Karnaugh map, can transform in to the following truth table:
b
0
1
1
0
The minterms (when output is 1) are: (0,0,0) (-a ~b & -c) This corresponds to the following minterms logic: (-a & -b & -c)"
Finally, based the above logic equation, write the Verilog code that could described by the Karnaugh map:
module top module(
input
input input c,
output out
);
assign out (-a -b -c);
endmodule
Example of generated problem with Karnaugh Map representation in the problem de- scription, We include detailed step-by-step analysis for reaching the correct solution.
Figure 8:
30

---

Published as a conference paper at ICLR 2025
D.2 STATE TRANSITION GRAPHS AND TABLES
Problem
Given the state assigned table shown below, implement the logic functions Y[Ø]
2.
// Present state y[2:0]
Next state Y[2:0] x=0, Next state Y[2:0] x=1 Output
// 000
010, 011
1
// 001
100,
// 01e
001,
100 1
// 011
011, 100
0
// 100
100, 001
0
Solution
The state transition is follows:
// state
Next state in=0, Next state in=1
Output
1/ ,,
1
1/ B E,
C
8
C B
E
11
D E
11
The transition logic is then:
A: next x ? D : C;
B: next ? C
C: next ? E :
D: next x ? E D;
E: next x ? B E;
The output is 1 for states: A, C.
Thus the output logic is: assign z y 1ly C ve corresponds to 001 (A), 011 (D).
Finally, below is the Verilog code for the finite state machine:
module top. module (
input clk,
input x,
input [2:0] y,
output reg YO, output reg z );
reg [2:0] next_state: parameter A= B=1, (=2, D=3, E=4; comb begin (y)
A: next_state next_state
C;
E:
C: next_state = x E B;
D: next state
E D;
E: next state ? B E;
default: next state 'x;
endcase
end assign y == A =
assign Y0 ( next_state || next seat_state == );
endmodule
Figure 9: Example of problem with state transition tables.
31

---

input clk, Published as a conference paper at ICLR 2025
Problem
This is a Moore state machine with four states, four inputs, and one output.
Implement this state machine in Verilog. Reset is an active high synchronous
reset state
1/ D (out=0) in0=1 > D
1/ D (out=0)
in0=0 > A
// C (out=0)
-in=1
// (out=0)
-inl=0
// B (out=l)
-in2=1
// B (out=1)
-in2-0 B
1/ A (out=0)
-in311 B
1/ (out=0) 1n3-0
C
Solution
The finite state machine has four inputs, and the state transition logic is follows:
D: next in0 ? D : A;
C: next = inl ? D : B;
B: next in2 ? 0 B;
A: next in3 ? B
The output is 1 for states: B.
Thus the output logic is: "assign out state = B );²,
Finally, below the Verilog code for the finite state machine:
...
module top_module (
input reset,
input in0,
input inl,
input in2, input in3,
output out );
parameter D=0, C=1, B=2, A=3; reg state;
reg next;
comb begin
case (state)
D: next in0 D A;
C: next inl B;
B: next in2 B;
A: next in3 C;
default: next 'x;
endcase
end lways @ @(posedge clk) begin
(reset) state <= D;
else state < next;
end
assign out state );
endmodule
Figure 10: Example of problem with Moore FSM.
32

---

Published as a conference paper at ICLR 2025
Problem
The following diagram is Mealy machine. Implement in Verilog using one-hot encoding. Resets into state A and reset is asynchronous active high.
1/ x=0 (z=8)-
11 A x=1 (z=1) C
1/ B x=0 (2=1) C
1/ x=1 (2=0) B
1/ x=0
(z=0)
1/ C x=1 (z=0) D
1/ D x=0 (z=1) C
1/ X=1 (z=0) B
Solution
From the transition diagram, we have the following transition logic: // state next state in=0, next state in=1
// D,
B/B C, B //C C, D //. c, B
Thus the state transition logic is follows:
A: next x C : D;
B: next = x ? B C;
C: next x ? C;
D: next x B
The output 1 states: (A, x), (B, -x), (D, -x). state = x 11 ( state & -x
Thus the output logic is: 'assign
11 state = D & -x );¹.
Finally, below is the Verilog code for the finite state machine:
module top_module ( input clk, input areset, input output
); parameter A=2 --2' B=2'b01, C=2'b10, 0=2'b11; reg [1:0] state;
reg [1:0] next_state;
always_ comb begin
case (state)
A: next_state next_state
C;
C: next_ state D C;
D: next_state B C;
default: next_state 'x;
endcase
a lways @ posedge clk, posedge areset) begin if (areset) state <= A;
else state next state;
assign state == A & x II
state == B & -x 11 ( state == D & -x
);
endmodule
Figure 11: Example of problem with Mealy FSM.
33

---

Published as a conference paper at ICLR 2025
Problem
The following is the state transition table for Moore state machine with one input, one output, and four states. Use the following one one-hot state encoding: A =4' b0001, B=4 be41b C=4'b0100, D=4'b1000. Derive state transition and output logic equations by inspection assuming one hot encoding. Implement only the state transition logic and output logic (the combinational logic portion) for this state machine.
// state Next state in=0, Next state in=1
Output
1/ B,
0
1/ B 8 C
1
11 C D A
1/ B
C
Solution
Based on the state transition table, we can obtain the next state from observing the (previous state) and column (input).
Next state A on the following (row, column): (A, in=1) (c, in=l). This
correspond to the following logic: 'state[A] & in || state[C] in'
Next state is B on the following (row, column): : (A, in=0) (B, in=0) (D, in=0). This
correspond to the following logic: 'state[A] & -in II state(B) & -in || state(
-in'.
Next state is C on the following (row, column): (B, in=1) (D, in=1). This
correspond to the following logic: 'state[B] & in || state(D] in'.
Next state is D on the following (row, column): (C, in=0). This correspond to the following logic: "state[C] & -in'.
The output is 1 for states: B, C.
Thus the output logic is: "assign out state[B] || state[C] );².
Finally, below is the Verilog code for the finite state machine:
module top_module ( input in,
input [3:0] state,
output reg [3:0] next state,
output )
parameter A=θ, B=1, C=2, D=3;
assign next. state[A [A] state[A] in || state[C] in;
assign state_) state[A] -in || state[B] & -in II state[D] & -in;
assign stat_state[C] state[B] in state[D] in;
assign stat_state(D] state[C] ~in;
assign out state(B] || state[C] );
endmodule
Figure 12: Example of problem with in-edge focused solution shown in Figure
34

---

Published as a conference paper at ICLR 2025
D.3 WAVEFORMS
Problem
This is a combinational circuit, Read the simulation waveforms to determine what the circuit does, then implement it.
// time
9
// Ons
0
0
// 5ns
0
0
0
// 10ns
0
0
// 15ns
0
// 28ns
0
1
// 25ns
1
0
// 30ns
1
1
// 35ns
0
0
// 40ns
Q01111100
1
// 45ns
0
// 50ns
1
// 55ns
0
// 60ns
1
0
1/ 65ns
// 70ns
1
// 75ns
0
0
// 80ns
1
1
// 85ns
0
// 98ns
1
Solution
Based on the simulation waveform, can transform in the following truth table:
a
b
c
0
0
0
0
0
0
0
0
1
0
1
1
0
The minterms
(when output 1) are: (a & -b -c & -d)
(1,0,0,0) =>
(1,0,0,1) (a -b & & d)
(1,0,1,1) (a & - & & d) (1,1,0,1) (a
-c d)
This corresponds to the following minterms logic:
'(a & -b & -c & -d) (a & -b & -c & d) (a & -b c d) (a & b & -c & d)
Finally, based on above logic equation, can now write Verilog code:
module top module(
input
input
35

---

Published as a conference paper at ICLR 2025
input input
output
);
assign -c d);
(a & -b -c -d) (a & -c d) (a -b & & (a
endmodule
Figure 13: Example of waveform problem for combinational circuits.
Problem
This is a sequential circuit. Read the simulation waveforms to determine what the circuit does, then implement it.
// time
clk
reset
out
// Ons
1
X
// 5ns
1
1
// 10ns
1
// 15ns
// 20ns
8
// 25ns
0
// 30ns
0
// 35ns
// 40ns
// 45ns
8
// 50ns
// 55ns
// 68ns
// 65ns
1/ 78ns
8
// 75ns
// 80ns
// 85ns
// 90ns
Simulation waveform redacted
Solution
From the waveform, we have the following transition logic and output logic: // state Next state in=0, Next state in=1
Output
1/ C, B
1
C/C B,
D
0
11 D
A
1
11 A
C,
B
0
Thus the state transition logic is follows:
D: next in
C;
C: next in B;
B: next in A D;
A: next in C;
The output 1 for states: D,
Thus the output logic is: "assign out state == I| state == );
Finally, below is the Verilog code for the finite state machine:
36

---

Published as a conference paper at ICLR 2025
module cop_module ( input clk, input in,
input reset, output
parameter D=O, C=1, B=2, A=3;
reg state;
reg next;
clwa begin
case (state)
D: next in ? B : C;
C: next in ? D B;
B: next in ? A D;
in ? B
default: next 'x;
endcase
end
Lways @( @(posedge clk) begin
if (reset) state <= D; else state <= next;
end
assign out state || state );
endmodule
Figure 14: Example of a waveform problem for sequential circuits.
37

---

Published as a conference paper at ICLR 2025
E PROMPT TEMPLATES
E.1 SYNTHETIC DATA GENERATION
E.1.1 SELF-INSTRUCT
Your goal is create high-quality Verilog problem.
Guidelines for designing the problem description:
1. This should be *completely self-contained** providing all the contextual information one needs
to understand and solve the problem.
that any specific context, variables, code snip- pets pertinent to this problem are explicitly included.
Assume common verilog knowledge, but
Do not include the code snippet in the problem.
4. The problem should be desinged for the programmers to solve it with one verilog module.
5. The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags.
Now, Please use your creativity to create brand new high-quality Verilog problem.
Figure 15: Prompt used to generate initial 50 seed problems for
Your goal is to create high-quality Verilog problem.
Guidelines designing prohlem description:
This should **oompletely self-contained**t providing all the contextual information one needs to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that any specific context, variables, or code snip- pets pertinent to this problem explicitly included.
Do not include the code snippet in the problem.
problem should desinged the programmers solve with one verilog module. The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags.
Below shows some examples:
<PROBLEM>
(seed problems)
</PROBLEM>
Now, Please use your creativity to create brand new high-quality Verilog problem.
Figure 16: Prompt used for Self-Instruct.
38

---

Published as a conference paper at ICLR 2025
E.1.2 ISS-INSTRUCT
Your goal is create high-quality Verilog problem.
Guidelines for designing the problem description:
1. This should *completely self-contained"*, providing all the contextual information one needs to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that specific context, variables, code snip- pets pertinent this problem explicitly included.
Do not include the code snippet in the problem.
The problem should be designed for the programmers to solve with one Verilog module.
Guidelines the problem description format: The problem description section should enclosed within <PROBLEM> </PROBLEM> tags.
Please increase difficulty of the given programming test question a bit. You can increase the diffi- culty using, but not limited to, the following methods:
1. Your new problem should not directly solved by the original code snippet.
2. You can also change the bit-width requiremnt, how to reset internal signals (if applicable), and whether the solution needs clock signal (combinatorial versus sequential logic). If you do have reset method that is synchronous clock, make to add the clock signal the problem module input.
Add new constraints and requirements to the original problem, adding approximately 10 additional words.
4. Replace commonly used requirement in the programming task with less common and more specific one.
the original problem can be solved with only few logical steps, please more reasoning steps.
Now, Please gain inspiration from the following random code snippet to create a high-quality Verilog problem.
Code snippet for inspiration:
(code snippet)
Output:
Figure 17: Prompt used for OSS-Instruct. We also include prompts inspired from Evol- Instruct (Luo et al. 2024h) to increase problem difficulty.
39

---

Published as a conference paper at ICLR 2025
E.1.3 DOCU-INSTRUCT
Your goal is create high-quality Verilog problem.
Guidelines for designing the problem description:
This should *completely self-contained"*, providing all the contextual information one needs to understand and solve the problem.
2. Assume common verilog knowledge, but ensure that specific context, variables, code snip- pets pertinent this problem explicitly included.
Do not include the code snippet in the problem.
The problem should be designed for the programmers to solve with one Verilog module.
Guidelines the problem description format: The problem description section should enclosed within <PROBLEM> </PROBLEM> tags.
Now, Please gain inspiration from the following textbook or wikipedia snippet to create a high-quality Verilog problem. The information might not be directly related to Verilog, but try to be make the problem relevant as possible to the textbook issue discussed.
Textbook snippet for inspiration:
(document snippet]
Output:
Figure 18: Prompt used for Docu-Instruct with Wikipedia and textbooks.
am going to give you concept and some descriptions about that concept. Based on the descrip- tions and concept name, determine if the concept belongs to one of the following categories:
Hardware description and modeling Verilog.
Fundamental constructs such as modules, ports, and wires specific to Verilog. Synthesis and optimization techniques employed in hardware design using Verilog.
Simulation tools and methodologies for verifying Verilog-based hardware designs.
Common design patterns best practices in Verilog efficient hardware implementation.
Programming concepts like loops, functions related to Verilog.
Hardware related concepts such as finite state machines that could he implemented Verilog.
Algorithms that could be implemented in hardware, such as Fourier Transforms.
Concept: (Wikipedia title)
Description: (Wikipedia content)
Do not make assumptions and only respond "Yes" if you are certain that the (Wikipedia title) is re- lated hardware design Verilog coding language.
Your answer should start with "Yes" or "No".
Figure 19: Prompt used to filter Verilog related Wikipedia pages.
40

---

Published as a conference paper at ICLR 2025
E.1.4 NON-TEXTUAL REPRESENTATIONS
Your goal is create high-quality Verilog problem. Specifically, we would like to test the skills of understanding Karnaugh maps and state transition diagrams. The problem description section should be enclosed within <PROBLEM> </PROBLEM> tags.
Now, please gain inspiration from the following random code snippet create high-quality Ver- ilog problem. Remember that the problem you generated must include Karnuugh maps the format above. The random code snippet MUST be related to the solution. Your problem statement should be short and succinct (no more than 5 sentences) and you MUST generate a Karnaugh map in the problem description. Your problem description should not describe the Karnaugh map in words and should assume that the student need decipher the Karnaugh map solve the problem.
Code snippet for inspiration:
...
(cnde snippet]
Below two examples how represent Karnaugh map related questions in purely textual for- mat. should NOT use following to generate problem but only consider the style.
<PROBLEM>
Given the state assigned table shown below, implement the finite state machine.
Reset should synchronous active high reset the FSM to state 000.
// Present state y[2:0] Next state y[2:0] x=0, Next state y[2:0] x=1, Output
// 000
000,
// 001
001, 100
0
// 010
010, 001
0
// 011
001, 010
// 100
011, 160
</PROBLEM> <PROBLEM>
Implement the circuit described by the Karnaugh map below.
//
1/
1/
00
11
01
1
11
11
11
10
<PROBLEM>
Figure 20: An prompt example to encourage LLMs to generate questions with Karnaugh Maps.
41

---

Published as a conference paper at ICLR 2025
E. PROMPTS FOR SAMPLING SOLUTIONS WITH LLM GENERATED PROBLEMS
Your goal is solve Verilog problem by completing one verilog module.
Guidelines for writing the solution (verilog module) and formatting:
1. The module should be **completely self-contained**, fulfilling all the requirements needed to solve the problem.
Ensure the correctness the syntax and functionality.
The module name must be "top_modie _op_mmoule".
4. Do not add blackslash in front of underscore symbol.
5. The solution must be valid Verilog code.
The solution section should he enclosed within <SOLUTION> </SOLUTION> tags.
reasoning section should enclosed within <REASON> </REASON> tags. Write comments in the solution section. Try to embed the reasoning in comments. Always try to write the corresponding comments before the code.
The module should be * **ooppetely self-contained**, do not include import outside the module and define everything inside the module.
Below shows example:
Problem description:
Build counter that counts from 999, inclusive, with period of 1800 cycles. The reset input active high synchronous, and should reset the counter to 0. Solve the problem by completing the following module.
Output:
<REASON>
To solve the problem of building a counter that counts from 0 999 with reset input that is active high synchronous, you will need a 10-bit register (since 2^10=1024 and 999 1624). The counter should increment on each clock cycle and wrap around back to e when it reaches 1000. The reset input, when active, should reset the counter
the next clock edge.
</REASON>
Here the Verilog code for the module: <SOLUTION>
module vop_module(
input clk, input reset, output reg [9:0] q);
// This block executes on the positive edge of the clock signal. lways @(posedge clk) begin
// Checks the reset is active. This synchronous with the clock. if (reset) begin
// Reset the counter to when reset is active 0;
// counter has reached 999, needs to wrap around the next cycle.
end else if (q == 999) begin
// Reset to 0 after reaching 999
q 0;
// other cases, just increment the counter. end else begin
// Increment the counter
end end endmodule </SOLUTION>
42

---

Published as a conference paper at ICLR 2025
Now, please solve the following Verilog problem. will also attach a reference code snippet which was used as an inspiration to generate the problem. The provided code may not directly solve the problem you should use it only reference. Reference code:
(code snippet)
Problem description:
(in context examples]
Output:
Figure 21: Prompt used for sampling solutions for synthetic data generation. We include a in context example to encourage models to include reasoning traces. Prompts in blue only included for problems generated from a code snippet.
E. PROMPTS FOR VERIFYING SOLUTIONS
Check if the given Verilog module is a valid solution to the problem. The output should be "True" or "False" and be enclosed within <VALID> </VALID> tags and the explanation in <REA- SON></REASON> tags.
Now check the following:
<PROBLEM>
(problem)
<PROBLEM>
<SOLUTION>
[solution]
<SOLUTION>
Figure 22: Prompt used for verifying solutions.
43

---

Published as a conference paper at ICLR 2025
E.2 PROMPTS FOR TARGETED CODE REPAIR
E.2. ERROR REPORT
Here a Verilog problem description:
(problem description]
Here is erroneous implementation:
(error code]
Here a correct implementation:
(correct code)
Generate detail error report.
The emrrr report should describe the common emorr type and output the code category. The port should also detailed enough beginners to repair the erroneous implementation step by step.
Output:
Figure 23: Prompt for Error Report generation.
Here is a Verilog problem description:
(problem description)
Here is erroncous implementation: ..*
(error code]
Here is the error report:
(error report]
..*
Now fix the erroneous implementation and give me the correct code. Output:
Figure 24: Prompt for Error Report self-consistency validation. The generated code fix will be evaluated for functional correctness. Error reports whose code fixes do not pass will be filtered.
44

---

Published as a conference paper at ICLR 2025
E.2.2 ERROR INJECTION
Your goal is create an error-fixing Verilog practice problem for programmers. You will demonstrate type of error that is commonly made by programmers.
Create error repair practice problem with three components:
Problem description
Erroneous implementation
3. Hints for fixing
Here is an example:
<EXAMPLE>
The following Verilog module is intended to implement the specification below. However, there is bug in the code which causes incorrect results. Please fix the bug make the module work intended.
Erroneous Implementation:
// Verilog code with the injected error module example module ( input wire clk, input wire reset, output reg [3:0] counter
);
// Intended functionality:
// This module should count from 0 to 15 and then wrap around.
always @(posedge clk posedge reset) begin
if (reset) begin
counter Abb000:;
end else begin
counter <= counter 1'bl; // Error injectedd Should be 4'b1
end end
endmodule
Hints for Fixing:
1. Verify the bit-width of the counter and the increment operation.
Check initialization and wrapping condition the counter. Ensure that the addition operation correctly handles the 4-bit counter.
</EXAMPLE>
Now, here is the commonly made
(error report]
...
Inject the above error into the following module and create an error repair practice problem. Check if is possible to inject the error. If not, create the problem with the given error alone and ignore the module in the code snippet.
(code snippet)
Output:
Figure 25: Prompt used inject targeted errors open-source code in code Repair data. We also prompt the L.M to self-verify the error could be injected to the code snippet.
45

---

Published as a conference paper at ICLR 2025
Karmaugh Maps and Truth Tables
State Transition Graphs
Tables
Somple randoen minterms Step1, Sample variables T'A minterms 11. 2.5, don't Leares 17) Configurations
ON
Step1. Sample
SOP ferm (-4 &- - & &
Configuratins
Trath tuble
Step2. Construct
Representations
and Problemos
11013
1101010
0111010
0
1
Karnaugh
0/011 01l
1|011 xip
< #1 11
Step2. Construct
Represrntations
and Probloms
A AA
38 (eumr)) -ayrion
(out))) -ays--
(eat-d)
groph.
Stat tramsliis cife.
(eatri)) -im-Bio
(est-1)) -a
uedale medaly
alvays postt kode
Inpur 3,
Ingast
la BB
It must C-C.
Step3, Construct Solution
Step3, Coastruct
Clans-
Solution
ondggn E- ()((4 [+ eademsdikk
caloxc
(atatenn)) (statenD)))
Figure 26: Correct-by-construutii for Karnaugh maps and truth tables.
Figure 27: Correct-yy-connsruutiinn for finite- state machines.
Waveforms
mudule top
always comth begia
(diale)
imput Input
1. Obtain Previous Solutions
uutput
R
endrase
assign I- (-a -b O-a & -e) & endmodale
end
assige out (state O) (statemDD)
Step 2. Simulate with Template Bench
0
Code Testbench
Verilog Simulator
Waveform VCD File
Combinatorial ciremit.
Sequential circuit.
A time elkk
Step Construct Waveform Problems
# Ons
0 1 0
I! Ons 0
10ns
A 10mc
15ns
Figure 28: Correct-by-construcc for waveforms.
46


Reviews:
[{"review_id": "RUca4uytEj", "reviewer": "Reviewer_8TDV", "summary": "This paper does a thorough evaluation of LLMs for verilog code generation. They first analyze existing model performance on Verilog code generation tasks, identify that \"non-textual representations\" are commonly mis-reasoned about, use this to motivate two new methods for improving SDG for verilog code gen tasks, and test their approach against other SDG approaches. They find that their method outperforms baselines.", "questions": "* Fig 1 is kind of confusing. Why choose checkpoints 1 and 2? Would we hope for the pass@k for checkpoint 2 to be higher than for chkpt 1?  This scatter-plot resembles a confusion matrix-- why choose the scatter plot representation over a different option? The value of figure 1 is made more apparent once we see figure 5. Maybe the two could be presented closer to one another in a camera-ready. How were the \"solvable\" and \"unsolvable\" regions chosen? \n* L319: how do we know that the ability to self-correct (validating via self-consistency) is due to a good error report, and not the model's ability to correct independent of the error report? Especially since the examples from which error reports are generated did yield both correct and incorrect generations, to start with.\n* is the amount of training data consistent between all rows of Table 6?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* This paper presents a clear discussion of an important and under-explored topic. Low-level programming languages are an appealing area in which to automate code reasoning, and programs in HDLs are notoriously difficult to verify. \n* thorough evaluation in terms of comparison to other SDG methods and other baselines. Appropriate ablations further convey the value of all components of their approach.\n* the code repair generation process is compelling, and validation well-grounded in existing literature. I anticipate that it's highly transferrable to other domains of data as well. \n* the combination of using hand-crafted methods for highly-underrepresented or challenging concepts (\"non-textual elements\") and automated self-consistency-based methods for intermediate concepts (generating the repair data) paints a cohesive picture for SDG, especially in this domain.", "weaknesses": "* the data generation processes for Karnaugh maps, state-transition diagrams, and waveforms are pretty hand-crafted. This makes this method difficult to transfer to other identified model weakness categories, and requires human-tuning to identify the best data gen method per category. This approach also may not work as well, if at all, on some categories. (For example, the findings of L461 that indicate the Waveforms problems do not improve as much as the other approaches.) An automated method for designing the data construction may scale better. (out of scope for this paper though, and I would not consider this a reason for rejection)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E7zoqJ8CXf", "reviewer": "Reviewer_5aNd", "summary": "The paper introduces CraftRTL, a novel approach to Verilog code generation by leveraging a combination of synthetic data generation and targeted code repair to improve accuracy and robustness.", "questions": "1. Could the authors discuss how this method for Verilog-specific elements might be adapted for other HDLs or general programming languages?\n\n2. Figures 2, 4, 5, and 6, along with Tables 4 and 5, could benefit from clearer formatting and structure. Could the authors enhance these visuals to improve readability and clarify how the best results are highlighted across different model types?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The primary contributions of this paper include the introduction of correct-by-construction data generation, which focuses on non-textual data representations that are essential for Verilog code and often challenging for LLMs. By incorporating Karnaugh maps, state-transition diagrams, and waveforms, the model’s capacity to interpret and generate these complex data formats improves. The experimental results demonstrate notable improvements over previous approaches on multiple benchmarks​.", "weaknesses": "The methods presented, particularly the correct-by-construction data targeting non-textual representations, are tailored heavily to Verilog-specific constructs such as Karnaugh maps, state-transition diagrams, and waveforms. While this adaptation effectively improves performance for Verilog code generation, the approach may have limited applicability to other hardware description languages or general programming languages that do not rely on these specific data formats. A broader discussion on how these techniques could be generalized would strengthen the paper's impact.\n\nSeveral figures and tables in the paper, notably Figures 2, 4, 5, and 6, as well as Tables 4 and 5, suffer from presentation clarity issues. Figures lack a cohesive and clear structure, making it difficult for readers to follow the exact steps. In Tables 4 and 5, the inconsistent formatting of model types and unclear emphasis on the best-performing results within each category lead to potential confusion in understanding the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M62Xpio6Mw", "reviewer": "Reviewer_sdjN", "summary": "This paper discusses two main issues when LLMs  handling Verilog code: models have difficulty handling non-textual elements in problem statements and models making \"minor\" programming mistakes. To address these issues, the authors specifically created a transformed non-textual dataset and code repair dataset to fine-tune the model. The results demonstrate that the fine-tuned Starcoder2-15B surpasses the prior state-of-the-art results in Pass@1 performance, achieving improvements of 3.8\\%, 10.9\\%, and 6.6\\% on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.", "questions": "1. Why focus solely on karnaugh maps, state-transition diagrams, and waveforms? They do not represent all types of non-textual representations.\n2. It is essential to ensure that the generated error report can effectively guide the model in correcting errors. How do the authors validate its effectiveness?\n3. In the \"Targeted Code Repair Dataset\" section, I suggest the author to  provide classification and proportion of the \"minor\" errors. Additionally, were any additional data augmentation measures taken for high-frequency errors during dataset construction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper conducts a detailed empirical analysis of the two main issues in Verilog code.\n2. The paper provides a thorough comparison with existing methods and shows good performance.", "weaknesses": "1. The main contribution of this work lies in constructing a fine-tuning dataset to address non-textual data and minor error issues. The technical contribution of the paper is limited.\n2. What is the specific definition of \"minor\" errors, and what common characteristics do they share?\n3. The font size of Figures 2 and 4 is too small to read.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPOWCAijp7", "reviewer": "Reviewer_uCA7", "summary": "- This paper performs a thorough analysis of fine-tuned LLMs on Verilog code, and revealing two main challenges of automated Verilog code generation\n- This paper creates a large number of correct-by-construction data to ensure solution correctness, incorporating Karnaugh Maps, state-transition diagrams, and waveforms\n- This paper develops an automated framework that utilizes LLMs to generate error reports from benchmark problems\n- Its evaluation results demonstrate that models fine-tuned with our data achieve state-of-the-art performance on Verilog coding, outperforming prior SOTA results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively", "questions": "- Checkpoint Selection: what is the selection criteria of your checkpoints in Figure 1 and Figure 5? You mentions \"two consecutive checkpoints\" in Line 434. Additionally, you only fine-tune your model for one epoch (Line 364), so at least one checkpoint might not see all training data. Whether such difference affects your results?\n- Application Scenario: This paper addresses an important problem in Verilog code generation utilizing domain knowledge of Verilog. So I am curious how such approach is applied to similar tasks, e.g., code generation without sufficient training data?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper is clearly written and easy to comprehend\n- This paper is well-motivated and address an important downstream task, automated Verilog code generation\n- This paper includes a comprehensive and reliable data construction pipeline\n- This paper conduct a comprehensive evaluation on three LLMs with SOTA baselines", "weaknesses": "Actually I like this paper, especially the data construction section; however, there are still some minor concerns:\n\n- \"Quality Assurance with LLM Validation\" (Line 317): Please provide more evidence about the choice of LLM validation. What is the rationale (or its limitations) of not using deterministic validation approaches, e.g., model checking?\n- In Section 2.3, you mention \"significant variability in the model’s pass rate on specific benchmark problems across different checkpoints\" (Line 164), while the results in Figure 1 indicates a highly positive correlation. Also, 15% discrepancies is also acceptable between two checkpoints. Can you provide me with a stronger evidence to support this claim, e.g., Pearson Correlation Coefficient, or explain why such difference is significant in this task?\n- Application scenario: This paper mainly utilizes domain specific patterns of various types of Verilog while it might be difficult when applied to similar tasks, e.g., code generation without sufficient training data.\n- Reproducibility Statement: this subsection exceeds the 10 pages limit. I think it should be placed within the first 10 pages or directly moved to appendix\n- Availability: this paper does not provide an available artifact", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)