CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair

Authors: Mingjie Liu, Yun-Da Tsai, Wenfei Zhou, Haoxing Ren
Primary area: generative models
Keywords: verilog code generation, synthetic data generation, large language models

Abstract:
Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making ''minor'' mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8\%, 10.9\%, 6.6\% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1729490159296

thread_type:
official_review

<thread_object>
creator: reviewer_uca7

<content>
summary:
- This paper performs a thorough analysis of fine-tuned LLMs on Verilog code, and revealing two main challenges of automated Verilog code generation
- This paper creates a large number of correct-by-construction data to ensure solution correctness, incorporating Karnaugh Maps, state-transition diagrams, and waveforms
- This paper develops an automated framework that utilizes LLMs to generate error reports from benchmark problems
- Its evaluation results demonstrate that models fine-tuned with our data achieve state-of-the-art performance on Verilog coding, outperforming prior SOTA results by 3.8%, 10.9%, 6.6% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively

strengths:
- This paper is clearly written and easy to comprehend
- This paper is well-motivated and address an important downstream task, automated Verilog code generation
- This paper includes a comprehensive and reliable data construction pipeline
- This paper conduct a comprehensive evaluation on three LLMs with SOTA baselines

weaknesses:
Actually I like this paper, especially the data construction section; however, there are still some minor concerns:

- "Quality Assurance with LLM Validation" (Line 317): Please provide more evidence about the choice of LLM validation. What is the rationale (or its limitations) of not using deterministic validation approaches, e.g., model checking?
- In Section 2.3, you mention "significant variability in the model’s pass rate on specific benchmark problems across different checkpoints" (Line 164), while the results in Figure 1 indicates a highly positive correlation. Also, 15% discrepancies is also acceptable between two checkpoints. Can you provide me with a stronger evidence to support this claim, e.g., Pearson Correlation Coefficient, or explain why such difference is significant in this task?
- Application scenario: This paper mainly utilizes domain specific patterns of various types of Verilog while it might be difficult when applied to similar tasks, e.g., code generation without sufficient training data.
- Reproducibility Statement: this subsection exceeds the 10 pages limit. I think it should be placed within the first 10 pages or directly moved to appendix
- Availability: this paper does not provide an available artifact

questions:
- Checkpoint Selection: what is the selection criteria of your checkpoints in Figure 1 and Figure 5? You mentions "two consecutive checkpoints" in Line 434. Additionally, you only fine-tune your model for one epoch (Line 364), so at least one checkpoint might not see all training data. Whether such difference affects your results?
- Application Scenario: This paper addresses an important problem in Verilog code generation utilizing domain knowledge of Verilog. So I am curious how such approach is applied to similar tasks, e.g., code generation without sufficient training data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730065548893

thread_type:
official_review

<thread_object>
creator: reviewer_sdjn

<content>
summary:
This paper discusses two main issues when LLMs  handling Verilog code: models have difficulty handling non-textual elements in problem statements and models making "minor" programming mistakes. To address these issues, the authors specifically created a transformed non-textual dataset and code repair dataset to fine-tune the model. The results demonstrate that the fine-tuned Starcoder2-15B surpasses the prior state-of-the-art results in Pass@1 performance, achieving improvements of 3.8\%, 10.9\%, and 6.6\% on VerilogEval-Machine, VerilogEval-Human, and RTLLM, respectively.

strengths:
1. The paper conducts a detailed empirical analysis of the two main issues in Verilog code.
2. The paper provides a thorough comparison with existing methods and shows good performance.

weaknesses:
1. The main contribution of this work lies in constructing a fine-tuning dataset to address non-textual data and minor error issues. The technical contribution of the paper is limited.
2. What is the specific definition of "minor" errors, and what common characteristics do they share?
3. The font size of Figures 2 and 4 is too small to read.

questions:
1. Why focus solely on karnaugh maps, state-transition diagrams, and waveforms? They do not represent all types of non-textual representations.
2. It is essential to ensure that the generated error report can effectively guide the model in correcting errors. How do the authors validate its effectiveness?
3. In the "Targeted Code Repair Dataset" section, I suggest the author to  provide classification and proportion of the "minor" errors. Additionally, were any additional data augmentation measures taken for high-frequency errors during dataset construction?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649746070

thread_type:
official_review

<thread_object>
creator: reviewer_5and

<content>
summary:
The paper introduces CraftRTL, a novel approach to Verilog code generation by leveraging a combination of synthetic data generation and targeted code repair to improve accuracy and robustness.

strengths:
The primary contributions of this paper include the introduction of correct-by-construction data generation, which focuses on non-textual data representations that are essential for Verilog code and often challenging for LLMs. By incorporating Karnaugh maps, state-transition diagrams, and waveforms, the model’s capacity to interpret and generate these complex data formats improves. The experimental results demonstrate notable improvements over previous approaches on multiple benchmarks​.

weaknesses:
The methods presented, particularly the correct-by-construction data targeting non-textual representations, are tailored heavily to Verilog-specific constructs such as Karnaugh maps, state-transition diagrams, and waveforms. While this adaptation effectively improves performance for Verilog code generation, the approach may have limited applicability to other hardware description languages or general programming languages that do not rely on these specific data formats. A broader discussion on how these techniques could be generalized would strengthen the paper's impact.

Several figures and tables in the paper, notably Figures 2, 4, 5, and 6, as well as Tables 4 and 5, suffer from presentation clarity issues. Figures lack a cohesive and clear structure, making it difficult for readers to follow the exact steps. In Tables 4 and 5, the inconsistent formatting of model types and unclear emphasis on the best-performing results within each category lead to potential confusion in understanding the experimental results.

questions:
1. Could the authors discuss how this method for Verilog-specific elements might be adapted for other HDLs or general programming languages?

2. Figures 2, 4, 5, and 6, along with Tables 4 and 5, could benefit from clearer formatting and structure. Could the authors enhance these visuals to improve readability and clarify how the best results are highlighted across different model types?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662950358

thread_type:
official_review

<thread_object>
creator: reviewer_8tdv

<content>
summary:
This paper does a thorough evaluation of LLMs for verilog code generation. They first analyze existing model performance on Verilog code generation tasks, identify that "non-textual representations" are commonly mis-reasoned about, use this to motivate two new methods for improving SDG for verilog code gen tasks, and test their approach against other SDG approaches. They find that their method outperforms baselines.

strengths:
* This paper presents a clear discussion of an important and under-explored topic. Low-level programming languages are an appealing area in which to automate code reasoning, and programs in HDLs are notoriously difficult to verify. 
* thorough evaluation in terms of comparison to other SDG methods and other baselines. Appropriate ablations further convey the value of all components of their approach.
* the code repair generation process is compelling, and validation well-grounded in existing literature. I anticipate that it's highly transferrable to other domains of data as well. 
* the combination of using hand-crafted methods for highly-underrepresented or challenging concepts ("non-textual elements") and automated self-consistency-based methods for intermediate concepts (generating the repair data) paints a cohesive picture for SDG, especially in this domain.

weaknesses:
* the data generation processes for Karnaugh maps, state-transition diagrams, and waveforms are pretty hand-crafted. This makes this method difficult to transfer to other identified model weakness categories, and requires human-tuning to identify the best data gen method per category. This approach also may not work as well, if at all, on some categories. (For example, the findings of L461 that indicate the Waveforms problems do not improve as much as the other approaches.) An automated method for designing the data construction may scale better. (out of scope for this paper though, and I would not consider this a reason for rejection)

questions:
* Fig 1 is kind of confusing. Why choose checkpoints 1 and 2? Would we hope for the pass@k for checkpoint 2 to be higher than for chkpt 1?  This scatter-plot resembles a confusion matrix-- why choose the scatter plot representation over a different option? The value of figure 1 is made more apparent once we see figure 5. Maybe the two could be presented closer to one another in a camera-ready. How were the "solvable" and "unsolvable" regions chosen? 
* L319: how do we know that the ability to self-correct (validating via self-consistency) is due to a good error report, and not the model's ability to correct independent of the error report? Especially since the examples from which error reports are generated did yield both correct and incorrect generations, to start with.
* is the amount of training data consistent between all rows of Table 6?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731978810048

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updates on Paper Revision

comment:
We thank all reviewers for questions and suggestions. We have made modifications to our paper to address comments from the reviewers and uploaded a new pdf version. In summary and response to all reviewers, we have made the following modifications:

1. We have greatly revised the figures and tables to enhance the visual and readability. Specifically we have:
- Merged figures to Figure 1 (Reviewer 8TDV), and provided further details in Appendix A10 (Reviewer uCA7).
- Redrawn Figure 3 with abbreviated text, enlarged bolded text fonts for improved readability (Reviewers 5aNd, sdjN). Also removed original Figure 2 now replaced with Figures 26,27, 28 in Appendix.
- Redrawn Figure 4 in high-resolution tikzplot (Reviewer 5aNd)
- Removed confusing captions and highlighted only best results for Table 4,5 (Reviewer 5aNd).

2. Added requested new results and details
- Classification and proportion of minor errors in Appendix A9 (Reviewer sdjN).
- Clarifications on effectiveness of error report in error fix on Page 6 Line 313: “whereas directly prompting the LLM without detailed error reports could resolve only 13% of the errors” (Reviewers 8TDV, sdjN, uCA7)

3. Added Appendix B for further discussions and broader impacts. In this section we address the concerns on technical contribution (Reviewer sdjN), generalizability of our proposed methods to other HDLs or programming languages (Reviewers 5aNd, uCA7), and significance on focusing to non-textual representations (Reviewer sdjN).


Although our work is focused on the narrow domain of Verilog coding, we believe that our proposed methods could be generalizable (details in Appendix B), and be of value to the broad ICLR research community. We hope our revised manuscript and rebuttal could offer clarification and hopefully resolve the many valid concerns raised by reviewers.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces methods for training code LLMs on an important hardware description language, doing a deep dive on the specific kinds of errors made on those problems and addressing them through correct-by-construction synthetic data generation. The primary strength is that it addresses an important problem and has a robust empirical evaluation, while the primary weakness is that the method is not particularly conceptually creative and relatively narrow in its practical implications. I recommend accepting this paper due to it's practical importance, and the fact that researchers in the ML and HDL communities could likely build on these models and methods.