DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model
Authors: Yi Liu
Changran XU
Yunhao Zhou
Zeju Li
Qiang Xu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Model
Program Representation Learning
Verilog Understanding and Generation
Primary area: foundation or frontier models, including LLMs

Abstract: Recent advancements in large language models (LLMs) have shown significant potential for automating hardware description language (HDL) code generation from high-level natural language instructions. While fine-tuning has improved LLMs' performance in hardware design tasks, prior efforts have largely focused on Verilog generation, overlooking the equally critical task of Verilog understanding. Furthermore, existing models suffer from weak alignment between natural language descriptions and Verilog code, hindering the generation of high-quality, synthesizable designs. To address these issues, we present DeepRTL, a unified representation model that excels in both Verilog understanding and generation. Based on CodeT5+, DeepRTL is fine-tuned on a comprehensive dataset that aligns Verilog code with rich, multi-level natural language descriptions. 
We also introduce the first benchmark for Verilog understanding and take the initiative to apply embedding similarity and GPT Score to evaluate the models' understanding capabilities. These metrics capture semantic similarity more accurately than traditional methods like BLEU and ROUGE, which are limited to surface-level n-gram overlaps. By adapting curriculum learning to train DeepRTL, we enable it to significantly outperform GPT-4 in Verilog understanding tasks, while achieving performance on par with OpenAI's o1-preview model in Verilog generation tasks.

Published as a conference paper at ICLR 2025
DEEPRTL: BRIDGING VERILOG UNDERSTANDING AND GENERATION WITH A UNIFIED REPRESENTATION MODEL
Yi Liu Changran Xu Yunhao Zhou Zeju Li Qiang Xu
'The Chinese University of Hong Kong National Technology Innovation Center for EDA {yliu22, 2j1124, @cse cuhk edu. hk (xxuchangran, yunhaoz as gmaail.omm
ABSTRACT
Recent advancements in large language models (LLMs) have shown significant potential for automating hardware description language (HDL) code generation from high-level natural language instructions. While fine-tuning has improved LLMs' performance in hardware design tasks, prior efforts have largely focused on Verilog generation, overlooking the equally critical task of Verilog understand- ing. Furthermore, existing models suffer from weak alignment between natural language descriptions and Verilog code, hindering the generation of high-quality, synthesizable designs. To address these issues, we present DeepRTI, a unified representation model that excels in both Verilog understanding and generation. Based on CodeTS++ DeepRTL is fine-tuned on comprehensive dataset that aligns Verilog code with rich, multi-level natural language descriptions. We also in- troduce the first benchmark for Verilog understanding and take the initiative apply embedding similarity and GPT Score to evaluate the models' understand- ing capabilities. These metrics capture semantic similarity more accurately than traditional methods like BLEU and ROUGE, which are limited to surface-level n-gram overlaps. By adapting curriculum learning train DeepRTL, we enable it to significantly outperform GPT-4 in Verilog understanding tasks, while achieving performance on par with OpenAl's ol o1-preview model in Verilog generation tasks. Our code and datasets are available at
INTRODUCTION
The development of powerful large language models (LLMs), such OpenAl's GPT-4 (Achiam al. 2023). has brought transformative benefits to diverse fields (Wei et al. 2024 Jin et al. 2023), including electronic design automation (EDA). These models help streamline the hardware design process by generating hardware description language (HDL) code, like Verilog, from user-defined specifications (Pearce al. 2020; Chang al. 2023). The use of LLMSs in EDA has opened new avenues for agile chip design, wherein hardware designers can specify requirements through natural language prompts, potentially boosting both creativity and efficiency in chip design processes.
Despite the adaptability of commercial LLMs like GPT-4 in the EDA domain, the proprietary na- ture of many designs necessitates the development of tailored model, which requires fine-tuning a specialized model to ensure data security and customization for specific needs. Recent studies have attempted fine-tuning open-source LLMs for Verilog generation, demonstrating great poten- tial for automated generation of Verilog code from high-level prompts (Thakur et al.) (2024) Chang al., 2024b) /Zang et al. 2024). However, these efforts often focus solely on Verilog generation, neglecting the equally critical task of Verilog understanding, i.e., summarizing high-level function- ality from Verilog code snippets using natural language. This capability is essential for effective communication among hardware designers, as it helps decipher complex code written by others, facilitating collaboration and comprehension. Moreover, even for Verilog generation, these works fail to establish strong alignment between natural language and Verilog code, which potentially harms the models' performance. For example. Thakur et al. (2024) do not incorporate paired data of natural language and Verilog code in their dataset. CCang et al. (002b)) propose mapping the

---

Published as a conference paper at ICLR 2025
Verilog Abstract Syntax Tree (AST) directly to natural language, but this approach is limited to line- level translation and produces descriptions that lack high-level semantics. To further bridge this gap, Zhang et al. 2024) introduce the MG- VG-Vlrilog dataset with multi-level descriptions alongside corre- sponding code samples, but its small size and reliance on I. -LMMA2-70B-Ch for annotations raise quality concerns. Such poor alignment between natural language and Verilog code can degrade the generation performance, leading to generation of non-synthesizable or non-functional Verilog code.
To address these challenges, we introduce DeepRTL, a unified representation model that bridges Ver- ilog understanding and generation. Achieving this requires an extensive collection of high-quality, cerdwarr--peciffie datasets, which are scarce the open-source community. To this end, we have meticulously curated comprehensive Verilog dataset that ensures strong alignment between Verilog code and natural language across multiple levels. This dataset includes both open-source and propri- etary Verilog design data. For the open-source data, we adopt the thain-of-thouglee (Co'T) approach and use GPT-4, the most advanced model available, to generate precise natural language descriptions of the Verilog code. Human evaluations have verified that these annotations are approximately 90% accurate, underscoring the dataset's 's high quality and reliability for training. For the proprietary data, engage team of professional hardware designers provide detailed annotations, which capture intricate design features and further boost the dataset's 's quality. This comprehensive dataset enables us develop DeepRTI. capable of both understanding and generating Verilog code. By integrating high-quality annotations, the model enhances efficiency and accuracy in various design tasks.
We are the first integrate the task of Verilog understanding into our model, addressing signifi- cant gap left by previous works that focus exclusively on Verilog generation. These earlier efforts lack benchmarks to evaluate LLMs' understanding capabilities of Verilog code, prompting us to introduce the first benchmark for Verilog understanding. Our benchmark comprises one hundred di- verse, high-quality Verilog designs and we have collaborated with professional engineers to develop precise high-level functional descriptions, which have been meticulously cross-checked by multiple designers to ensure their accuracy. In the software domain, traditional metries like BLEU (Papineni) et al. 2002) and ROUGE (Lin) 2004) scores have been commonly used to assess the similarity be- tween generated code summaries and ground truth annotations (Wang et al. 2023a). However, these metrics focus primarily on lexical overlap and often fail to capture the true semantic meaning of the descriptions. To tackle this issue, we take the initiative to apply embedding similarity and GPT score for evaluation, both of which assess semantic similarity more effectively. Embedding similar- ity utilizes vector representations for semantic alignment, while GPT score uses advanced LLMs to assess the semantic coherence between descriptions. These metrics provide more accurate means of evaluating generated descriptions against the ground truth annotations.
In our work, we employ CodeTT5+ (Wang al. 2023a), a family of encoder-decoder code foundation LLMs pre-trained on extensive software code, as the foundation to fine-tune on our dataset. Since the dataset includes hierarchical code summaries across multiple levels-iine, block, and module, providing both detailed and high-level functional descriptions- training. This begins with fine-tuning on line-level and block-level data, subsequently advancing to more complex module-level content. Such structured approach enables the model to incremen- tally build foundational knowledge, which significantly enhances its performance in both Verilog understanding and generation tasks. In Verilog understanding, DeepRTI. significantly outperforms GPT-4 across all metrics on the newly established understanding benchmark. In Verilog generation, it achieves comparable performance to OpenAl's ol -l-prvieww, the latest model designed for complex reasoning tasks including programming on the latest generation benchmark by byChang et al. (2024)).
adapt curriculum learning for
RELATED WORKS
Register Transfer Level in EDA. Register Transfer Level (RTL.) critical abstraction in EDA that describes the flow of data between registers and the logical operations on that data. This level is typically expressed using HDLs, with Verilog being the most widely used HDL in the industry. Con- sequently, the terms HDL and Verilog are used interchangeably in this work. In modern hardware design, engineers begin with specifications in natural language, which are then manually translated into HDLs before synthesizing circuit elements (Blocklove et al.) 2023). This manual translation process is prone to human errors, leading to potential design flaws and inefficiencies. Automating this translation can significantly reduce errors and streamline the design process. Recent develop-
2

---

Published as a conference paper at ICLR 2025
ments in artificial intelligence (AI) have enabled machine-baseee end-to-end translations, making this automation possible. And the ability to understand and generate Verilog code is crucial for advanc- ing this automation in hardware design. For an introduction of Verilog, please refer to Appendix [A]
LLMs for EDA. Recent advancements in LLMs have significantly impacted EDA, marking a trans- formative shift in hardware design (Chen et al. 2024). RReearchhrrs have examined the utilization LLMS for Verilog code generation, with benchmarking results presented in Thakur et al. (2023); Liu al. (2023b); [uu et al. (024) showing the potential of these models to mitigate the design chal- lenges faced by hardware developers. Furthermore, significant achievements have been achieved in fine-tuning for Verilog code generation (Chang et al.. 2024b] Thakur et al.. 2024). general RTL generation (Blocklove et al.] (023), and EDA tool script generation (Liu et al. [222aa) [Wu et al.) 2024). By reducing the need for extensive expertise in specific hardware, L.LMs enable hardware developers to quickly design intricate hardware systems (Fu et al. 2023).
Fine-tuning LLMs for Verilog Generation. Despite the great potential of ofate-of-the-art (SOTA) LLMs, e.g., OpenAl's GPT-4 (Achiam et al., (2023), in generating Verilog code, relying solely on them is insufficient due to the proprietary nature of hardware design. Besides, they are still limited in their ability to generate practical hardware designs (Fu et al. 2023). To address these limitations, recent studies have fine-tuned open-source LLMs on curated hardware design datasets (Liu et al.) 2023b Chang et al. 2024b Thakur et al., 2024 Zhang et al. 2024), which has been shown to im- prove L.MMs performance in generating Verilog code. However, effective use of L.LMs in hardware design requires high-quality, domain-specific data. Unfortunately, existing publicly available hard- ware datasets are often limited in size, complexity, or detail, hindering the effectiveness of LLMs in hardware design tasks. For example, datasets used in Thakur et al. (2023); [u et al. (2024) con- tain fewer than 200 data points, making them suitable only for benchmarking rather than effectively fine-tuning. Meanwhile, other datasets, such as those employed in [iu et (2023b); Thakur et al.) (2044), are overly simplistic, which hinder effective fine-tuming of LLMs. To improve alignment be- tween natural language and Verilog code, Chann et al. al. (2024b) translate Verilog files to an AST and then map nodes to natural language with predefined template. However, this method is limited to line-level Verilog code and the template-basce descriptions lack semantic information. Furthermore, the MG-Verilog dataset (Zhang et al.] 2024), despite featuring multi-level descriptions alongside code samples, is limited in size and its reliance on ALLMM2--7B-CHI for annotations raises quality concerns about the dataset. These alignment issues may hinder the fine-tuned LLMs' performance, leading to the generation of sonssytthesizabee or non-functional hardware source code. To address the limitations of previous studies, we introduce novel high-quality dataset that aligns natural lan- guage with Verilog code at multiple levels: line, block, and module. It includes both detailed and high-level descriptions. integrating open-source and proprictary code enhance its diversity and applicability. Unlike prior efforts focused solely on Verilog generation, we are the first to consider the crucial task of Verilog understanding. This comprehensive dataset enables the development of a unified representation model, DeepRTL, which excels in both Verilog understanding and generation, paving the way for significant advancements in hardware design automation.
Curriculum Learning. Curriculum learning is a training strategy inspired by human learning, where models are exposed to simpler tasks before advancing to more complex ones. This approach has been shown to accelerate convergence and improve model performance, particularly for tasks with hicrarchical difficulty levels. Initially introduced in Bengio et al. (2009). curriculum learning has been applied to various domains, including natural language processing (Xu et al. 2020), com- puter vision (Wang et al., (023b), and reinforcement learning (Narvekar et 22020). Recent work has demonstrated its efficacy in fine fine-tuning LLMs, where progressively increasing task complexity helps the models better capture intricate patterns (Campos) 2021). Notably, [a et al. (2024) apply curriculum learning code language models, achieving significant improvements in the accuracy of code execution tasks. In this work, we adapt curriculum learning to train DeepRTL, utilizing our structured dataset with descriptions at varying levels of detail. This approach significantly enhances the model's capabilities in both Verilog understanding and generation.
3 DATASET AND UNDERSTANDING BENCHMARK
In this section, we introduce our dataset designed to enhance Verilog understanding and generation, which aligns natural language with Verilog code across line, block, and module levels with detailed
3

---

Published as a conference paper at ICLR 2025
Original
Refined
Qualified
if
if
(higiual conmunst
Step
2042
if 2048 sokons
Filler
if
X
Comment
Entract
Commented Codk
Tinee: T3;
comtents value
Ls high
Step
Lsse: Liner en Liner-11411 11123:
Sanwarize
Step Dinil
Specification
Functional Description
'cordic sincos gene canDIC
nodule ieproving implorents cohOTC algorithe iths ca calculate sine and cosine values for given angle,
calculate and cosine vn luns for given angle specified by the Frequency Control Word (ree)
process
several handled
specific subaodulesive Nnin..
tevalvinng phase accuru lan,on, quadrant inng ROR butterfly operatiinns and output gene ratiion.
eethase Accumulat lencew The "adder" subendale adds the
Fox to the current phase accusala lator valus, lacinulating phase ......
How
Figure 1: The overview of the data annotation process. We employ the CoT approach and the SOTA LLM, GPT-4, for annotation. Annotations span three levels-line, block, and module- both detailed specifications and high-level functional descriptions.
and high-level descriptions. By integrating both open-source and proprietary code, the dataset offers diverse and robust collection that spans broad spectrum of hardware design complexities. We employ GPT-4 along with the CoT approach for annotation, achieving about 90% accuracy human evaluations, confirming the dataset's ''s high quality. We also introduce the first benchmark for Verilog understanding, setting a new standard for evaluating LLMs' capabilities in interpreting Verilog code.
3.1 DATASET SOURCE
Our dataset comprises both open-source and proprietary Verilog code. For the open-source part, we gather .v files from GitHub repositories using the keyword Veri log. These files segmented into individual modules, each representing a distinct functional unit within the Verilog design. This segmentation crucial given the limited context length of current LLMs, improving the efficiency and accuracy of the subsequent annotation and fine-tuning processes. We employ MinHash and Jac- card similarity metrics (Yan et al. 2017) deduplicate these modules and exclude those predom- inantly made up of comments or lacking complete module and endmodule structures. Finally, total of 61,755 distinct Verilog modules. For the proprietary portion, we (IPs) that enhance the variety and functional diversity of our dataset. This component includes a total 213 high-quality, industry-standard Verilog modules. These IPs not only offer range of advanced functions but also provide unique insights that complement the open-source data. Integrating these elements ensures comprehensive dataset that captures a wide spectrum of hardware design practices.
this process results incorporate a set of purchased intellectual properties
3.2 DATASET ANNOTATION
We employ different annotation strategies for open-source and proprietary code. For open-source code, we utilize the CoT approach with the SOTA LLM, GPT-4, to provide annotations at multiple levels. As illustrated in Figure [ initially remove all comments from the original Verilog code
4

---

Published as a conference paper at ICLR 2025
(resulting in refined code) to avoid training complications from incorrect misleading comments. If the token count of a complete module exceeds 2048, the maximum context length for CodeTS+, utilize GPT-4 to segment the module into smaller, manageable blocks such always blocks. If the resulting blocks still exceed 2048 tokens, we will discard them. For modules and blocks with token count below 2048 (qualified code), we then use GPT-4 to add informative comments, result- ing in commented code (Step I). From this commented code, we can extract line-level descriptions (pairings of single lines of code with natural language descriptions). To guarantee the accuracy and relevance of the inline comments, we use GPT-44-mini to rigorously check each comment, ensuring that all line-level descriptions are strictly confined to the context of their respective lines without incorporating any extraneous or irrelevant information. For example, consider the line "O =I1;" " annotated with "Assign the va lue of I1 to the output O when S is high. ". Since we cannot deduce from this single line that O is the output and S is related, such de- scriptions are deemed inaccurate and are consequently excluded from the dataset to maintain training effectiveness. In Step 2. we use GPT-4 to generate detailed specification for the commented code from Step1. This specification includes two main components: a summary of the code's s function- ality (what it does) and comprehensive explanation of the implementation process (how it works). Finally, in Step 3, we combine the qualified code from Step 1 with the detailed specification gener- ated in Step 2 to create high-level functional descriptions. To ensure precision, to focus on the qualified code, using the detailed specification only as reference. The resulting high- level descriptions succinctly summarize the code's 's functionality (what does) and provide concise overview of the implementation process (how it works). This annotation phase is the most critical and challenging as it demands that the model captures the code's 's high-level semantics, requiring profound understanding of Verilog. In current benchmarks and practical applications, users typ- ically prompt the model with high-level functional descriptions rather than detailed specifications. Otherwise, they would need to invest significant effort in writing exhaustive implementation details, making the process time-consin consuminn and requiring extensive expertise. For detailed prompts used in this annotation process, please refer to Appendix B And detailed explanation of why we discard Verilog modules or blocks exceeding 2048 tokens can be found in Appendix C
instruct GPT-4
Given the industrial-grade quality of the proprietary code, we engage professional hardware engi- neers to maintain high annotation standards. Adhering to rigorous industry-level standards, these experts ensure precise and accurate annotations, capturing intricate details and significantly enhanc- ing the dataset's value for advanced applications. Unlike GPT-generated annotations, these human- annotated ones incorporate an additional layer of granularity with medium-detail block descriptions. For detailed annotation standards and processes, please refer to Appendix D
Table 1: The overall statistics of the annotation results for our dataset.
Comment Level
Granularity
Count
Line Level
N/A
434697
High-level Description
892
Block Level
Medium- Detail Description
1306
Detailed Description
894
High-level Functional Description
59448
Module Level
Detailed Specification
59503
We present the overall statistics of the annotation results in Table Additionally, Figure illustates an example of our comprehensive annotation for a complete Verilog module. Notably, the overall dataset encompassing descriptions of various details across multiple levels is used for training. A similar work to ours is the MG-Verilog dataset introduced by /Zhang et al. (2024), including 11 11,000 Verilog code samples and corresponding natural language descriptions at various levels of details. However, it has several limitations compared to ours. Firstly, MG-Verilog is relatively small in size and lacks proprietary Verilog code, which diminishes its diversity and applicability. Secondly, employs direct annotation rather than the CoT approach, which we have found to enhance annotation accuracy as demonstrated in Section 3.3 Besides, our annotation is more comprehensive than that of MG-Verilog, which lacks granularity. We cover line, block, and module levels with both detailed and high-level descriptions, ensuring a strong alignment between natural language and Verilog code. Lastly, MG- Verilog relies on the open-source LLM LLaMA2-70B. Chat for annotation, whereas use the SOTA LLM GPT-4. In Section 4.4 we demonstrate that LLaMA2- 7-B--hhh has a poor understanding of Verilog code, leading inferior annotation quality in MG-Verilog.
5

---

Published as a conference paper at ICLR 2025
Commented Code
Line Level
initiellatiinn
scblee 0.am aww.ssi.eee teabrr
<by
sugrel
Disgard
signal fackang loel
Acllor rost sugna)
and
Instcalacetio tas signal Cective
see;
data 13,
leitt
bara capar
caeetive los)
Accept
revt, see current
wstr,
Read sttrece
Block Level
11. Nomurs Jegas bagura feg [15:8] 1. 11
High Description:
Loon varishlls for Iteractoo
imploment conditional
sperstioss allowns dynanic medifictionn of manory contenta when write aruble signal activated
neat nen see // Initializz rect ase to content enetry contents
Ivrate_onnn 12) begin 24 write crabled foctive lowt
anden
Medium-detall Decrriptinn:
Iterate throust sewer desti
ferite add 1
begin
This always blok pesposiblee for updating ned mem nagietr becca the caman vallos of mam, wrilc en. s. whee sidh and weke dats. Next, men regiser ssed to genernte men aggul. Detailed Description:
bague // write aseress watches corment unden
if tterate throgg data
and axetge dats to searyy
DATA 90070
tepin
nest neals + 11 weste_ lo // design patt nest, zea
This menory writing biuck Verilo ITI Whet dor write arblle signal 'wrii en a sative
DATAYZ2MM: apsate
sed
low, nerres throngh the momory 'ment and whites
ond
wrile daui as the matchel writ salr aadress while essuring that the data bis alian peopetly as per 1AATA, WInTH The ree of the monory contents are preserved is.
erciddule
Module Level
Fanctional Deseription:
This Vonlig mochle implements FIFO B block aonfigrabll dats withh, depth, and memory made
Speeificatiun:
This Venling module implements FIFO RAM block with configurable data with, deptt, and memory mode. It supports wnite and read operations, with die ahility to pipeline the wrlte and read addresses and dew The monory can be restt asyschooously or synchossuslly
Figure 2: An example of our comprehensive annotation for a complete Verilog module.
3.3 DATASET EVALUATION
To ensure the quality of our dataset, we assess annotations generated from the CoT process. We randomly sample 200 Verilog modules and engage four professional Verilog designers to evalu- ate the accuracy of annotations at various levels. This human evaluation indicates that annotations describing high-level functions achieve an accuracy of 91%, while those providing detailed speci- fications attain an accuracy of 88%. For line-level annotations, the accuracy is 98%. Additionally, we compare the CoT method with the direct annotation approach, where annotations are generated straightforwardly from the original code. This direct annotation method yields only 67% accuracy, highlighting the significant advantage of integrating the CoT process.
Recent studies in natural language processing (NLP) have demonstrated that LLMs fine-tuned with synthetic instruction data can better understand natural language instructions and show improved alignment with corresponding tasks (Wang et al. [022] Ouyang et al.) 2022 Taori al.] 2023). It is important to note that in our work, we also utilize data generated by language models for fine-tuning, including annotations at various levels. While not all annotations are perfectly accurate, we achieve a commendable accuracy of approximately 90%. Motivated by Waag et al. (2022), we treat those inaccuracies data noise, and the fine-tuned model this dataset still derives significant benefits.
3.4 UNDERSTANDING BENCHMARK
As the first work to consider the task of Verilog understanding, we introduce a pioneering bench- mark to evaluate LLMs' capabilities in interpreting Verilog code. This benchmark consists of 100 high-quality Verilog modules, selected to ensure comprehensive coverage of diverse hardware func- tionalities, providing a broad assessment scope across different types of hardware designs. We have engaged four experienced hardware engineers to provide precise annotations on each module's 's func- tionalities and the specific operations involved in their implementations. These initial annotations are then rigorously cross- versffee by three additional engineers to guarantee accuracy and establish high standard for future model evaluations. This benchmark fills critical gap by providing stan- dardized means to assess LLMs on interpreting Verilog code and will be released later. For detailed examples included in the benchmark, please refer to Appendix F

---

Published as a conference paper at ICLR 2025
Instruction Constraction
Example
Instruction:
High-Lav
Medale Level
Lenerata a Mloesterel suerany for Veritog modile.
Ganemiion
<herines Verilig exoules
Ourpot:
Maliundectei
Block Levd
tuue wodule surnery
Unikrdtndding
Instruction:
fased or the decaiieed baock sumwary, coda for the descrabed bllako
Veriloy
Detiled
Level
teputt
<betailed lock summary by hurare
Outpat:
Task
Gramularity
Comment Level
Curriculum Leaming
Deseription
To
Single
Complete Module
Human-
Detailed
High-level
GPT-armotated
Human-annorased
Figure 3: The overview of the instruction construction process and the curriculum learning strategy. For instruction construction, we integrate various settings, e.g., task type, granularity, and comment level, create tailored instructions for specific scenarios. The curriculum learning strategy involves three hierarchical stages: training progresses from line-level to module-level code (1st stage), tran- sitioning from detailed to high-level descriptions at each level (2nd stage), and advancing from GPP-annotated to human-annotated descriptions for each granularity (3rd stage).
MODEL AND EVALUATION
In this section, we introduce DeepRTL and elaborate on the preparation our instruction tuning dataset and how adapt curriculum learning for training. Additionally, detail the benchmarks and metrics used to evaluate our model's 's performance both Verilog understanding and generation tasks. To accurately assess the semantic precision of the generated descriptions, we take the initiative to apply embedding similarity and GPT score for evaluation, which are designed to quantitatively measure the semantic similarity between the model's 's outputs and the ground truth.
4.1 MODEL.
In our work, we have chosen to fine-tune CodeT5+ (Wang et al. 2023a), a family of encoder-decoder code foundation LLMs for wide range of code understanding and generation tasks. CodeT5+ em- ploys a "shallow encoder and deep decoder" architecture (Li al. 2022), where both encoder and decoder initialized from pre-trained checkpoints and connected by cross-attention layers. We choose to fine-tune CodeTT5 for its extensive pre-training on a vast corpus of software code, with the intent transfer its acquired knowledge to hardware code tasks. Also, the model's flexible archi- tecture allows for the customization of various training tasks, making it highly adaptable for specific downstream applications. Furthermore, CodcT5+ adopts an efficient fine-tuning strategy where the deep decoder is frozen and only the shallow encoder and cross- cross-attention layers are allowed to train, significantly reducing the number of trainable parameters. Specifically, we have fine-tur tune-tnedd two ver- sions of CodeT5+, 16b), on our dataset, resulting in DeepRTL-220m and DeepRTL- respectively. For more infor- mation on the model selection, please refer to Appendix F
(CodeTT5+222))
and instructcodet5p- 16t (CodeT5++-
4.2 INSTRUCTION TUNING DATASET
During the fine-tuning process, adopt the instruction tuning strategy to enhance the adaptability of LLMs, which is particularly effective when handling diverse types of data and tasks. Given that

---

Published as a conference paper at ICLR 2025
dataset features descriptions at multiple levels and our model is fine-tuned for both Verilog un- derstanding and generation tasks, there is diversity in both the data types and tasks. To accommodate this diversity, we carefully design specific instructions for each scenario, ensuring the model can ad- just its output to align with the intended instructions. Figure 3 illustrates how we combine various settings, e-g., task type, granularity, and comment level, to construct tailored instructions for each specific scenario, fostering a structured approach to instruction-based tuning that optimizes the fine- tuning efficacy. For details on the instructions for different scenarios, please refer to Appendix G
4.3 CURRICULUM LEARNING FOR DEEPRTL
We adapt curriculum learning for the fine-tuning process, leveraging structured dataset that fea- tures descriptions of various details across multiple levels. Initially, the model is fine-tuned on line- level and block-level data, subsequently progressing module-Ievel data. At each level, we start by aligning the detailed specifications with the code before moving to the high-level functional descrip- tions. And fine-tuning typically starts with GPT-annotated data, followed by human-annotated data for each annotation granularity. [ggure]]provide egy because particular focus is placed on aligning Verilog modules with their high-level functional descriptions, which poses the greatest challenge and offers substantial practical applications. This curriculum learning strategy enables the model to incrementally build knowledge from simpler to more complex scenarios. As result, the models demonstrate impressive performance across both Verilog understanding and generation benchmarks. Note that we exclude the cases in the bench- marks from our training dataset. We primarily follow the instruction tuning script of CodeTS the fine-tuning process, with a modification to expand the input context length to the maximum of 2048 tokens. We utilize the distributed framework, DeepSpeed, to efficiently fine-tune the model across a cluster equipped with eight NVIDIA A800 GPUs, each with 80GB of memory. During inference, we adjust the temperature to 0.8 for understanding tasks and to 0.5 for generation tasks, while other hyperparameters remain at their default settings to ensure optimal performance. Further details the adopted curriculum learning strategy are provided in Appendix H
an illustration of this process. We adopt such strat-
4.4 UNDERSTANDING EVALUATION
For evaluating LLMs' capabilities in Verilog understanding, we utilize the benchmark introduced in Section 3.4 The evaluation measures the similarity between the generated descriptions and the ground truth summaries. Previous works usually use BLEU (Papineni et al. 2002 and ROUGE (Lin) 2004) scores for this purpose (Wang et al. 2023a). BLEU assesses how many n-grams, i.e., se- quences of words, in the machine-generated sion). In contrast, ROUGE counts how many n- r-grams from the reference appear in the generated text (focusing on recall). However, both metrics primarily capture lexical rather than semantic similarity, which may not fully reflect the accuracy of the generated descriptions. To address this limitation, we take the initiative to apply embedding similarity and GPT score for evaluation. Embedding sim- ilarity calculates the cosine similarity between vector representations of generated and ground truth descriptions, using embeddings derived from OpenAl's satt-embedding---aaagge model. Meanwhile, GPT score uses GPT-4 to quantify the semantic coherence between descriptions by assigning sim- ilarity score from 1, where indicates perfect semantic alignment. These metries provide a more nuanced evaluation by capturing the semantic essence of the descriptions, thus offering more accurate assessment than previous methods. For details on the prompt used to calculate the GPT score, please refer to Appendix !
generated text appear in the reference text (focusing on preci-
4.5 GENERATION EVALUATION
To evaluate LLMs' capabilities in Verilog generation, we adopt the latest benchmark introduced by Chang et al. (2024a), which an expansion based on the previous well-established RTLLM benchmark (Lu et al. (2244). The benchmark by Chang et al. (2024a) encompasses a broad spectrum of complexities across three categories: arithmetic, digital eircuit logic, and advanced hardware designs. This benchmark extends beyond previous efforts by incorporating a wider range of more challenging and practical Verilog designs, thus providing a more thorough assessment of the models' capabilities in generating Verilog code.
https://gi http://ggtttuubuo thub. com/ http:/alesssoccoo wwweeeecr//eeeeeeeeeeeeeeeeeeeee
8

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results Verilog understanding using the benchmark proposed in Section 3.4
BLEU-4 denotes the smoothed BLEU-4 score, and Emb. Sim. represents the embedding similarity metric. Best results are highlighted in bold.
Model
BL.U-4
ROUGE-1
ROUGE-2
ROUGE-I.
Emb. Sim.
GPT Score
GPT-3.5
4.75
35.46
12.64
32.07
0.802
0.641
GPT-4
5.36
34.31
11.31
30.66
0.824
0.683
-preevie
6.06
34.27
12.25
31.01
0.806
0.643
CodeT5+-220m
0.28
7.10
0.34
6.18
0.313
0.032
CodeT3+-16b
0.10
1.37
0.00
1.37
0.228
0.014
LLaMA2-70B-CHE
2.86
28.15
10.09
26.12
0.633
0,500
DeepRTI--220m 1-2pmir--22m
11.99
40.05
20.56
37.09
0.793
0.572
DeepTTL-16b-diret
11.06
38.12
18.15
34.85
0.778
0.533
DeepRTI.-222 -220mm DeepRTI.-16h
18.66
47.69
29.49
44.02
0.837
0.705
18.94
47.27
29.46
44.13
0.830
0.694
The evaluation focuses on two critical aspects: syntax correctness and functional accuracy. We use the open-source simulator iverilog (Williams & Baxter 2002) to assess both syntactic and functional correctness of Verilog code generated by LLMs. For the evaluation metric, we adopt the prevalent @ass@k l metric, which considers problem solved if any of the & generated code samples pass the compilation or functional tests (Pei et al.] 2024). For this study, we set k values of and 5, where higher Pass @ l score indicates better model performance. To further delineate the models' capabilities, we track the proportion of cases that pass out of 5 generated samples and compute the average the success rate. For syntax correctness, this success rate measures the proportion of code samples that successfully compile and, for functional accuracy, the fraction that passes unit tests.
EXPERIMENTAL RESULTS
5.1 BASELINE MODELS
For the baseline models, we select OpenAl's GPT-4-turbo (GPT-4) and GPT-3.5-ture (GPT-3.5), as well as the ol-preview model, OpenAl's latest reasoning model designed to address complex problems across diverse domains, including programming. These models are chosen for their status as the most advanced Verilog generation (Chang et al. 2024bb Thakur et al., 2024 Liu et al. 2024). For comparison with models specifically fine-tuned on Verilog, please refer Appendix
LLMs currently available, with demonstrated excellence in
5.2 VERILOG UNDERSTANDING
As shown in Table DeepRTT. consistently outperforms GPT-4 across all evaluation metrics. Tradi- tional metrics like BLEU and ROUGE offer inconsistent assessments due to their inability to capture semantic similarity accurately: while DeepRTL-166 excels in BLEU-4 and ROUGE-L, DeepRTL- 220m leads in ROUGE-1 and ROUGE-2. In contrast, embedding similarity and GPT score provide a more accurate assessment of the models' capabilities understanding Verilog code. Compared to CodeT5+, the performance of DeepRTL-direct, which is trained directly without curriculum learn- ing, highlights the effectiveness of our dataset. And the subsequent improvements when employ- ing the curriculum learning strategy underscore its benefits. Additionally, the poor performance of LLaMA2-70B-CHI underscores the unreliability of the MG-Verilog annotations Zhang et al.) (2024). To further validate our model's performance, that DeepRTL-22000 GPT-4, and ol ol-preview achieve accuracies of 78%, 72%, and 67%, respec- tively. These results align closely with the embedding similarity affirming the effectiveness of these evaluation methods. We find that DeepRTL-220m outperforms DeepRTL-16b, likely due to the CodeTS+-220m's pre-training on a large corpus of paired soft- ware code and natural language data, which fosters better alignment between code and language. In contrast, CodeT5+-168 is primarily pre- pre-trained on software code data and then fine-tuned on synthetic instruction data, lacking robust code code-language alignment. Moreover, despite its smaller size. DeepRTL- -20m surpasses many billion- parommetr models, underscoring the high quality of dataset and the effectiveness of the curriculum learning strategy.
have conducted human evaluations, which show
GPT score metrics, further

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results Verilog generation. Each cell displays the percentage of code samples, out of five trials, that successfully pass compilation (syntax column) or functional unit tests (function column). Best results are highlighted in bold.
Benchmark
GPT-35
GPT4
of ol-pevview
DeupRTI-220m
DespRTI
syлаax
syntax
Johmonn,Cummie
40SS
0%
100%
D74
10075
1007S
0%
100%
ziu
0%
0%
0%6
016
0%
0%
0%
0%
076
0%
devect
605
275
100%
100%
100%
075
1002
0%
100%
0%
lioq_div
100%
UR8
0%
100n5
0%6
Logic
mux
100%
100%
100%s
40%
100%
100%
10005
100%
100%
100%
Parallel2serial
90%
0%
100%
0/
100%
0%
10075
0%
100/
176
40%
100%
100R5
100%
nabttshfter
60%
100%
1001
100%
100%
Sprral2prallle
605
0%
100%
07A
100%
20%
10075
0%
1007
17%
Wto 16
100%
0%
2016
196
100%
096
100%
0%
100%5
056
100%
0%
40%
1009
0%
00%
2075
2074
407
40%%
1007
0%
607
806
5076
08
07A
100%
100%
1007
100%
100%
100%
stdey_ 3256
T006
4275
1856
100%
079
1079
100P5
Arithmetic
1076
1006
407%
407%
100%
100%
1076
7076
10026
8026
dav Tihh
092
106
115
075
098
096
026
025
molii TRbil
808
09%
100%
2074
100%
100%
100%
0%
100%
02%
mulli lipe
678
205
100%
100%
100%
1005
1007
100%
1007%
100%
molh moliipipe.Bit Ahit
08
0%
08
07A
07%
08
08
0%
1/6
07%
409
40%
80F6
80%
1001
100%
100%
100%
1x4eystolia
1006
100%
100%
100%
100%
100%
100%5
10025
10035
282aystolic
096
098
016
096
016
09
0%6
0%
0%6
0%
4s4epuitilace
09
05%
195
075
0PA
07
096
075
Advanced
100%
100%S
macge
0%
016
0%
0%
Sataie fum
1006
09
1007
60%
100%
100%
100%
100%
1009
100%
Saune Full
20%
0%
80%
20%
100%
1009
100%
100%
100%
100%
4stane
40%
100%
100%
100%
2stite
100%
80F5
375
Sucuess Rose
60.65%
2000%
63.87%
27.74%
78.06%
38,055
78.06%
38.13%
76.13%
38.6%
Pass
32.369
51.61%
29,038
74.19%
70.07%
82.366
Pass
80.65%
33.48%
77.42%
45.18%
30.65%
31.6%%
80.65%
41.496
77.42%
38.77%
5.3 VERILOG GENERATION
Given the inferior performance of models like CodeTT5 and DeepRTI--direct in the Verilog under- standing task, comparison focuses the GPT series models. As shown in Table 3 OpenAl's ol-preview, the latest model designed to tackle complex tasks including programming, achieves the highest performance across all metrics. Nevertheless, our DeepRTL model exhibits comparable per- formance to ol -prevviww on several metrics and significantly surpasses GPT-4 in syntax correctness, Pass@1 1 functional accuracy, and overall functional success rate. Notably, DeepRTL consistently generates highly accurate code in successful cases, often achieving 100% pass rate among the five generated samples, which underscores its reliability for practical applications. Furthermore, con- sidering that OpenAl's models benefit from vast parameter sizes and extensive pre-training across diverse datasets, the performance of our more compact DecpRTL model is particularly impressive.
CONCLUSION
In this work, we introduce DeepRTL, novel unified representation model that bridges Verilog un- derstanding and generation. It is fine-tuned on meticulously curated dataset featuring multi-level natural language descriptions of Verilog code, encompassing line, block, and module levels with both detailed and high-level functional descriptions. DeepRTL not only addresses the gaps in previ- ous methods focused solely on Verilog code generation but also ensures strong alignment between Verilog code and natural language. Moreover, establish the first benchmark for evaluating LLMs' capabilities in Verilog understanding. To overcome the limitations of traditional metrics like BLEU and ROUGE, which primarily assess lexical similarity. we apply embedding similarity and GPT score for evaluating the model's understanding capabilities. These metrics are designed to evaluate the semantic similarity of deseriptions more accurately, thus better reflecting the precision of gener- ated descriptions. By implementing curriculum learning strategy, DeepRTT. demonstrates superior performance in both Verilog understanding and generation tasks. Specifically, it surpasses the SOTA LLM, GPT-4 across all understanding metrics and achieves performance comparable to OpenAl's ol -reviiw in terms of syntax correctness and functional accuracy for Verilog generation.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the General Research Fund of the Hong Kong Research Grants Council (RGC) under Grant No. 14212422 and 14202824, and in part by National Technology Innovation Center for EDA.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1ige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. Chip-chat: Challenges and opportunities in conversational hardware design. In 2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD), pp. 1-6. IEEE, 2023.
Daniel Campos. Curriculum learning for language modeling, arXiv preprint arXiv:2108.02170. 2021.
Kaiyan Chang, Ying Wang, Haimeng Ren, Mengdi Wang, Shengwen Liang, Yinhe Han, Huawei Li, and Xiaowei Li. Chipgpt: How far are we from natural language hardware design. arXiv preprint 2005 2023.
Kaiyan Chang, Zhirong Chen, Yunhao Zhou, Wenlong Zhu, Haobo Xu, Cangyuan Li, Mengdi Wang, Shengwen Liang, Huawei Li, Yinhe Han, et al. Natural language is not enough: Benchmarking multi-modal generative for verilog generation. arXiv preprint arXiv:2407.0888 08473, 2024a.
Kaiyan Chang, Kun Wang, Nan Yang, Ying Wang, Dantong Jin, Wenlong Zhu, Zhirong Chen, Cangyuan Li, Hao Yan, Yunhao Zhou, et al. Data is all you need: Finetuning Ilms for chip design via an automated design-data augmentation framework. arXiv preprint arxiv:2003. /202, 2024b.
Lei Chen, Yiqi Chen, Zhufei Chu, Wenji Fang, Tsung- Yi Ho, Yu Huang, Sadaf Khan, Min Li, Xingquan Li, Yun Liang, al. The dawn of ai-native eda: Promises and challenges of large circuit models. arXiv preprint arXiv:2403.072577 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2xXiv2247..27788 2024.
Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, and Yingyan Celine Lin. Gpr4aigchip: Towards next-generation ai accelerator design automation via large language models. In 2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD), pp. 1-9. IEEE, 2023.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi. Yu Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming- the rise of code intelligence. arXiv preprint arXiv:2401.141994 2024.
Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis. and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436. 2019.
Bu Jin, Xinyu Liu, Yupeng Zheng, Pengfei Li, Hao Zhao, Tong Zhang, Yuhang Zheng, Guyue Zhou, and Jingjing Liu. Adapt: Action-aware driving caption transformer. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7554-7561. IEEF, 2023.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser. Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition- code generation with alphacode. Science, 3786622)) 2022.

---

Published as a conference paper at ICLR 2025
Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 74-81,22 2004.
Mingjie Liu, Teodor-Dumine Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. Chipnemo:
Nomain-adapted Ilms for chip design. arXiv preprint arXiv:2311.00111 2023a.
Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. Verilogeval: Evaluating large language models for verilog code generation. In 2023 IEEE/ACM International Conference Computer Aided Design (ICCAD), pp. 1-8. IEEE, 2023b.
Shang Liu, Wenji Fang, Yao Lu, Jing Wang. Qijun Zhang, Hongee Zhang, and Zhiyao Xie. Rtlcoder: Fully open-source source and efficient llm-assisted rtl code generation technique. IEEE Transactions on Aodd Design of Integrated Circuits and Systems, 2024.
Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. Rtllm: An open-source benchmark for design rtl generation with large language model. In 2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC), pp. 722-727. IEFF, 2024.
Marwa Na, Kamel Yamani, Lynda Lhadj. Riyadh Baghdadi, et al. Curriculum learning for small code language models. In Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 4: Student Research Workshop), pp. 531-5431- 2024.
Sanmit Narvekar, Bei Peng. Mattco Lconetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone.
Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-50, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Hammond Pearce, Benjamin Tan, and Ramesh Karri. Dave: Deriving automatically verilog from english. In Proceedings of the 2020 ACM/IEEE Workshop Machine Learning for CAD, pp- 27-32, 2020.
Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. Betterv: Controlled verilog generation with discriminative guidance. arXiv preprint 2XXi:24422.03375 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt and Siddharth Garg. Benchmarking large language models for automated verilog rtl code generation. In 2023 Design, Automation & Test in Europe Conference & Exhibi- tion (DATE), pp. 1-6. IEEE, 2023.
Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan Gavitt, Ramesh Karri, and Siddharth Garg. Verigen: A large language model for verilog code generation. ACM Transactions Design Automation of Electronic Systems, 29(3):1-31, 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct Aligning language models with self-generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07990 07922,, 2023a.
12

---

Published as a conference paper at ICLR 2025
Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficient- train: Exploring generalized curriculum learning for training visual backbones. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5852-58644 2023b.
Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative Ilm-agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15077- 15087, 2024.
Stephen Williams and Michael Baxter. Icarus verilog: open-source verilog more than year later. Linux Journal, 2002(999;3, 2002.
Haoyuan Wu, Zhuolun He, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, and Bei Yu. Chateda: A large language model powered autonomous agent for eda. IEEE Transactions Computer-Aided Design of Integrated Circuits and Systems, 2024.
Benfeng Xu, Licheng Zhang, Zhendong Mao, Quan Wang, Hongtao Xie, and Yongdong Zhang. Cur- riculum learning for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. -195-6104, 2020.
Ziqi Yan, Jiqiang Liu, Gang Li, Zhen Han, and Shuo Qiu. Privmin: Differentially private minhash for jaccard similarity computation. arXiv preprint
2017.
Yongan Zhang, Zhongzhi Yu, Yonggan Fu, Cheng Wan, et al. Mg-verilog: Multi-grained dataset towards enhanced llm-assisted verilog generation. arXiv preprint arx1v:200..01910, 2024.
13

---

Published as a conference paper at ICLR 2025
A INTRODUCTION OF VERILOG
Verilog is the most widely used hardware description language (HDL) for modeling digital inte- grated circuits. enables designers specify both the behavioral and structural aspects of hardware systems, such as processors, controllers, and digital logic circuits. Verilog operates at relatively low level, focusing on gates, registers, and signal assignments- each representing physical hardware components. While Verilog supports behavioral constructs (e.g,, i if-else, case) that are some- what similar to software programming languages, their use constrained by synthesizable coding styles required for hardware implementation. Verilog differs from software programming languages like Python and C++ in several key ways:
1. Parallelism: Verilog inherently models hardware's concurmet nature, with multiple state- ments executing simultaneously. In contrast, software languages like Python typically fol- low a sequential execution model.
2. Timing:
Timing is fundamental concept in Verilog that directly influences how digital circuits are designed and simulated. Verilog relies on clocks to synchronize sequential logic behaviors, enabling the precise modeling of synthronous circuits. In contrast, software programming languages generally do not have an inherent need for explicit timing.
3. Syntax and Constructs: Verilog's syntax is tailored to describe the behavior and structure of digital circuits, reflecting the parallel nature of hardware. Key constructs of Verilog include:
The basic unit of Verilog, used to define a hardware block or component. Each module Verilog encapsulates inputs, outputs, and internal logic, and modules can be instantiated within other modules, enabling hierarchical designs that mirror the complexity of real-world systems. And each module instantiation results in the generation of a corresponding circuit block.
Modules:
Always block: In an al ways block, circuit designers can model circuits using high- level behavioral descriptions. However, this does not imply that a broad range of pro- gramming language syntax is available. In practice, Verilog supports only limited subset of programming-likk constructs, primarily f- elss se and case statements. Statements in multiple a.l. alwae ways blocks are executed in parallel and the resulting cir- cuit continuously performs its operations.
Sensitivity list: In an a always block, the sensitivity list specifies the signals that trigger the block's execution when they change.
Assign statements: assign statements are used to describe continuous assignments of signal values in parallel, reflecting the inherent concurrency of hardware.
Registers (reg) and Wires (wire): reg is used for variables that retain their values (e.g., flip-flops or memory), and wire is used for connections that propagate values through the circuit.
In contrast, software programming languages like C, Python, Java employ more con- ventional syntax for defining algorithms, control flow, and data manipulation. These lan- guages use constructs like loops (for, while), conditionals (if, else), and functions methods for structuring code, with data types such integers, strings, and floats for variable storage.
B PROMPT DETAILS FOR COT ANNOTATION
As shown in Figure we present the detailed prompts used in our annotation process. For each task, we supplement the primary prompt with several numaan-rvviewed input-output pair examples, serving in-context learning examples to enhance GPT's understanding of task requirements and expectations. These examples will serve as guidance for the model to correctly interpret and execute tasks in accordance with the prompt, ensuring more accurate and contextually relevant outputs.
14

---

Published as a conference paper at ICLR 2025
System Prompt
You expert handware design with deep inderstanding of Verilog RTL Now lange langinge model code genenttio tioks specifically fonssal on Varilag RTI. codes And I want you to hep me procesa the new dists, transforming into suitaall formatt that con serve as imputs and outpuis for the model.
Qualified Code to Commented Code Prompt
Add detailed comments the following Verilog code, explaining each statement's purpose and fiunctionality.
Commented Code to Specification Prompt
As an experienced digital IC design esginess, you've been tasked with exeating comprhhenaive desiga spacification document for the RTL module provided above. Platse analyze the RTL content and generate detailed document with the following siructure:
Modale Nome: Clearly
official
module.
2. Module Fumetionality: Provids thorough description the modale's i primary functions and objectives, Include the types of sigmals it cun
process, behavior under nomaal operating conditions, und any potestaal exception handling mechanisms.
Ouput/Output Port Speci ficatio:: list and deserite all input and output signals, control signals, and status signals. For each, spocify the datt type.
level, timing séationsilass parpose,
goves ning their changes.
Iniernal Working Principles Explain the module's interal mechanisms, Describe logical implementation process, including:
Division of logic functionality (combinational and sequential logic)
Signal data flow and control flow
Special functional units (c.g. registees, anthmeeic logic units, state machines)
Algorithm and Loge Implamentaion Detaik:: Provide detailad explanation of the spacific logie implemented intemal code blocks. This may
invelve algorihmin principles, implemenationn sieps, and parameter configuratins.
Insemall Signal Definitione: ist explein signals defined within the module.. Include their purposes, lifeeyeles, miles goveming their changes, relationships with other module componentts.
Please ensure that your explanation is clear, conciso, and technically accurate, suitable for other esgineers to understand and isplemment the design.
Specification to Functional Description Prompt
Please act as professional Verilog designer. Your task is to summarize concise yet accurate flacctiona description of the provided V Verilog code ('oadé' part). along with specification ('specificatioo) purt). This description should offer detailed explanation of the overall functionality, highlightnng the key elemenns operations the code.
Summarize the funntionnllt of the Varilog cade using the 'oodd' pust na the grimary referenes, with the 'ppeeificationn part serving as supelementary information. In case of any conflicts between the two, petuntize the information from the 'codé' part. When summariving descrigtion, keep following points in mind:
Focus clarity precissio, ensuring
description
easy to understand technicallly socurate.
2. Use professional tentinology related to HEDAA RTL.
3. Aveid detiiled operation comments, and insterd focus on the overall fanctionality and purpose of the module. explicitly
implanented within the Do add about the module's chameteristies or behavior that are not directly evident from code.
include information
information make secamptions
5. If subasduuee are found in the uput code, provide combined descrption of the general flnctionniiies of all suhmodules. Avoud detailed internal logio descriptions within submodules and use quilifiers like "mushe" prevent specific finctional coumpttons besed on names. Integrate the functionality descriptions all submsdules into overall module flnctional deserition.
Aveid speculation
spacific product medels, such GPU munufietuer and medel numbers.
7. Do not menionn any decailed internal logic variable transfomaaioons.
The description should include snewers the following question with professional temms/terms in Verilog/DAA..TI on functionality implemented by the provided Verilog code. accurately reflects the specific opecaion or module deseriledd by the code. module
the professional or functional term this code? Please ensure reply
If code canner be deseribed in terms of functionality, such as in cases of interface definitionn or simple variable definitiuns, do not infer based on variablle queeifiction providad, Instead, formet oulput generating response:
with does not perform any logical functions." refor the following example
Must respoed in one coculeet sentence and do not use bullet points or numbered lists, Try keep the length answer under 100 tokens. Format the ourput to begin with: The module implements...
Hese's example output iormat
referesee:
module (module name] implements [ovenil finction description] [Oytional: If cannot be deseribad in terms funetionality such as eases interfice definitons or simple variable deciniionss does not perform any logical functions.
Filtering Line-level Description Prompt
You expert handware design with deep inderstonding of Verilog RTL codes. Now I generation task specifically focused Verilog
lange langugge model for code comment
codes.
Your tisk evaluste relevance the code and cortespondngg commest. Please note the commentt should be diveetly related to code, and the comment should be explicifly staeed the code I provide.
Do not include any speculative information, such inferring funerionality from variable names that the code itself does not explicilly show. If so, generate "Trae" otherwise, generate ""aalse", Your answer should stictly "Trr"" "aalse" other content allowed generated.
Figure 4: Detailed prompts used in the CoT annotation process.
C DISCARDING VERILOG CODE EXCEEDING 2048 TOKENS
In the main submission, we state that Verilog modules and blocks exceeding 2048 tokens are ex- cluded, as 2048 is the maximum input length supported by CodeTT+. Beyond this limitation, several additional factors motivate this decision:
15

---

Published as a conference paper at ICLR 2025
Distribution of Code Token Lengths of the Verilog Generation Benchmark
Mean: $12.8
Mediam: 009.5
10
Frequency
250
500
750
1000
1250
1500
1750
Number of Tokend
Figure 5: The distribution of the token lengths of the generation benchmark by Chang et al. (2024a).
1. Generation Capabilities of Existing LLMs Are Limited to Small Designs
Existing benchmarks for Verilog generation, including the one used in our work (Chang et al. 2024a), do not include designs exceeding 2048 tokens, with the maximum token length observed in the benchmark being 1851. As shown in Table 3 of the main submis- sion, even the state-of-ther -ortt---rttttt LLM, ol -preview, is capable of accurately generating only simple designs and struggles with more complex ones. Figure Jiluutajillustratess the token length distribution across the benchmark, further justifying our decision to exclude Verilog mod- ules and blocks exceeding 2048 tokens.
2. Segmentation Common Practice
Segmenting longer code into smaller chunks that fit within the predetined context window and discarding those that exceed it is a widely accepted practice in both Verilog-related research (Chang et al. [0224b Pei et al. 2024) and studies on software programming lan- guage (Wang et al., 2023aa. This approach ensures compatibility with current LLMs while maintaining integrity and usability of dataset. It worth noting that the default maximum sequence length in CodeT5+ is 512 tokens, and our work extends this limit to 2048 tokens to better accommodate Verilog designs.
3. Empirical Findings and Practical Challenges
Our experiments reveal important empirical observation: existing LLMs, such as GPT- 4, consistently produce accurate descriptions for shorter Verilog modules but struggle with correctness when handling longer ones. Specifically, During the annotation process, we divide the dataset into two sections: Verilog designs with fewer than 2048 tokens, and de- signs with token lengths between 2048 and 4096 tokens. Our human evaluation finds that descriptions for Verilog designs with fewer than 2048 tokens approximately 90% accu- rate, while descriptions for designs with token lengths between 2048 and 4096 tokens have accuracy rates of only 60%-70%. And accuracy further decreases for designs exceeding 4096 tokens. Since our datasets rely on --e-geeeettedd annotations, restricting the dataset Verilog modules within the 2048-token limit helps maintain the quality and accuracy of annotations. This, in turn, facilitates higher-quality dataset creation and more efficient fine-tuning. For the potential negative impact of incorporating Verilog designs larger than 2048 tokens, please refer to Appendix K And we examine the impact of varying context window lengths in Appendix
D STANDARDS AND PROCESSES FOR MANUAL CODE ANNOTATION
Given the industrial-gret grade quality of the proprietary code, employ professional hardware engi- neers for manual annotation. We have established the following standards and processes to guide engineers in crafting accurate and detailed descriptions with example annotations shown in Figure
16

---

Published as a conference paper at ICLR 2025
1. Standards:
The hardware engineers required to provide descriptions at both the module and block levels.
For module-level descriptions, two levels are defined:
i. H (High-level): The role of this module in the overall design (IP/Chip).
11. D (Detailed): What functions this module performs (overview) and how it is im- plemented (implementation details). This description should adhere to a top-down structure and consist of approximately 2-5 sentences.
Note: If the summary statements for H and D are identical, both must be provided. For block-level descriptions, particularly a ways blocks, descriptions are required at three distinct levels:
(High-level): The role of this block in the overall design (e.g., across modules). i. M (Medium-detail): Contextual explanations.
111. D (Detailed): Descriptions specific to the block following top-down structure.
If details are absent, they may be omitted; do not guess based on signal names.
2. Processes:
Initially, we provide engineers with set of descriptions generated by GPT-4 for reference. They are then expected to revise and enhance these GPT-generated descriptions using their expertise and relevant supplementary materials, such as README files and register tables.
D _ap__pprt_bbeeeeeeeeeeeeee uar___cns7...
Module--evel comment
This module impleants the uant PxrO real and write lugit.
// Dr. This Verilog nodute tap tements par paraneterized RMN with confiqurable sata width, death, and mcmory wode, It supoorts both read Ludes
speral
wiility pipelime iun read and write iinalls besed exeury modie, suttpllever function sipelinang Lindnn signalls based the clock and reset signals.
combinationol. processes
It
sequential process for upsating the nenory and sipe
D_aob_uart_bo
rstui,, EndtLn,
wr. wndr, wndr.
Black-ievel comment
17 20 This always block Lop lements the logi: of writing to the FIFO butfer whan recesving choracter presaring for transmitting
// This Verilin code denigg to the uplate proces
saed fneet see with contit iomel write
opera: iuns. inltially, "nect mea is ass ansigned the value of "mes'. A arite operation to "rest ave" is only performed if "write unn active tow. The aemory adoress wstching 'erste adar" is upsated with value of "write werite data" Here, data is uritten bit-by-bit to Increnents
"DATA MIDIe wfter each iteration.
always S(men writscmnnn or aräie_addr writu, begin PHDC mood_ 124101
3:
noet__ee som mem:
1f
1'bel
for 8:
1 DEPTI:: 1 1) begin
burite_ador MTomi;
moxt_ront Nr begin and
WITAL
and and
endnodule
Figure 6: Human-annotated examples for the proprietary code.
E EXAMPLES OF VERILOG UNDERSTANDING BENCHMARK
To construct a high-quality benchmark, we first remove comments from the original code, and then submit it to experienced hardware engineers for annotation, ultimately producing the code and de- scription pairs as shown in Figure
17

---

Published as a conference paper at ICLR 2025
notele Mht_rlog_f_ceeccrr..e
acovle peis_ 21--127--111111---111 Lestcff_iinntoo #1 paramster PATTERCKEOIT
rodile Event/Niset snpet
5'165,
paraecter TC0
polppt rising_edge,
cutpat rallangedpp,, 1:
1:
input insut
cls,
locatzeras FLOP_DELAY 192;
rop INATEMULLDIII11: patterr_e
(1:9] rep_i 2110;
cutput 6:
1:
aasige rising (ore
baays glazsecge clkl began
relifel;
ratiang_ooge retliii
Qualified code
inpet EUR:
if(ratl began
61-reaifal::
(1-re_.)1))) regi(e)) (reg_i (110
TAATHRB_NDDTHE N2DTHEL uoll;
inotial 2'e0;
end eise begin pattem_o wTCo patters_o
giposedpe
(cun) to; andmodule DrLOP_DELAY
begin
raq_ate) and
(c2)
androda
with clack snable ane asynchrerous clear scrulation dolay for data capture on rasing eage clock."
esoule
signal, resesting increworting it by deassertion rot casal EML with da
The noda le
unploworts countor that oxerates the rising coge
ff onte"
transstions of the irput signal, edges, falling edgea, and bath based on the sams led ieput sagnak at
acoale Exent_Paaaa Lements detector for
talning edge
Manually verified description
Figure 7: Examples from the Verilog understanding benchmark.
F MODEL SELECTION
In this work, choose CodeT5+, family of cncoder-decoder code foundation models, as the base model for training DecpRTL for two primary reasons. First, as we aim to develop a unified model for Verilog understanding and generation, T5-like models are particularly well-suited due to their ability to effectively handle both tasks, as evidenced by Wang et al. (2023a). Second, the encoder component of CodeT55+ enables the natural extraction of Verilog representations, which can be potentially utilized for various downstream tasks in EDA at the RTL stage. Examples include PPA (Power, Performance, Area) prediction, which estimates the power consumption, performance, and area of an RTL design, and verification, which ensures that the RTL design correctly implements its intended functionality and meets specification requirements. Both tasks are crucial the hardware design process. This capability distinguishes it from decoder-only models, which are typically less suited for producing standalone, reusable intermediate representations. In future work, we plan to explore how DeepRTLL can further enhance productivity in the hardware design process.
To further demonstrate the superiority of CodeT5+ as base model, we fine-tune two additional models,
1, (deepseek-coder) (Guo et al. 2024h and Llama-3.2- BB Instruc (1lama-3.2) (Dubey al. 2024), using the same dataset as DeepRTL and the adopted curriculum learning strategy.
In Table 4 and Table 5 we present the performance of both the original base models and their fine- tuned counterparts Verilog understanding and generation tasks. The improvement in performance from the original base models to the fine-tuned models highlights the effectiveness of our dataset and the curriculum learning- based fine-tuning strategy. Compared the results in Table 2 and T the superior performance of DeepRTL-220m on both tasks, despite its smaller model size, underscores the architectural advantages of our approach.
G INSTRUCTIONS FOR DIFFERENT SCENARIOS
Figure [ presents detailed instruction samples for different scenarios, following the instruction con- struction process illustrated in Figure Additionally, it includes a special module-level task, which involves completing the source code based on the functional descriptions of varying granularity and the predefined module header.
3 inst
18

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation results on Verilog understanding using the benchmark proposed in Section 3.4 BLEU-4 denotes the smoothed BLEU-4 score, and Emb. Sim. represents the embedding similarity metric. Specifically, this table presents the performance of decoder-only models, where "long" indicates models fine-tuned on the dataset containing longer Verilog designs, and those fine-tuned specifically Verilog. indicates performance evaluated on designs shorter than 512 tokens.
Model
BLEU-4
ROUGE-1
ROUGE-2
ROUGE-L
Emb. Sim.
GPT Score
deepseek-coder (original)
1.04
21.43
4.38
19.77
0,678
0.557
deepseek-coder (fine-tuned)
11.96
40.49
19.77
36.14
0,826
0,664
deepseek-coder (long)
11.27
40.28
18.95
35.93
0.825
0.649
1lama-3.2 (original)
0.88
19.26
3.60
17.64
0.615
0.449
3.2 (fine-tnned)
12.11
39.95
19.47
35.29
0,825
0.620
1lama-3.2 (long)
11.32
39.60
18.67
34.94
0.814
0.610
RTLCoder VeriGen
1.08
21.83
4.68
20.30
0,687
0,561
0.09
6.54
0.35
6.08
0.505
0.311
DeepRTI-220m-5122
14.98
44.27
23.11
40.08
0,780
0.567
DeepRTL-220m
18.74
48.41
29.82
45.01
0.855
0.743
Table 5: Evaluation results on Verilog generation. Each cell displays the percentage of code sam- ples, of five trials, that successfully pass compilation (syntax column) or functional unit tests (function column). This table presents the performance of decoder-only models, where "o" denotes the original model and "f" denotes the fine-tuned model.
doepseck-coder (o)
deepseck-coder (f)
Ilama-3.2 (o)
Ilama-3.2 (f)
Benchmark
syntux
function
syntax
function
syntax
function
syntax
function
Counteon.Countrr
100%
0%
100%
0%
100%
0%
100%
0%
alu
0%
0%
0%
0%
0%
0%
0%
0%
edge detect
60%
0%
80%
20%
60%
0%
80%
0%
Treq div
80%
0%
100%
0%
80%
0%
100%
0%
Logic
mux
60%
0%
100%
100%
60%
0%
60%
60%
paralle12scrial
80%
0%
100%
0%
80%
0%
100%
0%
pulse detect
0%
80%
40%
60%
20%
60%
40%
right_shfteer
20%
0%
80%
80%
20%
0%
40%
40%
pcrial2paralel
100%
0%
100%
0%
100%
0%
100%
0%
width 8to
100%
0%
100%
0%
100%
0%
100%
0%
accu
80%
0%
100%
0%
80%
0%
100%
0%
adder. 16bit
20%
0%
40%
20%
20%
0%
20%
20%
adder I 16bit esa
0%
0%
0%
20%
0%
20%
20%
20%
adder 32bit
0%
0%
20%
0%
0%
0%
20%
20%
adder_6dhit
0%
20%
0%
0%
00
40%
0%
Arithmetic
Adder_Sbit
40%
0%
80%
20%
40%
0%
60%
20%
dibil
0%
20%
0%
0%
0%
mubti_1bbit
60%
0%
50%
0%
60%
0%
80%
0%
multi_booth Sooth
40%
0%
60%
0%
40%
0%
60%
0%
multi multi_pipp Abit
100%
0%
100%
100%
100%
0%
100%
100%
multi pppeeppppeeeeee 8bit
0%
0%
0%
0%
0%
0%
0%
0%
1x2nocpe
60%
0%
20%
40%
60%
20%
60%
20%
1x4systolic
20%
0%
100%
100%
20%
0%
20%
20%
2x 2x2systolic
0%
0%
0%
0%
0%
0%
0%
0%
4xcsspatiaacce
0%
0%
0%
0%
0%
0%
fimm
80%
0%
100%
100%
80%
0%
100%
100%
Advanced
macpe
0%
0%
0%
0%
0%
0%
0%
0%
Sstate.Ism
80%
0%
100%
20%
80%
0%
100%
100%
3state Ism
0%%
0%
100%
80%
20%
20%
100%
100%
Astate_fsm
80%
0%
100%
40%
80%
20%
100%
20%
2state f'sm
60%
0%
100%
20%
60%
0%
100%
20%
Success Rate
44.52%
0.00%
63.87%
25.81% 11%
45. 16%
3.23%
58.71%
22.58%
Pass @
12.90%
0.00%
61.29%
22.58%
12.90%
0.00%
54.84%
19.35%
Pass @
67.74%
0.00%
80.65%
48.39%
70.97%
16.13%
80.65%
48.39%
H FURTHER EXPLANATION OF THE ADOPTED CURRICULUM LEARNING STRATEGY
Our dataset includes three levels annotation: line, block, and module, with each level containing descriptions that span various levels of detail- from detailed specifications high-level functional
19

---

Published as a conference paper at ICLR 2025
Understanding Task
Generation Task
Generate high-level. for
Verilog nodule.
Instraction:
the sorressonding Verslog described sodale.
high-level module
Input:
laput:
Verslog
<higgh-leve? module
<ligh
Outpat:
summary>
Output:
-Qualifiee Verilog module-
Module Level
Generate detailed
Instruction:
Veritog
led rodule the corresponding Verillng code for the described sodale.
Instruction:
sodule.
Input:
=Qualified Verileg nodules
Input:
<Nediur-level module sumserye
Outpat:
<Octailed summary>
Output:
Verilog
Instraction:
Generate a high-level summary for the given
Based on the high-level block surwary, generate the zorresponding Verilog code for the described block.
Instruction:
Verilag block.
Input:
-Quallifee Verilog blocke-
thigh-tevel block sunmarys
Imput:
Output: etiih-level level block sumraryo
Output:
-Qualifies Verilog blocks
Generate vediur-tevel surmary given black.
on levell black surmary, generate correspending Verilag block.
Block Level
Input:
Ieput:
siualiffeed Verslog block-
block summarys
<tediuu-tevel block summary>
Output:
Output:
-Qualified Verilog block-
Generate detailed sumsrry for the given Verileg
Instruction:
detailed block the corresponding Verslogg code for the described block.
Instruction:
block.
-ifellified Verslog blocks
Input:
-Nodiux-level block summarys
<betailed block sumarys
<Qualifies Verilog black>
Verilog sing
Instruction:
Generate a sing Lnggee-iin Line conment for the given
Based on the singte-1ine conment, generate the corresponc Verilog single-Lire code.
Instruction:
described
Line Level
Input:
Input:
<bingl e-1 lnne codes
<Singlu
Output:
Output:
<Sing Le-Line comments
<Sing Le-ngle-inee code>
Instruction:
Based on the <Level of Granularity: functional descriation and defined nodule head, complete the Verilog code.
Special Case
Input:
<todule
-ctodule heads
Output:
<Qlalified le
Figure 8: Instruction tuning data samples for different scenarios.
descriptions. And the entire dataset utilized for training. To fully leverage the potential of this dataset, we employ curriculum learning strategy, enabling the model to incrementally build knowl- edge by starting with simpler cases and advancing to more complex ones.
The curriculum learning strategy involves transitioning from more granular to less granular annota- tions across hierarchical levels, which can be conceptualized as tree structure with the following components (as shown in Figure 9h:
1. Hierarchical Levels (First Layer)
The training process transitions sequentially across the three hierarchical levels-line, block, and module. Each level is fully trained before moving to the next, ensuring solid foundation at simpler levels before addressing more complex tasks.
2. Granularity of Descriptions (Second Layer)
Within each hierarchical level, the annotations transition from detailed descriptions high- level descriptions. This progression ensures that the model learns finer details first and then builds an understanding of higher-level abstractions.
20

---

Published as a conference paper at ICLR 2025
Training Process
Training Data
Line Level
Block Level
Module Level
13
High-Teeve Functional
Detailed Descriptions
Medium-Dettilld Descriptions
High-Level Level Descriptions
Detailed Spocifications
Descriptions
14
15
GPT- Amsotated
Humun- Annotated
GPT- Anmotated
Human- Annotated
Human- Annotated
GPT- Annotated
Hummn- Ampocated
GPT- Annotated
Human- Amnotated
Amnotated
Figure The adopted curriculum learning strategy visualized as a structure. Specifically, the terminals of the tree, enclosed by blue dotted boxes, represent specific training datasets. Our curriculum learning strategy follows a pre-order traversal of this tree structure.
3. Annotation Source Transition (Third Layer)
At each level and granularity, training starts with GPT-annotated data and is followed by nuuman-annottted data. This sequence leverages large-scale machine- machine-generated annotations first and refines the model with high-quality, human--urater data.
4. Instruction Blending
Each terminal node in this tree represents a specific training dataset, which blends tasks for Verilog understanding and Verilog generation. This enables the model perform well across diverse tasks.
The training process mirrors pre-order traversal of this tree structure:
1. Starting at the root, training begins with the line level.
2. The model progresses through the second layer (detailed, medium-detail, and high-level descriptions).
3. Within each granularity, training transitions through the third layer (GPT-annotated data first, followed by human-annotater data).
4. Once the line level is complete, the process repeats for the block level and then the module level.
To validate the effectiveness this strategy, we conduct an ablation study where the model is trained on the entire dataset all at once without progression. The results, presented in Table of the main submission, demonstrate that the curriculum learning strategy significantly outperforms this base- line approach. Moreover, to the best of our knowledge, this is one of the first applications of curriculum- like training strategy in the code-learning domain. Unlike existing Verilog-related mod- els that establish simple and weak alignments between natural language and Verilog code (Chang ett al. 2024b), or general software code datasets like CodeSearchNef (Husain et al. 2019) that only provide single-level docstring annotations, our approach incorporates multi-level and multi- granularity annotations in structured training process.
I PROMPT FOR CALCULATING GPT SCORE
To calculate the GPT score, input the model's generated descriptions (modelooutput) and the ground truth annotations (ground_truth to GPT-4, using the prompt displayed in Figure I0 This metric is designed to assess the semantic accuracy of the generated functional descriptions.
21

---

Published as a conference paper at ICLR 2025
You professional Verilop designer that needs evaluate the sinilarity between textual. functional summaries describing Verilog code.
The first surmary is the ground truth description of the Verilog code, and the second cummary is the generated description of the Verflog code.
Please read the following summaries and provide sinilarity score between e and I based on how sinitar the two surearies are in teras of describing the functionality of the Veritog code, where è neans completely dissivitar and veans identical.
Note that you should strictly only output the score without any additional information. Suesary 1: (ground truen) Suesary 2: prodel_outpurt.
Figure 10: Prompt used to calculate the GPT score.
J COMPARISON WITH MODELS SPECIFICALLY TRAINED ON VERILOG
To further demonstrate the superiority of DeepRTL. we conduct experiments comparing it with mod- els specifically trained on Verilog. We do not select (Chang et al. [022b Zhang et al. (024) for com- parison, as their models are not open-sourced, and it is non-trivial to reproduce their experiments. Additionally, the reported performance in their original papers is either comparable to, and in some cases inferior to, that of GPT-3.5. In Table and Table 66 we show the performance of two state- of-the-art art Verilog generation models, mine-tuned-codege--Jee 16B- Verilog ( (VeriGen) (Thakur al. 2024 on both Verilog understanding and generation benchmarks. It is noteworthy that RTT.Coder fine-tuned on DeepSeek-coder-66777 7B, and VeriGen is fine-tuned on CodeGen-mulli- eter sizes than DeepRTL-220mg Despite this, the superior performance of DeepRTL-220m further underscores the effectiveness of our proposed dataset and the adopted curriculum learning strategy.
CeeoCoo-DDeeeeee... 1 (Liu et al. 2024 and
both of which have significantly larger param-
Table 6: Evaluation results on Verilog generation. Each cell displays the percentage of code samples, out of five trials, that successfully pass compilation (syntax column) or functional unit tests (function column). This table presents the performance of models specifically trained on Verilog.
Benchmark
RTLCader
VeriCien
lnscton
Foncton
Johnson.
100%
alu
0%
0%%
02%
0P%
detec
100%
100%
100%
20%
div
80%
17%
100%
07S
Logic
purallelZeerial
60%
40%
0%
100%
80%
20%6
0%
pulse detect
20%
075
10%
0%
nigi-shiffer
N016
80%
100%
100%
Sparalll
0%
80%
width. Bin
60%
07%
100%
0%%
0%
0%
0%%
0%
X0%%
Mfler_lohrruse
R0%
80%%
02%
02%
adher_Shht
80%
096
0P6
026
20%
19%
40%6
Arithmetic
dity diy_Tahi
0%
19%
19%
milti
80%
075
30%
025
milli booth
20%
1PA
20%
028
mulli pupe. Kovi
0%
0%%
02%
0%
1x2nncpe
40%
40%
100%
100%
100%
100%
20%
StSsystolic
0%
096
088
4xdspusialace
0%
0%
0%
0%6
Advanced
magge
Tum
0%
60%
0%
R0PS
19%
30%
096
Sotate
40%
80%
Scrate fim
80%
0%
X0%
37%
4crate_fim
80%
19
80%
20%
Zstate
0%
0%
Successs Rate
18.39%
20.00%
30974
12.6%%
Pass
41,945
16.13%
48.39%
9.08%
71.42%
33.48%
70.97%
32.30%
https::/hugginggfaas
cww.ce.ccc......ooooooooooooooee www.oo/sallaaa thee wow.riinen-utnee-e----e--eeeeeee 16- 1og
22

---

Published as a conference paper at ICLR 2025
Table 7: Evaluation results Verilog generation. Fach cell displays the percentage of code samples, out of five trials, that successfully pass compilation (syntax column) or functional unit tests (function column). This table presents the performance of decoder-only models fine-tuned on the dataset containing longer Verilog designs.
deepseck-ewler
Tlamo- 2.2
Benchmark
symlax
Fonction
syntix
Tunction
Joinsonn Comnter
T00R
0%
100%
USS
alu
0%
0%
ddger
80%
19%
80%
0S5
freq.div
100%
19%
100%
0%
Logic
porallalSeeril
100%
10075
100%
19%
1009
60%
60%
ORS
pulse detect
80%
40%
00%
40%6
nghht shiller
80%
80%
10%
1176
Sprralltrralltt
TOORE
08%
100%
widtt
100%
0%
100%
0020
100%
076
1007
025
allkr_Tobt Jobn
20%
207
107
2026
JUbi
0%
07%
2074
2026
atfle_6-lhtt
08
07
078
076
Arithmetic
adder_Sbit
80%
2016
60%
20%
div
mill1 Tohi
R05
07%
80%
076
maltt booth
60%
0%
00%
0%6
mmili..npppe Aboi
1007
1004
100%%
1007
1x 2nociee
40%
40%
60%
2056
184cystolic
20%
20%
20%6
2056
Saystll
4odeportalace
026
Advanced
15m
100%6
100%
100%
100%
1PA
1PA
1P%
078
T00%
100%
Swate Ism
R0%
80%
100%
1006
Jorate Ism
T00%
096
100%
096
2state_fsm
TOPEE
20%
100%
30%
Staccess
20.05%
57.42%
21.94%
Pass
38.71%
19.35%
38.71%
10.35
77.42%
38.71%
77.42%
45.10%
K NEGATIVE IMPACT OF INCORPORATING VERILOG DESIGNS EXCEEDING 2048 TOKENS
Notably, the maximum input length for DeepSeek-coder is 16k tokens, while for L.aMM-3.22, is 128k tokens. To assess the potential negative impact of including Verilog designs exceeding 2048 tokens, we conduct an ablation study in which we do not exclude such modules for these two models and instead use the dataset containing longer designs for training. As shown in Table 7 and by comparing the results in Table 4 the performance of the fine-tuned models on both Verilog understanding and generation tasks significantly degrades compared to the results in Table where these models are fine-tuned using the same dataset as DeepRTí. This further validates the rationale behind our decision to exclude Verilog modules and blocks exceeding 2048 tokens.
L ADDITIONAL EXPERIMENTS INVESTIGATING THE IMPACT OF VARYING CONTEXT WINDOW LENGTHS
To address concerns regarding the potential bias introduced by excluding examples longer than 2048 tokens, investigate the impact of context window length. Specifically, exclude all Verilog modules exceeding 512 tokens and use the truncated dataset to train a new model, DeepRTL-220m: 512 utilizing the curriculum learning strategy, which has a maximum input length of 512 tokens. We then evaluate both DeepRTTL-220m-35 samples, where the module lengths are below 512 tokens, and present the results in Table For the generation task, DeepRTL-220m-511 shows near-zero performance, with nearly 0% accuracy for both syntax and functional correctness. This result refutes the concern that "a model accommodating longer context windows could potentially offer superior performance on the general task, but not for this tailored dataset." " as it does not hold true in our case.
and DeepRTL--220n on Verilog understanding benchmark
23


Reviews:
[{"review_id": "BeI8dUD79P", "reviewer": "Reviewer_hfVY", "summary": "This paper proposes a dataset and a model for verilog generation and understanding. It carefully describes the annotation process for the dataset and presents an extensive battery of experimental results. Overall, the paper seems valuable to me, although I should clarify that I am well-versed in code generation, but not in Verilog so I may be missing some context with related work.", "questions": "None", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- As a whole, the work seems extensive and relatively careful, from conceptualization to base data collection, human annotation, model training, and evaluation.\n- I am not an expert in EDA, but it seemed like the work was novel from the point of view of such a dataset and model not existing previously.\n- The experimentation is extensive, comparing a fairly large number of models with various evaluation metrics.", "weaknesses": "- As someone who is not well-versed in Verilog, I would have appreciated an explanation of the basics of the language, what is its basic syntax, characteristics, etc. But there was not very much explanation in the paper.\n- Conceptually, the work was rather straightforward and I did not get many clear research insights from the paper. For this paper I am not extremely concerned about this though, as the work seems valuable nonetheless, and could serve as a base for future research.\n- It was not clear how much of the work will be released for the research community to build on. It seems that some of the data may be released, but presumably the proprietary data will not be? And also it wasn't clear about the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V0HbAzQDBT", "reviewer": "Reviewer_UcZL", "summary": "This paper deals with the task of code understanding and generation in the context of generation of hardware description language (HDL) code. In particular, this work focuses on the generation of Verilog code.\nThe model is based on an existing CodeLLM (authors used CodeT5+), which was fine tuned with a new augmented dataset created for this purpose. The dataset comprises both open and proprietary verilog codes, which were augmented (commented and summarised) by GPT-4. \nTwo models are trained using a  progressive training strategy based on CodeT5+ models. For the understanding benchmark, models are evaluated in terms of BLUE and ROUGE, as well as embedding similarity and GPT score. Results show an improved performance over competitors and baseline models. For the generation part, the models are evaluated on a Verilog generation benchmark introduced by Chang et al. 2024, and compared with GPT series models showing competitive performance against the best, o1-preview and surpassing GPT3.5 and GPT4.", "questions": "* Verilog is not a particularly known language. Authors could have explained a bit more its nature, syntax and usage.\n\n* Figure 1, although it helps to understand the flow of data collection, it’s not particularly clear. The fact that the flow goes to the top-left in opposition to the common flow for reading (top to bottom and left to right) makes it unclear. Also, which part is used for training? Only after distil?\n\n* Line 388-392: these lines and Figure 3 describe the progressive training. This explanation is not clear. Are the authors just feeding the model with more to less granular annotations? That could be an example of Curriculum learning. Please clarify and add references if needed.\n\n* why the authors didn’t compared the performance of the new models with Zhang et al., 2024, , Chang et al. 2024b, Liu et al. (2023b); Thakur et al. (2024),", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Original approach, focusing on both generation and understanding tasks on a low resource code language as Verilog, specifically designed for hardware description.\nThe approach seems reasonable. The field of application is needed and follows the ultimate goal of improving electronic design automation.", "weaknesses": "The work lacks clarity. Particularly, the dataset collection and the training regime are not completely clear and their figures do not clarify the issue (see below).\nExperiments seem reasonable but all baselines and competitors weren’t trained specifically on verilog. Since the current work cites other previous approaches, experiments could have compared to them as well (or explain why was not possible)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uxzJpO1DNj", "reviewer": "Reviewer_Aczy", "summary": "The paper makes a contribution to the field of hardware design automation by addressing both the generation and understanding of Verilog code using large language models (LLMs). While previous studies primarily focused on the generation aspect, this work recognizes the importance of understanding Verilog code and proposes a unified representation model, DeepRTL, built on an enhanced CodeT5+ architecture. This model is trained on a specifically curated dataset that tightly aligns natural language descriptions with Verilog code, aiming to improve the semantic alignment between the two. Additionally, the paper introduces the first benchmark specifically for Verilog understanding and develops two novel metrics, embedding similarity and GPT score, to capture semantic similarities more effectively than traditional n-gram-based metrics like BLEU and ROUGE. In comparative assessments, DeepRTL surpasses GPT-4 in Verilog understanding tasks and matches the performance of OpenAI’s o1-preview model in code generation tasks.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel task for evaluating LLMs in hardware design, focusing on Verilog understanding—prior work mainly focuses on generation. It introduces new training datasets, evaluation benchmarks, and establishes baselines for this new task.\n\n2. DeepRTL, the model proposed in this paper, uniquely good at both the generation and understanding of Verilog, making it different from other models in the hardware design domain.\n\n3. The methodology for creating a natural language-code parallel corpus via prompt engineering with GPT-4 is innovative and shows promise for broader application in fields where parallel corpora are lacking.\n\n4. The diagrams in this paper describes the proposed methods clearly and intuitively.", "weaknesses": "1. The reason for selecting T5-like models as the base for DeepRTL is not empirically validated. It remains unclear whether the observed performance gains in Verilog understanding are due to T5's encoder-decoder architecture or the synthesized dataset used for fine-tuning. Comparative analysis with a decoder-only model, such as LLaMa-3-1B or DeepSeekCoder-1.3B, using the same dataset for finetuning would provide clearer insights.\n\n2. The paper does not evaluate the impact of varying context window lengths, which is important given that CodeT5+ supports a limited token count (2,048 tokens), while actual Verilog code often exceeds this length. Dropping examples longer than 2,048 tokens will also bias the results in favor of DeepRTL, which is based on CodeT5+. A model accommodating longer context windows could potentially offer superior performance on the general task, but not for this tailored dataset.\n\n3. The evaluation metrics for code understanding—embedding similarity and GPT score—are solely based on GPT models, leading to potential bias, as evidenced by the inflated scores of GPT-3.5, GPT-4, and o1-preview models shown in Table 2. This overlap may make the comparisons bias in favor of GPT-family models.\n\n4. The evaluation of code generation lacks a comprehensive set of baselines. Despite mentioning various Verilog generation models in the related work section, these models are absent from the comparative analysis in Table 3.\n\n5. The fine-tuning dataset includes proprietary code that cannot be released publicly, and the benchmarks used are also developed by the authors. The absence of shared code, data, or models in the publication hinders reproducibility and make it impossible to assess potential data contamination and bias in evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kUERzSjBUk", "reviewer": "Reviewer_vHqv", "summary": "This paper introduces a novel dataset and model training for Verilog understanding and generation, as well as a new high-quality benchmark for the understanding task.\n\nThe authors provide a large Verilog dataset based on a large quantity of crawled open source code that is processed into code and natural language descriptions via GPT4, as well as a smaller amount of hand-curated code-description items from proprietary sources.\\\nThey also introduce a new benchmark for Verilog understanding, consisting of 100 manually verified, high-quality code-description pairs.\n\nFor experiments, the authors train CodeT5+ -based models of sizes 220M and 16B on their newly introduced dataset, using \"progressive training\" and evaluate model performance in terms of Verilog understanding and generation capabilities.\\\nExperiments show that models trained in this manner outperform strong baselines on various metrics.", "questions": "- Why throw away dataset items that are longer than 2,048 tokens? It is true that this is the maximum input length for CodeT5+; however, why make a choice about the dataset based on the (essentially arbitrary) choice of model used in the specific experiments here?\\\nModern LLMs, including open source ones such as Llama, have context sizes way beyond 2,048 tokens.\n\n**Comment after Rebuttal**\nI have adjusted my score to \"marginally below acceptance threshold\", rather than an outright \"reject\" based very good rebuttals.\\\nA agree with the authors on certain parts of my original criticism; however, the current manuscript requires some significant amount of re-work to be \"acceptable\", especially wrt. the original claims and presented support for decisions such as base model choice, and length restrictions of the dataset.\n\n**Comment after further Rebuttal**\nThe authors have done an exceptional job at responding to reviewers' concerns, and addressed them in an updated manuscript. After the changes, I believe this paper is in a state that is acceptable; I have updated my scores accordingly.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A novel dataset and benchmark for highly specialised programming code (Verilog); this might be interesting as it provides a new and interesting resource for a programming language that does not have as much attention as others such as Python, Jave, or C++.", "weaknesses": "Beyond the curation of an interesting new dataset, there is very limited novelty to this work; it seems like authors might be somewhat unfamiliar with the current state of the field of LLMs/Machine Learning, including ML for code:\n- Fine-tuning a CodeT5 model on domain-specific code has been done.\n- The \"progressive training\" is just curriculum learning, which is well-established in the field.\n- Similarity scores based on vector similarity are as old as Word2Vec, if not older.\n- Similarities/evaluations with LMs or LLMs (here \"GPT Score\") are well-established, e.g., see \"LLM as a judge\", BERT Score, etc.\n\nThis seems like it would be a very nice paper for a specialised Verilog/hardware spec conference, but may be of limited value for a venue like ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)