Title: Refining Specs For LLM-Based RTL Agile Design
Primary area: datasets and benchmarks
Keywords: prompt engineering, LLM assisted design, RTL design, benchmark

Abstract: Large language models (LLMs) are increasingly employed to assist in agile register-transfer-level (RTL) hardware design. This is a labor-intensive stage in developing FPGA-based acceleration services or prototyping ASICs, and successful automation can largely shorten the development cycle. However, benchmarks are reporting a relatively low functional correctness rate when generating simple modules of less than 100 LOC (in Verilog), questioning the practicality of current LLMs in complex real-world designs.

This paper points out that this low accuracy is affected by using low-quality descriptions as prompts in both training datasets and benchmarks. First, the testbench (TB) strictly enforces behaviors that are absent from the natural language descriptions (NLDs), leading to false negatives. Second, existing automatically generated NLDs are usually too detailed in implementation, which is not suitable for both training and benchmarking. We show by experiment that, when provided with detailed specifications, general-purpose LLMs can achieve a high pass@5 rate (89\% on RTLLM, 96\% on VerilogEval-Human) without expensive finetuning or post-generation self-fixing. This implies that LLMs are enough capable of generating small RTL modules in action. Our tools can improve the quality of existing and future LLM-for-RTL datasets, and assist agile RTL designers to create qualified specifications.

TL;DR: LLM-refined specs can help LLM RTL design agents to achieve over 90% pass@5 rate on RTLLM and VerilogEval benchmarks.