Title: VeriLoC: Line-of-Code Level Prediction of Hardware Design Quality from Verilog Code
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hardware Design

LLM Embedding

Congestion

Timing
Abstract: Modern chip design is complex, and there is a crucial need for early-stage prediction of key  design-quality metrics like timing and routing congestion directly from Verilog code (a commonly used programming language for hardware design). It is especially important yet complex to predict individual lines of code that cause timing violations or downstream routing congestion. Prior works have tried approaches like converting Verilog into an intermediate graph representation and using LLM embeddings alongside other features to predict module-level quality, but did not consider line-level quality prediction. We propose VeriLoC, the first method that predicts design quality directly from Verilog at both the line- and module-level.  To this end, VeriLoC leverages recent Verilog code-generation LLMs to extract local line-level and module-level embeddings, and trains downstream classifiers/regressors on concatenations of these embeddings. VeriLoC achieves high F1-scores of 0.86-0.95 for line-level congestion and timing prediction, and reduces the mean average percentage error from 14%-18% for SOTA methods down to only 4%. We believe that VeriLoC embeddings and insights from our work  will also be of value for other predictive and optimization tasks for complex hardware design.


Peer reviews:
comments: SUMMARY:
This paper proposes a method for predicting hardware design quality metrics (timing violations and routing congestion) by combining LLM-based Verilog code embeddings with tree-based models (XGBoost/LightGBM).

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is well-structured, and figures (e.g., pipeline overview) effectively illustrate the workflow.
- The methodology is technically sound, with thorough ablation studies (e.g., dimensionality reduction, embedding granularity) supporting design choices.
- Addresses a real-world pain point in RTL design verification, potentially reducing EDA tool runtime.

Weaknesses:
- Straightforward application of embedding LLM. LLM + dimensionality reduction + classifier is not fundamentally novel outside hardware contexts, adding little value to the field of neural information processing.

QUESTIONS:
- How does the model perform on designs with different coding styles (e.g., generated RTL vs. handwritten)?
- Are there plans to integrate this into industry workflows (e.g., as a plugin for Vivado/VSCode)? What are the inference latency and memory overheads?
- How would NeurIPS audiences be interested in this? I honestly suggest to submit to EDA venues (DAC, DATE, ICCAD, etc.).

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

comments: SUMMARY:
This paper proposes VeriLoC, a novel method for predicting timing and routing congestion directly from Verilog code at both the line and module levels. By leveraging LLM-based embeddings, VeriLoC achieves strong performance improvements over prior work. The approach aims to provide early-stage design-quality prediction and may be useful for other hardware optimization tasks.

STRENGTHS AND WEAKNESSES:
strength:
- A key strength of this work is its novel focus on predicting design quality at the line level of RTL code, which has not been explored in previous studies. The results also demonstrate that LLM-based embeddings can effectively capture the semantic information needed for accurate line-level quality prediction.
- The paper is very clearly written, making the methodology and results easy to understand.
- This work is that the authors have open-sourced their code, which will facilitate reproducibility and encourage further research in this area.

weakness:

- My main concern is the practical applicability of this approach, as directly predicting timing and congestion from RTL code alone is inherently inaccurate. In practice, both timing and congestion are heavily influenced by downstream logic synthesis and physical implementation algorithms, which are not reflected at the RTL level. Therefore, I also find the analysis in Figures 7-9 to be not entirely convincing or clearly justified.

QUESTIONS:
- Why were context windows larger than 5 not tested or reported in Table 2?
- The timing prediction task is evaluated using binary classification metrics. Could the authors clarify why binary classification was chosen for this task, and what the criteria are for defining positive and negative cases?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper has proposed a workflow using llm to directly extract line-level and module-level information of verilog codes as a predection step to evaluate design quality at the very early stage of chip design. The objective is to predict metrics such as timing, congestion before all the time-consuming backend flows. In this work, the authors proposed using LLM as verilog encoder, followed with various line-level classifier/regressor module for different performance related task. The authors claims the finer line-level information and inter-line level information extractions contribute to higher metric prediciton/classification accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\t1. The encoding strategy prosposed in this paper includes finer granularity. The concatenation of line-level rtl information and module-level information is richer than simple embedding of module rtl\n\t2. The paper has made a good point about hardware design data is very scarce publicly, thus very difficult to reply on pretrained LLM for direct performance prediction for the lack of hugh quality data in pretrain data. And the embedding stage + performance prediction/classification allowing for resource-limited training.\n\t3. The paper has shown sota performance prediction witht he help of finer rtl information over the previous baselines.\n\nWeaknesses:\n\t1. The importance of each single line and how much will it affect the backend metrics such as congestion/timing is still beyond my imagination. For example, in Figure. 4, the author provided visualization of actual vs preducted WNS. But what if the backend EDA algorithms changes, will the strong alignment still hold? Such discussion or proof is missing, which will make the statement less convincing. For example, if the placement & routing algorithm changes or even if the technology nodes changes, will the WNS be the same? And for each line in the RTL, the code itself cleary did not change? How will the model adjust to these without additional information.\n\t2. A following concern is that The model did not provide ablative experiments on the line-level/module-level information. You need to provide it to prove the extra information does help to prediction the features bettern instead of just adopting a stronger model to fit the dataset better.\n\t3. Some more details about the training are not clear to me.  How exactly do you label the data of each line-level? How exacly do you train the encoder? or you just train encoder and decoder at the same time with the label of each module?\n\t4. This is a bit misleading calling it self LLM-based RTL performance prediction. To me, this is just a traditional machine learning based RTL performance prediction with the help from pretrained large language model.\n\nQUESTIONS:\n1. As listed in the Weakness.\n2. I also like to ask about the training cost, and the overall cost compared with baseline methods.\n3. I want to know why you chose the exact model architecture?\n4. Do you need to pretrain the encoder or not? Why do you call it LLM when it is just an  transformer model? unless it is pretrained on huge amount of general language data？\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose using embeddings from Verilog-code generation LLMs to predict routing congestion and timing violations for hardware designs at the line- and module-level. They demonstrate the performance of their method, dubbed VeriLoC, on the OpenABCD RTL/Verilog code dataset using several metrics, including F1 scores for line-level congestions and timing classification, as well as the $R^2$ and mean-average percentage error for the timing violation regression. They compare with several other SOTA methods, and find VeriLoC outperforms them. They also present additional studies related to the impact of Verilog code length and LLM size.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* It appears to be the first model addressing the line-level prediction of timing violations and routing congestion, making it quite novel.\n* Comparisons to previous SOTA methods, including MasterRTL and RTL-Timer, for the timing violation prediction, indicating improved performance.\n* Code is public, improving reproducibility and reusability.\n\nWeaknesses\n* No comparison is made to a recent SOTA method called CircuitFusion, where the authors cite difficulty due to the different datasets used for training.\n* It's unclear how well this model can generalize to other synthesis tools or timing constraints.\n\nQUESTIONS:\n* How reliable are the line- and module-level labels generated for the dataset? How often are congestion and/or timing violations attributed to the wrong line or module? Would another synthesis tool or different settings give (significantly) different labels?\n* I understand the ABCD dataset is public, but will the additional annotations you generated also be made public? This would allow benefit future work.\n* How difficult is it to evaluate CircuitFusion on your annotated ABCD dataset? Is the issue that it would require retraining?\n\nLIMITATIONS:\n* Authors should address how much they expect the model to generalize, e.g. to different timing constraints or synthesis tools. Would retraining be necessary for small changes? Consider discussing this in the paper or running some additional studies along this line.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}