CIRCUIT: A Benchmark for Circuit Interpretation and Reasoning Capabilities of LLMs

Authors: Lejla Skelic, Yan Xu, Matthew Cox, Wenjie Lu, Tao Yu, Ruonan Han
Primary area: datasets and benchmarks
Keywords: Large Language Models (LLMs), benchmarking, analog circuits, dataset creation, evaluation metrics

Abstract:
The role of Large Language Models (LLMs) has not been extensively explored in analog circuit design, which could benefit from a reasoning-based approach that transcends traditional optimization techniques. In particular, despite their growing relevance, there are no benchmarks to assess LLMs’ reasoning capability about circuits. Therefore, we created the CIRCUIT dataset consisting of 510 question-answer pairs spanning various levels of analog-circuit-related subjects. The best-performing model on our dataset, GPT-4o, achieves 48.04\% accuracy when evaluated on the final numerical answer. To evaluate the robustness of LLMs on our dataset, we introduced a unique feature that enables unit-test-like evaluation by grouping questions into unit tests. In this case, GPT-4o can only pass 27.45\% of the unit tests, highlighting that the most advanced LLMs still struggle with understanding circuits, which requires multi-level reasoning, particularly when involving circuit topologies. This circuit-specific benchmark highlights LLMs' limitations, offering valuable insights for advancing their application in analog integrated circuit design.

Decision: reject
Reviewer scores: 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730509360013

thread_type:
official_review

<thread_object>
creator: reviewer_ahcw

<content>
summary:
This paper introduces a benchmark designed to evaluate the circuit interpretation and reasoning capabilities of large language models (LLMs). 
The benchmark comprises 510 question-answer pairs covering various areas of analog circuit design. 
Evaluations conducted on three representative LLMs provide some insights into the reasoning abilities of state-of-the-art models.

strengths:
The paper presents an intriguing study on the capabilities of large language models (LLMs) in circuit interpretation and reasoning and offers some insights to advance research in leveraging LLMs for analog design automation. 

The dataset, comprising over 500 question-answer pairs sourced from a variety of courses and textbooks, appears substantial and non-trivial, providing a solid foundation for evaluating LLM performance in this domain.

weaknesses:
The paper’s technical contributions appear limited, focusing primarily on prompt engineering. 

Additionally, the benchmark used raises concerns: although it consists of 500 question-answer pairs, these questions are sourced from textbooks, which overly simplify the real-world challenges of analog circuit design. Analog circuit design involves multiple critical stages, such as topology generation, device sizing, and layout design, and it is unclear how the current benchmark addresses these complexities.

Furthermore, the results seem unreasonable. Instead of reporting only the overall accuracy, a more detailed breakdown of circuit performance across varying complexities should be presented. Understanding how the model handles complex analog circuit problems, as opposed to simpler ones, would provide far more meaningful insights into its effectiveness.

The insights are limited. It does not show how to improve the capabilities of LLM in addressing analog design automation.

questions:
How does the benchmark address the different design stages of analog circuits? 
What is the success ratio of each level circuit (e.g., 1, 3, 5) based on the benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730509805949

thread_type:
official_review

<thread_object>
creator: reviewer_buqo

<content>
summary:
This paper focuses on the application of large language models (LLMs) in analog circuit design. Given that analog circuit design highly depends on human expertise and faces a labor shortage problem, the authors aim to evaluate the capabilities of LLMs in this field by creating the CIRCUIT dataset. The paper details the construction of the dataset, evaluation metrics and methods, and conducts experiments to evaluate multiple LLMs. Finally, it discusses the significance, limitations, and future directions of the research results.

strengths:
1. The article constructs a dataset for analog circuits to evaluate the capabilities of LLM in analog circuits, promoting the exploration and research of LLM in analog circuits.
2. The construction of the dataset is relatively comprehensive, covering different difficulties, and considering diagrams and netlists.
3. The experiment is relatively detailed, testing and validating the current mainstream models and conducting analysis and summary.

weaknesses:
1. The dataset focuses more on evaluating the LLM's answers to analog circuit questions, similar to math problems. However, it cannot verify the LLM's ability to assist in analog circuit design. For example, use a LLM to evaluate the quality of circuit design in terms of power, performance, and area (PPA).
2. The dataset lacks diversity in design and does not cover the entire analog circuit design process. For example, circuit structure design, device selection, test verification and physical design, etc.
3. The benchmark lacks an evaluation of the overall performance of the circuit.

questions:
1. How does the benchmark results help to improve LLM for EDA-aided design?
2. The current dataset focuses more on formula derivation and mathematical calculations, but may lack an understanding of how to evaluate LLM's understanding of the quality of analog circuit design. For example, evaluate the ability of large language models in predicting circuit PPA, ultra-parameter tuning of circuits, and test verification.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730547254247

thread_type:
official_review

<thread_object>
creator: reviewer_5hra

<content>
summary:
This paper presents a Circuit Interpretation and Reasoning Capabilities (CIRCUIT) dataset to evaluate LLMs in understanding and reasoning about analog circuits. The authors conduct a series of experiments to assess the performance of various LLMs in understanding analog circuits and their topologies from diagrams and netlists.

strengths:
1. This paper is the first to evaluate the ability of LLMs in circuit design domain and summarizes the limitations of existing LLMs according to the results. The authors summarize the limitations of existing models based on the experimental results. The qualitative analysis of errors offers valuable insights for improving LLMs.
2. This paper is well organized and clear.

weaknesses:
1. The sources of problems are limited. Most of problems are collected from textbooks published more than 10 years ago, which primarily assess general knowledge of analog circuits. It should be beneficial to construct more challenging problems about complex analog IC designs from various sources, such as handmade questions according to modern IC datasheets. 
2. The categories in this dataset are imbalance. In Figure 3, most of Basic questions are in level 1, while RF questions are more challenging. Besides, there is a lack of detailed analysis regarding LLM performance across different levels and categories.

questions:
1. How to differentiate between errors where LLMs have insufficient background knowledge about circuits and those where incorrect reasoning occurs? Do errors caused by insufficient knowledge fall under Direction Errors? 
2. A minor question about Table 4: how is the error rate calculated? I saw that the sum of each column or raw do not equal 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659237600

thread_type:
official_review

<thread_object>
creator: reviewer_japt

<content>
summary:
This paper introduces a CIRCUIT dataset. The dataset contains 510 questions and is evaluated via three LLM based models.

strengths:
* Evaluation results looks extensive, with results analysed in detail to have a comprehensive understanding of their performane on the dataset introduced.

weaknesses:
* The dataset size may not be sufficient enough to fully evaliate the capabilities of LLMs in analogue circuit design.
* While the evaluation results do show there is imrovements in terms of accuracies on analogue circuit related questions, it is unclear how well does this translate in designing analogue circuits.
* The paper only tests on solutions using LLM only, it is not clear how circuit design using combination of different models could utilize this as a benchmark. This seems more like how to train a LLM using questions on how analogue circuits operate.

questions:
please see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732687750196

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revisions in Response to Reviewer Feedback

comment:
We took the reviewer feedback and implemented several changes to our work. The main changes are highlighted in red. These include:

1. **Clarified the benchmark scope** and how it helps advance LLMs in analog circuit design in the Introduction and Related Work sections.
2. **Moved per-category accuracies** from the Appendix to the main paper body, **added a table of per-level accuracies**, and briefly discussed these results in the Human Evaluation section.
3. **Reorganized the Human Evaluation section** slightly and cut content for clarity.
4. **Emphasized the technical contributions** of our work in the Discussion section and reflected in more detail on future work. Also, slightly reorganized and cut content for coherence.
5. **Revised the Limitations** of our work based on reviewer feedback.
6. **Refined the Conclusion** for improved coherence.

We appreciate the reviewers' feedback, which helped us improve and clarify our contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper describes a benchmark for understanding analog circuits, which consists of 510 questions. Different LLMs are evaluated. The setup requires numbers (like the voltage) and the diagram of the circuit in the netlist format, as well as the question/answer pairs.

The accuracy of the models like GPT-4 is around 50%

The usefulness of creating new benchmarks for LLM very clear, and this paper adds one of them. However, I think that the contribution is not so big: you can expect such kind of results for any university-level course, be it topology, algebraic geometry or circuit understanding: they all can be transformed into such microbenchmarks. 
The paper does not provide any recommendations or attempts to increase the accuracy. 
What happens, if we finetune the model on such circuit data? What are typical errors and in which part? 
The benchmark also does not show any fundamental limitations of LLM for this scenario.