QCircuitNet: A Large-Scale Hierarchical Dataset for Quantum Algorithm Design

Authors: Rui Yang, Yuntian Gu, Ziruo Wang, Yitao Liang, Tongyang Li
Primary area: datasets and benchmarks
Keywords: AI for science, quantum computing, quantum oracle construction, quantum algorithm design

Abstract:
Quantum computing is an emerging field recognized for the significant speedup it offers over classical computing through quantum algorithms. However, designing and implementing quantum algorithms pose challenges due to the complex nature of quantum mechanics and the necessity for precise control over quantum states. 
Despite the significant advancements in AI, there has been a lack of datasets specifically tailored for this purpose. 
In this work, we introduce QCircuitNet, the first benchmark and test dataset designed to evaluate AI's capability in designing and implementing quantum algorithms in the form of quantum circuit codes. Unlike using AI for writing traditional codes, this task is fundamentally different and significantly more complicated due to highly flexible design space and intricate manipulation of qubits. 
Our key contributions include: 
1. A general framework which formulates the key features of quantum algorithm design task for Large Language Models.
2. Implementation for a wide range of quantum algorithms from basic primitives to advanced applications, with easy extension to more quantum algorithms.
3. Automatic validation and verification functions, allowing for iterative evaluation and interactive reasoning without human inspection.
4. Promising potential as a training dataset through primitive fine-tuning results.

We observed several interesting experimental phenomena: fine-tuning does not always outperform few-shot learning, and LLMs tend to exhibit consistent error patterns. QCircuitNet provides a comprehensive benchmark for AI-driven quantum algorithm design, offering advantages in model evaluation and improvement, while also revealing some limitations of LLMs in this domain.

Decision: reject
Reviewer scores: 5, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730518513765

thread_type:
official_review

<thread_object>
creator: reviewer_qghz

<content>
summary:
This paper presents QCircuitNet, the benchmark and test dataset for evaluating AI’s ability to design and implement quantum algorithms in quantum circuit codes.

strengths:
+ Interesting direction of using AI/LLM for developing quantum circuits for a particular quantum algorithm
+ Dataset can be useful to some extent

weaknesses:
+ Not clear what exactly are the new contributions in addition to the “run_and_analyze” function.

questions:
This paper claims that existing application benchmarks fail as a dataset for AI because they did not capture the design patterns of each algorithm, ignore post-processing and construction of different oracles. However, after reading the paper, I could not find what exactly are the “design patterns” of a quantum algorithm. The paper did not clarify any circuit features, or algorithm features, or device features. 

Meanwhile, It seems this work only focuses on oracle-based quantum algorithms, which represents a very small group of quantum algorithms that were well-known and developed many years ago (and is also not the main focus of the quantum computing/algorithm community).

It seems to me this claimed ‘dataset contribution’ is only a different way of selling the application benchmarks by (i) collecting the standard descriptions of particular quantum algorithms from text or existing paper; (ii) running in Qiskit-Aer to obtain standard output as the reference; (iii) adding metric measurement functions for the feedback/reward/gradient, which can be easily achieved using the MQTBench templates online. With that, I don’t think the technical contribution is sufficient for an ICLR publication. 

Last but not least, the word ‘post-processing’ is confusing, as in quantum computing, post-processing usually refers to error-mitigation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646873499

thread_type:
official_review

<thread_object>
creator: reviewer_o2k6

<content>
summary:
This paper introduces QCircuitNet, a benchmark dataset specifically designed to evaluate AI's capability in designing and implementing quantum algorithms as quantum circuit codes.

strengths:
1. A proposed method effectively captures quantum algorithms, situated between pure mathematical formulas and natural language.

weaknesses:
1. The dataset contains only classic algorithms and lacks generalizability.
2. Lacks tests for code completion.

questions:
1. Why call "QCircuitNet"? I mean it is more like a neural network rather than a dataset or benchmark. This name is confusing.
2. How many algorithms are currently included in the dataset? Does each algorithm contain test data with different number of qbits?
3. "The total computation cost is approximately equivalent to two days on an A100 GPU." you mean to test one model or all the experiments add up cost 2days? 
4. Can you incorporate "chain of thought" in this benchmark?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730678997702

thread_type:
official_review

<thread_object>
creator: reviewer_z41s

<content>
summary:
The authors present a new dataset, QCircuitNet, for Quantum Algorithm Design, aimed at enhancing the design and implementation of quantum algorithms using LLMs. The authors a structured framework that allow the LLMs to apply to the quantum algorithm discovery. QCircuitNet features built-in functions for automatic validation and verification of algorithms, supporting iterative evaluation without human intervention. Experiments results highlight the evaluation of LLMs for quantum algorithm discovery with dataset, showcasing its potential as a valuable resource in the field.

strengths:
1. QCircuitNet is the first dataset specifically designed for evaluating LLMs in quantum algorithm design.

2. The authors provide a structured framework that encapsulates key features of quantum algorithm design, allowing LLMs to work effectively with complex quantum tasks.

3. The paper presents valuable experimental findings regarding the performance of LLMs in quantum algorithm discovery.

weaknesses:
1. The paper primarily addresses quantum circuit design as a language modeling task, which may limit the scope of quantum algorithm generation and overlook other important methodologies.

2. The authors do not address Variational Quantum Algorithms, which are significant in near term quantum computing, indicating a gap in the dataset's comprehensiveness.

3. Given the limited research on LLMs for quantum circuits, the necessity of creating this dataset at this stage may be questioned.

4. The evaluation metrics used are heavily influenced by natural language processing. It raises the question of whether there might be a more intrinsic approach to integrating quantum into these metrics.

questions:
1. Why did you choose to focus primarily on quantum circuit design as a language modelling task? 

2. Are there alternative methodologies or frameworks that could also be explored for quantum algorithm discovery with your dataset?

2. What is your rationale for excluding VQAs from QCircuitNet? How do you see their importance in the context of quantum algorithm design?

3. Have you considered developing metrics that are more aligned with quantum computing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730802890409

thread_type:
official_review

<thread_object>
creator: reviewer_utsv

<content>
summary:
The paper introduces QCircuitNet, a novel dataset aimed at facilitating AI-driven quantum algorithm design. This dataset provides benchmarks and tools to evaluate the ability of LLMs to generate and validate quantum algorithms in quantum circuit code. QCircuitNet provides an automatic verification framework that ensures circuit validity.

strengths:
The paper introduces QCircuitNet, and it provides a comprehensive structure, including benchmarks, automatic validation, and compatibility with various algorithms. Additionally, it offers a unique framework for large language models by formulating quantum algorithms as programming tasks.

weaknesses:
See Questions.

questions:
1. In QcircuitNet, what is the specific number of circuits, and what is the range of qubits? As a benchmark, the description of the dataset is unclear.

2. Directly using LLMs to design quantum algorithms remains challenging. For workflows like Figure 2, can LLMs only learn specific quantum algorithms? However, with the limited algorithms/circuit types in the current benchmark, it seems insufficient to evaluate LLMs’ ability to design arbitrary quantum algorithms.

3. The current setup relies on classical simulations for verification, which limits scalability and slows down processes, especially for higher qubit counts. If real quantum computers are used, significant noise will be present. In such cases, how can effective verification be ensured?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731814823539

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We would like to thank the reviewers for recognizing that QCircuitNet is the first dataset which provides a comprehensive framework for benchmarking quantum algorithm design with LLMs. The main concerns focus on the following aspects: 
- **Generalizability to advanced quantum algorithms**: One core consideration to design the framework of QCircuitNet is to ensure easy extension to advanced quantum algorithms. In Section 4.1.2 of the original manuscript, we have provided a concrete example by implementing Generalized Simon's Problem, a more advanced version of the standard Simon’s problem and an active area of research in recent years [1, 2]. It effectively demonstrates the generalizability of QCircuitNet to advanced algorithms.
- **Scalability**: We choose classical simulation over real quantum computers as default verification methods also due to consideration of noises at NISQ era. As quantum hardware advances, these noise-related issues may gradually be alleviated. While we acknowledge that the speed of classical simulation could be a bottleneck for large qubit numbers, the time requirements remain manageable at the current scale of our benchmark. Additionally, quantum algorithm design is not inherently dependent on the number of qubits. For instance, the structure of the Deutsch-Jozsa algorithm - where Hadamard gates are applied before and after the oracle - remains consistent regardless of qubit count. In this sense, simulations and verifications on smaller qubit instances can effectively reveal the model’s capabilities for a given algorithm to a meaningful extent.
- **A few clarifications issues**: Detailed clarifications have been made in individual response to the reviewers.

[1] Z. Ye, Y. Huang, L. Li, and Y. Wang. Query complexity of generalized simon’s problem. Information and Computation, 281:104790, 2021.

[2] Z. Wu, D. Qiu, J. Tan, H. Li, and G. Cai. Quantum and classical query complexities for generalized simon’s problem. Theoretical Computer Science, 924:171–186, 2022.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732194486982

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We would like to express our gratitude for the time and effort you have dedicated to reviewing our paper. We have carefully considered your feedback and provided a detailed response in rebuttal. We are eager to hear your opinions and to address any further concerns you may have.

Thank you once again for your dedication, and we look forward to your response.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a dataset to test AI capabilities to design and implementing different quantum algorithm in the form of quantum circuits. The authors also fine-tune a model on this dataset and analyze the performance.
It is unclear, what audience this approach is for. It does not show any principal limitations of existing LLM (it did not see the QC domain -> it is not performing well, so lets fine-tune). It is also not combined with other methodologies and is left with a smart prompting of existing models. 

It also does not provide any new insights (faster/better/new) algorithms in the quantum computing field. 
So, I would consider this work incremental, and this concerns is shared by the reviewers