SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Classical Simulation of Quantum Circuits Using Reinforcement Learning: Parallel Environments and Benchmark
Xiao-Yang Liu $^{1,2}$  and Zeliang Zhang $^{2*}$
$^{1}$ Rensselaer Polytechnic Institute,  $^{2}$ Columbia University,
liux33@rpi.edu, XL2427@columbia.edu, hust0426@gmail.com
Google's "quantum supremacy" announcement [3] has received broad questions from academia and industry due to the debatable estimate of 10,000 years' running time for the classical simulation task on the Summit supercomputer. Has "quantum supremacy" already come? Or will it come in one or two decades later? To avoid hasty advertisements of "quantum supremacy" by tech giants or quantum startups and eliminate the cost of dedicating a team to the classical simulation task, we advocate an open-source approach to maintain a trustable benchmark performance. In this paper, we take a reinforcement learning approach for the classical simulation of quantum circuits and demonstrate its great potential by reporting an estimated simulation time of less than 4 days, a speedup of  $5.40 \times$  over the state-of-the-art method. Specifically, we formulate the classical simulation task as a tensor network contraction ordering problem using the K-spin Ising model and employ a novel Hamiltonian-based reinforcement learning algorithm. Then, we evaluate the performance of classical simulation of quantum circuits. We develop a dozen of massively parallel environments to simulate quantum circuits. We open-source our parallel gym environments and benchmarks.
Google proudly announced "achieving quantum supremacy" [3] with its 53-qubit Sycamore circuits back in 2019, which was later challenged by researchers claiming to have pulled ahead of Google on classical computers. Quantum supremacy [46] aims to demonstrate that a programmable quantum device can solve a problem that no classical computer can solve in any feasible amount of time, irrespective of the usefulness of the problem. As illustrated in Fig. 1, for the problem of random number generation, Google's "quantum supremacy" announcement [3] relied on an estimated simulation time of 10,000 years on the Summit supercomputer, while existing works reduce it to less than 21 days and scale the number of quantum qubits up to around 100. This raises the debate: Has "quantum supremacy" already come? Or will it come in one or two decades later? Our goal is to use machine learning methods to derive the best performance curves for the classical simulation task, to settle
Figure 1: Running time of different quantum circuit simulation methods.
down the present debate and suggest
*Zeliang Zhang completed this work as a research assistant at Columbia University.
that achieving "empirical quantum supremacy" requires continuing quantum hardware developments without an unequivocal first demonstration.
There are several existing works on the classical simulation of quantum circuits [8]. One promising method is using tensor networks [42] since quantum circuits can be naturally represented as tensor networks, i.e., executing a quantum gate is mathematically a tensor contraction operation. Therefore, establishing a benchmark curve for the classical simulation task is mathematically searching for the optimal tensor network contraction ordering (TNCO), which is a combinatorial optimization problem [25], a variant of the well-known traveling salesman problem (TSP) where reinforcement learning (RL) algorithms [33] have shown powerful capability. We are motivated to take a reinforcement learning approach to establish state-of-the-art performance.
There is a debate on the estimate of running time for classical simulation tasks, as illustrated in Fig. 2. Huang et al. [21] used a heuristic tensor network approach on a computing cluster and estimated the simulation time to be 21 days. Meirom et al. [34] used a reinforcement learning algorithm where the policy network used a graph neural network and reported an order-wise reduction compared to the best heuristic method. However, there is no available dataset for training and benchmarking machine learning algorithms, and it is reasonable to maintain a publicly trustable performance curve for the coming quantum supremacy.
To avoid hasty advertisements of "quantum supremacy" by tech giants or quantum startups and eliminate their cost of dedicating a team to the classical simulation of quantum circuits, establishing a standard benchmark is important. In this paper, we take a reinforcement learning (RL) approach for the classical simulation of quantum circuits and demonstrate its great potential by reporting an estimated simulation time of less than 4 days, a speedup of  $5.40 \times$  over the state-of-the-art method. The result demonstrates that the "quantum supremacy" claim still lacks an unequivocal first demonstration. Specifically, we adopt the K-spin Ising model for the classical simulation of quantum circuits, i.e., the tensor network contraction order (TNCO) problem, and employ the Hamiltonian-based reinforcement learning algorithm to minimize the number of multiplications. Then, we establish stan-
dard criteria to evaluate the simulation performance for Google's Sycamore circuits. We develop a dozen of massively parallel environments for training and evaluating RL agents. We release multiple datasets, including tensor-train, synthetic, and sycamore quantum circuits, and benchmark curves, including OPT-Einsum, Cotengra, and our RL method, on Github at https: //github.com/XiaoYangLiu-FinRL/RL4QuantumCircuits.
Figure 2: Debatable time estimates of classical simulation of Google's Sycamore circuits.
We hope the AI/ML community and quantum physics community will collaborate to maintain reference curves for validating an unequivocal first demonstration of "empirical quantum supremacy".
Random circuit sampling: Random Circuit Sampling (RCS) [35] is an approach that has garnered considerable attention within the quantum computing community. It involves the generation of random quantum circuits, which are subsequently subjected to measurement procedures. By sampling the measurement outcomes, RCS enables the evaluation of the distribution of classical output probabilities, providing valuable insights into the computational power of noisy intermediate-scale quantum devices. While many works [27, 38, 35] demonstrate the hardness of RCS, highlighting its computational complexity, the reinforcement learning-based method [34] presents a promising way to accelerate the classical simulation of RCS.
Quantum circuits for random number generation task: One notable example of the application of quantum circuits [53] is the random number generation. It is NP-hard since simulating a probability distribution on classical computers involves an exponential time complexity. For the problem of random number generation, Google's Sycamore circuit [3] was a milestone in quantum computing, which used the random circuit sampling technique to achieve the random selection process and claimed to demonstrate "quantum supremacy". However, it is hard for most researchers to access
the computing resources of quantum circuits for study. It is critical to simulate the quantum circuits and establish the dataset and benchmark performance to verify such scientific claims.
Classical simulation of quantum circuits: In the field of classical simulation, several approaches have been proposed to simulate quantum circuits, including the matrix representation methods [8], variational algorithms [51], and tensor network methods [42]. Among these, the tensor network method leverages the mathematical framework of tensor networks to approximate quantum states and perform efficient simulations. Tensor network methods, such as the matrix product state (MPS) [16] and projected entangled pair state (PEPS) [59], have shown promising results in simulating both one-dimensional and two-dimensional quantum systems. In this paper, we take the tensor network method for simulating quantum circuits.
Tensor network contraction [48] and the search for the optimal contraction path [4] have garnered significant attention in computational physics and quantum information theory. Various techniques [9] are employed to identify the most efficient way to contract tensors, minimizing computation cost. These advancements enable efficient simulation of quantum circuits. There are two popular open-source libraries, OPT-Einsum [13] and Cotengra [18] (CTG-Greedy and CTG-Kahypar), while Cotengra achieves state-of-the-art performance for most tensor networks.
Ising model: The Ising model [11] can serve as a unified formulation of combinatorial optimization problems, including graph coloring, maximum cut problems, and traveling salesman problems. The Ising model describes a system of interacting spins that can be either up or down, representing binary variables in studied problems. By mapping the studied problem onto an Ising model, the problem can be reformulated as finding the configuration of spins that minimizes the corresponding energy function [57]. The tensor network contraction problem can be analyzed into one kind of combination problem, especially the TSP. Thus, it motivates us to model the tensor network contraction problem as the Ising model and learn the optimal tensor contraction path by minimizing the energy function to reduce the computation complexity.
$\mathbf{AI} / \mathbf{ML} + \mathbf{X}$ : In recent years, the integration of artificial intelligence (AI) and machine learning (ML) techniques in scientific research has revolutionized various domains [12, 26, 6, 23]. The application of AI/ML in scientific fields, often referred to as  $\mathrm{AI + X}$  or machine learning for science, has brought about significant advancements and novel approaches to solving complex problems [54]. For example, tensor network factorization can be used for probabilistic modeling [15]. The quantum entanglement can be built as a fundamental connection with deep neural network design [28]. Quantum K-spin Hamiltonian Regularization is proposed to stabilize the reinforcement learning process [29]. A recent study uses the tensor network-based quantum circuits for image classification [19].
3 Classical Simulation of Quantum Circuits Using Tensor Networks
We use uppercase calligraphic letters to denote tensors, e.g.,  $\mathcal{X} \in \mathbb{R}^{I \times J \times K}$ , uppercase and lowercase boldface letters to denote matrices and vectors, e.g.,  $\pmb{X} \in \mathbb{R}^{I \times J}$  and  $\pmb{x} \in \mathbb{R}^I$ .
3.1 Quantum Circuits
Qubits: A qubit in a superposition state can be represented as  $|\phi \rangle = \alpha_0 |0\rangle + \alpha_1 |1\rangle$ , where  $\alpha_0, \alpha_1 \in \mathbb{C}$ , and  $|\alpha_0|^2 + |\alpha_1|^2 = 1$ . For  $n$  qubits, we use a linear combination of  $2^n$  coefficients and states  $|0\dots 0\rangle$ ,  $|0\dots 1\rangle$ ,  $\dots$ $|1\dots 1\rangle$ , respectively,
$$
| \psi \rangle = \alpha_ {0 \dots 0} | 0 \dots 0 \rangle + \dots + \alpha_ {1 \dots 1} | 1 \dots 1 \rangle , \tag {1}
$$
where  $\alpha_{0\dots 0},\dots,\alpha_{1\dots 1}\in \mathbb{C}$ , and  $|\alpha_{0\dots 0}|^2 +\ldots +|\alpha_{1\dots 1}|^2 = 1$
Quantum gates: Single- and double-qubit quantum gates are building blocks of quantum circuits,
- Single-qubit gate:
$$
\sqrt {\boldsymbol {X}} = \frac {1}{\sqrt {2}} \left[ \begin{array}{l l} 1 & - i \\ - i & 1 \end{array} \right], \sqrt {\boldsymbol {Y}} = \frac {1}{\sqrt {2}} \left[ \begin{array}{l l} 1 & - 1 \\ 1 & 1 \end{array} \right], \sqrt {\boldsymbol {W}} = \frac {1}{\sqrt {2}} \left[ \begin{array}{l l} 1 & - \sqrt {- i} \\ \sqrt {- i} & 1 \end{array} \right], (2)
$$
- Double-qubit gate:
$$
\operatorname {f S i m} (\theta , \phi) = \left[ \begin{array}{c c c c} 1 & 0 & 0 & 0 \\ 0 & \cos \theta & - i \sin \theta & 0 \\ 0 & - i \sin \theta & \cos \theta & 0 \\ 0 & 0 & 0 & \exp^ {- i \phi} \end{array} \right], \tag {3}
$$
where  $\theta \approx \pi /2$  and  $\phi \approx \pi /6$  are used in the Sycamore quantum circuits.
Quantum circuits consist of a sequence of quantum gates. For a given initial state  $|\psi_0\rangle$  and a quantum circuit  $U = U_{m}\dots U_{1}$ , the final state is  $|\psi \rangle = U_m\times \dots \times U_1|\psi_0\rangle$ . Fig. 3 illustrates an example of 4 qubits and a circuit  $U$  of  $m = 2$  cycles. The initial state  $|\psi_0\rangle = |0000\rangle$ . The quantum circuit  $U$  takes an initial state  $|\psi_0\rangle$  of  $n$  qubits as input, performs  $m$  cycles of gate operations, and outputs a bit-string  $i_1\dots i_n$  of length  $n$ .
In the  $i$ -th cycle, two operations are performed:
1. A single-qubit gate  $\pmb{R}_i^j \in \mathbb{C}^{2 \times 2}$  randomly selected from set  $\{\sqrt{X}, \sqrt{Y}, \sqrt{Z}\}$  is applied to  $|\psi_{i-1}\rangle_j$ , resulting in state  $|\psi_i\rangle_j = R_i|\psi_{i-1}\rangle_j$ .
2. Then, execute a two-qubit quantum gate  $U_{i}^{p} \in \mathbb{C}^{4 \times 4}$  to  $|\psi_{i-1}\rangle_{p}$  and  $|\psi_{i-1}\rangle_{p+1}$  and obtain a new state  $|\psi_{i}\rangle_{p,p+1}$ . Specifically, during an odd cycle, qubit pairs with indices originating at 0, such as (0,1), (2,3), are chosen. For even cycles, qubit pairs
Figure 3: A quantum circuit where the classical simulation lies in the middle.
with indices starting at 1 are selected, including pairs such as  $(1,2),(3,4)$ . This systematic approach ensures the appropriate application of the  $U_{i}^{p}$  operation on the designated qubit pairs.
Before measurement, each qubit  $|\psi_{m}\rangle_{j}$  undergoes a random single-qubit quantum gate  $R_{m + 1}^{j}$ . The random binary bit-string  $i_{1}\dots i_{n}$  is sampled from the probability distribution  $P = | \langle i_1i_2\dots i_n|\left|\psi_{m + 1}\right\rangle |^2$ , which is obtained from the measurement outcome. More details about the random circuit sampling can be found in Appendix A.
3.2 Random Circuit Sampling
Quantum supremacy: Quantum supremacy refers to a major milestone in quantum computing, representing the point at which a quantum computer can solve a specific problem that is practically infeasible for classical computers to solve within a reasonable timeframe. It is a demonstration of the superior computational power of quantum systems compared to classical counterparts. Achieving quantum supremacy signifies the ability of a quantum computer to perform computations exponentially faster than even the most powerful classical supercomputers.
A quantum circuit defines a distribution: Quantum circuit is a sequence of quantum operations or gates applied to a set of qubits. These gates manipulate the quantum state of the qubits, transforming them according to the specific operations performed. At the end of a quantum circuit, measurements are typically performed on the qubits, extracting classical information from the quantum system. The results of these measurements are probabilistic, meaning that they occur with certain probabilities. Therefore, a quantum circuit defines a distribution by specifying how these probabilities are distributed among the different possible measurement outcomes.
For example, given a quantum circuit  $U$ , the initial state  $|0\rangle$ , and the output sampling bit-string  $x = a_{1}a_{2}\dots a_{n}\in \{0,1\}^{n}$ , the modeled probability is  $p(x) = |\langle x|U|0\rangle |^2$ . The Google team used [1] the following noise model for the noisy samples produced by the Sycamore circuits,
$$
N _ {c} (x) = \phi P _ {C} + (1 - \phi) 2 ^ {- n}, \tag {4}
$$
where  $\phi$  is a fidelity parameter describing the quality of the sample,  $P_{C}$  is the probability distribution defined by the quantum circuit. The parameter  $\phi$  is estimated by follows,
$$
\phi = \prod_ {g \in \mathcal {G} _ {1}} \left(1 - e _ {g _ {1}}\right) \prod_ {g \in \mathcal {G} _ {2}} \left(1 - e _ {g _ {2}}\right) \prod_ {q \in \mathcal {Q} _ {1}} \left(1 - e _ {q}\right), \tag {5}
$$
where  $\mathcal{G}_1$  is the set of single-qubit gate,  $\mathcal{G}_2$  is the set of double-qubits gate, and  $\mathcal{Q}_1$  is the set of qubits. Google set [1]  $e_{g_1} = 0.16\%$ ,  $e_{g_2} = 0.62\%$  and  $e_q = 3.8\%$ .
XEB [2]: XEB stands for "Linear Cross-Entropy Benchmarking" and is a metric used to assess the performance of quantum processors. It quantifies the fidelity of a quantum computer's output compared to a classical reference model. Mathematically, XEB is defined as:
$$
\mathcal {F} _ {X E B} (x) = 2 ^ {n} \frac {1}{M} \sum_ {x \in \mathcal {D}} p (x) - 1, \tag {6}
$$
where we have a collection of  $M$  random bit-strings  $\mathcal{D} = \{x_1,\dots,x_M\}$ . If a random quantum circuit runs without errors, we have  $\mathcal{F}_{XEB} = 1$ . If bit-strings are sampled from a classical uniform distribution, we have  $\mathcal{F}_{XEB} = 0$ .
By evaluating the XEB metric, researchers can assess the performance of quantum circuits. It provides a quantitative measure of how well a quantum computer reproduces the expected outcomes and is a valuable tool for evaluating and improving quantum computing technologies.
3.3 Classical Simulation Using Tensor Networks
The classical simulation task aims to efficiently calculate  $\pmb{U} = \pmb{U}_m\cdots\pmb{U}_1$  in (7) on classical computers, as marked red in Fig. 3. It is mathematically a combinatorial optimization problem tensor network contraction ordering (TNCO).
$$
| \psi \rangle = \underbrace {U _ {m} \times \cdots \times U _ {1}} _ {\text {c l a s s i c a l s i m u l a t i o n t a k}} | \psi_ {0} \rangle . \tag {7}
$$
Tensor contraction operation: Given two tensors  $\mathcal{X} \in \mathbb{R}^{I \times J \times K}$  and  $\mathcal{Y} \in \mathbb{R}^{K \times M \times N}$ , their contraction results in a 4D tensor  $\mathcal{Z} \in \mathbb{R}^{I \times J \times M \times N}$  where
$$
\mathcal {Z} _ {i, j, m, n} = \sum_ {k = 1} ^ {K} \mathcal {X} _ {i, j, k} \mathcal {Y} _ {k, m, n}, \tag {8}
$$
which takes  $IJKMN$  multiplications. Using the tensor diagram representation, as in Fig. 4, a node denotes a tensor and an edge denotes a tensor contraction operation.
Tensor network representation: Leaving the input quantum bits and the final measurement out, the quantum circuit  $\mathbf{U} = \mathbf{U}_m \cdots \mathbf{U}_1$  can be represented as a tensor network. Specifically, a single-qubit gate  $\mathbf{R}$  is represented as a matrix (a 2D tensor), while a double-qubit quantum gate  $\mathbf{U}$  is represented as a 4D tensor. Using tensor diagrams, a quantum circuit in Fig. 3 is mapped into a tensor network in Fig. 4. Note that the final result of the classical simulation task is an 8D tensor shown in Fig. 4. We provide more details about the tensor representations in Appendix B.
Simulation efficiency: We would like to illustrate that the contraction ordering for calculating  $\mathbf{U} = \mathbf{U}_m \cdots \mathbf{U}_1$  is critical to the simulation efficiency. Different orderings may result in a significant difference in the number of multiplications. The difference may be hundreds of orders for quantum circuits with a large number of qubits and many cycles [34]. For a simple example in Fig. 4, two contraction orderings (blue and red) for the same quantum circuit involve 976 and 5056 multiplications, respectively.
4 Tensor Network Contraction Ordering Using Reinforcement Learning
First, we formulate the classical simulation task as a combinatorial optimization problem, i.e., tensor network contraction order (TNCO). Then, we adopt a K-spin Ising model whose Hamiltonian is used as the loss function to train a policy network via curriculum learning. We also provide a pool of implementation tricks to improve training efficiency and boost performance.
Tensor network contraction order (TNCO). Given a tensor network  $G = (V, E)$ , a contraction path  $P = (e_1, \ldots, e_{n-1})$ ,  $e_t \in E_t$ , and a corresponding sequence of graphs  $(G_1, \ldots, G_{n-1})$ , the
Figure 4: For the quantum ciruits in Fig. 3, two contraction orderings (blue and red) involove 976 and 5056 multiplications, respectively.
goal is to find a path  $P$  with minimum cost,
$$
P ^ {*} (G) = \operatorname {a r g m i n} _ {P} \sum_ {t = 1} ^ {n - 1} R _ {t} \left(e _ {t}\right) \tag {9}
$$
$$
\begin{array}{l} \text {s . t .} P = (e _ {1}, \dots , e _ {n - 1}), e _ {t} \in E _ {t}, \end{array}
$$
where  $E_{t}$  is the set of remaining edges between tensors,  $G_{t}$  is the tensor network after  $t$ -th tensor contraction and the reward  $R_{t}$  is defined as the number of multiplications for the tensor contraction along edge  $e_{t}$ . This formulation is consistent with [34].
Consider an example in Fig. 5, graph  $G_{1}$  has  $V_{1} = (1,2,3,4), E_{1} = (k,j,m,i,s), w = \{K,J,M,I,S\}$ . Assuming that the first contraction operation is on index  $m$  in  $G_{1}$ , tensors 3 and 4 are contracted into tensor 34 at a computation cost of  $IJMS$  multiplications. Then, the graph is updated to  $G_{2}$  with  $V_{2} = (1,2,34), E_{2} = (k,ij,s), w = \{K,IJ,S\}$ . Assuming the second con
Figure 5: An example of tensor network contraction.
traction operation is on index  $k$  in  $G_{2}$ , tensor 1 and 2 are contracted into tensor 12 at a computation cost of  $SKIJ$  multiplications. The updated graph  $G_{3}$  has  $V_{3} = (12,34)$ ,  $E_{3} = (sij)$ ,  $w = \{sij\}$ . Finally, tensors 12 and 34 are contracted into a real number using  $SIJ$  multiplications. The total number of multiplications is  $IJMS + SKIJ + SIJ$ .
4.2 The Proposed Reinforcement Learning Method
First, we reformulate (9) into an Ising model. Then, we extend it to a K-spin Ising model, which allows training a policy network using the curriculum learning method [5].
TNCO using Ising model: The Ising model of TNCO problem uses  $N^2$  spins  $x_{u,j}$ , where  $u$  denotes the tensor and  $j$  denotes its order in the TNCO path. We use  $J_{u,v}^{i}$  to denote the cost introduced by the tensor contraction between  $u$  and  $v$  for the  $i$ -th order, i.e., the number of multiplications. The energy of the original TNCO problem has three terms. The first term requires each tensor to appear at least once in the TNCO path. The second term requires there are exactly two tensors selected at order  $j$  along a path. The third term measures the contraction cost at order  $j$ . These are encoded in the following Hamiltonian:
$$
H (\boldsymbol {x}) = \sum_ {i = 1} ^ {N - 1} \left\{\left(2 - \sum_ {u = 1} ^ {N - i} \boldsymbol {x} _ {u, i}\right) ^ {2} + \sum_ {u = 1} ^ {N} \sum_ {v = 1} ^ {N} J _ {u, v} ^ {i} \boldsymbol {x} _ {u, i} \boldsymbol {x} _ {v, i} \right\}. \tag {10}
$$
As shown for example, in Fig. 6, it can be computed by  $H(x) = (2 - \pmb{x}_{1,1} - \pmb{x}_{2,1} - \pmb{x}_{3,1} - \pmb{x}_{4,1})^2 + (2 - \pmb{x}_{1,2} - \pmb{x}_{2,2} - \pmb{x}_{3,2})^2 + (2 - \pmb{x}_{1,3} - \pmb{x}_{2,3})^2 + w_1(1,4)\pmb{x}_{1,1}\pmb{x}_{4,1} + w_1(1,2)\pmb{x}_{1,1}\pmb{x}_{2,1} + w_1(2,3)\pmb{x}_{2,1}\pmb{x}_{3,1} + w_1(2,4)\pmb{x}_{2,1}\pmb{x}_{4,1} + w_1(3,4)\pmb{x}_{3,1}\pmb{x}_{4,1} + w_2(1,3)\pmb{x}_{1,2}\pmb{x}_{3,2} + w_2(1,2)\pmb{x}_{1,2}\pmb{x}_{2,2} + w_2(2,3)\pmb{x}_{2,2}\pmb{x}_{3,2} + w_3(1,2)\pmb{x}_{1,3}\pmb{x}_{2,3}$
Figure 6: Illustration of TNCO problem: conventional Ising model vs. our K-spin Ising model.
TNCO using K-spin Ising model: Our solution takes K steps to solve the TNCO problem, i.e.,  $\pmb{x}^1 \rightarrow \pmb{x}^2 \rightarrow \dots \rightarrow \pmb{x}^K$ , which can be encoded in the following Hamiltonian,
$$
H \left(\boldsymbol {x} ^ {1}, \dots , \boldsymbol {x} ^ {K}\right) = \sum_ {k = 1} ^ {K} H \left(\boldsymbol {x} ^ {k}\right) + \sum_ {k = 1} ^ {K} \sum_ {i _ {1} \in V ^ {1}} \dots \sum_ {i _ {k} \in V ^ {k}} J _ {i \dots i} ^ {1 \dots k} \boldsymbol {x} _ {i _ {1}} ^ {1} \dots \boldsymbol {x} _ {i _ {k}} ^ {k}, \tag {11}
$$
where  $J_{i_1\dots i_k} = \sum_{k = 2}^{K}\gamma^{K - k}w(i_1,i_2)w(i_2,i_3)\dots w(i_{k - 1},i_k)$ .