SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Policy network: We build the policy network with a transformer neural network. We represent the tensor network with an undirected graph, demonstrated as a symmetric matrix  $M$ . If  $i$ -th and  $j$ -th tensors are connected with a shared index  $d$ , then  $M_{i,j} = d$ . The policy network takes  $M$  as input and generates the contraction ordering  $P = (e_1, \dots, e_{n-1})$ , where  $e_t \in E_t$  is the connected edge between  $u$ -th and  $i$ -th tensors  $x_{u,i}$ .
Challenges: Several challenges for using RL algorithms include the wide dynamic range, huge search space, slow convergence due to Heavy-tailed cost distribution, incorporating existing solvers, and credit assignment problem [34]. The sampling bottleneck is a major challenge in training RL agents [32]. Next, in Section 4.3, we develop a pool of implementation tricks to overcome these challenges, including massively parallel gym-environment, learning to optimize, dual replay buffers, swarm intelligence, and curriculum learning.
4.3 Implementation Tricks
Massively parallel gym-environment. The sampling bottleneck is a major challenge in training RL agents. We implement a massively parallel gym-environment to accelerate the sampling efficiency. We initialize  $N$  independent environments and generate and forward different tensor network contraction paths to these environments at each step. Each environment performs the contraction operations according to the given path, yielding the next state and calculating the number of multiplications as rewards. We store these transitions, including the tensor networks and their corresponding multiplications, in the replay buffer.
Learn to optimize. We adopt the learn-to-optimize (L2O) strategy [10] for training an RL agent. Instead of using conventional optimizers like Adam or Nestrov, we employ a long-short-term memory (LSTM) network. It takes the current gradient and loss value as inputs and produces gradients. This L2O strategy effectively accelerates the convergence process, particularly in the presence of a Heavy-tailed cost distribution. This approach enables faster and more efficient training by dynamically adapting the parameter updates based on the current gradient and loss value.
Dual replay buffers. We maintain two replay buffers, one saves the tensor contraction ordering from the agent in an iterative rollout manner, and the other saves high-quality tensor contraction orders so far. The reason for two replay buffers is that we want to avoid high-quality tensor contraction ordering being deleted by the rule of first in, first out (FIFO) due to the GPU memory limit. We use the Hamiltonian value to measure the quality. We sample from two replay buffers alternatively during the training process, which stabilizes the RL training.
Swarm intelligence. We train multiple optimizers since different optimizer is easily stuck into local optima [7, 39]. To mitigate the impact, different optimizer shares knowledge with others regularly, thus escaping local optima and approaching the global optima. Specifically, during training,
optimizers keep the top- $N$  solutions while optimizers share them. Rather than directly using the orderings for training, the optimizer will manually add some noise to the shared orders to search for the solution in the neighborhood of that local optimum.
Curriculum learning. We apply curriculum learning method [5, 17, 62]. With an acceptable performance on small-scale tensor network contraction problems, the optimizer has successfully escaped multiple local optima. The optimizer may approach the global optima when varying the studied tensor networks from small to large scale. We have two tricks for measuring the problem scale: the number of iterations and parameter freezing strategies.
- The first trick is varying the number of iterations,  $K$ . A large  $K$  may result in insufficient optimization, while in contrast, small-scale optimization problems corresponding to a small  $K$  may experience over-fitting. Therefore, starting with an initial large  $K$ , we gradually decrease  $K$  during the training process.
- The second trick is masking network parameters, i.e., freezing different ratios of parameters. Specifically, at first, we randomly freeze  $90\%$  of the parameters and optimize the rest  $10\%$  parameters, which is easy to optimize. Then, we gradually decrease the ratio of masked parameters. It can help the optimizer escape the local minima.
Note that the two curriculum learning tricks can be integrated with the swarm intelligence trick to boost performance.
5 Performance Evaluation and Benchmark
We present experimental results on both synthetic tensor networks and Google's Sycamore circuits. Further, we investigate the scalability performance of our method for large-scale quantum circuits that have a similar structure to Google's Sycamore circuits.
5.1 Baseline Methods
We provide the following five baselines for verification purposes:
- OE-GREEDY [13]: an open-source solver from OPT-Einsum library, which uses the greedy search algorithm to find the optimal tensor contraction path.
- CTG-Greedy [18]: an open-source solver from the Cotengra library, which uses the greedy search algorithm to find the optimal tensor contraction path.
- CTG-Kahypar [18]: A robust graph partitioning-based solver from the Cotengra library, which achieves state-of-the-art results in many tensor networks.
- ACQDP [21, 22]: Using the stem optimization, including the hypergraph partitioning, local optimization, and dynamic slicing, to search the optima tensor contraction order<sup>2</sup>.
- RL-TNCO [34]: an RL approach combined with graph neural networks to search the optimal tensor contraction order, which is one of the state-of-the-art methods for the classical simulation of quantum circuits<sup>3</sup>.
Experiment setup. We conducted all experiments on a DGX-2 server with NVIDIA A100 GPUs, each of which consists of 48 GB device memory. There are two Intel(R) Xeon(R) Gold 5118 CPUs. Each of CPUs has 12 cores @2.30GHz supporting 24 hardware threads. There are 128 GB DDR4 memories on the server. We set the learning rate  $\eta$  as  $3 \times 10^{-3}$  with a decay factor 0.9 (reduce the learning rate every 1,000 epochs), and the total number of epochs is 1,000,000. We set  $K = 3$  for our K-spin Ising model. The number of parallel environments is 2048, while the large-scale cases use 1024. When the number of tensors increases, the GPU memory consumption increases; therefore, we use 1024 environments for large-scale tensor networks.
It is not generalizable to all tensor networks. We only add ACQDP's performance in Sycamore circuits.
3The codes are not public, so we took results from Table 3 of the paper [34].
5.2 Verification on Synthetic Tensor Networks
Synthetic tensor-train networks. We tested tensor-train networks with tensors from 400 up to 2000. From Table 1, our RL-Ising method achieves a speedup of  $2 \times$  over the CTG-Kahypar method for tensor size from 400 to 1000. Both OE-Greedy and CTG-Greedy cannot work for the problem instances with over 1,500 tensors, while the RL-Ising method exhibits good scalability, outperforming CTG-Kahypar by a speedup of  $1.73 \times$  for tensor sizes 1500 and 2000.
Table 1: Results on synthetic tensor-train networks.
Tensors	400	600	800	1000	1500	2000
Scale	×10120	×10180	×10241	×10301	×10451	×10602
OE-Greedy [13]	17.22	27.67	4.44	3.83	-	-
CTG-Greedy [18]	10.33	16.60	2.67	4.28	-	-
CTG-Kahypar [18]	10.23	16.60	4.67	4.26	14.12	4.57
RL-Ising	5.16	8.28	2.14	2.14	7.01	2.29
Synthetic random tensor networks. We generate random tensor networks with the same settings in [34]. Random networks were generated with varying tensors, namely 25, 50, 75, and 100. The connections in these networks were assigned an average degree of 3, with the degrees of individual nodes independently drawn from the set  $\{2, 3, 4, 5, 6\}$ . We report the mean and median results of 5 random instances for each size. As given in Table 2, our RL-Ising method outperforms all baselines with a speedup of  $3.98 \times$
Table 2: Number of multiplications for synthetic random tensor networks.
Qbits	25	50	75	100
Scale	×104	×107	×1010	×1012
OE-Greedy [13]	53.7/27.7	75.4/11.3	104/4.5	5296/26.4
CTG-Greedy [18]	40.3/20.3	12.8/ 4.2	8.3/0.9	27.9/ 2.2
CTG-Kahypar [18]	46.4/24.8	13.4/ 4.3	4.1/0.4	54.2/ 1.2
RL-TNCO [34]	13.1/12.5	3.2/ 1.8	1.2/0.2	5.5/ 1.8
RL-Ising	12.5/11.4	2.5/ 1.5	0.7/0.1	4.9/ 1.1
5.3 Benchmark Performance on Google's Sycamore Circuits
Google's Sycamore circuits [3] has 53 qubits and  $m$  cycles,  $m = 12, 14, 16, 18, 20$ . As shown in Fig. 3, each cycle has one layer of random single-qubit gates and one layer of two-qubit gates. In different cycles, the two-qubit gates are applied to different pairs of quantum qubits.
The results are summarized in Table 3. Specifically, the RL-Ising method outperforms the RL-TNCO method with a speedup of  $2.84 \times$ . With 20 cycles, our RL-Ising method has a speedup of  $4.64 \times$  over the CTG-Kahypar method,  $5.40 \times$  over the ACQDP method, and
Table 3: Number of multiplications for Google's Sycamore circuits.
Cycles	m = 12	m = 14	m = 16	m = 18	m = 20
Scale	×1010	×1012	×1013	×1016	×1018
OE-Greedy [13]	6.23 × 107	4.77 × 107	7.74 × 1012	6.21 × 1010	9.59 × 108
CTG-Greedy [18]	1.16 × 107	1.91 × 107	1.42 × 1010	3.71 × 107	4.19 × 107
CTG-Kahypar [18]	2.55 × 103	1.41 × 102	1.03 × 104	48.0	6.69
ACQDP [21]	1.09 × 103	71	1.15 × 104	25.8	6.65
RL-TNCO [34]	5.44	7.39	-	-	3.49
RL-Ising	1.31	1.07	9.27	12.98	1.23
$2.42 \times$  over the RL-TNCO method. The estimated running time of ACQDP is 21 days [21, 22]; therefore, the estimated running time of our RL-Ising method is  $21 / 5.40 \approx 3.9$  days.
We provide a comparison of different methods in Fig. 7. With an increasing number of cycles, the number of multiplications in the log scale increases linearly, which corresponds to exponential trends (as illustrated in Fig. 1). Both RL methods, the RL-TNCO method and our RL-Ising method, are much lower than existing heuristic methods. Furthermore, the RL-Ising method is lower than that of the RL-TNCO method, with an improvement of 0.384 orders.
Take-home message I: The "quantum supremacy" claim still lacks an unequivocal first demonstration, since Google's announcement [3] was under serious questions due to the debatable estimate of 10,000 years' running time for the classical simulation task on the Summit supercomputer.
Figure 7: Number of multiplications (in log-scale) by different methods.
Table 4: Number of multiplications on large-scale quantum circuits.
Qbits	100	200	300	400	500
Scale	×1021	×1023	×1025	×1028	×1033
OE-Greedy [13]	8.31 × 109	2.26 × 109	6.90 × 1011	7.55 × 1011	4.34 × 109
CTG-Greedy [18]	3.34 × 108	4.24 × 108	5.37 × 109	4.49 × 109	1.38 × 109
CTG-Kahypar [18]	3.96 × 104	2.31 × 104	3.51 × 103	6.25 × 102	4.18 × 102
RL-Ising	3.85	5.12	2.27	3.98	1.84
5.4 Scale Up to Large-Scale Quantum Circuits
Scalability is very important for evaluating future hardware developments. We scale up to large-scale quantum circuits with 100, 200, 300, 400 and 500 qubits, respectively. There are 20 cycles for each generated quantum circuit, taking a similar structure as in Fig. 3. As given in Table 4, our RL-Ising method surpasses all the baseline methods with 227 speedups. With increasing the scale of quantum circuits, the performance gap between our method and the runner-up method CTG-Kahypar becomes large. It further shows the strong scalability of our RL-Ising method.
Therefore, we believe that reinforcement learning methods have great potential for finding the best performance curve.
Take-home message II: To validate "empirical quantum supremacy" for future quantum hardware developments, the machine learning community is expected to play a critical role in maintaining publicly trustable benchmark performances with open-source training datasets..
In this paper, we have demonstrated the potential of a reinforcement learning approach for the classical simulation of quantum circuits. We reported an estimated simulation time of less than 5 days, which is a remarkable speedup of  $4.62 \times$  over the state-of-the-art heuristic methods. We conduct extensive experiments to evaluate the classical simulation performance. Moreover, we have developed parallel gym environments and benchmarks, which are openly accessible as open-source resources.
However, the provided environments may not cover all types of classical simulation tasks. The best performance is an open question, which asks for continuing efforts. This project may initiate long-term collaborations from the AI/ML and quantum physics communities to maintain reference curves for validating the "empirical quantum supremacy" and drive continuing hardware advancements.
[57] Bram Vanhecke, Jeanne Colbois, Laurens Vanderstraeten, Frank Verstraete, and Frédéric Mila. Solving frustrated Ising models using tensor networks. Physical Review Research, 3(1):013041, 2021.
[58] F. Verstraete and J. I. Cirac. Valence-bond states for quantum computation. Phys. Rev. A, 70:060302, Dec 2004.
[59] Zhi-Yuan Wei, Daniel Malz, and J Ignacio Cirac. Sequential generation of projected entangled-pair states. Physical Review Letters, 128(1):010607, 2022.
[60] Yiqing Zhou, E Miles Stoudenmire, and Xavier Waintal. What limits the simulation of quantum computers? Physical Review X, 10(4):041038, 2020.
[61] Qingling Zhu, Sirui Cao, Fusheng Chen, Ming-Cheng Chen, Xiawei Chen, Tung-Hsun Chung, Hui Deng, Yajie Du, Daojin Fan, Ming Gong, et al. Quantum computational advantage via 60-qubit 24-cycle random circuit sampling. Science bulletin, 67(3):240-245, 2022.
[62] Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, and Dongyan Zhao. Combining curriculum learning and knowledge distillation for dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1284–1295, 2021.
Appendix A Classical simulation task
It should be needed to sample one million samples, achieving the required XEB.
Fig. 3 shows a corresponding circuit example. It has  $n = 2$  qubits for the initial state  $|\psi_0\rangle$  and  $m = 2$  cycles. The sampling process using the quantum circuit is computed as follows,
1. In the first cycle, random selected single-qubit quantum gates are first applied to all the four qubits of  $|\psi_0\rangle$ ; Then, the double-qubit quantum gates are applied to  $R_1^0 |\psi_0\rangle_{0,1}$  and  $R_1^2 |\psi_0\rangle_{2,3}$ , respectively, and obtain  $|\psi_1\rangle$ ;
2. In the second cycle, random selected single-qubit quantum gates are first applied to all the four qubits of  $|\psi_1\rangle$ ; Then, the double-qubit quantum gates are applied to  $\pmb{R}_2^1 |\psi_1\rangle_{1,2}$ , and obtain  $|\psi_2\rangle$ ;
3. Next, random selected single-qubit quantum gates are first applied to all the four qubits of  $|\psi_2\rangle$ , and obtain  $R_2^j |\psi_1\rangle_j$ ,  $j = 0,1,2,3$ ;
4. Last, we have a measurement  $\alpha_{i_1i_2i_3i_4} = \langle i_1i_2i_3i_4|\psi_3\rangle$ . The sampled output is a bit-string  $\alpha_{i_1i_2i_3i_4}$ .
Algorithm 1 Random circuit sampling
1: Input: initial state  $|\psi_0\rangle$ , number of cycles  $m$ , single-qubit quantum gate  $\{\sqrt{X},\sqrt{Y},\sqrt{W}\}$ , double-qubit quantum gate  $U_{i}$   
2: for  $i = 1,\ldots,m$  do  
3: for  $j = 0,\ldots,n-1$   
4:  $R_i^j \gets$  randomly select from  $\{\sqrt{X},\sqrt{Y},\sqrt{W}\}$ ,  
5:  $|\psi_i\rangle_j = R_i^j |\psi_{i-1}\rangle_j$ ,  
6: end for  
7: for  $j = (i+1)\%2,\ldots,n/2-1$   
8:  $p \gets$  compute the manipulate quantum bit index  $2j + (i+1)\%2$   
9:  $|\psi_i\rangle_{p,p+1} = U_i^p |\psi_{i-1}\rangle_{p,p+1}$ ,  
10: end for  
11: end for  
12: for  $j = 0,\ldots,n-1$   
13:  $R_{m+1}^j \gets$  randomly select from  $\{\sqrt{X},\sqrt{Y},\sqrt{W}\}$ ,  
14:  $|\psi_{m+1}\rangle_j = R_{m+1}^j |\psi_m\rangle_j$ ,  
15: end for  
16: Obtain a measurement by  $\alpha_{i_1i_2\dots i_n} = \langle i_1i_2\dots i_n|\psi_{m+1}\rangle$ ,  
17: Output: a bit-string  $\alpha_{i_1i_2\dots i_n}$ .
Quantum circuits: There have been many quantum circuits proposed as follows,
- Sycamore quantum [3]: It consists of 53 qubits and 20 cycles. For the Boson sampling problem, it only needs 200 seconds to finish this task, while it needs 10,000 years for classical simulation.
- Jiu Zhang [60]: It consists of 74 qubits. For the Gaussian Boson Sampling problem, it can use 200s to finish up to a million times compared with classical simulations.
- Zuchongzhi [61]: It has 60 qubits with the number of 24. The achieved sampling task is about 6 orders of magnitude more difficult than that of Sycamore in the classic simulation.
Appendix B Data Generators and Reinforcement Learning Environments.
B.1 Tensor Network Representations
We use the unidirectional graph to represent the tensor network. Specifically, for a given tensor network with  $n$  tensors, we use a symmetric  $n \times n$  matrix  $M$  to represent the dimensional relationships between tensors, which is named the dimension matrix. Specifically, if the tensors with index  $i$  and  $j$  are connected with a shared dimension as  $d_{ij}$ , then we set  $M_{ij} = M_{ji} = d_{ij}$ , otherwise, it is
Figure 8: Example of the tensor network contraction environment.
set as 0. We also denote a connection matrix  $C$ , where  $C_{ij} = C_{ji} = 1$  indicates the connection between the  $i$ -th tensor and  $j$ -th tensor.
B.2 Tensor Network Contraction
We employ tensor network contraction as the underlying computational framework in our quantum circuit simulation. Given a specific tensor network structure and a specified contraction order, the environment is designed to provide the number of multiplications required during the tensor contraction process.
Specifically, as illustrated in Figure 8, we consider a tensor network comprising 4 tensors connected in a grid structure. When performing a tensor contraction between tensors  $C$  and  $D$ , it is necessary to compute the number of multiplications involved. From the unidirectional graph representation, tensor  $C$  has dimensions  $M \times J$ , tensor  $D$  has dimensions  $S \times I \times M$ , and a shared edge with dimension  $M$  exists.
To contract tensors  $C$  and  $D$  along the edge with dimension  $M$ , the number of multiplications can be computed as  $\frac{(MJ) \times (SIM)}{M}$ . This expression takes into account the multiplication of the dimensions  $MJ$  from tensor  $C$  and  $SIM$  from tensor  $D$ , divided by the shared dimension  $M$ .
Upon completing the contraction, it is essential to update the unidirectional graph representation. The contracted tensor formed by the contraction of  $C$  and  $D$  retains an independent edge with dimension  $M$  ( $M_{33} = M_{34} = M_{43} = M_{44} = M$ ). The connected edges between the contracted tensor ( $C$  or  $D$ ) and other tensors are determined by multiplying the original edge dimensions between them ( $M_{23} = M_{24} = M_{32} = M_{42} = IJ$ ). This update to the unidirectional graph ensures accuracy and reflects the changes resulting from the contraction step.
By incorporating the update of the unidirectional graph and computing the number of multiplications, we complete the current simulation step.
B.3 Verification
Unified representation: We use a tuple consisting of the contracted tensor indices to represent the tensor contraction order. The resulting tensor is labeled as a new index. As shown for example in Fig. 5, we first write down (3,4) for the first contraction operation. We label the generated tensor 34 as 5. Next, we use (1,2) to label the second tensor contraction operation and 6 to label the resulting tensor 12. Last, we contract the tensor 12 and 34, where we use (5,6) to represent the contraction record. Thus, we have the contraction order  $\{(3,4),(1,2),(12,34)\}$
Calculating the number of multiplications: Given the tensor contraction order, we will compute the number of involved multiplications. In Fig. 5, using different optimizers, we can get a specific contraction order, like  $\{(3,4),(1,2),(12,34)\}$ . We first contract the 3-th and 4-th tensor, of which the number of multiplication is  $IJMS$ . Then, for the contraction order  $(1,2)$ , we have the number of multiplication as  $SKIJ$ . Last, for the contraction order  $(12,34)$ , we have the number of multiplication as  $SIJ$ .
(a) Tensor-train network.
(b) Tensor-ring network.
Figure 9: Illustration of the tensor train and tensor ring network.
B.4 Datasets for Different Tensor Networks
In this section, we will first introduce the data structure used to describe the relationships between different tensor nodes. Then, we will demonstrate the generation process of the dataset for different tensor networks.
Data Structure: The tensor network can be represented as an undirected graph  $G = \{V, E, w\}$  and is amenable to be stored using the adjacent table, which can be defined as follows (C-format),
class TensorNetwork {
    int V; // number of tensor nodes
    int E; // number of edges
    Queue[] adj; // adjacent nodes for each node
    TensorNetwork(int V) { // init the graph
        this.V = V; this.E = 0;
        this.adj = new Queue[V];
        for (i = 0; i < V; i++) {
            this.adj[i] = new Queue {};
        }
    }
    void AddEdge(int v, int w) { // add the edges
        if (w > v) v, w = w, v;
        this.adj[v].enqueue(w);
        this.E++;
    }
    void Build(...); // differs in different tensor networks
Specifically, we use  $V$  and  $E$  to save the number of tensor nodes and edges, respectively. The variable  $adj$  is a queue with flexible length, where  $adj[j]$  is also a queue to save the adjacent tensor nodes. Given the total number of tensor nodes  $V$ ,  $TensorNetwork(V)$  is used to initialize the undirected graph, where each tensor node is assigned an empty queue. The  $addEdge(\cdot, \cdot)$  is invoked to add the connected relationship between the connected tensor nodes  $v$  and  $w$ . We make a simplified assumption that only the larger index  $w$  can be added to the adjacent queue of  $v$ . Different tensor network differs in the implementation of the  $Build(\dots)$  function, which invokes the  $addEdge(\cdot, \cdot)$  depending on the specified tensor network structure.
Tensor-Train Network or Matrix Product States: The matrix product state (MPS) [41, 56] or tensor-train (TT) represents a tensor as a chain-like contraction of third-order tensors with the head and tail as matrices. The tensor diagram of MPS/TT is shown in Fig. 9(a).
MPS has been widely used in modeling quantum circuits such as [47, 16, 45]. In the Build(...) function, for each tensor node  $i$ , we need to use the AddEdge(i, i + 1) to add the  $i + 1$ -th node to the adjacent lists of  $i$ -th node,  $i < |V| - 1$ . We can set up different numbers of tensor nodes  $N$  to generate the dataset for MPS tensor network.
void TensorNetwork::Build() { // Build for MPS
    for (i = 0; i < this.V-1; i++) this.AddEdge(i, i+1); }
The generation code for the tensor-train network is saved in https://github.com/XiaoYangLiu-FinRL/RL4QuantumCircuits/blob/main/datasets/mps/mps_generate.py.
Tensor ring network: The tensor ring (TR) represents a high-order (or high-dimensional) tensor by a sequence of 3rd-order tensors that are multiplied circularly, whose tensor diagram notation can be represented in Fig. 9(b).
The tensor-ring network has been utilized to simulate the quantum circuit in [44].
The main difference between the tensor ring and the MPS tensor network is that the first and last tensor nodes in tensor ring network are also connected. Thus, only a minor modification of MPS generation can be applied to generate the tensor ring network.
void TensorNetwork::Build(){ // Build for tree tensor  
for (int i = 0; i < this.V; i++)  
this.AddEdge(i, (i+1)%this.V); }
The generation code for the tensor-ring network is saved in https://github.com/XiaoYangLiu-FinRL/RL4QuantumCircuits/blob/main/datasets/tr/tr_generate.py.
Tree Tensor Network: Tree tensor network (TTN) [37, 52, 55] or Hierarchical Tucker (HT) is a generalization of MPS that encodes a tree entanglement structure. The diagram notation of a TTN can be represented in Fig. 10.
Figure 10: Tree tensor network.
Tree tensor network has been used to model the quantum [24, 50], and quantum chemistry [36, 37].