Title: ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Autoregressive neural network

tensor network

quantum many-body physics

variational Monte Carlo
Abstract: Quantum many-body physics simulation has important impacts on understanding fundamental science and has applications to quantum materials design and quantum technology. However, due to the exponentially growing size of the Hilbert space with respect to the particle number, a direct simulation is intractable. While representing quantum states with tensor networks and neural networks are the two state-of-the-art methods for approximate simulations, each has its own limitations in terms of expressivity and inductive bias. To address these challenges, we develop a novel architecture, Autoregressive Neural TensorNet (ANTN), which bridges tensor networks and autoregressive neural networks. We show that Autoregressive Neural TensorNet parameterizes normalized wavefunctions, allows for exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. We demonstrate our approach on quantum state learning as well as finding the ground state of the challenging 2D $J_1$-$J_2$ Heisenberg model with different systems sizes and coupling parameters, outperforming both tensor networks and autoregressive neural networks. Our work opens up new opportunities for quantum many-body physics simulation, quantum technology design, and generative modeling in artificial intelligence.

ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation

Abstract

Quantum many-body physics simulation has important impacts on understanding fundamental science and has applications to quantum materials design and quantum technology. However, due to the exponentially growing size of the Hilbert space with respect to the particle number, a direct simulation is intractable. While representing quantum states with tensor networks and neural networks are the two state-of-the-art methods for approximate simulations, each has its own limitations in terms of expressivity and inductive bias. To address these challenges, we develop a novel architecture, Autoregressive Neural TensorNet (ANTN), which bridges tensor networks and autoregressive neural networks. We show that Autoregressive Neural TensorNet parameterizes normalized wavefunctions, allows for exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. We demonstrate our approach on quantum state learning as well as finding the ground state of the challenging 2D J 1 -J 2 Heisenberg model with different systems sizes and coupling parameters, outperforming both tensor networks and autoregressive neural networks. Our work opens up new opportunities for quantum many-body physics simulation, quantum technology design, and generative modeling in artificial intelligence.

1 Introduction

Quantum many-body physics is fundamental to our understanding of the universe. It appears in high energy physics where all the fundamental interactions in the Standard Model, such as quantum electrodynamics (QED) and quantum chromodynamics (QCD), are described by quantum mechanics. In condensed matter physics, quantum many-body physics has led to a number of rich phenomena and exotic quantum matters, including superfluids, superconductivity, the quantum Hall effect, and topological ordered states (Girvin & Yang, 2019). As an application, quantum many-body physics is crucial for new materials design. The electronic structure problem and chemical reactions in quantum chemistry are governed by quantum many-body physics. The recent development of quantum computers is also deeply connected to quantum many-body physics. A multi-qubit quantum device is intrinsically a quantum many-body system, such that progress on quantum computer engineering is tied to our understanding of quantum many-body physics (Preskill, 2021).

All information of a closed quantum manybody system is captured by the wavefunction, whose properties are described by the famous Schrödinger equation. An important tool to study and understand quantum many-body physics is to classically simulate the wavefunction. However, the wavefunction is a high dimensional function in Hilbert space, whose dimension grows exponentially with the number of particles. For example, for qubit systems where each qubit has two degrees of freedom), the wavefunction of 300 qubits will have dimension 2 300 , which is larger than the number of atoms in the observable universe. Furthermore, the Schrödinger equation is a complex-valued

ANTN

Bridging the gap

𝜓𝜓

(

𝑥𝑥1

,

𝑥𝑥2

,

𝑥𝑥3

,

𝑥𝑥4

)

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

𝜓𝜓

(

𝑥𝑥1

,

𝑥𝑥2

,

𝑥𝑥3

,

𝑥𝑥4

)

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

𝑥𝑥1

𝑥𝑥2

𝑥𝑥3

𝑥𝑥4

𝜓𝜓

(

𝑥𝑥1

,

𝑥𝑥2

,

𝑥𝑥3

,

𝑥𝑥4

)

TN

Pro: Physics inductive bias

Con: Limited expressivity

Pro: High expressivity

Con: No physics prior

high-dimensional equation, which is challenging to solve or simulate in general.

A number of algorithms have been developed to simulate quantum many-body physics, including quantum Monte Carlo, tensor networks, neural network quantum states, and quantum computation. In particular, computing the ground state of quantum many-body systems is of great interest. One important approach is the variational principle, which provides an upper bound for the ground state energy. To apply the variational principle successfully, one must design an ansatz that can represent and optimize the wavefunction efficiently. Tensor networks and neural network quantum states (Carleo & Troyer, 2017) are the two main state-of-the-art methods that can be applied with the variational principle for quantum many-body simulation. However, tensor networks usually suffer from an expressivity issue in systems with more than one dimension, while neural network quantum states usually lack inductive bias from the underlying physics structure and have sign structure challenges in the representation.

In this paper, we develop a novel architecture, Autoregressive Neural TensorNet (ANTN), to bridge neural network quantum states and tensor networks, achieving the best of both worlds. In particular, our contributions are threefold:

Develop ANTN with two variants called 'elementwise' and 'blockwise,' which each naturally generalize the two state-of-the-arts ansatzes, tensor networks (TN) and autoregressive neural networks (ARNN), to provide proper inductive bias and high expressivity.

Prove that ANTN is normalized with exact sampling, has generalized expressivity of TN and ARNN, and inherits multiple symmetries from ARNN.

Demonstrate our methods on quantum state learning and variationally finding the ground state of the challenging 2D J 1 -J 2 Heisenberg model, outperforming both TN and ARNN.

2 Related Work

Tensor Networks (TN) represent high-dimensional wavefunctions using low-rank tensor decomposition, notably matrix product state (MPS) (Vidal, 2003, 2004), PEPS (Verstraete & Cirac, 2004), and MERA (Vidal, 2007). They capture the entanglement structure of physical systems and, with algorithms like the density matrix renormalization group (DMRG) (White, 1992), are used for state simulations and real-time dynamics. However, their expressivity can be limited in systems with more than one dimension and systems with high entanglement. In machine learning, TN appears as tensor train (Oseledets, 2011) and CP decomposition methods.

Neural Network Quantum State (NNQS) leverages neural networks for high dimensional wavefunction representation (Carleo & Troyer, 2017). It's been demonstrated that many quantum states can be approximated or represented by NNQS (Sharir et al., 2021; Gao & Duan, 2017; Lu et al., 2019; Levine et al., 2019a; Luo et al., 2021b,a; Deng et al., 2017; Huang & Moore, 2021; Vieijra et al., 2020), and it has yielded state-of-the-art results in computing quantum system properties (Gutiérrez & Mendl, 2020; Schmitt & Heyl, 2020; Vicentini et al., 2019; Yoshioka & Hamazaki, 2019; Hartmann & Carleo, 2019; Nagy & Savona, 2019; Luo et al., 2021b, 2022a). Key advancements in NNQS include ARNN that improves sample efficiency and gradient estimation, and the development of

neural networks that adhere to the underlying symmetries of quantum systems (Luo et al., 2021b; Hibat-Allah et al., 2020; Choo et al., 2019; Luo & Clark, 2019; Hermann et al., 2019; Pfau et al., 2020; Luo et al., 2021a, 2022b, 2021b; Chen et al., 2022). Despite its potential, NNQS faces challenges such as the lack of physics prior and the sign structure problem. While there are attempts to integrate NNQS with TN, including matrix product state with neural network backflow (Lami et al., 2022) and generalizing MPS to RNN (Wu et al., 2022), the former can not produce normalized wavefunctions, while the latter only builds on tensor contractions that lacks the nonlinear activation functions of standard neural network wavefunctions.

3 Background

3.1 Quantum Preliminaries

In this work, we focus on qubit systems in quantum many-body physics. The wavefunction or quantum state ψ of the system is a normalized function ψ : Z n 2 → C with ∑ x | ψ ( x ) | 2 = 1 , where n is the system size. The input to the wavefunction is an n -bit string x ∈ { 0 , 1 } n . Therefore, the wavefunction ψ can be viewed as a complex-valued vector of size 2 n , with Dirac notation ⟨ ψ | and | ψ ⟩ correspond to a conjugate row vector and a column vector respectively, and ψ ( x ) = ⟨ x | ψ ⟩ . Because the size of the wavefunction grows exponentially with the system size n , a direct computation quickly becomes intractable as the system size increases. The goal of NNQS is to design a compact architecture that can approximate and optimize the wavefunction efficiently.

3.2 Quantum State Learning

Given a quantum state | ϕ ⟩ , we are often interested in finding | ψ θ ⟩ that is closest to | ϕ ⟩ given the variational ansatz. In quantum mechanics, the closeness of two quantum states is measured by the quantum fidelity F = |⟨ ϕ | ψ θ ⟩| 2 = | ∑ x ϕ ∗ ( x ) ψ θ ( x ) | 2 where ∗ refers to complex conjugation. Quantum fidelity satisfies 0 ≤ F ≤ 1 , with F = 1 corresponding to exact match, and F = 0 orthogonal quantum states. Finding F max for a given ansatz allows us to quantify how good the ansatz can be used to approximate that state. In practice, minimizing -log F is usually better than maximizing F itself. This can be achieved by enumerating all the basis x in small systems and stochastically in large systems such as quantum state tomography (see Appendix A.1).

3.3 Variational Monte Carlo

For a given quantum system with n qubits, the Hamiltonian ˆ H can be written as a Hermitian matrix of size 2 n × 2 n . The ground state energy E g of the system is the smallest eigenvalue of ˆ H and the ground state is the corresponding eigenvector. For large system sizes, finding the ground state directly is usually impossible. In this case, the variational principle in quantum mechanics provides an upper bound on the ground state energy E g . For all (normalized) wavefunctions | ψ ⟩ , it is evidental that E g ≤ ⟨ ψ | ˆ H| ψ ⟩ . Therefore, finding the | ψ θ ⟩ that minimizes E θ = ⟨ ψ θ | ˆ H| ψ θ ⟩ gives the lowest upper bound of the ground state energy. For large system sizes, we can stochastically evaluate and minimize

where H x , x ′ refers to the matrix element of ˆ H and we interpret | ψ θ ( x ) | 2 as a probability distribution. The summation over x ′ can be efficiently computed given x since the Hamiltonian is usually sparse. The gradient ∇ θ E θ can also be calculated stochastically in a similar fashion (see Appendix A.2).

4 Method

4.1 Preliminaries

Autoregressive Neural Network Wavefunction. The autoregressive neural network (ARNN) (Fig. 1 left) parameterizes the full probability distribution as a product of conditional probability distributions

where x = ( x 1 , . . . , x n ) is a configuration of n qubits and x <i = ( x 1 , . . . , x i -1 ) is any configuration before x i . The normalization of the full probability can be guaranteed from the normalization of individual conditional probabilities. The ARNN also allows for exact sampling from the full distribution by sampling sequentially from the conditional probabilities (see Appendix B.1). The autoregressive constructions for probabilities can be easily modified to represent quantum wavefunction by (1) replacing p ( x i | x <i ) with a complex-valued conditional wavefunction ψ ( x i | x <i ) and (2) using the following normalization condition for the conditional wavefunctions: ∑ x | ψ ( x i | x <i ) | 2 = 1 (Sharir et al., 2020; Hibat-Allah et al., 2020; Luo et al., 2021b). Similar to the case of probabilities, ARNN automatically preserves the normalization of wavefunctions and allows for exact sampling (see Appendix B.1). Because of this, ARNN is often more efficient when training and computing various quantum observables compared to other generative neural networks.

Matrix Product State (MPS). MPS (also known as tensor train) is a widely used TN architecture to study quantum many-body physics (Vidal, 2003, 2004). The MPS defines a wavefunction using n rank-3 tensors M α i -1 α i x i with α i -1 (or α i ) the index of the left (or right) bond dimensions and x i the configuration of the i th qubit. Then, the full wavefunction is generated by first choosing a particular configuration x = ( x 1 , . . . , x n ) and then contracting the tensors selected by this configuration (Fig. 1 right) as

where the left-and-right-most bond dimensions are assumed to be D ( α 0 ) = D ( α n +1 ) = 1 so are not summed.

4.2 Autoregressive Neural TensorNet (ANTN)

Elementwise and Blockwise Construction for ANTN. Both ARNN and MPS are powerful ansatzes for parameterizing quantum many-body wavefunctions. Albeit very expressive, ARNN lacks the physics prior of the system of interest. In addition, since wavefunctions are complex-valued in general, learning the sign structure can be a nontrivial task (Westerhout et al., 2020). MPS, on the other hand, contains the necessary physics prior and can efficiently represent local or quasi-local sign structures, but its expressivity is severely limited. The internal bond dimension needs to grow exponentially to account for a linear increase in entanglement. It turns out that MPS representation is not unique in that many MPS actually represent the same wavefunction; and if we choose a particular constraint (without affecting the expressivity), MPS allows for efficient evaluation of conditional probability and exact sampling (Ferris & Vidal, 2012) (see Appendix B.2). Because both these ansatzes allow for exact sampling, it is natural to combine them to produce a more powerful ansatz. Therefore, we develop the Autoregressive Neural TensorNet (ANTN) (Fig. 1 (middle)). In the last layer of the ANTN, instead of outputting the conditional wavefunction ψ ( x i | x <i ) , we output a conditional wavefunction tensor ˜ ψ α i -1 α i ( x i | x <i ) for each site. Defining the left partially contracted tensor up to qubit j as ˜ ψ α j L ( x ≤ j ) := ∑ α <j ∏ j i =1 ˜ ψ α i -1 α i ( x i | x <i ) , we can define the (unnormalized) marginal probability distribution as

where ∗ denotes complex conjugation. Then, the (normalized) conditional probability can be obtained as q ( x j | x <j ) = q ( x ≤ j ) / ∑ x j q ( x ≤ j ) . We construct the overall wavefunction by defining both its amplitude and phase according to

with q ( x ) =: ∏ n i =1 q ( x i | x <i ) and the phase ϕ ( x ) =: Arg ∑ α ∏ n i =1 ˜ ψ α i -1 ,α i ( x i | x <i ) . In other words, we define the amplitude of the wavefunction through the conditional probability distributions and define the phase analogous to the standard MPS.

We develop two different constructions for ANTN that differ in the last layer on how to construct conditional wavefunction tensors.

The elementwise ANTN is given by

where f NN ( x i , α i -1 , α i | x <i ) is the complex-valued output. The blockwise ANTN is given by

where the complex-valued output f NN ( x i | x <i ) is broadcasted over α . This results in a lower complexity that allows us to use a larger maximum bond dimension.

Transformer and PixelCNN Based ANTN. Our construction above is general and can be applied to any standard ARNN. Depending on the application, we can use different ARNN architectures. In this work, we choose the transformer and PixelCNN depending on the specific tasks. The transformer (Vaswani et al., 2017) used here is similar to a decoder-only transformer implemented in (Luo et al., 2020), and The PixelCNN we use is the gated PixelCNN (Van den Oord et al., 2016) implemented in (Chen et al., 2022).

MPS Initialization. Since our ANTN generalizes from TN, both the elementwise and the blockwise can take advantage of the optimized MPS from algorithms such as DMRG (White, 1992) as an initialization. In practice, we can initialize the TN component with the optimized DMRG results of the same bond dimension (similar to (Lami et al., 2022; Wu et al., 2022)). The MPS Initialization can also be thought of as a pretraining of the ANTN. This is a nice feature since it provides a good sign structure and inductive bias from the physics structure, which does not exist in the conventional ARNN.

Limitations. In this work, we only integrated MPS into the ANTN, where MPS may not be the best TN in various settings. Besides MPS, many other TNs also allow efficient evaluation of conditional probabilities and exact sampling, such as MERA (Vidal, 2007) and PEPS (Verstraete & Cirac, 2004), or cylindrical MPS for periodic boundary conditions. In fact, the recently developed TensorRNN (Wu et al., 2022) can also be viewed as a type of TN and can be integrated into our construction for future work. In addition, while the ANTN construction is general in terms of the base ARNN used, our current study only focuses on transformer and PixelCNN. Lastly, as shown later in Sec. 5.1, the current implementation of ANTN has an additional sampling overhead that is linear in system size which can be avoided.

5 Theoretical Results

5.1 Exact Sampling and Complexity Analysis of ANTN

Theorem 5.1. Autoregressive Neural TensorNet wavefunction is automatically normalized and allows for exact sampling.

Proof. This is a direct consequence that we defined the amplitude of the wavefunction through normalized conditional probability distributions q ( x i | x <i ) . (See Appendix B.1 for the detailed sampling procedure.)

Complexity Analysis. We first note that for MPS, the number of parameters and computational complexity for evaluating a bitstring x scales as O ( nχ 2 ) , where n is the number of particles and χ = D ( α ) is the (maximum) bond dimension of MPS. The sampling complexity scales as O ( n 2 χ 2 ) . The DMRG algorithm has a computational complexity of O ( nχ 3 ) (White, 1992). The number of parameters and computational complexity of ARNN depends on the specific choice. Assuming the ARNN has n ARNN parameters and a single pass (evaluation) of the ARNN has a computational complexity of c ARNN , then the sampling complexity of the ARNN is O ( nc ARNN ) in our current implementation. The ANTN based on such ARNN would have O ( n ARNN + nχ 2 h dim ) parameters for the elementwise construction, and O ( n ARNN + nχ 2 + nh dim ) parameters for the blockwise construction. The computational complexity for evaluating and sampling bitstrings scales as O ( n s ( c ARNN + nχ 2 h dim )) for elementwise construction and O ( n s ( c ARNN + nχ 2 )) for blockwise construction with s = 0 for evaluation and s = 1 for sampling.

We note that the complexity analysis above is based on our current implementation, where the sampling procedure for all algorithms has an O ( n ) overhead compared to evaluation. It is possible to remove it by storing the partial results during the sampling procedure.

Then, each gradient update of the variational Monte Carlo algorithm requires sampling once and evaluating N con times where N con is the number of connected (non-zero) x ′ 's in H x , x ′ given x , which usually scales linearly with the system size. Then, the overall complexity is O ( N s ( N con c eva + c samp )) with N s the batch size, c eva the evaluation cost and c samp the sampling cost. Usually, the first part dominates the cost.

The difference in computational complexities and number of parameters between elementwise ANTN and blockwise ANTN implies that blockwise ANTN is usually more economical than elementwise ANTN for the same hidden dimsnieon h dim and bond dimension χ . Therefore, for small bond dimensions, we use the elementwise ANTN for a more flexible parameterization with a higher cost. In contrast, for large bond dimensions, we use the blockwise ANTN, which saves computational complexity and improves the initial performance (with DMRG initialization) of the blockwise ANTN at the cost of less flexible modifications from the ARNN. We also note that compared to the state-ofthe-art MPS simulation, even for our blockwise ANTN, a small bond dimension usually suffices. For example, in the later experimental section, we use bond dimension 70 for blockwise ANTN while the best MPS results use bond dimension 1024, which has much more parameters than our ANTN. In general, our ANTN has much fewer parameters compared to MPS due to the O ( χ 2 ) parameters scaling which dominates at large bond dimension.

5.2 Expressivity Results of ANTN

Theorem 5.2. Autoregressive Neural TensorNet can have volume law entanglement, which is strictly beyond the expressivity of matrix product states.

Proof. We proved in Thm. 5.3 that ANTN can be reduced to an ARNN, which has been shown to have volume law entanglement that cannot be efficiently represented by TN (Sharir et al., 2021; Levine et al., 2019b). Hence, ANTN can represent states that cannot in general be represented by MPS efficiently; thus it has strictly greater expressivity.

Theorem 5.3. Autoregressive Neural TensorNet has generalized expressivity over tensor networks and autoregressive neural networks.

Proof. Thm B.2of Appendix B.4 shows that both TN and ARNN are special cases as ANTN and Thm B.3of Appendix B.4 shows that ANTN can be written as either a TN or an ARNN with an exponential (in system size) number of parameters. Thus, ANTN generalizes the expressivity over both TN and ARNN.

5.3 Symmetry Results of ANTN

Symmetry plays an important role in quantum many-body physics and quantum chemistry. Many of the symmetries can be enforced in ARNN via the two classes of the symmetriesmask symmetry and function symmetry .

Definition 5.1 (Mask Symmetry) . A conditional wavefunction ψ ( x i | x <i ) has a mask symmetry if ψ ( x i | x <i ) = 0 for some x i given x <i .

Definition 5.2 (Function Symmetry) . A conditional wavefunction tensor ψ ( x i | x <i ) has a function symmetry over a function F if ψ ( x i | x <i ) = ψ ( F ( x i ; x <i ) | F ( x <i )) where F ( x ≤ i ) := { F ( x 1 ) , F ( x 2 ; x 1 ) , . . . , F ( x i ; x <i ) } .

Here, we list several symmetries that ANTN inherits from ARNN and show the proofs in Appendix B.4.

Theorem 5.4. Autoregressive Neural TensorNet inherits mask symmetry and function symmetry from autoregressive neural networks.

Corollary 5.4.1 (Global U(1) Symmetry) . Autoregressive Neural TensorNet can realize global U(1) symmetry, which conserves particle number.

Corollary 5.4.2 ( Z 2 Spin Flip Symmetry) . Autoregressive Neural TensorNet can realize Z 2 spin flip symmetry such that the wavefunction is invariant under conjugation of the input.

MPS (4)

MPS(16)

MPS (64)

MPS (256)

ANTN (4)

0.0

0.2

0.4

0.6

0.8

1.0

Fidelity

(a)

MPS (4)

MPS(16)

MPS (64)

MPS (256)

ANTN (4)

0.0

0.2

0.4

0.6

0.8

1.0

Fidelity

(b)

Shallow Random Circuit

no sign rule

sign rule

6 Experiments

6.1 Quantum State Learning

As stated previously, ANTN generalizes both ARNN and TN to take advantage of both the expressivity and the inductive bias. Here, we test the ANTN on learning physically important quantum states to demonstrate this ability. In this task, we use the transformer neural network for ARNN and ANTN due to the 1D structure of the system.

Experiments on expressivity. We first test the expressivity of the ANTN by learning a class of well-known high-entangled states-random permuted Bell states. A 2-qubit Bell state is defined as ( | 00 ⟩ + | 11 ⟩ ) / √ 2 , which has 1 bit of entanglement. For a system size of n , we randomly pair qubits in the first half of the system with the qubits in the second half of the system to be Bell states. This process creates quantum states with n/ 2 bit of entanglement between the first and second half of the system. It can be shown that a bond dimension of 2 n/ 2 is required for MPS to fully represent such a system. In Fig. 2 (a), we plot the quantum fidelity of learning 16-qubit random Bell states. As the figure shows, MPS cannot represent the states without reaching the required bond dimension of 256 = 2 8 . The ARNN and ANTN, on the other hand, can represent such states without limitations from the bond dimension.

Experiments on inductive bias. We then test the physics inductive bias of the ANTN. One of the physics inductive biases of MPS is that it can represent wavefunctions with local or quasi-local sign structures that can be hard for neural networks to learn. These wavefunctions can have a fluctuating sign depending on the configuration x . Here, we use (real-valued) shallow random circuits to mimic the short-range interaction and generate states with sign structures. We test the algorithms on these states both with and without the 'sign rule', which means that we explicitly provide the sign structure to the algorithm. As shown in Fig. 2 (b), the fidelity of MPS only depends weakly on the sign rule, whereas the fidelity of ARNN can change drastically. Our ANTN inherits the property of MPS and is thus not affected by the sign structure. Furthermore, being more expressive, ANTN with a bond dimension of 4 already performs better than MPS with a bond dimension of 16.

6.2 Variational Monte Carlo

We further test our algorithm on finding the ground state of the challenging 2D J 1 -J 2 Heisenberg model with open boundary condition. The model has a rich phase diagram with at least three different phases across different J 2 /J 1 values (Capriotti et al., 2004) (Lante & Parola, 2006). In addition, the

Elementwise

16

36

64

100

144

Number of qubits

0.01

0.00

0.01

J

2 =0.2

J

2 =0.5

J

2 =0.8

complicated structure of its ground state makes it a robust model on which to test state-of-the-art methods. Here, we use the PixelCNN for ARNN and ANTN due to the 2D geometry of the system. The 2D J 1 -J 2 Hamiltonian is given by

where subscript refers to the site of the qubits, ⟨· , ·⟩ is the nearest neighbour and ⟨⟨· , ·⟩⟩ is the next nearest neighbour. σ ⃗ i · σ ⃗ j = X i ⊗ X j + Y i ⊗ Y j + Z i ⊗ Z j with X , Y and Z the Pauli matrices

We will fix J 1 = 1 and vary J 2 in our studies.

Experiments on inductive bias of J 1 -J 2 Heisenberg Model.

As shown previously, it can be challenging for neural networks to learn the sign structures. The ground state of the J 1 -J 2 model also has a sign structure, which can be partially captured by the Marshall sign rule(Marshall, 1955). The sign rule is exact at J 2 = 0 and becomes worse for large J 2 . This could introduce bias to the neural network wavefunction if directly applied. We compare the restricted Boltzmann machine (RBM) (from NetKet (Vicentini et al., 2022)), our implementation of gated PixelCNN, and the two different constructions of ANTN with and without the sign rule.

The result is shown in Table 1. Since our approach is based on the variational principle discussed in Sec. 3, it provides an upper bound on the exact ground state energy; therefore, a lower energy implies a better state. As shown in the results, both RBM and PixelCNN improve significantly with the application of the sign rule at J 2 = 0 . 2 and J 2 = 0 . 5 , but the results deteriorate at J 2 = 0 . 8 . This is expected because the sign rule becomes increasingly inaccurate as J 2 increases, especially past J 2 = 0 . 5 . Our ANTN, on the other hand, does not require the sign rule in both constructions. As an additional comparison, we note that both of our ANTN constructions achieved better results compared to the recently developed matrix product backflow state (Lami et al., 2022), which uses the sign rule and has a per site energy of -1 . 9267 at J 2 = 0 . 5 .

Ablation study on J 1 -J 2 Heisenberg Model. We scan across many J 2 for the model without using the sign rule. In this experiment, we compare the performance of six models to compute the ground state energy of the J 1 -J 2 Hamiltonian system with J 1 = 1 fixed and J 2 = 0 . 2 -0 . 8 with 0.1 increments, covering three different phases of the model. The first three models are TN models, using the MPS with bond dimensions of 8, 70, and 1024. For up to 4 × 4 system, the MPS results with bond dimension 1024 are exact and can be regarded as a benchmark. The fourth model is a PixelCNN pure neural network; the fifth and sixth models are elementwise and blockwise ANTN models. Thus the experiment compares the ANTN against the two previous state-of-the-art techniques.

Table 2 summarizes the per site ground state energy computed by different models at different J 2 's in three distinct phases. Our results provide strong evidence that ANTN integrating TN and ARNN outperforms each base model, achieving state-of-the-art results.

Comparison between ANTN and MPS. In all cases the ANTN surpasses the MPS with the corresponding bond dimension on which they are based. For example, even in the 10 × 10 system with J 2 > 0 . 5 , where MPS with a bond dimension of 1024 outperforms the ANTN, the elementwise ANTN still significantly improves on the base MPS with a bond dimension of 8 and the blockwise ANTN improves on the base MPS with a bond dimension of 70. It is consistent with Theorem. 5.3 and Theorem. 5.2 that ANTN is more expressive than TN.

In addition, the ANTN models scale well for larger systems. Figure 3 visualizes the scalability of ANTN compared to MPS. The figure plots the difference in per site energy between the elementwise ANTN with bond dimension 8 and MPS with bond dimension 1024 for J 2 = 0 . 2 , 0 . 5 , 0 . 8 where lower energies signify better performance for the elementwise ANTN. Compared to MPS, the ANTN models compute better ground state energies as the system size grows larger. As the system size increases, the elementwise ANTN with bond dimension 8 starts to outperform MPS with bond dimension 1024. Even at J 2 = 0 . 6 , where the elementwise ANTN is slightly worse than MPS, the difference gets significantly smaller at 12 × 12 compared to 10 × 10 (Table 2). In fact, the elementwise ANTN achieves such a performance using only ∼ 1% the number of parameters of MPS. We note that for large J 2 , the system goes into a stripped phase (Nomura & Imada, 2021), which can be less challenging for MPS to represent. Nevertheless, in almost all cases our ANTN still outperforms MPS on the 12 × 12 system. MPS has a cost dominated by the bond dimension (quadratic for memory and cubic for computational), which limits its use in practice for large system sizes that require large bond dimensions. According to the complexity analysis in Sec. 5.1, ANTN has lower complexity than TN and thus scales better for larger systems.

Comparison between ANTN and ARNN. The elementwise and blockwise ANTN models also consistently outperform the pure neural network PixelCNN model. This agrees with Theorem 5.3 that both elementwise and blockwise ANTN have a generalized expressivity compared to ARNN. In Appendix C we further compare ANTN with ARNN and find that: (a) it is challenging to surpass ANTN by ARNN even with an increased number of parameters; (b) the improvement from ANTN mainly comes from the effective inductive bias of MPS structure instead of the DMRG initialization; (c) ANTN has a favorable scaling compared to ARNN and MPS for reaching accurate results.

The details of all experiment setups and hyperparameters can be found in Appendix D.

7 Conclusion

In this paper, we developed Autoregressive Neural TensorNet, bridging the two state-of-the-art methods in the field, tensor networks, and autoregressive neural networks. We proved that Autoregressive Neural TensorNet is self-normalized with exact sampling, naturally generalizes the expressivity

of both tensor networks and autoregressive neural networks, and inherits proper physics inductive bias (e.g. sign structures) from tensor networks and various symmetries from autoregressive neural networks. We demonstrated our approach on quantum state learning and the challenging 2D J 1 -J 2 Heisenberg model with different system sizes and couplings. Our approach achieves better performance than both the original tensor network and autoregressive neural network while surpassing tensor networks with large bond dimensions as the system size increases. In addition, our approach is robust, independent of sign rule. Besides scientific applications, since both tensor networks and autoregressive neural networks have been applied to machine learning tasks such as supervised learning and generative modeling, our novel approach holds promise for better performance in these domains due to its exact sampling, expressivity, and symmetries.

Broader Impact

Our Autoregressive Neural TensorNet, blending tensor networks and autoregressive neural networks, could advance our grasp of quantum phenomena, potentially fueling scientific breakthroughs. It enhances quantum material design and quantum computing through improved simulations and control of quantum states. This technology may also inspire new machine learning models for handling high-dimensional data. However, possible ethical and societal impacts, such as the use for chemical weapon development, require careful scrutiny.

References

Barrett, T. D., Malyshev, A., and Lvovsky, A. Autoregressive neural-network wavefunctions for ab initio quantum chemistry. Nature Machine Intelligence , 4(4):351-358, 2022.

Capriotti, L., Fubini, A., Roscilde, T., and Tognetti, V. Ising transition in the two-dimensional quantum j 1- j 2 heisenberg model. Physical review letters , 92(15):157202, 2004.

Carleo, G. and Troyer, M. Solving the quantum many-body problem with artificial neural networks. Science , 355(6325):602-606, 2017. doi: 10.1126/science.aag2302. URL https://www.science. org/doi/abs/10.1126/science.aag2302 .

Chen, Z., Luo, D., Hu, K., and Clark, B. K. Simulating 2+ 1d lattice quantum electrodynamics at finite density with neural flow wavefunctions. arXiv preprint arXiv:2212.06835 , 2022.

Choo, K., Neupert, T., and Carleo, G. Two-dimensional frustrated J 1 -J 2 model studied with neural network quantum states. Phys. Rev. B , 100:125124, Sep 2019. doi: 10.1103/PhysRevB.100.125124. URL https://link.aps.org/doi/10.1103/PhysRevB.100.125124 .

Deng, D.-L., Li, X., and Das Sarma, S. Quantum entanglement in neural network states. Physical Review X , 7(2), May 2017. ISSN 2160-3308. doi: 10.1103/physrevx.7.021021. URL http: //dx.doi.org/10.1103/PhysRevX.7.021021 .

Ferris, A. J. and Vidal, G. Perfect sampling with unitary tensor networks. Phys. Rev. B , 85:165146, Apr 2012. doi: 10.1103/PhysRevB.85.165146. URL https://link.aps.org/doi/10.1103/ PhysRevB.85.165146 .

Gao, X. and Duan, L.-M. Efficient representation of quantum many-body states with deep neural networks. Nature Communications , 8(1):662, Sep 2017. ISSN 2041-1723. doi: 10.1038/ s41467-017-00705-2. URL https://www.nature.com/articles/s41467-017-00705-2 .

Girvin, S. M. and Yang, K. Modern condensed matter physics . Cambridge University Press, 2019.

Greensmith, E., Bartlett, P. L., and Baxter, J. Variance reduction techniques for gradient estimates in reinforcement learning. J. Mach. Learn. Res. , 5:1471-1530, December 2004. ISSN 1532-4435.

Gutiérrez, I. L. and Mendl, C. B. Real time evolution with neural-network quantum states, 2020.

Hartmann, M. J. and Carleo, G. Neural-network approach to dissipative quantum many-body dynamics. Phys. Rev. Lett. , 122:250502, Jun 2019. doi: 10.1103/PhysRevLett.122.250502. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.250502 .

Hermann, J., Schätzle, Z., and Noé, F. Deep neural network solution of the electronic schrödinger equation, 2019.

Hibat-Allah, M., Ganahl, M., Hayward, L. E., Melko, R. G., and Carrasquilla, J. Recurrent neural network wave functions. Phys. Rev. Research , 2:023358, Jun 2020. doi: 10.1103/ PhysRevResearch.2.023358. URL https://journals.aps.org/prresearch/abstract/10. 1103/PhysRevResearch.2.023358 .

Huang, Y. and Moore, J. E. Neural network representation of tensor network and chiral states. Phys. Rev. Lett. , 127:170601, Oct 2021. doi: 10.1103/PhysRevLett.127.170601. URL https: //link.aps.org/doi/10.1103/PhysRevLett.127.170601 .

Lami, G., Carleo, G., and Collura, M. Matrix product states with backflow correlations. Physical Review B , 106(8), aug 2022. doi: 10.1103/physrevb.106.l081111. URL https://doi.org/10. 1103%2Fphysrevb.106.l081111 .

Lante, V. and Parola, A. The ising phase in the j1-j2 heisenberg model. Physical Review , 2006.

Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architectures. Physical Review Letters , 122(6), Feb 2019a. ISSN 1079-7114. doi: 10.1103/physrevlett.122. 065301. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122. 065301 .

Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architectures. Physical review letters , 122(6):065301, 2019b.

Lu, S., Gao, X., and Duan, L.-M. Efficient representation of topologically ordered states with restricted boltzmann machines. Phys. Rev. B , 99:155136, Apr 2019. doi: 10.1103/PhysRevB.99.155136. URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.99.155136 .

Luo, D. and Clark, B. K. Backflow transformations via neural networks for quantum manybody wave functions. Physical Review Letters , 122(22), Jun 2019. ISSN 1079-7114. doi: 10. 1103/physrevlett.122.226401. URL https://journals.aps.org/prl/abstract/10.1103/ PhysRevLett.122.226401 .

Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via a probabilistic formulation, 2020.

Luo, D., Carleo, G., Clark, B. K., and Stokes, J. Gauge equivariant neural networks for quantum lattice gauge theories. Physical review letters , 127(27):276402, 2021a.

Luo, D., Chen, Z., Hu, K., Zhao, Z., Hur, V . M., and Clark, B. K. Gauge invariant autoregressive neural networks for quantum lattice models, 2021b. URL https://arxiv.org/abs/2101.07243 .

Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via a probabilistic formulation. Phys. Rev. Lett. , 128:090501, Feb 2022a. doi: 10. 1103/PhysRevLett.128.090501. URL https://link.aps.org/doi/10.1103/PhysRevLett. 128.090501 .

Luo, D., Yuan, S., Stokes, J., and Clark, B. K. Gauge equivariant neural networks for 2+ 1d u (1) gauge theory simulations in hamiltonian formulation. arXiv preprint arXiv:2211.03198 , 2022b.

Marshall, W. Antiferromagnetism. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences , 232(1188):48-68, 1955. ISSN 00804630. URL http: //www.jstor.org/stable/99682 .

Nagy, A. and Savona, V. Variational quantum monte carlo method with a neural-network ansatz for open quantum systems. Phys. Rev. Lett. , 122:250501, Jun 2019. doi: 10.1103/PhysRevLett.122. 250501. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122. 250501 .

Nomura, Y. and Imada, M. Dirac-type nodal spin liquid revealed by refined quantum many-body solver using neural-network wave function, correlation ratio, and level spectroscopy. Physical Review X , 11(3), aug 2021. doi: 10.1103/physrevx.11.031034. URL https://doi.org/10. 1103%2Fphysrevx.11.031034 .

Oseledets, I. V . Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):2295-2317, 2011. doi: 10.1137/090752286. URL https://doi.org/10.1137/090752286 .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems , pp. 8026-8037, 2019.

Pfau, D., Spencer, J. S., Matthews, A. G. D. G., and Foulkes, W. M. C. Ab initio solution of the manyelectron schrödinger equation with deep neural networks. Phys. Rev. Research , 2:033429, Sep 2020. doi: 10.1103/PhysRevResearch.2.033429. URL https://journals.aps.org/prresearch/ abstract/10.1103/PhysRevResearch.2.033429 .

Preskill, J. Quantum computing 40 years later. arXiv preprint arXiv:2106.10522 , 2021.

Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jones, M., Klein, A., Milechin, L., Mullen, J., Prout, A., Rosa, A., Yee, C., and Michaleas, P. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC) , pp. 1-6. IEEE, 2018.

Schmitt, M. and Heyl, M. Quantum many-body dynamics in two dimensions with artificial neural networks. Physical Review Letters , 125(10), Sep 2020. ISSN 1079-7114. doi: 10.1103/physrevlett.125. 100503. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.125. 100503 .

Sharir, O., Levine, Y., Wies, N., Carleo, G., and Shashua, A. Deep autoregressive models for the efficient variational simulation of many-body quantum systems. Physical Review Letters , 124(2), jan 2020. doi: 10.1103/physrevlett.124.020503. URL https://doi.org/10.1103% 2Fphysrevlett.124.020503 .

Sharir, O., Shashua, A., and Carleo, G. Neural tensor contractions and the expressive power of deep neural quantum states, 2021.

Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems , 29, 2016.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30: 5998-6008, 2017.

Verstraete, F. and Cirac, J. I. Renormalization algorithms for quantum-many body systems in two and higher dimensions, 2004. URL https://arxiv.org/abs/cond-mat/0407066 .

Vicentini, F., Biella, A., Regnault, N., and Ciuti, C. Variational neural-network ansatz for steady states in open quantum systems. Physical Review Letters , 122(25), Jun 2019. ISSN 1079-7114. doi: 10.1103/physrevlett.122.250503. URL https://journals.aps.org/prl/abstract/10. 1103/PhysRevLett.122.250503 .

Vicentini, F., Hofmann, D., Szabó, A., Wu, D., Roth, C., Giuliani, C., Pescia, G., Nys, J., VargasCalderón, V., Astrakhantsev, N., and Carleo, G. NetKet 3: Machine learning toolbox for many-body quantum systems. SciPost Physics Codebases , aug 2022. doi: 10.21468/scipostphyscodeb.7. URL https://doi.org/10.21468%2Fscipostphyscodeb.7 .

Vidal, G. Efficient classical simulation of slightly entangled quantum computations. Physical Review Letters , 91(14), oct 2003. doi: 10.1103/physrevlett.91.147902. URL https://doi.org/10. 1103%2Fphysrevlett.91.147902 .

Vidal, G. Efficient simulation of one-dimensional quantum many-body systems. Physical Review Letters , 93(4), jul 2004. doi: 10.1103/physrevlett.93.040502. URL https://doi.org/10.1103% 2Fphysrevlett.93.040502 .

Vidal, G. Entanglement renormalization. Phys. Rev. Lett. , 99:220405, Nov 2007. doi: 10.1103/PhysRevLett.99.220405. URL https://link.aps.org/doi/10.1103/PhysRevLett. 99.220405 .

Vieijra, T., Casert, C., Nys, J., De Neve, W., Haegeman, J., Ryckebusch, J., and Verstraete, F. Restricted boltzmann machines for quantum states with non-abelian or anyonic symmetries. Physical Review Letters , 124(9), Mar 2020. ISSN 1079-7114. doi: 10.1103/physrevlett.124.097201. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.124.097201 .

Westerhout, T., Astrakhantsev, N., Tikhonov, K. S., Katsnelson, M. I., and Bagrov, A. A. Generalization properties of neural network approximations to frustrated magnet ground states. Nature Communications , 11(1):1593, Mar 2020. ISSN 2041-1723. doi: 10.1038/s41467-020-15402-w. URL https://doi.org/10.1038/s41467-020-15402-w .

White, S. R. Density matrix formulation for quantum renormalization groups. Physical review letters , 69(19):2863, 1992.

Wu, D., Rossi, R., Vicentini, F., and Carleo, G. From tensor network quantum states to tensorial recurrent neural networks. arXiv preprint arXiv:2206.12363 , 2022.

Yoshioka, N. and Hamazaki, R. Constructing neural stationary states for open quantum manybody systems. Phys. Rev. B , 99:214306, Jun 2019. doi: 10.1103/PhysRevB.99.214306. URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.99.214306 .

Appendices

A Additional Background

A.1 Quantum State Learning

In this section, we discuss more details about quantum state learning. As mentioned in the main paper, the quantum fidelity

measures the closeness of the two (normalized) quantum states | ϕ ⟩ and | ψ θ ⟩ , with ∗ denoting complex conjugation. By minimizing -log F , one can obtain the | ψ θ ⟩ closest to the target state | ϕ ⟩ (see Appendix D for optimization details). For small system sizes ( ≲ 20 qubits), it is possible to enumerate all the basis x and compute -log F exactly as in the case of this work. As mentioned in the main paper, we learn the random Bell state and shallow random circuit to test the expressivity and physics inductive bias of the ansatz. More specifically, the states are generated as follows

Random Bell State. A two-qubit Bell state is defined to be the state ( | 00 ⟩ + | 11 ⟩ ) / √ 2 . We use the following algorithm to generate a n -qubit random Bell state ( n is a multiple of 2 ).

where bell _ state( · , · ) creates a Bell state given the two qubits and product _ state([]) creates the tensor product state for a list of individual states. Each two-qubit Bell state in the random Bell state forms between the left half and right half of the system, allowing a maximum entanglement across a cut in the middle of the system.

Shallow Random Circuit. The shallow random circuit generates random states using the following algorithm.

where n l is the number of layers, and random _ gate( · , · ) generates a random unitary gate on the two qubits. The random _ gate( · , · ) function is realized by first generating a (real-valued) Gaussian random matrix of size 4 × 4 , following a QR decomposition of the matrix to obtain the unitary part as the random unitary gate.

Each of the above algorithms defines a distribution of states, which we average over multiple realizations to compute the mean and standard deviation of the learned quantum fidelity.

However, for large system sizes, the Hilbert space is too large, so one has to evaluate the fidelity stochastically. This can be achieved by rewriting

Which allows the fidelity to be evaluated by sampling from | ϕ | 2 . The gradient of -log F can be written as

Alternatively, one can sample from ψ θ ( x ) by writing the complex conjugated

This time, we have to be careful with

Then,

We can further use the variance reduction technique (Greensmith et al., 2004) from reinforcement learning and write

where the minus 1 comes from the mean of the coefficient in front of ∇ θ log ψ ∗ θ ( x )

One can verify that without the assumption that | ψ θ ⟩ is normalized (i.e. F = |⟨ ϕ | ψ θ ⟩| 2 / ⟨ ψ θ | ψ θ ⟩ ), Eq. A.7 can also be obtained without applying the variance reduction formula.

In the main paper, due to the small system size N = 16 , we enumerated all the bases when evaluating the quantum fidelity.

A.2 Variational Monte Carlo

As shown in the main paper, given a Hamiltonian ˆ H (and its matrix elements H x , x ′ )

Since the Hamiltonian ˆ H is usually sparse, give x , one only needs to sum up a small number of x ′ 's for the numerator (usually linear in system size), allowing Eq. A.9 to be evaluated efficiently. Analogously, we can evaluate the gradient as

Furthermore, it is possible to reduce the variance by either directly applying the variance reduction formula (Greensmith et al., 2004), or explicitly normalizing | ψ θ ⟩ (minimizing ⟨ ψ θ | ˆ H| ψ θ ⟩ / ⟨ ψ θ | ψ θ ⟩ ) to obtain

where ⟨ ψ θ | ˆ H| ψ θ ⟩ can be approximated by applying Eq. A.9 on the same batch x . In this work, we use Eq. A.11 as the gradient of the loss function (see Appendix D for optimization details).

B Additional Theoretical Results

B.1 Exact Sampling from Conditional Probabilities

Suppose a full probability distribution over multiple qubits is written as a product of conditional probabilities as

with x <i = ( x 1 , . . . x i -1 ) , then sampling a bitstring x from the probability distribution can be obtained by sequentially sampling each conditional probability as Since each sample is sampled independently using this algorithm, a batch of samples can be obtained in parallel, allowing for an efficient implementation on GPUs. In addition, the exact sampling nature of the algorithm makes it not suffer from the autocorrelation time problem of standard Markov chain Monte Carlo.

This same algorithm applies to sampling from | ψ ( x ) | 2 with a simple replacement p ( x i | x <i ) → | ψ ( x i | x <i ) | 2 . This is a direct consequence of

x i

Algorithm 3 Exact Sampling from Conditional Probabilities/Wavefunctions

B.2 Exact Sampling from MPS

As mentioned in the main text, matrix product state (MPS) with particular constraints, also allows for exact sampling. Recall that MPS defines a wavefunction as

Let's first assume that the wavefunction is normalized. It is easy to notice that any gauge transformation

with A being an invertible matrix leaves the resulting wavefunction invariant. Here, we suppress the α indices and view them as the indices for matrices. It has been shown that by fixing this gauge redundancy (Vidal, 2003, 2004), we can restrict all the M matrices to satisfy the following condition

where δ · , · is the Kronecker delta function. The matrices M that satisfy this condition are called isometries, and the MPS is called in the right canonical form (to be distinguished from the left canonical form). In the right canonical form, each M α i -1 α i x i can be interpreted as a basis transformation ( x i , α i ) → α i -1 that is part of a unitary matrix.

Theorem B.1. Defining the left partially contracted tensors

If the matrix product state is in the right canonical form, then the marginal probability of the wavefunction satisfies

Proof. Let's define the right partially contracted tensors analogously

We will first show that

We can prove this by induction:

· Base case:

which directly comes from Eq. B.5 by noticing that D ( α n ) = 1 so it can be ignored.

then,

· Inductive step: Write

Assuming then

Using this result, we can then prove our desired result about the marginal probability. Let's write

Corollary B.1.1. Matrix product state allows for efficient exact sampling.

Proof. Thm. B.1 shows that marginal probability distribution q ( x ≤ i ) can be evaluated efficiently. The conditional probability can be evaluated efficiently either as q ( x i | x <i ) = q ( x ≤ i ) /q ( x <i ) or (equivalently) as an explicit normalization of the marginal probability distribution q ( x i | x <i ) = q ( x ≤ i ) / ∑ x i q ( x ≤ i ) . Then, Algorithm 3 can be used to sample from the full distribution.

B.3 Autoregressive Sampling Order

Algorithm 3 samples from conditional probabilities, which requires an ordering of the system to be defined. For 1D systems, we use a linear ordering such that qubit 1 corresponds to the leftmost qubit, and qubit n corresponds to the rightmost qubit. This ordering is natural for 1D MPS and transformer, as well as the transformer-based ANTN. For 2D systems, we use a snake (zig-zag) ordering. In this ordering, the qubit at the 2D location ( i, j ) is defined to be the i × L y + j th qubit, where L y is the number of qubits along the second dimension. This ordering is inherently defined from the masked convolution for PixelCNN and PixelCNN-based ANTN. The MPS is reshaped to satisfy the same ordering for 2D systems. The same 2D ordering can be generalized to other tensor networks as well.

B.4 Additional Details of Expressivity and Symmetry Results of ANTN

Theorem B.2. Both tensor networks and autoregressive neural networks are special cases of Autoregressive Neural TensorNet.

Proof. Recall that the ANTN defines the wavefunction from its amplitude and phase

ANTN reduces to MPS. If we don't modify the tensors with neural networks, ˜ ψ α i -1 ,α i ( x i | x <i ) reduces to M α i -1 ,α i x i and ˜ ψ α j L ( x ≤ j ) reduces to M L α j x ≤ j . It is then straightforward that Eq. B.19 reduces to Eq. B.15 and thus the probability distribution of ANTN reduces to that of MPS. Analogously, Eq. B.21 reduces to the phase of an MPS, and thus ANTN wavefunction reduces to an MPS wavefunction.

ANTN reduces to ARNN. If we set all D ( α ) = 1 , then ˜ ψ α i -1 α i ( x i | x <i ) reduces to ψ ( x i | x <i ) and all the summations over α can be dropped. In this case, q ( x ≤ j ) reduces to ∏ j i =1 | ψ ( x i | x <i ) | 2 and q ( x j | x <j ) reduces to | ψ ( x j | x <j ) | 2 , which is the same as the standard ARNN. In addition, the phase ϕ ( x ) reduces to Arg ∏ n i =1 ψ ( x i | x <i ) which is also the same as ARNN. Thus, ANTN wavefunction reduces to ARNN wavefunction.

Notice that we only showed one type of TN, i.e. MPS, but the proof can be easily generalized to any TN that allows evaluation of marginal probabilities.

Theorem B.3. Autoregressive Neural TensorNet can be written as either a tensor network or an autoregressive neural network with an exponential (in system size) number of parameters.

Proof.

where and

ANTN as TN. In ANTN, the base ARNN outputs the conditional tensors ˜ ψ α i -1 α i ( x i | x <i ) to form the final wavefunctions. These conditional tensors can be written as ˜ ψ α i -1 α i x 1 ,x 2 ...,x i in tensor notation. It directly follows from the shape of these tensors that they contain χ 2 · 2 i elements (where χ 2 is from the two bond dimensions and 2 i is from all the qubits at or before the current qubit i ). According to Theorem 5.2 ARNN has volume law entanglement while TN doesn't. The conditional tensors generated from ARNN do not permit efficient tensor decomposition. Therefore, a tensor network almost has to parameterize all full conditional tensors, resulting in ∑ N i =1 χ 2 · 2 i ∼ O ( χ 2 · 2 N ) parameters in total.

ANTN as ARNN. In ANTN, the conditional probability is calculated from the marginal probability as shown in Eq. 4 as

The summation over α i 's contains χ 2 i -1 terms, each of which can be viewed as a marginal (quasi-)probability q α 1 ,α ′ 1 ,...,α i ( x ≤ i ) generated by the underlying ARNN of ANTN. This

effectively creates a weight matrix on top of a conventional ARNN, whose shape is h dim × χ 2 i -1 × 2 (where h dim is the hidden dimension, and 2 is the local dimension of each qubit) on top of a conventional ARNN, resulting in ∑ N i =1 2 · h dim · χ 2 i -1 ∼ O ( h dim · χ 2 N ) parameters.

Definition B.1 (Mask Symmetry) . A conditional wavefunction ψ ( x i | x <i ) has a mask symmetry if ψ ( x i | x <i ) = 0 for some x i given x <i .

Theorem B.4. Autoregressive Neural TensorNet inherits mask symmetry from autoregressive neural network.

Proof. The mask symmetry results in ψ ( x ) = 0 for certain x satisfying the condition while preserving the normalization of ψ ( x ) . For ANTN, we can directly set q ( x i | x <i ) = 0 for the x i given x <i . This results in ψ ( x ) = 0 for the same x .

Corollary B.4.1 (Global U(1) Symmetry) . Autoregressive Neural TensorNet can realize global U(1) symmetry, which conserves particle number.

Proof. A global U(1) symmetry in a qubit system manifests as a conservation of the number of 0 's and number of 1 's in x . Equivalently, ψ ( x ) = 0 for x violates such conservation. (Hibat-Allah et al., 2020) has shown that autoregressive neural networks can preserve global U(1) symmetry as a mask symmetry, which ANTN inherits.

Definition B.2 (Function Symmetry) . A conditional wavefunction tensor ψ ( x i | x <i ) has a function symmetry over a function F if ψ ( x i | x <i ) = ψ ( F ( x i ; x <i ) | F ( x <i )) where F ( x ≤ i ) := { F ( x 1 ) , F ( x 2 ; x 1 ) , . . . , F ( x i ; x <i ) } .

Theorem B.5. Autoregressive Neural TensorNet inherits function symmetry from autoregressive neural networks.

Proof. We can apply the function symmetry on the left partially contracted tensors instead of the conditional wavefunctions. Here, we show that this produces the desired conditional probabilities and phase. Applying the function symmetry on the left partially contracted tensors results in

This implies that the marginal probability satisfies q ( x ≤ j ) = q ( F ( x ≤ j )) after contracting over the index α j , which then results in the correct symmetry on the conditional probabilities

as the conditional probability is just the marginal probability normalized at site j . The phase, on the other hand,

which satisfies the same function symmetry from Eq. B.23.

Corollary B.5.1 ( Z 2 Spin Flip Symmetry) . Autoregressive Neural TensorNet can realize Z 2 spin flip symmetry such that the ANTN wavefunction is invariant under conjugation of the input.

Proof. Spin flip symmetry exists for quantum chemistry systems that do not couple spin up and spin down electron, so that the wavefunction is invariant to inputs when spin up and down are flipped. (Barrett et al., 2022) has shown that autoregressive neural network can preserve Z 2 spin flip symmetry as a function symmetry, which ANTN inherits.

Corollary B.5.2 (Discrete Abelian and Non-Abelian Symmetries) . Autoregressive Neural TensorNet can realize discrete Abelian and Non-Abelian symmetries.

Proof. Gauge symmetry is a local symmetry such that the function is invariant under local transformation. (Luo et al., 2021b) has shown that autoregressive neural networks can preserve discrete Abelian and non-Abelian symmetries as either the mask symmetry or the function symmetry, which ANTN inherits.

C Additional Experiments

In this section, we perform additional experiments to further compare ANTN and ARNN, and show that

It is challenging to surpass ANTN by ARNN even with increased number of parameters.

The improvement from ANTN mainly comes from the effective inductive bias of MPS structure instead of the DMRG initialization.

ANTN has a favorable scaling compared to ARNN and MPS for reaching accurate results.

Table 3: Energy per site ↓ , tensor network parameters N params , TN , neural network parameters N params , NN , total parameters N params and runtime T training for 10 × 10 system with various algorithms. Elementwise (EW) and blockwise (BW) are two constructions of ANTN with PixelCNN as the underlying ARNN. For MPS, the number in the parenthesis labels the bond dimension. For PixelCNN, the two numbers label the number of layers and hidden dimension respectively. For ANTN, the numbers label (in this order) the number of layers, hidden dimension, and bond dimension. For PixelCNN, we also test pertaining (P) using MPS result (with bond dimension the last number in parenthesis), and for EW, we in addition test random initialization (RI) of the MPS part. The algorithms in boldface are new experiments not previously shown in the main paper. Best energies for each J 2 are also highlighted in boldface. The MPS with DMRG algorithm is run on CPUs, while the PixelCNN, elementwise (EW) and blockwise (BW) ANTN are trained using GPUs, with PixelCNN (P) (7-48) and EW (RI) (7-48-8) using A100 GPUs (labeled with † ), and the rest using V100 GPUs.

10 4

10 5

10 6

10 7

10 8

Number of parameters

1.95

1.90

1.85

1.80

1.75

Energy per site

PixelCNN

EW

BW

10×10

12×12

0

50

100

150

200

250

300

Runtime [hrs]

PixelCNN

EW

BW

10×10

12×12

C.1 Comparison between ANTN and ARNN with More Layers

In Table 3 and 4 we include additional results.

In Table 3, we perform 2 addition tests: a) pretrain pure PixelCNN by learning MPS results; b) train ANTN(elementwise) without DMRG initialization. We find that DMRG pretraining negatively affects the result, while ANTN without DMRG initialization performs as good as with DMRG initialization. This is understandable, as learning MPS results with complex sign structure is essentially the same task as learning a quantum state from shallow random circuit, which has been shown a difficult task for ARNN in Figure 2(b) of the original paper. Since the MPS result is an approximation of the actual ground state, learning this state actually makes a worse initialization for PixelCNN. (We note that the PixelCNN was originally initialized from the result of smaller system sizes, as we described in Appendix D under transfer learning.) On the other hand, the ANTN, despite not using DMRG initialization, still retains the ability to represent flexible sign structures. In addition, since DMRG is just an optimization algorithm to train MPS, the gradient-descent-based algorithm should still train the MPS part of ANTN reasonably well without DMRG initialization.

In Table 4, we test a series of ARNNs with different layers (up to 20 layers) and hidden dimensions (up to 72 hidden dimensions) such that the largest ARNN tested has more parameters than ANTN. We found that all the ARNN energies are not as good at ANTN (which is only based on 7 layers and 48 hidden dimensions), despite that ARNNs have more parameters and take a longer time to train using the same GPU (see new data below). Furthermore, we observe that an increase of parameters can help ARNN to improve energy in general, but this improvement hits a diminishing return potentially due to difficulties in optimization as the number of parameters increases. In addition, we further tested the (approximate) sign rule and found that it could help ARNN obtain better energies but still worse than ANTN without the sign rule. This indicates that the ANTN has the advantage of inheriting the flexible sign structure from MPS, avoiding the manual bias of adding the sign rule. The new results have provided strong support for our expectation, both from an expressivity perspective and from the fact that ANTN has a better physics inductive bias than ARNN for optimization. We have also added a plot for the new benchmark in the updated manuscript. Although the added layer improves the result to some extent, the PixelCNN fails to beat ANTN in terms of energy calculation. This is consistent with Theorem 5.3 that ANTN is more expressive than ARNN.

C.2 Comparison of Runtime and Number of Parameters

In Table 3, Table 4, Table 5 and Figure 4, we show the number of parameters and runtime of each algorithm. In summary, MPS has the most parameters, and is the slowest with DMRG algorithm, taking as long as 2 weeks. The ANTN is much more efficient compared to MPS while obtaining better energies, and it is comparable to pure ARNN while obtaining much better energies. Within the two types of ANTN, the elementwise construction, despite using more parameters, runs faster than the blockwise construction. We note that since TNs and ARNNs use the parameters very differently, a direct comparison of the two types of parameters may be unfair. Therefore, we list both the total number of parameters and the parameters for the individual parts (TN part and ARNN part).

D Experimental Details and Reproducibility

We note that while the following details can be used to reproduce our results, additional adjustments may be possible to further improve our results. The code associated with this paper is available at https://github.com/antn2023/ANTN

Implementation. The transformer (Vaswani et al., 2017) used in this work is a decoder-only transformer implemented in (Luo et al., 2020), and the PixelCNN used is the gated PixelCNN (Van den Oord et al., 2016) implemented in (Chen et al., 2022) but without the channelwise mask.

Hyperparameters. For the transformer in quantum state learning, we use 2 layers, 32 hidden dimensions, and 16 attention heads, whereas for the PixelCNN in variational Monte Carlo, we use 7 layers with dilations 1, 2, 1, 4, 1, 2, and 1 for each layer and 48 hidden dimensions. For additional experiments with more layers, we always use dilation of 1 for the layer 8 and beyond.

Initialization. Throughout the experiment, we initialize the weight matrices of the last fully connected layer of the ARNN and ANTN to 0, and initialize the bias according to the following rule:

The bias of ARNN is always randomly initialized.

For the random Bell state learning, the bias of ANTN is randomly initialized

For the rest of the experiment, the bias of ANTN is set to 0.

The rest of the parameters of ARNN and ANTN are randomly initialized according to PyTorch (Paszke et al., 2019)'s default initialization scheme.

In addition, for the random Bell state learning experiment, we do not initialize the ANTN with MPS (and hence the random bias), whereas for the shallow random circuit learning experiment, we initialize the ANTN with an MPS that comes from explicit truncation of the full wavefunction, and for the variational Monte Carlo experiment, we initialize the ANTN with DMRG optimized MPS.

Optimization. Through the experiment, we use the Adam optimizer with an initial learning rate of 0.01. For quantum state learning experiments, we train the ARNN and ANTN with full basis for 2000 iterations, where the learning rate is halved at iterations 600, 1200, and 1800. In case the full basis cannot be computed in one pass due to GPU memory constraint, we divide the full basis into several batches and accumulates the gradient in each iteration. For variational Monte Carlo experiments, we train the ARNN and ANTN stochastically until the energy converges. During the training, we halve the learning rate at iterations 100, 500, 1000, 1800, 2500, and 4000. In each experiment, we choose the maximum batch size that can be fed into the GPU memory, while using accumulation steps to keep the effective batch size around 10000 throughout the training.

Transfer Learning. We in addition take advantage of transfer learning. For J 2 = 0 . 2 , 0 . 5 , and 0 . 8 , we begin from scratch with the 4 × 4 system. Then, an N × N system is initialized from an ( N -2) × ( N -2) system by keeping all but the weights in the last layer. Then, for other J 2 , the neural network is initialized from one of J 2 = 0 . 2 , 0 . 5 , and 0 . 8 that is closest to the current J 2 of the same system size by keeping all but the weights in the last layer. We find this allows the optimization to converge faster than always starting from scratch.

Computational Resources. The models are mainly trained using NVIDIA V100 GPUs with 32 GB memory and Intel Xeon Gold 6248 CPUs, with some models trained using NVIDIA A100 GPUs with 80 GB memory.

References

Barrett, T. D., Malyshev, A., and Lvovsky, A. Autoregressive neural-network wavefunctions for ab initio quantum chemistry. Nature Machine Intelligence , 4(4):351-358, 2022.

Capriotti, L., Fubini, A., Roscilde, T., and Tognetti, V. Ising transition in the two-dimensional quantum j 1- j 2 heisenberg model. Physical review letters , 92(15):157202, 2004.

Carleo, G. and Troyer, M. Solving the quantum many-body problem with artificial neural networks. Science , 355(6325):602-606, 2017. doi: 10.1126/science.aag2302. URL https://www.science. org/doi/abs/10.1126/science.aag2302 .

Chen, Z., Luo, D., Hu, K., and Clark, B. K. Simulating 2+ 1d lattice quantum electrodynamics at finite density with neural flow wavefunctions. arXiv preprint arXiv:2212.06835 , 2022.

Choo, K., Neupert, T., and Carleo, G. Two-dimensional frustrated J 1 -J 2 model studied with neural network quantum states. Phys. Rev. B , 100:125124, Sep 2019. doi: 10.1103/PhysRevB.100.125124. URL https://link.aps.org/doi/10.1103/PhysRevB.100.125124 .

Deng, D.-L., Li, X., and Das Sarma, S. Quantum entanglement in neural network states. Physical Review X , 7(2), May 2017. ISSN 2160-3308. doi: 10.1103/physrevx.7.021021. URL http: //dx.doi.org/10.1103/PhysRevX.7.021021 .

Ferris, A. J. and Vidal, G. Perfect sampling with unitary tensor networks. Phys. Rev. B , 85:165146, Apr 2012. doi: 10.1103/PhysRevB.85.165146. URL https://link.aps.org/doi/10.1103/ PhysRevB.85.165146 .

Gao, X. and Duan, L.-M. Efficient representation of quantum many-body states with deep neural networks. Nature Communications , 8(1):662, Sep 2017. ISSN 2041-1723. doi: 10.1038/ s41467-017-00705-2. URL https://www.nature.com/articles/s41467-017-00705-2 .

Girvin, S. M. and Yang, K. Modern condensed matter physics . Cambridge University Press, 2019.

Greensmith, E., Bartlett, P. L., and Baxter, J. Variance reduction techniques for gradient estimates in reinforcement learning. J. Mach. Learn. Res. , 5:1471-1530, December 2004. ISSN 1532-4435.

Gutiérrez, I. L. and Mendl, C. B. Real time evolution with neural-network quantum states, 2020.

Hartmann, M. J. and Carleo, G. Neural-network approach to dissipative quantum many-body dynamics. Phys. Rev. Lett. , 122:250502, Jun 2019. doi: 10.1103/PhysRevLett.122.250502. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.250502 .

Hermann, J., Schätzle, Z., and Noé, F. Deep neural network solution of the electronic schrödinger equation, 2019.

Hibat-Allah, M., Ganahl, M., Hayward, L. E., Melko, R. G., and Carrasquilla, J. Recurrent neural network wave functions. Phys. Rev. Research , 2:023358, Jun 2020. doi: 10.1103/ PhysRevResearch.2.023358. URL https://journals.aps.org/prresearch/abstract/10. 1103/PhysRevResearch.2.023358 .

Huang, Y. and Moore, J. E. Neural network representation of tensor network and chiral states. Phys. Rev. Lett. , 127:170601, Oct 2021. doi: 10.1103/PhysRevLett.127.170601. URL https: //link.aps.org/doi/10.1103/PhysRevLett.127.170601 .

Lami, G., Carleo, G., and Collura, M. Matrix product states with backflow correlations. Physical Review B , 106(8), aug 2022. doi: 10.1103/physrevb.106.l081111. URL https://doi.org/10. 1103%2Fphysrevb.106.l081111 .

Lante, V. and Parola, A. The ising phase in the j1-j2 heisenberg model. Physical Review , 2006.

Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architectures. Physical Review Letters , 122(6), Feb 2019a. ISSN 1079-7114. doi: 10.1103/physrevlett.122. 065301. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122. 065301 .

Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architectures. Physical review letters , 122(6):065301, 2019b.

Lu, S., Gao, X., and Duan, L.-M. Efficient representation of topologically ordered states with restricted boltzmann machines. Phys. Rev. B , 99:155136, Apr 2019. doi: 10.1103/PhysRevB.99.155136. URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.99.155136 .

Luo, D. and Clark, B. K. Backflow transformations via neural networks for quantum manybody wave functions. Physical Review Letters , 122(22), Jun 2019. ISSN 1079-7114. doi: 10. 1103/physrevlett.122.226401. URL https://journals.aps.org/prl/abstract/10.1103/ PhysRevLett.122.226401 .

Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via a probabilistic formulation, 2020.

Luo, D., Carleo, G., Clark, B. K., and Stokes, J. Gauge equivariant neural networks for quantum lattice gauge theories. Physical review letters , 127(27):276402, 2021a.

Luo, D., Chen, Z., Hu, K., Zhao, Z., Hur, V . M., and Clark, B. K. Gauge invariant autoregressive neural networks for quantum lattice models, 2021b. URL https://arxiv.org/abs/2101.07243 .

Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via a probabilistic formulation. Phys. Rev. Lett. , 128:090501, Feb 2022a. doi: 10. 1103/PhysRevLett.128.090501. URL https://link.aps.org/doi/10.1103/PhysRevLett. 128.090501 .

Luo, D., Yuan, S., Stokes, J., and Clark, B. K. Gauge equivariant neural networks for 2+ 1d u (1) gauge theory simulations in hamiltonian formulation. arXiv preprint arXiv:2211.03198 , 2022b.

Marshall, W. Antiferromagnetism. Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences , 232(1188):48-68, 1955. ISSN 00804630. URL http: //www.jstor.org/stable/99682 .

Nagy, A. and Savona, V. Variational quantum monte carlo method with a neural-network ansatz for open quantum systems. Phys. Rev. Lett. , 122:250501, Jun 2019. doi: 10.1103/PhysRevLett.122. 250501. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122. 250501 .

Nomura, Y. and Imada, M. Dirac-type nodal spin liquid revealed by refined quantum many-body solver using neural-network wave function, correlation ratio, and level spectroscopy. Physical Review X , 11(3), aug 2021. doi: 10.1103/physrevx.11.031034. URL https://doi.org/10. 1103%2Fphysrevx.11.031034 .

Oseledets, I. V . Tensor-train decomposition. SIAM Journal on Scientific Computing , 33(5):2295-2317, 2011. doi: 10.1137/090752286. URL https://doi.org/10.1137/090752286 .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems , pp. 8026-8037, 2019.

Pfau, D., Spencer, J. S., Matthews, A. G. D. G., and Foulkes, W. M. C. Ab initio solution of the manyelectron schrödinger equation with deep neural networks. Phys. Rev. Research , 2:033429, Sep 2020. doi: 10.1103/PhysRevResearch.2.033429. URL https://journals.aps.org/prresearch/ abstract/10.1103/PhysRevResearch.2.033429 .

Preskill, J. Quantum computing 40 years later. arXiv preprint arXiv:2106.10522 , 2021.

Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jones, M., Klein, A., Milechin, L., Mullen, J., Prout, A., Rosa, A., Yee, C., and Michaleas, P. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC) , pp. 1-6. IEEE, 2018.

Schmitt, M. and Heyl, M. Quantum many-body dynamics in two dimensions with artificial neural networks. Physical Review Letters , 125(10), Sep 2020. ISSN 1079-7114. doi: 10.1103/physrevlett.125. 100503. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.125. 100503 .

Sharir, O., Levine, Y., Wies, N., Carleo, G., and Shashua, A. Deep autoregressive models for the efficient variational simulation of many-body quantum systems. Physical Review Letters , 124(2), jan 2020. doi: 10.1103/physrevlett.124.020503. URL https://doi.org/10.1103% 2Fphysrevlett.124.020503 .

Sharir, O., Shashua, A., and Carleo, G. Neural tensor contractions and the expressive power of deep neural quantum states, 2021.

Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems , 29, 2016.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. Advances in neural information processing systems , 30: 5998-6008, 2017.

Verstraete, F. and Cirac, J. I. Renormalization algorithms for quantum-many body systems in two and higher dimensions, 2004. URL https://arxiv.org/abs/cond-mat/0407066 .

Vicentini, F., Biella, A., Regnault, N., and Ciuti, C. Variational neural-network ansatz for steady states in open quantum systems. Physical Review Letters , 122(25), Jun 2019. ISSN 1079-7114. doi: 10.1103/physrevlett.122.250503. URL https://journals.aps.org/prl/abstract/10. 1103/PhysRevLett.122.250503 .

Vicentini, F., Hofmann, D., Szabó, A., Wu, D., Roth, C., Giuliani, C., Pescia, G., Nys, J., VargasCalderón, V., Astrakhantsev, N., and Carleo, G. NetKet 3: Machine learning toolbox for many-body quantum systems. SciPost Physics Codebases , aug 2022. doi: 10.21468/scipostphyscodeb.7. URL https://doi.org/10.21468%2Fscipostphyscodeb.7 .

Vidal, G. Efficient classical simulation of slightly entangled quantum computations. Physical Review Letters , 91(14), oct 2003. doi: 10.1103/physrevlett.91.147902. URL https://doi.org/10. 1103%2Fphysrevlett.91.147902 .

Vidal, G. Efficient simulation of one-dimensional quantum many-body systems. Physical Review Letters , 93(4), jul 2004. doi: 10.1103/physrevlett.93.040502. URL https://doi.org/10.1103% 2Fphysrevlett.93.040502 .

Vidal, G. Entanglement renormalization. Phys. Rev. Lett. , 99:220405, Nov 2007. doi: 10.1103/PhysRevLett.99.220405. URL https://link.aps.org/doi/10.1103/PhysRevLett. 99.220405 .

Vieijra, T., Casert, C., Nys, J., De Neve, W., Haegeman, J., Ryckebusch, J., and Verstraete, F. Restricted boltzmann machines for quantum states with non-abelian or anyonic symmetries. Physical Review Letters , 124(9), Mar 2020. ISSN 1079-7114. doi: 10.1103/physrevlett.124.097201. URL https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.124.097201 .

Westerhout, T., Astrakhantsev, N., Tikhonov, K. S., Katsnelson, M. I., and Bagrov, A. A. Generalization properties of neural network approximations to frustrated magnet ground states. Nature Communications , 11(1):1593, Mar 2020. ISSN 2041-1723. doi: 10.1038/s41467-020-15402-w. URL https://doi.org/10.1038/s41467-020-15402-w .

White, S. R. Density matrix formulation for quantum renormalization groups. Physical review letters , 69(19):2863, 1992.

Wu, D., Rossi, R., Vicentini, F., and Carleo, G. From tensor network quantum states to tensorial recurrent neural networks. arXiv preprint arXiv:2206.12363 , 2022.

Yoshioka, N. and Hamazaki, R. Constructing neural stationary states for open quantum manybody systems. Phys. Rev. B , 99:214306, Jun 2019. doi: 10.1103/PhysRevB.99.214306. URL https://journals.aps.org/prb/abstract/10.1103/PhysRevB.99.214306 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper approaches the problem of designing wave function ansatz for the ground state estimation of the Heisenberg model. This problem introduces several challenges including the rich parameterization of the wave function, sampling of the states from the parameterized wave function, and incorporating symmetries into the model. There are two main approaches to this problem: autoregressive neural networks and tensor networks.\nThe former allows for fast sampling and incorporating symmetries. The latter incorporates problem-specific inductive bias.\n\nThe authors propose an autoregressive model which incorporates tensor networks thus, hopefully, inheriting the best from both approaches. Shortly, the proposed model at every step of auto-regression yields tensors instead of scalars. This allows for a richer class of wave function representations while enjoying the properties of auto-regressive models such as exact sampling.\n\nThe authors study the proposed model both theoretically and empirically. The theoretical contribution of the paper is the proof of exact sampling and invariance properties. For the empirical study, the authors demonstrate the performance of the proposed ansatz for quantum state learning (fidelity maximization) and for the Heisenberg model (variational Monte Carlo).\n\nSTRENGTHS:\nI would say that the paper corresponds to all NeurIPS standards:\n- The paper approaches an important problem with clear applications. \n- The proposed method is novel and sound.\n- The experimental study is extensive.\n- It is well-written.\n\nWEAKNESSES:\nMy only concern regarding the empirical study is the comparison in the number of parameters. Indeed, the main goal of the paper is to propose a richer family of models, where we would expect better approximation properties for the same number of parameters as in concurrent approaches. I believe that this can be better illustrated with experiments by reporting the number of parameters along with the energy values. Right now, the authors fix the shapes for the tensor network and then add a neural network \"on top\" of it. Therefore, it is not very clear how to compare the models. Maybe adding a plot (e.g., number of parameters vs energy) would improve the readability of the experimental section.\n\nOther comments:\n- line 127. Typo (articles).\n- line 144. Typo (the last index of $\\alpha$).\n- line 153. Typo (many MPS represent)\n- line 160. Typo (up to qubit $j$). Also, the notation with the summation over $\\alpha$ is not very clear here.\n- line 271. Typo.\n\nQUESTIONS:\nI have no questions for the authors.\n\nLIMITATIONS:\nThe limitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes generic Autoregressive Neural TensorNet (ANTN), for quantum many-body simulation. In order to achieve high expressivity, sign structure preserving, physics inductive bias, accurate sampling, and symmetry preservation, ANTN combines tensor networks and autoregressive neural networks. The author demonstrates the improved performance of ANTN on quantum state learning and learning ground state energy.\n\nSTRENGTHS:\n1. The proposed architecture absorbs the advantages of two state-of-the-art methods i.e., tensor networks and autoregressive neural networks. The idea is straightforward but novel and interesting.\n2. This paper well illustrates its differences and advantages over tensor network based and neural network quantum state based methods for quantum many body simulation.\n3. It provides a comprehensive theoretical analysis and the experimental results show that the proposed method can still maintain good expressive ability with fewer bond dimensions.\n\nWEAKNESSES:\n1. The author solely compares and reports the bond dimensions taken up by the ANTN and MPS/DMRG approaches in Fig 1, Tab 1 and 2. Given that the first half of the ANTN is a sophisticated autoregressive neural network, reviewer concerns about that the author may evade a fact that whether the order of magnitude of the parameters of the autoregressive neural network could surpass the bond dimension of the following tensor network. As a consequence, the existence of the NN parameters will cancel out the efficiency brought on by the reduction in bond dimension.\n2. Although the combination of autoregressive neural network and tensor network to build a new neural network quantum state method that absorbs the advantages of both is a delightful idea, reviewers are worried that such method will also inherit the disadvantages of both. For example, tensor networks are likely to be very tricky to characterize a highly entangled quantum state [1]. At the same time, traditional tensor networks may also perform poorly in learning the ground state energy of a long-range Hamiltonian [2,3]. These problems are also an important reason why some researches abandon the tensor network and embrace the neural network-based quantum states. The reviewer is pleased if the authors could clarify these points, and it would be preferable if they could provide some explanations and comparisons in the experimental or theoretical part.\n\n[1] Deng D L, Li X, Sarma S D. Quantum entanglement in neural network states[J]. Physical Review X, 2017, 7(2): 021021.\n\n[2] Glasser I, Pancotti N, August M, et al. Neural-network quantum states, string-bond states, and chiral topological states[J]. Physical Review X, 2018, 8(1): 011006.\n\n[3] Xiao T, Huang J, Li H, et al. Intelligent certification for quantum simulators via machine learning[J]. npj Quantum Information, 2022, 8(1): 138.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nThe author does describe the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors introduce ANTN (Autoregressive Neural Tensor Net) as a novel architecture that combines features from both ARNN and Tensor Networks for simulating challenging quantum many-body systems. The main objective of this research is to utilize an ARNN-type network to generate a conditional wavefunction tensor rather than a direct conditional wavefunction, as commonly done in standard ARNNs. This latter can be interpreted as a probability density. Figure 1 provides a clear visualization of the idea.\n\nBy integrating ARNN and TN within this framework, the authors aim to leverage the advantages of both methods, thereby achieving enhanced expressivity and a stronger adherence to physics inductive bias. To demonstrate the superiority of their approach, the authors present several theorems highlighting its advantages. While detailed proofs are available in the appendix, the main text often provides concise explanations, which significantly support the flow of the manuscript and complement the discussed results.\n\nThe method's performance is initially evaluated by benchmarking its ability to learn 2-qubit Bell random states and real-valued shallow random circuits. Finally, the authors compare their ANTN approach to state-of-the-art methods in solving the ground state of a 2D $J_1-J_2$ Heisenberg model with open boundary conditions. This particular problem is known to be challenging due to its phase diagram exhibiting at least three distinct phases as $J_1$ and $J_2$ are varied. The results presented in Table 1 and Table 2 demonstrate that the ANTN method consistently outperforms existing methods across different system sizes and parameter choices for $J_2$.\n\nSTRENGTHS:\n- The paper is scientifically sound and well-written, with a comprehensive and meticulous literature review that covers all relevant works in the field.\n- The paper's structure is highly effective, resulting in a smooth and enjoyable reading experience.\n- Section 3 stands out as it provides a thorough overview of the essential components required to comprehend the more theoretical discussion starting from Section 4. This inclusion enhances the accessibility of the paper, making it approachable even for readers who are not thoroughly familiar with the field, such as myself.\n- The concept of combining the inherent flexibility of ARNN with the strengths of TN within a single framework is undeniably compelling.\n\nWEAKNESSES:\n- The manuscript frequently mentions the concept of \"expressivity,\" but it is never explicitly defined or explained. It would greatly benefit the readers to dedicate a paragraph to clarify its meaning and highlight its significance, thereby implicitly assessing the advantages of the proposed approach. Specifically, Theorem 5.2 is not entirely clear to me. It would be helpful to have a clear definition of expressivity to shed light on this point.\n- The paper repeatedly refers to the \"sign structure\" problem, but this issue is never explicitly defined or explained. While referencing relevant literature is helpful, the paper should strive to be self-contained when discussing this critical aspect. Given its importance in evaluating the efficacy of the proposed method in the experimental section, a paragraph discussing this concept would be valuable for readers less familiar with it.\n- Although ANTN combines properties from both ARNN and TN, being a parametrized neural network, it is expected that ANTN also requires training. However, the main text does not discuss this in great detail. A comprehensive discussion on training ANTN in the main text is essential to provide a complete understanding of the methodology.\n- The manuscript lacks experiments that benchmark the runtime and memory complexity. It would be beneficial to compare ANTN with state-of-the-art TN methods in terms of these metrics. Such comparisons would enhance the overall evaluation of ANTN and its performance.\n\nQUESTIONS:\n- How is the ANTN model trained? While intuitions are provided in Sections 3.2 and 3.3 (with additional details in the appendix), a more specific discussion would enhance understanding of the proposed framework. Additionally, it would be beneficial to explore the limitations of training ANTN. Is the training process significantly more challenging compared to plain ARNN? Can you provide an estimate of the computational time required to train a network for a specific downstream task?\n- The authors frequently mention that ARNNs lack the physics prior of the system under study. However, it raises the question of the extent to which this holds true. As correctly stated by the authors, inductive physics biases, such as symmetries, can actually be incorporated into pure ARNN networks to incorporate the desired physics prior. Could the authors provide some comments on this? \n- I wonder whether it would be interesting to replicate the analysis in Table 1 and Table 2 by fixing $J_2$ and modifying $J_1$ instead. Would substantial differences in the results be expected? This question arises because, as I understand it, $J_1$ modulates the interaction between nearest neighbors, potentially leading to collective behavior that may be more challenging to capture.\n- How do runtime and memory requirements scale in the experiments presented in Section 6.2? The insightful complexity analysis discussed in Section 5 suggests that it would be valuable to empirically visualize these factors through numerical experiments.\n- Is there a specific reason why DGRM results are not shown in Table 1, while RBM results are missing in Table 2?\n- In Figure 3, it appears that the DGRM method outperforms ANTN for medium-sized systems and increasing values of $J_2$. Since the bond dimension is fixed for both methods, do you have any intuition regarding how this plot would change as the bond dimension varies? Intuitively, one might expect that having a larger bond state for DGRM would improve its performance, particularly for larger system sizes. It would be intriguing to perform a detailed scaling analysis as a function of the bond dimension, acknowledging the potential increase in complexity and the enhanced capability of TN methods to capture more features, potentially leading to improved results.\n- In the broader impact section, the authors suggest that the ANTN approach may be applicable to other standard machine learning tasks. However, in such tasks, like image generation, for instance, there might not be strong physics prior to leverage using the TN representation. It would be interesting to hear the authors' perspectives on the benefits of applying their framework to standard machine learning benchmarks and how it could be advantageous in those scenarios.\n- Can this idea of combining TN with a generative type of neural networks be extended to other generative models?\n\nLIMITATIONS:\nIn addition to the concerns mentioned in the weaknesses and questions sections, which implicitly highlight some limitations of the current manuscript, I believe that the primary limitation of this work may lie in its scalability and the training difficulties of ANTN compared to MPS. Although I am not an MPS expert, I am aware that sampling (and training) for an ARNN can be slow and computationally expensive, particularly when dealing with larger systems. While runtime may also pose a challenge for MPS, as increasing the bond dimension can become a computational bottleneck, I am uncertain whether this would be a minor issue compared to training, and sampling from, an ANTN when considering extremely large systems.\n\nA few minor comments: \n- In the introduction, it would be good to already have the relevant references being cited when a field/topic is firstly mentioned and introduced. \n- The Acronyms MERA and PEPS are never introduced in their extensive form. Would be good to do so when they are mentioned for the first time in line 72 of page 2. \n- Line 271 page 7:  previous -> previously\n- Line 287 page 7:  comes -> come\n- Line 690 page 19: staring -> starting", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Autoregressive Neural TensorNet (ANTN): a novel blend between Matrix Product States (MPS) and AutoRegressive Neural Network. Theoretically, this paper shows that ANTN parameterizes normalized wavefunctions, allows for exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. Experimentally, this paper shows that ANTN is more efficient than MPS and better biased than ARNN in quantum state learning, and show that ANTN outperforms MPS and ARNN in approximating the ground state of the 2D J1-J2 Heisenberg model with systems sizes up to 12x12.\n\nSTRENGTHS:\nThe proposed architecture is novel, and importantly simple and intuitive therefore conveying a potential of enjoying “the best of both worlds” – an MPS above a very expressive AR neural network that creates a strong input representation. The core observation that one can construct such an architecture while retaining the ability to perform exact efficient sampling, is strong. \nThe experimental results are non-trivial. The clean experiments in 6.1 tell an interesting story of efficiency with respect to MPS and bias with respect to ARNN. In section 6.2, it is an impressive achievement to consistently surpass ARNN on the one hand and MPS on the other on the challenging J1J2 task. Moreover, the positive trend with system size is promising. Overall, the experimental results in this paper bring an appetite to try this new architecture on larger systems and on more problems.\n\nWEAKNESSES:\nOverall, I find this paper to be not well structured. My main disappointment with the writing is the lack of details on the experimental part of the paper. It is by far more convincing than the theoretical part, but unfortunately does not receive enough space and details.\n\nFor example, it is unclear to me what ANTN architecture was used for the experiments. Section 4 mentions the use of both PixelCNN and Transformer as the ARNNs underlying the ANTN. However, both tables 1 and 2 do not mention which variant was used, do not show results for both variants, and mention only the PixelCNN baseline. A long look at the experimental details in the paper body and in the appendix did not really help. It makes it very hard to assess the strength of the results when it is not clear which ARNN was used. Moreover, I am missing external performance references in order to assess the strength of the results. The field of many body quantum simulations has matured to the point where I expect new architectures to compare to external SOTA results; otherwise, it is not easy to judge results that are given relatively to baselines implemented by the authors, especially in the haziness / opacity of the current detailing of the experimental setup. Comparison to existing results on J1J2 will help convince me that the reported gains are meaningful. Besides that, much more details on the experimental setup on which the results were attained are required for this potentially important experimental outcome to be verifiable.           \n\nTheoretically, Theorem 1 is important; Section 5.2 is disappointing and does not motivate using ANTN from an expressivity standpoint; and section 5.3 is not tied to the specific experiments conducted later on is used in the remainder in the paper (so though possibly interesting it does really contribute to the practical case advocating the use of ANTN). \n\nElaborating, Theorem 5.2 does not provide an expressivity result that allows for reasoning regarding the benefits of ANTN. Specifically, it isn’t clear **by how much** ANTN is stronger than ARNN or MPS. The simple theoretical arguments provided do not make it clear, for example, whether ANTN is stronger than adding one more layer to the ARNN. Therefore, from an expressivity standpoint, the authors did not make a convincing case for the benefits of ANTN. Theorem 5.3 simply mentions the strength of ARNN as shown in other works, but does not contribute to the case of why ANTN is better than the existing ARNN from an expressivity standpoint. An experiment of ARNN with one more layer may very well show that the gains can also be pretty easily achieved with an existing architecture, which would be disappointing. (as a remark it isn’t clear how many layers are in the experiments on ARNN, the appendix gives some details but it isn’t clear if Transformer of PixelCNN was used).\n\nQUESTIONS:\nMultiple references to the “Inductive bias of the physics structure of MPS” is pretty vague. When you discuss initializing the MPS with the optimized DMRG results of the same bond dimension that kind of makes sense, but then you could also initialize the ARNN with an optimized physics related representation, so I don’t see an advantage of MPS in this “optimzed initialization” regard. As for randomly initialized MPS vs ARNN, I don’t see that a-priori the former is better suited than the latter. In particular, all of the relevant physical symmetries of ATNN listed in section 5.3 seem to be inherited from ARNN and not from MPS. On the other hand, the experiments of section 6.1 mention “wavefunctions with local or quasi-local sign structures” and “shallow random circuit to mimic the short-range interaction and generate states with sign structures”. NeurIPS is ultimately a non-physics conference and such statements are hard to decipher for me as well as for broader readership -- what exactly is the task you ran? how exactly is it related to interesting physical systems? . Especially given the important role of this point in the paper’s main argument regarding the benefits of MPS, I find it to be insufficiently explained. \nCan you please elaborate and be more specific on what MPS inductive biases contribute to the physics suitability of ANTN? \n\nFurther issues:\nWhat is h_dim? I don't see it defined.\nLine 63: state-of-the-arts \nLine 153: missing  punctuation after “unique”\nLine 311: especially pass -> especially past\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}