ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation
Authors: Zhuo Chen
Laker Newhouse
Eddie Chen
Di Luo
Marin Soljacic
Venue: NeurIPS 2023 poster (2023)
Keywords: Autoregressive neural network
tensor network
quantum many-body physics
variational Monte Carlo
TL;DR: We develop the Autoregressive Neural TensorNet, combining autoregressive neural networks and tensor networks, to advance challenging quantum many-body physics simulations.

Abstract: Quantum many-body physics simulation has important impacts on understanding fundamental science and has applications to quantum materials design and quantum technology. However, due to the exponentially growing size of the Hilbert space with respect to the particle number, a direct simulation is intractable. While representing quantum states with tensor networks and neural networks are the two state-of-the-art methods for approximate simulations, each has its own limitations in terms of expressivity and inductive bias. To address these challenges, we develop a novel architecture, Autoregressive Neural TensorNet (ANTN), which bridges tensor networks and autoregressive neural networks. We show that Autoregressive Neural TensorNet parameterizes normalized wavefunctions, allows for exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. We demonstrate our approach on quantum state learning as well as finding the ground state of the challenging 2D $J_1$-$J_2$ Heisenberg model with different systems sizes and coupling parameters, outperforming both tensor networks and autoregressive neural networks. Our work opens up new opportunities for quantum many-body physics simulation, quantum technology design, and generative modeling in artificial intelligence.

ANTN: Bridging Autoregressive Neural Networks and Tensor Networks for Quantum Many-Body Simulation
Zhuo Chen 123
Laker Newhouse Eddie Chen Di Luo 12357 Marin Soljačié13
NSF Al Institute for Artificial Intelligence and Fundamental Interactions
Center for Theoretical Physics, Massachusetts Institue of Technology
3 Department Physics, Massachusetts Institute of Technology
*Department of Mathematics, Massachusetts Institute of Technology
'Department of Physics, Harvard University {chenzhuo, lakern, ezchen, diluo,
Cmiit. edu
Abstract
Quantum many-body physics simulation has important impacts on understanding fundamental science and has applications to quantum materials design and quan- tum technology. However, due to the exponentially growing size of the Hilbert space with respect the particle number, direct simulation is intractable. While representing quantum states with tensor networks and neural networks are the two state-of-the-art methods for approximate simulations, each has its limitations in terms of expressivity and inductive bias. To address these challenges, we develop novel architecture, Autoregressive Neural TensorNet (ANTN), which bridges tensor networks and autoregressive neural networks. We show that Autoregressive Neural TensorNet parameterizes normalized wavefunctions, allows for exact sam- pling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits variety of symmetries from autoregressive neural net- works. We demonstrate our approach on quantum state learning as well as finding the ground state of the challenging 2D J1-J2 Heisenberg model with different systems sizes and coupling parameters, outperforming both tensor networks and autoregressive neural networks. Our work opens up new opportunities for quan- tum many-body physics simulation, quantum technology design, and generative modeling in artificial intelligence.
Introduction
Quantum many-body physics is fundamental to our understanding of the universe. It appears in high energy physics where all the fundamental interactions in the Standard Model, such as quantum electrodynamic (QED) and quantum chromodynamics (QCD), described by quantum mechanics. In condensed matter physics, quantum many-body physics has led to a number of rich phenomena and exotic quantum matters, including superfluids, superconductivity. the quantum Hall effect, and topological ordered states (Girvin & Yang 2019). As an application, quantum many-body physics is erucial for new materials design. The electronic structure problem and chemical reactions in quantum chemistry are governed by quantum many-body physics. The recent development of quantum computers also deeply connected quantum many-body physics. multi-qubit quantum device is intrinsically a quantum many-body system, such that progress on quantum computer engineering is tied understanding of quantum many-body physics (Preskill 2021
"quull contribution corresponniig author: diluo@ mit.edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

All information of a closed quantum many. body system captured by the wavefunc- tion, whose properties are described by the famous Schrödinger equation. An impor- tant tool to study and understand quantum many-body physies is to classically simu- late the wavefunction. However, the wave- function a high dimensional function in Hilbert space, whose dimension grows expo- nentially with the number of particles. For example, for qubit systems where each qubit has two degrees of freedom), the wavefunc- tion of 300 qubits will have dimension 2300, which is larger than the number of atoms in the observable universe. Furthermore, the Schrödinger equation is a complex-valued high-dimensional equation, which is challenging to solve or simulate in general.
ANTN Brigging gap
ARNN Pro: High expeesaivity Con: No physicx price
TN
Pro. Physics inductive bis Con:: Limited expresiiity
Figure I: Diagrammatic representation of autoregres- sive neural network (ARNN), tensor network (TN) and our Autoregressive Neural TensorNet (ANTN).
number of algorithms have been developed to simulate quantum many-body physics, including quantum Monte Carlo, tensor networks, neural network quantum states, and quantum computation. In particular, computing the ground state of quantum many-body systems is of great interest. One important approach is the variational principle, which provides an upper bound for the ground state energy. To apply the variational principle successfully, one must design an ansatz that can represent and optimize the wavefunction efficiently. Tensor networks and neural network quantum states (Carleo & Troyer, 2017) are the two main state-of-the-arr methods that can be applied with the variational principle for quantum many-body simulation. However, tensor networks usually suffer from an expressivity issue in systems with more than one dimension, while neural network quantum states usually lack inductive bias from the underlying physies structure and have sign structure challenges in the representation.
In this paper, we develop novel architecture, Autoregressive Neural TensorNet (ANTN), to bridge neural network quantum states and tensor networks, achieving the best of both worlds. In particular, contributions are threefold:
Develop ANTN with two variants called "elementwise" and "blockwise," which each natu- rally generalize the two state-of of-theearts ansatzes, tensor networks (TN) and autoregressive neural networks (ARNN), provide proper inductive bias and high expressivity.
Prove that ANTN is normalized with exact sampling, has generalized expressivity of TN and ARNN, and inherits multiple symmetries from ARNN
Demonstrate our methods quantum state learning and variationally finding the ground state of the challenging 2D J1-J2 Heisenberg model, outperforming both TN and ARNN.
2 Related Work
Tensor Networks (TN) represent high-dimensional wavefunctions using low-rank tensor decompo- sition, notably matrix product state (MPS) (Vidal) 2003] (004). PEPS (Verstraete Cirac 2004). and MERA (Vidal, (007). They capture the entanglement structure of physical systems and, with algorithms like the density matrix renormalization group (DMRG) (White) 1992), are used for state simulations and real-time dynamics. However, their expressivity can be limited in systems with more than one dimension and systems with high entanglement. In machine learning, TN appears as tensor train (Oseledets) 2011) and CP decomposition methods.
Neural Network Quantum State (NNQS) leverages neural networks for high dimensional wave- function representation (Carleo & Troyer 2017). It's been demonstrated that many quantum states
can be approximated or represented by NNQS (Sharir et al. [202]] Gao Duan 2017 et al. 2019 Levine et al., 2019a Luo et al. 202 al Deng et al. [017]] & Moore [021] Vieijra et al.
2020), and has yielded state-of the- results in computing quantum system properties (Gutiérrez
Mendi| 2020 Schmmitt& & Heyl 2020 Vicentini et al.
2019 Yoshioka & Hamazaki 2019 Hartmann
Carleo 20099; Nagy & Savonal 2019: Luo al.
202 2022a). Key advancements in NNQS
include ARNN that improves sample efficiency and gradient estimation, and the development of

---

neural networks that adhere to the underlying symmetries of quantum systems Luo et al. 202 bb
Hibat-Allah al. 2020 Choo al. 2019 Luo Clark 2019 Hermann al. 2019 Piau et al. 2020
Luo et al. 2021 2022b 202 Chen et al. 2022). Despite its potential, NNQS faces challenger such as the lack physics prior and the sign structure problem. While there are attempts to integrate NNQS with TN, including matrix product state with neural network backflow (Lami et al. 2022) and
generalizing MPS to RNN (Wu et al. 2022), the former can not produce normalized wavefunctions, while the latter only builds tensor contractions that lacks the nonlinear activation functions of standard neural network wavefunctions.
3 Background
3.1 Quantum Preliminaries
In this work, we focus on qubit systems in quantum many-body physies. The wavefunction or quantum state & of the system is normalized function ab : 2n + with Σn, [=(x)] = 1, where
is the system size. The input to the wavefunction is n-bit string E {0, 1) Therefore, the wavefunction té can be viewed as a complex-valued vector of size 2", with Dirac notation (v) and (ı5) correspond Because the size of the wavefunction grows exponentially with the system size n, a direct computation quickly becomes intractable the system size increases. The goal of NNQS is to design a compact architecture that can approximate and optimize the wavefunction efficiently.
conjugate
vector and a column vector respectively, and (<x) (æ (z)
3.2 Quantum State Learning
Given a quantum state (o), we are often interested finding (V0) that is closest to (Ø) given the variational ansatz. In quantum mechanics, the closeness of two quantum states is measured by the quantum fidelity J' (olvb))
IE d² (=)))(s( where refers complex conjugation. Quantum fidelity satisfies < P < 1, with F orthogonal quantum states. Finding F Fmax for given ansatz allows to quantify how good the ansatz can be used to approximate that state. In practice, minimizing log F is usually better than maximizing itself. This can be achieved by enumerating all the basis in small systems and stochastically in large systems such as quantum state tomography (see Appendix A.I A.I.
corresponding to exact match, and F 0
3.3 Variational Monte Carlo
For given quantum system with n qubits, the Hamiltonian H. can be written as Hermitian matrix of size 2" x 2 The ground state energy Eg of the system is the smallest eigenvalue of 7L and the ground state the corresponding eigenvector. For large system sizes, finding the ground state directly is usually impossible. In this case, the variational principle in quantum mechanics provides an upper bound on the ground state energy Eg. For all (normalized) wavefunctions (v>), it evidental that
E, < (wb ()) Therefore, finding the (eb)) that minimizes Ee = (the |H (be) gives the lowest upper bound the ground state energy. For large system sizes, can stochastically evaluate and minimize
(wba) (llo)) = Σ [2)(2))) H. (o (xx)
Eer Hoz xe th (o('x)
E B via (2)
where Ha. refers to the matrix element H and we interpret (x) 2 as a probability distribution. The summation over æ' can be efficiently computed given since the Hamiltonian is usually sparse. The gradient Eg can also be calculated stochastically in similar fashion (see Appendix (A.)).
4 Method
4.1
Preliminaries
Autoregressive Neural Network Wavefunction. The autoregressive neural network (ARNN) (Fig. D
left) parameterizes the full probability distribution product of conditional probability distributions
3

---

as
p(x) = p x<s),
(2)
where æ: (œ1 ( Fn,) is a configuration of n qubits and Bei (21,..., ;;-1)) is any configura- tion before Tj. The normalization of the full probability can be guaranteed from the normalization of individual conditional probabilities. The ARNN also allows for exact sampling from the full distribution by sampling sequentially from the conditional probabilities (see Appendix [.J]. The autoregressive constructions for probabilities can be easily modified to represent quantum wavefune- tion by (1) replacing p( <i)) with a complex-valuce conditional wavefunction (x and (2)
using the following normalization condition for the conditional wavefunctions: Σn (
(Sharir et al. 2020: Ailat et al., 2020; Luo et al. 2021b). Similar to the case of probabilities, ARNN automatically y preserves the normalization of wavefunctions and allows for exact sampling (see Appendix B. Because of this, ARNN is often more efficient when training and computing
various quantum observables compared other generative neural networks.
Matrix Product State (MPS). MPS (also known as tensor train) is widely used TN architecture to study quantum many-body physics (Vidal 2003 2004). The MPS defines wavefunction using rank-3 tensors Ma; - with Qç-1 (or (4) the index of the left (or right) bond dimensions and z; the configuration of the ith qubit. Then, the full wavefunction is generated by first choosing particular
configuration a = (Iı,..., Fn) and then contracting the tensors selected by this configuration (Fig. right)
((x)= = Σ
M
Man-1aa
ΣII Mar-oa
(3)
where the left-and-right-noost bond dimensions are assumed to be D(oo) = D(ax+1) = so are not summed.
4.2 Autoregressive Neural TensorNet (ANTN)
Elementwise and Blockwise Construction for ANTN. Both ARNN and MPS are powerful ansatzes for parameterizing quantum many-body wavefunctions. Albeit very expressive, ARNN lacks the physics prior of the system of interest. In addition, since wavefunctions are complex-valued in general, learning the sign structure can be a nontrivial task (Westerhout et al. 2020). MPS, on the other hand, contains the necessary physics prior and can efficiently represent local quasi-local sign structures, but its expressivity severely limited. The internal bond dimension needs to grow exponentially to account for a linear increase in entanglement. It turns out that MPS representation is not unique in that many MPS actually represent the same wavefunction; and if we choose a particular constraint (without affecting the expressivity), MPS allows for efficient evaluation of conditional probability and exact sampling (Ferris& & Vidal 2012) (see Appendix B.2). Because both these ansatzes allow for exact sampling, it is natural combine them to produce more powerful ansatz. Therefore, we develop the Autoregressive Neural TensorNet (ANTN) (Fig. [ (middle)). In the last layer of the ANTN, instead of outputting the conditional wavefunction wavefunction tensor Ja.-1a ds (I, (i)) for each site. Defining the left partially contracted tensor up to qubit j as V (æ=²))= := Ea II
we output conditional
1 20.-100 10, (x, (1),, we can define the (unnormalized) marginal
probability distribution as
((xs)) = Σ
3i ( (x<i)"
ay
where denotes complex conjugation. Then, the (normalized) conditional probability can be obtained as ((>,|><<< ) = / Σx, (( We construct the overall wavefunction by defining both its amplitude and phase according to
(xx) = ((x))())) ko())
(5)
with ((x) =: IT= ((x)( and the phase ((x) =: Arg 1 (1
(te)) In other
words, we define the amplitude of the wavefunction through the conditional probability distributions and define the phase analogous to the standard MPS.

---

We develop two different constructions for ANTN that differ in the last layer on how to construct conditional wavefunction tensors.
The elementwise ANTN is given by
Go 1 (3 (2))
10: [NN 01-1,
(6)
where fNN a1 x <<) is the complex-valued output. The blockwise ANTN is given by
jan (xi)))))))) M + [NN((a))
where the valued output JNN (=+) is broadcasted over a. This results in a lower complexity that allows us larger maximum bond dimension.
Transformer and PixeICNN Based ANTN. Our construction above general and can applied to any standard ARNN. Depending on the application, we can use different ARNN architectures. In this work, we choose the transformer and PixeICNN depending on the specific tasks. The transformer (Vaswani et al.) 2017) used here is similar to decoder-only transformer implemented in (Luo et al., 2020), and The PixcICNN we use is the gated PixeICNN (Van den Oord et al. 2016) implemented in (Chen et al. (022).
MPS Initialization. Since our ANTN generalizes from TN, both the elementwise and the blockwise can take advantage of the optimized MPS from algorithms such as DMRG (Whitel 1992) as an initialization. In practice, we can initialize the TN component with the optimized DMRG results of the same bond dimension (similar to (Lami et al.) [022:] Wu et al., (022)). The MPS Initialization can also be thought of as a pretraining of the ANTN. This a nice feature since it provides good sign structure and inductive bias from the physics structure, which does not exist in the conventional ARNN.
Limitations. In this work, we only integrated MPS into the ANTN, where MPS may not be the best TN in various settings. Besides MPS, many other TNs also allow efficient evaluation of conditional probabilities and exact sampling, such as MERA (Vidal 2007) and PEPS (Verstraete & Cirac 2004 or cylindrical MPS for periodic boundary conditions. In fact, the recently developed TensorRNN (Wu et al., 2022) can also be viewed as type of TN and can be integrated into our construction for future work. In addition, while the ANTN construction general in terms of the base ARNN used, our current study only focuses on transformer and PixeICNN. Lastly, shown later in Sec. Sec.5.1T the current implementation of ANTN has an additional sampling overhead that is linear system size which can be avoided.
Theoretical Results
5.1 Exact Sampling and Complexity Analysis of ANTN
Theorem 5.1. Autoregressive Neural TensorNet wavefunction is automatically normalized and allows for exact sampling.
Proof. This is direct consequence that we defined the amplitude of the wavefunction through normalized conditional probability distributions (x (See Appendix B.1 for the detailed sampling procedure.)
Complexity Analysis. We first note that for MPS, the number of parameters and computational complexity for evaluating a bitstring æ scales as O(nx²), where n is the number of particles and X = D(a) is the (maximum) bond dimension of MPS. The sampling complexity scales as Onn² x(n²x²). The DMRG algorithm has a computational complexity of (nxx)) (White) 1992). The number of parameters and computational complexity of ARNN depends on the specific choice. Assuming the ARNN has NARNN parameters and single pass (evaluation) of the ARNN has computational complexity of CARNN. then the sampling complexity of the ARNN is (RCARRNN) in our current implementation. The ANTN based on such ARNN would have O + hadmm parameters for the elementwise construction, and O(HARNN + rex² + ruhai)) parameters for the blockwise construction. The computational complexity for evaluating and sampling bitstrings scales as (CARNN + 22x2haim) )) for elementwise construction and O "CARRNN - =nx?) )) for blockwise construction with 0 for evaluation and for sampling.
5

---

We note that the complexity analysis above is based on our current implementation, where the sampling procedure for all algorithms has an (n) overhead compared evaluation. is possible to remove it by storing the partial results during the sampling procedure.
Then, each gradient update of the variational Monte Carlo algorithm requires sampling once and evaluating Non times where Noon is the number of connected (non-zero) æ' in Hz, given æ, which usually scales linearly with the system size. Then, the overall complexity is ONN, Ceva Cssmp) with N₂ the batch size, Ceva the evaluation cost and Csamp the sampling cost. Usually, the first part dominates the cost.
The difference in computational complexities and number of parameters between elementwise ANTN and blockwise ANTN implies that blockwise ANTN is usually more economical than elementwise ANTN for the same hidden dimsnieon haim and bond dimension X. Therefore, for small bond dimensions,
the elementwise ANTN for more flexible parameterization with higher cost. In contrast, for large bond dimensions, we use the blockwise ANTN, which saves computational complexity and improves the initial performance (with DMRG initialization) of the blockwise ANTN at the cost of less flexible modifications from the ARNN. We also note that compared to the state-of- the-art MPS simulation, even for our blockwise ANTN, small bond dimension usually suffices. For example, in the later experimental section, we use bond dimension 70 for blockwise ANTN while the best MPS results use bond dimension 1024, which has much more parameters than our ANTN. In general, our ANTN has much fewer parameters compared to MPS due to the O(x²) parameters scaling which dominates at large bond dimension.
5.2 Expressivity Results of ANTN
Theorem 5.2. Autoregressive Neural TensorNet can have volume law entanglement, which is strictly beyond the expressivity of matrix product states.
Proof. We proved in Thm. 5.3 that ANTN can be reduced to an ARNN, which has been shown to have volume law entanglement that cannot be efficiently represented by TN (Sharir et al.) 1 Levine et al. 2019b). Hence, ANTN can represent states that cannot in general be represented by MPS efficiently; thus has strictly greater expressivity.
Theorem 5.3. Autoregressive Neural TensorNet has generalized expressivity over tensor networks and autoregressive neural networks.
Proof. Thm B.2pf Appendix B.4 shows that both TN and ARNN are special cases as ANTN and Thm Bf Appendix [.4 shows that ANTN can be written as either TN or an ARNN with an exponential (in system size) number of parameters. Thus, ANTN generalizes the expressivity over both TN and ARNN.
5.3 Symmetry Results of ANTN
Symmetry plays an important role in quantum many-body physics and quantum chemistry. Many of the symmetries be enforced in ARNN via the two classes of the symmetries mask symmetry and function symmetry.
Definition 5.1 (Mask Symmetry). conditional wavefunction (x///<>> has a mask symmetry if (z; ( = O for some given Øei.
Definition 5.2 (Function Symmetry). A conditional wavefunction tensor (x,())))) has a func- tion symmetry over a function F if (Iz,((=))) *(F(z; xx) F(x<i)) where F(x == (F(x1),F(22:) ".,,,::::").....F(((;(;.... &(,;
Here, we list several symmetries that ANTN inherits from ARNN and show the proofs in Ap- pendix B.4
Theorem 5.4. Autoregressive Neural TensorNet inherits mask symmetry and function symmetry from autoregressive neural networks.
Corollary 5.4.1 (Global U(I) Symmetry). Autoregressive Neural TensorNet can realize global U(1) symmetry, which conserves particle number.
Corollary 5.4.2 (Z.2 Spin Flip Symmetry). Autoregressive Neural TensorNet can realize Z, spin flip symmetry such that the wavefunction is invariant under conjugation of the input.

---

Random Bell State
(b)
Shallow Random Circuit
0.8
0.8
Edelty
0.6
0.6
0.4
0.4
0.2
0.2
sign rule
0.0
0.0
MPS (4)
MPS(AS)
MPS (64)
MPS (258)
ARNN
ANTN (4)
MPS (4)
MPSite)
MPS (64)
MPS (296)
ARNN
ANTI (4)
Figure 2: Fidelity i quantum state learning with 16 qubits for TN (MPS), ARNN (transformer) and ANTN (elementwise construction with transformer+MPS). (a) Learning random Bell states. (b) Learning real-valued depth-4 random circuit with and without sign rule. The error bar denotes the standard deviation (not the standard error of the mean) of the fidelities over the random states sampled from the corresponding distribution. The mean and standard deviation calculated from 10 random states. The numbers inside the parentheses denote the bond dimension.
Corollary 5.4.3 (Discrete Abelian and Aon-Abelian Symmetries). Autoregressive Neural TensorNet can realize discrete Abelian and Non-Abelian symmetries.
Experiments
6.1 Quantum State Learning
As stated previously, ANTN generalizes both ARNN and TNN to take advantage both the expressivity and the inductive bias. Here, test the ANTN learning physically important quantum states to demonstrate this ability. In this task, we use the transformer neural network for ARNN and ANTN due to the ID structure of the system.
Experiments on expressivity. We first test the expressivity of the ANTN by learning a class of well-known high-entangled states random permuted Bell states. A 2-qubit Bell state is defined as (100) + / which has 1 bit of entanglement. For system size of n, randomly pair qubits in the first half of the system with the qubits in the second half the system to be Bell states. This process creates quantum states with n/2 bit of entanglement between the first and second half of the system. It can be shown that bond dimension of 2n/ is required for MPS to fully represent such a system. In Fig. (a), we plot the quantum fidelity of learning 16-qubit random Bell states. As the figure shows, MPS cannot represent the states without reaching the required bond dimension of 256 28. The ARNN and ANTN, on the other hand, can represent such states without limitations from the bond dimension.
Experiments on inductive bias. We then test the physics inductive bias of the ANTN. One of the physics inductive biases of MPS is that it can represent wavefunctions with local or quasi-local sign structures that can be hard for neural networks to learn. These wavefunctions can have a fluctuating sign depending the configuration the short-range interaction and generate states with sign structures. We test the algorithms on these states both with and without the "sign rule". which means that explicitly provide the sign structure to the algorithm. As shown in Fig. (b), the fidelity of MPS only depends weakly on the sign rule, whereas the fidelity of ARNN can change drastically. Our ANTN inherits the property of MPS and thus not affected by the sign structure. Furthermore, being more expressive, ANTN with a bond dimension of 4 already performs better than MPS with bond dimension of 16.
Here,
(real-valued) shallow random circuits to mimic
6.2 Variational Monte Carlo
We further test our algorithm on finding the ground state of the challenging 2D Ji-J2 Heisenberg model with open boundary condition. The model has rich phase diagram with at least three different phases across different J2/ J1 values (Capriotti al. 2004) [Lante & Parola 2006). In addition, the

---

Energy per site
18x 18x8
Elementwis DMRG
Algonihmss
J2
J2 0.8
RBM (NS)
1,9609911
1.6128(20)
-. 75008
RBM (S)
1.1194((1)
(0)
00.7277 29)
PixeICNN (NS)
-2.21218(16)
1.. 1.7552((2
-.. (19882((6))
PixcICNN (S)
-2.23171(9)
-1888001 19]
1.83672(26)
1/ 022
Elementwise (8 NS)
22223690(4)
-1 1.93018(8)
-2. 00006 16)
J/ m.5
Elementwise S)
-2.23658(4)
-1.93102(7)
(110))2(11)
1/2 0.8
Blockwise (70 NS)
-2.23484(7)
-1.93000(7)
-1. (9348(13)
26 36
100
144
Blockwise (70 S)
-2.23517(6)
(.92880(9)
-1 1913)
Number qubiis
Table 1: Energy per site + for 8 x 8 system with various algorithms where elementwise and blockwise are two constructions of ANTN (with PixeICNN MPS). The bond dimensions for ANTN are labeled inside the parentheses. For each algorithm, both with the sign rule (S) and without the sign rule (NS) The best energy is highlighted in boldface and the second in italic.
Figure 3: Energy per site difference between elementwise ANTN with bond dimension and MPS with bond dimension 1024 optimized us- ing DMRG algorithm for various J2 and different system sizes from 4 x to 12 x 12.
test it
Energy per site L
10 10
Algorithms
J/ 0.2
J2 0.3
J2 = 0.4
J2 0.5
J2 0.6
J2 0.7
J2 0.8
MPS
-1.997537
1.893753
-1.797675
1..3/1326
1.716253
-.768225
-1860971
MPS (70)
2.191048
2.069009
-1.556480
1.866159
1.81629
L.854296
2.007845
MPS (1024)
2.255633
-2.138591
-2.031681
1. 1.998770
1. 1866561
-1 1.894371
2.062730
PixcICNN
2 (.22466((24)
2.12873(11)
(2)
(.770098(29)
1. (27)
-1811000(13)
1.98331(17)
Elementwise (8)
2.26034(6)
2.14450(()
1.94001 (2703727
1. 88888684110
(888643(7)
2. (49)
Blockwise (70)
8.25755(8)
2.14152(8) (8)
9.03919(
(.93842(42)
(5270(
1.8783((33)
2.05088(14)
Energy per
12
Algorithms
J, 0.2
32 0.3
Jg = 0.4
0.5
Jg 0.6
0.7
Jn 0.8
MPS (8)
-1998207
-1887531
-1.800844
1.733906
1.720619
L.186652
-1893916
MPS (70)
-2.185071
-2.394443
-1.944832
-1.851954
1.812450
-1 11853650
-2.030131
MPS (1024)
-2.264084
-2.141043
-2.027736
-1931244
-1858846
1.913483
-2. 09.09001
PixcICNN
2.224311
-.12611(223)
2.01768(21
(.74282(30)
1.776677 (16)
-1.85239
-2 2.00322(55)
Elementwise (8)
-2277446(27)
2.15537(6)
(.04437(7)
1 (.94686(6)
1. 1.85544(11 15
1 1.91391
2.009457(10)
Blockwise (70)
-.226152(50)
2. 1513555(
-2.04225 (8)
1 (.94288(3))
(14433)11888 (11)66(15)
(190551((12)
(34031313
Table 2: Energy per site J. for 10 x 10 and 12 x 12 system with various algorithms where elementwise and blockwise are two constructions of ANTN (with PixeICNN MPS). The bond dimensions for MPS and ANTN are labeled in parentheses. The MPS is optimized with DMRG algorithm. The best energy is highlighted boldface and the second best value is highlighted in italic.
complicated structure of its ground state makes it a robust model on which to test state-of-the-art methods. Here, we use the PixeICNN for ARNN and ANTN due to the 2D geometry of the system.
The 2D J1 Hamiltonian is given by
H = J1
Σ o. o + J2 (35
Σ o o
(8)
((.33)
where subscript refers to the site of the qubits, (.. )) is the nearest neighbour and ((.-)) )) is the next
nearest neighbour. o, o. = X, & X, + Y, & Y + Zi Ø Z, with X, Y and Z the Pauli matrices
0
0 1
1 0
X
Y
Z
(9)
0
-1
We will fix J1 = 1 and vary J2 in our studies.
Experiments inductive bias of Jı-J2 Heisenberg Model.
As shown previously, it can be challenging for neural networks to learn the sign structures. The ground state of the Jı-J2 model also has a sign structure, which can be partially captured by the Marshall sign rule(Marshal]] 1955). The sign rule exact at J2 = 0 and becomes worse for large J2- This could introduce bias to the neural network wavefunction if directly applied. We compare the restricted Boltzmann machine (RBM) (from NetKet (Vicentini et al.) (022)), our implementation of gated PixcICNN, and the two different constructions of ANTN with and without the sign rule.

---

The result is shown in Table Since our approach is based on the variational principle discussed in Sec. [3] it provides an upper bound on the exact ground state energy; therefore, lower energy implies better state. As shown in the results, both RBM and PixeICNN improve significantly with the application of the sign rule at J2- = 0.2 and J2_ = 0.5, but the results deteriorate at J2 = 0.8. This is expected because the sign rule becomes increasingly inaccurate J2 increases, especially past J2- = 0.5. Our ANTN, on the other hand, does not require the sign rule in both constructions. As an additional comparison, we note that both of our ANTN constructions achieved better results compared to the recently developed matrix product backflow state (Lami et al. 2022), which uses the sign rule and has per site energy of - 1.9267 at J2 = 0.5.
Ablation study on J1-J2 Heisenberg Model. We scan across many J2 for the model without using the sign rule. In this experiment, we compare the performance of six models to compute the ground state energy of the J1-J2 Hamiltonian system with J1 = fixed and J2 = 0.2-0.8 with 0.1 increments, covering three different phases of the model. The first three models are TN models, using the MPS with bond dimensions of 8, 70, and 1 1024. For up to 4 x system, the MPS results with bond dimension 1024 are exact and can be regarded as benchmark. The fourth model is PixeICNN pure neural network; the fifth and sixth models are elementwise and blockwise ANTN models. Thus the experiment compares the ANTN against the two previous state-of-the-arr techniques.
Table _ summarizes the per site ground state energy computed by different models at different Jz in three distinct phases. Our results provide strong evidence that ANTN integrating TN and ARNN outperforms each base model, achieving sftc-of-the-an results.
Comparison between ANTN and MPS. In all cases the ANTN surpasses the MPS with the corresponding bond dimension which they are based. For example, even in the 10 x 10 system with J2 > 0.5, where MPS with a bond dimension of 1024 outperforms the ANTN, the elementwise ANTN still significantly improves on the base MPS with a bond dimension of 8 and the blockwise ANTN improves on the base MPS with a bond dimension of 70. It is consistent with Theorem, 5.3 and Theorem. 5.2 that ANTN is more expressive than TN.
In addition, the ANTN models scale well for larger systems. Figure 3 visualizes the scalability of ANTN compared to MPS. The figure plots the difference in per site energy between the elementwise ANTN with bond dimension 8 and MPS with bond dimension 1024 for J2 = 0.2, 0.5, 0.8 where lower energies signify better performance for the elementwise ANTN. Compared to MPS, the ANTN models compute better ground state energies as the system size grows larger. As the system size increases, the elementwise ANTN with bond dimension 8 starts to outperform MPS with bond dimension 1024. Even at J2 0.6, where the elementwise ANTN is slightly worse than MPS, the difference gets significantly smaller at 12 12 compared 10x 10 (Table 2) In fact, the elementwise ANTN achieves such a performance using only ~ 1% the number of parameters of MPS. We note that for large J2. the system goes into stripped phase (Nomura & Imada (202]) which can be less challenging for MPS to represent. Nevertheless, in almost all cases our ANTN still outperforms MPS on the 12 X 12 system. MPS has cost dominated by the bond dimension (quadratic for memory and cubic for computational), which limits its use in practice for large system sizes that require large bond dimensions. According to the complexity analysis in Sec. 5.1 ANTN has lower complexity than TN and thus scales better for larger systems.
Comparison between ANTN and ARNN. The elementwise and blockwise ANTN models also consistently outperform the pure neural network PixeICNN model. This agrees with Theorem 5.3 that both elementwise and blockwise ANTN have generalized expressivity compared to ARNN. In Appendix C we further compare ANTN with ARNN and find that: (a) it is challenging to surpass ANTN by ARNN even with an increased number of parameters, (b) the improvement from ANTN mainly comes from the effective inductive bias of MPS structure instead of the DMRG initialization; (c) ANTN has a favorable scaling compared to ARNN and MPS for reaching accurate results.
The details of all experiment setups and hyperparameters can be found in Appendix D
7 Conclusion
In this paper, we developed Autoregressive Neural TensorNet, bridging the two state-of-the-art meth- ods in the field, tensor networks, and autoregressive neural networks. We proved that Autoregressive Neural TensorNet is self-normalized with exact sampling, naturally generalizes the expressivity

---

of both tensor networks and autoregressive neural networks, and inherits proper physics inductive bias (e.g. sign structures) from tensor networks and various symmetries from autoregressive neural networks. We demonstrated our approach on quantum state learning and the challenging 2D J1-J2 Heisenberg model with different system sizes and couplings. Our approach achieves better perfor- mance than both the original tensor network and autoregressive neural network while surpassing tensor networks with large bond dimensions as the system size increases. In addition, our approach is robust, independent of sign rule. Besides scientific applications, since both tensor networks and autoregressive neural networks have been applied to machine learning tasks such as supervised learning and generative modeling, our novel approach holds promise for better performance in these domains due to its exact sampling, expressivity, and symmetries.
Broader Impact
Our Autoregressive Neural TensorNet, blending tensor networks and autoregressive neural networks, could advance our grasp of quantum phenomena, potentially fueling scientific breakthroughs. enhances quantum material design and quantum computing through improved simulations and control of quantum states. This technology may also inspire new machine learning models for handling high-dimensional data. However, possible ethical and societal impacts, such the for chemical weapon development, require careful scrutiny.
Acknowledgement
The authors acknowledge support from the National Science Foundation under Cooperative Agree- ment PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaiff. org/). This material based upon work supported by the U.S. Department of Energy, Office of Science, National Quantum Information Science Research Centers, Co-design Center for Quantum Advantage (C2QA) under contract number DE-SC0012704. This material is also in part based upon work supported by the Air Force Office of Scientific Research under the award number FA9550-21-1-00 2018) and Lincoln Laboratory Supercomputing Center for providing (HPC, database, consultation) resources that have contributed to the research results reported within this paper. Some computations in this paper were run the FASRC cluster supported by the FAS Division of Science Research Computing Group at Harvard University. ZC acknowledges the DARPA 134371-5113608 award and NSF 10434.
The authors acknowledge the MIT SuperCloud (Reuther et al.)
References
Barrett, T. D., Malyshev, A., and Lvovsky, A. Autoregressive neural-network wavefunctions for ab initio quantum chemistry. Nature Machine Intelligence, 4(4):351-358, 2022.
Capriotti, L.., Fubini, A., Roscilde, T., and Tognetti, V. Ising transition in the two-dimensional
quantum j 1- heisenberg model. Physical review letters, (2(15)) 2004.
Carleo, G. and Troyer, M. Solving the quantum many-body problem with artificial neural networks. Science, (55(6335)) 606, 2017. doi: 10.1 2..116//ciecce.ag223002 URL ttte/doi/abs//1e http:/10..111///0 wwt./
aag2302
Chen, 7.,, Luo, D., Hu, K., and Clark, B. K. Simulating 2+ 1d lattice quantum electrodynamics finite density with neural flow wavefunctions. arXiv preprint wtx.xx::2220066000 2022.
Choo, K., Neupert, T., and Carleo, G. Two-dimensional frustrated J J1-22 model studied with neural network quamtum states. Phys. Rev. B, 100:125124, Sep 2019. doi: 10.1 11w.01/yseee..1..... 100.112224.. URL https: ///ink/ aps org/ /oo/10...... 1110/ 100. 125124
Deng, D.-L., Li, X., and Das Sarma, S. Quantum entanglement in neural network states. Physical Review X, 7(2), May 2017. ISSN 2160-3308. doi: 10.1103/physsevxx72010222 URL http
//dx, doi. orgo 10/ 1103/PhysBeex.....m 7222001
10

---

Ferris, J. and Vidal. G. Perfect sampling with unitary tensor networks. Phys. Rev. B, 85:165146. Apr 2012. doi: 10.1 www//oyysee........o 155.16646.. |ttpttsss:/ org/ 1103/ PhysRevB. 85. 165146
Gao, X. and Duan, L..M. Efficient representation of quantum many-body states with deep neural
networks. Nature Communications, 8(1):662, Sep 2017. ISSN 2041-1723. doi: 10. 1038/
s41467-017-00705-2 URL/ttpss://// :/ww://///wwww//// http://.atuueeee http://///....ttie///11///000000
Girvin, S. M. and Yang. K. Modern condensed matter physies. Cambridge University Press, 2019.
Greensmith, E., Bartlett, L., and Baxter, J. Variance reduction techniques for gradient estimates in reinforcement learning. J. Mach. Learn. Res., 5:1471-1530, December 2004. ISSN 1532-4435.
Gutiérrez, I. L. and Mendl, C. B. Real time evolution with neural-networt quantum states, 2020.
Hartmann, M. J. and Carleo, G. Neural-network approach to dissipative quantum many- dynamics. Phys. Rev. Lett. 122:250502, Jun 2019. doi: 10. 103/ 122.25002. URL https://journals: aps. org/ wwt.oooggp//bsaaatt.o.oo
1ww3/110y///e.eett... 122. 250502
Hermann, J., Schätzle, Z., and Noé, F. Deep neural network solution of the electronic schrödinger equation, 2019.
Hibat-Allah, M., Ganahl, M., Hayward, L. E., Melko, R. G., and Carrasquilla, Recurrent neural network wave functions. Phys. Rev. Research, 2:023358, Jun 2020. doi: 10.1103/ aps www.eseerccaabstreeeeepeeeeeeeee
MiisseeRescarcch222223333 URL
L103/PhyaRevResearchh 023358
Huang, Y. and Moore, J. E. Neural network representation of tensor network and chiral states. Phys. Rev. Lett. 127:170601, Oct 2021. doi: 10. 127.1 URL https
//link. aps org 10.0//11.... 1e1111/////heeiie
127. 170601
Lami, G., Carleo, G., and Collura, M. Matrix product states with backflow correlations. Physical Review B, 106(8), aug 2022. doi: 10. 1031100//yeeeeb..000. 106. UtL/ 1103%2Fphysrevb. 106. 1081111
Lante, V. and Parola, A. The ising phase in the jl-j2 heisenberg model. Physical Review, 2006.
Levine, Y.. Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architec- tures. Physical Review Letters, 122(6), Feb 2019a. ISSN 1079-7114. doi: 10. 1001100//pyyseellett.222 065301. URL
aps http://www.....pppppp/////////aa http:///.r//////aaaaaa/o/o//////
122
065301
Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement deep learning architec- tures. Physical review letters, 122(6),06330,, 2019b.
Lu, S., Gao, X., and Duan, L.-M. Efficient representation of topologically ordered states with restricted boltzmann machines. Phys. Rev. B, 99: Apr 2019. doi: 10. 10.1100//yyy .9.9 195111113 36. URL https:://ournaat wwterraasssap...oomm www.arrwwa...a.....ororo...a.a.e http://wwwaaa.a.pppoooooopptaaaa
99. 155136
Luo, D. and Clark, B. K. Backflow transformations via neural networks for quantum many- body wave functions. Physical Review Letters. 122(22). Jun 2019. ISSN 1079-7114. doi: 10. 1033/ppysrevlett.122.2222 PhysRevLett, 122.2264011
URL/httts://jjuuaal... http://waaaaaa....rrrp.rpp.
aps hptp.//...//p/////aattttt//.....
Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via probabilistic formulation, 2020.
Luo, D., Carleo, G., Clark, B. K., and Stokes, Gauge equivariant neural networks for quantum lattice gauge theories. Physical review letters, 127(27):276402, 2021a.
Luo, D., Chen, 7.,, Hu, K., Zhao, Z., Hur, M., and Clark, K. Gauge invariant autoregressive neural networks for quantum lattice models, 2021b. httpt:ttt//aaaae http:///aivv..g//a/////000//////

---

Luo, D., Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via probabilistic formulation. Phys. Rev. Letr, 128:090501, Feb 2022a. doi: 10. 1103/PhysRevl ett. 12t. 090000. httpt://t:///in aps http://i...a...o.oo/o///ao/a//// httt:////oo/......//////aaaaaaae 1103/hhyseevLett 2880090001
Luo, D., Yuan, S., Stokes, J., and Clark, B. K. Gauge equivariant neural networks for 2+ 1d u (1) gauge theory simulations in hamiltonian formulation. arXiv preprint arXiv:2211.03119 2022b.
Marshall, W. Antiferromanetism.. Proceedings of the Royal Society of London. Series A,
Mathematical and Physical Sciences, 232(1188);48-68, 1955. ISSN 00804630. URL http
7/www. jstor wrrg/stable//9968
Nagy, A. and Savona, V. Variational quantum monte carlo method with a neural-network ansatz for open quantum systems. Phys. Rev. Lett., 122:250501, Jun 2019. doi: 10.1100/hhyseeweet..22 250501. URL 250501
. aps. wwtg/prl/abon httgg/r//abstratt/Uu Phtoo////hyyeetttt.t.oom 122
Nomura, Y. and Imada, M. Dirac-type nodal spin liquid revealed by refined quantum many-body solver using netwolknetwork wave function, correlation ratio, and level spectroscopy. Physical Review X, 11(3), aug 2021. doi: 10. ww..100/physrevx.11..000
URL htttps://dii.org//
1103/2Fphysrevx 11. 031034
Oseledets, I. V. Tensor-train decomposition. SIAM Journal on Scientific Computtnn C 3365)2225--2377, 2011. doi: 10. 137/099752286. URL htt.:ttps:/// http:///oooo.o11..1./ 11t////00..177/007522255
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, segh-performance deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Pfau, D., Spencer, J. S., Matthews, A. G. D. G., and Foulkes, W.M. C. Ab initio solution of the many- electron schrödinger equation with deep neural networks. Phys. Rev. Research, 2:033429, Sep 2020. URL tttpps://journn: www.rnals.ass..oom www.:ppp..ggppepppeepppeeeeeeeee wwwgg//rreeeach//// abstract/10 http:////.......ooo/yyereeaaaaaa 2. 033429
doi: 10. I hew.100/yhyseeeesseeeeee2.0.333
Preskill, J. Quantum computing 40 years later. arXiv preprint arXiv:2106.10522, 2021.
Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jones, M., Klein, A., Milechin, L.., Mullen, J., Prout, A., Rosa, A., Yee, C., and Michaleas, Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018.
Schmitt, M. and Heyl, M. Quantum many- body dynamics in two dimensions with artificial neural net- works. Physical Review Letters, 125(10), Sep 2020. ISSN 1079-7114, doi: 10, 1033phyysseell 125. 100503. URL https://journals aps. pttppp/rr/pp/p/baatrt////00
124(2), jan 2020. doi: 10. 10.1103/phsrrevlet. 124.020503. URI. http://taaaatt...ooooo/eeeeeeere 125 2Fphysrevlett. 124.020503
100503
Sharir, O., Levine, Y., Wies, N., Carleo, G., and Shashua, A. Deep autoregressive models for the efficient variational simulation of many-body quantum systems. Physical Review Letters, 1103%
Sharir, O., Shashua, A., and Carleo, Neural tensor contractions and the expressive power of deep neural quantum states, 2021.
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O.,, Graves, A., et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.., Gomez, A. N., Kaiser, E.,, and Polosukhin, I. Attention is all you need. Advances in neural information processing systems, 30: 5998-6008, 2017.
Verstraete, and Cirac, J. Renormalization algorithms for quantum-many body systems two and higher dimensions, 2004. BtLihttps://axxinn httpa/////ooom/m/00000050555
12

---

Vicentini, F., Biella, A., Regnault, N., and Ciuti, C. Variational neural-network ansatz for steady states in open quantum systems. Physical Review Letters, 122(25), Jun 2019. ISSN 1079-7114. doi: 10.1100/physevveent.2 12t.:/eeet..222.2..... 22.250003. URI htt.httss///jouraallll aps pttps.rg/pp///bstrrct///// 1103/PhysRevlett. 122. 250503
Vicentini, F., Hofmann, D., Szabó, A.. Wu, D., Roth, C.. Giuliani, C., Pescia, G., Nys, J., Vargas-
1 quantum systems. SciPost Physics Codebases, aug 2022. doi: 1022166/sciposshhyxooeee.. URL
Calderón, V., Astrakhantsev, N., and Carleo, G. NetKet 3: Machine learning toolbox for many-body
httts://doi..rgg wtt.://............... 2146777sccppstppyssodddd
Vidal, G. Efficient classical simulation of slightly entangled quantum computations. Physical Review Letters, 91(14), oct 2003. doi: 10. 10.1 103/physreveeee.11...... 147902. URL https://doi. org/10
1103/2Pphysrevlettt.c 91 147902
Vidal, G. Efficient simulation of miediieenionnal quantum many- systems. Physical Review
Letters, 93(4), jul 2004. doi: 10.1 10.110//physreelettt...o. 93.040502. URL https://:///oi. org 1103%.
2Fphysrevlett.. 93. 040502 040002
Vidal, G. Entanglement renormalization.
Phys. Rev. Lett., 99:220405, Nov 2007. doi:
10. 11100//Pysseeeeett99922000 URL htRL:ttt::/// aps. wwt. 1103//1yyseeeeetttt 99. 220405
Vieijra, T., Casert, C., Nys, J., De Neve, W., Haegeman, J., Ryckebusch, J., and Verstraete, Restricted boltzmann machines for quantum states with non-abelian anyonic symmetries. Physical Review Letters, 124(9), Mar 2020. ISSN 1079-7114. doi: 10. 124pppysssseeeeee120000720 aps. wtw..p./p.........tt URL htttss://jornnn hpt..///ppp////aaaatttttt//o wwt.:///se///.eetttttt.omm 124. 097201
Westerhout, Astrakhantsev, N., Tikhonov, S., Katsnelson, M. I., and Bagrov, General- ization properties of neural network approximations to frustrated magnet ground states. Nature
Communicationns, 11(1) Mar 2020. ISSN 2041-1723 doi: 10.1038/s41461 1888//44467-220 15402-w.
URL httpttt/t//....... org/ 10. 1038/ 1038/ $41467 -20 10000-15502-44
White, S. R. Density matrix formulation for quantum renormalization groups. Physical review letters, 69(19)22863, 1992.
Wu, D., Rossi, R., Vicentini, F., and Carleo, G. From tensor network quantum states to tensorial recurrent neural networks. arXiv preprint arXiv:2206. 20t.22206./3633, 2022.
Yoshioka, N. and Hamazaki, R. Constructing neural stationary states for open quantum many- body systems. Phys. Rev. B, 99:214306, Jun 2019. doi: 10. 10.1100//hysRevB.. 999...99999..... 99.22143666 URL https://journals. aps. org/ ptbp /ttp://atatttt....////////////// .9.99493066
13

---

Appendices
A Additional Background
A.1 Quantum State Learning
In this section, we discuss more details about quantum state learning. As mentioned in the main paper, the quantum fidelity
F = (())bb))
Σ o" (x))e(())
(A.1)
measures the closeness of the (normalized) quantum states (o) and ((be), with denoting complex conjugation. By minimizing - log F, one can obtain the (ua) closest to the target state (ø) (see Appendix(]foo D optimization details). For small system sizes (S 20 qubits), is possible enumerate all the basis & and compute log F' exactly as in the case of this work. As mentioned in the main paper, we learn the random Bell state and shallow random circuit test the expressivity and physics inductive bias of the ansatz. More specifically, the states are generated as follows
Random Bell State. A two-qubit Bell state defined to be the state (J00) + /11))/v2. We use the following algorithm to generate n-qubit random Bell state (n is multiple of 2).
Algorithm Random Bell State Generation
shuffle((1 11... )) < shuffle _ + nl...n)
V <
x-1
while S n/2 do V.append (belddbbllleee bell_state(as,bb b.))
i i+i+1
end while
(b) + product _
return (e))
where bell soll_sttte(;.) creates Bell state given the two qubits and product product_state([) creates the tensor product state for list of individual states. Each two-qubit Bell state in the random Bell state forms between the left half and right half of the system, allowing a maximum entanglement across cut in the middle of the system.
Shallow Random Circuit. The shallow random circuit generates random states using the following algorithm.
14

---

Algorithm 2 Shallow Random Circuit State Generation
10)
[ [-]
while S ni do
i 4-1
while < n do
U random gate + 1)
()) - U (
(--0 2
end while
i
while < n do
random gate(i, +) 1)
( - U (b)
end while end while return (g))
where ny is the number of layers, and random random_gate(:.))) generates random unitary gate the two qubits. The random -aatam--_tttt)))) function is realized by first generating a vrauevv Gaussian random matrix of size x 4, following QR decomposition of the matrix obtain the unitary part as the random unitary gate.
Each of the above algorithms defines distribution of states, which we average over multiple realizations to compute the mean and standard deviation of the learned quantum fidelity.
However, for large system sizes, the Hilbert space is too large, so one has to evaluate the fidelity stochastically. This can be achieved by rewriting
J= > (m))v(() Σ (()))) (2(m))200 Vo(ar)
v0( |0)2 ((xx)
(A.2)
Φ(x)
Which allows the fidelity to be evaluated by sampling from written
10) The gradient of log F can be
Vof
E
2012
(eva(w) ((x)
da ((m)
Vg 9 log F 2R
2R
22R
Bx
log (ar))
E
(2))
(arl ((x))
((#)
(A.3)
Alternatively, one can sample from vo (x) by writing the complex conjugated
d(x)
((x)
Σ [2))
E lohol
(A.4)
(x)
This time, we have to be careful with
((x) (x)
((x)
Vof Σ Vo) = Σ 22)
Vo log (x)) E B0010
Ve log (z))
1hg (x)
(A.5)
Then,
((x))
log
-22 Vos
-2R
E R
(o(
log 2/g (x:)
(A.6)
E
We can further use the variance reduction technique (Greensmith et al. 2004) from reinforcement learning and write
((()) 2o (x)
V log = -2R e~
Vo log (x),
(A.7)
vio
E
15

---

where the minus I comes from the mean of the coefficient in front of V log 2ig
d(x)
(
(A.8)
100 (a
o(x²)
One can verify that without the assumption that (w)) normalized (i.e. F == Kél // (1bo (%a)))
Eq. A.7 can also be obtained without applying the variance reduction formula.
In the main paper, due the small system size N = 16, enumerated all the bases when evaluating the quantum fidelity.
A.2 Variational Monte Carlo
As shown in the main paper, given a Hamiltonian H (and its matrix elements Ha,
Ne (xx'
Ex Ha
a vo(()))
(wee Ho)
Σ Ha (2))
Σ 20.100
l (x))
En lvo.
1
wh
(A.9)
Since the Hamiltonian 7 is usually sparse, give æ, one only needs to sum up a small number of x' ss for the numerator (usually linear in system size), allowing Eq. A.9 to be evaluated efficiently. Analogously, we can evaluate the gradient as
Ve (bh (H(
2A Σ He
(ori vo
Vougg (x) vo)
20
Σ
2 Ha.
(A.10)
1 (xx
Vi (xx)
H dh (x')
2R E //ol
log (xr).
t/e (x:
Furthermore, is possible to reduce the variance by either directly applying the variance reduction formula (Geennsmith et al. 2004), or explicitly normalizing (ebe) (minimizing (WelFil)oo) / (welwe) obtain
Exe (o((')) (x)
Vo (wo (()o)) 2R E E 1000
(wol) (Wol7 V log (x))
(A.11)
where (thg H (bbo) can be approximated by applying Eq. A.9 on the same batch æ. In this work, we use Eq. A.Il 1.Ilas as the gradient of the loss function (sce Appendix D optimization details).
B Additional Theoretical Results
B.1 Exact Sampling from Conditional Probabilities
Suppose a full probability distribution over multiple qubits is written as a product of conditional
probabilities
p(x) = p(21,.......
2n = I p p
(B.1)
with æei (z1,... 11), then sampling bitstring a from the probability distribution can be obtained by sequentially sampling each conditional probability as Since each sample sampled independently using this algorithm, batch of samples can be obtained in parallel, allowing for an efficient implementation on GPUs. In addition, the exact sampling nature of the algorithm makes not suffer from the autocorrelation time problem of standard Markov chain Monte Carlo.
This same algorithm applies to sampling from (2)) with a simple replacement p(z,(1<2)) + [=<x, This is direct consequence of
1 a = II ( m22)
(x))):) = [ [<+)
(B.2)
and the fact that require Σx, (2,2//0)))) =1.
16

---

Algorithm 3 Exact Sampling from Conditional Probabilities/Waeetuiii
<< 0
i
while S n do
2<i + x Ti <<) x.append(z.) i f i+1
end while return
B.2 Exact Sampling from MPS
As mentioned in the main text, matrix product state (MPS) with particular constraints, also allows for exact sampling. Recall that MPS defines a wavefunction as
(xx)
Σ
M
Mon 10n
EI
M 10:
(B.3)
Let's 's first assume that the wavefunction is normalized. It is easy to notice that any gauge transforma- tion
(Ma,; Ma - (Ma, A, (1;
(B.4)
with A being an invertible matrix leaves the resulting wavefunction invariant. Here, we suppress the a indices and view them as the indices for matrices. It has been shown that by fixing this gauge redundancy (Vidal 2003, 2004), we can restrict all the M matrices satisfy the following condition
(M (M
Mac-1a
5 199f-1'
(B.5)
22,00
where 8., is the Kronecker delta function. The matrices M that satisfy this condition are called isometries, and the MPS is called in the right canonical form (to be distinguished from the left canon- ical form). the right canonical form, each M - can interpreted as a basis transformation (2i, oi) 05-1 that is part of unitary matrix.
Theorem B.1. Defining the left partially contracted tensors
ML
= EI M
(B.6)
OGS EI
If the matrix product state is in the right canonical form, then the marginal probability of the wavefunction satisfies
g(x51)= = Σ [(b())) = Σ MLE'
(MLRsS)
(B.7)
Proof. Let's define the right partially contracted tensors analogously
MR = EII
Ma: 19;
(B.8)
as 1=)
We will first show that
Σ M
(MROET)) d'-1) = 8 8ay-1
(B.9)
We prove this by induction:
Base case:
EMe. Ma.-1e 10n (Mai 101)) 8 Sam
MRa.- an-i
(M
(B.10)
Ti, which directly comes from Eq. B.5 by noticing that D(an) = 1 so it can be ignored.
17

---

Inductive step: Write
M
a--1 Σ
M 10 M
R
(B.11)
Assuming
Σ
MRag
(MRaesn))
Day,a't
(B.12)
#2
then
Σ [
(MRaZS)
Σ E
Ma
Moj-ing
(ME)
Σ
MR
(MRizs
maiil
M
(M
òay.at
Σ
M
Maj-10 10s
(M
2j.9j
8 00y-1,a)
(B.13)
Using this result, we can then prove our desired result about the marginal probability. Let's write
(b(x))
M M
Σ ML MR
(B.14)
1=1
then,
((xg)))
Σ |x(x)|1
ΣΣ
Mr MR
(Mress MR
#ssojin
Σ
ML
aj
(MLR'))
8ay,a
(B.15)
Σ
ML
(Mioss)
aj
Corollary B.1.1. Matrix product state allows for efficient exact sampling.
Proof. Thm. B.1 shows that marginal probability distribution g(ærgs) can be evaluated efficiently. The conditional probability (equivalently) as an explicit normalization of the marginal probability distribution g(v; (se))
be evaluated efficiently either q10; (( /q(x/)/)))))))
((xx//// Σx, (3)2))) Then, Algorithm 3 can be used sample from the full distribution.
B.3 Autoregressive Sampling Order
Algorithm 3 samples from conditional probabilities, which requires an ordering the system to be defined. For ID systems, and qubit n corresponds to the rightmost qubit. This ordering is natural for ID MPS and transformer, well the wansformer-basee ANTN. For 2D systems, we ordering, the qubit at the 2D location (i.j) is defined to be the x Ly + th qubit, where Ly is the number of qubits along the second dimension. This ordering is inherently defined from the masked convolution for PixelCNN and PixeICNN-based ANTN. The MPS reshaped to satisfy the same ordering for 2D systems. The same 2D ordering can be generalized other tensor networks well.
linear ordering such that qubit corresponds the leftmost qubit,
snake (zig-zag) ordering. In this
B.4 Additional Details of Expressivity and Symmetry Results of ANTN
Theorem B.2. Both tensor networks and autoregressive neural networks are special cases of Autore- gressive Neural TensorNet.
18

---

Proof. Recall that the ANTN defines the wavefunction from its amplitude and phase
xb(x) == (((
(B.16)
where
(x)) II { ==1
(B.17)
((
with g(0,|=<e)) = Ex, g(
(B.18)
(( = Σ
VZ (x W& (x-د))
(B.19)
aj
15 (xgs) = Σ
N 3ol- (a. (o2))
(B.20)
acj i=l
and
<(x) =: Arg Σ
II
30--1.00 1.00 (1z(
(B.21)
ANTN reduces to MPS. If don't modify the tensors with neural networks, Go 1.0g (z, ( reduces to M2;-1.00 and 12 (x=z) reduces to MLSs. It is then straight- forward that Eq. B.19 reduces Eq. B5 and thus the probability distribution of ANTN reduces to that of MPS. Analogously, Eq. B.21 reduces to the phase of an MPS, and thus ANTN wavefunction reduces to MPS wavefunction.
ANTN reduces to ARNN. If we set all D(a) = 1, then 3a.- (mea)))))))))))))))()) reduces to (xr (x) and all the summations over a can be dropped. In this case, g(mgg)) reduces to I (=x(((==))) and {{ reduces to [ (جم)) which is the same as the standard ARNN. In addition, the phase ((x) reduces to Arg I[:-1 (b(xa(at)))) which is also the same ARNN. Thus, ANTN wavefunction reduces ARNN wavefunction.
Notice that we only showed one type of TN, i.e. MPS, but the proof can be easily generalized to any TN that allows evaluation of marginal probabilities.
Ll
Theorem B.3. Autoregressive Neural TensorNet can be written as either tensor network autoregressive neural network with an exponential (in system size) number of parameters.
Proof.
ANTN TN. In ANTN, the base ARNN outputs the conditional tensors qia (2: x <i) form the final wavefunctions. These conditional tensors can be written as 2a1-19 2a1..2..... in tensor notation. It directly follows from the shape of these tensors that they contain x² 2' elements (where x2 from the two bond dimensions and 2' is from all the qubits at before the current qubit i). According to Theorem 5.2 ARNN has volume law entanglement while TN doesn't. The conditional tensors generated from ARNN do not permit efficient tensor decomposition. Therefore, a tensor network almost has to parameterize all full conditional tensors, resulting in ΣN x² 2 O(x2 2 parameters in total.
ANTN as ARNN. In ANTN, the conditional probability is calculated from the marginal probability as shown in Eq. as
(( = Σ 32 (x<))
x<)))
as
Σ
Jan (21) jaa aa x an
as of
(B.22)
The summation over os 's contains 26 terms, each of which can be viewed as marginal (quasi-)probabbiiiy gdt q ai (x<)) generated by the underlying ARNN of ANTN, This
19

---

effectively creates weight matrix on top of a conventional ARNN, whose shape is haim x X 2i-1
(where haim is the hidden dimension, and 2 the local dimension of each qubit) on top of conventional ARNN, resulting in Σm= 2 hdim x²==1 (hhimm x²²) parameters.
Definition B.1 (Mask Symmetry). A conditional wavefunction (xx(((=))) has mask symmetry if (I, xx.) O for some I, given Zzi-
Theorem B.4. Autoregressive Neural TensorNet inherits mask symmetry from autoregressive neural network.
Proof. The mask symmetry results in (xx)) O for certain æ satisfying the condition while preserving the normalization of ((r). For ANTN, we directly set q( (i)))) for the 2. given Eei.. This results in (x)) = 0 for the same a.
Corollary B.4.1 (Global U(1) Symmetry). Autoregressive Neural TensorNet can realize global U(1) symmetry, which conserves particle number.
Proof. A global U(I) symmetry in qubit system manifests as a conservation of the number of O's and number of I's in æ. Equivalently, (xx) for violates such conservation. (Hibat-Allah et al.) 2020) has shown that autoregressive neural networks can preserve global U(1) symmetry as a mask symmetry, which ANTN inherits.
Definition B.2 (Function Symmetry). conditional wavefunction tensor (xx x<s)) has func- tion symmetry over a function F if ((x,( = (F((;;;;e:) F(x<:)) where F(ag)) == (F(x1), F(r=)),,,
Theorem B.5. Autoregressive Neural TensorNet inherits function symmetry from autoregressive neural networks.
Proof. We can apply the function symmetry on the left partially contracted tensors instead of the conditional wavefunctions. Here, we show that this produces the desired conditional probabilities and phase. Applying the function symmetry on the left partially contracted tensors results in
1 20: (~5z) (F
(B.23)
This implies that the marginal probability satisfies () = ((F(z)))) after contracting over the index atj, which then results in the correct symmetry on the conditional probabilities
(و, = ((F((,))
(B.24)
as the conditional probability is just the marginal probability normalized at site j. The phase, on the other hand,
(xx) Arg EII (x.( Arg (Fga),
(B.25)
which satisfies the same function symmetry from Eq. B.23
Corollary B.5.1 (Z, Spin Flip Symmetry). Autoregressive Neural TensorNet can realize Ez spin flip symmetry such that the ANTN wavefunction is invariant under conjugation of the input.
Proof:
Spin flip symmetry exists for quantum chemistry systems that do not couple spin up and spin down electron, so that the wavefunction is invariant to inputs when spin up and down are flipped. (Barrett et al. 2022) has shown that autoregressive neural network can preserve Z2 spin flip symmetry as function symmetry, which ANTN inherits.
O
Corollary B.5.2 (Discrete Abelian and Non-Abelian Symmetries). Autoregressive Neural TensorNet can realize discrete Abelian and Non-Abelian symmetries.
Proof. Gauge symmetry a local symmetry such that the function invariant under local transfor- mation. (Luo et al. 2021b) has shown that autoregressive neural networks can preserve discrete Abelian and non-Abelian symmetries as either the mask symmetry or the function symmetry, which
ANTN inherits.
D
20

---

C
Additional Experiments
In this section, we perform additional experiments to further compare ANTN and ARNN, and show that
It is challenging to surpass ANTN by ARNN even with increased number of parameters.
The improvement from ANTN mainly comes from the effective inductive bias of MPS
structure instead the DMRG initialization.
ANTN has favorable scaling compared to ARNN and MPS for reaching accurate results.
Energy per site and additional information for 10 x 10 system
Algorithms
J2 = 0.2
J2 = 0.5
J2 0.8
N paramss Narroms NN
N parami
training
MPS (8)
-1.997377
L.734326
1.869871
1.2 x 10
0
1.2 10
hr
MPS (70)
-2.191048
L.866199
-2.007845
10a
8,8 x 10
hr
MPS (1024)
-2.255633
1.938770
-2.062730
1.7 x 10
0
1.7 x 10
242 hrs
PixeICNN (7-48)
-.22246(22)
1.74008(29)
-1.9833((7))
5.0 10$
5.0 x 10a
43 hrs
PixeICNN (P) (7-48)
-2.1920(9)
-1.6458(10)
-1.9468(6)
5.0 10
5.0 10
19 hrst
PixeICNN (8-48)
-2.23581
(1886922(16
-2.01093(16)
5.7 10
5.7 10
59 hrs
EW (7-48-8)
-2.26034
11.4000
-2.05707(43)
1.2 x 106
5.0 x 10a
1.7 10
89 hrs
EW (RI) (7-48-8)
-2.25946(6)
-2.05125 1.94003
1.2 x 10
5.0 10
1.7 X 10
46 hrs
BW (7-48-70)
(.25755(
1.93842 42) 2.05088
10
5.0 10
X 10 136
Table 3: Energy per site 4, tensor network parameters Naarams, TN, neural network parameters Nparams.NN. NNN total parameters N paaamm and runtime Tirsiiing for 10 10 system with various algo- rithms. Elementwise (EW) and blockwise (BW) are two constructions of ANTN with PixeICNN the underlying ARNN, For MPS, the number in the parenthesis labels the bond dimension. For PixeICNN, the two numbers label the number of layers and hidden dimension respectively. For ANTN, the numbers label (in this order) the number of layers, hidden dimension, and bond dimension. For PixelCNN, we also test pertaining (P) using MPS result (with bond dimension the last number in parenthesis), and for EW, we in addition test random initialization (RI) of the MPS part. The algorithms in boldface are new experiments not previously shown in the main paper. Best energies for each J2 are also highlighted in boldface. The MPS with DMRG algorithm is run on CPUs, while the PixeICNN, elementwise (EW) and blockwise (BW) ANTN are trained using GPUs, with PixeICNN (P) (7-48) and EW (RI) (7-48-8) using A100 GPUs (labeled with 1), and the rest using V100 GPUs.
-1.75
DMRG
DMRG
PixelCNN
CNN
Energy per site
-1.80
EW
EW
BW
BW
10 10
10 x 10
-1.85
12 12
12 x 12
-1.90
-1.95
10*
10
10
10a
100
150
200
250
Number of parameters
Runtime [hrs]
Figure 4: Energy per site total number of parameters and runtime (in hours) for various algorithms (MPS with DMRG algorithm, PixeICNN, elementwise (EW), and blockwise (BW) ANTNs) and system sizes (10 x 10 and 12 x 12) for J2 0.5. The ANTN construction uses PixeICNN as the underlying ARNN, The total number of parameters includes parameters from both the TN part and the ARNN part.
C.1 Comparison between ANTN and ARNN with More Layers
In Table 3 and 4 we include additional results.
21

---

Energy per site and additional information for 10 10 system J2 = 0.5
Algorithms
J2 = 0.5
params,, TN
N params, N
N params
Training
MPS (8)
-.7734226
1.2 x 10*
0
1.2 x 10
1 hr
MPS (70)
1.866159
8.8 x 10a
0
8.8 x 10
< 1 hr
MPS (1024)
-1.938770
1.7 x 10
0
1.7 x 10a
242 hrs
PixelCNN (7-48)
(.74499829)
0
5.0 x 10"
5.0 x 10
19 hrs (A100) or 43 hrs (V100)
PixeICNN (8-48)
1.86922 (66)
0
5.7 x 10
5.7 x 10
59 hrs (V100)
PixeICNN (9-48)
1.90021
0
6.3 x 10
6.3 x 10
21 hrs (A100)
PixeICNN (10-48)
1.90440(14) (4)
0
7.0 x 10
7.0 x 10
23 hrs (A 100)
PixeICNN (11-48)
-1 1.92826 11)
0
7.7 x 10
7.7 x 10"
26 hrs (A 100)
PixeICNN (12-48)
-1.92986(10 (0))
0
8.4 x 10
8.4 x 10
30 hrs (A 100)
PixeICNN (15-48)
(.92869
0
1.0 x 10
1.0 x 10
51 hrs (A100)
PixeICNN (20-48)
1.91537
0
1.4 x 106
1.4 x 10
66 hrs (A 100)
PixeICNN (8-72)
- 1.88536(199 19)
0
1.3 x 106
1.3 x 10
35 hrs (A 100))
PixeICNN (9-72)
(5))
0
1.4x x 10c
1.4 x 10
40 hrs (A100)
PixeICNN (I0-72)
1.90718(15)
0
1.6 x 10
1.6 x 10
47 hrs (A100)
PixeICNN (11-72)
1.92965
13)
0
1.7 x 10
1.7 x 10
52 hrs (A 100)
PixeICNN (12-72)
-191118(11)
0
1.9 x 10
1.9 X 10
57 hrs (A 100)
PixeICNN (S) (11-48)
1192915(122
0
7.7 x 10
7.7 x 10
31 hrs (A 100)
PixeICNN (S) (12-48)
1.93112(11
0
8,4 x 10"
8.4 x 10"
34 hrs (A (A100)
PixeICNN (S) (15-48)
1.93233(9)
1.0 x 10"
1.0 x 10
47 hrs (A 100)
PixeICNN (S) (20-48)
1.93058(10)) 10)
0
1.4 x 106
1.4 x 10
64 hrs (A 100)
PixeICNN (S) (11-72)
- (19)211(10)
0
1.7 x 10"
1.7 x 10
54 hrs (A100)
PixeICNN (S) (12-72)
1.92812(12)
0
1.9 x 10c
1.9 x 10
59 hrs (A (A100)
EW (7-48-8)
1.94001(())
1.2 x 10
5.0 x 10
1.7 x 10
46 hrs (A100) or 89 hrs (V100)
BW (7-48-70)
-1.38422
42)
8.9 x 10"
5.0 x 10
1.4 x 10
136 hrs (V100)
Energy per site 4, tensor network parameters NPprams, TN, neural network parameters N Nparame.NN. total parameters Nparams and runtime Teraining for 10 x 10 system at J2 = 0.5 with various algorithms. Elementwise (EW) and blockwise (BW) are two constructions of ANTN with PixeICNN as the underlying ARNN. For MPS, the number in the parenthesis labels the bond dimension. For PixeICNN, the two numbers label the number of layers and hidden dimension respectively. For ANTN, the numbers label (in this order) the number of layers, hidden dimension, and bond dimension. For PixelCNN, we also test the sign rule (S) The algorithms in boldface are new experiments not previously shown in the main paper. Best energies for each J2 are also highlighted in boldface. The MPS with DMRG algorithm is run on CPUs, while the PixeICNN, elementwise (EW) and blockwise (BW) ANTN are trained using GPUs
Table 4:
Additional information for 12 x 12 system
Algorithms
N pparams, TN
params, NN N Nparams
T Training
MPS (8)
1.8 x 10a
0
1.8 x 10*
1 hr
MPS (70)
1.3 x 106
0
1.3 x 106
< 1 hr
MPS (1024)
2.6 x 108
0
2.6 x 10
329 hrs
PixeICNN (7-48)
0
5.0 x 105
5.0 x 10a
82 hrs
EW (7-48-8)
1.7 x 106
5.0 x 10s
2.2 x 10th
183 hrs
BW (7-48-70)
1.3 x 106
5.0 x 10
1.8 x 10. 287 hrs
Table 5: Additional information for the number of parameters and runtime of the 12 x 12 experiment in the paper for various algorithms. The runtime is average over different J2's 's. The MPS with DMRG algorithm is run on CPUs, while the PixeICNN, elementwise (EW), and blockwise (BW) ANTN are trained using V00 GPUs The ANTN uses PixeICNN as the underlying ARNN. For MPS, the number in the parenthesis labels the bond dimension. For PixeICNN, the two numbers label the number of layers and hidden dimension respectively. For ANTN, the numbers label (in this order) the number of layers, hidden dimension, and bond dimension.
22

---

In Table B] we perform 2 addition tests: a) pretrain pure PixeICNN by learning MPS results; b) train ANTN (elementwise) without DMRG initialization. We find that DMRG pretraining negatively affects the result, while ANTN without DMRG initialization performs as good as with DMRG initialization. This is understandable, as learning MPS results with complex sign structure essentially the same task as learning a quantum state from shallow random circuit, which has been shown a difficult task for ARNN in Figure 2(b) of the original paper. Since the MPS result is an approximation of the actual ground state, learning this state actually makes worse initialization for PixeICNN. (We note that the PixeICNN was originally initialized from the result of smaller system sizes, as we described in Appendix D under transfer learning.) On the other hand, the ANTN, despite not using DMRG initialization, still retains the ability to represent flexible sign structures. In addition, since DMRG is just optimization algorithm to train MPS, the gradient-descent- besceet---bed algorithm should still train the MPS part of ANTN reasonably well without DMRG initialization.
In Table 4 we test series of ARNNS with different layers (up to 20 layers) and hidden dimensions (up to 72 hidden dimensions) such that the largest ARNN tested has more parameters than ANTN. We found that all the ARNN energies are not good at ANTN (which is only based on 7 layers and 48 hidden dimensions), despite that ARNNS have more parameters and take a longer time to train using the same GPU (see new data below). Furthermore, we observe that an increase of parameters can help ARNN to improve energy in general. but this improvement hits a diminishing return potentially due to difficulties in optimization the (approximate) sign rule and found that it could help ARNN obtain better energies but still worse than ANTN without the sign rule. This indicates that the ANTN has the advantage of inheriting the flexible sign structure from MPS, avoiding the manual bias of adding the sign rule. The new results have provided strong support for expectation, both from an expressivity perspective and from the fact that ANTN has better physics inductive bias than ARNN for optimization. We have also added a plot for the new benchmark in the updated manuscript. Although the added layer improves the result to some extent, the PixeICNN fails to beat ANTN in terms of energy calculation. This is consistent with Theorem 5.3 that ANTN is more expressive than ARNN
the number of parameters increases. In addition,
further tested
C.2 Comparison of Runtime and Number of Parameters
In Table 3. Table Table 5 and Figure 4 we show the number of parameters and runtime of each algorithm. In summary, MPS has the most parameters, and is the slowest with DMRG algorithm, taking as long as 2 weeks. The ANTN is much more efficient compared to MPS while obtaining better energies, and it is comparable to pure ARNN while obtaining much better energies. Within the two types of ANTN, the elementwise construction, despite using more parameters, runs faster than the blockwise construction. We note that since TNs and ARNNs use the parameters very differently. a direct comparison of the two types of parameters may be unfair. Therefore, we list both the total number of parameters and the parameters for the individual parts (TN part and ARNN part).
D Experimental Details and Reproducibility
We note that while the following details can be used to reproduce our results, additional adjustments may be possible to further improve our results. The code associated with this paper is available at https://github. www//oo///a/oeecc antnn023//22////////// ANTN
Implementation. The transformer Vaswani et al. 2017) used in this work is decoder-only transformer implemented in (Luo et al. 2020), and the PixeICNN used is the gated PixeICNN (Van den Oord et al. 2016) implemented in (Chen et al., 2022 but without the channelwise mask.
Hyperparameters. For the transformer in quantum state learning, we use 2 layers, 32 hidden dimensions, and 16 attention heads, whereas for the PixeICNN in variational Monte Carlo, we use 7 layers with dilations 1, 2, 1, 4, 1, 2, and for each layer and 48 hidden dimensions. For additional experiments with more layers, we always use dilation of for the layer and beyond.
Initialization. Throughout the experiment, we initialize the weight matrices of the last fully connected layer of the ARNN and ANTN to O, and initialize the bias according to the following rule:
The bias of ARNN is always randomly initialized.
* For the random Bell state learning, the bias of ANTN is randomly initialized
23

---

For the rest of the experiment, the bias of ANTN is set to O.
The rest of the parameters of ARNN and ANTN are randomly initialized according to PyTorch (Paszke et al., 2019)'s s default initialization scheme.
In addition. for the random Bell state learning experiment, we do not initialize the ANTN with MPS (and hence the random bias), whereas for the shallow random circuit learning experiment, we initialize the ANTN with an MPS that comes from explicit truncation of the full wavefunction, and for the variational Monte Carlo experiment, we initialize the ANTN with DMRG optimized MPS.
Optimization. Through the experiment, 0.01. For quantum state learning experiments, we train the ARNN and ANTN with full basis for 2000 iterations, where the learning rate is halved iterations 600, 1200, and 1800. In case the full basis cannot be computed in one pass due to GPU memory constraint, we divide the full basis into several batches and accumulates the gradient in each iteration. For variational Monte Carlo experiments, train the ARNN and ANTN stochastically until the energy converges. During the training, we halve the learning rate at iterations 100, 500, 1000, 1800, 2500, and 4000. In each experiment, we choose the maximum batch size that can be fed into the GPU memory, while using accumulation steps to keep the effective batch size around 10000 throughout the training.
the Adam optimizer with an initial learning rate of
Transfer Learning. We in addition take advantage of transfer learning. For J2 = 0.2, 0.5, and 0.8, we begin from scratch with the 4 x 4 system. Then, N x N system is initialized from an (N 2) 2) x (N - 2) system by keeping all but the weights in the last layer. Then, for other J2, the neural network is initialized from one of J2 0.2, 0.5, and 0.8 that is closest the current J2 of the same system size by keeping all but the weights in the last layer. We find this allows the optimization converge faster than always starting from seratch.
Computational Resources. The models are mainly trained using NVIDIA V100 GPUs with 32 GB memory and Intel Xcon Gold 6248 CPUs, with some models trained using NVIDIA A100 GPUs with 80 GB memory.
24

---

References
Barrett, T. D., Malyshev, A., and Lvovsky, A. Autoregressive neural-network wavefunctions for ab initio quantum chemistry. Nature Machine Intelligence, (4):351-338) 2022.
Capriotti, L.. Fubini, A., Roscilde, T., and Tognetti, V. Ising transition in the two-dimensional
quantum j 1- heisenberg model. Physical review letters, (2(15):15700,, 2004.
Carleo, G. and Troyer, M. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325) 606, 2017. doi: 00.116//ciinccc.uu222222 URL hRtthtts:///wrt stttpp:://ww..cceenc org/ http///o//abb///// 1126/ 111///////////eee science aag2302.
Chen, Z., Luo, D., Hu, K., and Clark, B. K. Simulating 2+ 1d lattice quantum electrodynamies finite density with neural flow wavefunctions. arXiv preprint arXiv:2212.06833 2022.
Choo, K., Neupert, T., and Carleo, G. Two-dimensional frustrated JI J1-J2 model studied with neural network quantum states. Phys. Rev B, 100: 125124, Sep 2019. doi: 10.1 1..100/hyysReBB.... 100.152244.
125124.
URL https: puttps://iine //link org. doi /100//100.. 111111101Ppp/ Pata/hhysRee..... 100. 125124
Deng, D.-L., Li, X., and Das Sarma. S. Quantum entanglement in neural network states. Physical Review X, 7(2), May 2017. ISSN 2160-3308,
doi: 10. 10.11000/hysreexx.0220222 URL http:
//dx org/ / wwtgg/...1103//11 1110//hyyseex...ttmm 7 7.00101
Ferris, A. and Vidal, G. Perfect sampling with unitary tensor networks. Phys. Rev. B, 85:165146, Apr 2012. doi: 10.1 11100//hyeee...8..6661. URL httth:t/tinnkkkk. aps. org//oi//1// 1103/
PhysRevB. 85 165146
Gao, X. and Duan, L.-M. Efficient representation of quantum many-body states with deep neural
networks, Nature Communications, 8(1):662, Sep 2017. ISSN 2041-1723. doi: 10310
s41467-017-00705-2 URL URLhttss:://wwt http://wwtttt..ett///////////e.t nature com /ttpa/acceeee/11177000000-000000
Girvin, S. M. and Yang. K. Modern condensed matter physics. Cambridge University Press, 2019.
Greensmith, E., Bartlett, P. L., and Baxter, J. Variance reduction techniques for gradient estimates in reinforcement learning. J. Mach. Learn. Res., 5:1471-1530, December 2004. ISSN 1532-4435.
Gutiérrez, I. L. and Mendl, C. B. Real time evolution with neural-network quantum states, 2020.
Hartmann, M. J. and Carleo, G. Neural-network approach to dissipative quantum many-body dynamics. Phys. Rev. Lett., 122:250502, Jun 2019, doi: Leet 122.250502. URL https: //jjarn///ualss aps .app.rrg/...o..................r httplpppaaapppppppppp www.11100/hyyseeLettt.. 122. 250502
Hermann, J., Schätzle, Z., and Noé, F. Deep neural network solution of the electronic schrödinger equation, 2019.
Hibat-Allah, M., Ganahl, M., Hayward, L. E., Melko, R. G., and Carrasquilla, J. Recurrent neural network wave functions. Phys. Rev. Research, 2:023358, Jun 2020. doi: 10.1 RhysRevResearcc.20023333 URL htLphttss:///onnma 1103/PhyaRevResearcch 2. 023358
aps http://reeeaaccaabbreeeeee
Huang, Y. and Moore, J. E. Neural network representation of tensor network and chiral states. Phys. Rev. Lett., 127:170601, Oct 2021. doi: 10. wew.:oo/Phyysee.e..............
URL https
//link. aps www.g/////iii 1ttp:/10..110///aaaaaa www./////////ettt..eeeoee 127. 12700700011
Lami, G., Carleo, G., and Collura, M. Matrix product states with backflow correlations. Physical Review B, 106(8), aug 2022. doi: 10. 103//hysrevb..000. 106.1081111111111111111111111111 https://////oo
1103//2physrevb... 106. 1081111
Lante, V. and Parola, A. The ising phase in the j1-j2 heisenberg model. Physical Review, 2006.
Levine, Y., Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement deep learning architec- tures. Physical Review Letters, 122(6), Feb 2019a. ISSN 1079-7114, doi: 10. 10.1100///ysr 122 065301. URL http:a///ooooaaaaaa.aoo/o/////aa http:////aasttttttooooooooeeeeee 122
065301
25

---

Levine, Y.. Sharir, O., Cohen, N., and Shashua, A. Quantum entanglement in deep learning architec- tures. Physical review letters, 122(6):06531,, 2019b.
Lu, S., Gao, X., and Duan, L.-M. Efficient reprrsettionnn of topologically ordered states with restricted boltzmann machines. Phys. Rev. B, 99:155136, Apr 2019. doi: 10.1133/PhysReBBB ,9. 199155166 136. URL
1ttp://..1////hhhheheeeee 99. 155136
Luo, D. and Clark, B. K. Backflow transformations via neural networks for quantum many- body wave functions. Physical Review Letters. 122(22). Jun 2019. ISSN 1079-7114. doi: 10. 1103/physrevlet. 222.226401. htR.httts://jouraa.... aps http:///r//aaassrrrtoo..... PhysRevLett. 122. 226401
Luo, D., Chen, Z., Carrasquilla, J., and Clark, K. Autoregressive neural network for simulating open quantum systems via probabilistic formulation, 2020.
Luo, D., Carleo, G., Clark, B. K., and Stokes. J. Gauge equivariant neural networks for quantum 2021a.
lattice gauge theories. Physical review letters,
Luo, D., Chen, Z., Hu, K., Zhao, Z., Hur, V. M., and Clark. B. K. Gauge invariant autoregresssve neural networks for quantum lattice models, 2021b. Utl.
Luo, D.. Chen, Z., Carrasquilla, J., and Clark, B. K. Autoregressive neural network for simulating open quantum systems via probabilistic formulation. Phys. Rev. Lett., 128:090501, Feb 2022a. doi: 10. 1103/PhysRevI.ett 22.009501. htt./attss///ikk.eee http:///wm....p...o//oo///...... http:////......1ooo///e/eeeeeea 190500
Luo, D., Yuan, S., Stokes, J., and Clark, B. Gauge equivariant neural networks for 1d u (1) gauge theory simulations in hamiltonian formulation. arXiv preprint arXiv:221 2022b.
Marshall, W. Antiferommanetism. Proceedings of the Royal Society of London. Series A.
Mathematical and Physical Sciences, 232((188):48 1955. ISSN 00804630. URL http:
//www.jstor org/ htg/staable/9963)
Nagy, A. and Savona, V. Variational quantum monte carlo method with a neural-network ansatz for open quantum systems. Phys. Rev. Letr. 122:250501, Jun 2019. doi: 10. 10.1103/hhsseee.ett.. 250501. URL https:: //jornas aps orggprr//// httrr//bbstactt/// 1103 11t.//1seeee/..ettttt.t.. 122 250501
Nomura, Y. and Imada, M. Dirac-type nodal spin liquid revealed by refined quantum many-body solver using netrol-network wave function, correlation ratio, and level spectroscopy. Physical Review X, 11)3) aug 2021. doi: 10.1103/physttexx111.333 physrexp 11. 031034
URL
Oseledets, I. V. Tensor-train decompasition. SIAM Journal on Scientific Computing, 33(5):2295-2317 2011. doi: 10. 10./01999777.286. URL //do: 11t1//10011170007/52222000
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, gegh-performncce deep learning library. In Advances in neural information processing systems, pp. 8026-8037, 2019.
Pfau, D., Spencer, J. S., Matthews, A. G. D. G., and Foulkes, W. M. Ab initio solution of the many- electron schrödinger equation with deep neural networks. Phys. Rev. Research, 2:033429, Sep 2020. URL
doi: 10. hwss:/hyseeeesseerre2.0333333330
aps org httg://rreeaach///ooo
abstract/10. 1103//hyyReeReesarcc. 2. 2.333429
Preskill, J. Quantum computing 40 years later. arXiv preprint arXiv:2106. (Tt::2006..0222..0 2021.
Reuther, A., Kepner, J., Byun, C., Samsi, S., Arcand, W., Bestor, D., Bergeron, B., Gadepally, V., Houle, M., Hubbell, M., Jones, M., Klein, A., Milechin, L., Mullen, J., Prout, A., Rosa, A., Yee, C., and Michaleas, P. Interactive supercomputing on 40,000 cores for machine learning and data analysis. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pp. 1-6. IEEE, 2018.
26

---

Schmitt, M. and Heyl, M. Quantum many-body dynamics in two dimensions with artificial neural net- works. Physical Review Letters, 125(10), Sep 2020. ISSN 1079-7114 doi: 10.1 h0.pp0///yyyeellet...k 100503. URL http:://oorrral.........o//p//pp https//accccat.110o///eeeeeeeaaa 125 100503
Sharir, O., Levine, Y., Wies, N., Carleo, G., and Shashua, A. Deep autoregressive models for the efficient variational simulation of many-body quantum systems. Physical Review Letters, 124(2), jan 2020. doi: 10. 103ppyysrevllt.. 124.020503. URL https:://oi..rrgge oro.:/.113/001.3/ 2Fphysrevlett. 124. 020503
Sharir, O., Shashua, A., and Carleo, G. Neural tensor contractions and the expressive power of deep neural quantum states, 2021.
Van den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is all you need. Advances in neural information processing systems, 30: 5998 2017.
Verstraete, F. and Cirac, J. Renormalization algorithms for quantum many body systems in two and higher dimensions, 2004. URL httth:////axxii http//avivov./////////. cond mat mat/0407066
Vicentini, F., Biella, A., Regnault, N., and Ciuti, C. Variational neural-network ansatz for steady states in open quantum systems. Physical Review Letters, 122(55), Jun 2019. ISSN 1079-7114. doi: 10. 10.1100/ppyseeeettt1222..2. 122.550033. URLLpttts://joornalll aps org/ 1103/PhySRevLett. 122. 250503
Vicentini, F., Hofmann, D., Szabó, A., Wu, D., Roth, C., Giuliani, C., Pescia, G., Nys, J., Vargas- Calderón, V., Astrakhantsev, N., and Carleo, G. NetKet 3: Machine learning toolbox for many-body
quantum systems. SciPost Physics Codebases, aug 2022. doi: 1021466kscipopphyssodeeb.. URL
https://doi.oo org/10 21 2166ttFFscipottphyyycddes
Vidal, G. Efficient classical simulation of slightly entangled quantum computations. Physical Review Letters, 91(14), oct 2003. doi: 10. 11100//pyyrreeeeel91 147902. URL https://aoi.orrg/1/
1103/2Fphysrevllett..s wt..:/eetettttt................. 9411
Vidal, G. Efficient simulation of one-dimensional quantum many-body systems. Physical Review
Letters, 93(4), jul 2004. doi: 10. 90 93.045502. URL httpss////ddo oroo/i..rrg/1... 1103%
2Fphysrevlett.99 940002
Vidal, G. Entanglement renormalization. Phys. Rev. Lenl, 99:220405, Nov 2007. doi: httL:htps:://iiie
10. 111.:PPhyceeeLc..9922000000 99. 220405
Vieijra, T., Casert, C., Nys. J., De Neve, W., Haegeman, J., Ryckebusch, J., and Verstraete, F. Restricted boltzmann machines for quantum states with non-abor non-abelian or anyonic symmetries. Physical Review Letters. 124(9), Mar 2020. ISSN 1079-7114. doi: 10.1:11.111000 1tt1100/ppysseeeee..2..00000
URL attps://journal.. aps wpt.aaag/p.///assrrttt//e////e/e http://ateee...1o1///ee/eeeetttt 124. 097201
Westerhout, T., Astrakhantsev, N., Tikhonov, S., Katsnelson, M. I., and Bagrov, General- ization properties of neural network approximations to frustrated magnet ground states. Nature Communicationss (11)::159, Mar 2020. ISSN 2041-1723. doi: 1038//414467-099 15402-w. 15402-w
URL httph:tt:/ //doi org/ 1038/ s4 1111467--200 -
White, S. R. Density matrix formulation for quantum renormalization groups. Physical review letters, 69619)22863, 1992.
Wu, D., Rossi, R., Vicentini, F., and Carleo, G. From tensor network quantum states to tensorial recurrent neural networks. arXiv preprint arXiv:2206. 100006//3333 /2363, 2022.
Yoshioka, N. and Hamazaki, R. Constructing neural stationary states for open quantum many- body systems. Phys. Rev. B, 99:214306, Jun 2019. doi: 10.1103/PhysRev3.99222 99..2299..999999966 URL. https::/jjounie aps.org/ httggg/bbbbssrreeee wwwp/bbstaatt/o.oooo httpe/aaaee.oo.111o////eere////e 99.214306
27


Reviews:
[{"review_id": "FsECmZEM0C", "reviewer": "Reviewer_2bnN", "summary": "This paper proposes Autoregressive Neural TensorNet (ANTN): a novel blend between Matrix Product States (MPS) and AutoRegressive Neural Network. Theoretically, this paper shows that ANTN parameterizes normalized wavefunctions, allows for exact sampling, generalizes the expressivity of tensor networks and autoregressive neural networks, and inherits a variety of symmetries from autoregressive neural networks. Experimentally, this paper shows that ANTN is more efficient than MPS and better biased than ARNN in quantum state learning, and show that ANTN outperforms MPS and ARNN in approximating the ground state of the 2D J1-J2 Heisenberg model with systems sizes up to 12x12. \n", "questions": "Multiple references to the “Inductive bias of the physics structure of MPS” is pretty vague. When you discuss initializing the MPS with the optimized DMRG results of the same bond dimension that kind of makes sense, but then you could also initialize the ARNN with an optimized physics related representation, so I don’t see an advantage of MPS in this “optimzed initialization” regard. As for randomly initialized MPS vs ARNN, I don’t see that a-priori the former is better suited than the latter. In particular, all of the relevant physical symmetries of ATNN listed in section 5.3 seem to be inherited from ARNN and not from MPS. On the other hand, the experiments of section 6.1 mention “wavefunctions with local or quasi-local sign structures” and “shallow random circuit to mimic the short-range interaction and generate states with sign structures”. NeurIPS is ultimately a non-physics conference and such statements are hard to decipher for me as well as for broader readership -- what exactly is the task you ran? how exactly is it related to interesting physical systems? . Especially given the important role of this point in the paper’s main argument regarding the benefits of MPS, I find it to be insufficiently explained. \nCan you please elaborate and be more specific on what MPS inductive biases contribute to the physics suitability of ANTN? \n\nFurther issues:\nWhat is h_dim? I don't see it defined.\nLine 63: state-of-the-arts \nLine 153: missing  punctuation after “unique”\nLine 311: especially pass -> especially past \n", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed architecture is novel, and importantly simple and intuitive therefore conveying a potential of enjoying “the best of both worlds” – an MPS above a very expressive AR neural network that creates a strong input representation. The core observation that one can construct such an architecture while retaining the ability to perform exact efficient sampling, is strong. \nThe experimental results are non-trivial. The clean experiments in 6.1 tell an interesting story of efficiency with respect to MPS and bias with respect to ARNN. In section 6.2, it is an impressive achievement to consistently surpass ARNN on the one hand and MPS on the other on the challenging J1J2 task. Moreover, the positive trend with system size is promising. Overall, the experimental results in this paper bring an appetite to try this new architecture on larger systems and on more problems.     \n\n", "weaknesses": "Overall, I find this paper to be not well structured. My main disappointment with the writing is the lack of details on the experimental part of the paper. It is by far more convincing than the theoretical part, but unfortunately does not receive enough space and details.\n\nFor example, it is unclear to me what ANTN architecture was used for the experiments. Section 4 mentions the use of both PixelCNN and Transformer as the ARNNs underlying the ANTN. However, both tables 1 and 2 do not mention which variant was used, do not show results for both variants, and mention only the PixelCNN baseline. A long look at the experimental details in the paper body and in the appendix did not really help. It makes it very hard to assess the strength of the results when it is not clear which ARNN was used. Moreover, I am missing external performance references in order to assess the strength of the results. The field of many body quantum simulations has matured to the point where I expect new architectures to compare to external SOTA results; otherwise, it is not easy to judge results that are given relatively to baselines implemented by the authors, especially in the haziness / opacity of the current detailing of the experimental setup. Comparison to existing results on J1J2 will help convince me that the reported gains are meaningful. Besides that, much more details on the experimental setup on which the results were attained are required for this potentially important experimental outcome to be verifiable.           \n\nTheoretically, Theorem 1 is important; Section 5.2 is disappointing and does not motivate using ANTN from an expressivity standpoint; and section 5.3 is not tied to the specific experiments conducted later on is used in the remainder in the paper (so though possibly interesting it does really contribute to the practical case advocating the use of ANTN). \n\nElaborating, Theorem 5.2 does not provide an expressivity result that allows for reasoning regarding the benefits of ANTN. Specifically, it isn’t clear **by how much** ANTN is stronger than ARNN or MPS. The simple theoretical arguments provided do not make it clear, for example, whether ANTN is stronger than adding one more layer to the ARNN. Therefore, from an expressivity standpoint, the authors did not make a convincing case for the benefits of ANTN. Theorem 5.3 simply mentions the strength of ARNN as shown in other works, but does not contribute to the case of why ANTN is better than the existing ARNN from an expressivity standpoint. An experiment of ARNN with one more layer may very well show that the gains can also be pretty easily achieved with an existing architecture, which would be disappointing. (as a remark it isn’t clear how many layers are in the experiments on ARNN, the appendix gives some details but it isn’t clear if Transformer of PixelCNN was used).\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We appreciate the reviewer's acknowledgment that our work is theoretically novel and experimentally nontrivial. The authors apologize that due to the character limit, we are unable to respond to all the questions and have to omit some contents. The authors would provide additional explanations in the follow-up discussions.**\n\n**As the reviewer points out, our proposed architecture ANTN gives rise to a strong input representation by integrating MPS with expressive ARNN and maintains efficient sampling. Meanwhile, we have demonstrated clear advantages over MPS and ARNN with various experiments.**\n\n>**Weaknesses:**\n\n>Overall, I find this paper to be not well structured. My main disappointment with the writing is the lack of details on the experimental part of the paper. It is by far more convincing than the theoretical part, but unfortunately does not receive .....\n  \n**The authors apologize for the oversight and thank the reviewer for nice suggestions on improving our writing. We will add relevant information explaining the different architectures used in different experiments in the captions as suggested. In terms of the architecture choice, all the experiments in Section 6.1 uses the Transformer and all the experiments in Section 6.2 uses the PixelCNN. The reason for such a choice is that the quantum state learning setup in Section 6.1 has a 1D structure, which makes Transformer a natural choice for the underlying ARNN, whereas in Section 6.2, the $J_1$-$J_2$ model has a 2D structure, making pixelCNN a better choice. This also makes a more fair comparison with the pure pixelCNN baseline.**\n\n>Moreover, I am missing external performance references ......\n\n**Thanks for the reviewers' suggestion and agree that comparisons to external SOTA results are necessary. In fact, we compared our results with three previous SOTA results in the original paper: RBM from NetKet, recent MPBS result, and standard DMRG result with $\\chi=1024$. In Table 1, we compared our results with the RMB results for $8\\times8$ system using the existing NetKet package which has been one of the SOTA packages for many models in quantum many-body systems. In addition, we also compared with one of the recent SOTA results using MPBS at $J_2=0.5$ (written in paper line 314), which is worse than our results. Moreover, DMRG has been the SOTA in this field for decades, especially under open boundary conditions which takes DMRG's advantage. Here, we benchmarked against DMRG with $\\chi=1024$, which contains $10^8$ parameters, 2 orders of magnitude larger than our model. In fact, the previous SOTA MPBS did not surpass DMRG with $\\chi=1024$, and it can be viewed as a milestone that our results go beyond DMRG for large systems. We will adjust the writing to make these comparisons more apparent. To clarify the details of the setup, the pixelCNN used in this work follows the implementation as mentioned in the main paper, which is implemented according to the gated pixelCNN, but slightly modified for quantum wave functions. Then, the neural networks are trained using the Adam optimizer with gradients calculated from the variational Monte Carlo algorithm (Appendix A.2), and training details are listed in Appendix C. As the reviewer suggests, we will also include additional implementation details on the experimental setup in the updated version to make sure the outcome is verifiable.**\n\n>Theoretically, Theorem 1 is important......\n\n**The authors thank the reviewer for the comment and the recognition of the importance of Theorem 1. The authors apologize that Section 5.2 writing may be too brief to highlight the expressivity of ANTN over both MPS and ARNN as an important motivation. To clarify, we have now provided detailed proof which shows that ANTN cannot be efficiently represented by either TN or ARNN to highlight this important result. (See global response.)**\n \n>and section 5.3 is not tied to the specific experiments ......\n  \n**Thanks for the reviewer's comment. Section 5.3 mainly serves as an important theoretical result that the symmetries that can be implemented in ARNN can also be implemented in ANTN. As implementing symmetries on ARNN is already non-trivial, with many researchers working in this direction, explicitly developing ANTN with symmetries can be an important and promising direction for future work.**\n\n>Elaborating, Theorem 5.2 does not provide ......\n\n**The authors thank the reviewer for the comment. We have provided an updated proof above, where it is shown that a general ANTN can be written as a TN or ARNN with an exponential (in system size) number of parameters The proof demonstrates that ANTN has stronger expressivity over ARNN and TN (see global response).**\n\n>Theorem 5.3 simply mentions......\n\n**Indeed, Theorem 5.3 is a theoretical result that shows ANTN has the nice feature that it can inherit the symmetries from ARNN. The motivation here is not to show ANTN is more expressive than ARNN, but how it can non-trivially incorporate various symmetries even with tensor network on top of ARNN.**\n\n>An experiment of ARNN......\n\n**The authors thank the reviewer for the suggestion of adding more layers to the ARNN. In the global response, the authors attached additional benchmarking results for ARNN with more layers, where ARNN still lacks behind ANTN, which further supports the updated proof of Theorem 5.2. Our results clearly demonstrate that ANTN is more expressive than ARNN both theoretically and experimentally (see global response).**\n\n>(as a remark ......\n\n**The authors apologize for the confusion. All the experiments in Section 6.1 use Transformer with 2 layers, and all the experiments in Section 6.2 use PixelCNN with 7 layers. The authors will update the caption to indicate the details of neural network architectures.**\n\n>**Questions:** \n\n**The authors will answer the questions in the follow-up discussions due to the character limit.**\n"}, {"review_id": "upZvxUpNdu", "reviewer": "Reviewer_Vtcz", "summary": "In this work, the authors introduce ANTN (Autoregressive Neural Tensor Net) as a novel architecture that combines features from both ARNN and Tensor Networks for simulating challenging quantum many-body systems. The main objective of this research is to utilize an ARNN-type network to generate a conditional wavefunction tensor rather than a direct conditional wavefunction, as commonly done in standard ARNNs. This latter can be interpreted as a probability density. Figure 1 provides a clear visualization of the idea.\n\nBy integrating ARNN and TN within this framework, the authors aim to leverage the advantages of both methods, thereby achieving enhanced expressivity and a stronger adherence to physics inductive bias. To demonstrate the superiority of their approach, the authors present several theorems highlighting its advantages. While detailed proofs are available in the appendix, the main text often provides concise explanations, which significantly support the flow of the manuscript and complement the discussed results.\n\nThe method's performance is initially evaluated by benchmarking its ability to learn 2-qubit Bell random states and real-valued shallow random circuits. Finally, the authors compare their ANTN approach to state-of-the-art methods in solving the ground state of a 2D $J_1-J_2$ Heisenberg model with open boundary conditions. This particular problem is known to be challenging due to its phase diagram exhibiting at least three distinct phases as $J_1$ and $J_2$ are varied. The results presented in Table 1 and Table 2 demonstrate that the ANTN method consistently outperforms existing methods across different system sizes and parameter choices for $J_2$.", "questions": "- How is the ANTN model trained? While intuitions are provided in Sections 3.2 and 3.3 (with additional details in the appendix), a more specific discussion would enhance understanding of the proposed framework. Additionally, it would be beneficial to explore the limitations of training ANTN. Is the training process significantly more challenging compared to plain ARNN? Can you provide an estimate of the computational time required to train a network for a specific downstream task?\n- The authors frequently mention that ARNNs lack the physics prior of the system under study. However, it raises the question of the extent to which this holds true. As correctly stated by the authors, inductive physics biases, such as symmetries, can actually be incorporated into pure ARNN networks to incorporate the desired physics prior. Could the authors provide some comments on this? \n- I wonder whether it would be interesting to replicate the analysis in Table 1 and Table 2 by fixing $J_2$ and modifying $J_1$ instead. Would substantial differences in the results be expected? This question arises because, as I understand it, $J_1$ modulates the interaction between nearest neighbors, potentially leading to collective behavior that may be more challenging to capture.\n- How do runtime and memory requirements scale in the experiments presented in Section 6.2? The insightful complexity analysis discussed in Section 5 suggests that it would be valuable to empirically visualize these factors through numerical experiments.\n- Is there a specific reason why DGRM results are not shown in Table 1, while RBM results are missing in Table 2?\n- In Figure 3, it appears that the DGRM method outperforms ANTN for medium-sized systems and increasing values of $J_2$. Since the bond dimension is fixed for both methods, do you have any intuition regarding how this plot would change as the bond dimension varies? Intuitively, one might expect that having a larger bond state for DGRM would improve its performance, particularly for larger system sizes. It would be intriguing to perform a detailed scaling analysis as a function of the bond dimension, acknowledging the potential increase in complexity and the enhanced capability of TN methods to capture more features, potentially leading to improved results.\n- In the broader impact section, the authors suggest that the ANTN approach may be applicable to other standard machine learning tasks. However, in such tasks, like image generation, for instance, there might not be strong physics prior to leverage using the TN representation. It would be interesting to hear the authors' perspectives on the benefits of applying their framework to standard machine learning benchmarks and how it could be advantageous in those scenarios.\n- Can this idea of combining TN with a generative type of neural networks be extended to other generative models?", "limitations": "In addition to the concerns mentioned in the weaknesses and questions sections, which implicitly highlight some limitations of the current manuscript, I believe that the primary limitation of this work may lie in its scalability and the training difficulties of ANTN compared to MPS. Although I am not an MPS expert, I am aware that sampling (and training) for an ARNN can be slow and computationally expensive, particularly when dealing with larger systems. While runtime may also pose a challenge for MPS, as increasing the bond dimension can become a computational bottleneck, I am uncertain whether this would be a minor issue compared to training, and sampling from, an ANTN when considering extremely large systems.\n\nA few minor comments: \n- In the introduction, it would be good to already have the relevant references being cited when a field/topic is firstly mentioned and introduced. \n- The Acronyms MERA and PEPS are never introduced in their extensive form. Would be good to do so when they are mentioned for the first time in line 72 of page 2. \n- Line 271 page 7:  previous -> previously\n- Line 287 page 7:  comes -> come\n- Line 690 page 19: staring -> starting\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is scientifically sound and well-written, with a comprehensive and meticulous literature review that covers all relevant works in the field.\n- The paper's structure is highly effective, resulting in a smooth and enjoyable reading experience.\n- Section 3 stands out as it provides a thorough overview of the essential components required to comprehend the more theoretical discussion starting from Section 4. This inclusion enhances the accessibility of the paper, making it approachable even for readers who are not thoroughly familiar with the field, such as myself.\n- The concept of combining the inherent flexibility of ARNN with the strengths of TN within a single framework is undeniably compelling.", "weaknesses": "- The manuscript frequently mentions the concept of \"expressivity,\" but it is never explicitly defined or explained. It would greatly benefit the readers to dedicate a paragraph to clarify its meaning and highlight its significance, thereby implicitly assessing the advantages of the proposed approach. Specifically, Theorem 5.2 is not entirely clear to me. It would be helpful to have a clear definition of expressivity to shed light on this point.\n- The paper repeatedly refers to the \"sign structure\" problem, but this issue is never explicitly defined or explained. While referencing relevant literature is helpful, the paper should strive to be self-contained when discussing this critical aspect. Given its importance in evaluating the efficacy of the proposed method in the experimental section, a paragraph discussing this concept would be valuable for readers less familiar with it.\n- Although ANTN combines properties from both ARNN and TN, being a parametrized neural network, it is expected that ANTN also requires training. However, the main text does not discuss this in great detail. A comprehensive discussion on training ANTN in the main text is essential to provide a complete understanding of the methodology.\n- The manuscript lacks experiments that benchmark the runtime and memory complexity. It would be beneficial to compare ANTN with state-of-the-art TN methods in terms of these metrics. Such comparisons would enhance the overall evaluation of ANTN and its performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Thanks for the appreciation of our work. We apologize that due to the character limit, we have to simply the response. We will provide additional explanations in the follow-up discussion.**\n\n>**Weaknesses:**\n\n>The manuscript frequently mentions the concept of \"expressivity,\" .....\n\n**Thanks for the comment. Indeed, there could be different definitions of \"expressivity\" based on different perspectives. Two typical important ideas of expressivity are related to entanglement and sign structure, which could be both incorporated into ANTN. In particular, the degree of entanglement is the bottleneck for MPS since it is designed to represent low entanglement quantum wave functions. The sign structure (or more generally the phase structure) stems from the fact that quantum wave function can have complex-number output, which requires more design of the neural network architecture. In this manuscript, we choose to define \"expressivity\" in a general way as a neural network's ability to represent (generic) quantum wave functions (potentially limiting the number of parameters). We also updated the proof of Theorem 5.2 in the global response. We have included the above discussion on expressivity in the updated paper.**\n\n>The paper repeatedly refers to the \"sign structure\" problem.....\n \n**Thanks for the suggestion. The sign structure (or more generally the phase structure) comes from the fact that quantum wave functions are L2-normalized functions. In other words, a quantum wave function can be viewed as a probability distribution with complex-valued phases ($\\psi(x)=\\sqrt{p(x)}\\exp{i\\theta(x)}$). For real-valued Hamiltonian, such phases are reduced to signs of $+1$ and $-1$ ($\\theta(x)=0$ or $\\pi$) (i.e. the sign structure). To capture such a structure, the neural network requires additional design to go beyond a probability representation. When the sign structure is known, the quantum wave function can be reduced to the probability distribution and can be solved more easily. However, such a structure is only analytically known for a limited number of models, and in the worst case it can be very random binary number distribution of $+1$ and $-1$ that are hard to represent. In this work, ANTN can learn this sign structure more efficiently due to the inductive bias from the MPS. We have included more discussion on the \"sign structure\" problem in the updated paper.**\n\n>Although ANTN combines properties......\n  \n**The ANTN is trained using standard gradient-based optimizers (Adam to be specific) with the gradient derived in Appendix A.2. We have added this in the updated paper to be clearer to the readers.**\n\n>The manuscript lacks experiments......\n\n**Thanks for the comment. Yes, it would be very beneficial to provide benchmarks regarding runtime and memory complexity. A theoretical analysis is provided in Section 5.1. And additional experimental runtime results are included in the global response. In practice, all of the experiments expect DMRG in this work can be efficiently run on NVIDIA V100 GPU with 32 GB memory using at least 2000 samples in parallel. The DMRG algorithm with a bond dimension of 1024 requires a substantial amount of memory (with $10^8$ parameters) and the optimization is hard to be efficiently parallelized. Therefore it is run on CPU (which is standard for the DMRG algorithm.) We have included these discussions in the updated paper.**\n\n>**Questions:**\n\n>How is the ANTN model trained...\n\n**In Sections 3.2 and 3.3, we provide the training loss functions. For quantum state learning, the gradient is simply the gradient of the loss function. For the variation Monte Carlo algorithm, the gradient is similar to the policy gradient of the setup in reinforcement learning, which we derive in Appendix A.2. Then, Adam optimizer was used to train the neural network, with details written in Appendix C. We hope this answers the reviewer's question.**\n\n>The authors frequently mention that ARNNs lack...\n\n**As discussed previously, there are different perspectives of expressivity and inductive bias of a quantum wave function. While symmetry could be Incorporated into ARNN, there are other important inductive bias such as low entanglement and sign structure, which may not be easy to capture by ARNN. Indeed, it is known that MPS is better at capturing low entanglement (such as low dimension system ground state) and sign structure. Hence, by bridging MPS and ARNN together into ANTN, ANTN is equipped with more inductive bias, not only multiply symmetries from ARNN, but also sign structure and various entanglement degree (both low entanglement from MPS and high entanglement from ARNN).**\n\n>I wonder whether ...\n\n**Thanks for the comment. Since finding the ground state corresponds to finding the smallest eigenvalue of the Hamiltonian matrix, multiplying the matrix by a constant does not change the problem (it only rescales the eigenvalue). Hence, the only physically relevant quantity is the ratio $J_2/J_1$.**\n\n>How do runtime......\n  \n**Already answered above**\n\n>Is there a specific reason why DGRM .. \n\n>In Figure 3, it appears that the DGRM..\n \n>In the broader impact section.. \n\n**We will answer these questions in the follow-up discussion due to the character limit.**\n  \n>**Limitations:**\n\n>In addition to...\n\n**The authors understand the concerns that the reviewer raises. However, ARNN (and ANTN) are not slower compared to MPS, especially with the ability of exact sampling. In fact, because ARNN and ANTN can be trained more efficiently on GPU compared to MPS (with DMRG algorithm) due to the lower memory requirement and easier parallelization, ARNN and ANTN can be faster in practice with runtime details included in the global response.**\n\n>**A few minor comments:**\n\n**Thanks for the suggestions and corrections, and we have updated the manuscript accordingly.**"}, {"review_id": "DY7VpwOzxc", "reviewer": "Reviewer_dUKK", "summary": "The paper proposes generic Autoregressive Neural TensorNet (ANTN), for quantum many-body simulation. In order to achieve high expressivity, sign structure preserving, physics inductive bias, accurate sampling, and symmetry preservation, ANTN combines tensor networks and autoregressive neural networks. The author demonstrates the improved performance of ANTN on quantum state learning and learning ground state energy.", "questions": "Please see the weaknesses.", "limitations": "The author does describe the limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed architecture absorbs the advantages of two state-of-the-art methods i.e., tensor networks and autoregressive neural networks. The idea is straightforward but novel and interesting.\n2. This paper well illustrates its differences and advantages over tensor network based and neural network quantum state based methods for quantum many body simulation.\n3. It provides a comprehensive theoretical analysis and the experimental results show that the proposed method can still maintain good expressive ability with fewer bond dimensions.", "weaknesses": "1. The author solely compares and reports the bond dimensions taken up by the ANTN and MPS/DMRG approaches in Fig 1, Tab 1 and 2. Given that the first half of the ANTN is a sophisticated autoregressive neural network, reviewer concerns about that the author may evade a fact that whether the order of magnitude of the parameters of the autoregressive neural network could surpass the bond dimension of the following tensor network. As a consequence, the existence of the NN parameters will cancel out the efficiency brought on by the reduction in bond dimension.\n2. Although the combination of autoregressive neural network and tensor network to build a new neural network quantum state method that absorbs the advantages of both is a delightful idea, reviewers are worried that such method will also inherit the disadvantages of both. For example, tensor networks are likely to be very tricky to characterize a highly entangled quantum state [1]. At the same time, traditional tensor networks may also perform poorly in learning the ground state energy of a long-range Hamiltonian [2,3]. These problems are also an important reason why some researches abandon the tensor network and embrace the neural network-based quantum states. The reviewer is pleased if the authors could clarify these points, and it would be preferable if they could provide some explanations and comparisons in the experimental or theoretical part.\n\n[1] Deng D L, Li X, Sarma S D. Quantum entanglement in neural network states[J]. Physical Review X, 2017, 7(2): 021021.\n\n[2] Glasser I, Pancotti N, August M, et al. Neural-network quantum states, string-bond states, and chiral topological states[J]. Physical Review X, 2018, 8(1): 011006.\n\n[3] Xiao T, Huang J, Li H, et al. Intelligent certification for quantum simulators via machine learning[J]. npj Quantum Information, 2022, 8(1): 138.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Summary:**\n>The paper proposes generic Autoregressive Neural TensorNet (ANTN), for quantum many-body simulation. In order to achieve high expressivity, sign structure preserving, physics inductive bias, accurate sampling, and symmetry preservation, ANTN combines tensor networks and autoregressive neural networks. The author demonstrates the improved performance of ANTN on quantum state learning and learning ground state energy.\n\n>**Strengths:**\n>The proposed architecture absorbs the advantages of two state-of-the-art methods i.e., tensor networks and autoregressive neural networks. The idea is straightforward but novel and interesting.\n>This paper well illustrates its differences and advantages over tensor network based and neural network quantum state based methods for quantum many body simulation.\n>It provides a comprehensive theoretical analysis and the experimental results show that the proposed method can still maintain good expressive ability with fewer bond dimensions.\n\n**The authors thank the reviewer for the appreciation of this work.**\n\n>**Weaknesses:**\n>The author solely compares and reports the bond dimensions taken up by the ANTN and MPS/DMRG approaches in Fig 1, Tab 1 and 2. Given that the first half of the ANTN is a sophisticated autoregressive neural network, reviewer concerns about that the author may evade a fact that whether the order of magnitude of the parameters of the autoregressive neural network could surpass the bond dimension of the following tensor network. As a consequence, the existence of the NN parameters will cancel out the efficiency brought on by the reduction in bond dimension.\n\n**The authors thank the reviewer for bringing up the question regarding the number of parameters for the MPS, ARNN and ANTN. The authors would like to clarify that for the experiments that we conducted, MPS with the bond dimension of $1024$ actually has the most parameters ($10^8$), given that the number of parameters for MPS scales as $N\\chi^2$ with $N$ the system size and $\\chi$ the bond dimension. The ARNN has fewer parameters ($5.0\\times 10^5$) compared to the ANTN ($10^6$), because we fixed the shape of the underlying ARNN when conducting the experiment. In the global response, we list out in detail the number of parameters and runtimes for the experiments. We find that our ANTN outperforms both ARNN and MPS even though ANTN has a comparable number of parameters as ARNN and much fewer parameters than MPS.**\n\n>Although the combination of autoregressive neural network and tensor network to build a new neural network quantum state method that absorbs the advantages of both is a delightful idea, reviewers are worried that such method will also inherit the disadvantages of both. For example, tensor networks are likely to be very tricky to characterize a highly entangled quantum state [1]. At the same time, traditional tensor networks may also perform poorly in learning the ground state energy of a long-range Hamiltonian [2,3]. These problems are also an important reason why some researches abandon the tensor network and embrace the neural network-based quantum states. The reviewer is pleased if the authors could clarify these points, and it would be preferable if they could provide some explanations and comparisons in the experimental or theoretical part.\n\n**Thanks for the question. The authors understand the reviewer's concern about combining ARNN and MPS. However, as shown in Theorem 5.2, the ANTN has generalized expressivity over both ARNN and MPS. In fact, both ARNN and MPS are special cases of ANTN. Therefore, ANTN does not inherit the disadvantages of both ARNN and MPS.  Moreover, we updated Theorem 5.2 in the global response, where we explicitly proved that neither ARNN nor MPS can efficiently represent ANTN. In addition, ANTN inherits the flexible sign structure from tensor network, which plays an important role for optimization. As we show in the new experiments in the one-page response, our ANTN achieves the best performance in those cases compared to RBM and ARNN with both sign rule and non-sign rule as well as MPS. Our new theoretical and experimental results strongly support that ANTN takes advantages of both MPS and ARNN. Regarding the specific example that the reviewer pointed out. ANTN is able to capture the entanglement very well. As we demonstrated in Figure 2(a), the ANTN achieves the same fidelity as the ARNN for random Bell states (maximally entangled). In an ANTN, the MPS and ARNN divide the work of representing a quantum state, with the MPS specializing in representing the quasi-local sign structure (Figure 2 (b)), and the ARNN focusing on representing the entanglement. Therefore, the ANTN actually inherits the advantages of both ARNN and MPS, which is exactly the reason that ANTN surpasses both MPS and ARNN in our experiments. We hope this explanation resolves the reviewer's concern.**\n\n>**Questions:**\n>Please see the weaknesses.\n\n>**Limitations:**\n>The author does describe the limitations.\n"}, {"review_id": "duPCTyHhun", "reviewer": "Reviewer_wT6D", "summary": "The paper approaches the problem of designing wave function ansatz for the ground state estimation of the Heisenberg model. This problem introduces several challenges including the rich parameterization of the wave function, sampling of the states from the parameterized wave function, and incorporating symmetries into the model. There are two main approaches to this problem: autoregressive neural networks and tensor networks.\nThe former allows for fast sampling and incorporating symmetries. The latter incorporates problem-specific inductive bias.\n\nThe authors propose an autoregressive model which incorporates tensor networks thus, hopefully, inheriting the best from both approaches. Shortly, the proposed model at every step of auto-regression yields tensors instead of scalars. This allows for a richer class of wave function representations while enjoying the properties of auto-regressive models such as exact sampling.\n\nThe authors study the proposed model both theoretically and empirically. The theoretical contribution of the paper is the proof of exact sampling and invariance properties. For the empirical study, the authors demonstrate the performance of the proposed ansatz for quantum state learning (fidelity maximization) and for the Heisenberg model (variational Monte Carlo).", "questions": "I have no questions for the authors.", "limitations": "The limitations are adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I would say that the paper corresponds to all NeurIPS standards:\n- The paper approaches an important problem with clear applications. \n- The proposed method is novel and sound.\n- The experimental study is extensive.\n- It is well-written.", "weaknesses": "My only concern regarding the empirical study is the comparison in the number of parameters. Indeed, the main goal of the paper is to propose a richer family of models, where we would expect better approximation properties for the same number of parameters as in concurrent approaches. I believe that this can be better illustrated with experiments by reporting the number of parameters along with the energy values. Right now, the authors fix the shapes for the tensor network and then add a neural network \"on top\" of it. Therefore, it is not very clear how to compare the models. Maybe adding a plot (e.g., number of parameters vs energy) would improve the readability of the experimental section.\n\nOther comments:\n- line 127. Typo (articles).\n- line 144. Typo (the last index of $\\alpha$).\n- line 153. Typo (many MPS represent)\n- line 160. Typo (up to qubit $j$). Also, the notation with the summation over $\\alpha$ is not very clear here.\n- line 271. Typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Summary:**\n> The paper approaches the problem of designing wave function ansatz for the ground state estimation of the Heisenberg model. This problem introduces several challenges including the rich parameterization of the wave function, sampling of the states from the parameterized wave function, and incorporating symmetries into the model. There are two main approaches to this problem: autoregressive neural networks and tensor networks. The former allows for fast sampling and incorporating symmetries. The latter incorporates problem-specific inductive bias.\n\n> The authors propose an autoregressive model which incorporates tensor networks thus, hopefully, inheriting the best from both approaches. Shortly, the proposed model at every step of auto-regression yields tensors instead of scalars. This allows for a richer class of wave function representations while enjoying the properties of auto-regressive models such as exact sampling.\n\n> The authors study the proposed model both theoretically and empirically. The theoretical contribution of the paper is the proof of exact sampling and invariance properties. For the empirical study, the authors demonstrate the performance of the proposed ansatz for quantum state learning (fidelity maximization) and for the Heisenberg model (variational Monte Carlo).\n\n> **Strengths:**\n> I would say that the paper corresponds to all NeurIPS standards:\n\n> The paper approaches an important problem with clear applications.\n> The proposed method is novel and sound.\n> The experimental study is extensive.\n> It is well-written.\n\n**The authors thank the reviewer for the appreciation of this work.**\n\n> **Weaknesses:**\n> My only concern regarding the empirical study is the comparison in the number of parameters. Indeed, the main goal of the paper is to propose a richer family of models, where we would expect better approximation properties for the same number of parameters as in concurrent approaches. I believe that this can be better illustrated with experiments by reporting the number of parameters along with the energy values. Right now, the authors fix the shapes for the tensor network and then add a neural network \"on top\" of it. Therefore, it is not very clear how to compare the models. Maybe adding a plot (e.g., number of parameters vs energy) would improve the readability of the experimental section.\n\n**Thanks for the comment. While it is hard to set the number of parameters of all the ansatz exactly the same, to address the question, we have conducted additional experiments with additional parameters in ARNN, and list the energies, number of parameters, and runtimes for the experiments conducted in this work in the tables and the figures in the global response. We note that our ANTN actually has a comparable number of parameters to ARNN and has much fewer number of parameters than bond 1024 DMRG, while at the same time achieving better performance than all of them. The results further confirm that our ANTN achieves efficient representation and optimization by integrating tensor network and ARNN.**\n\n\n> **Other comments:**\n\n>line 127. Typo (articles).\n>line 144. Typo (the last index of $\\alpha$).\n>line 153. Typo (many MPS represent)\n>line 160. Typo (up to qubit $j$). Also, the notation with the summation over $\\alpha$ is not very clear here.\n>line 271. Typo.\n\n**The authors thank the reviewer for pointing out the typos and we have fixed them in the updated version of the paper.**\n\n\n>**Questions:**\n>I have no questions for the authors.\n\n>**Limitations:**\n>The limitations are adequately addressed."}]

Author rebuttal:
**We would like to thank all the reviewers' comments and suggestions. We apologize that due to the character limit and math rendering issue, some explanations are simplified/omitted. We will provide additional explanations in the follow-up discussion period.**

**Reviewer 2bnN acknowledges our novel blend of MPS and ARNN for ANTN takes the best of both worlds and achieves non-trivial experimental performances. Reviewer Vtcz acknowledges that our ANTN which integrates both MPS and ARNN is "undeniably compelling". Reviewer dUKK acknowledges that our proposed ANTN absorbs the advantages of two state-of-the-art methods (MPS and ARNN) with comprehensive theoretical analysis and experimental results. Reviewer wT6D acknowledges that our approach is novel and sound for an important problem with extensive experimental support.**

**The main concerns of reviewers are on the theoretical side of our theorems and the details of experiments. We have addressed all the questions with both updated theory and new experimental data. In particular, our updated theorem 5.2 clearly demonstrates that ANTN has stronger expressivity than TN and DMRG, which is also further supported by our new experiments in terms of performance and number of parameters. (See one-page pdf)**

**We have also implemented all the writing suggestions from the reviewers and provided answers to each review. Based on our update, we appreciate if the reviewers can consider raising their scores.**

**On the theory side, reviewers expressed the concern that Theorem 5.2 is not strong enough. In response, we updated Theorem 5.2 to show that our ANTN has stronger expressivity over TN and ARNN. More specifically, a general ANTN can be written as a TN or ARNN with an exponential (in system size) number of parameters. The updated Theorem (simplified due to math rendering issue) is listed below.**

Updated Theorem 5.2.

Autoregressive Neural TensorNet has generalized expressivity over tensor network and autoregressive neural network.

Theorem: 
1. Both TN and ARNN are special cases as ANTN
2. A general ANTN can be written as a TN or ARNN with an exponential (in system size) number of parameters. 

Detailed proof (will add in appendix)
1. As already shown in the paper

2. (a) In ANTN, the base ARNN outputs the conditional tensors $\tilde{\psi}^{\alpha_{i-1} \alpha_i} (x_i|\boldsymbol{x_{<i}})$ to form the resulting wave functions. These conditional tensors, when explicitly written out, contain $\chi^2\cdot2^i$ elements (where $\chi^2$ comes from the two bond dimensions and $2^i$ from all the qubits at or before the current qubit $i$). Since it has been shown that ARNN has volume law entanglement while TN doesn't. The conditional tensors generated from ARNN do not permit efficient tensor decomposition. Therefore, tensor network almost has to parameterize the full conditional tensor resulting in $\sum_{i=1}^N\chi^2\cdot2^i\sim\mathcal{O}(\chi^2\cdot2^N)$ parameters in total for all qubits.
(b) In ANTN, the conditional probability is generated from the marginal probability as shown in Eq. 4. The summation over $\alpha_i$'s  contains $\chi^{2i-1}$ terms when expanded, each of which can be viewed as a (quasi-)marginal probability $q^{\alpha_1,\alpha_1',\dots,\alpha_i}(\boldsymbol{x_{\le i}})$ generated by the underlying ARNN of ANTN. Using conventional ARNN architecture, a weight matrix of shape $h_{\mathrm{dim}}\times\chi^{2i-1}\times 2$ (where $h_{\mathrm{dim}}$ is the hidden dimension, and 2 is the local dimension of the current qubit) is required to fully parameterize all the (quasi-)probabilities, leading to $\sum_{i=1}^N2\cdot h_{\mathrm{dim}}\cdot \chi^{2i-1}\sim\mathcal{O}(h_{\mathrm{dim}}\cdot\chi^{2N})$ parameters in the last layer in total.
Thus, ANTN generalizes the expressivity over both TN and ARNN and cannot be efficiently represented with either TN or ARNN. 

**On the experiment side, we answer the reviewer's questions by running additional experiments and providing additional details regarding the number of parameters and training time for various algorithms in the attached PDF. The detail is summarized below.**

1. **Does increasing the number of layers allow pure pixelCNN to beat ANTN?** In attached Table R.1, we include additional results using pure pixelCNN with more layers. Although the added layer improves the result to some extend, the pixelCNN fails to beat ANTN in terms of energy calculation. This is consistent with the updated Theorem 5.2 that ARNN cannot efficiently parameterize a generic ANTN. 

2. **How does the inductive bias/physics prior of MPS affects the result? Does the improvement come from the DMRG initialization or the MPS structure?** In attached Table R.1, we perform 2 addition tests: a) pretrain pure pixelCNN by learning DMRG results; b) train ANTN (elementwise) without DMRG initialization. We find that DMRG pretraining negatively affects the result, while ANTN without DMRG initialization performs as good as with DMRG initialization. This is understandable, as learning DMRG results with complex sign structure is essentially the same task as learning a quantum state from shallow random circuit, which has been shown a difficult task for ARNN in Figure 2(b) of the original paper. Since the DMRG result is an approximation of the actual ground state, learning this state actually makes a worse initialization for pixelCNN. (We note that the pixelCNN was originally initialized from the result of smaller systems sizes, as we described in Appendix C under transfer learning.) 

2. **How do the runtime and number of parameters for the different algorithms compare?** In Table R.1, Table R.2, and Figure R.1, we show the number of parameters and runtime of each algorithm. In summary, DMRG has the most parameters, and is the slowest algorithm, taking as long as 2 weeks. The ANTN is much more efficient compared to DMRG while obtaining better energies, and it is comparable to pure ARNN while obtaining much better energies. 
  

Meta-review: The reviewers acknowledge the innovation to combine autoregressive (AR) models and matrix product state (MPS) for second-quantized quantum state representation, and the resulting empirical advantages over both AR and MPS models. There are also concerns on the precision of the expressiveness theorem, the explanation of conveyed inductive bias or physical structure from MPS, and completeness of empirical results to support the claims. After some discussions with the reviewers, they found additional explanations and empirical results provided in rebuttal addressed some of the concerns, and the overall opinion tends to outweigh the contributions, hence I recommend an accept for this submission.

Nevertheless, it is important that the authors include the additional explanations and empirical results in the final version of the paper. One particular issue to note is that an ARNN model could also achieve exponential expressiveness using polynomially many parameters, so the claim on the expressiveness comparison in the general rebuttal may not be fair enough.
Decision: Accept (poster)