SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\nabla_ {\theta} - \log F = - 2 \Re \frac {\nabla_ {\theta} f}{f} = - 2 \Re \frac {\mathbb {E} _ {\boldsymbol {x} \sim | \phi | ^ {2}} \frac {\nabla_ {\theta} \psi_ {\theta} (\boldsymbol {x})}{\phi (\boldsymbol {x})}}{\mathbb {E} _ {\boldsymbol {x} \sim | \phi | ^ {2}} \frac {\psi_ {\theta} (\boldsymbol {x})}{\phi (\boldsymbol {x})}} = - 2 \Re \mathbb {E} _ {\boldsymbol {x} \sim | \phi | ^ {2}} \frac {\frac {\psi_ {\theta} (\boldsymbol {x})}{\phi (\boldsymbol {x})}}{\mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim | \phi | ^ {2}} \frac {\psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\phi \left(\boldsymbol {x} ^ {\prime}\right)}} \nabla_ {\theta} \log \psi_ {\theta} (\boldsymbol {x}). \tag {A.3}
$$
Alternatively, one can sample from  $\psi_{\theta}(\pmb {x})$  by writing the complex conjugated
$$
f ^ {*} = \sum_ {\boldsymbol {x}} | \psi_ {\theta} (\boldsymbol {x}) | ^ {2} \frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})} = \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})}. \tag {A.4}
$$
This time, we have to be careful with
$$
\nabla_ {\theta} f ^ {*} = \sum_ {\boldsymbol {x}} \phi (\boldsymbol {x}) \nabla_ {\theta} \psi_ {\theta} ^ {*} (\boldsymbol {x}) = \sum_ {\boldsymbol {x}} | \psi_ {\theta} (\boldsymbol {x}) | ^ {2} \frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})} \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})} \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}). \tag {A.5}
$$
$$
\nabla_ {\theta} - \log F = - 2 \Re \frac {\nabla_ {\theta} f ^ {*}}{f ^ {*}} = - 2 \Re \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})}}{\mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x} ^ {\prime})}{\psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}} \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}). \tag {A.6}
$$
We can further use the variance reduction technique (Greensmith et al., 2004) from reinforcement learning and write
$$
\nabla_ {\theta} - \log F = - 2 \Re \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \left[ \frac {\frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})}}{\mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x} ^ {\prime})}{\psi_ {\theta} (\boldsymbol {x} ^ {\prime})}} - 1 \right] \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}), \tag {A.7}
$$
where the minus 1 comes from the mean of the coefficient in front of  $\nabla_{\theta}\log \psi_{\theta}^{*}(\pmb {x})$
$$
\frac {\mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x})}{\psi_ {\theta} (\boldsymbol {x})}}{\mathbb {E} _ {\boldsymbol {x} ^ {\prime} \sim | \psi_ {\theta} | ^ {2}} \frac {\phi (\boldsymbol {x} ^ {\prime})}{\psi_ {\theta} (\boldsymbol {x} ^ {\prime})}} = 1. \tag {A.8}
$$
One can verify that without the assumption that  $|\psi_{\theta}\rangle$  is normalized (i.e.  $F = |\langle \phi |\psi_{\theta}\rangle |^2 /\langle \psi_{\theta}|\psi_{\theta}\rangle)$  Eq.A.7 can also be obtained without applying the variance reduction formula.
In the main paper, due to the small system size  $N = 16$ , we enumerated all the bases when evaluating the quantum fidelity.
A.2 Variational Monte Carlo
As shown in the main paper, given a Hamiltonian  $\hat{\mathcal{H}}$  (and its matrix elements  $\mathcal{H}_{\boldsymbol{x},\boldsymbol{x}'}$ )
$$
\langle \psi_ {\theta} | \hat {\mathcal {H}} | \psi_ {\theta} \rangle = \sum_ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \mathcal {H} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \psi_ {\theta} ^ {*} (\boldsymbol {x}) \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right) = \sum_ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \left| \psi_ {\theta} (\boldsymbol {x}) \right| ^ {2} \frac {\mathcal {H} _ {\boldsymbol {x} , \boldsymbol {x} ^ {\prime}} \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\psi_ {\theta} (\boldsymbol {x})} = \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\sum_ {\boldsymbol {x} ^ {\prime}} \mathcal {H} _ {\boldsymbol {x} , \boldsymbol {x} ^ {\prime}} \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\psi_ {\theta} (\boldsymbol {x})}. \tag {A.9}
$$
Since the Hamiltonian  $\hat{\mathcal{H}}$  is usually sparse, give  $x$ , one only needs to sum up a small number of  $x^{\prime}$ 's for the numerator (usually linear in system size), allowing Eq. A.9 to be evaluated efficiently. Analogously, we can evaluate the gradient as
$$
\begin{array}{l} \nabla_ {\theta} \left\langle \psi_ {\theta} \right| \hat {\mathcal {H}} \left| \psi_ {\theta} \right\rangle = 2 \Re \sum_ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \mathcal {H} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \left[ \nabla_ {\theta} \psi_ {\theta} ^ {*} (\boldsymbol {x}) \right] \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right) \\ = 2 \Re \sum_ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} | \psi_ {\theta} (\boldsymbol {x}) | ^ {2} \mathcal {H} _ {\boldsymbol {x}, \boldsymbol {x} ^ {\prime}} \frac {\nabla_ {\theta} \psi_ {\theta} ^ {*} (\boldsymbol {x})}{\psi_ {\theta} ^ {*} (\boldsymbol {x})} \frac {\psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\psi_ {\theta} (\boldsymbol {x})} \tag {A.10} \\ = 2 \Re \mathbb {E} _ {\boldsymbol {x} \sim | \psi_ {\theta} | ^ {2}} \frac {\sum_ {\boldsymbol {x} ^ {\prime}} \mathcal {H} _ {\boldsymbol {x} , \boldsymbol {x} ^ {\prime}} \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\psi_ {\theta} (\boldsymbol {x})} \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}). \\ \end{array}
$$
Furthermore, it is possible to reduce the variance by either directly applying the variance reduction formula (Greensmith et al., 2004), or explicitly normalizing  $|\psi_{\theta}\rangle$  (minimizing  $\langle \psi_{\theta}|\hat{\mathcal{H}} |\psi_{\theta}\rangle /\langle \psi_{\theta}|\psi_{\theta}\rangle)$  to obtain
$$
\nabla_ {\theta} \left\langle \psi_ {\theta} \right| \hat {\mathcal {H}} \left| \psi_ {\theta} \right\rangle = 2 \Re \mathbb {E} _ {\boldsymbol {x} \sim \left| \psi_ {\theta} \right| ^ {2}} \left[ \frac {\sum_ {\boldsymbol {x} ^ {\prime}} \mathcal {H} _ {\boldsymbol {x} , \boldsymbol {x} ^ {\prime}} \psi_ {\theta} \left(\boldsymbol {x} ^ {\prime}\right)}{\psi_ {\theta} (\boldsymbol {x})} - \left\langle \psi_ {\theta} \right| \hat {\mathcal {H}} \left| \psi_ {\theta} \right\rangle \right] \nabla_ {\theta} \log \psi_ {\theta} ^ {*} (\boldsymbol {x}), \tag {A.11}
$$
where  $\langle \psi_{\theta}|\mathcal{H}|\psi_{\theta}\rangle$  can be approximated by applying Eq. A.9 on the same batch  $x$ . In this work, we use Eq. A.11 as the gradient of the loss function (see Appendix D for optimization details).
B Additional Theoretical Results
B.1 Exact Sampling from Conditional Probabilities
Suppose a full probability distribution over multiple qubits is written as a product of conditional probabilities as
$$
p (\boldsymbol {x}) = p \left(x _ {1}, \dots x _ {n}\right) = \prod_ {i = 1} ^ {n} p \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) \tag {B.1}
$$
with  $\pmb{x}_{<i} = (x_1, \ldots, x_{i-1})$ , then sampling a bitstring  $\pmb{x}$  from the probability distribution can be obtained by sequentially sampling each conditional probability as Since each sample is sampled independently using this algorithm, a batch of samples can be obtained in parallel, allowing for an efficient implementation on GPUs. In addition, the exact sampling nature of the algorithm makes it not suffer from the autocorrelation time problem of standard Markov chain Monte Carlo.
This same algorithm applies to sampling from  $|\psi(\boldsymbol{x})|^2$  with a simple replacement  $p(x_i | \boldsymbol{x}_{<i}) \to |\psi(x_i | \boldsymbol{x}_{<i})|^2$ . This is a direct consequence of
$$
\psi (\boldsymbol {x}) = \prod_ {i = 1} ^ {n} \psi \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) \Rightarrow | \psi (\boldsymbol {x}) | ^ {2} = \prod_ {i = 1} ^ {n} | \psi \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) | ^ {2}, \tag {B.2}
$$
and the fact that we require  $\sum_{x_i}|\psi (x_i|\pmb{x}_{< i})|^2 = 1$
Algorithm 3 Exact Sampling from Conditional Probabilities/Wavefunctions
$\pmb {x}\gets []$ $i\gets 1$    
while  $i\leq n$  do  $\begin{array}{rl} & {\pmb{x}_{< i}\leftarrow \pmb{x}}\\ & {\pmb{x}_i\sim p(\pmb {x}_i|\pmb{x}_{< i})}\\ & {\pmb{x}.append(\pmb {x}_i)}\\ & {i\leftarrow i + 1} \end{array}$    
end while   
return  $\pmb{x}$
B.2 Exact Sampling from MPS
As mentioned in the main text, matrix product state (MPS) with particular constraints, also allows for exact sampling. Recall that MPS defines a wavefunction as
$$
\psi (\boldsymbol {x}) = \sum_ {\alpha_ {1}, \dots , \alpha_ {n - 1}} M _ {x _ {1}} ^ {\alpha_ {0} \alpha_ {1}} \dots M _ {x _ {n}} ^ {\alpha_ {n - 1} \alpha_ {n}} = \sum_ {\boldsymbol {\alpha}} \prod_ {i = 1} ^ {n} M _ {x _ {i}} ^ {\alpha_ {i - 1} \alpha_ {i}}. \tag {B.3}
$$
Let's first assume that the wavefunction is normalized. It is easy to notice that any gauge transformation
$$
\left(M _ {x _ {i}}, M _ {x _ {i + 1}}\right)\rightarrow \left(M _ {x _ {i}} A, A ^ {- 1} M _ {x _ {i + 1}}\right), \tag {B.4}
$$
with  $A$  being an invertible matrix leaves the resulting wavefunction invariant. Here, we suppress the  $\alpha$  indices and view them as the indices for matrices. It has been shown that by fixing this gauge redundancy (Vidal, 2003, 2004), we can restrict all the  $M$  matrices to satisfy the following condition
$$
\sum_ {x _ {i}, \alpha_ {i}} M _ {x _ {i}} ^ {\alpha_ {i - 1} \alpha_ {i}} \left(M _ {x _ {i}} ^ {\alpha_ {i - 1} ^ {\prime} \alpha_ {i}}\right) ^ {*} = \delta_ {\alpha_ {i - 1}, \alpha_ {i - 1} ^ {\prime}}, \tag {B.5}
$$
where  $\delta_{\cdot, \cdot}$  is the Kronecker delta function. The matrices  $M$  that satisfy this condition are called isometries, and the MPS is called in the right canonical form (to be distinguished from the left canonical form). In the right canonical form, each  $M_{x_i}^{\alpha_{i-1}\alpha_i}$  can be interpreted as a basis transformation  $(x_i, \alpha_i) \rightarrow \alpha_{i-1}$  that is part of a unitary matrix.
Theorem B.1. Defining the left partially contracted tensors
$$
M  ^ {\alpha_ {j}} := \sum_ {\boldsymbol {\alpha} \leq j} \prod_ {i = 1} ^ {j} M _ {x _ {i}} ^ {\alpha_ {i - 1} \alpha_ {i}}, \tag {B.6}
$$
If the matrix product state is in the right canonical form, then the marginal probability of the wavefunction satisfies
$$
q (\boldsymbol {x} _ {\leq i}) := \sum_ {\boldsymbol {x} _ {> i}} | \psi (\boldsymbol {x}) | ^ {2} = \sum_ {\alpha_ {j}} M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j}} \left(M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j}}\right) ^ {*}. \tag {B.7}
$$
Proof. Let's define the right partially contracted tensors analogously
$$
M  ^ {\alpha_ {j - 1}} := \sum_ {\boldsymbol {\alpha} \geq j} \prod_ {i = j} ^ {n} M _ {x _ {i}} ^ {\alpha_ {i - 1} \alpha_ {i}}. \tag {B.8}
$$
We will first show that
$$
\sum_ {\boldsymbol {x} \geq j} M _ {R \boldsymbol {x} \geq j} ^ {\alpha_ {j - 1}} \left(M _ {R \boldsymbol {x} \geq j} ^ {\alpha_ {j - 1} ^ {\prime}}\right) ^ {*} = \delta_ {\alpha_ {j - 1}, \alpha_ {j - 1} ^ {\prime}}. \tag {B.9}
$$
We can prove this by induction:
- Base case:
$$
\sum_ {x _ {n}} M _ {R _ {x _ {n}}} ^ {\alpha_ {n - 1}} \left(M _ {R _ {x _ {n}}} ^ {\alpha_ {n - 1}}\right) ^ {*} = \sum_ {x _ {n}} M _ {x _ {n}} ^ {\alpha_ {n - 1} \alpha_ {n}} \left(M _ {x _ {n}} ^ {\alpha_ {n - 1} ^ {\prime} \alpha_ {n} ^ {\prime}}\right) ^ {*} = \delta_ {\alpha_ {n - 1}, \alpha_ {n - 1} ^ {\prime}}, \tag {B.10}
$$
which directly comes from Eq. B.5 by noticing that  $\mathcal{D}(\alpha_n) = 1$  so it can be ignored.
- Inductive step: Write
$$
M _ {R \boldsymbol {x} _ {\geq j}} ^ {\alpha_ {j - 1}} = \sum_ {\alpha_ {j}} M _ {x _ {j}} ^ {\alpha_ {j - 1} \alpha_ {j}} M _ {R \boldsymbol {x} _ {\geq j + 1}} ^ {\alpha_ {j}}. \tag {B.11}
$$
Assuming
$$
\sum_ {\boldsymbol {x} \geq j + 1} M _ {R \boldsymbol {x} \geq j + 1} ^ {\alpha_ {j}} \left(M _ {R \boldsymbol {x} \geq j + 1} ^ {\alpha_ {j} ^ {\prime}}\right) ^ {*} = \delta_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}}, \tag {B.12}
$$
$$
\begin{array}{l} \sum_ {\boldsymbol {x} _ {\geq j}} M _ {R \boldsymbol {x} _ {\geq j} ^ {\alpha_ {j - 1}}} \left(M _ {R \boldsymbol {x} _ {\geq j}} ^ {\alpha_ {j - 1} ^ {\prime}}\right) ^ {*} = \sum_ {x _ {j}} \sum_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} M _ {x _ {j}} ^ {\alpha_ {j - 1} \alpha_ {j}} \left(M _ {x _ {j}} ^ {\alpha_ {j - 1} ^ {\prime} \alpha_ {j} ^ {\prime}}\right) ^ {*} \sum_ {\boldsymbol {x} _ {\geq j + 1}} M _ {R \boldsymbol {x} _ {\geq j + 1}} ^ {\alpha_ {j}} \left(M _ {R \boldsymbol {x} _ {\geq j + 1}} ^ {\alpha_ {j} ^ {\prime}}\right) ^ {*} \\ = \sum_ {x _ {j}} \sum_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} M _ {x _ {j}} ^ {\alpha_ {j} - 1 \alpha_ {j}} \left(M _ {x _ {j}} ^ {\alpha_ {j} ^ {\prime} - 1 \alpha_ {j} ^ {\prime}}\right) ^ {*} \delta_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} \\ = \sum_ {x _ {j}, \alpha_ {j}} M _ {x _ {j}} ^ {\alpha_ {j} - 1 \alpha_ {j}} \left(M _ {x _ {j}} ^ {\alpha_ {j} ^ {\prime} - 1 \alpha_ {j}}\right) ^ {*} \\ = \delta_ {\alpha_ {j - 1}, \alpha_ {j - 1} ^ {\prime}}. \tag {B.13} \\ \end{array}
$$
Using this result, we can then prove our desired result about the marginal probability. Let's write
$$
\psi (\boldsymbol {x}) = \sum_ {\boldsymbol {\alpha}} \prod_ {i = 1} ^ {n} M _ {x _ {i}} ^ {\alpha_ {i - 1} \alpha_ {i}} = \sum_ {\alpha_ {j}} M _ {L _ {\boldsymbol {x} \leq j}} ^ {\alpha_ {j}} M _ {R _ {\boldsymbol {x} > j}} ^ {\alpha_ {j}}, \tag {B.14}
$$
$$
\begin{array}{l} q (\boldsymbol {x} _ {\leq i}) = \sum_ {\boldsymbol {x} _ {> i}} | \psi (\boldsymbol {x}) | ^ {2} = \sum_ {\boldsymbol {x} _ {> j}} \sum_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j}} M _ {R \boldsymbol {x} _ {> j}} ^ {\alpha_ {j}} \left(M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j} ^ {\prime}} M _ {R \boldsymbol {x} _ {> j}} ^ {\alpha_ {j} ^ {\prime}}\right) ^ {*} \\ = \sum_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} M _ {L \boldsymbol {x} \leq j} ^ {\alpha_ {j}} \left(M _ {L \boldsymbol {x} \leq j} ^ {\alpha_ {j} ^ {\prime}}\right) ^ {*} \delta_ {\alpha_ {j}, \alpha_ {j} ^ {\prime}} \tag {B.15} \\ = \sum_ {\alpha_ {j}} M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j}} \left(M _ {L \boldsymbol {x} _ {\leq j}} ^ {\alpha_ {j}}\right) ^ {*}. \\ \end{array}
$$
Corollary B.1.1. Matrix product state allows for efficient exact sampling.
Proof. Thm. B.1 shows that marginal probability distribution  $q(\pmb{x}_{\leq i})$  can be evaluated efficiently. The conditional probability can be evaluated efficiently either as  $q(x_{i}|\pmb{x}_{<i}) = q(\pmb{x}_{\leq i}) / q(\pmb{x}_{<i})$  or (equivalently) as an explicit normalization of the marginal probability distribution  $q(x_{i}|\pmb{x}_{<i}) = q(\pmb{x}_{\leq i}) / \sum_{x_{i}} q(\pmb{x}_{\leq i})$ . Then, Algorithm 3 can be used to sample from the full distribution.
B.3 Autoregressive Sampling Order
Algorithm 3 samples from conditional probabilities, which requires an ordering of the system to be defined. For 1D systems, we use a linear ordering such that qubit 1 corresponds to the leftmost qubit, and qubit  $n$  corresponds to the rightmost qubit. This ordering is natural for 1D MPS and transformer, as well as the transformer-based ANTN. For 2D systems, we use a snake (zig-zag) ordering. In this ordering, the qubit at the 2D location  $(i,j)$  is defined to be the  $i \times L_y + j$  th qubit, where  $L_y$  is the number of qubits along the second dimension. This ordering is inherently defined from the masked convolution for PixelCNN and PixelCNN-based ANTN. The MPS is reshaped to satisfy the same ordering for 2D systems. The same 2D ordering can be generalized to other tensor networks as well.
B.4 Additional Details of Expressivity and Symmetry Results of ANTN
Theorem B.2. Both tensor networks and autoregressive neural networks are special cases of Autoregressive Neural TensorNet.
Proof. Recall that the ANTN defines the wavefunction from its amplitude and phase
$$
\psi (\boldsymbol {x}) := \sqrt {q (\boldsymbol {x})} e ^ {i \phi (\boldsymbol {x})}, \tag {B.16}
$$
$$
q (\boldsymbol {x}) = \prod_ {j = 1} ^ {n} q \left(x _ {j} \mid \boldsymbol {x} _ {<   j}\right) \tag {B.17}
$$
$$
\text {w i t h} q \left(x _ {j} \mid \boldsymbol {x} _ {<   j}\right) = \frac {q \left(\boldsymbol {x} _ {\leq j}\right)}{\sum_ {x _ {j}} q \left(\boldsymbol {x} _ {\leq j}\right)} \tag {B.18}
$$
$$
q \left(\boldsymbol {x} _ {\le j}\right) := \sum_ {\alpha_ {j}} \tilde {\psi} _ {L} ^ {\alpha_ {j}} \left(\boldsymbol {x} _ {\le j}\right) \tilde {\psi} _ {L} ^ {\alpha_ {j}} \left(\boldsymbol {x} _ {\le j}\right) ^ {*} \tag {B.19}
$$
$$
\tilde {\psi} _ {L} ^ {\alpha_ {j}} \left(\boldsymbol {x} _ {\leq j}\right) := \sum_ {\boldsymbol {\alpha} _ {<   j}} \prod_ {i = 1} ^ {j} \tilde {\psi} ^ {\alpha_ {i - 1} \alpha_ {i}} \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) \tag {B.20}
$$
$$
\phi (\boldsymbol {x}) =: \operatorname {A r g} \sum_ {\boldsymbol {\alpha}} \prod_ {i = 1} ^ {n} \tilde {\psi} ^ {\alpha_ {i - 1}, \alpha_ {i}} \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right). \tag {B.21}
$$
- ANTN reduces to MPS. If we don't modify the tensors with neural networks,  $\tilde{\psi}^{\alpha_{i - 1},\alpha_i}(x_i|\boldsymbol{x}_{< i})$  reduces to  $M_{x_i}^{\alpha_{i - 1},\alpha_i}$  and  $\tilde{\psi}_L^{\alpha_j}(\boldsymbol{x}_{\leq j})$  reduces to  $M_{L\boldsymbol{x}_{\leq j}}^{\alpha_j}$ . It is then straightforward that Eq. B.19 reduces to Eq. B.15 and thus the probability distribution of ANTN reduces to that of MPS. Analogously, Eq. B.21 reduces to the phase of an MPS, and thus ANTN wavefunction reduces to an MPS wavefunction.
- ANTN reduces to ARNN. If we set all  $\mathcal{D}(\alpha) = 1$ , then  $\tilde{\psi}^{\alpha_{i - 1}\alpha_i}(x_i|\pmb{x}_{< i})$  reduces to  $\psi (x_{i}|\pmb{x}_{< i})$  and all the summations over  $\alpha$  can be dropped. In this case,  $q(\pmb{x}_{\le j})$  reduces to  $\prod_{i = 1}^{j}|\psi (x_{i}|\pmb{x}_{< i})|^{2}$  and  $q(x_{j}|\pmb{x}_{< j})$  reduces to  $|\psi (x_{j}|\pmb{x}_{< j})|^{2}$ , which is the same as the standard ARNN. In addition, the phase  $\phi (\pmb {x})$  reduces to  $\operatorname{Arg}\prod_{i = 1}^{n}\psi (x_{i}|\pmb{x}_{< i})$  which is also the same as ARNN. Thus, ANTN wavefunction reduces to ARNN wavefunction.
Notice that we only showed one type of TN, i.e. MPS, but the proof can be easily generalized to any TN that allows evaluation of marginal probabilities.
Theorem B.3. Autoregressive Neural TensorNet can be written as either a tensor network or an autoregressive neural network with an exponential (in system size) number of parameters.
- ANTN as TN. In ANTN, the base ARNN outputs the conditional tensors  $\tilde{\psi}^{\alpha_{i - 1}\alpha_{i}}(x_{i}|x_{< i})$  to form the final wavefunctions. These conditional tensors can be written as  $\tilde{\psi}_{x_1,x_2\dots ,x_i}^{\alpha_i - 1\alpha_i}$  in tensor notation. It directly follows from the shape of these tensors that they contain  $\chi^2\cdot 2^i$  elements (where  $\chi^2$  is from the two bond dimensions and  $2^{i}$  is from all the qubits at or before the current qubit  $i$ ). According to Theorem 5.2 ARNN has volume law entanglement while TN doesn't. The conditional tensors generated from ARNN do not permit efficient tensor decomposition. Therefore, a tensor network almost has to parameterize all full conditional tensors, resulting in  $\sum_{i = 1}^{N}\chi^{2}\cdot 2^{i}\sim \mathcal{O}(\chi^{2}\cdot 2^{N})$  parameters in total.
- ANTN as ARNN. In ANTN, the conditional probability is calculated from the marginal probability as shown in Eq. 4 as
$$
\begin{array}{l} q (\boldsymbol {x} _ {\leq i}) = \sum_ {\alpha_ {i}} \tilde {\psi} _ {L} ^ {\alpha_ {i}} (\boldsymbol {x} _ {\leq i}) \tilde {\psi} _ {L} ^ {\alpha_ {i}} (\boldsymbol {x} _ {\leq i}) ^ {*} \\ = \sum_ {\alpha_ {1}, \alpha_ {1} ^ {\prime}, \dots , \alpha_ {i - 1}, \alpha_ {i - 1} ^ {\prime} \alpha_ {i}} \tilde {\psi} ^ {\alpha_ {1}} \left(x _ {1}\right) \tilde {\psi} ^ {\alpha_ {1} ^ {\prime}} \left(x _ {1}\right) ^ {*} \dots \tilde {\psi} ^ {\alpha_ {i - 1} \alpha_ {i}} \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) \tilde {\psi} ^ {\alpha_ {i - 1} ^ {\prime} \alpha_ {i}} \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) ^ {*}. \tag {B.22} \\ \end{array}
$$
The summation over  $\alpha_{i}$ 's contains  $\chi^{2i - 1}$  terms, each of which can be viewed as a marginal (quasi-)probability  $q^{\alpha_1,\alpha_1',\ldots ,\alpha_i}(\pmb {x}_{\leq i})$  generated by the underlying ARNN of ANTN. This
effectively creates a weight matrix on top of a conventional ARNN, whose shape is  $h_{\mathrm{dim}} \times \chi^{2i - 1} \times 2$  (where  $h_{\mathrm{dim}}$  is the hidden dimension, and 2 is the local dimension of each qubit) on top of a conventional ARNN, resulting in  $\sum_{i = 1}^{N}2 \cdot h_{\mathrm{dim}} \cdot \chi^{2i - 1} \sim \mathcal{O}(h_{\mathrm{dim}} \cdot \chi^{2N})$  parameters.
Definition B.1 (Mask Symmetry). A conditional wavefunction  $\psi(x_i | x_{<i})$  has a mask symmetry if  $\psi(x_i | x_{<i}) = 0$  for some  $x_i$  given  $x_{<i}$ .
Theorem B.4. Autoregressive Neural TensorNet inherits mask symmetry from autoregressive neural network.
Proof. The mask symmetry results in  $\psi(\pmb{x}) = 0$  for certain  $\pmb{x}$  satisfying the condition while preserving the normalization of  $\psi(\pmb{x})$ . For ANTN, we can directly set  $q(x_{i}|\pmb{x}_{<i}) = 0$  for the  $x_{i}$  given  $\pmb{x}_{<i}$ . This results in  $\psi(\pmb{x}) = 0$  for the same  $\pmb{x}$ .
Corollary B.4.1 (Global U(1) Symmetry). Autoregressive Neural TensorNet can realize global  $U(1)$  symmetry, which conserves particle number.
Proof. A global  $\mathrm{U}(1)$  symmetry in a qubit system manifests as a conservation of the number of 0's and number of 1's in  $\pmb{x}$ . Equivalently,  $\psi(\pmb{x}) = 0$  for  $\pmb{x}$  violates such conservation. (Hibat-Allah et al., 2020) has shown that autoregressive neural networks can preserve global  $\mathrm{U}(1)$  symmetry as a mask symmetry, which ANTN inherits.
Definition B.2 (Function Symmetry). A conditional wavefunction tensor  $\psi(x_i | x_{<i})$  has a function symmetry over a function  $F$  if  $\psi(x_i | x_{<i}) = \psi(F(x_i; x_{<i}) | F(x_{<i}))$  where  $F(x_{\leq i}) := \{F(x_1), F(x_2; x_1), \ldots, F(x_i; x_{<i})\}$ .
Theorem B.5. Autoregressive Neural TensorNet inherits function symmetry from autoregressive neural networks.
Proof. We can apply the function symmetry on the left partially contracted tensors instead of the conditional wavefunctions. Here, we show that this produces the desired conditional probabilities and phase. Applying the function symmetry on the left partially contracted tensors results in
$$
\tilde {\psi} _ {L} ^ {\alpha_ {j}} \left(\boldsymbol {x} _ {\leq j}\right) = \tilde {\psi} _ {L} ^ {\alpha_ {j}} \left(F \left(\boldsymbol {x} _ {\leq j}\right)\right). \tag {B.23}
$$
This implies that the marginal probability satisfies  $q(\pmb{x}_{\leq j}) = q(F(\pmb{x}_{\leq j}))$  after contracting over the index  $\alpha_{j}$ , which then results in the correct symmetry on the conditional probabilities
$$
q \left(x _ {j} \mid \boldsymbol {x} _ {<   j}\right) = q \left(F \left(x _ {j}\right) \mid F \left(\boldsymbol {x} _ {<   j}\right)\right), \tag {B.24}
$$
as the conditional probability is just the marginal probability normalized at site  $j$ . The phase, on the other hand,
$$
\phi (\boldsymbol {x}) = \operatorname {A r g} \sum_ {\boldsymbol {\alpha}} \prod_ {i = 1} ^ {n} \tilde {\psi} ^ {\alpha_ {i - 1}, \alpha_ {i}} \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right) = \operatorname {A r g} \tilde {\psi} _ {L} ^ {\alpha_ {n}} \left(\boldsymbol {x} _ {\leq n}\right), \tag {B.25}
$$
which satisfies the same function symmetry from Eq. B.23.
Corollary B.5.1 ( $\mathbb{Z}_2$  Spin Flip Symmetry). Autoregressive Neural TensorNet can realize  $\mathbb{Z}_2$  spin flip symmetry such that the ANTN wavefunction is invariant under conjugation of the input.