Lattice Convolutional Networks for Learning Ground States of Quantum Many-Body Systems
Authors: Cong Fu
Xuan Zhang
Huixin Zhang
Hongyi Ling
Shenglong Xu
Shuiwang Ji
Venue: Submitted to ICLR 2023 (2023)

Abstract: Deep learning methods have been shown to be effective in representing ground-state wave functions of quantum many-body systems. Existing methods use convolutional neural networks (CNNs) for square lattices due to their image-like structures. For non-square lattices, existing method uses graph neural network (GNN) in which structure information is not precisely captured, thereby requiring additional hand-crafted sublattice encoding. In this work, we propose lattice convolutions in which a set of proposed operations are used to convert non-square lattices into grid-like augmented lattices on which regular convolution can be applied. Based on the proposed lattice convolutions, we design lattice convolutional networks (LCN) that use self-gating and attention mechanisms. Experimental results show that our method achieves performance on par or better than the GNN method on spin 1/2 $J_1$-$J_2$ Heisenberg model over the square, honeycomb, triangular, and kagome lattices while without using hand-crafted encoding.

Under review as a conference paper at ICLR 2023
LATTICE CONVOLUTIONAL NETWORKS FOR LEARNING GROUND STATES OF QUANTUM MANY-BODY SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning methods have been shown to be effective in representing ground-star state wave functions of quantum many-body systems. Existing methods use convolu- tional neural networks (CNNs) for square lattices due their image-like structures. For non-square lattices, existing method uses graph neural network (GNN) in which structure information is not precisely captured, thereby requiring additional sand-ed sublattice encoding. In this work, we propose lattice convolutions in which a set of proposed operations are used to convert non-square lattices into grid-like augmented lattices which regular convolution be applied. Based on the proposed lattice convolutions, we design lattice convolutional networks (LCN) that use self-gating and attention mechanisms. Experimental results show that our method achieves performance on par or better than the GNN method on spin 1/2 J1 Heisenberg model over the square, honeycomb, triangular, and kagome lattices while without using hand-crafted encoding.
INTRODUCTION
Study of quantum many-body problems is of fundamental interests in physics. It is crucial for theoretical modeling and simulation of complex quantum systems, materials and molecules (Carleo et al., 2019). For instance, graphene, arguably the most famous 2D material, is made of carbon atoms on honeycomb lattice. Solving quantum many-body problems remains be very challenging because of the exponential growth of Hilbert space dimensions with the number of particles in quantum systems. Only approximation solutions are available in most cases. Tensor network (White) 1992] Schollwöck (211 Orús 2014 [Biamonte and Bergholm 2017) is one of the popular techniques to model quantum many-body systems but suffers entanglement problems (Choo et al., 2018). Variational Monte Carlo (VMC) (McMillan 1965) is a more general methodology obtain quantum many-body wave functions by optimizing a compact parameterized variational ansatz with data sampled from itself. But how design variational ansatz with high expressivity to represent real quantum states still an open problem.
Recently traditional machine learning models, such as restricted Boltzmann machine (RBM) (molenn
sky] 1986), has been used as variational ansatz (Carleo and Troyer [2017] Nomura et al. 201 7 Choo
ct al. 20188 Kaubrucgger et al. 20188] Choo al. 2020) Nomura 202 Chen et al. [0022). Following
this direction, some studies explore deep Boltzmann machines Gao and Duan 2017 Carleo et al.
2018, Pastori et al. 2019) and fully-connected neural networks represent quantum states (Saito
and 20188 Cai and Liu 2018; Saito 2017 2018 Saito and Kato 2018). Most recent studies also use CNN variational ansatz for square lattice systems (Liang al. [0088] Choo al. 2019
Zheng et al. 2021 Liang et al., 2021 Roth and MacDonald, 2021). And GNN has been applied to non-square lattices and random graph systems (Yang et al. 2020a] Kochkov et al. 2021).
In this work, we explore the potential of using CNN as variational anstaz for non-square lattice quantum spin systems. We propose lattice convolutions that use a set of proposed operations to convert non-square lattices into grid-like augmented lattices which any existing CNN architectures can be applied. Based on proposed lattice convolution, we design highly expressive lattice convolutional networks (LCN) by leveraging self galf-gating and attention mechanisms. Experimental results show that our method achieves performance on par or better than the GNN method over the square, honeycomb, triangular, and kagome lattice quantum systems on spin 1/2 JI-J2 Heisenberg model, prototypical

---

Under review as a conference paper at ICLR 2023
quantum many-body model of magnetic materials that captures the exchange interaction between spins.
Novelty and Significance. Our work proposes the first pure deep learning approach that does not require any prior knowledge of quantum physics to solve quantum many-body problems on different types of lattice systems. Our approach overcomes the shortcomings of previous neural quantum state methods, which not only require extensive prior knowledge but are also designed for a specific lattice or even specific regime. However, our method can be seamlessly applied different lattices and can still achieve competitive or even better performance than existing methods without introducing prior knowledge. As a result, our method possesses great generalizability in practice. This makes our approach of great value in the study of quantum many-body problems.
Relations with Prior Work. GNN (Kochkov et al. 2021 is proposed as the first and generic method that can be applied to various lattice shapes. To this end, it is natural that LCN uses the same experiment setting as GNN. While GNN uses different hand-crafted sublattice encoding techniques for different lattice structures, LCN only needs to augment different lattices in simple and principled way without any prior knowledge. This significantly enhances the generalization capability of LCN in practice. RRoth and Mac (2021) proposes a general framework called Group-CNN. However, it can only be easily applied to square and triangular lattices. Moreover, it still needs to consider specific symmetry groups for different lattice systems prior knowledge. Choo al. (2019) applies CNN on square lattice, but it needs to use specific quantum physics knowledge such as point group symmetry and the Marshall sign rule, which is the known sign structure of ground state. However, the Marshal sign rule only works for bipartite graphs (such as square lattice) and non-frustrated regimes.
2 BACKGROUND AND RELATED WORK
In quantum mechanics, a quantum state is represented as a vector in Hilbert space. This vector is linear combination of observable system configurations {c; known computational basis. In the context of spin 1/2 systems, each spin can be measured in two states, spin-up or spin-down, which are represented by and J, respectively. All the combinations of spins form basis. Given N spins, there are in total 2 configurations in the computational basis. Specifically, state can be written as
N
(+b) = Σ (()))))
where lc;) represents an array of spin configurations of N spins, e.g., 11 ... J. and vi(c) is the wave function, which is in general complex number. The summation is over all possible 2 spin configurations. The squared norm [ıb(),)2 corresponds to the probability of system collapsing to configuration & when being measured, and Σ; [eb(ca))2 = due to normalization.
2.1 GROUND STATES
The ground state of a quantum system its lowest-energy state. Usually, many physical properties can be determined by the ground state. Particle interactions within given quantum many-body system are determined by Hamiltonian, which an Hermitian matrix H in the Hilbert space. System energy and its corresponding quantum state governed by the time-indepenee ime-independent Schrödinger equation:
H | Flw).
which is an eigenvalue equation. The eigenenergy E is the eigenvalue of H and (τ) is the corre- sponding eigenvector. In principle, those can be obtained by eigenvalue decomposition given H. The lowest eigenvalue is called the ground state energy. and its associated eigenvector is called the ground state. The ground state and the ground state energy determine the property of the quantum system at zero temperature.
2.2 VARIATIONAL PRINCIPLE IN QUANTUM MECHANICS
Given a system of size N, the dimension of the Hamiltonian matrix is N x 2 Since the dimension of the matrix grows exponentially with system size, it is intractable to eigenvalue decomposition
2

---

Under review as a conference paper at ICLR 2023
directly even for relatively small systems. The state-of-the-art algorithm using Lanezos method, that explores the sparseness of H, can obtain the ground state energy and the ground state for N up to 48. For larger systems, common approach is to use variational principle to approximately solve the Schrödinger equation. According to the variational principle, the energy of any given quantum state is greater than equal to the ground state energy. So
optimize parameterized wave functions to make the energy as low as possible, Specifically, we can approximate the ground state of a Hamiltonian H by minimizing the variational term E shown as below:
(()))) (1/1))
E
2 Eo.
where E' is the expectation value of energy of a variational quantum state (&)) for given Hamiltonian H. and Eg is the true ground state energy. The state (8) takes the form of Eq. equation | Given H. the expectation value E is determined by the wavefunction (b(;,) that can be any parameterized
functions. The goal is to find the optimal function ti(c,) that minimizes E. The success of the variational method relies the expressivity of the parameterized function. Therefore it is natural to explore neural networks as variational ansatz of the wavefunction.
2.3 RELATED WORK
Variational quantum many-body states with wave functions given by neural networks are called
neural-networt quantum states, initially studied by Carleo and Troyer (2017). where they use RBM to represent many-body wave function. Subsequent studies (Choo et al. 2018 Saito 2018 Cai and Liu 2018) apply fully-connected neural networks as variational ansatz, which has been shown to be more effective than RBM methods. But these methods do not explicitly consider the structure information when applied to two-dimentional systems (Cai and Liu 2018). Motivated by the spatial symmetry of periodic quantum systems and successful practices of convolutional neural networks (CNNs) in computer vision (Krizhevsky al. 2012), Liang et al. (2018). Choo et al.) (2019) and
Szabó and Castelnovo 2020) use CNNs to represent quantum states for square lattices. Albergo et al.
(2021) and Boyda al. (2021) combine flow-based sampling with CNN for lattice field theories.
CNN is able to effectively represent highly entangled quantum systems (Levine et al.] 2019) than
RBM based representations (Deng al. 2017), which benefits from its information reuse. However,
CNN cannot be naturally used on non-grid like systems. Recently graph neural networks (GNNs)
have also been applied to represent wave functions (Yang et al., [2020a] Kochkov al., 202T]. GNNs can work with arbitrary geometric lattices or even random graph systems but structure information is not precisely captured. So additional hand-crafted sublattice encoding needed augment system configurations in order to respect underlying quantum symmetry (Kochkov et al. (2021).
Wave functions are usually cemplex-valuedd therefore is necessary to predict both amplitudes and phases. [Cho et al. (2019) use complex-valuse weights and biases to predict amplitudes and phases simultaneously, and design generalized activation function. Whereas amplitudes and phases are predicted separately using real-valued networks in Szabó and Castelnovo (2020); Kochkov al. (2021).
LATTICE CONVOLUTIONAL NETWORKS
While GNN can be naturally applied to non-square lattices, isotropic weights prevent it from capturing rich structure information. Therefore auxiliary hand-crafted structure encoding is needed to augment the original spin configuration input on lattices. We argue that CNN more suitable to model wave functions for lattice systems, which features repetitive local patterns.
In this section, we introduce LCN, a novel lattice convolutional network that has strong capability to model wave functions for non-square lattice systems without using any extra structure encoding.
3.1 MOTIVATION AND OVERVIEW
In this work, we focus on designing CNNs on four types of lattice, including square, triangular, honeycomb and kagome, which are the four of the most common lattice structures that describes two dimensional materials. The lattice structures are shown in Figure[] CNNs are known to be efficient feature extractors regular structures. Inside the network, each convolution layer applies
3

---

Under review as a conference paper at ICLR 2023
(a) Square
(c) Kagome
(b) Triangular
(d) Honeycomb
Original vertex
Virtual vertex
Lattice edge
Convolution kernel
Figure 1: Structures of four different lattices studied in this work. Shadow areas denote the regular convolution kernels. Red dots represent original vertices lattices, green dots represent padded virtual vertices. In (c) and (d), dashed lines denote the nearest neighbors of virtual vertices. On the right are different substructures captured by convolution. Note that shear directions won't It affect the result due lattice symmetry. Even though the lattices shown finite but they periodically arranged in whole space, which is realized by applying periodic boundary condition.
anisotropic filters on local regions, shared across the entire input. Through training, each filter learns to be sensitive to different local patterns. By stacking multiple convolution layers, the global structure information can be precisely captured.
straightforward to apply CNNs on a square lattice due to its image-like structure. Triangular lattices can be viewed sheared square lattices where every unit cell of the square lattice undergoes the same affine transformation. Therefore we can shear the convolution kernel accordingly to match the shape of the transformed unit cells. However, for lattices that cannot be converted into grids, such as honeycomb and kagome, the key to define the shape of the convolution kernel and optimize the weight sharing across convolution sites. In the proposed LCN, solve these challenges in principled way by converting non-square lattices into grid-like augmented lattices through set of operations such that regular convolution kernels can be applied.
3.2 AUGMENTED LATTICES
As mentioned in Scction 3.J triangular lattices can be seen as sheared square lattices, which implies that the local structure is same everywhere on the lattice, hence regular square kernels can be naturally applied. By contrast, honeycomb and kagome lattices have multiple local structures, making it difficult to share kernel weights across different structures. Moreover, different local structures are arranged in staggered manner, which impedes the information reuse among the same local structures.
Critically, we make the key observation that honeycomb and kagome lattices can be converted from triangular lattices by removing some vertices and edges. Conversely, honeycomb and kagome lattices can be viewed triangular lattices through augmenting virtual vertices back on the original lattices. As shown in Figure [ (c) and (d), virtual vertices (green dots) are inserted in the center of each hexagon sub-structure. Through this augmenting operation, can apply identical kerell everywhere on the lattices regardless original local structures but still can capture different structure information.
The advantages of this operations are two folds. First, during convolution, the virtual vertices participate in the convolution in the same way the original vertices. i.e., values of virtual vertices
4

---

Under review as a conference paper at ICLR 2023
also updated. By doing the virtual lattices gather and distribute the information from the original vertices, which can help increase receptive field and boost information exchange. Second, by adding virtual vertices, reuse, which is crucial to capture long-range spin correlations (Liang et al.) 2018). To some extent, augmenting with virtual vertices could enhance the wave function representation ability.
overlap the same convolution kernel in order enable information
3.3 LATTICE CONVOLUTIIONS
After augmentation, the lattice becomes either square or triangular, both of which are grid-structured. However, due to the characteristics of the input, additional processing steps are required in order to apply regular convolutions on the augmented lattice.
Boundary alignment (BA). Lattices with finite vertices often have irregular boundaries. However, in current deep learning libraries, convolutions image-like data typically requires the input grid to have equal number elements on each row. Therefore, we zero-pad the augmented lattices into parallelograms. In our experiments, square lattices already have regular boundaries so this step is omitted. The aligned boundary is drawn with solid line in Figure
Periodic padding (PP). For images, boundaries commonly zero-padded before convolutions to preserve the size of feature maps. Finite quantum systems often consider periodic boundary conditions that the lattice structure information, after padding the aligned boundaries with zero, we replace the padding values around the original lattice area with the values given by the periodic boundary condition. We can optionally do the periodic padding for the virtual vertices as well. In Figure the padded boundary is drawn with dashed line, the original lattice area is marked with pink shadow and the periodic padding is marked with green shadow.
be repeated fill the entire space. To preserve this important
Mask. Finally, after each convolution, clean up the artifacts introduced the two previous steps, we reset all vertices used for the boundary alignment and the periodic padding to zero. We do not reset the virtual vertices allow information passing through them, i.e., we only reset the vertices outside the original lattice area. We also conduct ablation study of mask operation in Appendix K.1 I.I
To summarize, the proposed lattice convolution applied on an input lattice U is defined as:
LatticeConv( U; W) Mask W = PP((BAAug(U)))
where W is the convolution weight matrix; PP, BA and Mask stand for the above mentioned three processing steps and Aug stands for the augmentation step defined in Section 3.2, The symbol denotes the regular convolution which defined as:
(W U'),, Σ
Σ
W mn U' i-m.J-n
(5)
where U' E Hxxxx xd denotes a H x W feature map with d input channels. W has shape (2s +
1) x (2s' 1) x d' x where d' is the number of output channels and (2s 1) (2s' + 1) is the size of the receptive field.
3.4 INSTANTIATIONS
Square. As shown in Figure (a), ) for square lattices, the 3 x 3 kernel receptive field contains all four nearest neighbors and four second near- est neighbors (or called next nearest neighbor in some references), which defined by the euclidean distance, instead of by connectivity.
Aligned boundary
boundary
Triangular. For triangular lattices, the recep- tive field of 3 x convolution kernel includes six nearest neighbors and two second nearest neighbors of center vertices, shown in Fig- ure (b). This structure pattern is same at positions across the lattice.
Original lattice area
paciding
Figure 2: Boundary alignment and periodic padding.
5

---

Under review as a conference paper at ICLR 2023
Honeycomb, For honeycomb lattices, the 3 x convolution kemnel centered on the original vertices captures all three nearest neighbors and two second nearest neighbors. As shown in the Figure | (d), depending the local structure, the mapping between the kernel weights and the original vertices can have two different situations. The kernel centered on virtual vertices captures six original vertices around it. For honeycomb lattice, we also apply the periodic padding for the virtual vertices.
Kagome. For kagome lattices, as shown in Figure T (c), the x 3 kernel can capture all nearest neighbors and some next nearest neighbors. The virtual vertices can receive information from eight original vertices around it. There are three local structures centered on original vertices. And we only apply the periodic padding for the virtual vertices on kagome lattices of size 36.
3.5 CNN VERSUS GNN
The convolution applies different weights for neighbors with different relative positions, which we argue is critical to capture the structure information. This is to the opposite with graph neural networks where same weights used for all one-hop neighbors define a graph on the square lattice by defining edges as the spatially nearest neighbors. For quantum many-body systems, the spatially second nearest neighbors play important role in defining the energy. But the two-hop neighbors on the graph will also include the third nearest neighbors. To be able to capture the structure information with GNNs, Kochkov et al. (202I) propose augment the vertices with sublattice encoding to explicitly provide structure information at input. Our experiments show that such encoding is indeed critical for GNNs. On the contrary, our lattice convolution accurately learns the ground state without any additional input. To some extent, the proposed lattice convolution can learn the structure encoding automatically in the kernel space.
graph. For example, we can
NETWORK ARCHITECTURE AND TRAINING
After constructing the grid-like input and defining the convolution operation, any existing convolution neural network architecture can be applied. We aim to design variational ansatz to have powerful expressivity and capture spatial long-range spin correlations. To this end, our model is developed based recent advanced deep learning modules and attention mechanisms. The details of main components are described below.
suncez--and-Exxiitatiin Block. Squeeze-and-Exciian the quality of representations produced by a network by explicitly modeling the channel-wise interdependencies. It utilizes squeeze operation to aggregate channel-wise global spatial information and excitation operation to capture channel-wise dependencies through self-gating recalibration. Details of formulation can be found in Appendix
(SE) block (Hu et al.) 2018) can improve
Non-Local Block. We find it is useful to incorporate spin-spin global interaction other than only locality interaction defined by lattice edges. Non-local operation (Wang al., 2018 designed to capture long-range dependencies. Specifically, non-local operation make features at one position attend all other position's features. Details of formulation can be found in Appendix !
SE-Non-Local Layer. Based on the SE block and non-local block, we propose SE-Non-Local Layer by further introducing skip connection and pre-activation techniques. Therefore, the resulting SE- Non-Local Layer consists of the following components: Normalization - Activation + LatticeConv
- SE Block + Addition >> Non-local Block. We apply LayerNorm (LN) (Ba al. 2016) as the normalization and ReLU as the activation function.
In our model. we predict amplitudes and phases separately with real-valued network, follow- ing Kochkov et al. (02T). We first use lattice convolution to transform original spin configuration input into embedding space and then stack multiple SE-Non-LLocal layers obtain the final latent representation. At last, we flatten the feature maps into vector to keep all information lattice vertex and use MLP to obtain the log amplitude and argument the wave function value. The overall architecture of our variational ansatz is shown in Figure 3 We also conduct ablation study network choice in ??
each
Training. As discussed in Section 2.2
use Variational Monte Carlo framework to optimize the variational ansatz iteratively by minimizing the statistic expectation of system energy (E). The data (spin configurations) sampled from the probabilistic distribution defined by variational ansatz

---

Under review as a conference paper at ICLR 2023
Latiet beev
Latricel
Squoree & Excetaaioo
Non local
Rel.I
Lathectonn
Squerre & Exceation
Non Non-local local
Flatten
Limear
LN
ReLL
Linear
arg(y)
log(hel)
Layens
Figure 3: Lattice Convolutional Networks architecture
(neural network wave function) itself. Then we stochastically estimate the gradient V V.. (Kochkov et al. 2021) to update network parameters w. Details of the training method can be found in Ap- pendix B and Appendix
EXPERIMENTS
In this section, we evaluate the proposed LCN on learning ground states of the spin- 1/2 J1-J2 Heisenberg model, where the input vertices represent quantum spins. We show that our model can accurately approximate the ground state energies and achieve on par or better results with GNN models.
Lattice. Following Kochkov et al. (2021). four kinds of lattice are used in our experiments including square, honeycomb, triangular and kagome. Periodic boundary conditions are used so that the neighborhood patterns of the boundary vertices are the same with the internal vertices. All the lattice geometries use in the experiments
be found in Kochkov et al. (2021 Appendix A.5.a).
J1-J2 Quantum Heisenberg model. The J1-J2 quantum Heisenberg model is the prototypical model for studying the magnetic properties of quantum materials. Its Hamiltonian matrix is given by:
Σ S. S, + J2 (1
Σ S. S. ((--5))
where S; = (ST, SF)
S;) are the spin-1/2 operators the i-th vertex. The spin operator S is an Hermitian matrix of dimension 2 defined as S 70j-1 s のe s f8N-j, where & stands for Kronecker product, / is the two-by-two identity matrix and rº is the two-by-two Pauli matrix for a = y, z. The term S, S, describes the antiferromagnetic exchange between the spin on site i and the spin site j. (+, +) denotes the nearest neighbors and ((:-)) denotes the second nearest neighbors, both in terms euclidean distances; Jz controls the interaction strength between the next nearest neighbors. The interaction strength between nearest neighboring spins is to 1 as the unit.
Setup. We test our models on four kinds of lattice with various sizes and Jz values. We compare the proposed LCN with GNN (Kochkov al. 2021) as well reference energies. We use the same references as in Kochkov et af. (2021). We also choose specific J2 values under which the ground state is much harder learn for the GNN model. The energy per site of the learned wave function is used as evaluation metric where the total energy of the system is divided by the number of spins the ground state.
(vertices). lower energy per site indicates more accurate approximation More implementation details can be found in Appendix D Considering we are proposing generic CNN network that fits multiple kinds of lattice without prior knowledge, we don't include other customized physics-incorporated networks for square lattice as baselines. But for completeness, we conduct experiments on full range of square lattice to compare with other state-of-the-.. models in Appendix L
Results. The experimental results are summarized in Table 1 For small systems (N 36), the reference energies are the exact ground state energies computed by direct diagonalizing the Hamiltonian matrix (Schulz et al. 1996. Albuquerque et al. 2011 Iqbal et al. 2016 Changlani et al. 2018p. For large systems (NV 98, 108), the exact diagonalization is computationally infeasible. Reference energies are calculated with quantum Monte Carlo (QMC) (Sandvik] 1997) and RBM+PP
(pair-product states) (Nomura and Imada 2021) for square with J2 0.0 and Jz 0.5. respectively,
7

---

Under review as a conference paper at ICLR 2023
Table 1: Estimated energy per site of the learned wave function (with error bars in parenthesis). Lower is better. Four kinds of lattice with various sizes are used for comparison. The J2 value controls next nearest neighboring interaction on the lattice, resulting in differences in the ground states. The GNN-2 model doubles the parameters and computation by using separate branches for predicting the amplitude and argument of the wave function. Best result in each row is denoted in bold (if two are the same, we only bold-face LCN for clarity), and LCN's s results that worse than GNN-2 but better than or same with GNN are underlined. We use to denote the results measured from plots their reference papers. For the reference ground state energies we also annotate the employed methods: for exact diagonalization, oo for infinite-size estimates, I for RMB+PP, 3 for QMC methods, and for DMRG methods.
Lattice
Size
J2
GNN
GNN-2
LCN (ours)
Reference Energy
0
-0.6788*
-0.6788(1)
-0.6788721
36
0.5
-0.5022(4)
-0.5023(5)
-0.5022(2)
-0.503810*
Square
0
-0.6708(0)
-0.6708(1)
0.671549(4)$
100
0.5
-0.4955(4)
-0,4960(5)
-0.4957(4)
-0,49629(1)}
0
-0.551*
-0.5516(1)
-0.5517*
32
Honeycomb
0.2
-0.4563(6)
-0,4564(7)
-0.4563(1)
-0.4567
-0,4421(4)
-0.5448
98
0.2
-0.4528(2)
-0.4536(5)
-0.4538(4)
-0.4527
0
-0,55889
-0.5601(4)
-0.56037341
36
0.08
-0.5221*
-0,527((4
-0.52861*
Triangular
0.125
-0.512(2)
-0.113188)
-05512(66
-0.515564*
0
-0.550888
-0.5519(4)
-0.4755(4)
-0.551
108
0.125
-0,500(9)
-0,5069(8)
-0,5110(4)
-0.5126*
0
-0.434(1)
-0.4338(6)
-0.4367(4)
-0.438376531
36
Kagome
-0.02
-0.4339*
-0.4384(5)
-0.4399T*
108
0
-0.4302(1)
-0.4314(7)
-0.4276(3)
-0.4386*
and density-matrr renormalization group (DMRG) (Igbal et al. 2016 Yan et al.) 2011 for triangular and kagome. In some cases, the estimates for infinite-size lattice are used as reference energies, extrapolated from exact diagonalizations for honeycomb or from DMRG for triangular. These reference energy of large systems can be seen as a very tight upper bound of true ground state energy. Results show that the wave functions learned by the proposed LCN consistently gives energies close to the reference ground state energies on both small and large systems. For small systems, LCN gives similar ground state energies with GNN on square and honeycomb lattices while achieves better ground state energies on triangular and kagome lattices. For large systems at J/2 = 0, LCN achieves similar ground state energies with GNN on the square lattice while falls behind on triangular and kagome lattices. For large systems Ja * O, LCN outperforms GNN on all tested lattices including square, honeycomb and triangular. The good performance proves that the proposed LCN is able to accurately represent the ground states of quantum many-body systems without explicit structure encoding.
Quantum State and Energy. The quality of the variational result is measured by the ratio (E- EA) where A is the gap of the Hamiltonian, i.e. the difference between the energies of the first excited state and the ground state. Usually for gapless model, the exact gap A is unknown. However, for the Heisenberg J1-J2 model, approximately improvement on order 1/N2 on the energy per spin could be considered significant in approaching the true ground state. As listed in Table our results can achieve this improvement in most cases. But even though energy value improvement is slightly less than order 1 1/N², there still could be dramatic improvements in approaching true ground-state we conduct experiment to show this on 12 node kagome lattice in Appendix G
Kernel Design Comparison. Apart from applying regular convolution kernels on augmented lattices, we also design special convolution kernels for original honeycomb, triangular and kagome lattices, which only capture the nearest neighbors of each center vertices. Details of these special kernel design can be found at Appendix E
8

---

Under review as a conference paper at ICLR 2023
We compare the performance of these two categories of kernel design small sys- tems over honeycomb, triangular and kagome lattices, as shown in Table 2 Experimen- tal results show that regular convolution kemels consistently outperform special kernels. We argue the reasons are two folds. First, the regular kernel directly captures interaction be- tween part of second nearest neighbors, which more helpful when J2 is not zero. Second reason is that information reuse is hindered to some extent for special kernels. For example, when using special kernels for original kagome lattices, three distinctive kernel shapes need to be used for capturing different local structures on lattices. And the same local structure has little overlap with each other, resulting in less overlap between same kernels. This impedes the information reuse among the same local struc- tures that is crucial for capturing long-range spin correlations (Liang et al. 2018).
Table 2: Performance comparison between regular kernel and special kernel on honeycomb, triangular and kagome lattices.
Regular Kernel
Special Kernel
Lattice
Size
J2
0
-0.5516(1)
-0.512(22
Honeycomb
32
0.2
-0.4563(1)
-0.447((3)
-0,5601(4)
-0.5501(6)
Triangular
36
0.08
-0.5273(4)
-0.5127(7)
0. 125
-0.5126(6)
-0.4936(7)
0
-0.4367(4)
-0.4207(6)
Kagome
36
-0.02
-0.4384(5)
-0.4253(6)
So we conclude that lattice augmentation together with regular kernels are necessary for processing quantum lattice systems. On one hand, lattice augmentation is general for these four kinds of lattice and does not need any prior domain knowledge compared with GNN's sublattice encoding. On the other hand, with virtual vertices added, regular kernels can be applied, which has many advantages over special kernels such as boosting information exchange and reuse, as described in Section 3.2
Different Kernels for Honeycomb. For hon- eycomb lattices, the special kernel has half less parameters than regular kernel. So for fairness comparison, we enlarge the special kernel to capture all next nearest neighbors and hence the number of parameters is similar to the regular kernel. As shown in T performance of the 2-hop special kernel is still worse than regular kernel with virtual vertices added. Another observation is that the 2-hop special kernel performs even slightly worse than 1-hop special kernel, which implies that enlarging the receptive field of special kernel cannot help capture further useful spin-spin correlation.
Table 3: Experiments on honeycomb with different kernel design.
Regular Kerne
Special Kernel
hop Special
Lattice
Size J2
Kemel
Honey- comb
0
-0.5516(1)
0.5512(22
0.5511(2)
32
0.2
-0.4563(1)
-0.4547(3)
-0.4539(4)
Network Architecture Comparison. We conduct experiments on kagome lattice to show the effect of Squeeze-and and-Excitation (SE) block and Nonlocal block, as shown in Table Adding SE block will improve performance upon using residual connection only. We hypothesize that SE blocks use squeeze operation to aggregate global spatial information and rescaling to recalibrate the importance of channels to capture channel-wise dependencies, which can implicitly capture long-range spin correlation. And non-local block makes spin at one position attend to all other spins, which explicitly capture spin-spin global interaction, so using non-local block can further improve performance.
Table 4: Network architecture selection
Lattice
Size
J2
Residual only
Residual+SE
Residual+Nonlocal
Residual--SE+NNnlocal
36
0
-0.4310(5)
-0.4323(5) -0.4153(6)
-0.4365(5)
-0.4367(4)
Kagome
108
0
-0.4090(5)
-0.4161(5)
-0.4276(3)
6 CONCLUSION
We propose lattice convolutions to process non-square lattices by converting them into grid-like augmented lattices through a set of operations. So regular convolution can be applied without using hand-crafted structure encoding, which is needed in the previous GNN method. And we design lattice convolutional networks that use self- self-gating and attention mechanisms to capture channel-wist interdependencies and spatial long-range spin correlations. which contribute to high expressivity of variational wave functions. We experimentally demonstrate the effectiveness of lattice convolution network wave functions and achieve performance on par or better than existing methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael S Albergo, Gurtej Kanwar, Sébastien Racanière, Danilo J Rezende, Julian M Urban, Denis Boyda, Kyle Cranmer, Daniel C Hackett, and Phiala E Shanahan. Flow-based sampling for fermionic lattice field theories. Physical Review D, (44(1))114500) 2021.
AF Albuquerque, David Schwandt, Balazs Hetenyi, Sylvain Capponi, Matthieu Mambrini, and AM Läuchli. Phase diagram of a frustrated quantum antiferromagnet on the honeycomb lattice: Magnetic order versus valence-bond erystal formation. Physical Review B, 84(2):024406, 2011.
Nikita Astrakhantsev, Tom Westerhout, Apoorv Tiwari, Kenny Choo, Ao Chen, Mark H Fischer, Giuseppe Carleo, and Titus Neupert. Broken-symmetry ground states of the heisenberg model on the pyrochlore lattice. arXiv preprint arXiv:2101.08787, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064500, 2016.
Jacob Biamonte and Ville Bergholm. Tensor networks in nutshell. arXiv preprint arXiv: 2017.
Denis Boyda, Gurtej Kanwar, Sébastien Racanière, Danilo Jimenez Rezende, Michael S Albergo, Kyle Cranmer, Daniel Hackett, and Phiala Shanahan. Sampling using su (n) gauge equivariant flows. Physical Review D, 103(7):074504, 2021.
Peter Broecker, Juan Carrasquilla, Roger G Melko, and Simon Trebst. Machine learning quantum phases of matter beyond the fermion sign problem. Scientific reports, 7(1):1-10, 2017.
Zi Cai and Jinguo Liu. Approximating quantum many-body wave functions using artificial neural networks. Physical Review B, 97(3)x03511 16,, 2018.
Giuseppe Carlco and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325);602-666, 2017.
Giuseppe Carleo, Yusuke Nomura, and Masatoshi Imada. Constructing exact representations of quantum many-body systems with deep neural networks. Nature communications, 9(1):1-11, 2018.
Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Reviews of Modern Physics, (14):0450,, 2019.
Hitesh J Changlani, Dmitrii Kochkov, Krishna Kumar, Bryan K Clark, and Eduardo Fradkin. Macro- scopically degenerate exactly solvable point in the spin-1/2 1/2 kagome quantum antiferromagnet. Physical review letters, 120(11): 17202, 2018.
Hongwei Chen, Douglas Hendry, Phillip Weinberg, and Adrian E Feiguin. Systematic improvement of neural network quantum states using lanczos recursion. arXiv preprint arXiv:2206.143300 2022.
Jing Chen, Song Cheng, Haidong Xie, Lei Wang. and Tao Xiang. Equivalence of restricted boltzmann machines and tensor network states. Physical Review B, 97(8)0051044 2018
Kenny Choo, Giuseppe Carleo, Nicolas Regnault, and Titus Neupert. Symmetries and many-body excitations with neural-network quantum states. Physical review letters, 121(16))16200, 2018.
Kenny Choo, Titus Neupert, and Giuseppe Carlco. Two-dimensional frustrated j 1-j with neural network quantum states. Physical Review B. 125,,, 2019.
model studied
Kenny Choo, Antonio Mezzacapo, and Giuseppe Carleo. Fermionic neural-network states for ab-initio electronic structure. Nature communications, 11(1):1-7, 2020.
Dong-Ling Deng, Xiaopeng Li, and Das Sarma. Quantum entanglement in neural network states. Physical Review X, ((2):221021, 2017.
10

---

Under review as a conference paper at ICLR 2023
Francesco Ferrari, Federico Becca, and Juan Carrasquilla. Neural gutzwiller sutzwiller-projectedd variational wave functions. Physical Review B, 100(12):125131, 2019.
Xun Gao and Lu-Ming Duan. Efficient representation of quantum many-body states with deep neural networks. Nature communications, 8(1):1-6, 2017.
Ivan Glasser, Nicola Pancotti, Moritz August, Ivan D Rodriguez, and Ignacio Cirac. Neural-network quantum states, string-bond states, and chiral topological states. Physical Review X, 8(1 8(1):01 1006, 2018.
Jiequn Han, Linfeng Zhang, and E Weinan. Solving many-electron schrödinger equation using deep neural networks. Journal of Computational Physics, 399:108929, 2019.
Mohamed Hibat-Allah, Martin Ganahl, Lauren E Hayward, Roger G Melko, and Juan Carrasquilla. Recurrent neural network wave functions. Physical Review Research. 2(2):023358, 2020.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
Yasir Iqbal, Wen-Jun Hu, Ronny Thomale, Didier Poilblanc, and Federico Becca. Spin liquid nature in the heisenberg 1-j 2 triangular antiferromagnet. Physical Review B, 93(14))14441, 2016.
Raphael Kaubruegger, Lorenzo Pastori, and Jan Carl Budich. Chiral topological phases from artificial neural networks. Physical Review B, (7(11))1951336 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Dmitrii Kochkov, Tobias Pfaff, Alvaro Sanchez-Gonzalez, Peter Battaglia, and Bryan K Clark, Learn- ing ground states of quantum hamiltonians with graph networks. arXiv preprint arX:v:2110.06390 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-11055 2012.
Yoav Levine, Or Sharir, Nadav Cohen, and Amnon Shashua. Quantum entanglement in deep learning architectures. Physical review letters, (22(6))0 2019.
Xiao Liang, Wen- Yuan Liu, Pei-Ze Lin, Guang-Can Guo, Yong-Sheng Zhang, and Lixin He. Solving frustrated quantum many-particle models with convolutional neural networks. Physical Review B, 98(10):104426, 2018.
Xiao Liang, Shao-Jun Dong, and Lixin He. Hybrid convolutional neural network and projected entangled pair states wave functions for quantum many- many-particle states. Physical Review B, 103(3): 035138, 2021.
Di Luo and Bryan K Clark. Backflow transformations via neural networks for quantum many-body wave functions. Physical review letters, 122(22):226401, 2019.
William Lauchlin McMillan. Ground state of liquid he 4. Physical Review, 138(2A):A442, 1965.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Ieml, 2010.
Yusuke Nomura. Helping restricted boltzmann machines with quantum-state representation by restoring symmetry. Journal of Physics: Condensed Matter, 33(17) 2021.
Yusuke Nomura and Masatoshi Imada. Dirac-type nodal spin liquid revealed by refined quantum mody-body solver using neural-network wave function, correlation ratio, and level spectroscopy. Physical Review X, 11(3):031034, 2021.
Yusuke Nomura, Andrew Darmawan, Youhei Yamaji. and Masatoshi Imada. Restricted boltzmann machine learning for solving strongly correlated quantum systems. Physical Review B, 96(20): 205152, 2017.
11

---

Under review as a conference paper at ICLR 2023
Román Orús. A practical introduction to tensor networks: Matrix product states and projected entangled pair states. Annals of Physics, 349:117-158, 2014.
Lorenzo Pastori, Raphael Kaubruegger, and Jan Carl Budich. Generalized transfer matrix states from artificial neural networks. Physical Review B, 99(16):165123 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Christopher Roth. Iterative retraining of quantum spin models using recurrent neural networks. arXiv preprint arXiv:2003.06228, 2020.
Christopher Roth and Allan H MacDonal.. Group convolutional neural networks improve quantum state accuracy. arXiv preprint arXiv:2 104.00004 2021.
Hiroki Saito. Solving the bose-bubbess model with machine learning, Journal of the Physical Society of Japan, 86(9):093011, 2017.
Hiroki Saito. Method to solve quantum few-body problems with artificial neural networks. Journal of the Physical Society of Japan, 87(7):074002, 2018.
Hiroki Saito and Masaya Kato. Machine learning technique find quantum many-body ground states of bosons on lattice. Journal of the Physical Society of Japan, 87(1):014001, 2018.
Anders W Sandvik. Finite-size scaling of the ground-state parameters of the two-dimensional heisenberg model. Physical Review B, (6(18):16778 1997.
Ulrich Schollwöck. The density-matrix renormalization group the age of matrix product states. Annals of physics, 326(1)966-192, 2011.
HJ Schulz, TAL. Ziman, and Didier Poilblane. Magnetic order and disorder in the frustrated quantum heisenberg antiferromagnet in two dimensions. Journal de Physique I, 6(5):675-703, 1996.
Or Sharir, Yoav Levine, Noam Wies, Giuseppe Carleo, and Amnon Shashua. Deep autoregressive models for the efficient variational simulation many-body quantum systems. Physical review letters, 124(2):020503, 2020.
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
James Stokes, Javier Robledo Moreno, Eftychios A Pnevmatikakis, and Giuseppe Carleo. Phases of two-dimensional spinless lattice fermions with first-quantized deep neural-network quantum states. Physical Review B, 10252)):05112,, 2020.
Attila Szabó and Claudio Castelnovo. Neural network wave functions and the sign problem. Physical Review Research, 2(3):03307,, 2020.
Peiyuan Teng. Machine-learing quantum mechanics: Solving quantum mechanics problems using radial basis function networks. Physical Review E, 98(3):033305, 2018
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7794-7803, 2018.
Steven R White. Density matrix formulation for quantum renormalization groups. Physical review letters, 69(1):228,3, 1992.
Simeng Yan, David A Huse, and Steven R White. Spin-liquid ground state of the s= 1/2 kagome heisenberg antiferromagnet. Science, (126603)):113311111 201
Li Yang, Wenjun Hu, and Li Li. Scalable variational monte carlo with graph neural ansatz. arXiv preprint ar arXiv:2011. 12445,, 2020a.
12

---

Under review as a conference paper at ICLR 2023
Li Yang, Zhaoqi Leng. Guangyuan Yu, Ankit Patel, Wen-Jun Hu, and Han Pu. Deep learning- enhanced variational monte carlo method for quantum many-body physics. Physical Review Research, 2(1))012039, 2020b.
Han Zheng, Zimu Li, Junyu Liu, Sergii Strelchuk, and Risi Kondor. Speeding up learning quantum states through group equivariant convolutional quantum ansätze. arXiv preprint 12.061,1,, 2021.
13

---

Under review as a conference paper at ICLR 2023
A
SUMMARY OF TEST LATTICES
Table 5: Test lattices settings with different lattice types, system sizes and J2 values.
Lattice
Number of Nodes
J2
36
0, 0.5
Square
100
0, 0.5
32
0, 0.2
Honeycomb
98
0, 0.2
36
0, 0.08, 0.125
Triangular
108
0, 0.125
36
0, -0.02 0
Kagome
108
B TRAINING ALGORITHM
Algorithm Training Algorithm of Lattice Convolutional Networks
1: Input: Lattice structure L. number of spin sites N, Lattice convolution network who with trainable parameter 0, learning rate a, Markov Chain batch size B, initial annealing step 8, measure step m
2: Set different random seeds for each Markov chain
3: Randomly initialize spin configurations
[+] BxN with equal number of +1 and -1
4: Ć < Metropolis Hastings Metropolis-Hasttngg((( C, s)
5: repeat
6: Etotal 0
7: for = to do
8: Exstal x- Erstal + Ej Hij
v(e)) p(e.)
9:
end for
10:
Eo +- Eiotal
$ aV E.
12:
C Metropolis-HastinssSie č, m)
13:
until Eo is converged
C VARIATIONAL MONTE CARLO
As discussed in Section 2.2 we use Variational Monte Carlo approach to optimize the variational ansatz by minimizing the statistic expectation of system energy. The equation is shown below:
(1b Hw)
En (1la)) (cal H(c)) (c/l1)
(1b1))
(w/1)
((())) EE
Hig #c;) vee))
(7)
Σ, (c:)2
Σ His a (cj) (c))
= E
where the energy expectation computed over the probability distribution D(c) Σ. [. ple)) and E, M4 stc.) refereed to local energy. Specifically, at each optimization step, we first Markov-Chaie Monte Carlo (MCMC) method sample system configurations ; from target
14

---

Under review as a conference paper at ICLR 2023
distribution, which is given by the neural network called variational wave function. Then stochastically estimate the gradient V [Kochkov et al. 2021) to update network parameters w. Mcanwhile evaluate energy expectation in every optimization step by taking average of all the local energy associated with each sampled configuration. The matrix multiplication between and (15) can be performed efficiently because of the sparseness of Hamiltonian, which is determined by the lattice topology and the specific quantum system. Given a quantum system of N spin-1/2, the dimension of H 2 however the typical number of nonzero values in each row is only of order N.
D IMPLEMENTATION DETAIL
We use the same CNN architecture for all lattices. The convolution operation in each CNN layer replaced with the proposed lattice convolution according to the lattice type. Pre-activation is not used for triangular lattices. For kagome lattice size 108, mask processing after convolution not used.
We implement our models and VMC procedures in PyTorch (Paszke et al. 2019) and all models were trained on NVIDIA RTX A6000. For efficiency we implement the VMC procedures with GPU parallelization. A batch of configurations intervals to minimize the correlation between consecutive samples where the interval equal to the size of the system. We further initialize each MCMC chain with different random seed to maximize independence between chains. During training. we use the stochastic gradient estimated from each mini-bated of samples and optimize the models with the Adam optimizer (Kingma and Ba 2015). The optimization generally converges within 30,000 steps. We save the model with the lowest stable training energy for testing, where the energies in neighboring steps have small variance. During testing, the energy is estimated from 200,000 configurations sampled from equilibrated Markov chains. Hyperparameters for optimization can be found Appendix []] To estimate the stochastic error bars, we group Markov chains into 100 bins. The error bar is computed as the standard deviation of average energies from each bin.
sampled in parallel at each step. Samples kept
E SPECIAL KERNEL. DESIGN
We can design special convolution kernel structures based on different repetitive local patterns of different lattices, where each type of kernels only capture all the nearest neighbor nodes of the center vertices.
Honeycomb
Triangular
Kagome
Figure 4: Special Kernel Design.
Honeycomb. We consider the convolution kernel that covers the nearest neighbors. For every vertex the honeycomb lattice, its nearest neighbors always form an equilateral triangle. However, for two adjacent vertices, the orientations of the triangles formed by their neighbors different. Specifically, two neighbors can be related by a 180-degree rotation. As a result, as shown in Figure [[a we rotate the convolution kernel by 180 degrees when going from vertex one of its neighbors.
Triangular. For each center vertex, all of its nearest neighbors form a hexagon and this pattern is naturally design hexagon shape convolution kernel to
repetitive across the whole lattice. So capture this pattern, as shown Figure [4b] practice, we implement this kernel by masking two positions of the square kernel the diagonal cornrr
15

---

Under review as a conference paper at ICLR 2023
Kagome. We observe that Kagome lattice has three different repetitive local patterns across the space, where each pattern can capture nearest neighbors around the center vertices in the pattern. So we design three different kernels corresponding to these three patterns, shown in Figure 4(c) And these three kernels do not share weights with each other.
F PERIODIC PADDING VERSUS ZERO PADDING
periodic padding is used before convolution in order to consider the periodic boundary condition for lattices. We compare the performance between periodic padding and zero padding, shown in Table which shows that periodic padding is essential for achieving better results.
As discussed in Section 3.3
Table 6: Performance comparison between periodic padding and zero padding.
Periodic padding
Zero padding
Lattice
Size
J2
-0.5516(1)
00.5511
Honeycomb
32
0.2
-0.4563(1)
-0.4542(3)
-0.5601(4)
-0.5109(6)
Triangular
36
0.08
-0.5273(4)
-0.5095(S)
0.125
-0.5126(6)
-0.5091(5)
-0.4367(4)
-0.4356(5)
Kagome
36
-0.02
-0.4384(5)
-0.4349(4)
G QUANTUM STATE OVERLAP AND ENERGY
As described in main experimental results, our goal is to identify the ground-state. There could be dramatic improvements in approaching true ground-state even though the corresponding energy value improvement is small. We conduct an experiment on small kagome lattice with 12 nodes to clarify this, where true ground state can be obtained. shown in Figure we can observe that even though energy accuracy has 0.9% improvement (from -0.4492 to -0.4533). the ground state overlap (up to 1) can increase by 10% (from 0.9087 to 0.9971).
H MODEL CAPACITY COMPARISON
In this section, we compare the number of model parameters between Lattice Convolutional Networks (LCN) and GNN (Kochkov et al., 202T) model over different lattices, as shown in Table For triangular and kagome lattices with 36 nodes, LCN with only 0.29M and 0.49M trainable parameters is able achieve better performance than the GNN model with 0.86M parameters. And on honeycomb and square lattices, LCN with 0.36M and 0.28M parameters yields same results with the GNN model. So we argue that LCN is more expressive and parameter efficient than GNN model on various small lattices.
As for the lattices with larger nodes, the number of parameters in LCN increased, which is caused by the final MLP layer and boundary alignment operation the augmented lattices. Since we flatten the final feature maps into a vector which is then processed by a MLP layer, the number parameters will increase with the size of the network input. Also, need to zero-pad padd the augmented lattices into parallelograms due to the irregular boundaries of original lattice geometry structures, which then increases the size of the input feature map.
16

---

Under review as a conference paper at ICLR 2023
-0.444
1.0
-0.446
0.8
Overlap: pred pred Wead)
Energy per site
-0.448
0.6
-0.450
0.4
-0.452
0.2
2500
5000
7500
10000
12500
15000
17500
20000
Training iteration
Figure 5: Improvement state overlap on kagome lattice
Table 7: Model parameters of Lattice Convolutional Networks and GNN. For square lattice of size 100, the number for LCN is counted with 4 SE-Non-Local layers. We also use a 3-layer model which slightly smaller and contains 1.0M parameters. See Appendíx for details.
Lattice
Size
LCN (Ours)
GNN
36
0.28M
0.86M
Square
100
1.1M
0.86M
32
0.36M
0.86M
Honeycomb
98
1.9M
0.86M
36
0.29M
0.86M
Triangular
108
1.9M
0.86M
36
0.49M
0.86M
Kagome
108
4.6M
0.86M
I NETWORK COMPONENTS
Squeeze-and-Exiine Excitaaton Block. The process of SE-block can be represented as (Hu et al. 2018):
2. Fsg (u.)
ΣΣ ue ue(i)))
W
(8)
8 Fez (z, W) = (g(2, W)) = (W26
X. Fscale (ue, 3c) selle
where u. HW denote the c-th output feature map of convolution operator. 2, denote the e-th element of channel descriptor z squeezed from u, 8 refers to ReLU (Nair and Hinton 2010) activation function and refers sigmoid function. WN R CC and W₂ E - is the dimension reduction factor. The final output x, computed by channel-wise multiplication between 8. and ug-
17

---

Under review as a conference paper at ICLR 2023
Non-Local Block. Response at position, denoted as ys, is computed weighted of features at all positions. The output of each position is formulated as (Wang et al. 2018)
f (x,, = ((x))² Fo(s))
1 yi ((x)
Σ (x,, (x)
2. = W.ys +2,
where y is a linear embedding that transform the input feature map x: y (x:) W And / (x,, xj) is embedded gaussian. 0 (x:) Wøx; and (x,) = Wax, are two embeddings to compute dot product similarity. ((x) = Ex₁ f (x, xy) is normalization factor. z, the final output of this block position by adding residual connection.
J HYPERPARAMETERS
Table 8: Hyperparameters for optimization. The learning rate is multiplied by 0.1 every decay step. During training, gradient norm is clipped from beginning for square and honeycomb lattices. For triangular and kagome lattices, gradient norm clipped after first learning rate decay. For all experiments, we use 3x3 convolution kernel and 64 output channels for each layer.
Number
Learning rate
Gradient norm clip
Lattice
Size
J2
SE- Non-Local layers
Batch size
Learning rate decay steps
0
2
500
le-3
20000, 40000. 60000
0.5
2
500
le-3
20000, 40000, 60000
Square
200
Se-4
8000, 12000, 16000
100
0.5
200
Se-4
8000, 12000, 16000
2
500
le-3
20000, 40000, 60000
32
Honeycomb
0.2
2
500
1e-3
20000, 40000, 60000
0
4
100
7e-4
8000, 12000, 16000
0.2
4
100
7e-4
10000, 16000, 22000
1000
lc-3
every 4000
0.08
1000
le-3
every 4000
Triangular
0.125
1000
le-3
every 4000
2
200
le-3
every 4000
None
108
0.125
2
200
le-3
every 4000
None
0
2
1000
le-3
every 4000
Kagome
-0.02
2
1000
le-3
every 4000
108
2
200
1e.3
every 4000
K ABLATION STUDY
K.1 MASK OPERATION
As described in Section 3.3 we reset all vertices used for the boundary alignment and the periodic padding to zero after each convolution. The vertices used for boundary alignment do not belong to the original lattice area and periodic padding, they are only used for transforming original lattice area to regular shape where regular square kernel can be applied on the boundary. Also the vertices used for periodic padding do not belong to original lattice area. So this motivates us to use mask operation. But in practice we found mask operation could be optional choice but it indeed has benefit in certain cases. For example, mask operation could improve performance on kagome lattice of size 36, as shown in Table 9
18

---

Under review as a conference paper at ICLR 2023
Table 9: Performance comparison between mask and non-mask
Lattice
Size
J2
Mask
No Mask
0
-0.4367(4)
-0.4364(6) -0.4336(4)
Kagome
36
-0.02
-0.4384(5)
Table 10: Full range J2 on 36 sites square lattice. Results for exact energies and CNN are taken from Choo et al. 2019). Results for RBM+PP are taken from Nomura and Imada (022)) Results for RBM+Lanczos are taken from (Chen et al.) 2022) (only J2 0.5, 0.55, 0.6 use one Lanczos step). Our method is the only one without any prior physical knowledge incorporated in the model.
Square
Jj 02
41 0.5
0.55
Exst
Sahalt olll (1og Charees 20r1
0.678872
0,589846
0,529545
0,53310
0,495178
0.40329
0.585457
0,714560
CNN
40.77 6767733
4. 598991191
10.52936(1)
-0514531
-0.00185(1)
4.4966222
4040025(
-0 -0.58500:11
-0 -0.1131(())
RBM
- (16766612)
-0, 599044(3)
(.529687())
2.51552
-0.50376(3)
0.4552244)
-0.0313155
4.59641
-0.1129(1))
RIIM+4
Nortre ard mino 2021
0.32972611
-0305700111
0
LCN (oun)
0.67788))
0.5990X))
-0.52951))
0.3151()
..50222))
.0.491622
.0.45402)
-0.585112)
.0.11993)
L FULL RANGE J2 ON SQUARE LATTICE
For completeness, we compare with other ibysics-incorporated methods targeting on square lattice. We conduct experiments for full range J2 on square lattice of size 36, as shown in Table TO Compared with CNN (Choo et al. 2019), our method consistently performs well on full range of J2, especially for frustrated regime (J2 R 0.5) and small J2 while without using any prior physical knowledge. For large J2, CNN is better than ours, it might be because they enforce C₁ symmetry which is important in striped order phase at large J2 (Choo et al., 2219).. Besides, CNN incorporates Marshall sign rule which defines the sign structure the ground state wavefunction for square lattice at J1=0 or J2=0. Marshal sign rule only works for bipartite graph (such square lattice) and non-frustrated regimes. the prior sign rule is violated (J2=0.5), CNN result is worse than ours, as expected.
The latest SOTA methods RBM+PP (Nomura and Imada 2021 and RBM+Lanczos (Chn et al.) 2022) have very good results on square lattice. RBM+PP is a hybrid method which heavily relies intense physical knowledge (GNN (Kochkov al. 202T) uses some results from RBM+PP as reference energy). RBM+PP combines the restricted Boltzmann machine and the pair-product wavefunction. The pair-produce wavefunction has been used in physies for long time and is known that the pair-product wavefunction excellently captures the ground state of the Heisenberg model on a square lattice. Therefore the combination RMB+PP very effective on square lattices. However, it is unclear how it performs on other lattices. RBM+Lanczos incorporates spin flip, translation, and lattice point group symmetries for square lattice in the network. And uses Lanczos iterations to further improve results. But computation cost of Lanczos method increases dramatically with Lanczos steps, and Lanczos correction will become smaller for larger systems.
M GNN WITH AND WITHOUT SUBLATTICE ENCODING.
To illustrate the necessity of using hand hand-crafted sublattice encoding in the GNN model, we try to implement the GNN model based details provided in Kochkov et al. (202I) and reproduce the results. We conduct ablation study on sublattice encoding and summarize the results in Table TT For triangular and kagome lattices, cannot get valid results without using sublattice encoding (either too large or too small). One possible reason is that the distribution of learned wave function is very hard for accurate sampling. On square and honeycomb lattices, sublattice encoding is essential for achieving better results. So we can draw the conclusion that the hand-crafted sublattice encoding is important for GNN model.
19

---

Under review as a conference paper at ICLR 2023
Table 11: Reproduced GNN performance with and without sublattice encoding.
GNN with sublattice
GNN without sublattice
Lattice
Size
J2
Square
36
0.0
-0.6776(6)
-0.6536(11)
Honeycomb
32
0.2
-0.4551
-0.4498(48)
0.0
-0.5571(5)
N/A
Triangular
36
0.08
-0.5253(5)
N/A
0.125
-0.5119(6)
N/A
Kagome
36
0.0
-0.4345(5)
N/A
20


Reviews:
[{"review_id": "mHhRwwG5Qx", "reviewer": "Reviewer_maSg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates application of neural networks based on lattice convolutions as variational ansatz for a quantum many body problem on a set of lattices (including non-square variations such as triangular, honeycomb and kagome). Authors test their approach on commonly studied Heisenberg models with up-to second nearest neighbor interactions. They compare their results to other methods compatible with non-square geometry such as graph neural networks finding their method competitive.", "summary_of_the_review": "Overall a solid paper that achieves good results on a set of challenging benchmark problems (that are still an open research topic). The proposed method would be of value to the research community interested in applying machine learning to problems in many body quantum physics, as well as could be of interest to a wider ML for science audience.\n\nThe paper features good results and authors make logical & clear efforts to support the claims. There are a few additional ablation studies that could improve the paper from a physics insight standpoint: (on small system size on Kagome lattice including nonlocal information seems to noticeably help - does this still hold for larger system sizes?)\n\nOne aspect in which I disagree with the authors is their structural comparison with graph neural networks. Both the GNN and the proposed LCN options attempt to take advantage of the lattice symmetries to make the ansatz more suitable for the problem at hand. For a general lattice symmetry this comes at a price of limited representability (assuming standard CNN-based or graph network based implementations) which in general hurts the variational energies. Both GNN and the proposed LCN address this tension by selecting a sublattice, which in GNN is encoded as additional input features and in the LCN case is determined by the choice of the unit cell for the underlying convolutions (both uses the same heuristic that does not involve any “domain knowledge”). The real advantage of the proposed method, from my perspective, is that the paper uses regular convolutions that are likely to be much more efficient on modern hardware than message passing steps, enabling faster iteration and better variational energies given the same compute budget.\n\nI’m also a believer that statements of “primacy” and “novelty” at ICLR submissions are generally not load bearing and should be discouraged.\n\nMinor comments, confusions, suggestions:\n* Related work: “to augment system configurations in order to respect underlying quantum symmetry” - I don’t think graph neural networks have any “quantum symmetries”, am I misunderstanding?\n* Introduction: “suffers entanglement problems”\n* Consider switching network ablation study (ideally extended :) ) from the appendix and reproduction of the GNN ablation study, since the former is new and provides an interesting piece of evidence.\n\nSuper minor nits:\n* Variational principle in quantum mechanics (Lanczos method has been shown up to ~48)\n* Background and related work: “all combinations of spins form a basis” (remove “a set”)?\n", "strength_and_weaknesses": "Strengths:\n* Authors provide results on a range of model problems and obtain competitive variational energies\n* The proposed method compares favorably to relevant baselines\n* Proposed architecture should be able to utilize hardware accelerators well\n\nWeaknesses:\n* Additional modeling choices: (squeeze-and-excitation block, non-local block, SE-non-local layer) are studied in the appendix and mostly on small system size. As an exciting addition, it would be great to see whether this holds true for larger problem (108 spins) (interesting regardless of the outcome). Does this ablation depend on the underlying physics of the system? (e.g. ordered vs disordered)\n* The primary difference with respect to a GNN baseline is misrepresented/obscured (see main review for details)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ODBs6-OuLnl", "reviewer": "Reviewer_aQ26", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the context of Variational Monte Carlo for finding the ground state of quantum many-body systems, convolutional neural networks have proven to be a powerful ansatz for solving square lattices. This paper proposes a method for extending ConvNet quantum states to non-square lattices by projecting them onto square lattices with non-uniform paddings to fill the holes in the projection. Prior works on neural networks for non-square lattices have focused on graph neural networks instead, which relied on hand-crafted features to encode the geometry of the lattice. By building on the success of ConvNets, this work aims to offer a simpler, more principled approach. The experimental results on the challenging J1-J2 Heisenberg model demonstrate that the proposed method can perform as well or better in most cases without relying on any hand-crafted features.", "summary_of_the_review": "While the proposed method is novel and offers a principled solution to a non-trivial problem, the presentation of the results and prior works, as well as the lack of support for the core arguments leads me to recommend rejecting it. It could be that with a major revision addressing the presentation issues, and perhaps improving some of the results that showed GNN to be superior, the paper could be accepted.", "strength_and_weaknesses": "Strengths:\n\n* Simpler more principled novel method that builds on the power of ConvNets, instead of trying to adapt GNN to less conventional settings, and using less external knowledge on the given problem. This could be crucial when tackling more intricate systems without the need to carefully tune the method to them.\n\n* Clearly written.\n\n* Extensive experiments demonstrating that in *most* (but not all) cases lattice ConvNets (LCN) can achieve better accuracy than the competing GNN approach.\n\nWeaknesses:\n\n* It is clear that GNN perform significantly better than LCN in some cases, where LCN results achieve a relative error of ~10^-2 (compared to GNN). This raises questions about whether LCN might lack the needed inductive bias that GNNs utilize in their construction. Given the stated goal of a method that relies less on prior knowledge, these negative results might show that LCNs might not yet fulfill this role.\n\n* There are a few issues with the presentation of the results. For one, the \"reference energy\" column is a bit problematic and requires more careful discussion. Specifically, in many cases there is not yet a consensus that a correct estimate of the energy is presently available (e.g., for J1/J2 = 0.5 for 10x10 lattice), and so the last column in some cases is not really a reference that can be used to evaluate how well LCN solve the task, but is in fact a competing neural network that solves it more effectively. Another issue is that it should be noted that the specific paper on GNN they compare against uses a completely different optimization method (IT-SWO) and not the conventional VMC -- when comparing architectures it is better to ensure all other factors remain the same. Finally, in cases where the competing method is better, it should be in bold-face to emphasize that instead of keeping the entire row in regular font.\n\n* The claim regarding LCN not needing any prior knowledge unlike previous works is a bit misleading. First, because they do incorporate prior knowledge into the architecture, and second, because some of the examples of prior knowledge they try to avoid (Marshal sign rule) are not at all addressed by their proposed method. Namely, the locality in the convolutional layers assumes mostly short-range interactions -- a true departure from the assumption of locality would use only the non-local layers (which is really self-attention by another name). The authors note the use of symmetry in previous works as an example of prior knowledge, but periodic symmetry is part of the LCN architecture -- so it is not clear what is different here. As for the use of the Marshal sign rule, while indeed other works have pointed out the difficulty of learning the sign structure without relying on some prior knowledge or heuristic, this work does not propose any arguments for why LCN helps with this challenging problem.\n\n* Some of the hyperparameters in the experiments are missing: How many channels / kernels were used in each layer? Is the window size of the convolutional layers always 3x3? Are the number of MCMC chains equal to the batch size, or are there fewer chains and multiple samples are produced per step?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3QWMgp-Wt0D", "reviewer": "Reviewer_xp3P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose lattice convolutions to convert non-square lattices into grid-like augmented lattices where regular convolution can be applied. Based on this and self-gating and attention, they design lattice convolutional networks and apply the networks to some experiments.", "summary_of_the_review": "Learning ground-states of quantum many-body systems by deep neural networks is an interesting topic. But it seems that the approach in the paper of adapting the usual CNNs to non-square lattices can be easily replaced by using a linear transformation. The authors should convince the readers why this simple approach does not work. ", "strength_and_weaknesses": "Learning ground-states of quantum many-body systems by deep neural networks is an interesting topic. Adapting the usual CNNs to non-square lattices for replacing graph neural networks is an interesting idea. But a linear transformation should easily transfer a non-square lattice to the square one. Unless I miss something, the problem seems to be easy. The authors should convince the readers why this simple approach does not work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject