Towards LLM4QPE: Unsupervised Pretraining of Quantum Property Estimation and A Benchmark
Authors: Yehui Tang
Hao Xiong
Nianzu Yang
Tailong Xiao
Junchi Yan
Venue: ICLR 2024 spotlight (2024)
Keywords: quantum property estimation
pretraining
finetuning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Estimating the properties of quantum systems such as quantum phase has been critical in addressing the essential quantum many-body problems in physics and chemistry. Deep learning models have been recently introduced to property estimation, surpassing  conventional statistical approaches. However, these methods are tailored to the specific task and quantum data at hand. It remains an open and attractive question for devising a more universal task-agnostic pretraining model for quantum property estimation. In this paper, we propose LLM4QPE, a large language model style quantum task-agnostic pretraining and finetuning paradigm that 1) performs unsupervised pretraining on diverse quantum systems with different physical conditions; 2) uses the pretrained model for supervised finetuning and delivers high performance with limited training data, on downstream tasks. It mitigates the cost for quantum data collection and speeds up convergence. Extensive experiments show the promising efficacy of LLM4QPE in various tasks including classifying quantum phases of matter on Rydberg atom model and predicting two-body correlation function on anisotropic Heisenberg model.

Published as a conference paper at ICLR 2024
TOWARDS LLM4QPE: UNSUPERVISED PRETRAINING OF QUANTUM PROPERTY ESTIMATION AND A BENCH- MARK
Yehui Tang', Hao Xiong', Nianzu Yang', Tailong " Yan
'Department of Computer Science and Engineering, Shanghai Jiao Tong University
Institute of Quantum Sensing and Information Processing, Shanghai Jiao Tong University
'Hefei National Laboratory, Hefei, China
{ychui tang, taxuexh, yangnlanzu, ta a jtu. edu.
ABSTRACT
Estimating the properties of quantum systems such as quantum phase has been critical in addressing the essential quantum many-body problems in physics and chemistry. Deep learning models have been recently introduced to property esti- mation, surpassing conventional statistical approaches. However, these methods are tailored to the specific task and quantum data at hand. It remains open and attractive question for devising a more universal task-agnostic pretraining model for quantum property estimation. In this paper, propose LLM4QPE, large language model style quantum task- tank-agi pretraining and finetuning paradigm that 1) performs unsupervised pretraining diverse quantum systems with differ- ent physical conditions; 2) uses the pretrained model for supervised finetuning and delivers high performance with limited training data, on downstream tasks. It mit- igates the cost for quantum data collection and speeds up convergence. Extensive experiments show the promising efficacy of LLM4QPE in various tasks includ- ing classifying quantum phases of matter on Rydberg atom model and predicting two-body correlation function on anisotropic Heisenberg model.
INTRODUCTION
Estimating quantum system properties such as quantum phase is essential for verifying and evalu- ating quantum technologies (Huang et al., 2020; Gočanin et al., 2022), which is often in the form of many-body problems. Precise estimation of generic quantum systems is challenged due to the exponential complexity inherent in describing quantum many-body systems (Gebhart et al., 2023). Fortunately, physical systems of interest such as those generated by the dynamics of local Hamilto- nians are not generic, since their particular structure guarantees that the full complexity of Hilbert space is in principle not required for their accurate description (Carrasquilla et al., 2019). Ac- cordingly, statistical (including sarning-based approaches have emerged to characterize quantum systems from traditional Density Functional Theory (DFT) (Hohenberg & Kohn, 1964), Quantum Monte Carlo (QMC) (Ceperley & Alder, 1986), to advanced variational methods e.g. Tensor Net- works (TNs) (Orús, 2019) and Neural Network Quantum States (NNQS) (Zhang & Di Ventra, 2023).
There are basically two categories of variational methods for quantum property estimation (QPE). The first category refers to the TNs and NNQS which formulate QPE as an optimization problem where the quantum state is approximately represented by a parameterized wave function. The pa- rameterized wave function is updated by minimizing the expectation values of relevant observable estimators, based on either density matrix renormalization group (DMRG) algorithm (White, 1992) or variational Monte Carlo (VMC) (McMillan, 1965). Afterwards the interested properties can be analyzed by preforming algebra operations on the wave function. Another line of research resorts to neural networks to serve as universal functions for directly approximating quantum system proper- ties (Gilmer et al., 2017; Kawai & Nakagawa, 2020; Xiao et al., 2022), which we call NNQPE. The input to the neural networks is the measurement results of the quantum state, and the output the
Correspondence author. Work partly supported by NSFC (62222607), Shanghai Municipal Science and Technology Major Project (2021SHZDZXX000), SITU Trans-med Awards Research (STAR) 20210106.

---

Published as a conference paper at ICLR 2024
property of interest. The parameters optimized using gradient descent. The goal of NNQPE is to accurately characterize the properties of the quantum state using as few identical copies and mea- surements possible. Compared with the TNs. this class of methods could more easily display non- local correlations, allowing in principle to capture quantum states with higher entanglement (Huang et al., 2022). Moreover, rather than TNs and the NNQS where additional computational overheads is required to extract the properties given the optimized parameterized wave function, NNQPE can directly predict the properties for unknown quantum states.
However, NNQPE suffers generalization ability issue, especially given limited measurement data for training (Gebhart al., 2023). Although the generalizability could be improved by training the models based on extensive measurement data and corresponding labels, the labeling process, i.e., accurately estimating properties of quantum systems requires computational and memory resources that increase exponentially with the system size (Carleo et al., 2019). In particular, the labeling efforts for quantum systems are intensive. For example, DFT suffers from self-interaction error and delocalization error, making it difficult to represent quantum states with strong correlations (Verma & Truhlar, 2020). The sign problem (Loh Jr et al., 1990) implies that it is intractable for QMC to evaluate properties for large systems or systems with low temperatures (Troyer & Wiese, 2005; Huang et al., 2022). The maximum bond dimensions of TNs for precisely preserving the proper- ties of quantum states such as the entanglement entropy scales exponentially w.r.t. the evolution time (Brandao & Horodecki, 2015). In conclusion, the labeling process is hard to complete classi- cally due to the inherent separation between quantum and classical computing.
Furthermore, despite the significant promise of NNQPE, their application in harnessing advanced machine learning techniques for quantum physics remains in its early stages. Current models of NNQPE are tailored and trained for particular quantum systems and specific tasks. This approach contrasts sharply with the era of Large Language Models (LLMs) (Radford et al., 2018; Brown et al., 2020), which have achieved general-purpose language generation and understanding capabilities. In the realm of L.L.s, pretraining serves as the primary method for capturing general language understanding and afterwards finetuning is adopted to adapt the model to accomplish specialized tasks. This distinction highlights the nascent yet evolving nature of applying sophisticated machine learning strategies within the quantum physics domain.
In fact, with the increasing scale of the quantum devices, a vast amount of quantum data are pro- duced by quantum measurement (Brydges et al., 2019). Such data holds intricate details about the system. An open question is designing a versatile model, which undergoes extensive pretraining to master these quantum intricacies. The success of deep learning in handling high-dimensional data sheds lights on answering this question. First, the sheer volume of quantum data from measurements allows for the extraction of meaningful patterns and representations (Anshu & Arunachalam, 2024). Second, the universal approximation capabilities of neural networks suggest that given sufficient data and computational resources, it's possible model the complex, nonlinear relationships inher- ent in quantum systems (Carleo et al., 2019; Gebhart et al., 2023). Lastly, the task-agnostic nature of pretraining (Liu et al., 2023) aligns with the quantum realm's diversity, where a single model can learn hidden features across various systems and physical conditions. This feasibility is further supported by the principle of transfer learning (Weiss et al., 2016), where knowledge gained one context can significantly benefit task-specific applications.
In this paper, we introduce an LLM-style task-agnostic pretraining model for Quantum Property Estimation named LLM4QPE. This model is pretrained by leveraging vast (unlabeled) quantum data, across diverse quantum systems of the same family govern by different physical conditions. For the downstream tasks, finetune LLM4QPE quantum phases of matter and predicting two-body correlation function. We also consider two fam- ilies of quantum model including the Rydberg atom model and the anisotropic Heisenberg model. The results show its promising power for tackling QPE problems especially in scenarios with limited data availability. The contributions are:
two typical QPE tasks including classifying
1) Departure from most existing supervised learning QPE models reliant on restricted, task-specific labeled quantum data, we propose LLM4QPE, to our best knowledge, the first LLM-style model for quantum property estimation. Its unsupervised pretraining is fulfilled by maximizing the expected log likelihood of measurement bit strings, which is entirely unsupervised and agnostgc.
2

---

Published as a conference paper at ICLR 2024
2) We develop the novel architecture of our LLM4QPE model. Specifically, embed the batch-style discrete measurement records to a continuous space, a trainable LSTM embedding layer is attached to the transformer decoder. The for handling diverse quantum data stemming from experiments under varying physical conditions, enabling prediction of the property of quantum systems of the same family.
architecture provides innate framework
3) We collect a set of quantum data from simulations for unsupervised pretraining and supervised finetuning. For pretraining, the dataset consists of quantum state measurement records, the size of which scales linearly w.r.t. the system size and the number of measurements, along with the values of physical condition variables determining the evolution of quantum systems. Downstream tasks utilize set of data generated from quantum systems of the same family, with additional system properties serving as labels for tasks like phase classification and correlation prediction.
4) We verify the superiority of our approach by empirical studies on two QPE tasks: classifying quantum phases of matter on Rydberg atom model and predicting two-body correlation function on anisotropic Heisenberg model, given limited measurements on resource-limited device.
PRELIMINARIES OF QUANTUM STATE AND QUANTUM MEASUREMENT
We introduce basic concepts of quantum computing. Please refer to (Nielsen & Chuang, 2010) for more details. We put the details related work to Appendix A.
Quantum State and Density Operator. The quantum bit named as qubit is the basic unit of the quantum system. We call the ensemble of all qubits in (sub)system the quantum state. The qubit is in superposition and becomes deterministic once the measurement is performed on it. How a quantum state is described mathematically depends on the chosen basis state. For example, by using two orthogonal computational basis states' [0] = [b_ and [1] = [f]. one qubit can be described mathematically as linear combination (の) = a0) + 311) = [3] in the space C2, where a. B E C are the amplitudes satisfying la|² + [3] 1. An alternate formulation for describing the quantum state is possible using& a tool known as the density operator or density matrix. For example, the density matrix of (0) is Po = [0}(0] (b where (0) denotes the conjugate transpose of [0). For generic L-qubit quantum state, it can be described by the so called wave function:
M
M
()) = Σ
((I,....,OL)) 01))o1,.......... OL.)
(1)
d==1
where L ZZ > C maps fixed configuration o (o1,..., of L qubits to complex number satisfying EM EM (o1...., 12 1, and 0; E (1....,,, M) }
is one of the M possible outcomes by performing quantum measurement on the i-th qubit. The wave function is formulated
in complex Hilbert space where the vector representation of the quantum state (N)) CM4 and its
density matrix (xb) E CM xM which becomes astronomical for large L.
Quantum Measurement. It converts some of the quantum information into classical form (for fur- ther processing), as described by a set of measurement operators {Om \M satisfying Σm Om = I, where M is the total number of operators. Measuring qubit leads collapse of the wave function and produces potentially yield different outcomes. The possible outcomes correspond to the indices m of measurement operators. Concretely, upon measuring the qubit p. the probability of getting the result vn is given by p(m) tr(p0m) m) For a quantum state with L qubits, the common strategy is to measure each of the qubits in parallel (Leibfried et al., 1996; Jullien et al., 2014). According to the born rule of quantum mechanics, such measurement procedure outputs measurement string ( where og E (I....,M with probability (o)....,, as given in Eq.
3 LLM4QPE
3.1 OVERVIEW
As shown in Fig. 1, our model involves two steps: pretraining and finetuning. For pretraining, the model is fed with unlabeled D,,, and undergoes fully unsupervised training. Subsequently, the pretrained parameters transferred to the supervised finetuning phase, where all the parameters
Computational basis states are also referred to as the 7-basis states in some literature.
3

---

Published as a conference paper at ICLR 2024
Measurement
KI -ildeerggeeee Loss Cunmu
Classification Negularization Exp.
Ground
Records
S
I Lincar Projection
Token
Feature Aggregation
Embeddiigss
E.
E:
R
E
Linear Projection & Softmax
Position
Embeddiings
R2
Layer Normalization
Transfer
Condition
Layer Normalization
Fere-foorwad Network
Embeddiigs
Ee
Food-forward Network
Multi-head Attention
Oatput
Embeddings
Masked Multi-head Atteation
makee
sabotttnn poo
Physical
Conditions
Unlaheled Measurement
Records and Physical Conditions
Unseen Quantum Data
a) Input Embedding
b) Pretraining
c) Finetuning
Figure 1: Pretraining and finetuning of LLM4QPE. a) The output embeddings are the summation of token embeddings, condition embeddings and position embeddings. Three embeddings corre- spond encode discrete measurement records, continuous physical variables and qubit positions, respectively. The token embeddings are replaced with the LSTM embeddings while finetuning. b) The main part of the model is a multi-layer transformer decoder. Pretraining is entirely unsuper- vised. The output target is to approximate the classical distribution of the wave function. e) The model for finetuning and pretraining share the same structure. The pretrained parameters are trans- ferred to the finetuning stage and updated towards task specific supervised loss.
updated using labeled data D, for various downstream tasks with their task-specific supervised losses. Finally, we evaluate our LLM4QPE using dataset De. Each downstream finetuning model possesses separate parameters, even though they initially share the same pretrained parameters. One of the most notable aspects of our model is the consistent structural similarity between pretraining and finetuning. with only few small modifications when handling different downstream tasks.
The description of the quantum data is discussed in Sec. 3.2. We make an analogy between quantum data and text that, each measurement outcome o of a qubit is analogue to the token, and the number of the possible outcomes M is likely to the vocabulary size [V]. A measurement string Ø, which resembles the sentence in texts, is projection of the entire quantum system with correlative effects among them. The collection of measurement records R comprised of many measurement strings from various physical conditions are akin to the corpus gathered from various sources and genres. In fact, these have also been mentioned implicitly in (Sharir et al., 2020; Hibat-Allah et al., 2020; Cha et al., 2021; Zhang& & Di Ventra, 2023). Yet existing works are still confined to the single task for training and testing, involving
pretraining. Our model, in contrast, draws inspiration from L.MMs to handle quantum data. Specifically, the data type and data collection strategies are described in Sec. 3.2 and more details can be found in Appendix
Given the generated datasets, we first discuss how to unsupervisely pretrain LLM4QPE in Sec. 3.3. Afterwards the pretrained parameters are updated towards a supervised loss for different tasks, as presented in Sec. 3.4.
3.2 DESCRIPTION OF THE QUANTUM DATASET GENERATED FROM SIMULATION
We first provide the definition of the quantum dataset in Def. in which the procedures of quantum dataset generation are provided. An casy-to-understand flowchart also provided in Fig, 2.
Definition 1 (Quantum Dataset). The quantum dataset is described D {s.). Each sample S; = (R,,, Ci, P:) contains the measurement records R,, the physical condition variables e; and the (optional) system property variables pt. Let L denote the number of qubits, K represent the number of copies of each quantum state and M denote the number of possible outcomes by performing m measurement on a single qubit. We explain their meaning detail below.
c; E CRC represents the physical condition variables controlling the evolution of the quantum system. These variables can be directly obtained when initializing quantum experiments. The types of the variables could be system size, coupling strength of Hamiltonians, etc.
2)
The measurement records, denoted as R, E z are outcomes generated by quantum mea- surement. A quantum state is generated by evolving the system under a fixed physical condition
4

---

Published as a conference paper at ICLR 2024
Arbitrary K-outcome quantum measurements
Messure the by given (On)a
sample with probubility tr(pOm.)
Qu antatnn Evolution
c)
Quhlis
Data Type
Messurement Records
collect
Plysiical Conditions
Database
System Properties
Physical Conditions
Mensurement Strings
Figure 2: Process of generating the quantum dataset. a) For each qubit of the quantum system, we perform quantum measurement using operators {Omm JM m== and obtain an integer outcome m with probability p(m). b) Consider the quantum system govern by different physical conditions. Quantum measurements are performed on an ensemble of identical quantum states evolved under each of fixed physical conditions. Measurement can be done parallel for all the qubits of single copy of the quantum state and outputs a measurement string, This process is applicable and feasible to existing digital and analog quantum computers. e) The collected data are structured and packed into a series of tensors, which can be efficiently stored into classical devices and easy to process.
specialized by Ct Afterwards quantum measurement is performed independently each qubit in parallel using a set of measurement operators [0 )m= Performing measurement on qubits results in measurement string, represented σ (. ... OL) where each or {1,.... M) The measurement procedures above are repeated K times for each copy of the quantum state. Finally, collect K L measurement outcomes and store them within R,,
3) (Optional) Certain system property P: E IR. represents the statistics of the quantumn system conditioned on Cú such as the quantum phase, correlation function, entanglement entropy, purity, etc. The exact values of Pt can be calculated by classical post-processing by analyzing the either the wave functions or measurement statistics. We treat these properties as supervised labels which used for finetuning the model.
It should be mentioned that the process of quantum dataset generation above is closed to Wang et al. (2022). The difference is that LLM4QPE requires additional ground-truth labels of system proper- ties for finetuning, rather than the suggestions of Wang et al. (2022) in which the authors propose reconstruct the quantum state by unsupervised learning on measurement records, afterwards clas- sical shadow (Huang et al., 2020) is required to predict specific quantum properties. The two step strategy often introduces additional overheads. Furthermore, our experiments indicate that parame- ters in LLM44PF are specifically optimized for corresponding objectives such quantum phase of matters and correlation function, which often leads to superior performance in our numerical results.
3.3 UNSUPERVISED PRETRAINING
Unlike the previous studies (Czischek et al., 2022: Zhang & Di Ventra, 2023) which consider the pre- training warmup process to find suitable initialization for model's parameters and then finetune the model on the specific system with the same learning objective as pretraining. Instead, LLM4QPE regards the pretraining as the avenue master the quantum intricacies across different systems of the same family. The pretrained parameters can be transferred towards various downstream tasks. LLM4QPE is pretrained in fully unsupervised manner, as illustrated in Fig. lb.
Quantum Data for Pretraining. The quantum dataset D, [R,, J used for pretraining is constructed using the strategy discussed in Sec. 3.2. Here we discuss how to reorganize the data to adapt to LLM4QPE's unsupervised pretraining. Let Kp be the number of measurement strings used for pretraining. We stack all the input measurement records { Nr along the first dimension
and output Ejn X, where each row is measurement string σ e z We also construct the matrix Cin E Kx where each row is the values of physical condition variables e₁ e RC For both the Rydberg atom model and the anisotropic Heisenberg model, we fix Np 100 and Kp = 1024. For each training iteration, we randomly sample Bp rows of E, and C..- Such that the n.
input of the model is Co) E Em, Cp E Cin) with batch size Bp-
5

---

Published as a conference paper at ICLR 2024
Input Embeddings. As shown in Fig. la, consider three types of embeddings input to capture the hidden patterns of the quantum system: token embeddings, condition embeddings and position embeddings. Since each element of the measurement string σr, is discrete integer a E {l..... ......... M) which resembles to the token in NLP, we use learned embeddings to convert the measurement string Or with additional start token s and output the token embeddings E, E I2 ((L+1)x where is the feature dimension. We empirically find that encoding the physical condition into the model can further improve the performance. A Feed-Forward Network (FFN) with one hidden layer is used to embed the physical condition C₁ into the feature vector E. E B It is treated sentence-level embedding which will be added to all of the L measurement tokens, and we call it the global embedding. Subsequently, the input embeddings are the (broadcasting) summation Eout = E, + E. + E, where Ep is the positional embeddings as the same as (Vaswani et al., 2017). Eour is then processed by deeper layers in the discussion below.
Model Architecture. As depicted in Fig. 1b, the main part of LLM4QPE is a multi-layer trans- former decoder which originates from (Vaswani al., 2017). The input is the embedding E and the output is H E (L+1)x which are high-order representations of all the measurement strings and the conditional variables in a batch. Please refer to (Vaswani et al., 2017) for more details on transformer. For pretraining, given fixed qubit configuration o (01.... (o...... L.). LLM4QPE at-
tempts to approximate the classical distribution p( oL) = | (o)..... oL) in Eq. 1. Such joint distribution is approximated by factorizing it into a product of conditional probabilities:
4
=
- ,1,
(2)
1=1
The parameters are optimized by minimizing the average negative log- likeliilldooo loss:
unsuup
Σ -logp(ot.....zle)) By (arcc)ED
(3)
which corresponds to the maximization of (conditional) likelihoods concerning the observed mea- surement outcomes. Pretraining is entirely unsupervised, enabling the model to be trained on ex- tensive quantum data that encompass a wide range of physical conditions. To maintain the physical validity that restricts the output distribution to be normalized, a general strategy is employed to fix the last layer as the linear projection with softmax activation function, such that the output distribu- tion satisfies M
Eot=1 p(a = (see Appendix C for proof).
3.4 SUPERVISED FINETUNING
The self-attention mechanism in the transformer allows LLM4QPE to model wide range of down- stream tasks, whether it involves classifying quantum phases of matter or predicting the entangle- ment entropy of quantum states. This adaptability achieved simply by replacing the relevant inputs and outputs as needed. Rather than the two-step model (Wang et al., 2022) that uses the pretrained model to generate new measurement records conditioning on the physical variables and then predicts quantum properties based on classical shadow (Huang et al., 2020). LLM4QPE is an end-to-end task-agnostic pretrained model to provide property estimation for the quantum system.
Quantum Data for Finetuning and Input Embeddings. The dataset Dj = {RR, Pý p,) N, are generated using the random seed different from the seed for generating D,, Then we split D, to construct train/test dataset D,/D.. It is ensured that the sampled physical conditions for pretraining will not appear in finetuing, i.e. Cj & {c,} for E (1....,N, for finetuning are sampled from the same distribution as the pretraining. The details about the data collection can found in Appendix B. Unlike the pretraining where the input measurement records is a sentence-lewe level vector a _.4, the input of fine-tuning becomes a batch of measurement records X, E Z where K2 is the number of measurement strings. The reason for such change can be explained through both intuitive and rational perspectives. Intuitively, single measurement string cannot reflect the whole picture of the quantum system. Rationally, predicting the properties of the quantum system in classical computers generally requires exponential number of measurements with respect to the system size L (Gebhart et al., 2023). Even though for some quantum systems with low entanglement, the number stills grows quasi-polynomially with L (Huang et al., 2022). Accordingly, the input of the model is replaced with ((3,,c) 0z), P. ) where the tuple (Xj, c2) is the input, P;
Note that the physical conditions
6

---

Published as a conference paper at ICLR 2024
PixceICNN
NN-Classical Shadow
LLM4QPE w/o pretrain
LLM4QPE
L= 19
L= 25
L L 31
Weighted FI Score (%)
95.0
95
95
92.5
90
87.5
85
64
128
236
1024
128
1024
#1 Measurement Strings
# Measurement Strings
# Measuuemment Strings
Figure 3: Comparison of weighted Fl score w.r.t. number of measurement strings on Rydberg atom model.
the corresponding label and B, is the batch size used for supervised finetuning. The embedding is also distinct from that of pretraining. The learned token embeddings for the measurement string d, is not feasible for the batch-style records X,. To deal with it, a Long Short-Term Memory (LSTM) layer attached in front of the decoder, as depicted in Fig. Ic. The LSTM layer converts the discrete measurement records X; and outputs high-order embeddings E E IR B xLxd The additional
embeddings including physical condition embeddings and positional embeddings are transferred
from pretraining. The output embedding is the summation given as E = Em + E. Ep
transferred
Feature Aggregation and Output Projection. The output of the L-layer transformer decoder is H E B. For a specific downstream task, the decoder initialized with the pretrained parameters and all the parameters representation for each the B, training samples, a feature aggregation layer is attached after the last multi-head attention layer. This layer converts the hidden feature H along the second axis and output H' E BBdd Finally, additional linear projection layer is employed project the feature into H" B..PP along with task-dependent activated function which is taken be tanh for predicting the correlation function, since we have the prior that each element of the label p, is in the range [-1, 1] (See Appendix for details). While the too-oof is adopted for classifying quantum phases of matter.
finetuned towards supervised loss. To obtain the feature
Learning Objective. The properties estimation for the quantum system are treated as the supervised learning tasks. Tow types of tasks are considered in this paper, including classifying quantum phases of matter and predicting correlation function. The former belongs to regression task, while the latter be regarded architecture within LLM4QPE. We seamlessly integrate task-specific inputs and ground-truth labels into LLM4QPE and proceed finetune all model's parameters in end-to-end manner. Given that the training samples are {(X,, Cj), P, 1-21 where B, is the batch size. For classifying quantum phases of matter, Рj is the one-hot label. We minimize the observed data negative log- likelihood which yields supervised loss for classification (with P classes):
classification task. For each supervised task, maintain consistent
Suu
1
Σ
Σ I(pp 1]log (fo (x))
Br
jE(L..... N.)
where I[-] is an indicator function, N₁ is the size of training dataset and f- denotes the prediction of the model with parameters 0 be optimized. For predicting the correlation, p, the continuous valued label. We adopt the Root Mean Square Error (RMSE) loss:
Cuup
Coups
Espp
1
Σ
Σ (X,) - -P/.)²²
(5)
Be
je((....NN.
N.).....)...
Detailed description of task specifci finetuning can be found in the experiment section.
EXPERIMENTS
In this section, we present the finetuning results on two quantum property estimation tasks including classifying quantum phases of matter and predicting correlation function. Two families of quan-
7

---

Published as a conference paper at ICLR 2024
Table 1: Classification accuracy quantum phases matter on the Rydberg atom model with varied system
size L and varied training size N,, where y is fixed to be 1024. The best results are highlighted in bold.
Method
N₂ 25
N;
64
N, 100
N; 25
N; 64
N; 100
N; 25
N; 64
N, 100
RBF Kernel
91.75
92.29
93.25
88.43
92.27
94.2
88.32
90.79
92.75
NTK
92.12
92.58
93.79
89.17
94.14
95.39
86,99
92.03
92.71
PixelCNN
92.18
92.79
92.98
88.91
91.59
94.73
85.29
92.21
92.98
NN-shadow
91.73
92.64
93.61
90.57
91.32
95.91
86.38
91.79
92.51
LLM+QFE
94.14
93.38
95.95
93.95
96.51
96.05
87.95
94,95
96.67
LLM4QPE pretrain
90.85
95.35
95.27
1.0
1.2
quaao LEMANNS qaião LEMAQUE Qliuh
L1MRgF persoin
LLM prezain gaiika)
LLMA wia prezzin guiiio)
1.0
0.8
Text Weighted P1 Store
19
713
pretrain
Training
1=26 25
713
71.
0.6
w/o
0.6
364
171
203
1 31 w/o pretrain
0.4
813
413
0.4
240
222
300
300
300
0.0
192
172
165
164
128
256
512
1024
200
600
800
1000
Messstement Strings for Pre-tratning LLMA4QP
Epoch
The required number of epochs for each respective model to attain 90% of its peak weighted Fl score where NN 100.
Figure 5:
Figure 4: The evolution of training loss and test weighted F1 score with increasing training epochs where N₂ = 100 and K = 1024.
tum models are considered - the Rydberg atom model (Bernien et al., 2017) and the anisotropic Heisenberg model (Kranzl al., 2023).
As baseline methods, we basically consider the classical shadow (Huang et al., 2020) a learning free protocol for constructing the representation an unknown quantum state. Besides, we compare with some kernel methods including Radial Basis Function (RBF) Kernel (Huang et al., 2022) and Neural Tangent Kernel (NTK) (Huang et al., 2022). We further consider some advanced deep learn- ing based methods, such as PixeICNN (Sharir et al., 2020) and a classical shadow based generative model (NN-shadow) (Wang et al., 2022) for comparison.
4.1 CLASSIPYING QUANTUM PHASES OF MATTER ON RYDBERG ATOM MODEL
We first consider the Rydberg atom model with different system size L E {19, 25, 31). We pre- train LLM4QPE for different system sizes separately with a fixed number of sampled physical conditions N, 100. Each physical condition variable C; is 4-dimensional vector denoted as [L, Ri Ro/a: where A is the detuning of laser, 92 the Rabi frequency and Ro/a is the interaction range. The values of these four variables can be obtained directly when initializing the (simulated) quantum experiments. For each physical condition we generate Ky measurement strings based on computational basis measurement operators, such that the total number of possible mea- surement outcomes M 2. Then LLM4QPE pretrained with dataset D,- The pretrained pa- rameters are transferred to finetune the model using Dr, where the number of sampled physical con- ditions N, E (25, 64, 100) and the number of measurement strings K₃ E (64, 128, 256, 512, 1024). We fix the size of D. for evaluation to be N₂ 10000. Following (Bernien et al., 2017), sider three categories of quantum phase, i.c., Disorder, Zz. Z3 to establish the label p,. which is a 3-dimensional one-hot vector. More details about the data generation can be found in Appendix B.
We also take evaluation without pretaining the LLM4QPE: all the parameters are initialized ran- domly in uniform distribution [-1, 1]. We use accuracy and weighted Fl score metries for 3-class classification for evaluation of our models and baselines. The results are listed in Tab. and LLM4QPE achieves the best mean accuracy except for setting I. = 31 with N, 25.
8

---

Published as a conference paper at ICLR 2024
Table 2: RMSE of predicting the correlation on the anisotropic Heisenberg model with varied system size L and training size N₁. y is fixed to 64, The best results are in bold.
2-8
I. 10
J- 12
Mediod
N,
50
90
N, 30
N. 50
No = 90
N: 30
No
N; 14:
Shadow
RBF Kernel
0.2085
0,3077
0
0.2104
02131
6,2079
0.2039
0.1931
021157
NTK
0.2062
0,2064
0.2552
0.2095
0.3085
0,2097
0.2141
0.1922
0.2105
PNNKKNN
0.2257 0.2257-0015
0,257740.019
0:223-00024
0,295
0.229
0:208810004
0.2300
0.2367-00
NN-shaooo
0.3089 0.02
1.208-10:015
0,9157-0012
0.20860.0077
0:3154-0017
0.190010.01
0.2057
10039
0.201100.019
0,3100 0.036
LLMIQSE
0.176 BJR32
6.199-00122
6.199 H082
0.178748121
0.176M 1J115
LLMAQPE wil pretrais
0.3043-00227
02207100006
0.14922027
022799500055
0.194420013
0.1981230.025
0,2040:0028
0:2091200001
0.3026--0027
Table 3: Ablation study results on condition embedding and LSTM embedding. We consider N₂-
64 with Ky = 1024 for the Rydberg model, and N; 50 with Ky = 64 for the Heisenberg model.
Rydberg
=
Heisenberg
==8
L=
L= 12
original wo cond. embed.
93_38
96.51
94.95
original
0.1612
0.1949
0.1787
93.29
95,96
93.52
w/o cond. embed.
0.1906
0.2095
0.1981
w/o LSTM emhed.
90.75
92.18
89.65
w/o LSTM emhed.
0.1929
0.1997
0.1904
Fig. 3 shows the performance on varied Ky. LLM4QPE achieves the best weighted F1 score across all systems and in particular, outperforms by a large margin when K5 64. The results indi- cate that pretrained LLM4QPE can handle the input when few number of measurement records are available, which is greatly instrumental due to the expensive and time-consuming (simulated) quantum experiments. We further plot the training dynamics of LLM4QPE with and without pre- training throughout the training epochs in Fig. 4. The curves indicate that the pretraining enables much faster convergence of supervised loss and achieves better finetuning accuracy. Meanwhile, the required number of epoch for the model to attain 90% of its peak weighted Fl score is provided in Fig. 5. It reflectw that within the same system size L, the pretrained LLM4QPE converges faster than the non-pretrained version, with a lower training error and a higher test weighted Fl score.
4.2 PREDICTING CORRELATION FUNCTION ON ANISOTROPIC HEISENBERG MODEL
Next we consider regression task predicting correlation on the anisotropic Heisenberg model. This quantum model inherits the long-range interactions between every two quantum sites, leading to a complex dynamics which hard to be simulated by classical computers (Orús, 2019). We restrict the system size T. E {8, 10, 12) due to memory limitations. The ground states of quantum systems with different physical conditions are calculated by eigenvalue decomposition. For each physical condition we generate K2 measurement strings based on Pauli-6 measurement operators such that MM = 6. Then we pretrain the LLM4QPE for different system sizes independently with training size Np 100.
For model's 's finetuning, we vary the number of generated training samples N₂ E (20, 50, 90} and fix the measurement strings K 64. The dataset used for evaluation is generated with N₂ 200. To obtain the ground-trute -ttuthttuuthha labels, We calculate true values of the two-body correlation functions and collect them the supervised labels, which is an L x L continuous-valuee matrix where each entry is in the range [-1, 1]. The RMSE results is reported in Tab. 2. LLM4QPE outperforms baselines in all settings. The learning-basee models baselines often fail surpass the predictive accuracy of learning-free classical shadow. While our pretrained LLM4QPE stands out by a remarkable margin.
Finally, we study the effects of condition embedding and the LSTM embedding on both Rydberg atom model and anisotropic Heisenberg model. Note that we replace the LSTM with a fully con- nected layer with same input/output dimension. The results are given in Tab. 3, where the results consistently show that both embedding techniques contribute to some positive effects and suggest that these two techniques can both help to leverage useful information from input quantum data.
5 CONCLUSION AND OUTLOOK
This paper proposes task-agnostic unsupervised pretraining approach for estimation of the prop- erties of the quantum systems via quantum datasets. The core of our approach is a transformer encoder enabling to learn useful hidden information in a fully unsupervised pretraining procedure. The pretrained parameters can be transferred to solving downstream tasks, leading to more effective classifying quantum phases and predicting correlation function on resource-limited device given limited measurement information.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper proposes a novel approach for estimating the properties of quantum systems inspired by LLMs. The authors acknowledge the potential ethical implications of this research, such as the misuse of quantum data, the bias or error in the estimation results, and the impact on the development of quantum technologies. The authors have followed the best practices for data collection, model design, and evaluation, and have disclosed the sources of funding and the conflicts of interest. The authors also adhere to the principles of research integrity and comply with the relevant laws and regulations. The authors hope that this research will contribute to the advancement of quantum science and benefit the future research.
REPRODUCIBILITY STATEMENT
The generated quantum data of the Rydberg atom model and the anisotropic Heisenberg model available at https: //gi thub. com/ / aabelleel 1231 (ape-date The code to train the model and analyze the experimental results available from the first author on reasonable request.
REFERENCES
Bloqade.jl:
Package for the quantum computation and quantum simulation based the neutral- atom architecture, 2023.
Anurag Anshu and Srinivasan Arunachalam. survey on the complexity of learning quantum states. Nature Reviews Physics, 6(1):59-69, 2024.
Ville Bergholm, Josh Izaac, Maria Schuld, Christian Gogolin, Shahnawaz Ahmed, Vishnu Ajith, M Sohaib Alam, Guillermo Alonso-Linaje, AkashNarayanan, Ali Asadi, et al. Penny- lane: Automatic differentiation of hybrid cuantum-classical computations. arXiv preprint arXiv:1811.04968. 2018.
Alejandro Bermúdez, Luca Tagliacozzo, Germán Sierra, and P Richerme. Long-range heisenberg
models in quasiperiodically driven crystals of trapped ions. Physical Review B, 95(2):22443,, 2017.
Hannes Bernien, Sylvain Schwartz, Alexander Keesling, Harry Levine, Ahmed Omran, Hannes Pichler, Soonwon Choi, Alexander Zibrov, Manuel Endres, Markus Greiner, et al. Probing many-body dynamics on a 51-atom quantum simulator. Nature, (7766))))579- 2017 2017.
Fernando GSL Brandao and Michal Horodecki. Exponential decay of correlations implies area law.
Communications mathematical physies, 333:761-798, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Tiff Brydges, Andreas Elben, Petar Jurcevic, Benoît Vermersch, Christine Maier, Ben P Lanyon, Peter Zoller, Rainer Blatt, and Christian F Roos. Probing rényi entanglement entropy via random- ized measurements. Science, 364(6437):26--666) 2019.
Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science, 355(6325):002660) 2017.
Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Vogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Re- views of Modern Physics, 9144):0450 2019.
Juan Carrasquilla, Giacomo Torlai, Roger G Melko, and Leandro Aolita. Reconstructing quantum states with generative models. Nature Machine Intelligence, 1(3):155-161, 2019.
David Ceperley and Berni Alder. Quantum monte carlo. Science, 5556473), 1986.
10

---

Published as a conference paper at ICLR 2024
Peter Cha, Paul Ginsparg, Felix Wu, Juan Carrasquilla, Peter L McMahon, and Eun-Ah Kim. irrention-based quantum tomography. Machine Learning: Science and Technology. 3())001LT01, 2021.
Philippe Corboz. Variational optimization with infinite projected entangled-pair states. Physical Review B, 94(3):035133, 2016.
Stefanie Czischek, M Schuyler Moss, Matthew Radzihovsky, Ejaaz Merali, and Roger G Melko. Data-enhanced variational monte carlo simulations for rydberg atom arrays. Physical Review B, 105(20):205008, 2022.
G Mauro D' Ariano, Matteo GA Paris, and Massimiliano F Sacchi. Quantum tomography. Advances in imaging and electron physics, 128:206-309, 2003.
Yuxuan Du, Yibo Yang, Tongliang Liu, Zhouchen Lin, Bernard Ghanem, and Dacheng Tao. Shad- ownet for data-centric quantum system learning. arXiv preprint arXiv:2308. 1290, 2023.
Xun Gao and Lu-Ming Duan. Efficient representation of quantum many-body states with deep neural networks. Nature communications, 8(1):662, 2017.
Valentin Gebhart, Raffaele Santagati, Antonio Andrea Gentile, Erik M Gauger, David Craig, Natalia Ares, Leonardo Banchi, Florian Marquardt, Luca Pezzè, and Cristian Bonato. Learning quantum systems. Nature Reviews Physics, 5(3): 141-156, 2023.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Aleksandra Gočanin, Ivan Supić, and Borivoje Dakić. Sample-efficient inclieeenndeendeent quantum state verification and certification. PRX Quantum, 3(1):010317, 2022.
James Gubernatis, Naoki Kawashima, and Philipp Werner. Quantum Monte Carlo Methods. Cam- bridge University Press, 2016.
Mohamed Hibat-Allah, Martin Ganahl, Lauren E Hayward, Roger G Melko, and Juan Carrasquilla. Recurrent neural network wave functions. Physical Review Research, 2(2):023358, 2020.
Pierre Hohenberg and Walter Kohn. Inhomogeneous electron gas. Physical review, 136(3B):B864, 1964.
Hsin-Yuan Huang, Richard Kueng, and John Preskill. Predicting many properties of quantum system from very few measurements. Nature Physics, 16(10):0000-1055 2020.
Hsin-Yuan Huang, Richard Kueng. Giacomo Torlai, Victor V Albert, and John Preskill. Provably ef- ficient machine learning for quantum many-body problems. Science, 37766613)) 2022.
T Jullien, P Roulleau, B Roche, A Cavanna, Y Jin, and DC Glattli. Quantum tomography of an electron. Nature, 5 514(7524))603-600 2014.
Hiroki Kawai and Yuya O Nakagawa. Predicting excited states from ground state wavefunction by supervised quantum machine learning. Machine Learning: Science and Technology, 1(4):045027, 2020.
Florian Kranzl, Stefan Birnkammer, Manoj Joshi, Alvise Bastianello, Rainer Blatt, Michael Knap, and Christian F Roos. Observation of magnon bound states in the long-range, anisotropic heisen- berg model. Physical Review X, 13(3):031017, 2023.
Dietrich Leibfried, DM Meekhof, BE King. CH Monroe, Wayne M Itano, and David Wineland. Experimental determination of the motional quantum state of a trapped atom. Physical review letters, :7221),, 1996.
Laura Lewis, Hsin- Yuan Huang, Viet T Tran, Sebastian Lehner, Richard Kueng, and John Preskill. Improved machine learning algorithm for predicting ground state properties. Nature communica- fions, 15(1 (195, 2024.
11

---

Published as a conference paper at ICLR 2024
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023.
EY Loh Jr, JE Gubernatis, RT Scalettar, SR White, DJ Scalapino, and RL Sugar. Sign problem in the numerical simulation of many-electron systems. Physical Review B, 41(13)93301, 1990.
William Lauchlin McMillan. Ground state of liquid he 4. Physical Review, 138(2A):A442, 1965.
Michael A Nielsen and Isaac Chuang. Quantum computation and quantum information. Cam- bridge university press, 2010.
Román Orús. Tensor networks for complex quantum systems. Nature Reviews Physics, 1(9):538- 550, 2019.
David Perez-Garcia, Frank Verstraete, Michael M Wolf, and Ignacio Cirac. Matrix product state representations. arXiv preprint quantphh/0001997, 2006.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Kristof T Schütt, Michael Gastegger, Alexandre Tkatchenko, K-R Müller, and Reinhard J Maurer. Unifying machine learning and quantum chemistry with deep neural network for molecular wavefunctions. Nature communications, 10(1):5024, 2019.
Or Sharir, Yoav Levine, Noam Wies, Giuseppe Carleo, and Amnon Shashua. Deep autoregressive models for the efficient variational simulation of many-body quantum systems. Physical review letters, 124(2)022003,, 2020.
GI Struchalin, Ya A Zagorovskii, EV Kovlakov, SS Straupe, and SP Kulik. Experimental estimation of quantum state properties from classical shadows. PRX Quantum, 2(1):010307, 2021.
Giacomo Torlai, Guglielmo Mazzola, Juan Carrasquilla, Matthias Troyer, Roger Melko, and
Giuseppe Carleo. Neural-network quantum state tomography. Nature Physics, 14(5):4474400 2018.
Matthias Troyer and Uwe-Jens Wiese. Computational complexity and fundamental limitations to fermionic quantum monte carlo simulations. Physical review letters, 94(17):17020) 2005.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Pragya Verma and Donald G Truhlar. Status and challenges of density functional theory. Trends in Chemistry, 2(4))302-31 -188 2020.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, Carey, Îlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020. doi: 10.
Haoxiang Wang, Maurice Weber, Josh Izaac, and Cedric Yen-Y Yu Lin. Predicting properties of quantum systems with conditional generative models. arXiv preprint arXiv:2211.16943 2022.
Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. A survey of transfer learning. Journal of Big data, 3(1):1-40. 2016.
Steven R White. Density matrix formulation for quantum renormalization groups Physical review letters, (99))2886,, 1992.
12

---

Published as a conference paper at ICLR 2024
Dian Wu, Lei Wang, and Pan Zhang. Solving statistical mechanics using variational autoregressive networks. Physical review letters, 122(8):080602, 2019.
Ya-Dong Wu, Yan Zhu, Ge Bai, Yuexuan Wang, and Giulio Chiribella. Quantum similarity testing with convolutional neural networks. Physical Review Letters, 130(21):2 2023.
Tailong Xiao, Jingzheng Huang, Hongjing Li, Jianping Fan, and Guihua Zeng. Intelligent certifica- tion for quantum simulators via machine learning. upj Quantum Information, 8(1):138, 2022.
Ting Zhang. Jinzhao Sun, Xiao-Xu Fang, Xiao-Ming Zhang, Xiao Yuan, and He Lu. Experimental quantum state mcasurement with classical shadows. Physical Review Letters, 127(20):20050,, 2021.
Yuan-Hang Zhang and Massimiliano Di Ventra. Transformer quantum state: A multipurpose model for quantum many-body problems. Physical Review B. 107(7)005147, 2023.
Yan Zhu, Ya-Dong Wu, Ge Bai, Dong-Sheng Wang, Yuexuan Wang, and Giulio Chiribella. Flexible learning quantum states with generative query neural networks. Nature Communications, 13 (1):6222, 2022.
13

---

Published as a conference paper at ICLR 2024
A RELATED WORK
A.1 LEARNING-FREE METHODS FOR QPE
Estimating the properties of the quantum system is a long-standing problem in quantum physics (D' Ariano al., 2003). The main challenge that the complexity of describing the quantum system using classical computers typically scales exponentially with respect to the sys- tem size (Nielsen & Chuang, 2010). Even though, in fact, the quantum systems studied in physical experiments generally can be described by limited number of physical variables. This restriction leads to the studied quantum systems occupy only small part of the exponentially large Hilbert space (Carrasquilla et al., 2019), such that they can be characterized by some classical methods within an acceptable error.
Traditional algorithms including the QMC (Ceperley & Alder, 1986) and DFT (Hohenberg & Kohn, 1964) has made success for investigating the electronic structure (or nuclear structure), principally the ground state of many-body systems, such atoms, molecules, and the condensed phases (Gu- bernatis et al., 2016). However, these methods have scalability issues and are difficult to be used to deal with large-scale quantum many body problems. An alternative is class of TNs methods (Orás, 2019) based on variational method and shows unprecedented performance in analyzing the char- acteristics of ground state. These methods including Matrix Product State (MPS) (Perez-Garcia et al., 2006) and Projected Entangled Pair States (PEPS) (Corboz, 2016). This class of methods approximates the wave function by decomposition of the high-order wave functions into multiple low-rank tensors. It is then possible to analyze properties of the quantum state by taking algebra operations on the wave function. Recently, the classical shadow protocol (Huang et al., 2020) sug- gests to use random measurements to characterize the quantum properties. Classical shadow has facilitated applications such as direct fidelity estimation (Struchalin et al., 2021) and state function prediction (Zhang et al., 2021).
A.2 LEARNING-BASED METHODS FOR QPE
With the continuous development of machine learning technologies, neural network based meth- ods have emerged to tackle the QPE problems. These methods can be categorized into two classes according to the purpose. The methods (Carleo & Troyer, 2017; Gao & Duan, 2017; Torlai et al., 2018; Schütt et al., 2019; Hibat-Allah et al., 2020; Zhang & Di Ventra, 2023) of the first class are called Neural Network Quantum State (NNQS), which replace the tensor used in TNs with neural network as a parametric function approximator of quantum many-body wave functions. The pa- rameterized wave function is updated by minimizing the expectation values of relevant observable estimators, based on either density matrix renormalization group (DMRG) algorithm (White, 1992) variational Monte Carlo (VMC) (McMillan, 1965). Afterwards the interested properties can be analyzed by preforming algebra operations on the wave function. Another line of research (Gilmer al., 2017; Kawai & Nakagawa, 2020; Xiao et al., 2022) is known as Neural Network Quantum Property Estimation (NNQPE). NNQPE directly optimizes the parameters towards a specific learn- ing objective which represents a certain property of quantum systems such as the quantum phase.
For both NNQS and NNQPE, different neural network ansatz corresponds to solve quantum many- body problems with different physical structures. Examples include restricted Boltzmann machine (RBM) (Carleo & Troyer, 2017), recurrent neural networks (RNNs) (Carrasquilla et al., 2019), con- volutional neural networks (CNNs) (Wu et al., 2019; Sharir et al., 2020; Wu et al., 2023), and transformers (Cha et al., 2021: Wang et al., 2022; Zhang & Di Ventra, 2023; Du et al., 2023).
Our work is closely related to NNQPE. While ours employs a unsupervised pretraining to extract the hidden information of the quantum systems govern by different parameters. We find empirically that this scheme can make the model perform better under a limited number of copies of quantum states and measurements. The recent work proposed by Zhu et al. (2022) implements similar pretraining strategy for learning of quantum states, whereas our approach differs from it by avoiding assumptions about knowing the prior frequency about the measurement strings.
14

---

Published as a conference paper at ICLR 2024
B DETAILS OF THE QUANTUM DATASET GENERATION
quantum dataset is a collection of data that describes quantum systems and their evolution. The collection of quantum data must take into account the following factors: 1) the method of data collection must be feasible on quantum devices and not contradict the disciplines of quantum me- chanics; 2) the process of data collection is completely automated and does not require experienced experts to organize and label it and 3) the data must be structured and can be stored on resource- limited classical devices, thus can be easy to be processed by the machine learning techniques with- out further post-processing. The quantum dataset we established satisfies these three points. It is also worth mentioning that our model can be used centralized infrastructure process all these data uniformly, thanks to the unsupervised pretraining design of the model.
In this paper, we conduct simulated experiments generate the quantum dataset in classical com- puters. For the anisotropic Heisenberg model, quantum measurement is performed using the Pauli-6 measurement operators such that M = 6, whereas computational basis measurement operators are employed for the Rydberg atom model leading to M = 2. Assume that variables describ- ing the physical condition lives in a finite continuous space within the physical restriction. Let D = [R, c; J denote the quantum dataset used for pre-training and Dj = {R,, c,), pi 1 for fine-tuning, where [p,l = N, and [Dy] Ny. For pre-training the model, we first uniformly
sample a number of points {c,lc, E F) Afterwards we conduct simulated experiments for each c, and collect the corresponding measurement records. The system property P, is not needed since the pre-training phase is fully unsupervised. While for fine-tuning, we initialized the experi- ments with another random seed and sample N, physical conditions also within space F, resulting
in fcyle, E F Note that We also collect the measurement records for each cj. The difference part is that we additionally calculate the system property py and use it as supervised labels. We further split the D into D, and De for training and evaluation respectively with varied separation and the experimental configurations of the dataset generation
ratio. Details of the are discussed below.
B. RYDBERG ATOM MODEL
Rydberg atom model is a programmable quantum simulators capable of preparing interacting qubit systems (Bernien al., 2017). Such quantum model can be effectively described two-level quantum system consisting the ground state (g) ((0)) and the Rydberg state (r)). The quantum dynamics of this model is governed by the Hamiltonian
Ω HRydberg Σ -
VO T.
E
(6)
T
where o₂ is the Pauli-X matrix, P is the Rabi frequency, A is the detuning of a laser, Vo is the Rydberg interaction constant, j is the Rydberg interaction constant and H is the position vector of the site i. n, = (r.) (ryl is the occupation number operator at site i, and o- = =gg) + In:) (gil describes the coupling between the ground state (9.) and the Rydberg state |r;) at position
We follow the recent work in (Wang et al., 2022) to generate the quantum dataset. We refer the readers to their paper for details. Here we briefly introduce the main procedures. We consider the Rydberg atom model with system size L E {19, 25, 31). We fix the interaction constant Vo = 862690 2n MHz um and vary the value of Ω E [0,5] and A E [-10, 15] to get different physical conditions c, where e is a 4-dimensional vector in the form [4, 2, Ro/a), where Ro/a denote the interaction range with R2 (Vo/ Then the approximate ground state for diffident physical condition is prepared by the tool Bl Blogade. ji (blo, 2023). This tool can also output the measurement strings and the true phase of each physical condition. The measurement operators are for the quantum measurement, such that the total number of the possible outcomes is M = 2. In this paper, three different phases are considered including the Disordered phase, Z2 Ordered phase and Za Ordered phase. We sample N,- = 100 physical conditions with K, 1024 measurement strings for pre-training, and N; E [25, 64, 100} physical conditions with Ky E (64, 128, 256, 512, 1024] for fine-tuning. The number of physical conditions for evaluation is fixed to be Ne = 10000. The supervised labels for fine-tuning are hot encoded vectors of the true phases such that the dimension (number of classes) of p is 3. Note that it is ensured that the sampled physical conditions for pre-training will not appear in fine-tuning.
chosen to the computational basis (10) [1) [1)
15

---

Published as a conference paper at ICLR 2024
B.2 ANISOTROPIC HEISENBERG MODEL
Exploring the effects of these long-range interactions of the quantum system is essential for un- derstanding the quantum mechanics (Bermúdez al., 2017). In this paper, we consider the recent progress for the long-range interactions with the experimentally realized power-law exponent of the anisotropic Heisenberg model (Kranzl et al., 2023). The dynamics of the anisotropie Heisenberg model is determined by the Hamiltonian
Σ Ju + tojo ho
Heissabberg
where of is the Pauli matrix operated on the i-th site, h determines the Ising interactions be- tween the magnons, and Jij is the long-range interaction strength satisfying Jij J/li - jl We follow the configuration of (Kranzl et al., 2023) to geenrate the quantum dataset. The val- ues of h and J are fixed with and 369 rad/s, and we vary the value of E (1, uniformly. is extremely hard to characterize the quantum system with long-range interactions using the existing computing techniques. Thus we restrict the system size L E {8, 10, 12). For all the systems we consider the number of measurement strings used for pre-training as K₂ = 1024 100. For model's finetuning,
and fix the number of sampled physical conditions N, we vary the number of generated training samples N, E {20, 50, 90) and fix the measurement strings Ky 64. The physical condition is defined
vector whose dimension C = I.2, The problem of finding the ground state is viewed as the eigenvalue decomposition problem and we obtain the ground state for each sampled physical condition by the scipy (Virtanen et al., 2020) built-in functions. The measurement records and the true values of the wwo-body correlation function and the entanglement entropy are obtained using the pennylane (Bergholm et al., 2018) tool- box. We consider the Pauli-6 POVM measurement operators with M 6 outcomes, which are given as MPmuli-6 = { [1)}, {|+),1-)}, flr), } stand for the eigenbasis of the Pauli operators dz: ux. and yy re- spectively. For the task of predicting the correlation matrix, the ground-truth label is L matrix and each element of the matrix is the expectation value of the observable
in which each element is the coupling strength Jij for i,j E {I....,L} L}.
(0), [1)(1), x [+}++,, --X(1, x (rl, x [X J. and
Oū ===3 (0'001 + tojor + to'o!).
(8)
Thus each element can be written as tr(pOig) in the range [-1, 1]. where p the density matrix of the ground state for each sampled physical condition. We flatten the correlation function matrix to be the L'-dimensional continuous-valued vector and treat it as the supervised label for fine-tuning. While for the task of predicting the entanglement entropy, the label is a real number which can be calculated as - log(tr(r²)) where the left-half subsystem with system size T.22 of the quuut quantum system.
C POOF OF THE NORMALIZED OUTPUT DISTRIBUTION
In the main text, we claim that the output (classical) distribution satisfies
M
M
Σ
Σ
or.) =
(9)
as long as the last linear projection layer uses the softmax activated function. The proof is given below.
The softmax activated function performed on the model's 's output, which is the product of condi- tional probabilities p(a o1) = I p(o,|o,-1,... o1). It is easy to check the claim holds for I. 1. Given that the claim also holds for L. k. For L 1, the following equation then be hold:
Σ
1.
(10)
16

---

Published as a conference paper at ICLR 2024
Table The RMSE of predicting the second-order Rényi entanglement entropy for the anisotropic Heisenberg model. We sample Np 100 physical conditions with Kp for pre-training.
1024 measurement strings
18
64
Kg
126
386
5,
2-3
1934
Ag 3X
613
R 105
Rg 64
Ar 10.
52 325
R BE36
158532
14277
1205%
13079
131155
13570
1,650
24324
230725
0.8114
8.2330
18200
0.200
832231
18873
382299
1.801
58361
037311
033889
382145
038885
03088
388455
1.9455
038319
038221
821844
831147
180084
03062
18728
836775
025124
006647
19455
125225
109m
0.855
03171
83105
18165
02045
00033
149454
19044
114291
02942
0.2531
Such that
M
M
Σ
Σ
01
0++1=1
Σ
Σ
0k+1))
#1=1
0k-1=1
k+1
Σ
Σ
II
L
-
(11)
0h-1
Σ
Σ
II
(o
Σ #411=1
Σ
L (o
.. 01)
The proof then complete.
D ADDITIONAL NUMERICAL RESULTS
D.1 RESULTS OF PREDICTING THE ENTANGLEMENT ENTROPY
We take an additional downstream task: predicting the second-order Rényi entanglement entropy - -log(tr(p3 (pa) for the anisotropic Heisenberg model, where A is the left-half subsystem with system size L/2 of the L-qubit quantum system. The number of training size is set to be N, = 90 and the predicted RMSE results given in Tab. 4. It can be observed that pre-traine pre-training remains effective for predicting the entanglement entropy of the anisotropic Heisenberg model.
D.2 MODEL SENSITIVITY TO THE NUMBER OF MEASUREMENTS
In Sec. 4, study the relationship between the number of measurements and the classification accuracy of quantum phase of matters on Rydberg atom model. It is empirically evident in Fig. that achieving linear growth in classification accuracy requires an exponential increase in the number of measurements per training example. Beyond the scaling related to number of measurements, we dive into further research on the scaling relationship between accuracy and the size of the training set (i.e., the number of sampled physical conditions which determine the dynamics of the quantum systems). We constrain the number measurement per example to 256 (since we find that a large value makes the accuracy reach saturation) and the results on the 3 31-qubit system are listed in the Tab. 5. The results show that the accuracy approximately exhibits linear growth w.r.t. training size. This finding is consistent with theoretical results presented in (Huang et al., 2022; Lewis et al., 2024), which demonstrate that there exists a polynomial scaling relationship between model performance and the size of training dataset.
17

---

Published as a conference paper at ICLR 2024
Table 5: Classification accuracy of quantum phases of matter on the Rydberg atom model with varied training size N₁, where L = 31 and K 256. The results are averaged over 3 runs with different random seeds.
N2 = 20
N₂ 40
N₂ 60
N, 80
LLM4QPE LLM4QPE w/o pretrain
82.05
87.24
89.16
90.63
79.17
81.78
85.96
88.47
Table Classification accuracy of quantum phases of matter on the 31-qubit Rydberg atom model. The pre-trained parameters are transferred from the model trained smaller system size. The training size is set to be N, 100, and the number of measurements K 1024.
LLM4QPE (pre-trained on 19-qubit system)
95.74
LLM4QPE preettrinned on 25-qubit system)
96.13
LLM4QPE (pre-trained on 31 31-qubit system)
96.67
LLM4QPE w/o pre-train
94.32
Table 7: Classification accuracy of quantum phases of matter on the 19-qubit Rydberg atom model. The training size is be N, 100, and the number of measurements Ky 1024.
no OOD
OOD
LLM4QPE LLM4QPE w/o pre-train
95.95
84.82
93.35
94.23
D.3 FINE TUNING THE MODEL. WITH OUT-OF-DISTRIBUTIIN DATASET
In this section, we consider fine tuning the LLM4QPE with out-of-distribution (OOD) dataset, which means the dataset used for fine-tuning and the dataset used for pre-training come from different distributions.
Here, we consider two different configurations to make the fine-tuning dataset from the pre-training one: the first is to re re-generate the fine-tuning data by modifying the physical variables and the second is to fine tune the model based on the parameters transferred from the model pretrained on fewer qubits. In the following, we consider the Rydberg atom model.
First, we take the evaluation that fine-tuning the model 31 31-qubit system by using he parameters pre-trained on 19 and qubit system. Note that the number of qubits also a physical variable and see if model parameters trained on small-scale systems could transfer and help model characterize larger-scale systems. The results are listed in Tab. 6. It is evident that pre-trained parameters transferred from small-scale systems also useful for large-scale systems.
Second, we modify the detuning of laser from [-10, 15] (which is exactly used in the paper) to [-20,-10] U [15, 25] generate OOD fine-tuning dataset, on Rydberg atom model with 19 qubits. The classification accracy are listed in Tab. 7. The pre-trained one fails to perform better than the LLM4QPE w/o prentrain. The main reason that the modified detuning values have driven the quantum evolution into very different dynamics and the pre-trained model learns less knowledge about Whether pre-training of LLM4QPE remains beneficial for OOD quantum datasets other settings remains open question, and will be further explored in our future work.
E LIMITATIONS
In this study, we concentrate on the classification quantum phases of matter and the prediction of correlation functions for the Rydberg atom model and the anisotropic Heisenberg model, respec- tively. While the LLM4QPE model offers flexibility for addressing various quantum many-body challenges, such reconstructing the density matrix. Our focus here is primarily on pretraining the model with a fixed number measurement strings. The impact of varying the number measure- ment strings the model's performance presents fascinating area for exploration. Additionally,
18

---

Published as a conference paper at ICLR 2024
the LLM4QPE model is characterized by relatively small parameter count (tens of thousands of parameters) when compared to the significantly larger parameter sets large language models. Due to the constraints imposed by the model's size, our pretraining efforts are confined to quantum sys- tems govern by Hamiltonians from the same family. Looking forward, there is an anticipation to develop more robust model, enriched with greater number of parameters, through learning datasets generated from diverse families of quantum systems.
19


Reviews:
[{"review_id": "EJlYYNVdNs", "reviewer": "Reviewer_1o9o", "summary": "The submission introduces Q-TAPE, a versatile pre-trained model for quantum systems, aiming to enhance property estimation. Drawing inspiration from Large Language Models' success in other domains, Q-TAPE offers several key advantages: it leverages a rich set of quantum data for comprehensive training, adopts an unsupervised and task-agnostic approach, excelling in classifying quantum phases and predicting correlation functions, especially with limited data, and seamlessly transfers knowledge from pre-training to specific properties estimation tasks. Extensive experiments validate Q-TAPE's efficacy across various quantum tasks. Furthermore, the authors commit to making the source code openly available, fostering further research and applications in quantum property estimation.", "questions": "The questions are listed in Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Q-TAPE presents a novel and promising approach to understanding quantum systems. Diverging from previous methods that primarily rely on supervised learning, Q-TAPE advocates an unsupervised pretraining combined with downstream task finetuning, a methodology akin to the training of large language models. This paradigm shift offers a more experimentally-friendly approach, emphasizing adaptability and robustness. The impressive results achieved, especially with quantum systems involving up to 31 qubits, underscore the significant potential of Q-TAPE, solidifying its strengths in advancing quantum property estimation.", "weaknesses": "Several minor concerns about the submission are listed below. \n\nFirstly, it is essential to empirically investigate the scaling behavior of the required number of measurements for each training example concerning the qubit count. A polynomial scaling that ensures satisfactory performance is desirable, as an exponentially scaling behavior may impede the practical utility of Q-TAPE for larger quantum systems.\n\nSecondly, there is a need for a discussion on the storage method employed for the training dataset and the corresponding memory cost. For larger qubit counts, as the number of measurements increases, the memory cost may become prohibitively expensive even though only the measurement results and the measured operators are stored. It would be beneficial to know if the authors have developed advanced methods to address this challenge, as the bottleneck of employing deep learning for quantum problems could shift to data management.\n\nLastly, some missing references that are pertinent to the content of the submission should be discussed, ensuring a comprehensive and well-referenced presentation of the work. Concrete examples include Refs [1] and [2], which separately address the ability of deep neural networks to simultaneously accomplish multiple tasks and the fundamental role of datasets in quantum system learning.\n\n[1] Wu, Ya-Dong, et al. \"Learning and Discovering Quantum Properties with Multi-Task Neural Networks.\" arXiv preprint arXiv:2310.11807 (2023).\n[2] Du, Yuxuan, et al. \"ShadowNet for Data-Centric Quantum System Learning.\" arXiv preprint arXiv:2308.11290 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iPFAKL9G8c", "reviewer": "Reviewer_qDBp", "summary": "In this work, the authors leverage a pre-trained LLM strategy to construct the Q-TAPE architecture to deal with quantum data, which involves an unsupervised pre-trained model and fine-tuning for specific tasks. Numerous experiments have been conducted to show the promising efficacy of the Q-TAPE.", "questions": "(1) Why not directly use quantum machine learning like quantum neural networks to deal with the quantum datasets? since the hybrid quantum-classical architecture can attain competitive empirical results. \n\n(2) Is there some out-of-distribution quantum data being existed in the datasets? If so, how to deal with the case.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work leverages the pre-training strategy that has attained triumph in large language models. \n\n2. This work aims at building quantum datasets that can be conceptually equivalent to the corpus used to train LLMs.", "weaknesses": "1. The idea of generating and collecting quantum data is important, but quantum data is more fitted to quantum machine learning models like quantum neural networks and quantum graph neural networks, instead of the classical ones. \n\n2. Even though the work somehow makes the generated quantum data suitable for the classical LLM architecture, the new hybrid quantum-classical architecture is not interesting. For classical-to-quantum data conversion, a simple quantum tensor encoding, amplitude encoding, or a more advanced method of the quantum kernel can efficiently deal with quantum data generation. If you could collect quantum datasets, the use of quantum machine learning models is more important and that is the most advantage of quantum neural networks. \n\n3, Similar techniques of pre-training strategy have been shown in many quantum machine learning works. For example, Yang's quantum NLP work has demonstrated the effectiveness of pre-training hybrid LLM for dealing with quantum data. Moreover, Qi's recent work has provided a solid theoretical understanding of pre-training LLM for quantum machine learning. Unfortunately, those important and very related works are not cited in this work. \n\nRef. [1] Yang, C.H.H., Qi, J., Chen, S.Y.C., Tsao, Y. and Chen, P.Y., 2022, May. When Bert meets quantum temporal convolution learning for text classification in heterogeneous computing. In IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8602-8606). IEEE.\n\nRef. [2] Jun Qi, Chao-Han Huck Yang, Pin-Yu Chen, Min-Hsiu Hsieh, \"Pre-Training Tensor-Train Networks Facilitate Machine Learning with Variational Quantum Circuits,\" arXiv:2306.03741v1. \n\n4. In the experiments, the authors do not discuss the in-distribution and out-of-distribution quantum data. For the out-of-distribution quantum data, the proposed pre-training LLM architecture cannot ensure a good performance. So, some new quantum-aware optimization algorithms need to be more deeply investigated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BJZyt188Jd", "reviewer": "Reviewer_2sNf", "summary": "There is a lot of hope for using machine learning to advance quantum physics. Many supervised machine learning algorithms have been studied and proposed for important foundational problems in quantum physics, such as classifying quantum phases of matter and predicting properties of quantum systems. Inspired by the power of unsupervised models (such as LLMs and pre-trained models in CV), the authors present a pre-trained model Q-TAPE for quantum physics. The authors showed that the pre-trained model enables strong performance with limited training data, which can be very useful as large-scale experimental data are hard to obtain.", "questions": "I can understand how language has many universal features, which makes pre-training very useful. However, I don't yet see what kind of features in quantum physics problems are \"universal.\" Could the authors provide some visualizations of the pre-trained model to showcase what kind of universal features are learned by Q-TAPE? Could the authors also provide more discussions (and possibly an example) to illustrate why pre-training is useful in quantum physics? I believe the authors' argument that pre-training can be very useful in quantum physics, but I am not entirely sure how.\n\nAs noted in the Limitations paragraph, the work would be stronger if the authors could provide experiments for a few more downstream tasks. I think predicting entanglement entropy should not be too hard, and I would like to see if pre-training helps there.\n\nIt would also be interesting to see for what downstream tasks the pre-trained model Q-TAPE is not particularly helpful now. From the perspective of building along this line of thinking, I think illustrating the shortcomings of the current model is actually very useful.\n\nThere are some typos in the Limitations paragraph in Section 5: \"... for expereiments.Though\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Machine learning for quantum many-body physics is an important research topic that has been subject to extensive studies in recent years. As of now, a good pre-trained model for quantum physics problems is unavailable. This work could initiate a fruitful line of research to build a powerful pre-trained model for quantum many-body physics.\n\nThe extensive experiments conducted in this work showcase the potential for using a good pre-trained model to significantly reduce the amount of experimental data needed. For example, in the largest system size, L = 31, and smallest experimental data of Fig. 3, the performance improved by >15% using the pre-trained model.", "weaknesses": "The authors only considered two downstream tasks (classifying phases and predicting properties).\nMore tasks would make the work even stronger.\n\nThe work is purely empirical (which I find to be a minor issue for this direction of research).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KQjJfboAo", "reviewer": "Reviewer_TiJ4", "summary": "This manuscript introduces a framework named QTAPE designed to address the challenge of quantum property estimation. The authors utilize an neural network-based model to leverage different types of quantum data to make the estimation.  Additionally, they showcase the model's effectiveness by deploying it to estimate properties in various many-body systems.", "questions": "- See \"Weakness\" section above.\n- In the section concerning the prediction of correlation functions for the anisotropic Heisenberg model, it would be beneficial if the authors could include an analysis of the prediction MSE for correlation functions associated with two quantum sites at **different** distances.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This manuscript employs a framework composed of unsupervised pre-training and supervised fine-tuning phase for estimating the properties.\n\n- The proposed method demonstrates versatility, as evidenced by its effectiveness across various types of tasks, as illustrated by the numerical experiments. \n\n- The proposed model exhibits satisfactory performance even in large-scale quantum systems.\n\n- The presentation of the paper is excellent.", "weaknesses": "- While the authors emphasize the unsupervised pre-training as a key novelty of their proposed method, I just feel that the approach they employ in the pre-training phase, involving the regeneration of measurement results, follows a similar logic to that found in [1]. In [1], the model generates measurement results from incomplete measurement results. Although the proposed model here incorporates additional input information, such as physical conditions, I still believe the authors should cite this previous work. \n- Furthermore, the fine-tuning phase of the model shares similarities to the framework described in [2]. Despite the results demonstrating the superiority of the proposed method in Table 1, it would be valuable for the authors to engage in a more in-depth discussion regarding the distinctions between their model and this prior work. Given that Q-TAPE without pretraining also outperforms NN-Classical Shadow, it would be appreciated if the authors could provide further analysis or intuitive explanations for the results to highlight the novelty of the proposed model.\n\n[1] Zhu, Y., Wu, Y. D., Bai, G., Wang, D. S., Wang, Y., & Chiribella, G. (2022). Flexible learning of quantum states with generative query neural networks. Nature Communications, 13(1), 6222.\n\n[2] Wang, H., Weber, M., Izaac, J., & Lin, C. Y. Y. (2022). Predicting Properties of Quantum Systems with Conditional Generative Models. arXiv preprint arXiv:2211.16943.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)