Junhua Zeng1* {jh.zenggdut@gmail.com}, Chao Li2* {chao.li@riken.jp}, Zhun Sun3 {zhun.sun@tohoku.ac.jp}, Qibin Zhao2
1,4{qibin.zhao@riken.jp}, and Guoxu Zhou {gx.zhou@gdut.edu.cn}
improve the overall performance over a plain AlphaZero agent (seeMethods and Supplementary Information for details).

*Equal contribution 1Guangdong University of Technology, China; 2RIKEN-AIP, Japan;

Played
game

(1)

L

Fig. 4 | 1D MERA. The entanglement entropy of a block of length L for a multiscaleentanglement renormalization ansatz (MERA) is upper-bounded as SL ≤ logχ × ∂ΩL, with
∂ΩL the boundary of region ΩL in the tensor network (that is, the number of links crossedby the blue dashed line) and χ the bond dimension. It is easy to realize that ∂ΩL = O(logL),and therefore one has that SL = O(log L) for the 1D MERA. This calculation matches the
behaviour from conformal field theories (CFTs) in (1 + 1)D, and corresponds, precisely,
to the lattice version of the Ryu–Takatanagi prescription to compute the entanglement
entropy in anti-de Sitter (AdS)/CFT196. Entanglement is thus the area in holographic spaceof the minimal surface separating the two regions. This is one of the key observations that
motivates the analogy between MERA and AdS/CFT.

(1)

(1)

(2)

(2)

(2)

(3)

(3)

(3)

(u , v , w )(u , v , w )(u , v , w )Machine learning
Several promising connections between TNs andmachine learning have been put forward. In REF.16, it was
Updatedshown that deep learning architectures can be underLearningmodel
stood using the language of quantum entanglement.
To name a couple of examples, convolutional networks
Policy headcorrespond to specific cases of TTNs, and recurrent neuPlayed games(u, v, w)buffer
ral networks correspond to MPS. More generically, theSamplewhole machinery of quantum information and entanrandom state
glement theory can be applied to understand neural
Value head
networks in new ways. One must, however, be careful,since in general neural networks are characterized by
Training labels
Network input
Neural network
Pre-generated
nonlinear functions, whereas TNs are linear and theresynthetic
169
demonstrations
fore obey the superposition principle. In REF. , it was
FIG.is1.anOverview
the machine
shown that there
equivalenceof
betweenrestrictedlearning workflow for a TTN classifier. A data instance x from a classical data set is transformed into(0)by feeding
as shown The neural network (bottom box) takes
a quantum|!type(x)!
previously played games and synthetic demonstrations. The updated network
Fig. 2superposition,| Overview of AlphaTensor.
Boltzmannmachines (astate
simpleof neural
network)each element of the data vector xi through a local map φ(xi ) defining a qubit
and TN schematicallystates. Boltzmann
werealso shownis sent to the actors (top box), where it is used by the MCTS planner to generate
as inputa tensor
S t , and outputs
onmachines
the Bloch
sphere.
The isometric tensors of the TTN (triangles) define a coarse graining of collectionsof thesequantumfeaturesamples (u, v, w) from a distribution
to be connectedtosomeclassesofTNstatesinarbitraryoverpotential
next
actions
to
play,
and
an
estimateofthefuturereturns
new games.
vectors at progressively higher levels of scale. At the highest level of scale, the projection of the extracted feature vector onto a collection ofdimensions170. In REFS171–174, MPS and TTNs were used(for example, of −Rank (S t )). The network is trained on two data sources:a classification decision.
weight vectors "W" | defines
for supervised and unsupervised
learning tasks of classifying images. Finally, in REF.175, it was discussed how
also
ensuresany entanglementin the quantum ML
This has the appealing property that each data vector x is
Nature | Vol 610 | 6 October 2022 | 49
quantumstatescircuits
basedon MPSthat
and TTNs
could bemodel arises
fromcorrelationsin an ensemble of data andmapped into a weighted superposition of all correlations
used to implement
machinelearning
tasks in near-term
176
explored
how probabilistic
quantumnot
devices,andaREF.
fromprioriassumptions
about preexisting correlations
xi x j · · · xk of orders 0 through L with each xi appearing at
graphical models motivate the concept of generalizedforindividual
data
vectors
[28].
We
will
parametrizethemapmost once, and so a tensor-network weighting vector in thisTNs, in which information from a tensor can be copied
froman L-dimensional
data vector x to an ensemble
space can efficiently select out the most relevant correlations
and reusedin otherparts of the network,classicalthus allowingof L oftwo-levelsystems (qubits) as
from this exponentially large set. The factor a can be used tofor new types
variational wavefunctions.

Representation for complex quantum systems(Orus, Nature Phys.’19)

Emergent geometry
In the framework of holography, there is a very suggestive connection between entanglement, TNs and quantum gravity: it looks like the MERA is a lattice realizationof a space with some geometry, in which the curvature
is somehow linked to entanglement. The observation
implies that space- time geometry may emerge fromthe underlying structure of entanglement in complex
quantum states. An instance that has been studied in somedetail is the possible relation between MERA and the
AdS/CFT or gauge/gravity duality162. This connection
between TNs and quantum gravity was originally noticed
in REF.15, and later investigated in other studies163–166.
More specifically, for a scale- invariant MERA, thetensors in the bulk can be understood as a discretized
AdS geometry, whereas the indices at the boundary
correspond to the local Hilbert spaces obtained after adiscretization of a CFT (FIG. 4). The connection can bemade more formal by taking the continuum MERA41
and evaluating the metric of the resulting smooth spacein the bulk, with the curvature of the geometry being
linked to the density of disentanglers164,165. Currently,
the connection is intriguing and has motivated a lotof research, especially from the string theory community. In particular, there have also been claims that
MERA does not correspond to an AdS geometry, butrather to a de Sitter geometry167. In recent work, however, it was shown that MERA is neither AdS nor deSitter, but rather a lightcone geometry168. Although
this connection is certainly suggestive and remarkable,

Tensor-network learning models

Discovering faster matrix multiplication(AlphaTensor, Fawzi et al., Nature’22)

Tensor Network Structure Search (TN-SS):

Various TN models

Boltzmann machine
A specific type of neuralnetwork in which the target isto reproduce some Gibbs
thermal probabilities.

MERGE
Linguistic operation introducedby Noam Chomsky, whichpicks up two entities (forexample, noun and adjective)
and produces a new one
from the two (for instance,
noun phrase).

546 | SEPTEMBER 2019 | VOLUME 1



Language models
L
2!
$
From the perspective of computational
linguistics,(0)|! (x)! =φ ( j) (x j )|i j !.
(1)
probabilistic language models used for speech and text i j
j=1 Thisi j =1
recognition were found to have a TN structure.
is aconsequence of the fact that Chomsky’s MERGE operation
can be understooda physicalcoarse-grainingis
of accomplishedinforThat is, asthe
parametrizationin terms of local
( j)
.
Such
probabilistic
modelsusually
have
the element into a superposimation177
maps φ (x) mapping a single data
form of a TTN or even an MPS, for example, loop-freetionthisof matches
qubit states(seeobservation
Fig. 1). In
TNs. In turn,the empiricalthatorder to ensure that the full(0)
mapneural! networks(x) maps
dataat language
instance into a normalized vector
convolutional
are eachquite good
processing.
In connection
withwetherequire
results mentionedin Hilbert
space,
thatin the previous section, it is clear that this is indeed so$ ' which'2
because such neural networks are TTNs,
encode' = 1 ∀ x.
'φ ( j) (x)(2)
i found in
the renormalization group structure of language
177
i
REF. , and are therefore naturally well-suitedfor the task.

FurtherThis
topicscondition is satisfied by the phaselike encoding
There are many other interesting results related to TN)(π x − xstates and methods, and I cannot attempt to summarize
min
φ
,
(x)
=
cos
0 I outline a few that I
all of them here. Nevertheless, here
2
x
−
x
max
min
believe are particularly relevant.

(3)

that has been used in Refs. [28,36,46,63] to encode data forquantum-inspired ML applications and is shown schematically by the Bloch sphere representations of each classicaldata element in Fig. 1. In Ref. [56] a quantum-inspired
algorithm using MPSs (known in the numerical analysis community as tensor trains [54]) found good performance in
certain learning tasks using the map
φ1 (x) = ax.

Language Spacewith a ∼ O(0.1), which satisfies Eq. (2) and so is suitablexfor quantum discriminative applications. Noting that a xmax&
1 ∀ x, we can use the small-angle approximation to findx
+ O(a3 ),(6)
φ0 (x) = 1 + O(a2 ), φ1 (x) = axmax
and so this map has the same essential features as Eq. (4). A
comparison of model performance utilizing the maps (5) and(4) will be given in Sec. III.

(4)

We now turn to building an unsupervised feature extractor
as a TTN acting on the encoded quantum data description,
following Ref. [29]. This procedure is depicted graphically by
the dotted blue box in Fig. 1. We assume that we have a collection of M training data vectors {xm }, m = 1, . . . , M, that has
been encoded into a collection of training states {|!(xm )!}.
Using this map and these states, we would like to build amodel f (x) = "W |!(x)! parametrized by a weight vector |W !
in the many-body Hilbert space, to perform some machinelearning task (e.g., classification). Given our collection oftraining vectors, the optimal weights for a given task can be

Search for the most suitable TNmodel for the task

Unknown algorithms

Prior Arts…
…

Human Experts Research Work ow

Figure1. A basic
workflow to research.
illustrate how human experts do
Building LLM-agent to mimic humanexperts
for innovative

TNGA: Genetic Algorithm (Li and Sun, ICML’20)
TNLS: Stochastic Search (Li et al., ICML’22)
TnALE: Alternating Enumeration (Li et al., ICML’23)
GREEDY: Greedy Algorithm (Hashemizadeh et al., arXiv, 2020)
SVDinsTN: SVD-like Optimization (Zheng et al., CVPR’24)
Bayesian TN-SS (Zeng et al., NN, 2024)……………………

innovation in research.

…

…

…

Global architecture of tnGPS. We conceptualize tnGPS
as a system where the inputs are known TN-SS algorithms
encoded in Python, and the outputs are newly discovered
TN-SS algorithms. Inspired by the way human experts conduct innovative research, tnGPS is designed with a global
pipeline similar to the one depicted in Figure 1. In thisthe LLMResults
acts as an agent
to performthe functions
4.design,
ExperimentalWork
ow of tnGPS
of each phase shown in Figure 1, replacing human experts.
Below, we introduce theNumerical
specific implementationof eachEvidence
block in detail.
Idea Pool, also referred to as “the pool” for brevity
throughout the paper, is defined as a set of algorithms described as (algorithm, score).
Here,algorithm contains various implementations of the function GenerateSamples( · ) as defined in Eq. (2) of a TN-SS
algorithm using Python, and score is a scalar value indicating the algorithm’s performance in the TN-SS problem.
To help LLMs understand TN-SS algorithms effectively, allalgorithms in the pool are standardized with a unified
interface, as depicted in Figure 2. Additionally, the text
in Figure 2 will serve as the “pilot” for constructing theprompt, as discussed at the end of this section.

follow a similar …
hypothesis or principle. In Idea Dropout…
…
(ID), this phase models the behavior of human experts by…
filtering out certain ideas from the pool, allowing them to fo…
cus only on the most interesting ones for deeper study. The
and RSE, of different algorithms on both the training andmetrics used…
in
ID
are
typically
multivariate,
encompassing
tnGPS:
Discovering
Unknown
TensorNetwork
Structure
SearchAlgorithms
via
Large
Language
Modelscomponents
resource consumption, or unexpected output formats occur,
Settings
oftnGPS.
Weuse
threeexisting
TN-SS
algorithms
ically,we extend TNGA by relaxing its binary constraint
test sets. As shown in Table 2, the three algorithms gen…
clarifies
therestriction.
#
Output:Independently
improvements
overtheseandAlgorithms
that will increase their
the sandbox willmake
immediately
terminate theprocess
Knowledge
Categorization(KC)
PromptKnowledge
categorisation
(KC).
In
this
we
for encoding
the phase,
topology with integers
ascredone in the preKnowledge
Recombination(KR)
Prompterated by tnGPS achieve comparable performance on the# new_individuals: List, len new_individuals_numbers, contains integer
factorssuchaspersonal
interests,
performance,
trends,orasinputs:
TNGA
(Li
&
Sun,
2020),
GREEDY(Hashempractical
performance(notfromon thecode
efficiency,
Insights gained from the generated algorithms: vious works (Li et al., 2022; 2023). For TNLS, we fix the training set, while ‘Ho-2” outperforms other algorithms on
remove theproblematic code
the job
queue
awaiting readability and parallel# numpy vectors. Each vector's len is the same as the len of the vectors in
Figure4. Thelevel).
key prompt
in knowledge
recombination
(KR).
to
run
the
code
in
a
relatively
isolatedsetting
implementation.
processing
You are usedencouraged
to be creative
to incorporate
novel ideas. environment
Figure 3. The
key promptFurthermore,used inTheknowledge
categorisation(KC).
permutation
andset the cluster
template used inconthe algorithm to# history_populations.
elements are within
range
average in the test set. Figure 10 visually demonstrates thatateclustersofalgorithms
using
LLMs.
Each
izadeh
et
al.,
2020)
and
TNLS
(Li
et
al.,
2022).
The
evalFinally,
thecomplete
prompts
used
in
KC,
KR,
II
and
even
randomness
due
to
the
large
scale
of
the
idea
pool.
tnGPS,
can leverageinsights
gainedfromthe existingalgorithms
and enabling
the embedded
knowledgeinfor novel
generation
Algorithm
1: # sentence
centroid specifies the goal and the orange sentence# [1,hyperparameters[‘code_upperbound']].
be a complete graph,simultaneous search
for TN
The
purplethe LLMs
curves associatedwith thealgorithm
three algorithms
generated byusing
small-scaletraining
data
beforeconducting
high-cost…
The purple sentence specifies theFiguregoal6. oftheprompt.
topology
andranks.
In GREEDY, we piece
modify its objective
4. areExperimental
Resultsfollowing the same architecture as shown
tnGPS tend to reach smaller values of the objective function
uation
phase
in
tnGPS
calculates
Eq.
(1)
for
each
generThe key prompt used in diversity injection (DI). Theclarifies
Non-Markovian
searching
dynamicDI
constructed•
tains
similaralgorithms,
representingadistincttherestriction.
return new_individuals(omitted)function from RSE to the function in (1), and further allow
purple sentence specifies the goal and the orange sentence clarifies
formalnumerical
experiments.
If
programerrors,abnormalcompared to other methods.
Inverseannealingformutation•In
this
section,we
use
severalbenchmarks
todemonstrateatedalgorithm,
averagingtheresults
overtheimagesthe
In this work,
innovationisconsideredasanoperationthatthe restriction.
the algorithminto bothincrease and decrease the ranks duringinFigure8.
Itconsistsofthreeparts:
interfacedescriptionofknowledge
on
how
to
solve
TN-SS.
In
doing
so,
we
first
New insights gained from the generated algorithms. The
that tnGPS
can discover new algorithms that outperform
resource
consumption,orunexpected
outputformats
occur,
Algorithm
N:5.
# centroid
Gaussian
perturbation
mutationthesearch.
•FigureThe
keypromptused
inincremental
innovation(II).
To select a preferred algorithm in ID, we need to implement …
codes of Ho-1,2,3 are provided in Appendix B. These codes
training
set.
In
Eq.
(1),
we
set
=
5
and
use
the
same
SOTA methods
for
TN-SS. goal-oriented
Additionally, we conductprompts
abla(Figure
2),
those
followingalistof
generates
new
ideas
from
existing
ones.
As
shown
in
Figthat
the
LLM
tends
to
improvecode
aspectssuch
aseffiation. To meet these requirements, we designed the promptBest
structure
crossoverthe•sandbox
will and
immediately
terminate
the process
Theandparameter
settingsof the algorithms
aretheas follows: inreveal several new insights on how to improve the effectivetion aexperiments
to evaluate the specifies
impactof eachcomponentThe
purplesentence
the
goal
and
the
orangesentence
simplify
initializetheclusters
with
each
algorithm
in
Give
me
novel
‘GenerateSample’thatismethodologicallydifferent
fromtheFigure2.
The
promptused
forinterfacedescription.
the roulette selection twice. First,
weperform
selectionatshown in Figure 7. It’s important to note that the prompt inciency,
compression
ratio
functionasinprevious
work
(Li
Sun,
TNGA,
we&
setstructure↵ = 100, of= the
5, thedata
elimination rate toalgorithms
andscores(e.g.,
Figures
3,
4,
5,
6),
and
formatness and efficiency of solving TN-SS problems. First, thewithintnGPS onYou
the discovery
performance.
readability,
andparallelism,
which
are
not
the
focus
The
algorithm
discovered
bytnGPS
implicitly
re
ects
the
hiddenabove
algorithms.
areencouragedtobecreativetoincorporate
novel
ideas
removetheproblematic
code
from
the
job
queue
awaiting
Figure 7 was developed through aure
trial-and-error
process,
1,
we
model
innovationasatwo-stageprocess
consisting
clarifies
thestack
restriction.
10%, and the mutation probability to 25%; in TNLS, we set
but
do
not
simplymethods
together.
the knowledge level. This involves
selectingthecentroids
pool.
Once
a
new
algorithm(e.g.
one
generated
bytnGPS)
Boundary
mutation2020).
The
hyperparameters
required
in
tnGPS
are
listedIn TnALE, the number of samples in each iteration is deterthis
phase.
To
prevent
this,weincludespecific
instruc•
making it dependent on the specific LLM used and our code-of
restriction(Figure
7).
These
three
parts
are
concatenated
c
=
0.99;
and
in
TnALE,
we
set
L
=
0,
L
=
15,
r
=
1,
4.1. Natural Images Compression
implementation.
mined by other hyperparameters.
writing conventions. As a result, manualadjustments
mayofknowledge
recombination(KR)andincremental
innovaand
D
=
1.
For
all
methods(including
those
generatedof clusters to determine whichbe necessary
clusters
will be considered. tions(highlighted
inaimorange
in Figure5)totoguide the LLM
in
Table
1.
For
this
experiment,
we
set
m
=
2,
n
=
1,
togethertoprovidecomprehensive
instructions
to
the
LLM.
is
coming,we
promptthe
LLM
toevaluatethemethodologwhen using different LLM models.
In
thisexperiment,
we
to
use
TN-SS
algorithms
Diversity
injection(DI).
We
implementDIbyleveraging
Algorithm
1:#
centroid
Incremental
Innovation(II)perform
Promptselection
Diversity
searchfor optimal
topology
andInjection
ranks improvements
for a tensor(DI)networkPrompt
away
from
these
unintended
and focus
on the values
Second, …
withintheselectedclusters,
we
at
7
tio
(II)
These
phasesarekeyingredientstocreate
↵
=
↵
=
100,
c
=
5
and
#Iter
=
30.
Additionally,we12(omitted)Concluding
Remarks(TN)LLM
to represent
naturalimagesnew
with fewer
parameters. in the algorithm pool. In
thetocreateclusters
ical
similarity
betweenthenewalgorithm
andtheclustertargeted
modifications.
…
4.
Experimental
Results
Figure 6. The key prompt used in diversity injection (DI). The
Algorithm level
N: # centroid
the algorithm
to choose the… preferred algorithm
fromselectgpt-4-1106-preview
as
the
LLM
model,
applybeginning
Data preparation.
We randomly
select14
images fromin not only
research
but
also
in
strategy
andentrepreneurInterface
description
…
doing
so,
we
designthepromptas
Figure6,
instructing
purplesentence
specifies
the
goal
and
the
orangesentence
clarifies
centroids,
which
are
the
algorithms
with
the
best
scoresinthe BSD500 dataset (Arbelaez et al., 2010). These images=============
each cluster.
This “bi-level” processis
repeated
until
the
…
ing
a
temperatureof
0.7
uniformly
acrossall
prompts.
After
…
In
this
section,we
use
severalbenchmarks
todemonstrateareconvertedtograyscaleandresized
to
256
⇥
256
pixels.
New Algorithm:
the
LLM
to
generate
TN-SS
algorithms
thataremethodthe
restriction.
ship(Rubin&Abramson,
2018;
Xiao
et
al.,
2022).
More
A
list
of
algorithms(andscores)tnGPS:a
LLM-driven
frameworkfor
discovering
new
TN-SS
algorithms.
…
•
Each imageeach
cluster.
The
newalgorithm
isthenassignedan
index
desired number of algorithms has been selected.
Algorithm
1: is then reshaped into an order-8 tensor by the
implementation,
weselectthe
top-threealgorithms
from
the
that
tnGPS
can
discovernewalgorithms
thatoutperform+==============
default Python reshaping
function.
The 14pre-processed
ologicallydistinctfrom
thecentroidsofexisting
clusters.
…
1
specificallyspeaking,
referstotraining
theof generatingSOTApool
Which algorithm in the above is methodologically most similar to the newalgorithm?
Purposeandrestriction
instructions.
imagesare split intoKR
two sets:
4 imagesforandprocess10(omitted)(excluding
the
input
algorithms),
termed
Ho-1
,
Ho-2,
to
declareits
clustermembership.
The
key
promptusedinmethodsfor
TN-SS.
Additionally,
weconduct
ablaJust
give
me
the
function
number
with
no
other
words.
Knowledge recombination (KR).
Guidedbytheworkflow
Following
this,
the
newly
createdalgorithms
from
DI
serve
Figure 7. The prompt used for format restriction of the output.
images for testing.
“Pilots”
for
the
nal
ation.
ToN:meet theserequirements,
we
designed the prompt
Algorithm
tnGPS
isdesignedbyprompting
LLMs
to
mimic
human
experts.
•ideas
bymerging,
integrating,orreconfiguringexisting
The purple sentence specifies the goalnew
and the orange
sentence
and
Ho-3,
as
the
outputs
oftnGPS.
The
three
algorithms
tionexperiments
toevaluatetheimpactofeachcomponent…
Formatrestriction
endof tnGPS. for
Settings
We usenew
three existing
TN-SS
algorithms
this
phase
is
illustratedin
Figure3.
shown in Figure 1, KR is a fundamental
ascentroids
clusters.
The
newly
createdalgorithms
clarifies the restriction.phase to generate
shown
in Figureimportant
to notethe promptinIndependently
make improvements
overGREEDY
these Algorithms
that thatwill increase
theiras inputs:
TNGA
(Li7.
& It’sSun, 2020),(Hashemprompt
construction
willbeevaluated
andcomparedwiththeexisting
TN-SS
withintnGPS
on
the
discovery
performance.
ideas.
II
involves
gradualimprovements
or
minor
modifipractical
performance
(not
on
the
code
efficiency,
readability
andparallelnovel TN-SS algorithms in tnGPS.
thisprompts
phase,
from
DI
serve
as centroids
new clusters.process,Since relyizadeh etal., 2020)and TNLS(Li et al., 2022). for
The evalFinally, theIncomplete
usedthein KC,input
KR, II and
Figure7
wasdeveloped
throughatrial-and-error
processing
level).
are calculates
encouraged
to(1)
befor
creativeto incorporate novel ideas.
Figure 3. The key prompt usedinconstructed
knowledge
categorisation(KC).
uation phase
inYou
tnGPS
Eq.
each
generalgorithms.
LLMs
provide
us
new
ideas
of
solvingmore
broad
tensorproblems.
DI arefollowing
the same architecture
as shown•
The
Full
PromptIdea
dropout(ID).
The
dropout
isconducted
byrandomly
consists of N 0 pairs of (algorithm,
score).
Theing
on existingknowledge
can lead 1toformalizes“path depencations
to existingideas.
Additionally,
atedsolely
algorithm,
averagingtheresults overthe images in theFigure
in Figure 8. It consists of threeparts:
interfacedescription…

Exploring the enormous language space of
LLMs for autonomous TN-SS algorithm
discovery, saving human experts from the “The limits of mylabor-intensive algorithm design process and language meanletting them focus on more challenging the limits of my world.”— from “Tractatus Logico-Philosophicus”
problems.

In this work, the main contributions are:…

…

Incremental Innovation (II): Make gradualimprovements to existing ideas.

…

When TN-SS Meets LLMs

042408-3

Motivation:…

…

K n o w l e d g e R e c o m b i n a t i o n ( K R ):Generate new ideas by merging existingones.

Experiments (Evaluation): Test andscore ideas to validate their potential.

Algorithm Space…

…

Diversity Injection (DI): Introduce new,orthogonal ideas through brainstorming or
external feedback.

B. Construction of unsupervised feature extractor)(π x − xmin
φ1 (x) = sin
2 xmax − xmin
www.nature.com/natrevphys

φ0 (x) = 1,

scale the data and avoid numerical overflow. Unfortunately,
this map does not satisfy the condition in Eq. (2) and so is notsuitable for application on quantum hardware. We can define
a generalization of Eq. (3) as
( x )
( x )
φ0 (x) = cos a, φ1 (x) = sin a
,
(5)
xmax
xmax

Idea Dropout (ID): Filter out lessinteresting ideas to focus on the mostpromising ones.

Indices...

…

Change of basis

Singular value

Artificial intelligence
In this section, I will comment on the recent observationthat neural networks (such as those used in deep learning) are in fact particular cases of TNs, and on the use of
MPS to improve some methods of artificial intelligence.
Additionally, I will outline the result that syntactic relations in language have a TN structure that is inheritedby probabilistic language models.

ΩL

Knowledge Categorization (KC): Re ne
ideas into knowledge clusters.

ActingSingular value

the role played by TNs in the quantization of gravity
remains unclear.

tnGPS is designed with a pipeline of prompts to harness
LLMs in order to discover novel TN-SS algorithms, mimicking the innovation process of human experts.

…

…

…

PHYSICAL REVIEW A 104, 042408 (2021)

TREE-TENSOR-NETWORK CLASSIFIERS FOR MACHINE …

…

…

REVIEWS

Change of basis
T n (Fig. 1a) is the tensor representing the matrix multiplication bilinear
operation in the canonical basis. The same bilinear operation can be
expressed in other bases, resulting in other tensors. These different

Idea Pool: Gather information throughliterature reviews and paper retrieval.

…

A tensor network (TN) is a mathematical representation of a large,
complex object that can be decomposed into simpler, interconnected parts.

How Human Experts Conduct
Innovative
ResearchtnGPS:
Discovering Unknown
Tensor Network Structure Search Algorithms via Large Language Models…

Vision and Contributions

Carlo tree search (MCTS) planning procedure. The network takes as
input a state (that is, a tensor S t to decompose), and outputs a policy
and a value. The policy provides a distribution over potential actions.
As the set of potential actions (u(t), v(t), w(t)) in each step is enormous,
we rely on sampling actions rather than enumerating them21,22. The
value provides an estimate of the distribution z of returns (cumulative
reward) starting from the current state S t . With the above reward
scheme, the distribution z models the agent’s belief about the rank of
the tensor S t. To play a game, AlphaTensor starts from the target tensor(T n) and uses the MCTS planner at each step to choose the next action.
Finished games are used as feedback to the network to improve the
network parameters.
Overcoming the challenges posed by TensorGame—namely, an enormous action space, and game states described by large 3D tensors
representing an abstract mathematical operation—requires multiple
advances. All these components, described briefly below, substantially…

Synthetic demonstrations
Although tensor decomposition is NP-hard, the inverse task of constructing the tensor from its rank-one factors is elementary. Hence,
we generate a large dataset of tensor-factorization pairs (syntheticR
demonstrations) by first sampling factors {(u(r ), v (r ), w (r ))}r =1 at random,Rand then constructing the tensor D = ∑r =1 u(r ) ⊗ v (r ) ⊗ w (r ). We train the
network on a mixture of supervised loss (that is, to imitate synthetic
demonstrations) and standard reinforcement learning loss (that is,learning to decompose a target tensor T n) (Fig. 2). This mixed training
strategy—training on the target tensor and random tensors— substantially outperforms each training strategy separately. This is despite
randomly generated tensors having different properties from the target tensors.

…

and the Internet of Things in Manufacturing, China.

Tencent Inc., China; Key Laboratory of Intelligent Detection…

R

Parameters: {u(r ), v (r ), w (r )}r = 1: length-n2 vectors such thatR
Tn = ∑r = 1 u(r ) ⊗ v (r ) ⊗ w (r )Input: A, B: matrices of size n × n
Output: C = AB(1) for r = 1, …, R do(2) mr ← (u(1r )a1 + ! + u(nr2)an 2) (v (1r )b1 + ! + v (nr2)bn 2)(3) for i = 1, …, n2 do(R )(4) ci ← w (1)i m1 + ! + w i mR
return C

TnALE: Solving Tensor Network Structure Search with Fewer Evaluations

1155 C. Experiment details
1156
3
4
1157 C.1. Low-rank structure of the optimization landscape1158 To verify the low-rank structure of the optimization landscape of (1), we empirically check the singular values of the1159 landscape tensor using the synthetic data. To be specific, we re-use the fourth-order tensor in the experiment for TN-PS, i.e.,
1160 TR (order-4) in Table 5. Here we remove the influence of unknown permutations and calculate the objective for all possible
1161 combinations of values of the TN-ranks. As a result, for each data, we have a landscape tensor (a tensor whose entries are
1162 values of the objective function) of order-4, and the modes of the tensor corresponding to the four TN-ranks. Figure 3 (a)1163 shows the singular values of the landscape tensor unfolded along different modes on average. We see that the landscape1164 tensor provides a significant low-rank structure in these data. We also depict the complete landscape (contour line, unfolded
1165 along the first two modes) with respect to Data A in Figure 3 (b). We can see that the obviously repeated pattern shown in
1166 the figure is the main reason leading to the low-rank structure of the landscape.
Knowledge
Idea Pool
1167
Idea Dropout (ID)
Categorisation (KC)1168(
, SCORE，CONTEXT)
161169
1 tnGPS: Discovering Unknown
45 Language Models
TensorNetwork Structure Search Algorithms via Large
2
1170
14
(
, SCORE，CONTEXT)
40
1171
12
1172
35
environment
to
run
the
code
in
a
relatively
isolatedsetting1173
30(, SCORE，CONTEXT)
101174
Algorithm 1: # centroid
25
using small-scale training data
before conducting high-cost
tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language1175
Models8201176(omitted)formal numerical experiments.
If program errors, abnormal
6
15
6
1177
10
Diversity Injection (DI)resource
consumption,orunexpected
output formats occur,
Algorithm N: # centroid
4
1178
3
Function description:
5
Algorithm 1:1179
the sandbox will immediately terminate the process and5 Give me a novel ‘GenerateSample’ that is methodologically different from thedef GenerateSample(history_populations, fitness_scores, best_individual,
2
10
20
30
40
1180
new_individuals_numbers, current_iteration, maximum_iteration, hyperparameters):
Algorithm
1 score: You are encouraged to be creative to incorporate novel ideas
abovealgorithms.
Indicesqueue awaiting
remove the problematic code from the job
1181
tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models
but do not
simplystack
methods
together.
(omitted)(a) Averaged singular values for the 4th-order landscape tensor.
(b) Optimization landscapes (the inverse 1/f (x)) wrt. the1182# GenerateSample: Function takes in integer vectors and output integer vectors.
implementation.
tensor of order-4 and correct permutation.
Algorithm N:
1183
Multiple-Stage Innovation# Inputs:1184
Experiments
Figure 3. Averaged singular values and Optimization landscapes for the tensor of order-4.
Algorithm
Nscore:# history_populations: Dictionary. Keys are integer strings from '1' to some1185
(Evaluation)#
Algorithms
1
to
N
are
implementationsofthe‘GenerateSample’
function. A lower score# larger value. Each key contains a list of integer numpy vectors.
1186
4 key prompt used in diversity injection (DI). The
Figure
6. Theimplies better
performance.
# fitness_scores: Dictionary. Keys are same as history_populations.
C.2. Details for the experiment of TN-PS (w.r.t., Table 1).
1187
Incremental
Knowledge
purplesentence
specifies
the
goal
and
the
orangesentence
clarifies
1188
Innovation(II) and what
Learning from their results, think about
what works
doesn’t,provideM(omitted)
Goal. (KR)In thisnovel
experiment, we intend to verify the superiority of TnALE in solving TN-PS problem.
Recombination
In this section, we use several benchmarks to demonstrate
methods
with lower scores. You are encouraged to be creative1189
to incorporate novel
therestriction.
Data generation. For the synthetic data of TR (order-4, order-6, order-8), PEPS (order-6), HT (order-6) and MERA
ideas
but
do
not
simplystack
methods
together.
# hyperparameters: Dictionary. Keys are strings contain the constants
1190
SCORE SCORE
SCORE
that
tnGPS
can
discovernewalgorithms
thatoutperform(order-8)
topology,
were-use
the
data
from
(Li
et
al.,
2022).
For
thegeneration
of
data
with
TW
(order-5)
structure,
we9. Objective vs. number of sample curves of different algorithms on four training images.
# used for computation. Default values should be provided using .get().
Figure1191
modemethods
to be 3, meanwhile,
the TN-ranks
are randomly selected{1, 2, 3}. Thenwe
SOTA
for TN-SS.
Additionally,
wefromconduct
abla1192 set the dimension of each tensor# Output:
drawprompt
samples from Gaussian distribution N (0, 1) as the values of core tensors. After contracting these core tensors
ation.
To
meet
these
requirements,
we
designed
1193 i.i.d.the# new_individuals: List, len new_individuals_numbers, contains integer
tion
experiments
to evaluate
theimpactof eachNaturalImage
Compression
Task:
TN-SS
to component
according to the tensor wheel
topology,
we uniformly
permutethe tensor
modesUseat random.
:
ideas.
Diﬀerent
colorsindicate
the
idea
from
diﬀerent
blocks.
1194# numpy vectors. Each vector's len is the same as the len of the vectors inshown
in TheFigure7.
It’s important
to note thatthe prompt(KR).
in nd
Figure4.
key
promptusedinknowledge
recombination
1195
tnGPS
on
the fordiscovery
performance.
TNsandtoTNLS
represent
naturalimages.
# history_populations. Furthermore, The elements are within range
Settings. In the experiment, within
we compact
implement
TNGA
comparison.
For
allthe methods, we use the same objective: workflow
indicators,
indicating
thethetransition
ofand
blocks.
1196orange
Figure7 wasdeveloped
througha trial-and-error# [1,hyperparameters[‘code_upperbound']].
The purple
sentence
specifiesgoal
thesentencefunctionprocess,asin Li & Sun (2020). Specifically, in the experiment the obejctive function of (1) is as follows:
1197
Objective:
clarifies
restriction.
makingitthedependent
on thespecificused: arrows
pointingfrom input
to output
for an LLM
operations.
return new_individuals
11198 and our code22
4.1.
NaturalImagesF (G, r) =+Compression·min
kX Zk / kX k ,
(46)
1199
✏(G, r)Z2T N S(G,r)
writing
conventions.
Asideas.
a result, manual adjustments may: clusters
containing similar| {z }|{z}
1200
compression
ratio (CR) we aim
relativeto
square(RSE)
be necessary when using different LLM models.
In thisexperiment,
useerrorTN-SS
algorithms to1201
that the LLM tends to improve code aspects
such as effi1202
Figure 2. The prompt used for interface description.
where X denotes the synthetic
tensor, and✏(G,r) represents
the compression
ratio
equalling
searchfor
optimal
topology
and
ranks
for
Settings
oftnGPS:
four
imagesfortraining,
tentoa tensor network
1203
ciency,readability,
andparallelism,
which
are not the focus images for testing.
tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models 1204
(TN)
torepresent
naturalwithfewerparameters.
Dimensionimages
of X✏(G, r) =.
Ho-1, Ho-2
and Ho-3
are algorithms
discovered
byontnGPSof
thisphase.
To
prevent
this,weincludespecific
instrucFigure
10.
Objective
vs.
numberofsamplecurvesofdifferent
algorithms
ten testing images.
1205
Dimension
sum
of
core
tensors
of
the
TN
under
(G,
r)
Provide runnable code that has implemented all your ideas (If any part1206
requires(highlighted
choice, use choicesfrom random).
Do not leaveanyguideplaceholder
for
tions
in
orangeinFigure
5)
to
the
LLM
Data
preparation.
We
randomly
select14
imagesfromby tnGPS), we set the upper limit for rank search to 4, theFunction description:
Table
1.
Parameters
involved
intnGPS.
Algorithm
me, all1:the functionality should be actually implemented. 1207
Your response
format parameter in (46) to be 200 and for the solver of the inner minimization, we select the Adam optimizer
number of iterations in searching to 20, and the number
Algorithm 1:#
centroid
We set the trade-off
def GenerateSample(history_populations, fitness_scores, best_individual,#Para. (X1000)
#Eva.
away
andKingma
focus
on(2014)
the withGaussian2
theBSD500
dataset(Arbelaezetal.,2010).
These
images1208<code>from these unintended improvementsofsamplesin each iteration to 100. For TNGA and theParameter
Description
process
Compression:anOut-of-Domain
Experimentnew_individuals_numbers,
current_iteration,
maximum_iteration,
hyperparameters):&Bathelearning
rate
of
0.001
and
use
Gaussiandistribution
N
(0,
0.1)
to
initialize
thecore
tensors.
Algorithm 1 score:(omitted)
1209
Your
code
algorithms generated by tnGPS, we initialize them with#Iter
Maximum iteration
targeted
modifications.
areconvertedtograyscaleandresized
to
256
⇥
256
pixels.
(omitted)
GenerateSample: Function takes in integer vectors and output integer vectors.
Algorithm N: ## centroid
same TN structures, which have TN ranks close to one but</code>↵1
Parameter in Eq. 3 for cluster selection
Algorithm N:with a 15% probability of changing each rank from 1 to
↵2Parameterin Eq. 3 for algorithm selection
Also,
you
don’t
need
to
add
any
other
words.
Each
image
is
then
reshaped
intoanorder-8
tensorbytheAblation
Experiments:# Inputs:=============
2. We then select the best TN structure from the TNGAm
Number of selected algorithms in ID
Algorithm N score:# history_populations: Dictionary. Keys are integer strings from '1' to some
initialization to initialize GREEDY, TNLS, and TnALE.
New Algorithm:
default Python reshaping function. The 14 pre-processednNumber of generated algorithms# Algorithms 1 to N are implementations of the ‘GenerateSample’ function. A lower score# larger value. Each key contains a list of integer numpy vectors.
impliesbetter performance.
# fitness_scores: Dictionary. Keys are same astnGPS:
history_populations.
cMaximum number of clusters
Algorithm
1:
Discovering Unknown Tensor Network Structure
SearchAlgorithms
via Large Language Models
Results. Figures 9 and 10 show how the value of the obimages are split into two sets: 4 images for training and 10==============
jective function (3) changes with increasing the number ofLearning from their results, think about what works and what doesn’t, provide M novel(omitted)
Which algorithm in the above is methodologicallymost similar to the new algorithm?
environment
to scores.
run thecodearein aencouraged
relatively
isolatedsettingto incorporate(omitted)samples for different TN-SS algorithms. Table 2 shows themethods
with
lower
You
to
be
creative
novel of the output.
Figure7.
The
promptused
forformatrestriction
imagesfortesting.
Algorithm
1:#
centroid
Just give me thefunction number
with nootherwords.contain the constants
using
small-scaletraining
data before
conducting high-cost…
ideas
but donot simply
stack methods
together.
searchonly
for the topology
orpermutation LLMs
of a TN, like
we GPT-4# hyperparameters:
Dictionary.
Keys
are strings
The
results
highlight
theimportance
of
the
'KR',
'II',
and
'DI'
averaged performance
metrics,
includingcompression ratio
MorepowerfulenhancetnGPS
performance
Algorithm
N:The
purplesentencethegoal and the orange sentence(omitted)formalnumerical
experiments. specifies
If program errors,
abnormal# used for computation. Default values should
be provided using .get().
extend them to fit the settings of this experiment. Specif-…

Neural network architecture
We propose a transformer-based23 architecture that incorporates
inductive biases for tensor inputs. We first project the S × S × S input
tensor into three S × S grids of feature vectors by using linear layers
applied to the three cyclic transpositions of the tensor. The main part ofthe model comprises a sequence of attention operations, each applied
to a set of features belonging to a pair of grids (Extended Data Figs. 3
and 4). This generalizes axial attention24 to multiple grids, and is bothmore efficient and yields better results than naive self-attention. The
proposed architecture, which disregards the order of rows and columns
in the grids, is inspired by the invariance of the tensor rank to slice
reordering. The final feature representation of the three matrices ispassed both to the policy head (an autoregressive model) and the value
head (a multilayer perceptron).

R

A meta-algorithm parameterized by {u(r ), v (r ), w (r )}r = 1 for computing
the matrix product C = AB. It is noted that R controls the number ofmultiplications between input matrix entries.

… …

Algorithm 1

Algorithm N: # centroid…

Give me a novel ‘GenerateSample’ that is methodologically different from theabove algorithms. You are encouraged to be creative to incorporate novel ideas
but do not simply stack methods together.

(1889-1951)

2

1# GenerateSample: Function takes in integer vectors and output integer vectors.
# Inputs:# history_populations: Dictionary. Keys are integer strings from '1' to some# larger value. Each key contains a list of integer numpy vectors.
# fitness_scores: Dictionary. Keys are same as history_populations.

# hyperparameters: Dictionary. Keys are strings contain the constants# used for computation. Default values should be provided using .get().

# Output:# new_individuals: List, len new_individuals_numbers, contains integer# numpy vectors. Each vector's len is the same as the len of the vectors in# history_populations. Furthermore, The elements are within range# [1,hyperparameters[‘code_upperbound']].

fi

fl

fl

fi

return new_individualsfi(omitted)

Algorithm
1:
Providerunnablecode that has implemented all your ideas (If any part
requireschoice,use choices from random). Do not leave any placeholder for
Algorithm
1 score:me, all the(omitted)
functionality should be actually implemented. Your response format<code>
Algorithm N:
Your code</code>Algorithm N score:# Algorithms
to N arethe ‘GenerateSample’ function. A lower score
Also,
you don’t1 need
toimplementations
add any otherofwords.
implies better performance.

Learning from their results, think about what works and what doesn’t, provide M novel
methods with lower scores. You are encouraged to be creative to incorporate novel
ideas but do not simply stack methods together.

Figure 4. The key prompt used in knowledge recombination (KR).
The purple sentence specifies the goal and the orange sentenceclarifies the restriction.

…

Function description:
def GenerateSample(history_populations, fitness_scores, best_individual,
new_individuals_numbers, current_iteration, maximum_iteration, hyperparameters):…

…

tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models…

1. We propose tensor-network-purposed GPT-driven structure search(tnGPS), a large language model (LLM)- driven automation framework
designed to automatically generate novel and effective TN-SS algorithms
tailored to speci c downstream tasks;
2. Experimental results demonstrate that the algorithms discovered by
tnGPS outperform existing TN-SS algorithms on benchmark data.
flfi

tnGPS: Discovering Unknown Tensor Network Structure Search
Algorithms via Large Language Models (LLMs)

02