MACS: Multi-Agent Reinforcement Learning for Optimization of Crystal Structures
Authors: Elena Zamaraeva
Christopher Collins
George R Darling
Matthew Stephen Dyer
Bei Peng
Rahul Savani
Dmytro Antypov
Vladimir Gusev
Judith Clymo
Paul G. Spirakis
Matthew Rosseinsky
Venue: NeurIPS 2025 poster ()
Keywords: multi-agent reinforcement learning
learning to optimize
geometry optimization
local optimization
atomic structure optimization
crystal structure
Primary area: reinforcement_learning
TL;DR: MACS: a new multi-agent reinforcement learning method for faster geometry optimization of crystal structures.

Abstract: Geometry optimization of atomic structures is a common and crucial task in computational chemistry and materials design. Following the learning to optimize paradigm, we propose a new multi-agent reinforcement learning method called Multi-Agent Crystal Structure optimization (MACS) to address the problem of periodic crystal structure optimization. MACS treats geometry optimization as a partially observable Markov game in which atoms are agents that adjust their positions to collectively discover a stable configuration. We train MACS across various compositions of reported crystalline materials to obtain a policy that successfully optimizes structures from the training compositions as well as structures of larger sizes and unseen compositions, confirming its excellent scalability and zero-shot transferability. We benchmark our approach against a broad range of state-of-the-art optimization methods and demonstrate that MACS optimizes periodic crystal structures significantly faster, with fewer energy calculations, and the lowest failure rate.

MACS: Multi-Agent Reinforcement Learning for Optimization of Crystal Structures
Elena Zamaraeva'
Christopher M. Collins"
George Darling
Matthew S. Dyer
Bei Peng" 33
Rahul Savani
Dmytro Antypov!
Vladimir V. Gusev
Judith Clymo
Paul G. Spirakis
Matthew J. Rosseinskyl2
'Leverhulme Research Centre for Functional Materials Design, University of Liverpool, UK
Department of Chemistry, University Liverpool, UK
School of Computer Science, University of Sheffield, UK
*Department of Computer Science, University of Liverpool, UK 'The Alan Turing Institute, London, UK
Abstract
Geometry optimization of atomic structures is a common and crucial task in computational chemistry and materials design. Following the learning to opti- mize paradigm, we propose a new multi-agent reinforcement learning method called Multi-Agent Crystal Structure optimization (MACS) to address periodic crystal structure optimization. MACS treats geometry optimization observable Markov game in which atoms are agents that adjust their positions to collectively discover a stable configuration. We train MACS across various compositions reported crystalline materials obtain a policy that successfully optimizes structures from the training compositions well structures of larger sizes and unseen compositions, confirming its excellent scalability and zero-shot transferability. We benchmark art optimization methods and demonstrate that MACS optimizes periodic crystal structures significantly faster, with fewer energy calculations, and the lowest failure rate. Code is available at https:::ss. /ttt::///tttii com/ htt///rcfmdd/acccc maacs
partially
approach against broad range of state-of-the
Introduction
In computational chemistry, geometry optimization of an atomic structure is the process of finding stable arrangement of atoms, following sequence of displacements in a 3-dimensional space until local energy minimum is reached [39 Within geometry optimization, our focus in this paper is on crystal structures, which are characterized by their periodicity. The arrangement of atoms within crystal directly determines its physical and chemical properties; hence the optimization of the crystal structure crucial to the discovery of new crystalline materials and finds applications in electronics, energy applications, information storage, and other domains.
In fact, an entire field within materials design, known crystal structure prediction (CSP), focuses on the computational prediction of stable crystal structures with desirable properties for subsequent synthesis in the laboratory [53] (44) Significant effort in a CSP workflow is focused on exploring the
"quual contribution
"orresponniig author, e-mail: Savan @ @liverpool.ac.uk
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

potential energy surface (PES) and performing geometry optimizations on candidate structures to minimize the energy and local atomic forces. In this context, the key requirement for an optimization method is the ability to quickly produce locally optimized equilibrium structure.
Existing approaches for geometry optimization of crystal structures include classical first- and second- order optimization methods such as the Daapeee--eecche--Gooaf--------
(BFGS) algorithm [4] 16 40] and the Conjugate Gradient [41] method, as well as methods tailored atomic structures such as the Fast Inertial Relaxation Engine (FIRE) [3] However, these methods often require either significant number of steps to optimize large structures or consuming calculations at each optimization step. This makes the time for optimization bottleneck in applications that require thousands of local optimization runs, such as CSP.
45 to improve the geometry optimization of crystal structures using mulli-agent reinforcement learning (MARL).. We observe that overall structural stability depends on the forces acting on individual atoms. The forces are partial derivatives (or gradients) of the energy with respect atomic positions, and all forces are zero at equilibrium where the structure achieves a local energy minimum. Both the energy and the forces are mostly determined by each atom's 's chemical properties and its surrounding local environment. Moreover, atoms of given element, as well as those of chemically related elements, tend to adopt similar chemical local environments, resulting in multiple similar local envirouments in an optimized crystal structure. Hence, it is natural to consider individual atoms as agents, moving independently but simultaneously to collectively discover an overall structure that is stable, i.e., a local minimum of the energy landscape. We therefore formulate crystal structure geometry optimization MARL problem, with the aim of learning decentralized policies for individual atoms to collectively discover a locally optimized structure.
In this work, we utilize the learning to optimize (L.2) paradigm 27
The energy (and most often forces) estimation is integrated in any optimization method, and various methods exist for this purpose. The available methods for computing energy/forces vary in their complexity and accuracy and, therefore, cost to compute. Lennard-Jones potentials (LJ) 226 [3] or the Müller-Brow methods, such as density functional theory (DFT), can estimate energies and forces for materials with high accuracy but at very much greater computational cost. We utilize CHGNet [12), machine learning interatomic potential model trained on DFT data, as offers fast and accurate energy and gradient estimates for crystal structures.
surface (37 are often used to model simple systems, while more complex
In this work, make the following contributions:
To the best of our knowledge, we are the first to apply MARL to address periodic crystal structure optimization. Our proposed method, Multi-Agent Crystal Structure optimization (MACS), presents novel formulation of periodic crystal structure optimization multi- agent coordination problem.
Our extensive experiments demonstrate that MACS optimizes the crystal structures signifi- cantly more efficiently than a wide range of state-of-the-art methods. These experiments cover diverse set of crystalline materials, including compositions with different elemental species, varying numbers of species, and distinct symmetry groups
MACS exhibits strong zero-shot transferability and scalability, maintaining efficiency in the optimization of larger structures from new, unseen compositions. Our work unlocks the potential of MARL for periodic crystal structure optimization.
2
Related Work
The L20 concept leverages machine learning to develop new optimization methods tailored to specific problems, and its application is rapidly expanding. This paradigm has been successfully applied to classical optimization challenges such as Bayesian swarm optimization (6,, black-box box
optimization [9] 28) adversarial training [55). or partial differential equations solving [88]
The geometry optimization of atomic structures is also target of the L,20 concept. In (2) the authors
propose a graph-based L.20 approach to optimization of finite, non-periodic atomic clusters using the IJ and Calcium silicate hydrate potentials [3], as well as the Stillinger-Wert Weber potentials (SW)
The authors show that their method achieves lower energy in optimized clusters compared to FIRE,

---

Adam [25]. and Gradient Descent [43] Another study [4 investigates the optimization of atomic
clusters with LJ, SW, and Gupta potentials 19, focusing on the minimization of energy in the cluster.
Molecular optimization tasks, when the main objective is to achieve stable molecule configurations in the least number of steps, are explored in [6 1 46 In (66), the authors utilize MARL train the MolOpt optimizer and benchmark it against three baselines: BFGS, FIRE, and MDMin [23]. The findings in [36] indicate that the MolOpt optimizer surpasses MDMin, exhibits performance comparable to FIRE, and is inferior to that of BFGS. Despite differences in the design of the Markov Decision Process (MDP) and the application distinct classes of chemical systems, we have added all baselines of [36] in our study to maintain consistency. In [46], the authors follow another approach, where using fixed optimizer variation of BFGS) they train the machine-leamingg potentials be more accurate during optimization.
Reinforcement learning (RI.) has also shown promise in other areas of computational chemistry, including the design of materials with specific properties [17 (24) and the optimization of the basin-
hopping routine in CSP [57]. For a comprehensive overview of other applications of RL in chemistry,
we direct the reader to the review in [42.
3 Preliminaries and Problem Formulation
3.1 Periodic Crystal Structures and Their Optimization
crystal structure is characterized by its unit cell, typically, parallelepiped, and the configuration of atoms within it. The unit cell repeats itself in all three dimensions, defining the infinite periodic arrangement of atoms (see Fig. 2 for a two-dimensional example).
Geometry or local optimization takes an initial structure as input and adjusts the positions of the atoms in the unit cell to achieve a structure where the energy is a local minimum. An efficient procedure to perform geometry optimization is crucial due to its extensive usage throughout computational chemistry. Atomic configurations physically stable structures of a material, and thus the properties of a material commonly depend on these structures. Therefore, calculations of optoelectronic, vibrational, mechanical, and energetic properties will begin with geometry optimization to achieve a local minimum.
local minimum on the potential energy surface (PES) represent
In the scope of crystal structure optimization, global and local optimization are distinguished by their targets. The global crystal structure optimization, which is the ultimate goal of CSP, aims to identify the global minimum energy structure for given composition, representing the most stable configuration. However, achieving globally optimal structure typically requires many iterations of structure generation or perturbation followed by local optimization (5 56 38 49)
Geometry optimization terminates under two conditions. The first condition, termed the condition of success, requires the norm of the maximum atomic forces within the structure to reach specified threshold. We use the threshold 0.05 eV/,, it represents typical use case in CSP applications (II. The second condition, termed the condition of failure, occurs when the maximum allowable number of optimization steps is reached without satisfying the condition of success. We set the maximum number of steps to 1000, which is generally sufficient to optimize the structures within this work using state-of-the-an methods such as BFGS, BFGS with line search, FIRE. Therefore, the problem of geometry optimization in our formulation is as follows.
Problem (geometry optimization): Given an initial crystal structure and the maximum of 1000 steps, autonomously adjust the positions of the atoms in the fixed unit cell to locally minimize all atomic forces in the structure to below 0.05 eV/À as quickly as possible.
3.2
Geometry Optimization Partially Observable Markov Game
We model geometry optimization of crystal structures as a partially observable Markov game POMG is multi-agent extension of MDP, with partial observability intro- (POMG) 32 31
Throughout this paper, all vectors and atomic positions are expressed in Cartesian coordinates, with distances and vector norms defined by the L2 norm.
3

---

store tuples (o us, og R
update evaluate
sample batch
Agent Sr
Qsi
Starting structure
Agent Ti
Replay
T
Agent O1
Q
Buffer
st
n/oi
Agent 02
Qaz
D/22
00r %2 o/3
Agent O3
Qai
U633
get rewards Rgg Ri
Q-networks
Energy Calculator
Figure I: Our overall MACS architecture. We use the SrTiO, example with five atoms: one atom of Sr. one atom of Tí, and three atoms of O. At each time step the current state sf is converted into individual observations ogg. of,- %6 o02. %0 that are passed to the agent (policy) networks. The agent networks output individual actions, which are then scaled and used as the atoms' displacements to update the structure. The energy calculator (CHGNet) provides the gradients for the updated structure to construct the next state g*+1 and compute the individual rewards. The policy training process follows the standard SAC 21 workflow with policy networks, Q-networks, and replay buffer.
duced for each agent. This means that each agent has a private local observation of the global state of the environment. POMG can be represented by tuple <A, S, O, U, T, R₂>, where A is set of N agents; S is the state space; O = 01 X ... x O is the joint observation space, where O; is the observation space of agent a;; U U₁
UN is the joint action space, where U; is the action space of a;; R₁ is the individual reward function that returns a scalar value to agent a; for a transition from state s $ state s' S after taking joint action E UJ; (s, 2) S x U S is the transition function, which determines the probability of transitioning to the next state s' E S given that agents take joint action U in state E S. The transition function deterministic in our problem setting.
By modelling the geometry optimization of crystal structures as a POMG, we treat each atom within periodic unit cell of structure All agents act independently and simultaneously to collectively discover a local minimum energy structure by maximizing their individual rewards. While each agent has different reward function, they are aligned toward a common objective as optimizing the position of one atom improves the relative positions of the surrounding atoms. We impose partial observability intentionally make the learning problem more tractable through the reasonable size of local observation spaces and to improve scalability to large numbers of atoms. The general scheme of MACS is presented in Fig.
individual agent, each with access only local observations.
Methodology
In this section, we formally introduce the proposed formulation of periodic crystal structure optimiza- tion POMG through defining the observation space, action space, and reward function. Then discuss our choice of the specific RL algorithm we use and its configuration.
Observations. Each atom in a crystal structure is surrounded by the neighboring atoms in the same unit cell, as well as their periodic images from the neighboring unit cells. Given this, we design the observation space so that each agent can observe its own features and the features of its k nearest neighbors. The & nearest neighbors refer to the atoms closest to the agent and enumerated order of increasing distance, either within its unit cell or from their periodic images in the neighboring unit cells. Fig. 2 shows dwo-dmmensional example of nearest neighbors of a specific atom in structure.

---

(b)
1.5
MACS
(e)
MACS
BFGS
2.0
BFGS
FIRE
FIRE
1.0
MDMin
1.5
MDMin
Density
BFGSLS
BFGSLS
FIRE-BFGSLS
FIRE+B
CG
1.0
CG
0.5
0.5
-7.50 0.0
-8.42
-8.00
-8.19
-8.00
-7.50
Energy
Energy
Figure 2: (a) Unit cells and nearest neighbors in two dimensions for a structure with three atoms. One atom is shown connected to its 12 nearest neighbors that belong to different unit cells; (b,c) The energy distribution of the local minima for the test sets of (b) SrTiO, with 80 atoms and (c) Ca2 0i with 96 atoms. The vertical line indicates the energy of the experimental structure.
To define the observation space, we start with defining the feature vector f. of agent aj at time step t: o, = concat((re, c log((g:))), d d
g 1
(1)
Here concal is the concatenation function; is the covalent radius of ag, which does not change during optimization and serves both to distinguish between different species and to carry important chemical information. The component ci is the action scaling factor that will be explained when defining the action space. g! is the gradient vector of agent a; at time step f, which is provided by CHGNet and scaled to avoid excessively large values (see Appendix A.3 log(Ig()) is used in the reward function and added to the feature vector to better capture the dependence of the reward on the observation. We call by felce-related the features that use the gradient vector that directly indicates the stability of an atom and its local environment. gf g!- is the change in the gradient vector from the previous time step & that reflects how successful the previous step was, and d is the displacement of a, at the previous time step ( - 1; these are history features designed to reflect the optimization dynamics. The force-related and history features prove their significance for the policy efficiency in Section 5.4
Now, we are ready to present the full observation of a; at time step / as follows: o = concol(f", o; ni ..... "".,,...,'' C ( [Iral.....
( r/....., (i....,,//.)) r/p))
Here, time step t, r......,ry ry are the relative (with respect to a;) positions of is nearest neighbors of 0g, and ......., , are the agents occupying those positions. Relative positions of the agent's s nearest neighbors and their feature vectors reflect the geometry and chemical properties of the agent's local environment. In this study, we use k = 12; therefore, the observation vector has length 204. The observation design is illustrated by an example Appendix [A.]]
Actions. A straightforward design for the action space would be to define actions as the agent's displacements. To ensure efficient learning, we propose using a scaling factor oi that depends on the gradient vector norm (gnorm) of a; at time step i to guide the order of magnitude of the action:
df = min( gg) Cmax).
Here Cmaa is tunable hyperparameter (in this study, Crazz 0.4) avoid excessively large steps. Given an action uf, E [=1,1]3 at time step f, the displacement of a; is as follows:
d = c.u,
The displacement can move an agent across the boundaries of the unit cell, in which case its position within the unit cell changes drastically. is crucial that neither observation vectors nor action vectors use the positions of atoms within the unit cell, as this approach allows smooth crossing between unit cells and the handling of multiple neighbors corresponding to the same agent from different unit cells.
Rewards.
The reward function should reflect the objective of reducing the atomic forces in a structure sufficiently low values. Hence, we propose using the gnorms of a, the current timestep and the next timestep t + 1 to construct the individual reward for a; as follows:
R log(lg!)) log(lg;+11
(5)
5

---

The logarithmic term helps to avoid skewing the policy toward short-term gains in the early stages of optimization, given the significant difference gnorms' magnitudes between non-optimized and optimized structures. We define an episode as the complete optimization of a structure and use the discount factor y 0.995 to encourage the policy to achieve the low forces more quickly.
Independent SAC. To train agents collectively discover locally optimized structure, use independent Soft Actor-Critic (SAC), which extends SAC [2T] from the single-agent to the multi- agent setting by treating all other agents as part of the environment. Hence, the multi-agent problem is decomposed into collection of simultaneous single-agent problems that share the same environment, SAC is chosen for its sample efficiency and its use of entropy regularization, which helps prevent early convergence to suboptimal policies and promotes balance between exploration and exploitation.
The MACS workflow with independent SAC is presented in Fig. Given a structure, CHGNet is used to estimate the forces acting on the atoms. First, the gradient vectors are scaled, and then the local observation vectors are constructed and normalized. Although may seem redundant to scale gradient vectors before normalization in the observations, this helps to preserve the gradient directions better and makes the training stable. The observation vectors then passed on the policy network. We use a standard SAC architecture proposed in [21] and implemented in RLlib [29]. and utilize the policy network and the twin Q-networks shared between all agents for efficient training. The policy and Q-networks are two-layered MLPs with ReLU activation functions. The policy network outputs three pairs (mean, std) for the action vector, which are passed through the tanh squashing to match the action space limits. The tuples <o',, u o R", > are stored in replay buffer with capacity of 10 million. The hyperparameter tuning details are provided in Appendix A
5 Experiments
In this section, we benchmark MACS against a set of methods commonly used for geometry optimiza- tion. We train MACS across diverse set of chemical systems and compare its performance against baselines using various evaluation metrics that assess the efficiency and reliability of the approach. We demonstrate the scalability and zero-shot transferability of our approach by applying the trained policy to optimize structures of unseen (larger) sizes within unseen compositions. Furthermore, we conduct ablation studies to investigate the influence of our observation space, action space, and reward function designs on the performance of MACS. Finally, we compare the results of optimization by MACS and the baselines through analyzing the energy distribution for the optimized structures.
5.1 Training and Testing Dataset Generation
We train MACS on set of six diverse chemical compositions Y2O3 [57] C12 M. SrTiO, D0]
Cag O [. CayAl,,ii001 200, and F1FeeFF1 [22]. These compositions vary in the number of elements (2-4) and in the number of atoms (5-80) required to describe their experimental structures. We generate training and testing structures using the Ab Initio Random Structure Searching pack- age (AIRSS) (30 During training, the initial pseudo random structures are generated the fly with the condition of belonging to one of the training compositions with equal probability and having 40 atoms with reasonable volume (see Appendix [or4 more details). For every composition on which the policy is trained, we generate three test sets of 300 structures each, with the structures containing K, 1.5K, and 2K atoms, where K is the size of the structures used during training.
To demonstrate the transferability of MACS, we generate test sets for three new compositions that do not participate in the training process Specifically, we select a composition from the training list, Ti,O,, and Sr, Sr.TioO10 For each of create two test sets: one with structures approximately the same size as
SrTiO,, and three from the same set of elements: Sry Sr,,, , Sry these new compositions, training structures and the other with structures twice the size.
5.2 Baselines and Evaluation Metrics
We benchmark MACS against six baselines: BFGS is quasi-Newton method that approximates the Hessian matrix based on gradient information; BFGSLS is variation of BFGS with line search (52] FIRE is a first-order method which is based on the molecular dynamics approach with additional velocity adjustments and adaptive time steps; FIRE+BFGSLS is hybrid approach where up to

---

250 steps of FIRE are followed by up to 750 steps of BFGSLS to fine-tune the structure [13 I MDMin is modification of the velocity- Verlet method with all masses of atoms equal to 1; CG stands for the conjugate gradient baseline, specifically, the Polak-Ribiere algorithm. All baselines are implemented in either the Atomic Simulation Environment package (ASE) [3] or SeiPy [48]. We allow the optimization of a structure to last up to 1000 steps or until the forces of all atoms are below the defined threshold.
A natural way to compare geometry optimization methods is by the time and number of steps required to optimize an initial configuration, hence our first two evaluation metrics are the mean number of steps among successful optimizations (N meaa) and the mean optimization time (Tmem), while the contribution of the training time is discussed separately in Appendix [B.2] We observe that for BFGSLS and CG, the number of steps is not equal to the number of energy calculations, as each step in both algorithms can involve more than one energy calculation. As energy calculations contribute significantly to the optimization time, we also use the mean number of energy calculations among the successful optimizations (Cmean) as an evaluation metric. Finally, we take into account the failure rate (P f) of each method to estimate their reliability.
Table 1: Performance comparison of MACS and the baselines on all test sets, covering T mean- Cmean- NN Nuvans and P F. The metries N Meaan and Cmean are presented either as a single value (if they are identical) or separated by a "," symbol (if they differ). The lowest values of T mean+ Cmeam and P are shown bold. The standard errors are provided in Appendix B.6
Toee (sce)
Nuxin: Caean
Compositonn
MACS
BFGS
FIRE
BFGSLS
FIRE+ BEGILL
MACS
BIGS
HRRE
MDMin
BEGSLS
FIRE+ BEGSLS
CG
atoms
40
18
48
42
74
28
38
64
121
313
262
442
137: 185
252; 267
122: :23:543
Y20g
60
32
85
92.
155
70
74
118
147
340
338
553
178: 781
174 357
145 (442
137
130
97
112
184
169
395
3933
206; 307
171
44
62
47
54
47
150
293
257
543
147: 177
232 :
158:716
S1e
112
79
355
307
176:201
280:291
:98:
74
175
120
315
110
117
275
230
414
391
745
213: 239
352; 365
283: 1269
40
94
109
248
65
102
134
143
255
314
675
133: 190
276:
141 557
SrT00g
60
90
163
202
461
138
200
250
179
316
379
719
169:255
321: 806
168 :81
30
142
242
366
672
214
332
452
206
329
466
765
199:317
355: 433
205 8337
48
59
135
119
264
68
120
156
146
270
324
623
136: 185
:31:317
151 :618
72
239
252
408
168: 249
756
96
163
276
412
705
195
324
499
205
353
467
762
193: 267
369:
213 213:876
124
98
132
111
274
246
135: 178
2:46
134: :62
KiFe
69
51
146
214
276
118
181
248
128
320
293.
506
163. 2555
209 5544
160:720
92
96
236
377
485
154
271
371
143
359
326
642
176: 236
3533 3916
181 :815
40
117
127
226
585
167
190
307
209
189
307
700
141: 264
269;
147:353
60
237
266
389
1068
276
422
572
264
230
382
755
165 296
316:39]
177:669
627
400
627
337
246
461
350
214:814
Compositions unseen during tralning
StuThO,
145
139
335
371
162:
319:353
:76
112
358
665
739
414
440
559
245
420
554
830
242: 397
427:
245 1003
SrgTig07
48
123
151
315
130
288
345
676
153: 210
209:
167 (682
96
159
369
497
800
366
375
477
227
382
501
817
212, 343
385: 449
223 9999
Stellighoo
34
.30
77
87
174
48
76
106
126
251
282
547
124: 173
256. 275
157 :377
68
113
202
238
498
149
205
307
186
310
408
729
179: 204
339: 385
182 /73:743
AVERAGE
66
175
239
207
315
366
151:253
317;361
179:7447
P (8)
0.36
9.64
46.19
0.82
18.22
The average takes average metric values for tests, mean the columm ahowe.
The average py across all compositions Py per test set is provided in Tabla]
5.3 MACS Policy Evaluation
We train MACS for 80,000 steps in total. The analysis of the variability of the policy trained starting from the different random seeds is provided in Appendix [..3 and confirms its consistent performance. After training, optimize the structures in the test sets using MACS and the baselines on the same hardware, allowing exactly one CPU per optimization (see Appendix B.] for more details).
Tab. shows the optimization results of MACS and the baselines across all test sets, covering the four evaluation metries (T meams N meaan Cmean- and PF) mentioned above. We observe that MACS is substantially faster and requires fewer energy caleulations than all baselines in nearly all test sets,
7

---

with only few exceptions. Specifically, on average, T meean and Cuean of MACS are 34% and 28% less than those of the best baseline, BFGSLS, respectively. We can also see that MACS has the lowest failure rate (Prr rr = 0. 0.36%) and performs comparably to BFGSLS on this metric. In terms of N mmaun BFGSLS performs slightly better than MACS, requiring 5% fewer steps on average. MACS performs comparably to CG and outperforms all other baselines. However, CG has a much higher failure rate (PF = 18.22%) than MACS. Both BFGSLS and CG involve multiple energy calculations per step, resulting in more total energy calculations and longer optimization time compared to MACS.
MACS consistently outperforms all baselines in T meean and Cmeun across all compositions and struc- ture sizes, except for CayAlzSig01a where MACS ranks first or second after BFGS (see also Appendix(B.8p This demonstrates the scalability of our method, as it maintains competitive perfor- mance as the structure size increases, Moreover, MACS exhibits excellent zero-shot transferability, as it outperforms all baselines in T mean and Cmean all sets of structures of compositions on which it was not trained.
Figs.2p [2p and [ show the energy distributions for local minima obtained by different methods. We can see that, when optimizing the same set of structures, MACS and the baselines sample from the same distribution of local minima. Figs. 3 and 31 show the energy evolution averaged over all successful optimizations for the test sets of the SrTiO, and AagAlzSig00 structures containing 80 atoms. It demonstrates that MACS decreases energy faster than the baselines or performs comparably to the best ones among them.
The analysis of the optimized structures based on interatomic distances is presented in Appendix B.7
5.4 Ablation Studies
We compare the MACS design proposed in Section 1 (referred to MACS) with its modifications.
Observations. We perform ablation experiments investigate the influence of the feature represen- tation in the observation space on our method's 's performance. Specifically, we compare the MACS atom feature vector provided in Eq. [ with the feature vectors in Eqs. to (referred to as feat.6, feat.7, feat.8, or f)a which use reduced feature representations that exclude some force-related or history features. For all setups, we evaluate the mean episodic reward and the mean episode length achieved by them across all compositions during training. As shown in Figs. B and P. MACS and feat.9 achieve the best performance, while feat.6 shows the worst performance. This demonstrates the importance of including force-related features in the observation space. We then evaluate the policies trained with different feature designs on optimizing the SrTiO, composition from the test sets. Tab. 2 shows that MACS performs sub- stantially better than feat.7 and feat.8 in T mean and N meaa- The design feat.9 shows marginally lower performance in Nuccan and better performance in T meman compared to MACS. In Appendix [.9 we further evalu- ate feat.9 on all remaining test sets and see that it takes on average 7.77% more opti- mization steps than MACS. These results show that both force-related and history features are crucial to the competitive performance of MACS.
a. = concat((rs, df
(6)
ft concat((ri, ci, log( (g(.)).)))))))
(7)
r concat((ri, ci log( gg [)] g/. gf gi
4-1
(8)
re concat([ri, ci, log( [)], d
Rewards. To investigate the influence of the reward function on the performance of our method, we explore two additional reward designs. For the first reward design (rew.10) we add a fixed penalty (tuned to -0.05) at each step to the reward used in Eq. [ to explore whether this encourages faster optimization:
R = log(Ig]) - log( + + penalty.
(10)
The second reward design (rew. 1 explores the effects of partial reward sharing by adding the average reward across all agents to the original individual reward used in Eq. 5
1 N
R log((g!)) - log(lg; (e(lgt++) +
(log(Ig]]) log(1s)* (1))
(II)
8

---

(a)
MACS
(b)
(c) 1.00
MAGS
BRGS
rew:10
MDMB
FIRE
200
0.75
BFGSLS
Reward
Reward
BRE+BFGGLS
0.50
CG
feat.6
feat.7
feat.8
0.25
act.12
(d)
MACS
(c) 1000
MACS
(f) 1000
MACS
DFGS
fesl..6
1ew.10
M BFGSLS
FIRE
Episode length
750
fealt
Epicods length
750
CG
FIRE CSLS
feal.99
act.11
500
250
250
100
200
300
40k
6ck
Step
Training steps
Training steps
Figure 3: (a, d) Energy evolution averaged over all successfully optimized structures of 80 atoms within compositions SrTiO3 (a) and CagAlySigO (d); (b,c.e.f) The ablation studies: the discounted episodic reward (b, e) and the mean episode length (c, f) across all compositions during training for MACS and its modifications.
Table 2: Performance comparison of MACS with varying feature representations (feat.7-9) in the observation space and varying reward functions (rew. 10, The best numbers are shown in bold. The design f is excluded due to its inability to successfully optimize structures.
Noess
Composition N
MACS
fost.7
feat.8
feat.9
MACS
fest.7
Scat.8
fear.
40
57
146
79
46
50
54
143
416
256
145
144
169
5/100g
60
90
147
236
AVERAGE
96
Pe (9)
0.22
show the mean episodic reward and mean episode length achieved by our method with different reward functions. Note that the episodic reward is normalized to make performance comparable between different reward functions. We can see that rew. and rew. perform similarly to MACS. We then evaluate the policies trained with the three reward functions optimizing the SrTiO; composition from the test sets. Tab, 2 shows that, MACS achieves superior performance than 10 and rew.1l in T meaan and N mean on the test set with the largest structures (80 atoms).
Fig. 3 and 3
Actions. We consider straightforward action design (act. 12) in which the action vector is used directly as atom's 's displacement (without the scaling factor ej as defined in Eq. 3h:
(12)
Fig. 3p shows that both action designs converge similar episodie rewards, but act. 12 is significantly less sample-efficien and stable. Moreover, Fig. 3 shows that act. 12 converges to higher average episode length, resulting in more optimization steps. This demonstrates that the scaling factor introduced in our action design is crucial to the competitive performance of MACS, enabling more sample-efficient and stable learning.
Conclusion
In this work, we present MACS, new MARL method for periodic crystal structure optimization. MACS introduces a novel model of geometry optimization as multi-agent coordination problem, an unexplored direction that poses unique challenges balancing expressive yet compact representations of chemical and geometric information, modelling complex atomic interactions, and enabling efficient, scalable policy learning. We conduct extensive experiments comparing MACS with various state- of-the-art methods and demonstrate that it learns policy capable of efficiently optimizing crystal

---

structures across diverse set of test cases, significantly outperforming all baselines. We show that on average across all test sets, MACS optimizes the structures 34% faster and with 28% fewer energy calculations than the strongest baseline, BFGSLS, preserving the lowest failure rate.
MACS demonstrates scalability and zero-shot transferability, as it is superior the baselines in the optimization of the structures of larger sizes and compositions that it does not encounter during training. In conclusion, MACS has the potential to evolve into a universal geometric optimizer for periodic crystal structures.
Limitations and future work. In this study, the only feature that differentiates atomic species is the covalent radius. Various atomic features and existing descriptor implementations [38] 47 will be considered in the future. The analysis of the observation space in Section (.44 confirmed that the history plays a crucial role in the efficiency of the method. Hence, the integration of recurrent neural networks into the method could improve MACS, and will investigate this matter further. Another promising direction for future work is learning to optimize the unit cell of the structure. We plan to treat the unit cell vectors as separate agents, optimizing them manner similar the way atoms optimize their positions. Finally, alternative approaches to estimating the energy of structures can be used in place of CHGNet. Two contrasting concepts can be considered: one involves using noisy energy and force evaluations to mitigate inaccuracies of machine-learning interatomic potential method, while the other relies using DFT, a more accurate, but also a more computationally expensive method. As CHGNet is trained on DFT, the Seit--wit--HGNee ceneetrrn--iiiieee with -FT workflow presents a promising concept for future study.
7
Acknowledgements
The authors would like to thank Professor Igor Potapov who gave valuable feedback that improved the quality of work.
The authors acknowledge funding from the Leverhulme Trust via the Leverhulme Research Centre for Functional Materials Design (RC-2015-036) and UK Engineering and Physical Sciences Research Council (EPSRC) through grant number EP/VO26887. This project has also received funding from the AI for Chemistry: Alchemy Hub (EPSRC grant EP/Y028775/1 and EP/Y028759/1), V. V. Gusev is supported by the Leverhulme Research Fellowship.
For the purpose of open access, the authors have applied Creative Commons Attribution (CC BY) license any Author Accepted Manuscript version arising from this submission.
References
[1] Kabir Ahuja, William H Green, and Yi-Pei Li. Learning to optimize molecular geometries using reinforcement learning. Journal of Chemical Theory and Computation, 17(2):818 818-82, 2021.
[2] Vaibhav Bihani, Sahil Manchanda, Srikanth Sastry, Sayan Ranu, and NM Anoop Krishnan. Stridernet: A graph reinforcement learning approach to optimize atomic structures on rough energy landscapes. In International Conference on Machine Learning, pages 2431-2451 PMLR, 2023.
[3] Erik Bitzek, Pekka Koskinen, Franz Gähler, Michael Moseler, and Peter Gumbsch. Structural relaxation made simple. Physical review letters, 97: 97010,, 2006.
[4] Charles George Broyden. The convergence of a class of double-rank minimization algorithms 1. general considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 1970.
[5] Christian Burnham and Niall J English. Crystal structure prediction via basin-hopping global optimization employing tiny periodic simulation cells, with application to water-ice. Journal of Chemical Theory and Computation, 15(6)338893390 2019.
[6] Yue Cao, Tianlong Chen, Zhangyang Wang, and Yang Shen. Learning to optimize in swarms. Advances in neural information processing systems, 32, 2019.
[7] Yu-Cheng Chang and Yi-Pei Li. Integrating chemical information into reinforcement learning for enhanced molecular geometry optimization. Journal of Chemical Theory and Computation, 19(23):8598-8660 2023.
10

---

[8] Tianlong Chen, Xiaohan Chen, Wuyang Chen, Howard Heaton, Jialin Liu, Zhangyang Wang. and Wotao Yin. Learning to optimize: A primer and benchmark. Journal of Machine Learning Research, (91189))1-59) 2022.
[9] Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy Lillicrap, Matt Botvinick, and Nando Freitas. Learning learn without gradient descent by gradient descent. In International Conference on Machine Learning, pages 748-756. PMLR, 2017.
[10] Christopher M Collins, George R Darling, and Matthew J Rosseinsky. The flexible unit structure engine (fuse) for probe structure-baset composition prediction. Faraday Discussions, 211:117-131, 2018.
[11] Christopher M Collins, Hasan M Sayeed, George R Darling, John B Claridge, Taylor D Sparks, and Matthew Rosseinsky. Integration of generative machine learning with the heuristic crystal structure prediction code fuse. Faraday Discussions, 356:85-103, 2025.
[12] Bowen Deng, Peichen Zhong, Kyujung Jun, Janosh Riebesell, Kevin Han, Christopher Bartel, and Gerbrand Ceder. CHGNet as a pretrained universal neural network potential for charge- informed atomistic modelling. Nature Machine Intelligence, 5(9):1031-104) September 2023.
[13] Jonathan PK Doye, Mark Miller, and David Wales. The double-funnel energy landscape of 1999.
the 38-atom lennard-jones cluster. The Journal of Chemical Physics, 1
[14] Roger Fletcher. A new approach to variable metric algorithms. The Computer Journal, 13(3):117-322, 1970.
[15] Julian D Gale and Andrew I Rohl. The general utility lattice program (gulp). Molecular Simulation, (955):211-341, 2003.
[16] Donald Goldfarb. A family of variable-metric methods derived by variational means. Mathe- matics of computation, 24(109):23-26, 1970.
[17] Prashant Govindarajan, Santiago Miret, Jarrid Rector-Brooks, Mariano Phiclipp, Janarthanan Rajefndran, and Sarath Chandar. Learning conditional policies for crystal design using offline reinforcement learning. Digital Discovery, 3(4):769-785, 2024.
[18] Daniel Greenfeld, Meirav Galun, Ronen Basri, Irad Yavnch, and Ron Kimmel. Learning to optimize multigrid pde solvers. In International Conference on Machine Learning, pages 2415 2423. PMLR, 2019.
[19] Raju P Gupta. Lattice relaxation a metal surface. Physical Review B, (3(1))66265, 1981.
[20] Vladimir V Gusev, Dunean Adamson, Argyrios Deligkas, Dmytro Antypov, Christopher M Collins, Piotr Krysta, Igor Potapov, George R Darling, Matthew S Dyer, Paul Spirakis, and Matthew Rosseinsky. Optimality guarantees for crystal structure prediction. Nature, 619(7968))68 72, 2023.
[21] Tuomas Haarnjja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-critic: Off policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[22] Anne-Marie Hardy, Antoine Hardy, and Gérard Ferey. Structure cristalline du bronze pseudo- quadratique Fo.: transition pyrochlore-quadratqque pour les composés KMM' X6. Acta Crystallographica Section B, 29(8): Aug 1973.
[23] Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Chris- tensen, Marcin Dulak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric D Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Leon- hard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter, Zhenhua Zeng, and Karsten W Jacobsen. The atomic simulation environ- ment-a python library for working with atoms. Journal of Physics: Condensed Matter, 29(27)227300,, 2017.
11

---

[24] Christopher Karpovich, Elton Pan, and Elsa A Olivetti. Deep reinforcement learning for inverse inorganic materials design. npj Computational Materials, 10(1):287, 2024.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[26] JE Lennard and Jones. On the determination of molecular fields. from the variation of the viscosity of gas with temperature. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 106(738):4414622 1924.
[27] Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint
2016.
[28] Xiaobin Li, Kai Wu, Xiaoyu Zhang, and Handing Wang. B2opt: Learning to optimize black-box optimization with little budget. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 18502-18510, 2025.
[29] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. In International Conference on Machine Learning, pages 3053-3062. PMLR, 2018.
[30] Leandro Liborio, Simone Sturniolo, and Dominik Jochym. Computational prediction of muon stopping sites using ab initio random structure searching (airss). The Journal of Chemical Physics, 148(13):13411)) 2018
[31] Michael L. Littman. Markov games as a framework for multi-agent reinforcement learning. In William W Cohen and Haym Hirsh, editors, Machine Learning Proceedings 1994, pages 157-163. Morgan Kaufmann, San Francisco (CA), 1994.
[32] Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[33] Enrico Masoero, E Del Gado, RJ-M Pellenq, F-J Ulm, and Sidney Yip. Nanostructure and nanomechanics of cement: polydisperse colloidal packing. Physical review letters, 109(15):155503, 2012.
[34] Amil Merchant, Luke Metz, Samuel S Schoenholz, and Ekin D Cubuk. Learn2hop: Learned optimization on rough landscapes. In International Conference on Machine Learning, pages 7643-7653. PMLR, 2021.
[35] Alexis W Mills, Joshua J Goings, David Beck, Chao Yang, and Xiaosong Li. Exploring potential energy surfaces using reinforcement machine learning. Journal of Chemical Information and Modeling, 62(13):3169 31799 2022.
[36] Rohit Modee, Sarvesh Mehta, Siddhartha Laghuvarapu, and U Deva Priyakumar. Molopt: Autonomous molecular geometry optimization using multiagent reinforcement learning. The Journal of Physical Chemistry B, 127(45):10095--0003 2023.
[37] Klaus Müller and Leo D Brown. Location of saddle points and minimum energy paths by constrained simplex optimization procedure. Theoretica chimica acta, 53:75 93, 1979.
[38] Artem R Oganov, Andriy O Lyakhov, and Mario Valle. How evolutionary crystal structure prediction works and why. Accounts of chemical research, 44(3):227-277, 2011.
[39] H Bernhard Schlegel. Geometry optimization. Wiley Interdisciplinary Reviews: Computational Molecular Science, 1(5):790-809, 2011.
[40] David F Shanno. Conditioning of quasi-newton methods for function minimization. Mathemat- ics of Computation, 24(111):647-656 1970.
[41] Jonathan Richard Shewchuk et al. An introduction to the conjugate gradient method without the agonizing pain. Technical report, Pittsburgh, PA, USA, 1994.
12

---

[42] Bhuvanesh Sridharan, Animesh Sinha, Jai Bardhan, Rohit Modee, Masahiro Ehara, and U Deva Priyakumar. Deep reinforeement learning in chemistry: A review. Journal of Computational Chemistry, 45(22): 188111886811888 2024.
[43] Frank H Stillinger and Randall A VaViolette. Local order in quenched states of simple atomic substances. Physical Review B, 34(8):5136, 1986.
[44] Frank H Stillinger and Thomas A Weber. Computer simulation of local order in condensed phases of silicon. Physical review B, 31():5266,, 1985.
[45] Ke Tang and Xin Yao. Learn to optimize a brief overview. National Science Review, 18)):nwae 132, 2024.
[46] Artem Tsypin, Leonid Anatolievich Ugadiarov, Kuzma Khrabrov, Alexander Telepov, Egor Rumiantsev, Alexey Skrynnik, Aleksandr Panov, Dmitry P Vetrov, Elena Tutubalina, and Artur Kadurin. Gradual optimization learning for conformational energy minimization. In The Twelfth International Conference on Learning Representations.
[47] Andrij Vasylenko, Dmytro Antypov, Sven Schewe, Luke M Daniels, John B Claridge, Matthew S Dyer, and Matthew J Rosseinsky. Digital features of chemical elements extracted from local geometries in crystal structures. Digital Discovery, 2025.
[48] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
[49] Junjie Wang, Hao Gao, Yu Han, Chi Ding, Shuning Pan, Yong Wang, Qiuhan Jia, Hui-Tian Wang, Dingyu Xing, and Jian Sun. Magus: machine learning and graph theory assisted universal structure searcher. National Science Review, 10(7):nwad128 2023.
[50] Daniel Widdowson and Vitaliy Kurlin, Resolving the data ambiguity for periodic erystals. In Advances in Neural Information Processing Systems, 2022.
[51] Daniel Widdowson, Marco M Mosca, Angeles Pulido, Vitaliy Kurlin, and Andrew I Cooper. Average minimum distances of periodic point sets foundational invariants for mapping periodic crystals. MATCH Communications' in Mathematical and in Computer Chemistry, 873))-529-559 2022.
[52] Philip Wolfe. Convergence conditions for ascent methods. it: Some corrections. SIAM Review, 13(2):185-18)) 1971.
[53) Scott M Woodley and Richard Catlow. Crystal structure prediction from first principles. Nature materials, 7(12):937-946, 2008
[54] Scott M Woodley, Graeme Day, and R Catlow, Structure prediction of crystals, surfaces and nanoparticles. Philosophical Transactions of the Royal Society A, (7.7(22186)) 18)), 2020.
[55] Yuanhao Xiong and Cho-Jui Hsieh. Improved adversarial training via learned optimizer. In Computer VEsV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part VIII 16, pages 85-100. Springer. 2020.
[56] Shiyue Yang and Graeme M Day. Exploration and optimization in crystal structure prediction: Combining basin hopping with quasi-random sampling. Journal of Chemical Theory and Computation, 1783):1988-1999, 2021.
[57] Elena Zamaraeva, Christopher M Collins, Dmytro Antypov, Vladimir V Gusev, Rahul Savani, Matthew Dyer, George R Darling, Igor Potapov, Matthew J Rosseinsky, and Paul G Spirakis. Reinforcement learning in crystal structure prediction. Digital Discovery, 183118810, 2023.
[58] Quan Zhou, Peizhe Tang, Shenxiu Liu, Jinbo Pan, Qimin Yan, and Shou-Cheng Zhang. Learning atoms for materials discovery. Proceedings of the National Academy of Sciences, 115(28):E641: E6417, 2018
13

---

42
2
a
a
a
3
a
a
a
a
a
a
d
a
11
a
a
ag
a
a
a
3
a
3
Figure 4: Two-dimensional structure with three atoms, and the three nearest neighbors for one of the atoms d1-
A
The MACS design and hyperparameter tuning
A.1
The nearest neighbors example
We consider two-dimensional example of structure with a square unit cell and three atoms a1 a2 03, where 02 and 03 relate to the same chemical element. We assume that the structure is undergoing optimization, and Fig. shows the atomic arrangement time step t. Here, for agent 01 the first nearest neighbor is a periodic image of atom a2 a neighboring unit cell, hence n 02; the second nearest neighbor is periodic image of a3 in another neighboring unit cell, and the third nearest neighbor atom dg itself, hence n nfa ag. Furthermore, rí rig, and ría denote the positions of the three nearest neighbors of @1, relative d1, L.e. vectors in Euclidean space from aj to each of its three nearest neighbors. Therefore, the observation vector for agent ai at time step looks follows:
o concat(fa, faz. fag. Sag. [|r (riah [rial], ri2. r(a).
(13)
During the optimization process, the list of k nearest neighbors is updated at each time step: existing nearest neighbors may update their relative positions and ordering the list, while some may leave the list and are replaced by new ones.
The amd package [50] 5T is used for the fast construction of the ordered list of the & nearest neighbors for all atoms on each step.
14

---

== 12
1000
k= 12
k=
k 15
Episode length
750
Reward
100
250
0.
20k
40k
60k
80k
40k
60k
80k
Training steps
Training steps
(a) The number of nearest neighbors (k)
(b) The number of nearest neighbors (k)
Figure 5: Mean episodic reward and mean episode length achieved by MACS with different numbers
of nearest neighbors considered in agent's local observation.
A.2 The number of nearest neighbors k:
We explore the number of nearest neighbors that should be considered in the local observation of an agent. We choose 10, 12, and 15 nearest neighbors and train three policies using MACS that differ only in this hyperparameter. Fig. shows that the three policies learned by MACS achieve comparable performance in both mean episodic reward and mean episode length during training. We then optimize the structures of the these policies. Tab. shows that the policy with k 15 has the highest failure rate and the longest optimization. The policies with k 10 and k 12 are comparable, with the latter being marginally better and therefore chosen for this study.
sets of compositions Y2O3, SrTiO3, and Siz 12 with
Table 3: Performance comparison of MACS with different numbers of nearest neighbors considered in agent's local observation. The best number is in bold, the standard errors in brackets.
Tman
mose
Composition
N
== 12
17(0)
18(3)
26(3)
123(2)
YyOs
60
34(1)
32(11
48(3)
150(3)
147(3)
171(5)
80
51(1)
48(1)
81(4)
179(4)
16933
201(6)
40
48(1)
57(2)
51(2)
148(3)
143(3)
164(4)
142113)
138(3)
215(5)
232(5)
C1gAlpSig0
60
231(7)
237(1))
245(3)
274(7)
264(7)
307(B)
80
380(11)
343116)
436(13)
333(8)
317(8)
367(9)
AVERAGE
60
122
120
140
201
196
224
Pyr (9)
0.56
0.788
3.52
15

---

9 max m
9 max
9 max 20
Gradient norm
20k
40k
60k
80k
Training steps
Figure 6: The gradient norm during training for MACS with different Jman.
300
5
1000
9max= + to 9max=20
Episode length
750-
200
Reward
9max
500
9mar
9max 20
100
250
0-
20k
40k
60k
80k
40k
60k Training steps
80k
Training steps
(a) gmaz
(b) gmax
Figure 7: Performance comparison of MACS with different values of Imaa=
A.3 Gradient vector scaling
Let go be the gradient vector provided by CHGNet or any other energy/gradients calculator for agent a, at time step The corresponding scaled gradient vector used for obsevatoons/eetinns/ewaass in our study is calculated as follows:
go go
|0 < 9mans
G
(14)
X
Amaz |
otherwise.
Here Jmax a tunable parameter. Our experiments showed that, while gradient vectors generally have components in the range [-50, 50] at the beginning of optimization, occasionally there can be vectors with components up to 500. Such gradient vectors significantly unbalance training and eventually lead to gradient explosion (see Fig. 6). In practice, there is no difference between large and extremely large gradient vectors, as they all indicate very undesirable atomic environment. Scaling the gradient vector to reasonable component values that preserve the direction helps mitigate this problem. Fig. shows the mean episodic reward and the mean episode length achieved by MACS during training with different values of Imaz. MACS with Jmar 20 converges the highest reward, which expected due to the higher gnorms at the beginning of the optimization, and hence, higher rewards. However, MACS with Jmaz 5 converges the lowest mean episode length while achieving high episodic reward, therefore, in this study, we use Jmar
16

---

Cmax =0 0.4
1000
mmax= 0.4
Ciax = 0.8
Cmax 0.8
200
Episode length
750
Reward
500
100
250
20k
40k
60k
80k
40k
60k Training steps
80k
Training steps
(a) Cmar
(b) Cmaa
Figure 8: Performance comparison of MACS with different values Cman-
1000
Reamm 2.T.T Em-10
2r.=00 2.T.
200
Episode length
750
4ra.:0:1.T_E::10
Reward
1..-6.1,7.--111
500
E--10
4ra.-0.08.T
3rae-0.08:11 000.000.8..--222.. 14
250
0
20k
40k
60k
BDk
40k
60k
Bok
Training steps
Training steps
(a) mean episodic reward
(b) mean episode length
Figure 9: Performance comparison of MACS with different action space bounds and target entropy values.
A.4 Hyperparameter tuning for actions
The scaling factor of is applied for the action of agent a; at time step 1 to guide the magnitude of the atom's displacement. The upper boundary for of is defined by hyperparameter Cmez avoid excessively large displacements. Fig- compares two variants of Cmaz and shows that the smaller Cmaz allows the policy to converge much higher mean episodic reward and lower episode length, and thus Cmax = 0.4 was chosen for this study.
We also explore straightforward approach for designing the action space: the action vector is used directly as the atom's 's displacement (Eq. 12), without the scaling factor cf- We consider different bounds the action space to investigate its effect on policy learning, namely, we consider -maax (max] 13 for different values of amar- We notice that, in SAC, the outputs of the policy network are tanh squashed and then scaled fit the action space limits. It leads different policy outputs' magnitudes for the same action vector, depending on the action space limits. To take this into account, also explore different values of the target entropy. We compare the mean episodic reward and the mean episode length achieved by MACS with different action space bounds and target entropy values. Fig. 9 shows that wider bounds on action space (amea 0.2) make training more stable, while narrower bounds on action space (amaz = 0.1) allow for achieving lower average episode length. We choose the best performing variant, namely, Omaz 0.1 with the target entropy T E= -1 for the ablation study.
17

---

Table 4: Hyperparameters used train MACS.
Hyparpanmetar
Trininn batch
Target entropy
-x
Truncate episodees
TRUE
Torget network update froquency
samples learning starts
Tau
0.001
Initial alpha
q
Actor learning rale
Criticc learning mec
0.0003
Eniropy leurning rate
0,0001
Replay buffeer capacity
10000000
Use priontiied replay buffer
0.4
Observation compunent noemallization
Number of neurest neighburs
Max steps in spisode
1000
1000
RBC=10M
RBC=1M
200
Episode length
750
RBC-5M
Reward
500
100
RBC=10M
250
RBC=1M
RBC 5M
20k
40k
60k
80k
40k
60k
80k
Training steps
Training steps
Replay buffer capacity (RBC)
(b) Replay buffer capacity (RBC)
1000
batch 8192
batch s.=1024
200
batch 5.=4096
Episode length
750
Reward
500
batch
250
batch 1024
batch = 4096
0.
6ôk
80k
40k
60k
80k
Training steps
Training steps
(c) Training batch size (batch s.)
(d) Training batch size (batch s.)
Figure 10: Performance comparison of MACS with different values of the replay buffer capacity and training batch size. The blue lines always indicate the values used in the paper.
A.5
Other hyperparameters
The list of hyperparameters used to train MACS is shown in Tab. Figs. 10 and 11 show the mean episodic reward and the mean episode length achieved by MACS during training for different
variations of the hyperparameters. We can see that the training batch size (Figs. 10c and 10db and the entropy learning rate (Figs. TIeland TTd play a crucial role.
18

---

T 8
1000
T_
E==7
T T_
200
T==99
T_
Episode length
Reward
500
100
250
20k
60k
80k
40k
60k Training steps
Training steps
(a) Target entropy (T_B) E)
(b) Target entropy (T E)
1000
200
Episode length
750
Reward
Ir=1e-4 E Ir=2e-4
100
E Ir=Se.5
Ir=1e-4 2r-4
250
Ir=5e-5
0.
20k
40k
60k
80k
40k
60k Training steps
80k
Training steps
(c) Entropy learning rate (E Ir)
(d) Entropy learning rate (E Ir)
1000
a/c Ir= 3e-4/3e-4
a/c 1/3e-4
200
a/c 1r-1e-/1e-
Episode length
750
Reward
500
100
a/c 4/3e--
250
a/c 1r=1e-4/3e-4
a/c Ir- 1e-4/e-4
20k
40k
60k
80k
40k
60k
80k
Training steps
Training steps
actor and critic Ir (a/c Ir)
(f) actor and critic Ir (a/c Ir)
1000
no twin
no twin
200
Episode length
800
Reward
100
400
200
60k
80k
40k
60k
80k
Training steps
Training steps
(g) twin Q-networks usage
(h) twin Q-networks usage
Figure 11: Performance comparison MACS with different values of the target entropy. entropy/ac- tor/critic learning rates, and twin Q-networks flag. The blue lines always indicate the values used in this study.
19

---

B Additional experimental details and results
B.1 Hardware usage for the training and testing purposes
We train MACS for ~80000 training steps in total using 40 concurrently running environments on Linux cluster node equipped with two 20-core Intel(R) Xeon(R) Gold 6138 CPUs (2.00 GHz) and 384 GB of memory. All baselines are not data-driven or trained; they are based on explicit, deterministic logic.
Optimization of the test sets using MACS and the baselines was performed on the same hardware used to train MACS. However, only a single CPU core was used for the optimization of the test sets, while the remaining 39 cores remained idle. This setup was chosen to ensure a fair comparison by preventing MACS from gaining advantage through parallelization, as the baselines parallelized.
B.2 Training time contribution to the experiments
The training time of MACS is 343,800 seconds and was not included in its evaluation. We evaluate the MACS policy based only on optimization time because including the combined training and testing time would make the evaluation highly sensitive to the size and number of test sets. In fact, the experiments could be extended by increasing the sizes/numbers the test sets until the relative contribution of training time becomes negligible, leading to the reported results.
Moreover, this setup reflects practical usage, as screening hundreds of thousands of structures is a typical use case in computational chemistry. Furthermore, the scalability and zero-shot transferability of MACS allow it to be trained on one set of compositions and then used to efficiently optimize larger structures from other compositions, without additional training, i.e., with zero training time in those cases.
It is ambiguous how to estimate the combined training and testing time, these phases were conducted under different conditions. Although the same hardware was used, the difference in the number of active cores (see Appendix B.T) raises the question of whether to report runtime using wall-clock time or CPU time, and which approach is fairer.
For reference, comparing the combined training and optimization time of MACS with the optimization time of the fastest baseline (BFGSLS), MACS is 3% faster in terms of wall-clock time but 126 times slower in terms of CPU time. Given the fact that training time can be omitted when the method is used in practice, we compare the methods only by the optimization time, as shown in Tab. !
B.3 Baselines
The baselines are accessed through the CHGNet package, which in turn interfaces with the opti- mization methods provided by the ASE package. We the BFGS, BFGSLS, FIRE, and MDMin implemented in ASE, while CG is implemented in SciPy and accessed through the CHGNet + ASE chain. The hyperparameters of the baselines are well-tuned and commonly used without modification; we adopt them as-is.
B.4 Random structures generation
We generate training and testing structures in the following way. Given composition, number of atoms, and a parameter v, we use the AIRSS package to generate a pseudo-random structure. AIRSS creates unit cell with random volume in the range ju 5%, + 5%) and places the atoms within this unit cell that the minimum distance between any atoms is A
During training, at the beginning of each episode, we randomly select a composition from the list of training compositions. We use the volume of the experimental structure for given composition as the parameter v in both training and testing to reflect the physics of the real material. Another parameter is the number of atoms in structure, which is selected to be closest to 40 atoms, given the composition.
20

---

Table The averaged discounted reward after 80,000 training steps.
Policy
Converged discounted episodic reward
Seed 1 (used in the paper)
236.0
Seed 2
237.4 4
Seed 3
235.6
Seed 4
234.8
Seed 5
235.2
Mean (std)
235.8 (0.9)
Seed 1
Seed 2
200
Seed 3
Seed 4
Seed 5
Reward
100
0
20k
40k
60k
Training steps
Figure 12: The discounted reward evolution for training MACS starting from five random seeds.
B.5 Variability of the MACS Policy
To identify the variability the policy trained by MACS, we independently trained five policies Seed 1 -eed each starting from different random seed while using the same hyperparameters and the time for training. Then we compared the converged discounted episodic rewards Tab, and their evolutions Fig. 12 The table shows that the policy trained by MACS consistently produces similar results across different random seeds.
B.6 Additional experimental data
Tabs. 8 extend the data presented in Tab. 1 Specifically, Tab, and Tab. 7abb7.cne compare MACS and all baselines in terms of T Tmean and Nmean respectively, including standard errors across all test sets, while Tab. Tab. reports PF for all test sets and methods. Figs. 13 and 14 show the energy distribution for the local minima obtained by the different methods for all test sets. Figs. [5 and show the energy evolution averaged by all successful optimizations of all test sets by the different methods.
B.7 The analysis of structures optimized by MACS
We calculated the atom-atom distances up to 6.00À within all 7,200 structures which were optimized using MACS. For each composition, we tabulated the distances, rounded to 2 decimal places as histograms, all of the histograms for individual compositions were then summed to give total atom-atom distance histogram for each composition. For each composition, we have then tabulated the shortest distance observed, and the distance of the first major peak in the histogram, both provided in Angstroms. We provide these results in Tab, 9
The shortest interatomic distances observed are plausible distances, the major observed peaks for each composition correspond to expected values when considering each chemistry. For example, the first peak 1.87 in SrTiO,, is close to that of the Ti-O distances in the experimentally observed structure of 1.95A. These observations in addition the distribution of energies shown in Fig. 13 lead us to the conclusion that the final structures produced by MACS do not contain unphysically short interatomic distances typical for the structures produced due the failure of the energy calculator.
21

---

Table 6: Comparison of MACS and the baselines by Tmean on all test sets, Standard errors are in brackets.
T Tieas (Se))
Compostion
stoms
MACS
BFGS
FIRE
MDMin
RFGSI S
FIRE+ BFGSLS
CG
18(1)
2812)
42(2)
74121
38(2)
64(4)
Y20g
32(1)
8513)
92133
135(3)
7462)
11806)
80
48(1)
137(4)
130(4)
207(4)
97(7)
(12(4)
184.9)
29(1)
62(2)
47(1)
116(3)
54(12)
42(3
81:3)
S1
66
51(1)
112:3)
58(2)
205(4)
6001)
79(2)
147(3)
74(2)
175(4)
12002
315(4)
110(2)
(17(2)
275(7)
57(12)
94(3)
10933
248(5)
6565)
(020)
13477)
SrT0O
1636)
20215)
46116)
200(21)
142(15)
242(8)
30648)
672(7)
214(16)
332(17)
452(20)
12)
135(5)
119(3)
264(5)
68L3)
(20(4)
1567)
TagTig07
72
106(12)
19966)
23944
479(7)
184(18)
252(5)
S01(14)
96
163(12)
270(7)
412(14)
705(7)
195(6)
324(33)
199(11)
46
31(1)
8213)
124(5)
155(2)
68(5)
9816)
132(7)
146(4)
96(12)
23666)
577(12)
485(8)
154.9)
271(77)
S71(16)
12715)
226071
58365)
19055)
237(14)
266(8)
389
1068(7)
278(16)
422(13)
57231)
80
343(16)
3338)
627(18)
1279(6)
400(7)
62767)
958(47)
Compositions unseen during the training
Tig
56
65(2)
145(5)
174(5)
361(5)
105(10)
(39(2)
234(12)
L12
189(13)
358(10)
665(18)
759(6)
414663)
440(5)
559233
Sig 0y
48
54(1)
123(4)
15164)
315(5)
8806)
130(3)
191(9)
369(10)
497(10)
77(3)
87(3)
174(4)
10K66)
113(12)
202(6)
238(6)
498(7)
149(3))
205(4)
301(16)
B.8 Composition targeted training
We train MACS on the only composition ( in which MACS was outperformed by a baseline (BFGS). We train the policy for the same period of time training. Tab, [0 confirms that MACS trained specifically on CazAlzSiz structures is superior to BFGS and MACS trained across all compositions in the training set in all metrics. The composition CagAlzSigO has the largest number of species, and its observation space can be more diverse than those of other compositions in this study. We suggest that longer training increasing the proportions of complex compositions during training can help MACS optimize them better.
did for the cross-compoiition
B.9 Additional ablation study
Observations. MACS and feat.9 achieve similar mean episodic reward and mean episode length during training, as well as similar performance in optimizing SrTIO, structures. We proceed with the optimization of all test sets using the feat.9 design and compare the results of MACS and feat.9 in We can see that the two policies are comparable in terms of T meaan- while MACS achieves an N mean that is 7.7% lower than that of feat.9, confirming the superiority of MACS.
Tab. II
'The optimization of test set with the Cuzs structures 88 atoms was interrupted by the cluster maintenance works and could not be finished due to the replacement of the hardware used for testing. Although optimization of the remaining structures of this test set could change the aggregated numbers, we rely on the comparison hetween all test sets to conclude that MACS consistently outperforms feat.9 as it consistently requires lower number of energy calculations than feat.9.
22

---

Table 7: Comparison of MACS and the baselines by N mean and Ceean on all test sets. The metrics Mmcan and Meean are presented either as a single value (if they are identical) or separated by a " symbol (if they differ). Standard errors are in the brackets.
Nuran : Cuean
Cumpusition
stomss
MACS
BFGS
FIRE
MDMin
BEGSLS
FIRE+ BFGSLS
OG
12113)
313(10)
262(4)
442(11)
139:2) 185(7)
252(3): 227(6)
(2212);
Y20g
14713)
360(10)
338(5)
553(12)
(2115)
325(1)
:12())812(12)
80
16913)
395(10)
393(4)
625(14)
20655)3071200
360(5) (00(14))
171(3)
150(3)
293/6)
257(4)
543(11)
147(3); 17766)
232(3) (44(3)
716613)))
S16
186(4)
35588)
307(5)
63301)
176(3) 20164)
280(3): 291(4)
198(4) (51
13015)
414(7)
745(12)
213(4) 239v49
(365(4)
183/6) 1269(26)
14313)
25588
$14.5)
625(12)
299(3)
572(1
SmT00
179(4)
316(10)
S7X6)
719(44)
(6(3) 255(14)
321(3) 40K(48)
16814) 681(1))
80
20015)
3298)
446(7)
765(17)
199(4) 317(26)
355(3) (43)
205(5) 857(21)
146(3)
270(9)
524.5)
623(13)
136/2); 185(10)
286(2) 317(11)
: 618(12)
Cay 0g
72
183(4)
31088
408(6)
707(15)
168(3) 249127)
335(3):3744)
186(3); 756(12)
205(4)
3539)
26709)
369(3) 443(23)
876(15)
111(2)
2748)
246(3)
501(10)
135(3) 178(7)
246(1) 263(6)
(34(3) (02(
KyRe5F15
128(3)
32077
293(4)
396(1)
239)53)
34N22)
(60(3) 720
92
143/2)
25966)
326(4)
642(10)
176(4) 236(13)
35313) 396(16)
18114): 815(20)
20915)
18955)
307(6)
700(3))
141(4): 264166)
:
L47(4) 553(15)
60
264(7)
230:5)
38266)
755(55)
165(3): 296 16)
:96())
17764) 669(15)
31708)
246(6)
461(7)
894(0)
189(3); 337v6)
350(3) (16)
214(6) 814(23)
Compositions unseen during the tralninn
Sry
56
172(4)
335(10)
371(6)
700(14) 850(16]
162(3)
319(3);
:7514):716115)
24515)
420(10)
554(8)
39735)
427(4) 500
245(4); 1000(19)
Sty T220y
153(4)
288(8)
345(5)
676(12)
2100100
3223))
16764) 682(17)
22715)
382(9)
501(7)
817(20)
212(4): (43(33))
383(3) 44997)
22315) 90X(19
126(3)
251(8)
282(5)
547(13)
256(3) (75(3))
557(17)
18615)
310(8)
408(5)
729(4))
179(4) 304(52)
339(3) (35(8)
(83(3) 743(14)
Table 8: Comparison of MACS and the baselines by P (%)
Compositicn
alems
MARL
FIRE
BEGSLS
BFGSLS
FIRE-
CG
40
0,33
5.67
7
17.33
0.33
1.33
19
Y20g
60
4.33
16
28.67
0.67
26.67
18:33
29.33
32.33
0.33
6.67
O12
0.33
2.67
2.67
11
88
0.33
56.67
3.33
2.33
3.33
0.33
15.67
S:D0g
60
0.33
2,39
46.33
0.67
0U67
21
1.33
26.67
1.33
22.33
10.33
CayTixU7
1.67
0.33
13
96
0.33
0.67
3.33
72.33
20
46
1.33
17.67
4.67
0.33
4.33
14
Kis
69
1.33
22.67
14
0
3
19.33
92
2:33
33.67
20.33
0.33
20
40
1.33
19
24.33
4.67
09.67
0.33
27
Compositions unseen during training
Sty110,
17.67
112
5.67
31
1.33
20.67
StyTi20
48
2.67
4
38.33
0.33
0.33
17.67
96
3.67
14
$1.33
20.33
SeaTi5000
34
0.33
4
18.67
0.67
14.67
2.33
8:33
55.67
17
23

---

Table 9: The shortest atom-aton distance observed and the distance of the first major peak in for the structures optimized by MACS. The total number counts for the distances observed in brackets.
Composition
Min distance
First peak
CagAlzSig01z
1.44 (4)
1.69 (5324)
CaaTizOr
1.38 (2)
1.85 (3924)
Cu2sS16
1.97 (2)
2.22 (4708)
K1FesFis
1.78 (2)
2.00 (5392)
Y2Os
1.45 (2)
2.22 (8218)
SrTiO,
1,40 (4)
1.87 (3898)
SryTiO,
1.41 (12)
1.84 (2850)
SraTizO,
1.41 (8)
(2506)
Sr,Ti,01u
1.41 (8)
1.86 (1950)
Table 10: Comparison of the MACS policy trained on the CagAl composition (MACS individual) with the MACS policy trained across all compositions in the training set (MACS) and BFGS.
T (see)
N
Composition
MACS
MACS individual
BEGS
MACS
MACS individual
BEGS
atoms
208
A1g
343
237
333
317
229
246
AVERAGE
232
164
242
264
196
222
Px (1)
1.78
0.33
0.89
Table 11: Comparison of the feat.9 design with MACS by T mean and Nmean- Standard errors are in brackets.
Timan (sec)
Nmvn
Compasition
N ateets
MACS
feat.
MACS
feat.)
40
18(1)
22(1)
121(3)
(4365)
Y20g
32(1)
39(3)
147(3)
168(4)
29(1)
2913)
150(3)
1128816
51(1)
54(2)
214(6)
74(2)
6713)
230(5)
338(30)
5Y(12)
4411)
143(3)
145(3)
SrD00
90112)
8112)
17914)
184(5)
19(
56(1)
146(3)
154(3)
Cayli,O7
9812)
163(12)
134(3)
205(4)
214(4)
31(1)
38(1)
111(2)
141())
KiFogFis
51(1)
64(2)
128(2
167(4)
96(12)
(12(2)
143(2)
183(1)
40
11764)
111(3)
20915)
18915)
CagAlgSig01a
343(16)
334(8)
317(8)
293(7)
Compositions unseen during training
SryTiO,
6311)
139(
174(4)
245(5)
247(5)
TryTig0y
54(1)
53(1)
15364)
96
15X(13)
150(4)
227(5)
229(5)
StoTiy00o
30(1)
3111)
126(3)
131(3)
183(12)
88(2)
186(5)
190(4)
AVERAGE
100
101
181
195

---

MACS BFGS
15
MAGS BFBS FIRE
1.5
BFGS FIRE MEMi
MAGS
FIRE MDMn BFGSLS
MDMin BFORLS
Denaiyy
Donsity
Deneity
1.0
FIRE+BFGSLS
FIRE-OFGSLS
CO
0.5
0.0
0.0
0.0
-4.0
Energy
Energy
Energy
(a) YzOg with atoms
(b) Y2O, with 60 atoms
(c) Y2O, with 80 atoms
MACS
MACS
MACS
1.5
BFGS
DFGS
BFGS
FIRE
FIRE
FIRE
MUMB
MOMB
1.0
MDWin
Density
1.0
BFGSLS
FBFOSS
BFGSLS
FIRE
Deneity
BFGSLS
FIRE-BFGSLS
0.5
0.5
0.0
0.0
3.50
Energy
Energy
Energy
(d) Cua with 44 atoms
CI2ySIa with 66 atoms
(f) CUasSIe with atoms
1.5
MACS
BFGS FIRE
FIRE
Denely
MDMn
BFGSUS
0.75
BFGSLS
MDMin
Density
MDMin
BFGSLS
FIRE-BPOLLS
FIRE+BFOSLS
FIRE-BFOSLS
CG
DG
0.26
0.00
0.00
Energy
Energy
Energy
SrTIOs with 40 atoms
(h) SrTiO, with 60 atoms
(i) SrTiO, with 80 atoms
2.0
MACS
Density
BFGSLS FIRE-BOOSLS CG
Densiyy
MDMin BFGSLS
Density
1.8
MDVin
BFGSLS
FIRE+BFOSLS
FIRE-BFGSLS
CG
CG
0.5
0.5
-8.19 -8.00
-7.50
-8.19
-3.00
-7.50
-8.19
-8.00
-7.50
Energy
Energy
Energy
(j) TizO, with 48 atoms
(k) Cas Tiz0, with 72 atoms
()) Cas Ti2O; with 86 atoms
BFGS
FIRE
ARE
FIRE
MDMin
MDMin
MDMin
Donaity
BFGSLS
Density
BFGSLS
Density
BFGSLS
FIRE- RRE-BFGSLS
FRE-BFFSLS
FIRE- BIRE-S
5.98
5.50
5.00
5.98
5.50
5.00
5.98
5.50
5.00
Energy
Energy
Energy
(m) Fis with 46 atoms
(n) KgFes Fis with 69 atoms
(o) K2Fcs Fis with 92 atoms
Figure 13: The energy distribution of the local minima obtained by all different methods. The vertical line indicates the energy of the experimental structure.
25

---

MACS BFGS
MACS 3FGS FIRE VDMI
MAGS
BFGS
FIRE
0.6
MDMi
Densiyy
Denalty
BFGSLS
0.4
0.2
0.0
0.0
0.0
Energy
Energy
Energy
(a) CasAl:Si;, with atoms
(b) CasAl:Si, with atoms
(c) CagAl22Sgg
80 atoms
1.00
FIRE
1.5
FIRE
MACS
BFGS
1.00
BPGS
MDMi
0.75
MOMI
Densly
0.75
BFGSLS
1.0
BFGSLS
Denaly
BPGELS
FIRE-B
BIRESS
CG
0.50
CG
0.26
0.25
0.00
0.0
0.00
Energy
Energy
Energy
(d) Srz TIO4 with 56 atoms
(e) Srz TiO, with 112 atoms
Sra Ti2O, with 48 atoms
MACS
MACS
MACS
BFGS
BFGS
BFGS
FIRE
FIRE
FIRE
MOMB
MOMi
1.0
MDVin
Densky.
1.0
BFGSLS
0.8
BFGSLS
Denslty
BFGSLS
FIRF
FIRF+BFOS
FIRF-AFOSI
0.4
0.5
0.0
0.0
-8.1-8.0
-7.5
Energy
Energy
Energy
(g) Sry TizO, with 96
(h) Sr, TigO with 34 atoms
(i) Sr, T1o with 68 atoms
Figure 14: The energy distribution of the local minima obtained by all different methods. The vertical line indicates the energy of the experimental structure.
26

---

MADS
DFGS
-7
FIRE
MDMB
T.
FIRE MDMB BFOSLS
MDWin
FIRE
Enargr
BFGSLS
Energy
Erergy
BFOSLS
RE+BFGSLS
FIRE+BPGSLS
FIRE-BBGSLS
OG
OG
-9
-9
-9
100
200
300
200
300
100
200
300
Step
Step
(a) Y2Og with 40 atoms
(b) YzO3 with 60 atoms
(c) Y20g with 80 atoms
MACS
MACS
MACS
BFGS
BFGB
BFGS
FIRE
FIRE
FIRE
-2
MDMin
2
MDMin
MDMin
Energy
BFGSLS
Energy
HRRE-B-GSLS
BFGSLS
Energy
GSLL-B--SSLS
BFGSLS
-4
0
100
200
Step
Step
Step
(d) C12 with 44 atoms
(e) CU2sSIG with 66 atoms
(I) C12gS16 with atoms
5
MAGS
-5
MAGS
-5
MAGS
BFGS
BFGS
BFGS
FIRE
Enargy
BFGSLS
FEE+BP BEELFSSSLS
Ererst
FIRE GFGS
BPGBLS
Erergt
FIRE-B/GGLS
BFOBLS
OG
CG
CC
-8
0
100
200
300
100
200
300
100
200
300
Step
Step
Step
(g) SrTiO, with 40 atoms
(h) SrTIO,; with 60 atoms
(i) SrTiOs with 80 atoms
MACS BFGS
-5
MACS
MDMin
MDMp BFGSLS
MDMin
Eneegy
BFGELS
REIBFGSLS
FIREBBFGLL
Evera
FIRELB
BFGSLS
CG
CO
CO
-8
-8
-8
100
200
300
100
200
330
100
230
300
Stop
Stop
Stop
(j) Cas Tz2O, with 48 atoms
(k) Cas TizO, with 72 atoms
()) Cag TizO, with 86 atoms
BFGS
BPGS
FIRE
FIRE
FIRE
MDMin
MDMi
MDMin
Enargr
BFGSLS
FIRE-BFGSLS
BFGSLS
Energy
BFGSLS
BIRE-S
-5
200
300
Step
Step
(m) KiFesFis with 46 atoms
(n) FıFesFis with 69 atoms
(0) KaFesFis with 92 atoms
Figure 15: Energy evolution averaged over all successfully optimized structures for all methods on the given test
27

---

MAGS
FIRE
FIRE
FIRE
MDMin
MDMp
MDMin
Energy
DFGSLS
FRE+BFOBLS
Energe
BFGSLS
FIRE+BFOSLS
Energy
BFGSLS
FIRE+B
-7
230
Stop
(a) CagAlgSin001 with 40 atoms
(b) CagAl2Si,00p with 60 atoms
(c) CagAlzSig with 80 atoms
MADS
MACS
BFGS
BFGS
BFGS
FIRE
FIRE
FIRE
MDMin
MDMp
MDWin
Energy
-6
BFGSLS
RE+BFGELS
Enerryy be
BFGSLS
FIRE+BFOSLS
BFGSLS
Step
Step
(d) Sr. TiO, with 56
(c) Sry Tio, with 112 atoms
Sry TigO, with 48
MACS
MACS
MACS
BFGS
BFGS
BFGS
FIRE
FIRE
FIRE
MDMp
MDMin
Enangl
BFGSLS
Eieery 8
BFGSLS
-8
100
300
100
200
300
Step
Step
Step
(g) Sra TizO7 with 96 atoms
(h) T13O10 with 34 atoms
(i) Sr. Ti3O1 with atoms
Figure 16: Energy evolution averaged over all successfully optimized structures for all methods on the given test set.
28

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the main abstract and introduction has accurately reflected the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of MACS in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
the paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The code used for this project provided the supplementary material. The code includes README that details the steps for reproducing our results.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
30

---

Answer: [Yes]
Justification: This paper has provided access to the code. Due to the size of the dataset, the testing sets are not provided. However, the instructions for the generation of comparable test set are given.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co cc/ caccc/puliccaee http://plllcgggsss/s/ppppso
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the paper specifies all the training and test model details and hyperparameters in Section 4 Section and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All methods were tested on the same sets of 300 initial structures for consis- tency. For each test set, every reported metric's value is based on the optimization of all 300 structures from this test. For all tables with means provided in the main text, we report the standard errors in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
31

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources used in this paper for training and testing are described in Appendix [I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
Justification: This research is conducted in the paper conform, in every respect, with the NeurlPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While our method can assist in new materials discovery, and new materials can be used in negative way to society, we do not address this issue in the paper as there is no direct connection.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
32

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks. The trained model is specific to the optimization of crystal structures only.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The usage of the assets documented including the version, the citation for the original paper, and the name of license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
33

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcoee cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: Our approach implemented a Python package and is assisted with documentation.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codd/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hatt:e///eerrpp..ccc///oeeerrrre for what should or should not be described.
35


Reviews:
[{"review_id": "GV03dcMwu7", "reviewer": "Reviewer_ytGL", "summary": "MACS (Multi-Agent Crystal-Structure optimization) frames periodic geometry relaxation as a partially observable Markov game. Each atom is an independent agent that observes itself and its 12 nearest neighbors, then outputs a 3-D displacement. Displacements are scaled by the atom’s current force norm; the reward is the log-drop in that norm. A single, shared policy is trained with independent Soft Actor-Critic and a replay buffer, using CHGNet for energy/force calls. Against six standard optimizers (BFGS, BFGS-LS, FIRE, FIRE+LS, CG, MDMin), MACS cuts mean optimization time (T_mean) by ~34 %, reduces the force-call budget (C_mean) by ~28 %, and posts the lowest failure rate (0.36 %) across 39 test sets, including zero-shot transfer to unseen compositions and larger cells.", "questions": "Questions: \n1. Did you tune BFGS-LS and CG for the experiments? \n2. Any preliminary results on treating lattice vectors as extra agents, as suggested in Limitations?", "limitations": "The paper notes that atomic descriptors are minimal and unit-cell vectors are fixed/ it could also discuss how reliance on CHGNet accuracy affects performance and how the policy behaves if force evaluations are noisy.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n- First MARL relaxer that treats every atom as an agent while respecting lattice periodicity.\n- Consistent wins on 39 test sets spanning seen and unseen chemistries and 2 × larger cells.\n- A single policy generalises to compositions and sizes never encountered during training.\n- Detailed ablations on features, reward, and action scaling show which choices matter (Fig. 3, Table 2).\n- Local observation graph, per-atom action scaling, and log-force reward are each justified by the ablations.\n- Code is promised. The appendix lists hyper-parameters, standard errors, and per-test PF, supporting reproducibility.\n\n**Weaknesses:**\n- minor clarity: repeated “otpimization/ Evalation” typos; Table 1 caption dense; Fig 3 colours hard to read.", "was_revised": true, "final_justification": "Overall Recommendation: 5 (Accept). The paper introduces a novel MARL formulation, per-atom agents under periodic boundary conditions, and demonstrates consistent improvements over strong classical optimizers across diverse test sets (including zero-shot and larger cells). The main limitations (minimal species encoding, fixed lattice vectors, reliance on CHGNet) are acknowledged. Presentation issues (a few typos, dense captions) are minor and easily fixed. Overall, the work is technically sound, original, and impactful.", "rebuttal": "**Summary Rebuttal**\n\nTo avoid repetition, we have formulated a shared response for the reviewers addressing the questions and concerns raised by more than one reviewer.\n\nWe believe that simplicity is a strength and have intentionally used the simplest MARL approach with independent learning. Instead of coordination via explicit communication, our method enables coordinated behavior through interactions among agents via interdependent rewards and overlapping local observations. This allows agents to adapt to each other’s behavior over time, resulting in implicit coordination. Another main reason is that independent methods tend to scale better to tasks involving a large number of agents and/or actions compared to centralized methods. This is crucial for our method’s scalability and applicability in computational chemistry, where the optimization of large structures is a major bottleneck in many computational methods.\n\nOur experiments demonstrate that, with a carefully designed setup of actions, observations, and rewards, independent learning can offer a significant improvement over state-of-the-art methods for crystal structure geometry optimization. We recognize the importance of convincing you that our baselines are strong enough for this claim to be valid, and we address this below.\n\nThe baselines we use, although classical, still form the majority of methods available in widely used packages for computational chemistry, such as ASE, VASP, GULP, and CHGNet. These are employed in thousands of computational research projects. In short, our baselines represent the standard choices used daily by practitioners working in computational chemistry. The lack of novel methods that consistently outperform these baselines in general settings motivated our research. We appreciate the references provided by Reviewer 6w1f; however, as we explain below in our rebuttal to 6w1f, none of these are appropriate for our task.\n\nFinally, we would like to emphasize the contribution of our paper in formulating a challenging, physically grounded optimization environment for a ubiquitous task in condensed matter science, and in proposing an efficient, scalable MARL method that opens new directions for further research.\n \n**Rebuttal to ytGL**\n\nDear Reviewer,\n\nThank you for taking the time to read our paper and provide valuable feedback. We will fix the typos and improve Table 1 and Figure 3 as suggested. The answers to your questions are below.\n\n> Question 1: Did you tune BFGS-LS and CG for the experiments? \n\n**Answer 1**. All baselines were used with the default hyperparameters defined in the ASE package, as they represent a typical use case.\n\n> Question 2: Any preliminary results on treating lattice vectors as extra agents, as suggested in Limitations?\n\n**Answer 2**. This is an interesting future work, as mentioned in our paper, and preliminary results are not yet available.\n\n> Limitations: The paper notes that atomic descriptors are minimal and unit-cell vectors are fixed/ it could also discuss how reliance on CHGNet accuracy affects performance and how the policy behaves if force evaluations are noisy.\n\n**Answer 3**. \nWe have conducted the additional analysis of the quality of the final structures produced by MACS, which is provided in the reply to reviewer 6w1f (Answer 4) and will be added to the Appendix. This analysis of the resulting structures reveals that they are physically acceptable.\nThe noisy force evaluations would be an interesting direction for future research, and we will mention it in the revised paper."}, {"review_id": "nNCD56fF3J", "reviewer": "Reviewer_6w1f", "summary": "This paper proposes MACS, a multi-agent reinforcement learning (MARL) framework for geometry optimization of periodic crystal structures. Each atom is modeled as an individual agent, observing local geometric and force-based features and adjusting its position to reduce atomic forces. The optimization is performed using independent SAC and guided by the CHGNet surrogate model for force estimation. The authors report substantial improvements over classical optimization baselines in terms of speed, energy evaluations, and failure rate, and demonstrate zero-shot transferability to larger or unseen compositions.", "questions": "See concerns:\n- I believe a short discussion would be helpful to better understand how strongly these issues might affect the proposed Markov game.\n- The confidence in MACS' performance would be substantially improved if stronger baselines were to be included.\n- Would it be possible to confirm the quality of the final crystal structures using DFT?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n---------\n- The formulation of geometry optimization as a MARL problem is intuitive, leveraging locality and structure in atomic interactions.\n- The pipeline is conceptually simple yet elegant.\n- Experiments are extensive, covering diverse compositions, varying sizes, and ablations on observation, action, and reward designs.\n\nConcerns\n---------\n- To me, the connection to learning to optimize is somewhat superficial: the RL algorithm is standard, the environment (CHGNet + atomic displacements) is fixed, and there is no higher-level learning over tasks.\n- The agents operate independently (using independent SAC). While effective, this ignores coordination issues known to affect MARL systems. I believe a short discussion would be helpful to better understand how strongly these issues might affect the proposed Markov game.\n- My main concern is that, although the results look strong, the baselines are limited to classical optimizers (e.g., BFGS, FIRE). There has been a number of works on geometric optimization, e.g., [1-3]. I believe the confidence in MACS' performance would be substantially improved if stronger baselines were to be included.\n- The energy estimations are based on CHGNet, which is itself a learned model. While fast, it introduces approximation errors. Through exploration, the RL agents might produce crystals that are o.o.d. of CHGNet's training data. It would be valuable to confirm the quality of the final crystal structures using DFT.\n\n\n[1] Zuo, Y., Qin, M., Chen, C., Ye, W., Li, X., Luo, J., & Ong, S. P. (2021). Accelerating materials discovery with Bayesian optimization and graph deep learning. Materials Today, 51, 126-135.\n\n[2] Li, C. N., Liang, H. P., Zhang, X., Lin, Z., & Wei, S. H. (2023). Graph deep learning accelerated efficient crystal structure search and feature extraction. npj Computational Materials, 9(1), 176.\n\n[3] Chen, S., & Zhang, S. (2022). A structural optimization algorithm with stochastic forces and stresses. Nature Computational Science, 2(11), 736-744.", "was_revised": true, "final_justification": "This is a copy of my last comment to the authors. To summarize:\n\n- my main concern was related to the (in my view) simplistic baselines. This discussion made me realize that the baseline seem appropriate, and thus properly support the benefit of MACS. It also showed my limited knowledge in the application field (as I come from an RL background).\n\n- my second concern focused on the appropriateness of evaluation. This was somewhat addressed by the authors (knowing that it is unreasonable to compute DFT calculations on all final structures during the rebuttal period).\n\n- my third concern was related to the independent nature of the agents in the MA system. Despite the fact that empirically, independent learners have been shown to perform well on SMAC, this does not mean that the MA target problem does not exist and does not affect the current application.\n\nIn light of all of this, I will raise my score from 4 to 5, as overall this work seems a solid advancement in a specific subfield of scientific discovery. I will also lower my confidence score from 3 to 2, as I feel I am missing part of the necessary relevant literature.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to read our paper and provide valuable feedback. In the following, we address your concerns and questions in detail.\n\n>Concern 1: To me, the connection to learning to optimize is somewhat superficial: the RL algorithm is standard, the environment (CHGNet + atomic displacements) is fixed, and there is no higher-level learning over tasks.\n\n**Answer 1**. Indeed, MACS is tailored specifically for geometry optimization and does not support optimization tasks from other areas of science. In the manuscript, we note that \"an entire field within materials design, known as crystal structure prediction (CSP), focuses on the computational prediction of stable crystal structures\" thus geometry optimization plays a fundamental role in computational materials design. Due to the highly diverse chemistry across different compositions and phase fields, we treat optimization within each composition or phase field as an individual task over which MACS learns to optimize.\n\n>Concern 2: The agents operate independently (using independent SAC). While effective, this ignores coordination issues known to affect MARL systems. I believe a short discussion would be helpful to better understand how strongly these issues might affect the proposed Markov game.\n\n**Answer 2**. Summary Rebuttal (at the beginning of the Rebuttal to ytGL) partly addresses this question. We would like to add that previous works [4, 5] have also shown that independent learning methods can perform just as well as or even better than state-of-the-art joint learning MARL approaches on multi-agent benchmark environments like the StarCraft Multi-Agent Challenge (SMAC), which consists of various challenging coordination tasks.\n\nFurther, the introduced action scaling coefficient $c_i$, defined by formula (3), restricts the order of magnitude of an agent's action based on its gradient norm. This makes transitions between states more predictable for agents, as each agent can rely on the gradients of neighboring agents in its local observation to estimate how far they **can** move from their current positions. Together with overlapping local observations and interdependent rewards, this enables efficient collective behavior without explicit coordination.\n\n[4] De Witt, C.S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P.H., Sun, M. and Whiteson, S.. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint arXiv:2011.09533, 2020. \n\n[5] Papoudakis, G., Christianos, F., Schäfer, L. and Albrecht, S.V.. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.\n\n>Concern 3: My main concern is that, although the results look strong, the baselines are limited to classical optimizers (e.g., BFGS, FIRE). There has been a number of works on geometric optimization, e.g., [1-3]. I believe the confidence in MACS' performance would be substantially improved if stronger baselines were to be included.\n\n> Question 2: The confidence in MACS' performance would be substantially improved if stronger baselines were to be included.\n\n**Answer 3**. We refer the reviewer to the Summary Rebuttal to answer the question. Also, in the papers [1-3], the structure optimization method provided is either designed for a different task or tailored to more restrictive conditions than those addressed in our paper. Specifically, in [1], the optimization method is applied under symmetry constraints to crystal structures **close to local energy minima**; in [2], the authors perform two-step structure optimization using two different energy calculators and **classical optimizers**; in [3], the authors consider the specific problem of structure optimization under **noise in gradient conditions** and introduce a two-staged optimization via the steepest-descent method with a fixed step size on each phase. The steepest descent is known to have slower convergence than other first-order methods, such as CG, one of the baselines in our study.\n\n> Concern 4: The energy estimations are based on CHGNet, which is itself a learned model. While fast, it introduces approximation errors. Through exploration, the RL agents might produce crystals that are o.o.d. of CHGNet's training data. It would be valuable to confirm the quality of the final crystal structures using DFT.\n\n**Answer 4**. The final energy distributions shown in Fig.12 of Appendix B.4 confirm the comparable final energy distribution among results of the optimization by MACS and the baselines. Performing density functional theory calculations on all of the final structures is impractical within the available time scale. However, we do agree with the reviewer, that assessing the quality of the resulting final crystal structures is an important task. For this we have performed an additional analysis of the final structures and suggest to add the following section to the Appendix:\n\n\"The analysis of structures optimized by MACS. \n\nWe calculated the atom-atom distances up to 6.00 Angstroms within all 7,200 structures which were optimized using MACS. For each composition, we tabulated the distances, rounded to 2 decimal places as histograms, all of the histograms for individual compositions were then summed to give a total atom-atom distance histogram for each composition. For each composition, we have then tabulated the shortest distance observed, and the distance of the first major peak in the histogram, both provided in Angstroms. We provide these results below with the total number of counts for the distances observed in brackets.\n\nComposition   |   Min Distance    |    First peak  |\n---|---|---|\nCa3Al2Si3O12 | 1.44 (4)                | 1.69 (5324)    |\nCa3Ti2O7         | 1.38 (2)                | 1.85 (3924)    |\nCu28S16           | 1.97 (2)                | 2.22 (4708)    |\nK3Fe5F15         | 1.78 (2)                | 2.00 (5392)    |\nY2O3                 | 1.45 (2)                | 2.22 (8218)    |\nSrTiO3              | 1.40 (4)                | 1.87 (3898)    |\nSr2TiO4            | 1.41 (12)              | 1.84 (2850)    |\nSr3Ti2O7          | 1.41 (8)                | 1.86 (2506)    |\nSr4Ti3O10        | 1.41 (8)                | 1.86 (1950)    |\n-------------------------------------------------------------------------\n\nThe shortest interatomic distances observed are plausible distances, the major observed peaks for each composition correspond to expected values when considering each chemistry. For example, the first peak at 1.87 Angstroms in SrTiO$_3$, is close to that of the Ti-O distances in the experimentally observed structure of 1.95 Angstroms. These observations in addition to the distribution of energies shown in Fig.12 of Appendix B.4 lead us to the conclusion that the final structures produced by MACS do not contain unphysically short interatomic distances that are typical for structures produced due to the failure of the energy calculator.\""}, {"review_id": "iFcwyebL4k", "reviewer": "Reviewer_V57T", "summary": "This paper introduces a multi-agent RL approach to optimizing crystal structure. The authors propose using independent SAC agents to represent each atom in the structure and introduce a state space which provides local information to each agent. Each agent is trained on independant reward functions simultaneously. The authors measure the performance of their proposed method against existing optimization techniques and find that their proposed method can achieve as good results as the best baseline while being significantly more efficient.", "questions": "(1) I am not 100% clear if the training computation is included in the metrics for comparing to baselines. My understanding is that it is not. Is this a fair comparison? It would be helpful if the authors could provide a argument for why this makes sense. \n(2) Failure rate $P_f$ is not defined and its positioning in Table 1 is a bit confusing. Is this the overall failure rate across all composition? Can this be reported per composition instead?\n(3) SAC produces a stochastic policy, however I do not see any discussion of evaulation over multiple seeds - was this done? \n(4) In line 295 it states that episodic reward is normalized to compare across reward functions - normalized in what way?", "limitations": "The authors address several limitiations, however I would add that the claim in line 319 seems too strong for the results provided in the paper since the test cases only consist of structures that contain the same element compositions as the training cases.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n(1) The paper is clearly written and easy to follow. Figure 1 provides a good and clear overview of the algorithm. Sufficient context is provided around crystal structure optimization for someone unfamiliar with the field to understand the application.\n\n(2) Experimental results indicate that the proposed method MACS outperforms existing methods in terms of efficiency while matching the best existing method in terms of failure rate.\n\nWeaknesses:\n(1) The novelty of the paper is limited. The primary contribution of the paper is formulating the optimization of crystal structure as a POMDP by defining an appropriate state and action space and reward function. Otherwise, the algorithm is a standard application of SAC. Additionally, while the paper refers to multi-agent RL, the actual algorithm treats each agent as a independent agent optimizing indepedent rewards. While an important and interesting application of RL, the core machine learning contribution is limited.\n(2) It is not clear to me if the baseline comparisons are entirely fair. For example - are the reported metrics $T_{mean}$ and $C_{mean}$ inclusive of the training steps required to train MACS? Finally MACS would produce stochastic policies but I do not see a discussion of the variability in policy performance across seeds.", "was_revised": true, "final_justification": "The results across seeds look consistent and my concerns over training time exclusion in the evaluation have been addressed, therefore I will raise my score to 4. I do not raise my score higher as I still think the novelty of the work is somewhat limited, however I think the contribution is technically solid and the performance results are good.", "rebuttal": "Dear Reviewer,\n\nThank you for all of your kind suggestions and questions. In the following, we answer all of your questions in detail.\n\n> Weakness (1): The novelty of the paper is limited. The primary contribution of the paper is formulating the optimization of crystal structure as a POMDP by defining an appropriate state and action space and reward function. Otherwise, the algorithm is a standard application of SAC. Additionally, while the paper refers to multi-agent RL, the actual algorithm treats each agent as a independent agent optimizing indepedent rewards. While an important and interesting application of RL, the core machine learning contribution is limited.\n\n**Answer to weakness (1)**: We would like to refer the reviewer to the Summary Rebuttal (at the beginning of the Rebuttal to ytGL) for the broad discussion. We acknowledge that exploring richer multi-agent interactions (e.g., via communication or centralized training) is a possible direction for future work.\n\n> Weakness 2: It is not clear to me if the baseline comparisons are entirely fair. For example - are the reported metrics and inclusive of the training steps required to train MACS? Finally MACS would produce stochastic policies but I do not see a discussion of the variability in policy performance across seeds.\n\n> Question (1): I am not 100\\% clear if the training computation is included in the metrics for comparing to baselines. My understanding is that it is not. Is this a fair comparison?..\n\n**Answer to Question (1)**: Indeed, the training time is not included in the comparison with the baselines. We suggest explicitly mentioning it in the main text and referring to a new section in the Appendix for the detailed discussion. The proposed text for that section is as follows:\n\n\"Appendix Section: Training Time Contribution to the Experiments.\n\nThe training time of MACS is 343,800 seconds and was not included in the evaluation of MACS. We evaluate the MACS policy based on optimization time only because including the combined training and testing time would make the evaluation highly sensitive to the size and number of test sets. In fact, the experiments could be extended by increasing the sizes/numbers of the test sets until the relative contribution of training time becomes negligible, leading to the reported results.\n\nMoreover, this setup reflects practical usage, as screening hundreds of thousands of structures is a typical use case in computational chemistry. Furthermore, the scalability and zero-shot transferability of MACS allow it to be trained on one set of compositions and then used to efficiently optimize larger structures from other compositions, without additional training, i.e., with zero training time in those cases.\n\nThere is also no fair way to estimate the combined training and testing time, since these phases were conducted under different conditions. Although the same hardware was used the difference in the number of active cores (see Appendix B.1) raises the question of whether to report runtime using wall-clock time or CPU time, and which approach is fairer.\n\nFor reference, comparing the combined training and optimization time of MACS with the optimization time of the fastest baseline (BFGSLS), MACS is 3\\% faster in terms of wall-clock time but 126 times slower in terms of CPU time. Given the fact that training time can be omitted when the method is used in practice, we compare the methods only by the optimization time, as shown in Table 1.\"\n\n> Question (2): Failure rate is not defined and its positioning in Table 1 is a bit confusing. Is this the overall failure rate across all composition? Can this be reported per composition instead?\n\n**Answer to Question (2)**. The failure rate notation $\\mathbf{P}_{F}$ is introduced on line 240. The failure rate reported in Table 1 represents the average across all test sets. Since the average failure rate is low and comparable to the best baseline, we provide failure rates per test set in Table 7 of Appendix B.4 and refer to it in the caption of Table 1. Table 7 shows that the failure rate remains consistently low across all test sets. We will add a note to Table 1 to clarify that the numbers provided there are averaged across the test sets to avoid the confusion.\n\n> (3) SAC produces a stochastic policy, however I do not see any discussion of evaulation over multiple seeds - was this done?\n\n**Answer to Question (3)**: We initially did not evaluate MACS on the same structures across multiple seeds, as we relied on a large number of randomly generated structures (300 per test set, across 24 test sets) to ensure reliable policy evaluation. Following the reviewer’s suggestion, we have conducted an additional experiment to assess variability due to the stochastic nature of SAC used by MACS. We propose adding the following section to the Appendix and referencing it in the main text:\n\n\"Appendix Section: Variability of the MACS Policy.\nWe evaluated the policy on a test set containing SrTiO$_3$ structures with 40 atoms by performing nine additional optimization runs, resulting in ten runs in total for this set. We measured variability in the average number of steps for successful optimizations, and the failure rate.\n\nThe failure rate was 0.33\\% in 2 runs, 0.66\\% in 1 run, and 0\\% in all others, confirming that failures remain rare.\n\nFor the number of optimization steps/energy calculations, we estimated (mean, std) among 10 seeds as (147.48, 3.18) which is well below the mean number of steps (190) in the BFGSLS, the best baseline in this test set.\"\n\n> Question (4): In line 295 it states that episodic reward is normalized to compare across reward functions - normalized in what way?\n\n**Answer to Question (4)**. All episodic rewards collected during training are normalized using min-max (unity-based) normalization to fit within the range $[0,1]$. This normalization avoids comparisons based on absolute reward values and instead allows assessment of convergence speed and stability.\n\n> Limitations: The authors address several limitiations, however I would add that the claim in line 319 seems too strong for the results provided in the paper since the test cases only consist of structures that contain the same element compositions as the training cases.\n\n**Answer to Limitations**. For any phase field with at least two elements, the number of possible compositions is infinite. In this context, the ability of MACS to apply a policy trained on one composition to optimize all structures within the same phase field makes it a truly universal optimizer for that field. To enable transfer learning across different phase fields, additional training on compositions from diverse phase fields will be required. We suggest to replace the sentence in line 319 by the following sentence:\n\"In conclusion, MACS has the potential to evolve into a universal geometric optimizer for periodic crystal structures.\""}, {"review_id": "m6jXtKth1r", "reviewer": "Reviewer_TGFN", "summary": "This paper presents MACS, a novel multi-agent reinforcement learning (MARL) method for optimizing periodic crystal structures. By formulating the geometry optimization task as a partially observable Markov game (POMG), the method treats atoms as decentralized agents and learns local policies that collectively minimize atomic forces in periodic structures. The authors benchmark MACS against state-of-the-art optimization methods such as BFGS, FIRE, CG, and others, and demonstrate significant improvements in optimization time, failure rate, and transferability to unseen systems.", "questions": "N.A.", "limitations": "N.A.", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strength: \n\n1. The problem of crystal structure optimization is well-motivated, and the experimental setup includes a reasonable benchmark against well-known baselines.\n\n2. Experimental results suggest improved generalization performance over several baselines.\n\nWeakness:\n\n1. Although the paper casts atoms as agents in a MARL setting, the agent design lacks meaningful interaction or interdependence typically expected in multi-agent systems. There is no explicit coordination mechanism beyond standard parameter sharing. This setup resembles multiple parallel optimizers more than a principled MARL approach.\n\n2. The “agent” in MACS functions more like a local function approximator on force minimization than a true autonomous or strategic decision-maker in a multi-agent setting. The use of the MARL framework here feels superficial and may be misleading in terms of the novelty claimed.\n\n3. The paper compares against standard MARL baselines, but does not include strong compositional or hierarchical RL baselines\n\n4. No theoretical insights or guarantees are provided on the benefits of compositionality, generalization bounds, or conditions under which MACS would outperform flat policies.", "was_revised": true, "final_justification": "Part of the unclearness is explained, but most of the weaknesses are still unaddressed.", "rebuttal": "Dear Reviewer,\n\nThank you for taking the time to read our paper and provide valuable feedback.\n\n> Weakness: Although the paper casts atoms as agents in a MARL setting, the agent design lacks meaningful interaction or interdependence typically expected in multi-agent systems. There is no explicit coordination mechanism beyond standard parameter sharing. This setup resembles multiple parallel optimizers more than a principled MARL approach.\n\n> Weakness: The “agent” in MACS functions more like a local function approximator on force minimization than a true autonomous or strategic decision-maker in a multi-agent setting. The use of the MARL framework here feels superficial and may be misleading in terms of the novelty claimed.\n\n**Answer to Weakness 1**. We would like to refer the reviewer to our Summary Rebuttal (at the beginning of the Rebuttal to ytGL) that addresses these questions.\n\n> Weakness: The paper compares against standard MARL baselines, but does not include strong compositional or hierarchical RL baselines\n\n**Answer to Weakness 2**. To the best of our knowledge, we are the first to apply MARL to address periodic crystal structure optimization (as mentioned in Lines 58-59 in our paper). All baselines are not MARL approaches, and we are not aware of any existing hierarchical RL methods used for this task. We believe hierarchical RL methods could be more suitable for solving global structure optimization, which is an interesting direction for future work.\n\n> Weakness: No theoretical insights or guarantees are provided on the benefits of compositionality, generalization bounds, or conditions under which MACS would outperform flat policies.\n\n**Answer to Weakness 3**.  We appreciate the reviewer’s point and agree that providing formal theoretical guarantees for performance or generalization in this setting remains an open challenge. Our primary contribution focuses on designing and validating an MARL-based framework that addresses the complexity of crystal structure optimization, which involves highly non-convex, high-dimensional energy landscapes with strong physical constraints. Deriving theoretical guarantees in such environments is non-trivial because reward functions and state spaces depend on material-specific physics rather than the synthetic or formal assumptions typically used in theoretical analysis. \n\nThat said, our results provide strong empirical evidence that the MARL framework preserves transferability within a phase field and maintains scalability for structures twice as large as the structures the policy was trained on. We agree that establishing theoretical conditions for guarantees on optimization time and quality - whether for existing methods or new ones - would be a significant achievement, which we leave for future research."}]

Author rebuttal:

Meta-review: This paper proposes a multi-agent reinforcement learning (MARL) framework for crystal structure optimization, where each atom is represented by an independent SAC agent operating with a locally defined state space. The agents are trained with independent reward functions, enabling simultaneous learning across atoms. The method is evaluated against classical optimization techniques and demonstrates performance comparable to the best baselines while being significantly more efficient.

The reviewers acknowledged the technical soundness and originality of the proposed MARL formulation, particularly the novel use of per-atom agents under periodic boundary conditions. The empirical evaluation was considered robust, with consistent results across seeds and comparisons that fairly demonstrated the benefits of the method. While some initial concerns were raised regarding the strength of the baselines, the appropriateness of the evaluation protocol, and the limitations of independent learners, these were either adequately addressed in the rebuttal or found less critical upon further discussion.

That said, some limitations remain: the novelty is perceived as somewhat limited by certain reviewers, and design choices such as minimal species encoding, fixed lattice vectors, and reliance on CHGNet constrain the generality of the approach. Presentation issues (e.g., dense captions, minor typos) were also noted but are easily fixable. One reviewer also highlighted their reduced confidence in assessing the work due to limited familiarity with domain-specific literature. Another reviewer, with a rating of 3, is not convinced on the theoretical insights or guarantees provided on the benefits of compositionality, generalization bounds, or conditions under which MACS would outperform flat policies. The authors acknowledge this limitation. My personal belief as an AC is this limitation is not severe enough to undermine the other strengths of the work.

Overall, the paper is seen as a solid contribution to scientific discovery through MARL, advancing the state of the art in crystal structure optimization.
Decision: Accept (poster)