Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
Authors: Rohan Shenoy
Evan Austen Coleman
Hans Gaensbauer
Elsa Olivetti
Venue: ICML 2025 poster (2025)
Keywords: NMR
MRI
elemental analysis
atomic abundance
carbon measurement
MRV
RL
magnetization
nuclear spin
pulse sequencing
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We train a policy which counts atoms of a given element in a sample more quickly than traditional approaches, by controlling a magnetic field to align their nuclear spins.

Abstract: Quantifying the elemental composition of a material is a general scientific challenge with broad relevance to environmental sustainability. Existing techniques for the measurement of atomic abundances generally require laboratory conditions and expensive equipment. As a result, they cannot be deployed *in situ* without significant capital investment, limiting their proliferation. Measurement techniques based on nuclear magnetic resonance (NMR) hold promise in this setting due to their applicability across the periodic table, their non-destructive manipulation of samples, and their amenability to *in silico* optimization. In this work, we learn policies to modulate NMR pulses for rapid atomic abundance quantification. Our approach involves three inter-operating agents which (1) rapidly align nuclear spins for measurement, (2) quickly force relaxation to equilibrium, and (3) toggle control between agents (1) and (2) to minimize overall measurement time. To demonstrate this technique, we consider a specific use case of low-magnetic-field carbon-13 quantification for low-cost, portable analysis of foodstuffs and soils. We find significant performance improvements relative to traditional NMR pulse sequencing, and discuss limitations on the applicability of this approach.

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
Rohan Shenoys 12 Evan A. Coleman
Hans Gaensbauer' Elsa A. Olivetti
Abstract
impurities management, and emissions monitoring (Norris et al., 2020). With such a simplification, this accounting process can be applied as an objective function for control- related tasks, which prevent those atoms from achieving wasteful or pollutant fates in industrial processes. The car- bon credit marketplace is macroscopic example, as it seeks to commodify carbon atoms by assigning a cost to their re- lease (as greenhouse gases, or GHGs) and reward to their sequestration (as e.g. carbonates, organic compounds, or pressurized gases) (Probst et al., 2024). However, such de- carbonization strategies face a major challenge: accurately and verifiably monitoring these elemental transactions in globally-scalable fashion (Coleman al., 2023; Macfarlane et al., 2024). Many techniques are commonly employed to effectively count atoms (measure atomic abundances) through chemical reactions and other destructive processes under controlled conditions, but these techniques share fun- damental limitations that hinder their effectiveness for cli- mate objectives broadly.
Quantifying the elemental composition of terial is a general scientific challenge with broad relevance to environmental sustainability. Exist- ing techniques for the measurement of atomic abundances generally require laboratory condi- tions and expensive equipment. As a result, they cannot be deployed in situ without significant cap- ital investment, limiting their proliferation. Mea- surement techniques based nuclear magnetic resonance (NMR) hold promise in this setting due to their applicability across the periodic ta- ble, their non-destructive manipulation of sam- ples, and their amenability to in sílico optimiza- tion. In this work, we learn policies to mod- ulate NMR pulses for rapid atomic abundance quantification. Our approach involves three inter- operating agents which (1) rapidly align nuclear spins for measurement, (2) quickly force relax- ation to equilibrium, and (3) toggle control be- tween agents (1) and (2) to minimize overall mea- surement time. To demonstrate this technique, we consider a specific use case of low-magnetic- carbon-13 quantification for low-cost, portable analysis of foodstuffs and soils. We find signifi- cant performance improvements relative to tradi- tional NMR pulse sequencing, and discuss limita- tions on the applicability of this approach.
These limitations motivate the development of measure- ment alternative which can meet the needs of climate change solutions. In this work, we propose a scalable data-driven method for rapid atomic abundance measurement in chal- lenging mcasurement environments. Our contribution is threefold. First, we train policies by reinforcement which shape and sequence magnetic field pulses for use in a sim- plified (low-fildd nuclear magnetic resonance (NMR) spec- troscope. Unlike incumbent procedures, this method is non- destructive, uses inexpensive and small-scale hardware, and generalizes to nuclear isotopes of many different elements. Secondly, we present a fast, robust simulator for generating large quantities of NMR spectroscopy data which is capable of reproducing the nuclear spin dynamics of many differ- ent samples in parallel when they are manipulated by an arbitrary magnetic field. Lastly, we present novel method to manipulate these spins for atomic abundance measure- ment, by training three inter-operating agents to orchestrate an NMR pulse sequence. We demonstrate its performance on the task of carbon- (*²C) abundance measurement in caffeinated water, using a simulated dataset derived from laboratory NMR spectra, and find a significant performance improvement relative incumbent techniques.
1. Introduction
Developing scientific solutions to mitigate climate change can, in certain cases, be reduced to the task of budgeting or counting atoms of specific elements. These interpreta- tions are commonly leveraged in multiple areas related to sustainability, including soil health measurement, material
Equua contribution 'Massachusetts Institute of Technology, Cambridge, MA Universiiy of California Berkeley, Berkeley, CA Correspondence to: Evan A. Coleman <ccol@ mit.eumit.duu>...
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance mcasurement
2. Preliminaries
NMR in this context, but data-driven methods are commonly applied to MRI to reduce the time and resources required for medical imaging and diagnostics. Magnetic resonance fingerprinting is one such technique: it uses a pattern recog- nition algorithm on labeled data to derive pulse sequence which can lead to direct imaging or segmentation of tissue properties (e.g. whether is cancerous). The authors of (Ma et al., 2013) first introduced this method and demonstrated it on samples of brain tissue, finding a significant reduction in the required measurement time relative to standard ap- proaches for the segmentation of grey matter from white matter (12.3 seconds versus nearly 10 minutes). More re- cently, deep learning was found to be highly effective at correcting electromagnetic interference in low-field MRI setup (using a 0.05 T magnetic field) without any loss in performance a typical MRI (usually 2 T) (Zhao et al., 2024). The authors showed that imaging of human subjects used less power and generated less noise with this approach, while requiring a comparable amount of measurement time.
Nuclear Magnetic Resonance (NMR) a physical phe- nomenon wherein the spins of atomic nuclei contained in a magnetic field are pulsed with electromagnetic radiation, causing them to precess before returning back in line with the magnetic field (Nishimura, 1996). This phenomenon is analogous to gently knocking a spinning top: it gyrates before eventually returning to stable rotation around the vertical axis. NMR spectroscopy is a measurement tech- nique based on this phenomenon, in which a substance is placed in strong background magnetic field and is exposed to a sequence of radio- radio-freqeency electromagnetic pulses which cause the atomic nuclei to radiate audiofrrqquency response signals at different frequencies according to their nuclear spin characteristics. The electromagnetic emission is measured at an axis perpendicular to the direction of the background magnetic field, giving a signal known as the free induction decay (FID). The Fourier transform of this signal is the NMR spectrum appropriate to the sample, and regression- baased techniques are used to compare this spec- trum to library of reference spectra for identification and analysis. To evaluate the relative abundance of the charac- terized material in different sample, one generally compares the integrals of their spectra in a fixed window of frequen- cies. Many different pulse sequences have been designed to target various properties of atoms and molecular bonds. The most basic such pulses simply rotate spins of a given nuclear isotope by a fixed polar angle relative to the background magnetic field, with the extent normally denoted in radians as e.g. a 2 or a π pulse (Nishimura, 1996). NMR may be applied to solid-phase samples, but is typically and more readily applied to liquids (Reif et al., 2021).
Reinforcement Learning (RL) for NMR. Some recent work has applied reinforcement learning techniques to gen- eral molecular structure prediction with 80% accuracy (Srid- haran et al., 2022), and to the classification of the geographic origins of wine with 94% accuracy (Korenika et al., 2024). However, neither of these works tuned the underlying NMR pulse sequence. Our proposed method is most similar to the carlier work in (Khaneja et al., 2005), which introduced gradient ascent pulse engineering (GRAPE). GRAPE is a form of model preeil-predietve control (Schwenzer et al., 2021) which demonstrated that optimal NMR pulse sequences for coupled nuclear spin systems could be designed by gradient descent on simulated data to minimize distance to gen- eral target state. However, the authors did not explicitly construct such a target state appropriate to abundance mea- surement." 1 In addition, GRAPE develops pulse sequences only in silico, and is only optimal given complete informa- tion about the intermediate states of the spins. It can become prohibitively computationally expensive for complex com- pounds containing large numbers of effective spins, which is an obstacle even in contexts where highly pure NMR spin systems are used, such as quantum computation (Jones, 2024). The output of GRAPE also requires an accurate
Magnetic Resonance Imaging (MRI) NMR spectroscopy is commonly applied in medicine as part of more sophis- ticated procedure known as MRI, which produces high- resolution anatomical imagery by associating FIDs to points in space, targeting the HH nuclei in water (Nishimura, 1996). NMR and MRI both have many prevalent applications in food science, environmental monitoring, and various engi- neering fields (Marcone al., 2013; Dais & Hatzakis, 2013; Soong et al., 2015; van der Klink& & Brom, 2000).
Machine Learning for Low-Rese MRI. A current area of emerging NMR and MRI research is centered around de- veloping and applying benchtop low-voltage or zero-field systems which use small permanent magnets or remove the background magnetic field entirely (Tayler et al., 2017: Zhao et al., 2024). These systems are of interest because of their reduced cost, as well as their ability to measure in situ. The approach thus represents strong candidate for atomic abundance measurement, since it addresses many of the scalability concerns of laboratory- atomic abundance analyzers. Machine learning has been applied sparingly to
Note for domain experts: To see why this is nontrivial, con- sider spin- & quantum system in the case of 2D readout. Achiev- ing a pulse which is agnostic the direction of the target state in the transverse plane requires studying multiple pulses with differ- ent such targets. A reward which averages across such target states does not work, because it gives the direction of the background
field Bg as a target, i.e. IC) = 1 I (11) + e 10)) = [1). In this scenario, an optimal pulse which maximizes (S S just B, = 0. Controlled experimental configurations to perform pulse fiee measurement are challenging to engineer, as they require rapidly and repeatedly switching between measuring the sample and measuring pure noise, similar to optical chopper setups.
2x
2

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
underlying physical model (Hamiltonian) for the spins in the sample, which is not generally possible for complex, long-chain molecules such as proteins (Hu et al., 2021).
2. The newly applied pulse, [B1,0,0] 0]
3. The constant background Bo field, [0, 10,0, Bo]
4. Instrument parameters: runtime T and temperature
5. The maximum observed magnetization observed
Our Contribution. This work is the first to apply modern reinforcement learning approaches to NMR pulse sequenc- ing for rapid atomic abundance measurement. Relative to methods such as GRAPE, we train policies by reinforcement on observables which are actually accessible in real NMR systems, so that our approach can exploit but does not rely on a complete underlying description. In addition, while we demonstrate the approach on simulated low-cost hardware, our method does not require an accurate pre-determined physical model of the underlying sample, and can be trained directly on real samples. We do not encounter computational barriers in our simulation-based results because we use the macroscopic Bloch equations to model the system, which are appropriate for large sample of material (Nishimura, 1 1996). These are straightforward to simulate, and simplify the design of objectives.
(max4 v MM + M2 thus far.
The temperature (which controls the instrument noise level) evolves as Tt+1 = Tt * C. where C ~ (1,, and C is redrawn at each timestep. While normally temperatures are held constant during runs, this choice of state space mimics the reduced capabilities of a low-cost in situ NMR spectrometer, and provides sufficiently realistic conditions for the agent to learn effective pulse modulation strategies.
We vectorize this environment using the SubprocVecEnv class in OpenAl Gymnasium, which runs multiple indepen- dent instances of the NMR in parallel. This considerably accelerates the training process by enabling effective train- ing over modern multi-core hardware cluster. Furthermore, the agent is able to explore diverse set of samples concurrently, mitigating overfitting to single sample or specific simulator configurations. This setup also facilitates running multiple instances of the same sample in parallel, meaning that the that the noise inherent to each simulation instance is averaged over many environments, leading more robust training and policies.
distributed
2.1. Simulator Design
To train policies, we emulate nuclear spin dynamics for a general NMR spectrometer. Our NMR simulator is struc- tured around an NMR environment that mimics a real-world spectrometer by restricting the agent's view to only high- level information that would be measured experimentally. Underneath this environment lies a spin simulator, which applies a first-order differential equation solver to the Bloch equations (see Appendix A), which model the time evolu- tion of the individual nuclear spin vectors that compose a specified sample. Given an applied magnetic pulse, a set of nuclear spin parameters, and the chosen time resolution, the spin simulator computes the response of the sample and produces the net M2 and My magnetization components of the sample. Before this output vector is passed back to the NMR environment, -yhuiin-Nqquist instrument noise is introduced. The profile of this noise is roughly constant in frequency space (which we model as Gaussian in the time domain), and has an amplitude determined by the tempera- ture of the readout, as well as its associated bandwidth and resistance (Perepelitsa, 2006).
2.2. Laboratory Data and Spin Set Generation
Each atomic spin in our simulation is defined by a tuple (y, T2), where y is the gyromagnetic ratio, Ti is the longitudinal (spin-lattice) relaxation time, and T2 is the transverse (spin-spin) relaxation time. 2 The gyromagnetic ratio 7 governs the rate at which a spin precesses around the static magnetic field By. The longitudinal relaxation time Ti describes how quickly the the spin's precession speed decays after excitation, while the transverse relaxation time T2 characterizes how rapidly spins lose phase coherence in the transverse plane, causing the observable signal to decay.
To build realistic spin set, we begin with the FID signal produced by actual NMR measurements. We first dissolved pure caffeine powder (Millipore Sigma, 99% purity) in wa- ter at a concentration of 0.057 mol/L. We then performed a serial dilution of this sample and collected 6 FIDs at progres- sively lower caffeine concentrations. We used the default ID NMR pulse (Single 5. 8000 scans, 2s acquisition time per scan) in a JEOL502 two-channel ECZ spectrometer op- erating at 500.44 MHz. We apply a Fourier transform to these FIDs to obtain NMR spectra, which exhibits sharp peaks corresponding to different atomic spins (7, Ti, Ta)
The NMR environment receives the net calculated magneti- zations and uses them to construct the state space presented to the agents. To demonstrate the approach, we assume the use of a two-axis magnetometer for signal readout, and a single-axis Faraday coil to generate the magnetic field for the pulse. though we emphasize that these are not require- ments for the proposed method and can be easily changed based on available hardware. Each state is represented four-element tuple in two dimensions, comprising:
NNoee for domain experts: As this paper is primarily ad- dressed to a machine learning audience, for simplicity we did not introduce the concept of chemical shift 8 and instead label spins by individually-deffeed effective gyromagnetic ratios 7.
1. The observed transverse magnetization (Ma, My)
3

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
present in the sample. From the peak frequencies (also known as chemical shifts), we infer the gyromagnetic ratios y of constituent spins. We use the exponential decay of the FIDs to derive the average T2 for the spins, and sample Tis from a Gaussian distribution as a source of training data variability.
in the NMR), at timestep 4. Intuitively, this phenomenon should occur when roughly all spins in the system have been coherently driven into the transverse plane, pointing in the same direction for short period of time. This alignment causes a sharp and high-frequency jump in the observed magnetization, which we refer to as a chirp.
Next, we simulate standard baseline -ieedimenii NMR experiment using the identified gyromagnetic ratios. At this stage, the frequencies of the peaks in the simulated spectrum match those of the real samples. However, the in- tensities (peak heights) typically do not. We therefore adjust the weights associated with each spin so that the simulated peak heights align with those in the empirical spectrum. These weights roughly represent the relative numbers of each class of spin in the overall sample. By iterating this process of matching the simulated spectrum to the measured spectrum, we arrive at a spin set whose composition closely aligns with the true samples'.
To formalize our approach, we define a Markov Decision Process (MDP), (S, A, 11 r),,, as follows:
The state space, S, is defined above.
The action space, A, is the continuous interval cor- responding to B1 E [-0.2, 0.2] Tesla, with Bo = 1 Tesla This magnetic field configuration is inexpen- sive to implement relative to those typically used in laboratory-grant NMR setups (generally ranging from 2 to 45 Tesla for Bo and up to 0.1 Tesla for Helmholtz coil-pulsed B,) (Moser et al., 2017; Con- radi, 2024). Importantly, unlike setups with stronger Bo. this choice of parameters can be achieved inexpen- sively using a Faraday coil' for B₁ and a neodymium- based permanent magnet for B, (Barón et al., 2023).
3. Experiments
In pursuit of framework for counting atoms with NMR, we explored and evaluated three reinforcement learning strate- gies which build one another. The guiding idea behind our approach is to devise policies capable of manipulating nuclear spins to squeeze the largest response possible out of a given sample, under realistic environmental conditions. The "loudness" " or intensity of this signal is expected to be directly proportional to the abundance of nuclear spins in a sample (Nishimura, 1996). First, we ask whether agent trained to maximize this signal can perform better than baseline NMR pulse. Second, we test whether a policy to quickly reset the system to its initial undisturbed state can permit this maximum response to be quickly reproduced. Lastly, we demonstrate that the decision to reset the system can also be delegated to an agent.
The policy 1l denotes the learned distribution of ac- tions conditioned on the observed state.
The discount factor I' is set to 0.99.
. The reward function at any timestep T' is given by:
FT
max 0StST
(M2, + M,,,).
(I)
To evaluate the efficacy of this approach, we train the PPO model (as described above) for 5M timesteps of length 0.2 ms across 50 distinct environments with o, 0.01. Each environment is initialized with a unique set of atomic spins determined by the procedure outlined in Section 2.1. We repeat model training on 5 fixed random seeds.
Figure I plots the maximum magnetization observed for a collection of samples versus the concentration of caffeine in those samples, comparing the learned pulse against the standard ID NMR pulse. The depicted relationship for the learned policy is steeper and more monotone than that of the ID NMR pulse, indicating that a more precise and direct relationship with concentration is achieved. The time required for these maximum concentrations to be reached was roughly 16% of the time required for a ID NMR pulse sequence to complete, representing a significant reduction. However, because the policy drives the sample to state of higher magnetization, it may require significantly more
For all policy trainings in these experiments, we employed the Proximal Policy Optimization (PPO) algorithm (Schul- man et al., 2017) provided in StableBBaselinee (Raffin et al., 2021). Additional details regarding this approach are pro- vided in Appendix B.
3.1. Model 1: Making Samples Chirp by Maximizing Transverse Magnetization
In the first experiment, we aim to design a pulse which ac- curately and quickly determines the maximum achievable magnetization response for a compound of interest. For- mally, we imagine placing a sample of material in mild background magnetic field along the 2 axis, and we reward an agent for achieving large values of M2 + M2 where Mr. and My.+ are the net observed magnetizations in the x and directions (the directions of the magnetic field sensors
"The field strength of solenoid made of wound copper wire is /NIIL, with μ the magnetic permeability of air, / the cur- rent flowing through the wire, and N' the number of coil turns in length L. For a 43 ga-ga enamel- copper wire of diameter 0.006 cm, the required current for singly-wound coil to achieve B, of 0.2 Tesla is 9.7 Amperes, comparable to the current drawn by a home microwave.
4

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
time to relax to its initial state. This is a problem if the experiment is be repeated many times in order to suppress measurement noise, suggesting the need for second policy which can accelerate the sample's 's return to equilibrium. We address this concern in Experiment 3.2.
environment. During this phase, the agent learns replicate the time-dependent behavior of the standard pulse sequence.
Once pre-training is complete, the neural network weights are transferred to the underlying network of the PPO agent. The agent is then trained for 5M timesteps in the environ- ment, following the same procedure used in the baseline experiment detailed in Section 3.1. This approach allows the agent to refine the standard ID NMR pulse using our reinforcement learning framework. Interestingly, the policy converged rapidly, within the first 300K timesteps, which suggests that the ID NMR pulse may be a local minimum for the loss function.
3.1.1. TRAINING WITH INCREASING NOISE
To understand the impact of increased noise on model ef- ficacy, we modify the training procedure by progressively increasing the instrument noise profile applied to the ob- served magnetizations. The training begins with an insub- stantial noise profile, allowing the model to initially focus on learning how its actions affect the spins without being hindered by noise. Every 500k timesteps, the noise level is multiplied by a factor of 10. This gradual increase in noise continues until the final 500k timesteps, at which point the noise profile is identical to that under which the model in the baseline Experiment 3.1 is trained. This staged approach introduces the agent to increasingly challenging conditions over time, while also preventing it from being overwhelmed by noise in the early stages. We find that this increased noise did not meaningfully change the capabilities of the learned policy at achieving a high magnetization, compared to the standard procedure used in Experiment 3.1. This indicates that the model is either inherently robust to noise variations or that the later stages of training, where noise levels match those in Experiment 3.1, dominate the learned policy. Consequently, our findings suggest that while pro- gressive noise increases may prevent early instability, it does not meaningfully improve final performance.
This model proved to be the most effective among our three chirping models. As shown in Figure 1, it achieves a sub- stantial increase in achieved magnetization, particularly in higher concentration samples. This suggests that leveraging the structured prior of the 11) NMR pulse sequence provides strong initialization, allowing the model to refine its policy more efficiently and exploit domain-specifn patterns that improve performance.
3.2. Model 2: Accelerated Spoiling
The standard ID NMR pulse sequence alternates between applying a sinusoidal pulse and allowing the system to re- set (i.e. not pulsing at all) during a fixed period. This ensures that the system resets the equilibrium state, so that repeated pulses (or scans) can generate statistically- independent samples of the same signal. The effect of repeated measurement is to suppress noise at rate pro- portional to v where n is the number of scans. While this conservative approach ensures accuracy, it is time-intensive. An approach known as spoiling can be employed to more quickly nullify residual transverse magnetization (Jehenson & Bloch, 1991).
3.1.2. DIRECT OPTIMIZATION OF STANDARD ID NMR PULSE SEQUENCE
In this experiment, we initialize the agent with a pre-trained policy derived from from the standard ID NMR pulse se- quence. This approach leverages the structure and known efficacy of the standard pulse sequence as a prior, rather than starting from scratch. By doing so, we seek understand whether features of existing approaches can be exploited to design effective pulse sequences by reinforcement. The pro- cess we apply here is analogous to the design of so-called "soft" NMR pulses, which are slight deformations of the standard or "hard" pulses (Brüschheiler et al., 1988).
In this experiment, we learn an accelerated spoiling policy. The MDP is the same as in Experiment 3.1, but the reward at any timestep t is modified to
Tz = (M2,, + M2.)) - 1[t = TI + +,, (2)
This reward penalizes the model for the current transverse magnetization at any step f, as well as the final transverse magnetization at the end of the episode.
To preitrain the model, we employ behavior cloning using the imitation learning framework from the sequences library (Raffin et al., 2021). Observation-acee
We find that the learned spoiling policy is highly effective at rapidly reducing transverse magnetization by actively dis- persing the spins from the transverse plane, which is how the agent receives the system state after the chirp. Com- pared to the standard method of simply applying no pulse, our approach achieves an up to 28% reduction in average magnetization over the evaluation period. This effect can be visualized in Figure 2, where the policy outperforms passive relaxation in driving the system to lower magnetization.
are generated by choosing the actions determined by apply- ing the ID NMR pulse sequence to the NMR environment, where the observations are determined by the state of the system, and the actions are the pulse values dictated by the standard pulse sequence. The model is trained for 50K epochs, with each epoch consisting of 2, 2,048 timesteps in the
5

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
Importantly, this accelerated relaxation occurs within a timescale that is just a fraction of the system's 72 relax- ation time, whereas in the standard protocol, the system is allowed to relax for multiple T2 periods. While the system is not reset to equilibrium within this shortened relaxation pe- riod, the suppression of magnetization is significant enough that we can simply terminate the pulse after a short spoil- ing time ty and achieve equilibrium more rapidly than the incumbent approach.
we demonstrate that the benefit of the spoiling model within this procedure arises solely from its ability to rapidly reduce the transverse magnetization.
To better understand the behavior of the chirp and spoil- ing models within the toggle framework, we visualize the models' actions as a function of the observed magnetiza- tions as well as the corresponding state densities in Figure 3. Across different environments and samples, we observe that the chirping model consistently reaches the maximum mag- netization in the same direction, validating the structured control it asserts in the system. Additionally, we find that the spoiling model applies a large RF pulse in a consistent manner until the magnetization is significantly reduced, at which point it will occasionally stop pulsing. The effect of this RF pulse is evident in the state density shift in the bottom two plots, where the distribution of magnetization states rapidly transitions in the direction of the RF pulse ap- plied by the spoiling model (the direction). These findings highlight the complementary roles of the chirping and spoil- ing models within our toggling framework. Crucially, we demonstrate that toggle between these models provides a su- perior strategy relative to simple relaxation for maintaining high magnetization and improving scan speed.
3.3. Model 3: Toggled Chirping and Spoiling
In standard NMR spectroscopy, the spoiling period must reset the transverse magnetization below a threshold, so that the sample is returned to its initial state before repeating the pulse. In this experiment, we consider the possibility of instead permitting partial reset, by training a third policy to choose which of the chirping or spoiling models has control of the environment. After chirping with the model trained in Section 3.1.2, the spoiling model from Section 3.2 can take over and push the spins toward equilibrium, before the next chirp re-establishes high magnetization. The MDP appropriate to this process has the same state space, but with an additional binary flag indicating which of the chirp or spoiling policies can control B1, as well as the time le since the chirping policy last had control and the time ts since the spoiling model last had control. The action space is change to that binary flag, although this action is only available 200 timesteps after the most recent toggle. The reward function at any given timestep T is taken as the change in maximum magnetization, with a penalty for long chirp-spoil sequences:
4. Limitations and Future Work
In this work, we introduced a novel technique to measure atomic abundances using NMR pulse sequences which are learned by reinforcement. While we demonstrate the suc- cess of our approach on simulated samples of caffeine dis- solved in water, it has fundamental limitations which con- strain its applicability and require discussion. Most crucially, the use of this method makes the implicit assumption that the samples being measured with a given pre-trained chirp- ing policy have same relative distributions of nuclei as the samples used for training the pulse. As a result, any such application of this measurement approach in the field will require periodic cross-checks against traditional NMR to ensure that distribution shift from the training data has not occurred. Moreover, in this work we treated NMR pulses as targeting individual atomic spins, but inter-nuclear interac- tions within a sample such as proton coupling (Nishimura, 1996) can cause other elements be activated by the applied magnetic field, leading to radio-frequency responses which interfere with our proposed procedure. Characterizing the performance of this proposed method as applied on real sam- ples of analytes of practical interest, in low-cost hardware, is the clearest and most intriguing avenue of future research. We note in addition that the results in Figure 3 (top left, black dots) suggest the existence of a preferred direction for abundance measurement in the transverse plane. would be of interest compare our method against a variant of GRAPE which uses this vector as target state.
KT
FT TT= = is x A max [|
- to x M|M (3)
KCT-to
In Figure 4, we demonstrate that our toggle model outper- forms the chirp-only model due to the repeated partial reset facilitated by the spoiling stage. This result highlights the importance of the interplay between the chirp and spoil- maximize mag-
ing models, which together enable netization, quickly diminish transverse magnetization, and subsequently restore high magnetization in cyclic manner.
To further validate the necessity of the spoiling model within this approach, we compare our toggle models performance against a modified chirp procedure where the maximum magnetization is artificially reset at the same points as the toggling transitions. This experiment reveals that simply setting the maximum magnetization to zero while the chirp is running is insufficient for maximizing magnetization over many runs. This limitation indicates that our MDP structure effectively learns to associate states with their maximum magnetization history, meaning that the learned policy par- tially depends on knowledge of prior magnetizations. Thus,
6

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
65
Chirp Model Pretrained on 1D NMR Pulse
Chirp Model w/ Increasing Noise
Default Chirp Model
60
1D NMR Pulse Sequence
Max Magnetization [au]
55
50
45
40
35
30
0.01
0.02
0.03 Molarity [mol/L]
0.04
0.05
Figure 1. The maximum magnetization achieved by our three chirping models, with baseline of the 1D NMR pulse sequence. The mean and standard error is across the five different fixed training seeds, as well as five environments (which can be interpreted as five independent scans). We see clear one-to-one relationship between molarity (concentration) and maximum magnetization our trained chirping policy. This was not the case for the incumbent ID NMR approach (black line).
3500
Model 2: Transverse Spoiling
No Pulse
3000
Magnetization [au]
2500
2000
1500
1000
0.02
0.04
0.06
0.08
Fraction of T2
Figure 2. Demonstration of the performance of our spoiling model in reducing the average magnetization in one environment compared to the standard approach of applying no pulse.

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
Strength
00
60
-40
-10
-50
My
-20
-20
-20
40
-20 0
20
60 80
-80
-60
-40
-20
0
20
40 80
80
Mx [su]
Mx laul
Figure 3. Heatmaps of the state-action relationship averaged over 50 independent scans of sample. Lines represent the observed state of the system, components the transverse magnetization. Lines are weighted by (Top) the action of the model, and (Bottom) time. Plots on the left visualize the chirping model's behavior, while those on the right show the spoiling model's behavior. In the top left, we include black dots to indicate where max magnetization was achieved in each environment.
80
70
60
50
30
Max Chirp
20
Max Chirp Reset
Max Toggled
Avg Chirp
10
Avg Chirp w/ Reset
Avg Toggled
2000
4000
6000
8000
Timesteps
Figure Depiction the maximum average magnetizations across three different approaches: (1) Chirping model's control; (2) Chirping model's control, but with the tracked max magnetization reset to whenever the toggle model acts; and (3) Toggle model's control.
8

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
Acknowledgments
unreliable, posing potential risks in applications such as carbon crediting and precision farming. Inaccurate carbon sequestration estimates could lead to financial misallocation in carbon markets, while errors in soil composition analysis might result in improper fertilizer application, negatively impacting both crop yields and environmental sustainability. Ensuring robustness across a broad range of sample varia- tions will therefore be critical for the practical deployment of this method.
This work was partially funded by the MIT Climate and Sustainability Consortium. The authors are grateful for the feedback of attendees of the Climate Change AI workshop at NeurlPS 2024, "Tackling Climate Change with Machine Learning.' The authors are additionally grateful to Glen Junor for support in the data collection process,
Impact Statement
References
This work presents novel reinforcement learning-based approach optimize nuclear magnetic resonance (NMR) spectroscopy for the purpose of atomic abundance measure- ment. We demonstrate that utilizing reinforcement learning to optimize the pulse sequence of these NMR systems can provide accurate measurements while reducing acquisition time. This approach has applications in environmental mon- itoring. materials science, and chemical analysis by making NMR-based sample property measurements more rapid, and allowing measurement to occur in-situ (Vyalikh et al., 2024; Simpson et al., 2018).
Barón, L.,, Urbina, R., Pêrez, M., Páez, C.,, and Fajardo, A. A low-complexity experimental characterization of the neodymium magnet grade. In 2023 IEEE 14th Latin America Symposium on Circuits and Systems (LASCAS), pp. 1-4, IEEE, 2023.
Brüschweiler, R., Griesinger, C., Sørensen, O., and Ernst, R. Combined use of hard and soft pulses for decoupling in two-dimensional nmr spec- troscopy. Journal of Magnetic Resonance (1969), 78(1):178-185, 1988. ISSN 0022-2364.
doi:
There are several potential implications of this work. First, the configuration of our simulator as a low-field NMR spec- troscope means that these techniques can be applied to sys- tems costing several hundred times less than a traditional NMR spectroscope (Michal, 2020). Second, the demon- strated efficacy of our methods within the broader NMR framework may build interest in machine learning-based pulse sequencing, revisiting and advancing prior discover- ies in the field which relied on model-predictio control (Khaneja et al., 2005). Nontrivial applications of interest for our setup include rapidly quantifying the water-soluble carbon content of soils, which could have significant im- plications for nature-based carbon crediting systems, and enable more accurate and scalable verification of soil car- bon sequestration. Due to its modularity, in agricultural settings this approach can additionally support data-driven optimization of fertilizer application and other precision farming techniques, improving food system sustainability and reducing its resource consumption.
http://2..oon//00000000002222222
URL https://www.sccenneddiccce ccteaeee/attacee/p////2222888889
Coleman, E., Tripathy, A., Sroka, S., Klein, L., Ferreira da
Silva, A., Rakhlin, M., Roa, B., and Diez, Carbon credits and credibility: A collaborative endeavor. 2023.
Conradi, M. S. Understanding magnetic fields, for nmr/mri. Journal of Magnetic Resonance Open, 20:100158 2024.
Dais, and Hatzakis. Quality assessment and authenti- cation of virgin olive oil by nmr spectroscopy: A critical review. Analyticu Chimica Acta, 765:1-27, 2013. 2013.. doi: 10. Hu, Y., Cheng, K., He, I.., Zhang, X., Jiang, B., Jiang, L.,, Li, C., Wang, G., Yang, Y., and Liu, M. Nmr-based methods for protein analysis. Analytical chemistry, 93(4): 1866-1879, 2021.
While this work demonstrates promising results, several limitations must be considered. First, all experiments were conducted in simulation using a simplified dataset, mean- ing that real-world implementation will likely present addi- tional challenges. Deploying this approach in a production NMR system would require significant computational re- sources and energy consumption train and deploy, i.e. it has environmental footprint which should be trans- parently documented. Additionally, the accuracy of this method depends on training the policy over a sufficiently diverse distribution of samples. If the model encounters out-of-distribution samples, its measurements could become
Jehenson, P. and Bloch, G. Elimination of surface signals by a surface-spoiling magnetic field gradient. thcoretical optimization and application to human in vivo nmr spec- troscopy. Journal of Magnetic Resonance (1969), 94(1): 59-72, 1991.
Jones, J. A. Controlling nmr spin systems for quantum computation. Progress in Nuclear Magnetic Resonance Spectroscopy, 2024.
Khaneja, N., Reiss, T., Kehlet, C., Schulte-Herbüüggeen T., and Glaser, S. J. Optimal control of coupled spin dynamics: design of nmr pulse sequences by gradient

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
ascent algorithms. Journal of magnetic resonance, 172 (2):296-305, 2005.
Reif, B., Ashbrook, S. E., Emsley, L.,, and Hong, M. Solid- state nmr spectroscopy. Nature Reviews Methods Primers, 1(1):2, 2021.
Korenika, A.-M. J., Jeromel, A., Tomaz, I., Jednačak, T., Rončević, S., Nemet, I,, Primožić, L., Hrenar, T., and Novak, P. Deep reinforcement learning classification of sparkling wines based on iep-ms and dosy nmr spectra. Food Chemistry: X, 21:101 162,,, 2024.
Schulman, J., Wolski, E. Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL htaps::/azxiv.ortt
Ma, D., Gulani, V., Seiberlich, N., Liu, K., Sunshine, J. L., Duerk, L.,, and Griswold, M. A. Magnetic resonance fingerprinting. Nature, 495(7440):187-192, 2013.
Schulman, J., Moritz, P., Levine, Jordan,, M., and Abbeel, P. High- diggh-dimensional continuous control using generalized advantage estimation, 2018. URL https: //rx/var.vvvvvva org org/abs/ 1506. 02438.
Macfarlane, M., Jia, R., Fricke, E., Vallicrosa, H., Yu, J., Mirzagholi, L.., Coleman, E., Olivetti, E., and Terrer, C. Nature- climate solutions: Current uncertainties and data gaps in the assessment of soil carbon sequestration potentials. 2024.
Schwenzer, M., Ay, M., Bergs, T., and Abel, D. Review on model predictive control: An engineering perspective. The International Journal of Advanced Manafacturing Technology, 117(5):3377-1349 2021.
Marcone, M. F., Wang, S., Albabish, W., Somnarain, D., Hill, A., and Nic, S. Diverse food-based applications of nuclear magnetic resonance (nmr) technology. Food Research International, 51(2):229-747 2013. doi: 10. 1016/j foodrrs..013..01.05
Simpson, J., Simpson, M. J., and Soong, R. En- vironmental nuclear magnetic resonance spectroscopy: An overview and a primer. Analytical Chemistry, 90(1):628-633 639, 2018. doi: 10.1021/acs.
7b03241. URL httph:ttp:///t http:::///oo.oo///////// orgg 102. accs. analchem. 7b03241. PMID: 29131590.
Michal, C. A. Low-cost low-field nmr and mri: Instru- mentation and applications. Journal of Magnetic Reso- nance, 319:106800, October 2020. doi: 10.1016/j.jmr. 2020.106800.
Soong, R., Simpson, M. J., and Simpson, A. J. In vivo nmr spectroscopy: toward real time monitoring of environ- mental stress. Magnetic Resonance in Chemistry, 53(9): 700-708, 2015. doi: 10. 10.1002/mrc.423
Moser, E., Laistler, E., Schmitt, F., and Kontaxis, G. Ultra- high field nmr and mri-the role of magnet technology to increase sensitivity and specificity. Frontiers in Physics, 5:33, 2017.
Sridharan, B., Mehta, S., Pathak, Y., and Priyakumar, U. D. Deep reinforcement learning for molccular inverse prob- lem of nuclear magnetic resonance spectra molecular structure. The Journal of Physical Chemistry Letlers, 13 (22):4924-49333 2022.
Nishimura, D. Principles of Magnetic Resonance Imag- ing. Stanford University, 1996. URL https://///// /books. google. com
Tayler, M. C., Theis, T., Sjolander, T. F,, Blanchard, W., Kentner, A., Pustelny, S., Pines, A., and Budker, D. In- vited review article: Instrumentation for nuclear magnetic resonance in zero and ultralow magnetic field. Review of Scientific Instruments, 88(9), 2017.
Norris, C. E., Bean, G. M., Cappellazzi, S. B., Cope, M., Greub, K. L., Liptzin, D., Rieke, E. L., Tracy, P. W., Mor- gan, C. L., and Honeycutt, C. W. Introducing the north american project to evaluate soil health measurements. Agronomy Journal, 12(():3195-321 2020.
2020.
van der Klink, J. and Brom, H. Nmr in metals, metal particles and metal cluster compounds. Progress in Nuclear Magnetic Resonance Spectroscopy, 36(2):89-201, 2000. ISSN 0079-6565. doi: http::/ww.aag//0ooo0oo00//666669
Perepelitsa, D. V. Johnson noise and shot noise. Dept. of Physics, MIT, 2006.
Probst, B. S., Toetzke, M., Kontoleon, A., Díaz Anadón, I.., Minx, C., Haya, B. K., Schneider, L., Trotter, P. A., West, T. A., Gill-Wiehl, A., et al. Systematic assessment of the achieved emission reductions of carbon crediting projects. Nature communications, 15(1):9562, 2024.
URL https://ww..cclenedlreettt cetance/artice///1///77755999999
Vyalikh, A., Münchgesang, W. and Velasco-Vélez. J.-J. De-
tection of electrocatalytical and -chemical processes by means of in situ flow nmr spectroscopy. Electrochemisttry Communications, 163: 107736, 2024. ISSN 1388-2481.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research. 22(268):1-8, 2021. URL http: //jmlr r. org/ wworg//aper///////// p2pprss/v2222-113333 html.
doi: http://dbbory///..oooccccccooomm URL httpps//www.ecienaaddeeeetttcc science/erttdd////////1332222229
10

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance mcasurement
Zhao, Y., Ding, Y., Lau, V., Man, C., Su, S., Xiao, L.., Leong,
A. T., and Wu, E. X. Whole-body magnetic resonance
imaging at 0.05 tesla. Science, 384(6696))aadm77666 2024.

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
A. Bloch equations
The Bloch equations describe the time evolution of magnetization observed in an NMR system. The relevant variables are:
Mz, My, M2, the components of the magnetization vector M.
Bg, By, Bz, the components of the external magnetic field B.
7, the nuclear spin's gyromagnetic ratio.
Ti, the longitudinal (spin-lattice) relaxation time.
T2, the transverse (spin-spin) relaxation time.
Mo. the equilibrium magnetization.
The corresponding system of coupled differential equations is:
d.M.
Ma
- -MMBB
(4)
T2
dM,, = rM, - -M,
My
(5)
dt
T2
dM, B, - M2
M2 - Mo
(6)
dt
Ti
B. Policy architecture and supporting information
Here, we outline the structuring of the Markov Decision Process (MDP) and the architecture of the Proximal Policy Optimization (PPO) agent as implemented in StableBaaeelines) (Raffin al., 2021). The parameters of the policy network and the training pipeline are contained in Algorithms and
In this work. structured our problem statement to employ a Markov Decision Process (MDP), standard formalism for decisionmaking under uncertainty which is frequently applied in the context of reinforcement learning. An MDP is defined by the tuple (S, A, s0, I',, with:
S: the set of possible states the system can occupy. A: the set of possible actions the agent can take. The system (S, A) is termed the environment, and it evolves with Iransition probabilities (P(s)s 64,aj a). which define the probability of that the environment evolves to state s after action taken state s'. In this paper, the transition probabilities effectively reduce delta functions when we disable Johnson-Nyquist noise, because the Bloch equations are deterministic.
II.a: the policy which determines the agent's behavior manipulate its environment. While deterministic policies possible, in our case the policy is randomly distributed and modeled with parameters 0 as IT... PP(ass) (o(a))). r(s, a): the reward function function which maps each state-action pair to scalar signal indicating the desirability of the state with respect to a specified metric. The agent's 's policy is trained to maximize this reward. Note that the reward functions used in this work implicitly depend on a but are specified only as functions of the state, r(s).
T [0, 1): the discount factor which is responsible for balancing immediate rewards against future rewards. multiplies contributions to the cumulative reward in each timestep, controlling the divergence of cumulative rewards over time and enabling the agent to drive the environment toward optimal outcomes on a long time horizon.
In many real-world settings the agent may not observe the full environment, but instead a state & which is subset of all available observables. This was the case for our setup, and explain which values are masked in the details to follow.
*We avoided the more standard notation Ih in the main body of the paper as avoid confusion with the angular shifts induced by ID NMR pulses, e.g. "pulsing with x" versus "pulsing by x."
12

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance measurement
At each timestep, the agent observes the current state St, chooses an action de ~ (-(e))))) according its policy, receives reward rz r(s,, az), and transitions to new state s++1 P((||ss, at). The goal of the agent is to learn parameter values that maximize the expected return, defined as the discounted sum of rewards:
20
Bs
Σ
r
The structure provided by an MDP allows for the development of gradient-based policy optimization methods such as the Proximal Policy Optimization (PPO) agent developed in this work. The agent's behavior is iteratively improved by updating a control policy which samples its actions probabilistically. The policy learned in a data-driven fashion through interactions with the environment (of which it observes a fraction, the state-space S) and estimating updates. As reward function will generally be noisy or sparse, can be difficult to estimate the payoff attributable a given state. In the PPO agent use in this analysis, we model it by value function Vø, which has learnable parameters d. To guide learning, it estimates the expected future return R starting from state s. Formally,
00
(s(s) R E
r
80 =
(8)
{=0
The parameters & are learned by minimizing the squared difference between the predicted value and the observed return:
E (Vo(s)) Rt) M
1
2
Lvalue
where the estimated return RG is defined as the value of the state at time t plus some estimate of advantage A.:
R= Vo((s) + A.
(10)
In our PPO implementation, the advantage A, is estimated from temporal differences 6, using Generalized Advantage Estimation (GAR) (Schulman al., 2018). This means that after collecting experience, we compute the following recursively,
using knowledge of future states:
5, = + (V) (st)
(11)
A. + TXA++1
(12)
where À is tunable parameter. These definitions enable the basic training pipeline outline in Algorithm 2, which learns á and @ as defined in Algorithm 1.
We list below the major similarities of the Stable-Baselines! PPO implementation with respect to the original work in (Schulman et al., 2017).
On-policy. Policy is updated based on the behavior of the very policy being trained, not on the behaviors of other policies. Samples are not reused as in off-policy approaches.
Clipping to constrain updates. Gradient clipping for actor by default, optional gradient clipping for value function (disabled by default). No trust region, no use of second-order derivatives or constraints to tune or stabilize.
Advantage estimation. By default, uses Generalized Advantage Estimation (GAE) (Schulman et al., 2018).
Actor-critic. Policy network ma is distinct and separate from value network Va-
Environment handling. Adapts to vectorized environments and allows for both continuous and discrete action spaces.
We emphasize that our policy network is a simple dense network, see Algorithm 1. We did not consider timeseries variants of this architecture which apply LSTMs or similar recurrent approaches condition model actions on the history of the state. We also note that the Bloch equations (written in Appendix A) are local in time t, that complete information about the state at any given time is sufficient to infer the next timestep (i.e. the dynamics are Hamiltonian). Within those equations, we effectively mask the Mo, Ti, T2, and ? values from the agent, so a non-recurrent architecture is not guaranteed to perfectly reconstruct or manipulate these dynamics. Nevertheless, it is interesting that our approach is performant as-is without such architectural considerations. Future work will investigate whether recurrent approaches improve the performance above this baseline.
13

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance mcasurement
Algorithm 1 Default PPO MIpPolicy Architecture in 3
1: Input:
Observation s R² (R 10 in the case of the toggle policy, to account for tc, tg, and the binary flag)
2: Actor network for magnetic field:
(o(s) <- MLPastorT (7 or 10, [64, 64),
MLP with 2 hidden layers of size 64 and tanh activations
log is trainable scalar, shared across actions
(a (s) = (s),
3: Actor network for binary toggle flag: ʃa(s) < MLPPato(10,, [64, 64),
MLP with 2 hidden layers of size 64 and tanh activations
(a(a = Bernoulli (f)()))
Equivalently, no (a = 1|s) = (fa(s)
4: Critic Network:
Vo(s) < MLPannti (7 10, [64, 64].
MLP with 2 hidden layers of size 64 and tanh activations
5: Initialization: Orthogonal
Gain 2 for all hidden layers
Gain 0.01 for actor output
Gain 1.0 for critic output
6: Stochastic action sampling: a a~ =o(als), a R2
14

---

Counting atoms faster: policy-based nuclear magnetic resonance pulse sequencing for atomic abundance mcasurement
Algorithm 2 PPO training pipeline in Stable- Basclines3, including hyperparameters used to produce the final results. Initialized policy and value function V 1: for each update iteration do
Require:
2: for each timestep = 0,..., T , do
> Collect experience over T Rsteps X Denvs transitions
3: Observe state 8t, take action de ~ (-(-1)))
4: Record (st, a, rt, log ((( (s))
5: end for
6: AT +-
7:
for each timestep = 1....,0 ,,.
Estimate advantages A, using GAE (Schulman et al., 2018)
8:
& +- rr (8)(1++)) (s)),,,
rT 0.99)
9:
A - 6; + IDA++1
(A 0.95)
10:
R₂ + A. + Vs((s)
> Compute returns
end for
12:
if advantage normalization enabled (True) then Ag f-
4, meen(A.)) si( id(
13:
end
14:
for K epochs do for each minibatch of size M = 64 do
15:
16:
(0) +- Ju [ [
ma(a, (8c)
b. Compute importance sampling ratio
17:
L CLIP (0) < min ppliicy
(e = 0.2)
b. Clip batch loss
18:
1 N7
Σ, LI
Compute policy loss
19:
if value clipping enabled (False) then Vpred +- Vols clip(VV(8t) Vola. - Evt) else Vpmed + (s():) end if
20:
21:
Livalue = 34 E. (Vpred Rie) Lentropy t 1
22:
Compute value loss
23:
N7 E, ((TT)(((t)
Compute entropy loss Total loss
24:
+ Lpolicy + Ci Lvalue + C2 C2
(c, = 0.5, C2 = 0.01)
25:
V Vel. - VoL. x min (l/V T,I,, max grad norm = 0.5)
D. Clip gradient norm Gradient step
[V.L[|
26:
0 0 VV (a=0.000 0.003)
27:
KT. <- E, (F) - log (0)
D Estimate KI. divergence
28:
if KLL 1.5 x (target KL o0) then break
> KL early stopping (disabled)
29:
end if
30:
end for
31:
end for
32:
end for
15


Reviews:
[{"review_id": "kApDDReQ86", "reviewer": "Reviewer_myr8", "summary": "The paper applies reinforcement learning to NMR data, to infer the elemental composition of a sample. They learn optimal policies to modulate NMR pulses for elemental abundance quantification. They present promising results on simulated data.\n\n## update after rebuttal\n\nI have raised my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback. Our responses are:\n\n1. **Regarding Weakness 1.** The reviewer states that the paper only presents application to simulated data. This is not totally accurate, for 2 reasons. **(I)** Firstly, in Experiment 1, the spin sets used in both training and evaluation are derived from genuine raw NMR data (a serial dilution of caffeine in water with a 90-degree pulse), and the Bloch model is a faithful representation of the underlying system. This is partly evidenced by the fact that, using the Bloch model, we can reproduce the empirically observed spectra exactly. **(II)** Secondly, we did investigate how to operate real-world NMR hardware with a control policy, but found it is not practical with current machines such as those used to produce the empirical data, as they are only operable with proprietary software, and purchasing a dedicated setup for modification would cost in the 6- to 7-figure range (USD). Such investment of funds and effort is challenging to justify without a proof-of-concept, which our results do provide. Furthermore, our results demonstrate concrete evidence that a high-fidelity, laboratory-grade NMR with expensive magnets, liquid nitrogen cooling, and similar luxury features can be potentially replaced with lower-quality, less expensive benchtop hardware. As a consequence of investigating this proof-of-concept, not only did we learn that the method has theoretical promise, but we also obtained evidence that the magnitude of investment required to explore hardware development could be much lower than this initial expectation. This finding is not obvious, and it represents an important motivating result which will be a precursor to a longer-term hardware development effort.\n\n2. **Regarding Weakness 2.** The reviewer comments that the machine learning content is weak, and that the paper is better suited to a venue focused on NMR. The ICML 2025 Call for Papers states the following regarding the Application-Driven Machine Learning track: “Application-Driven Machine Learning (innovative techniques, problems, and datasets that are of interest to the machine learning community and driven by the needs of end-users in applications such as healthcare, physical sciences, biosciences, social sciences, sustainability and climate, etc.)” As noted in the Related Work section, there is little existing work applying machine learning specifically to NMR pulse sequencing and atomic abundance measurement, so this work falls under the category of a “problem of interest” in the language of the Call. NMR is a common measurement technique applied within healthcare, analytical chemistry, and bioscience and is the basis of MRI technology. (The application specifically to sustainability is somewhat novel, though sustainability research has significant overlap with the aforementioned 3 fields.) While we agree that multiple field-specific venues could be acceptable for such a work, the key point we are trying to demonstrate and emphasize with this paper is that an application of machine learning unlocks capabilities which can impact many different areas of science in a unified fashion. In conjunction with the language of the Call, this widespread impact thanks to open-source frameworks and developments in reinforcement learning (including the Gymnasium framework and the PPO family of policy gradient methods used to produce our results) was our primary reasoning to submit to ICML 2025.\n3. **Regarding Question 1.** Given the diverse scientific backgrounds of potential readers for this work, we agree that it is appropriate to include additional definitions regarding MDPs, PPO-based agents, the designed objective functions, and potential alternatives. We have added an Appendix B where we provide these, and cite relevant related works.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "I am not sufficiently familiar with the literature to comment.", "essential_references_not_discussed": "I am not sufficiently familiar with the literature to comment.", "other_strengths_and_weaknesses": "The paper only presents application to simulated NMR data. It would be nice to have real data analyzed.\n\nThe machine learning content of the paper is weak, and it would seem that this paper is better suited to a specialized journal focusing on NMR.", "other_comments_or_suggestions": "I did not find any typos.", "questions_for_authors": "I understand the authors are using algorithms developed elsewhere. But given the readership of ICML I think readers would appreciate some more details were included about the algorithm used (things like PPO, MDP, etc.). Can the authors include a summarized self-contained description of these methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7NblenrLTK", "reviewer": "Reviewer_MkXf", "summary": "Counting atoms matters because it allows us to monitor and control the elemental makeup of materials, which is important for environmental sustainability applications. This paper presents a method that uses RL to optimise nuclear magnetic resonance (NMR) pulse sequences for faster and cheaper atomic abundance measurements.\n\n## Update after rebuttal\n\nI have increased my score by one point.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback. Our responses are:\n\n1. **Regarding Weakness 1:** lack of a baseline. In fact, we do use the standard 1D NMR pulse as a baseline, and we compare against it in all experiments. The baseline we use is the simplest and most common approach to analyze atomic abundances of solution-phase samples in an NMR. While more sophisticated pulse sequences are often performed in the case of chemical characterization (which investigates e.g. which molecular bonds are present and what their relative positions are), relative abundance quantification is typically done by (A) performing a 90-degree pulse (~0.5 ms), (B) waiting around a second (a few T2s) for the system to equilbrate, (C) taking a Fourier transform of the FID, and (D) integrating the NMR spectrum. However, as stated on line 194, the chirp in Experiment 1 achieves its maximum magnetization state in 16% of the time required for this 1D NMR pulse in step (A). This is a 6x speedup even before accounting for step (B), which conservatively takes 100x longer than step (A), if not longer [1]. The black line in Figure 1 demonstrates that no analogous shortcut is possible with the 1D NMR baseline, i.e. one would have to perform (A)-(D) to obtain an equivalent result. Similar acquisition time reductions have been achieved in data-driven MRI [2].\n2. **Regarding Question 1** and GRAPE [3] as a possible baseline. GRAPE requires specification of a target state. We are unaware of any standard state reported in the literature for atomic abundance measurement, but Footnote 1 on line 98 (page 2) shows how to derive one. The result seems paradoxical because it suggests performing no pulse at all. This seems strange, but is a consequence of differing assumptions and regimes of relevance. A zero pulse would work in theory, but requires different hardware which rapidly inserts and removes the sample from the magnetic field, so is not fully comparable to this approach. One can think of this as rapidly taking independent measurements of “signal plus background” and “only background” so that one can separate signal from the noise with a simple subtraction. Even if a zero pulse was found to perform better in simulation, engineering such hardware is vastly more challenging than a stationary approach; a relevant point of comparison is “magic angle spinning” in solid-state NMR where small cuvettes of material are spun at MHz frequencies [4]. We also emphasize that, besides performance considerations, in our approach there is no fundamental requirement of a faithful simulation of the underlying system. In a hardware instantiation, we can simply learn by reinforcement on real pre-characterized samples, which is not the case for GRAPE. The underlying reason for these differences is that GRAPE assumes the quantum mechanical regime is an appropriate description of the spins (i.e. samples contain only a few atoms or molecules so that quantum effects dominate), whereas RL on a Bloch simulator assumes a semi-classical description (i.e. samples contain multiple Avogadro’s numbers of atoms or molecules whose nuclear spins together behave classically). Areas of application appropriate to GRAPE include e.g. quantum computing [5,6] where individual molecules are manipulated into specific quantum states and then controlled as components of logic gates. Areas of application appropriate to our method include fields which commonly process macroscopic (gram-scale) samples appropriate to everyday contexts, such as food science, agriculture, and human health. However, the reviewer’s comment suggests a nontrivial avenue of further investigation which we had not realized earlier, and we are grateful for this insight. The top left of Figure 3 shows that a preferred direction is learned by the policy (black dots), providing an alternative target state for a GRAPE analysis than that derived in Footnote 1. It would be interesting to understand whether GRAPE with this target state is equivalent to our learned policy. We have added a sentence to the Future Work section incorporating this observation.\n\n[1] Joseph, David, and Christian Griesinger. \"Optimal control pulses for the 1.2-GHz (28.2-T) NMR spectrometers.\" Science Advances 9.45 (2023): eadj1133.\n\n[2] Ma, Dan, et al. Magnetic resonance fingerprinting. Nature, 495(7440):187–192, 2013. \n\n[3] Khaneja, Navin, et al. \"Optimal control of coupled spin dynamics: design of NMR pulse sequences by gradient ascent algorithms.\" Journal of magnetic resonance 172.2 (2005): 296-305.\n\n[4] Reif, Bernd, et al. \"Solid-state NMR spectroscopy.\" Nature Reviews Methods Primers 1.1 (2021): 2.\n\n[5] Jones, Jonathan A. \"Controlling NMR spin systems for quantum computation.\" Progress in Nuclear Magnetic Resonance Spectroscopy 140 (2024): 49-85.\n\n[6] Schulte-Herbrüggen, T., et al. \"Optimal control for generating quantum gates in open dissipative systems.\" Journal of Physics B: Atomic, Molecular and Optical Physics 44.15 (2011): 154013.", "claims_and_evidence": "There are 3 main claims (contributions) in the paper:\n1. \"Train policies by reinforcement which shape and sequence magnetic field pulses for use in a simplified (low-field) NMR spectroscope\". The method is claimed to be non-destructive and and generalisable to nuclear isotopes of many different elements.\n2. \"present a fast, robust simulator for generating large quantities of NMR spectroscopy data which is capable of reproducing the nuclear spin dynamics of many differ- ent samples in parallel when they are manipulated by an arbitrary magnetic field\"\n3. \"A novel method to manipulate these spins for atomic abundance measurement, by training three inter-operating agents to orchestrate an NMR pulse sequence.\"\n\nI think the experimental results back these claims in the sense that I can see there is an increased efficiency from metrics and graphs (e.g. model 2 reports 28% reduction in magnetisation, which I guess is good). However, as a non-expert with little to none knowledge in physics I can't assess how meaningful the experimental setup is and how significant the results are.", "methods_and_evaluation_criteria": "To my understanding, the simulator (based on Bloch equations) is calibrated using some real data (as described in Section 2.2). I have no idea if this is standard or if it makes sense for the things they are trying to demonstrate. \n\nThere aren't any baseline methods to compare the RL based approach to. Ideally, I'd like to see GRAPE (which this work is similar to), and potentially simpler baselines. E.g. would a random policy make sense here? Are there any \"heuristic\" policies that the field has developed previously?\n\nThe RL approach to policy learning itself is very standard - the authors use PPO.", "theoretical_claims": "there are no theoretical proofs in this work.", "experimental_designs_or_analyses": "I am not able to assess this.", "relation_to_broader_scientific_literature": "I cannot assess the impact to the broader scientific literature.", "essential_references_not_discussed": "I don't know this field.", "other_strengths_and_weaknesses": "On the ML side of things, the major weakness is lack of baselines.\n\nAs a non-expert I find it difficult to assess strengths and weaknesses more broadly.", "other_comments_or_suggestions": "NA", "questions_for_authors": "you mention GRAPE as a prior work related to what you do. Why is it not included as a baseline? More generally, what simpler baseline methods are there?\n\nMy review score is primarily due to the lack of baselines. I am not able to assess any of the physics, simulator setup etc.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "104j7sNXXx", "reviewer": "Reviewer_tJu3", "summary": "The paper presents another novel approach to performing NMR studies using reinforcement learning to modulate pulses. The authors study three different formulations: standard control of NMR pulse for spin alignment, relaxation of spins towards equilibrium, and allowing an agent to determine when relaxation (spoiling) should be allowed.\n\nThe trained agents could achieve the desired task efficiently in the experiments performed on simulated data for Carbon-13. This work generally represents a positive movement towards intelligent control of expensive procedures.\n\n## Update after rebuttal\n\nI was happy with the responses from the authors and feel the paper will make a great addition to the conference.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback. Our responses are:\n\n1. **Regarding Weaknesses** and the inclusion of additional real-world equipment deficiencies. We agree that the inclusion of thermal instrument noise is only part of the picture. A variety of limitations are worth modeling, including spin participation (how many spins experience the influence of the strong background $B_0$ field) and magnetic field inhomogeneities (due to e.g. a low-quality magnet) [1], as well as spatial perturbations of the sample during measurement. Our perspective is that a hardware implementation of the apparatus will be necessary in order to meaningfully generate profiles for these features in a Bloch simulator, but that producing a dataset of these effects as generated by real hardware is both feasible and a promising avenue of future research. \n2. **Regarding Question 1.** The main advantage of our approach over existing alternatives is the possibility of model-free development. We used a simulator as a proof-of-concept in this work to justify the construction of a bespoke hardware instantiation of the device and study its design parameters to reduce cost, but in a physical realization of the device we could simply learn a pulse by reinforcement on pre-characterized physical samples. Other approaches (including traditional NMR) require either analytical control or simulation of the underlying physics. In the Related Work, GRAPE is a result from optimal control, so is effectively the closest connection to MPC. However, GRAPE involves regression on a faithful model of the underlying system for a sample, which requires prior characterization and theoretical analysis of a compound of interest, and as such is only suited for sufficiently simple atomic or molecular systems that this characterization step can be performed. We have added a sentence explaining the connection between GRAPE and MPC in the Related Work and we welcome suggestions from the reviewer for relevant references to include which would further substantiate this connection.\n3. **Regarding Question 2.** As the focus of this work was on an initial demonstration of the application with comparison to an analytically-derived baseline, and PPO suffices for this purpose, we did not attempt to further optimize performance by introducing other approaches such as TRPO, SAC, or A2C/A3C. Per bullet point 3 in our response to Reviewer myr8, we have included an Appendix explaining PPO and its alternatives. The timesteps referred to in the paper are easily relatable to explicit time within the simulation. We measure T2 and T1 in real-world seconds and compare timesteps against these values where appropriate (e.g. Figure 2), but we did not do so everywhere because timesteps are the temporal unit most suitable to analyzing the optimization process. \n4. **Regarding Question 3.** Yes, techniques exist to extend Bloch simulations to account for a diverse range of molecular structure effects and it has been found that these are effective at improving MRI for molecularly complex samples such as brain tissue, e.g. [2]. Modification of the state descriptions and reward functions may be of interest for many reasons, an example is simultaneous tracking of mild impurities which one might reasonably treat as nuisance variables for the purpose of suppressing their radio-frequency response. In addition, whenever an analogous procedure exists in traditional physics-driven NMR to handle a specific such effect, we can optimize with respect to it in a similar fashion to the experiment in Section 3.1.2. Finally, we refer back to bullet point 2 above: wherever simulation-based methods fail, this method can in principle be applied directly on analytes of interest without an underlying physical representation of the system. This is not true of alternative approaches, though it comes with the caveats specified in Section 4 regarding distribution shift (lines 392-399, right side).\n\n[1] Ji, Yang, et al. \"Dynamic B0 field shimming for improving pseudo‐continuous arterial spin labeling at 7 T.\" Magnetic Resonance in Medicine 93.4 (2025): 1674-1689.\n\n[2] Singh, Munendra, et al. \"Bloch simulator–driven deep recurrent neural network for magnetization transfer contrast MR fingerprinting and CEST imaging.\" Magnetic resonance in medicine 90.4 (2023): 1518-1536.", "claims_and_evidence": "As far as the limitations of the evidence are clearly discussed, yes the claims are supported.", "methods_and_evaluation_criteria": "They make sense to the problem as a starting point from which more complex problems could be tackled.", "theoretical_claims": "There are no large theoretical claims made in the study.", "experimental_designs_or_analyses": "The experiments are designed well as a proof of concept.", "relation_to_broader_scientific_literature": "The use of agents to control and improve specific procedures is an emerging topic that will undoubtedly be of great interest to the community. From this perspective, the authors have made a positive contribution.", "essential_references_not_discussed": "I did not notice important missing work.", "other_strengths_and_weaknesses": "The idea is very interesting and well-formulated. Further, the decomposition of the problem into three stages is also well done. As the authors themselves mention, their data is limited not only in their use of simulation but also in that they study only a single species. Including real-world deficiencies of equipment and measurements would be a nice touch.", "other_comments_or_suggestions": "The paper is well structured, and there were no apparent problems with the writing or figures.", "questions_for_authors": "1. Perhaps the authors can explain the main benefits of the RL approach over a non-intelligent control. Are there other semi-automatic approaches to controlling the chirp signals that require no training but are still efficient, something from MPC, for example?\n2. What architectures were used in the studies, and how might this impact the results? Can time be introduced explicitly? \n3. Can the state descriptions and other aspects of the RL training be extended to include multi-atom effects?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tmQdRSwwz2", "reviewer": "Reviewer_PG82", "summary": "The paper uses reinforcement learning (Proximal Policy Optimization algorithm) to learn policies to modulate NMR pulses for rapid atomic abundance quantification. The authors developed three interacting agents to (1) align nuclear spins for measurement, (2) facilitate rapid relaxation to equilibrium, and (3) coordinate control between these processes to reduce overall measurement time. Experiments were conducted in a simulated NMR environment using low-magnetic-field carbon-13 quantification for cost-effective, portable analysis of foodstuffs and soils. The results indicate notable performance improvements compared to conventional NMR pulse sequences, and the study also discusses the limitations of this technique.\n\n## Update after rebuttal\n\nWhile the rebuttal has substantially improved my understanding and appreciation of the contributions, I think that the practical issues that are not addressed by the simulator, i.e. coupling, T2* effects and shielded vs deshielded protons, make the current approach less convincing to be used as a replacement for physical NMR machines. However, if used as a POC or to provide an estimate then it is reasonable. I look forward for future improvements where the most common practical issues are well captured by the simulator. I have increased my score by 1 point to Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback. Our responses are:\n\n1. **Regarding Claims and Methods.** The reviewer comments that the simulated approach is not satisfying. There is a precedent for the positive impact of simulated results in this field. Please refer to bullet point 4 in our response to Reviewer tJu3 as well as bullet point 1 in our response to Reviewer myr8.\n2. **Regarding Experimental Design 1.** To obtain an FID in the simulator, the individual spins $(\\gamma, \\text{T1}, \\text{T2})_i$ are separately simulated at first. The $(\\gamma, \\text{T1}, \\text{T2})_i$ for fixed $i$ is what we refer to as a “class of spin.” Subsequently, we perform a weighted sum (over index $i$) of each of these magnetizations in order to get the observed $M_x$ and $M_y$. The weights in this sum are derived by a simple least squares fit to the empirical data and their relative variances were low enough that we did not feel them important to consider as sources of uncertainty.\n3. **Regarding Experimental Design 2.** We agree that this is an important and subtle point. The agent does not have access to the net magnetization in the $B_0$ ($+z$) direction, and can only reasonably optimize to reduce transverse magnetization (in the $x$ and $y$ directions). The reviewer is correct that the state with $X$% of spins pointing in the $+z$ direction and $(1-X)$% of spins in the $-z$ direction parametrizes a line of maximum-reward states for the spoiling agent. If $X$ is nonzero, the spins in the $-z$ direction are unstable and precess back to $+z$, but this can cause longer-timescale oscillations in transverse magnetization than in the absence of active spoiling. This leads us to the rationale behind the second term in Equation (2) on lines 234-235: intermediate-timescale reductions in magnetization are insufficient, there must be a meaningful reduction in magnetization achieved by the end of the spoiling period.\n4. **Regarding Experimental Design 3.** The description remains valid. Relaxation times T1 and T2 are defined as constant rates of decay of transverse magnetization in the presence of a strong background field $B_0$ and the absence of a perturbing field $B_1$ (analogous to half-life in radioactivity). By actively spoiling with an applied field $B_1$, we force a faster rate of decay, but this does not change the underlying model description. It only means we can not use an RL-based pulse to measure relaxation times T1 or T2, but it is not necessary that we be able to do so.\n5. **Regarding Question 1** and spin-echo sequences. Our simulator can straightforwardly achieve a pi-pulse, this is just a matter of doubling the pulse time of the 90-degree pulse. It is intriguing to consider how introducing diverse pulse sequences into training (e.g. through teacher-forcing or by expanding exploration) could change the learned policy, especially given our response in Point 3. We are grateful for this suggestion, it is a very interesting avenue for future work and could unlock further performance benefits.", "claims_and_evidence": "The claims in the paper are sound but not entirely convincing to me. The proposed approaches are theoretically sound. However, if I understand correctly, experiments were conducted mostly in a simulated environment of NMR. While in this setting the paper has demonstrated significant gains in terms of speed, my main problem is that I am not entirely sure how well the approaches work in practice. Practical issues like coupling (acknowledged in the paper), T2* effects and shielded vs de-shielded protons are not captured by the simulator presented here.", "methods_and_evaluation_criteria": "The proposed methods and the evaluation criteria make sense to me. And indeed they are interesting. My only concern is that the paper seems to be incomplete as results mostly come from the simulated environment and there seems to be too little practical results other than the comparison of the chirping agents (sec 3.1) against collected 1D NMR pulses in section 2.2. However, these real data were used to train some of the agents.\n\nI would like to see results when applying spoiling and toggled chirping and spoiling to real NMR pulse sequencing processes, not in a simulated environment. However such results are not presented here.", "theoretical_claims": "The theoretical claims make sense to me. If backed up by experimental, non-simulated results this approach could be a game-changer in NMR pulse sequencing.", "experimental_designs_or_analyses": "I have a few questions regarding the experiments, in addition to the above comments:\n1. On lines 170-173: How are the weights used in the simulator? Could you elaborate the term \"class of spin\"?\n2. In section 3.2, does spoiling actually reset to the previous magnetization state for the sample even when net magnetization is 0? Let's say at 0 net magnetization, the number of protons whose direction is parallel to B0 is the same as the number of protons whose direction is anti-parallel to B0, would the spoiling agent create any zones where the protons are of the same polarity?\n3. How do tuples $(\\gamma, T_1, T_2)$ collected in section 2.2 remain valid for use in section 3.2 and 3.3 given that in 3.2 and 3.3 you have significantly cut down the relaxation time? Is there any guarantee that in practice this short amount of time is sufficient to achieve equilibrium after a period of high magnetization?", "relation_to_broader_scientific_literature": "If the proposed approaches work well in practice, it will be another example where reinforcement learning can substantially advance other fields.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "Could you give your opinion on modelling spin echo sequences (i.e. 180-degree pulses) in your simulator? Have you tried that?", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: **Summary.**

The topic of this work is to determine the elemental composition of a material using Nuclear Magnetic Resonance (henceforth NMR).

Specifically, the authors propose a reinforcement-learning based method (Proximal Policy Optimization) to modulate the NMR pulses for quantification of the element composition of a sample. 

Claims: their method is faster, non-destructive, more versatile and less costly than traditional ones.

They further demonstrate the applicability of their method through experiments.

**Important note.**

The authors mentioned that their paper should be assessed as an “application-driven” paper (special track).

**Strengths.**

* The method is superior to non-ML existing alternatives because it opens the possibility of model-free development. This is an important point for the “application-driven” paper track since “prior work outside the ML literature should be considered”.

**Weaknesses.**

* The ML content of the paper was deemed to be light (myr8). However, since this is an “Application-Driven ML Submission”, introducing a “problem of interest” to the ICML community can be a sufficient contribution.
* Understanding the paper is challenging for the non-domain-expert (MkXf).
* It was raised by PG82 that “coupling, T2* effects and shielded vs deshielded protons, make the current approach less convincing to be used as a replacement for physical NMR machines”. This demonstrates a weakness compared to alternative methods.

**Discussion and reviewer consensus.**

The author response was particularly effective at dispelling most of the reviewers’ concerns; everyone has increased their score, and there is a consensus that the paper could be accepted.

**Additional remarks.**

One concern was the lack of baselines in the experiments (MkXf); however, this concern was addressed: the authors use the standard 1D NMR pulse as a baseline in all their experiments, and argued that using so-called GRAPE was not appropriate here.

**Overall evaluation.**

If there is room, I think the paper would be a nice addition to the program.
Decision: Accept (poster)