Reinforcement Learning for Quantum Control under Physical Constraints
Authors: Jan Ole Ernst
Aniket Chatterjee
Tim Franzmeyer
Axel Kuhn
Venue: ICML 2025 poster (2025)
Keywords: reinforcement learning
quantum computing
quantum control
quantum dynamics
control theory
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We present an RL approach to improve quantum control of realistic quantum systems for a variety of applications by integrating physics informed constraints into the problem formulation.

Abstract: Quantum control is concerned with the realisation of desired dynamics in quantum systems, serving as a linchpin for advancing quantum technologies and fundamental research. 
Analytic approaches and standard optimisation algorithms do not yield satisfactory solutions for more complex quantum systems, and especially not for real world quantum systems which are open and noisy. 
We devise a physics-constrained Reinforcement Learning (RL) algorithm that restricts the space of possible solutions.
We incorporate priors about the desired time scales of the quantum state dynamics - as well as realistic control signal limitations - as constraints to the RL algorithm. 
These constraints improve solution quality and enhance computational scaleability. 
We evaluate our method on three broadly relevant quantum systems and incorporate real-world complications, arising from dissipation and control signal perturbations. 
We achieve both higher fidelities - which exceed 0.999 across all systems -  and better robustness to time-dependent perturbations and experimental imperfections than previous methods. 
Lastly, we demonstrate that incorporating multi-step feedback can yield solutions robust even to strong perturbations. Our implementation can be found at: https://github.com/jan-o-e/RL4qcWpc.

Reinforcement
Learning for Quantum Control under Physical Constraints
Jan Ole Ernst Aniket Chatterjee
Tim Franzmeyer *2 Axel Kuhn
Abstract
(laser pulses, microwaves etc.) to quantum system, to realise a desired time evolution (Glaser et al., 2015; Koch, 2016; Koch et al., 2022; Mahesh et al., 2022). Examples of such tasks include system initialisation, (quantum) state preparation, gate operation, state population transfer or state measurement. Quantum control enables performing such tasks with low error rates, which is particularly important for the realisation of fault tolerant quantum computing (Terhal, 2015). Isolated quantum systems exhibit unitary dynamics (i.e. reversible) which are comparatively easy to model for modest system sizes. Yet all real quantum systems are open, subject to some interaction with the environment and require the addition of irreversible non-unitary dynamics to realistically capture their evolution (Breuer & Petruccione, 2002).
Quantum control is concemed with the realisation of desired dynamics in quantum systems, serving a linchpin for advancing quantum technologies and fundamental research. Analytic approaches and standard optimisation algorithms do not yield satisfactory solutions for more complex quantum systems, and especially not for real world quan- tum systems which are open and noisy. We devise physics-constrained Reinforcement Learning (RL) algorithm that restricts the space of possible solutions. We incorporate priors about the desired time scales of the quantum state dynamics- as well as realistic control signal limitations - as con- straints to the RL algorithm. These constraints improve solution quality and enhance computa- tional scaleability. We evaluate our method on three broadly relevant quantum systems and in- corporate real-workd complications, arising from dissipation and control signal perturbations. We achieve both higher fidelities - which exceed 0.999 across all systems- - and better robustness to time-dependent perturbations and experimental imperfections than previous methods. Lastly, we demonstrate that incorporating multi-step feed- back can yield solutions robust even strong per- turbations. Our implementation can be found at petps::/githae com/ jan oeeeoooee////Aie
Motivated by such real-world experimental setups, tackle quantum control with physically realistic models. The com- bined unitary and non-unitary evolution of quantum systems is typically modelled by a master equation (Davies, 1974; Dirr et al., 2009), a first-order linear ODE also known as quantum I. Liouvillian or Lindbladian. Solving the master equation and controlling larger quantum systems computa- tionally expensive, growing quadratically with the quantum system size, limiting the use of standard optimisation meth- ods. While stochastic methods (Mølmer et al., 1993) can reduce the quadratic scaling to linear one, they require a large number of trajectories for high numerical accuracies. For large systems the only feasible option is to sample di- rectly from a quantum device. Experimental imperfections and noise arising from, e.g., signal distortion attenu- ation in optical and electronic setups, or due to inherent system imperfections (Burkard, 2009) pose additional challenges which existing approaches fail to address. In this work, we present a novel approach for controlling real- world open quantum systems, posing quantum control as Reinforcement Learning (RL) problem subject to physi- cal constraints. Specifically, we learn control policy that maximises the fidelity of the quantum control task, while re- moving control signals which result in overly fast quantum state dynamics from the space of possible solutions. A ma- jority of quantum control tasks, including those considered in this work, are concerned with adiabatically transferring population between quantum states (Král et al., 2007), such that the time evolution of the system is slow compared to the
1. Introduction
The optimal control of quantum systems important for enabling the development of quantum technologies such computing, sensing, and communication, and similarly plays an important role for quantum chemistry (Brif al., 2010) and solid state physics (Glaser et al., 2015). Quantum control requires the application of time-dependent signals
'Equal contribution CClarendon Laboratory. University of Ox- ford, United Kingdom Department of Engineering Science, Uni- versity of Oxford, United Kingdom. Correspondence to: Jan Ole Ernst ejannernst@@e physics.ox.ac.o
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reinforcement Learning for Quantum Control under Physical Constraints
System
Two Rydberg
Transmon
10
10-2
10
1.2
1.0
Update
Nonualise Time pes Update
Norton 19 Time pcy Update
1.0
10
-0.8
F
6 x 10-3.
0.6
-0.66
0.4
0.4
10
0.4
0.2
10-2
10-3
10-9
-0.0
0.0
-..0
0.25
0.50
0.75
1.00
2
6
0.25
0.50
0.75
1.00
Wmmx//100
Nawox 1000
Nywxx/
11000
max. left y-axis, solid red line) and normalised GPU time per RL. update (right y-axis, dotted blue line +lo) vs. permissible quantum solver steps Nmux Limiting Nmax- can be understood as placing an upper bound on the rate change of the quantum system evolution induced by the control signal- improves solution quality (lower infidelity) while also increasing computational efficiency (lower normalised time).
Figure 1: Minimum infidelity across hyperparameter combinations (1 (1-Fmax-
inverse energy gap of the states (E = hw) which facilitate the transfer. Quantum state dynamics which are fast induce leakage errors (decay outside of the desired quantum state space). Furthermore, oscillations in the quantum state populations severely limit the robustness of control solutions to any time-dependent noise real world experi- ments. In addition the hard constraint applied the space of possible solutions, cilitates smooth pulses and fixed amplitude endpoints with finite rise-time. Both characteristics are typically required for real-world implementation of quantum control signals. Lastly, we investigate using multi-step RL to address larger levels of system noise.
realising atomic quantum computers. Here, demonstrate robust control signals, even in the face of noise, unlike pre- vious approaches, and achieve higher fidelities at lower pulse energy than previous works. Lastly, we consider a superconducting Transmon quantum bit (qubit) (Egger et al., 8b) for qubit reset, for which we discover novel, physically-feasinn reset waveform which achieves an order of magnitude higher reset fidelity than any previous work.
introduce soft constraint that fa-
In conclusion, our work makes the following contributions:
1. We devise a computationally efficient RL implementa- tion that directly incorporates physical feasibility con- straints to enable discovery of experimentally realistic control signals. The improvements over standard RL algorithms are shown in Fig.
Incorporating physics- ahysies-based constraints into the RL problem only enhances solution quality but also significantly im- proves computational scalability. Control signals inducing fast quantum state dynamics require compute-heay. simu- lations and thereby longer computation times. Excluding these signals enables fast parallel optimisation of multiple hyperparameter configurations by removing simulation bot- tlenecks.
2. Fig. 1 demonstrates that our constraint the maximum number of simulation steps significantly improves com- putational scalability while simultancously improving solution quality.
Across three quantum systems, we outperform prior methods by achieving higher fidelities, lower pulse ener- gies, and greater robustness to time-dependent noise.
We validate approach three quantum control prob- lems. We begin with generalised electronic A system, com- mon in quantum dots, atoms, and circuit quantum electro- dynamics, revisiting a well known approach (Vitanov et al., 2017) for transferring population between states. Our im- plementation successfully learns realistic control signals with almost two orders of magnitude lower infidelity, and resilience dimendepndeent noise. We then explore the complex Rydberg gate (Lukin al., 2001), crucial for
2. Related Work
Several algorithms exist for devising optimal time- dependent control signals for quantum systems. Analytic methods like Lyapunov (Hou et al., 2012) are effective for small isolated systems but difficult to generalise to plex environments. Gradient-based methods which consider the piece-wise evolution of a quantum system, under piece-
2

---

Reinforcement Learning for Quantum Control under Physical Constraints
wise controls, such as Gradient Ascent Pulse Engineering (GRAPE) (Khaneja et al., 2005) Krotov (Reich et al., 2012) (which ensures monotonic convergence) work well on simple cost landscapes for a single objective with good initial guesses. Since gradients need to be evaluated exactly at each time-step for these methods, they are well suited to ideal simulations, but it is challenging to adapt them to closed-loop experimental optimisation where such gradi- ents can rarely be determined exactly and a large number of measurements are required. RL, to the contrary, requires no intermediate quantum state information and can perform optimisation with noisy samples. Other variations of global optimal control exist which consider the optimisation of the entire time dependent control signal at once (Giannelli et al., 2022b), which are prone to local minima. Global and local optimal control methods are explored in combination in (Go- et al., 2015). Methods which reduce the search space by decomposing the possible signal into basis (Caneva et al., 2011) are sensitive to basis choice (Pagano et al., 2024) and evolutionary algorithms (Brown et al., 2023) lack computa- tional scalability for larger systems or multiple objectives.
(Bukov et al., 2018; Metz & Bukov, 2023; Schäfer et al., 2020) to experimentally realistic systems, incorporating control signal noise into training as suggested by Schäfer et al. (2020). While Niu et al. (2019) find time-optimal gate sequences for superconducting qubits using trust re- gion policy-gradient methods, we advance this by enhancing computational scaleability by learning entire control signals in a single step and incorporating more complex noise mod- els. Our control pulses for typical A system go beyond existing work (Giannelli et al., 2022a; Norambuena et al., 2023) by incorporating realistic noise models and simultane- ous amplitude and frequency control to learn more optimal and realistic policies. Related work on optimising supercon- ducting qubit gates robust to noise with RL is presented in Nam Nguyen al. (2024), while enhanced superconducting qubit readout with RL is demonstrated in Chatterjee et al. (2025).
3. Background
3.1. Reinforcement Learning for Quantum Control
Machine learning has numerous applications in quantum science (Krenn et al., 2023). Numerous successful studies applied RL to the logical quantum ciruit level, to find op- timal error correction codes (Olle et al., 2024), real time error correction (Sivak et al., 2023) or circuit complication techniques (Van Der Linde et al., 2023; Quetschlich et al., 2023). In contrast to these works, our work focusses on lower-level pulse-level control.
Reinforcement Learning (RL) is a framework where an agent learns to make decisions by interacting with an envi- ronment to achieve a specific goal (Sutton & Barto, 1999). In quantum control, RL can be used to find the control actions a(t) that steer a quantum system toward target state Pdes- The key components in this RL setup are the state 8;, which is given as the density matrix p(t) of the quantum system, the control action a, applied to the sys- tem, a scalar reward r;, derived largely from the closeness of the system its target state and the policy x that maps states to actions. The objective is learn policy x² that maximises the expected cumulative expected reward, i.e.
We review prior ML. work, distinguishing between real de- vice sampling and numerical simulations. Baum et al. (2021)
devised an optimal gate set on a superconducting IBM quan- tum device. Reuer et al. (2023) and Porotti et al. (2022) use measurements and feedback to prepare quantum states, but generalisation to unseen environments is difficult. A model-based Hamiltonian learning approach was applied in Khalid et al. (2023), which provided important insights into sample efficiency. Although we focus on realistic quan- tum simulations, our RI. algorithm can be readily applied to physical experiments by replacing simulations with real- device sampling.
' E max, E Ei=077 Tt
Continuous Bandit Setting In the continuous bandit set- ting, the RL problem reduced to single-step episode. The agent selects one action a(t) in a continuous space [-1, 1]4 where d is the action dimension representing the number of discrete action samples in time, aiming to maximise the immediate reward based on the fidelity with respect to the target state. Specifically, the optimal action a* is given as a E arg max, E [r(a)], where r(a) is the reward obtained by applying action a.
Several studies have explored reinforcement learning (RI.) for controlling simulated quantum systems. While RL has been applied to discrete action space control (Paparelle et al., 2020; An et al., 2021; Zhang et al., 2019), these methods struggle in real-world settings with analog signals exhibit- ing finite response time' and in more complex systems. We extend prior work on controlling many-body systems
3.1.1. QUANTUM DYNAMICS SIMULATION
Assessing the quality of action att) in quantum control involves computing the fidelity (App. (12)), which mea- sures the overlap between the evolved quantum state and the target state. The state evolution under a(t) is obtained by numerically solving the master equation (App. (11)) for p()). the generalised quantum state at time t. For a de-
A particular limitation is the finite rise and fall time of elec- tronic or optical signals, which refers to the time required transi- tion from zero maximum amplitude (or vice versa), typically the order of nanoseconds or greater.
3

---

Reinforcement Learning for Quantum Control under Physical Constraints
tailed background on quantum dynamics and control, see App. Sec. 3. The state evolution is typically simulated using adaptive step-size solvers that implement higher-order Runge-Kur -utta methods (Hairer et al., 1993), which dynam- ically adjust their internal time steps based on local error estimates. If the error exceeds the numerical tolerance, the solver reduces its internal time step; if the error is suffi- ciently small, the time step is increased to enhance compu- tational efficiency. Therefore, control signals that lead to slower quantum state dynamics allow the adaptive solver to use larger time steps. Hence, slower quantum state dynam- ics require fewer solver steps and less computation time.
lation time across all hyper-parameter configurations, the slowest simulation among all learned policies determines the speed of the entire loop. We mitigate this bottleneck with a constrained RL algorithm that solves the quantum control problem subject to the condition that the required number of quantum simulation steps does not exceed a cho- sen threshold Nmax Formally, this constrained RL problem, for which the quantum simulation can be executed in fewer than Nma steps, is defined as:
T Σ 0=
T²" Emax E
rt
for a E т Naim(() < N
maaa
Sim
4. Methods
where т is the policy, re is the reward at time t, and NNim (a) is the number of solver steps required for conducting the quantum simulation for an action a sampled from policy Implementing this constrained RL algorithm prevents bottlenecks as it ensures that all simulations within the vec- torised run are completed within fixed maximal time frame. This approach enables efficient hyper-parameter exploration with minimal computational cost. The constraint, though seemingly restrictive, is physically motivated for adiabatic population transfer between quantum states (Král al., 2007). In adiabatic processes, the system evolves slowly relative to the inverse energy gap, requiring fewer solver steps. The maximal effective Rabi frequency, Reft = T"' sets a lower bound for Nmax via the adiabaticity condition Deit 5e >> 1 (Král et al., 2007). In practice, we increment Nmax long a significant decrease in infidelity is observed (see Fig. I for infidelities at different maximum solver steps for different quantum systems).
4.1. Coystas-oonstrained Reinforcement Learning
In practice, applying RL to efficiently find high-fidelity quantum control solutions requires restricting the space of possible solutions. First, we constrain signal bandwidth, and signal area, reflecting the limited instantancous band- width of electronics and signal components in experiments, and limitations in available signal power and duration. We implement these constraints via the reward function cf. Eq. 2 and Sec. 4.2, similar to the Lagrange Multiplier technique introduced in (Bhatnagar& & Lakshmanan, 2012).
Simulating the quantum system to compute the reward for the RL agent at each learning step, as detailed in Sec. 3.1.1, is computationally expensive. For complex quantum sys- tems and sub-optimal actions, this simulation can require an extremely large number of solver steps, far exceeding the computational time needed to update the RL agent. There- fore, secondly, constrain the policy to solutions that can be simulated within predefined number of maximum numerical solver steps, Nma. This constraint incorporates priors about the physical solution time scales into the algo- rithm, which incentivises adiabatic quantum state dynamics, yielding robust and interpretable solutions (cf. App. Fig. 9). Additionally, this constraint incentivises smoothness (requiring lower bandwidth) of signals, as smooth signals generally require fewer solver steps.
To summarise, the constrained RI. approach not only im- proves computational efficiency but also promotes the se- lection of physically realistic control signals (cf. App. Fig. 14). By enforeing these constraints, we achieve more in- terpretable quantum state dynamics (cf. App. Fig. 9) and prioritise adiabatic solutions, leading to experimentally fea- sible, robust and high fidelity control techniques.
4.2. Reward Shaping
RL optimisation algorithms are often sensitive to the choice of
We parametrise the control signal as combination of time- dependent amplitudes Ω; and time-dependent frequencies AS and introduce smoothness constraints that facilitate effi- cient learning and further improve computational efficiency. Smoother waveforms are easier to implement experimen- tally, offer clearer interpretation of the optimal quantum state evolution, and significantly speed up simulation times by reducing the number of required solver steps. To facili- tate smooth signal discovery, we apply a Gaussian convolu- tion filter to our control signal with a standard deviation to (cf. App. (20)) before simulating the quantum state dynam- ics which improves learning dynamics by favouring slower solution dynamics (an ablation over this is found in App.
(Henderson et al., 2018), necessitating hyper-parameter space searching to find optimal policies. We address this challenge by synchronously optimising con- trol policies for up to 1024 RL agents in parallel on a single GPU device, by implementing both the quantum solver and the RL algorithm using JAX (Bradbury et al., 2018), which features inst-in-time compilation and automatic differenti- ation. This allows for the compilation of the parallelised training and simulation loop end-to-end. However, in this synchronised parallel setup, the quantum simulation time needed per step is governed by the maximum quantum simu-
4

---

Reinforcement Learning for Quantum Control under Physical Constraints
Sec. G.4 Fig. 13). The reward function contains auxiliary smoothing penalties and is defined as
for superconducting Transmon qubits under bandwidth con- straints, essential for fast quantum circuit execution and scaling the volume of quantum gates that can be performed. Here, we discover a novel, pyysically-feasible reset wave- form which achieves an order of magnitude higher reset fidelity than any previous work. Fig. demonstrates the efficacy of our proposed method in finding higher-fideline solutions while reducing computational demand. Although in this work we simulate realistic and noisy quantum sys- tems, we detail in App. Sec. F how our setup can be applied to real systems directly.
La- - wF wo log (1 F) WoRReLU
( S Sea
1)
W..ReLU E s(Ai) Σ Sexe
1) W Abnen
Σ ((2,))
The first and most important reward-function term incen- tives high fidelity F = F(pin: Pdes) with respect to the desired final state pdes. This fidelity reward scales with the logarithm infidelity log (1 - F)). Next we define smooth- ness penalties, where (-) defines the ReLu function: ReLu(z)= 0f < 0 || ReLu(x)
x i >= 0. The smoothness S is compared to that of a reference signal Sbase (cf. App. Sec.G.4 Fig. 15 for a definition of and ab- lation over different smoothing functions). We introduce a smoothness penalty weighted by small coefficients Wa, wp, to balance fidelity, interpretability, and computational effi- ciency. Fig. 13 shows an ablation over various smoothing penalties. Contrary to the A system and Rydberg atom, the Transmon favours stronger smoothing penalties showing that our approach is adaptable to a wide variety of physical problem settings. Larger values of f, and W's Wg also re- duce the maximum required solver steps and thereby further enhances computational scaleability. The ability to achieve high-fidelity solutions across all environments at larger con- volution standard deviations, tor also demonstrates that we can find optimal signals compatible with realistic electronic control systems with limited instantaneous bandwidth (cf. App. Fig. 14).
5.0. EXPERIMENTAL IMPLEMENTATION
Constrained RL Implementation To enforce the con- straint Nim (a) < Nmmax on actions a sampled by policies I in the bandit setting, we assign a penalty reward ppnallyy In the bandit setting, Tppeally is assigned to any policy where Nsim (a) >= N and the value is chosen to be lower than any other possible reward in the environment, ensuring that the optimal policy cannot include states violating the constraint (Altman, 2021). This approach can be easily ex- tended to multi-step settings when the bounds of the reward function are known, which is the case here (Altman, 2021). The final reward function is then defined as
Penalty
if Nsm (a) >= Sim
L
(3)
Le
else
where La is defined in (2). Further details on the implemen- tation, including all relevant libraries, is found in App. Sec. G.I. All the code is fully open source at Ref. Ernst et al. (2025)
The final reward term penalises solutions with large pulse area (cf. App. Sec. C Fig. for an ablation over different area penalties for the A system), we set W = 0 for the Rydberg and Transmon problem settings. We introduce additional thysics-iiformer constraints which are problem specific and defined in App. Sec. C and App. Sec. D.
5.1. Population Transfer in Multi-level A System
Controlling quantum dynamics in multilevel systems is cru- cial for quantum information processing and relevant to solid-state physics and chemistry (Bergmann et al., 2019; Vitanov et al., 2017). We study a common experimental setup (Vitanov et al., 2017), the A system, featuring multiple ground and excited states, with the latter subject to sponta- neous decay. We consider two time-dependent control sig- nals with amplitudes Ωg Ωp which couple two electronic states with relative time-dependemt frequency detunings Ap and As (cf. App. Sec. C for more details). These four pa- rameters define the control fields in (10). While analytically optimal pulses exist for idealised three-level systems (Kuk- linski et al., 1989; Vasilev et al., 2009), we include excited state dissipation, parametrised by rate I', and an additional excited state detuned positively by Ax requiring cross-talk suppression (cf. App. Sec. C for details). This repre- sents a common physical configuration, describing nitrogen vacancy centres (Balasubramanian et al., 2009), quantum dots (Economou et al., 2012), and single atoms (Ernst et al., 2023). We present and benchmark results on optimising
5. Experiments
Overview We primarily focus on the continuous bandit RL setting, with supplementary experiments showing multi- step RL outperforms bandit methods under strong signal perturbations. We conduct experiments on three critical quantum control tasks relevant to quantum information pro- cessing. First, we address population transfer in multi-level systems, relevant to quantum chemistry and solid state physics, where we achieve high-fidelity population transfer in spite of dissipation and cross-talk. We explicitly show the effect of our physics driven constraints and the superior- ity of PPO over other RL alternatives in Fig. 2. Secondly, we optimise Rydberg gates in neutral atom quantum de- vices, focusing on enhancing gate fidelities and robustness to time-dependent noise, which is crucial for scalable quan- tum computing. Thirdly, we develop efficient reset protocols
5

---

Reinforcement Learning for Quantum Control under Physical Constraints
Comparison of Baselines: Mean Infidelity
F vs Update Steps
100
10
10-3
10-3
5000
10000
15000
20000
25000
30000
RL Update Steps
Constrained PPO
Constrained TD3
Coustrained DDPG
Vanilla PPO
Vanilla TD3
Vanilla DDPG
We compare three model-free RI. algorithms - PPO (Schulman et al., 2017), DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) in the noise-free A-system setting. Each algorithm run on the same Nvidia P100 GPU, using a single set of optimised hyper-pane parameeareeers per algorithm, with a common batch size and results averaged over two seeds (one s.d. shaded) for the same number of total steps. We compare standard (vanilla) versions of the algorithms which use T. F, to constrained formulation, using the reward defined in Eq. and incorporating step-size penalties (ef. Eq. 3). The constrained variants consistently outperform their vanilla counterparts, demonstrating the effectiveness of the constrained problem formulation and physically motivated loss function. Our implementation of PPO achieves the best performance, surpassing DDPG and TD3 by over an order of magnitude mean fidelity. Furthermore, PPO reaches mean fidelities exceeding 0.99 up to 100x faster than the alternative RL approaches.
Figure 2:
population transfer from one ground state another, fixing == 1, Qmaz 30 and Ax 100.
time with mean a and standard deviation (for details cf. App. Sec. G.5). Such noise typically arises from a variety of imperfections in the signal chain, as well quantum system level noise, such as magnetic field fluctuation or motion. Using unbiased (μr = 0) noise with various standard deviations exemplifies good robustness to low noise levels as shown in Fig (cf. App. Sec. C) where we attain > 0.99 mean fidelity for on a사 = 0.1. Further increasing leads significantly reduced population transfer fidelities which we address with multi-step RL in Sec. 5,4. Solutions for larger variety of system parameters and an extension to partial state transfer are shown in App. Sec. C.
We observe in Tab. that the fidelities F achieved in 4- level A system are significantly higher than state of the art and also more robust across different random initial seeds, highlighting the superiority of RL over methods which di- rectly differentiate the control action with respect to the fi- delity. We further find that the learned pulses are physically viable, while prior work (Giannelli al., 2022b; Brown et al., 2021) found infeasible solutions, which exhibit non- zero amplitudes at the start or end or have instantaneous parameter changes which cannot be realised on bandwidth limited hardware. Morcover, we note that our implemen- tation of PPO achieves the best fidelity > 0.999 and also achieves > 0.99 fidelity 100x times faster than TD3 (Fuji- moto et al., 2018) DDPG (Lillicrap et al., 2016) as shown in Fig. 2. Sweeping WA cf. (3) we find signals which have pulse extremely low pulse areas which approach the lower bound quoted in (Norambuena et al., 2023) (cf. Fig. 7 in the App.), implying low signal energy requirements. Example signals differ significantly for different pulse area penalties which is shown in Fig.
5.2. Rydberg Gates
Neutral atom quantum devices have shown promise for real- ising scalable, logical quantum computing (Bluvstein et al., 2023). The realisation of quantum computing requires a two- qubit gate (Nielsen & Chuang, 2010) which relies on the interaction of multiple atomic qubits which are brought in relative proximity detailed description of the Hamiltonian is provided in App. Sec. D) and addressed with laser beams. We consider optimisation of the Rydberg gate (Lukin et al., 2001) under realistic experimental conditions and signal perturbations.
Random fluctuations noise of either signal Qgy As/P are not detrimental to the overall fidelity. We implement an Uhleeen- noise process for both AspP and Asyr& a noise model which creates continuous noise 1/2 in
We consider the most widespread implementation of a Ryd- berg C-Z gate (a single photon Rydberg gate (Levine et al.,
6

---

Reinforcement Learning for Quantum Control under Physical Constraints
Method
F
Notes
Optimal Control (Giannelli et al., 2022b)
0.890
0.064
Using BFGS (Fletcher, 1987), n iter 1000.
Krotov (Goerz et al., 2019)
0.99 + 0.001
n
Analytic (Vasilev et al., 2009)
0.901
No uncertainty as analytic solution.
Reinforce (Brown et al., 2021)
0.930 0.034
Not exp. feasible
PINN (Norambuena et al., 2023)
0.83
No code available to benchmark
Vanilla DDPG (Lillicrap al., 2016)
0.985 0.004
Vanilla implies reward L
Vanilla TD3 (Fujimoto al., 2018)
0.625 = 0.01
Vanilla PPO (Schulman al., 2017)
0.989 + 0.0002
PPO (this work)
0.999 + 0.0003
Table I: We benchmark different methods for optimising coherent quantum population transfer in a multilevel A system. Averaged over 32 random sceds, our method achieves significantly higher than prior work with reduced sensitivity to the initial seed, yielding experimentally feasible controls. Timing comparison provided in App. Sec. G.3.
Amplitade Frequency (MHz)
Amplitudes and Detunings wA
Frequency (MIl))
Amplitudes and Detunings WA
30-
10.0
fts
Q2
20
7.5
A5
Ap
10
5.0
Qs
f2
2.5
Amplitade
A
-10-
Ap
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.70
0.75
0.80
0.85
0.90
0.95
1.00
Time (nt)
Time (ps)
Figure 3: Shown are example control signals generated for different pulse area penalties. For WA 0 (left), the algorithm seeks to maximise Ω at all times after fast rise and compensates cross-talk with frequency chirping. For w = 1 (right), we plot only the time interval [0.7, 1]. as the pulse amplitudes are zero otherwise. We show that the we discover pulses which reminisce of two interlcaved Gaussians, but exhibit non zero two-photon detuning As = 0 to cancel cross-talk (cf. App. Sec. C), which differs from the original experimental proposal for coherent population transfer (Vitanov al., 2017).
2019; Jandura & Pupillo, 2022) with a single pulse of am- plitude Qp and time-dependent frequency Ap which has known solutions. This is compared to the two-photon Ryd- berg C-Z gate which uses two time-dependent signals with amplitudes 2p, 22 and frequencies A A, As (akin to the A system). The single photon Rydberg gate is vulnerable to dimendependent noise which motivates the determination of an optimal pulse sequence for the two-photon Rydherg gate which exhibits superior robustness external perturbations (cf. App D). Finding optimal protocols which simultane- ously optimise both amplitude and frequency of Pump and Stokes beams is challenging since there exists a large num- ber of possible control signals in a large Hilbert space. A similar setup was addressed in Goerz al. (2014), however this did explicitly consider the decay of the Rydberg state and the optimised signals are extremely challenging to realise experimentally. Compared to Saffman et al. (2020)
delity F 0.9996 than the analytic solution F 0.99. well as the numerical solution F 0.997 and faster O. 0.25ps compared to the 1,us numerical solution. What is also remarkable is that for moderate levels of unbiased time- dependent amplitude and frequency noise (cf. App. Sec. D) we observe F > 0.999. Compared to Sun (2023), who ne- glect intermediate state decay, we achieve similar fidelities but with order of magnitude lower peak Rabi frequencies which implies lower laser power requirements. Moreover, we implement direct C-Z gate which does not require any additional single qubit rotations. We directly differ- entiated the input action with respect to the fidelity with BFGS (Fletcher, 1987) method over 1000 iterations and for 32 random initial seeds and achieved mean fidelity of 0.914 + 0.0742 (one s.d.) showing the superiority of RL to reliably achieve high fidelity solutions. The enhanced computational scaleability offered by our implementation could be used to optimise multi qubit gates with k > 2
we find a solution (cf. App. Fig. 10) which is higher fi-

---

Reinforcement Learning for Quantum Control under Physical Constraints
Amplitude / FFoquency (A.U.)
Amplitude and Detuning
Transmon State Population
10a
0.5
RL Amplitude
10-1
RL Detuning
0.0
Ana. Detuning
V 10-2
Ana. Fit to Amplitude
Ideal Evolution
0.5
RL Evolution
10-3
Smoothed Square Pulse
1.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time (1U))
x10
Time (u8)
x10
Figure 4: Optimal Waveform for Transmon Reset (left) discovered by RL and corresponding state evolution (right). The RL waveform (solid lines) amplitude evolution reminisces of a square-top Gaussian, with a smooth Hcaviside-detuning that accounts for dimedeependent frequency shifts. Equivalent reset performance is found by fitting a Hcaviside- and Gaussian square amplitude waveform (dashed lines), simplifying experimental calibration. Our approach shows reset errors of 0.03% matching the performance under an experimentally unrealistic ideal square pulse, and showing an order of magnitude improvement over smoothed square pulse.
reset
which are also robust.
frequency shift. While optimal control (Gautier et al., 2024) has been applied to the task of reset pulse optimisation, min- imal bandwidth constraints implied that no novel waveforms were found for improving the reset transition in a simple and realistic experiment. Using BFGS with direct differentiation of the input signal failed to optimise multi multi-objecttve reward functions satisfy realistic signal constraints. When op- timising solely for fidelity, it remained slow and prone to local minima due to the large search space of non- non-smooth actions. Similarly, we were unable to attain any reasonable learning with alternative RL algorithms other than PPO.
5.3. Transmon Reset
Superconducting quantum bits (qubits) have played cen- tral role in quantum computing breakthroughs, including the demonstration of quantum supremacy (Arute et al., 2019) as well as the suppression of errors with the sur- face code (Acharya et al., 2023). The Transmon (Koch et al., 2007), a widely used superconducting qubit, operates within its two lowest energy levels to form a qubit sub- space. Recent advances have extended Transmon lifetimes beyond 0.5 ms (Wang al., 2022), enabling longer quantum circuits and the implementation of егrоr correction codes. To maximise circuit operations within the qubit's lifetime, Transmons must be reset efficiently with high fidelity.
We apply our RL method optimise the Transmon re- set waveform under bandwidth constraints imposed by Gaussian-smookie smoothing (for further details cf. App. Sec. G.4), Considering state of the art parameters, given in the IBMQ experiment - qubit lifetime Zi of 500μs we find that our RL approach achieves 0.9997 fidelity under realis- tic bandwidth constraints shown in Fig. 4 (cf. App. E for further implementation details). This compared with a perfect square pulse which is not experimentally realis- tic without any smoothing, and a calibrated square pulse with smoothing which represents prior work (Egger et al., 201 8a). The RL waveform matches the theoretical optimal fidelity of the perfect square pulse and improves the fidelity of waveform used in prior work (Egger et al., 2018a) by an order of magnitude. In App. Sec. E we explicitly compare the results with the parameters used in Egger et al. (201 8a), and find that the RL-discovered reset waveform achieves the fidelity 0.997 of the ideal square pulse compared to the measured fidelity of 0.983. A fitted Heaviside detun- ing function from the RL-discovered waveform corrects the drive-induced Stark shift, simplifying experimental calibra-
Two main reset techniques exist: conditional reset (Ristè al., 2012), which follows state measurement, and un- conditional reset (Magnard et al., 2018), which is faster and more robust. We focus on optimising waveforms for unconditional reset (cf. App. Sec. E for further details). The reset rate is proportional to drive strength, theoretically favouring high-amplitude square pulses for maximum fi- delity. However, a drive-induced Stark shift alters the Trans- mon's resonance frequencies (Zeytinoğlu et al., 2015). In ideal conditions, square pulse with calibrated frequency can counter this shift. IBM demonstrated this approach experimentally, achieving 0.983 fidelity, while simulations under ideal conditions reached 0.996 fidelity (Egger et al., 20.8a). This mismatch could be explained by experimen- tally realistic bandwidth constraints as square pulses have finite rise and fall time, which induces a time-dependent
8

---

Reinforcement Learning for Quantum Control under Physical Constraints
tion, which we dub Heariside-Correeted Gaussian Square (HCGS) and explain further in App. Sec. E. Further results and extensions are provided in App. Sec. E.
6. Conclusion
In this work, we introduced a novel reinforcement learn- ing implementation for controlling open quantum systems by formulating quantum control as a constrained RL prob- lem. By integrating physics- based constraints that exclude control signals inducing overly fast quantum dynamics and enforcing smooth pulses with finite rise-time, we enhanced both the quality of control solutions and computational scal- ability. Our approach outperformed existing RL and non- RI methods on three key quantum control tasks, achieving higher fidelities and increased robustness to time-dependent noise. Achieving > 0.999 fidelity for the environments is significant insofar as this is often quoted the thresh- old for error free quantum computing with error correction (Gottesman, 2002; Fowler et al., 2012). We wish to high- light here, that especially for the Transmon qubit, we find novel waveforms that can be described with smooth func- tional parametrisation and realised with standard hardware. We are actively working on verifying the quality of our found solutions on physical devices. For future work, envision extending our approach to more complex quantum systems as required for fault tolerant quantum computing with error correction, this includes multi-qubit systems and higher-dimensional state spaces. Exploring adaptive con- straint mechanisms that adjust during the learning process could further improve performance in these large systems. Additionally, future work would extend this to quantum control tasks which require multiple sequential quantum gates or other concatenated control operations. Develop- ing generalised quantum control policies that incorporate system-dependeet observations during learning would en- able a single policy to adapt across diverse qubits and de- vices, significantly enhancing scalability. Validating these solutions real quantum hardware would accelerate the practical advancement of quantum technologies. We also wish to highlight that the methods presented here can ad- dress a variety of complex control tasks in real-world phys- ical systems, even in the presence of noise, imperfections, and parameter drifts.
5.4. Multi-Step Reinforcement Learning
Dandit RL
Multi-Step RI. Stepe, Js. 9)
Handit RI.
Multi-Step Stepe, Ho
10
10
20
30
40
50
60
RL Update Step
Figure 5: Comparison of mean infidelity for different values of Ho for bandit RL and multi-step RL. We observed 2x reduction in infidelity for larger noise bias Pe by using multi-step RL over the bandit setting.
We study the effectiveness of multi-step reinforcement learn ing strategies in achieving high fidelity control solutions under adverse noise conditions. Feedback on nanosecond timescales has been demonstrated experimentally (Álvarez al., 2022: Koch al., 2010), supporting this approach. This feedback can be realised by measuring classical signal noise without affecting quantum coherence. For example, in atomic quantum systems, laser intensity I can be monitored separately from the quantum system, as I x |S2|2 Changes in I directly modulate f2gpp and thereby provide feedback.
In multi-step RL, the agent aims to maximise cumula- tive rewards over multiple steps, unlike the bandit setting where actions are independent. In setup, at the start of each episode, a parameter μn is sampled uniformly from [_ to initialise an Smnttin-Uhlenbeed noise pro- cess (see App. Sec. G.5 (25)). The agent's 's control signal a, = (t) (amplitudes only) is affected by this noise, re- sulting in 2 = Ω + vt. In bandit RL, the agent does not observe the noise 24 and selects the action in one step. Conversely, in multi-step RL, each episode divided into four sections of 8 action samples corresponding to 0.25ps each. The agent initially observes O, O but receives the value of μe at times t = 0.25, 0.5, and 0.75ps (fur- ther implementation details are given in App. Sec. A.2). Fig. 5 illustrates that multi-step RL outperforms the bandit approach, especially as Pe increases beyond 10.
While our physics- conssriscon RL implemen- tation enhances computational efficiency and solution qual- ity, it may limit the exploration of control strategies that involve very fast and non-adiabee con-adiabatic quantum dynamics. The method's effectiveness also relies on accurate modelling of quantum systems, so models would first have to be estab- lished for black box systems more complicated real world devices. Although we address certain types of noise and perturbations, fully accounting for all experimental imper- fections is an area for future work and we could consider grey-box models of real devices.
Limitations

---

Reinforcement Learning for Quantum Control under Physical Constraints
Acknowledgements
Baum, Y., Amico, M., Howell, S., Hush, M., Liuzzi, M., Mundada, P., Merkh, T., Carvalho, A. R.,, and Biercuk, M. Experimental deep reinforcement learning for error- robust gate-set design on a superconducting quantum computer. PRX Quantum, 2(4), December 2021. ISSN 26913399. doi: 10. 10.110//PXQQaaum22200000 2105.01079 Publisher: American Physical Society.
JOE is grateful for helpful discussions with Daniel Puzzuoli and Ronan Gautier on GPU based quantum simulations and the authors are grateful to Christiane Koch for providing valuable feedback on this manuscript.
arXiv:
The authors would also like to acknowledge the use of the University of Oxford Advanced Research Computing (ARC) facility in carrying out this work (Richards, 2015). JOE & AK acknowledge financial support provided by the Christ Church Research Centre and the EPSRC funded QC13 Quan- tum Hub.
Bergmann, K., Nägerl, H.-C., Panda, C., Gabrielse, G., Miloglyadov, E., Quack, M.. Seyfang, G., Wichmann, G., Ospelkaus, S., Kuhn, A., Longhi, S., Szameit, A., Pirro, P., Hillebrands, B., Zhu, X.-F., Zhu, J., Drewsen, M., Hensinger, W. K., Weidt, S.,, Halfmann, T., Wang, H.- L., Paraoanu, G. S., Vitanov, N. V., Mompart, J., Busch, T., Barnum, T. J., Grimes, D. D., Field, R. W. Raizen, M. G., Narevicius, E., Auzinsh, M., Budker, D., Pálfff,, A., and Keitel, C. H. Roadmap on stirap applications. Journal of Physics B: Atomic, Molecular and Optical Physics, 52220))00200, September 2019. ISSN 1361- 6455, doi: 10. 10. 10.00088//361-64455433339955555 URL http: / //dx. doi. org/ 10. 1088 11111//8166111151111111 ab3995.
Impact Statement
This paper presents work whose goal is to advance the field of quantum control by applying machine learning tech- niques. We devise novel, higher fidelity, and more robust time dependent control signals for existing quantum sys- tems. This work is unlikely directly and significantly influence the applications of quantum technologies in the near term. However, in the long run, it may contribute to the development of advanced quantum technologies, which can have wide ranging impacts.
Bhatnagar, S. and Lakshmanan, K. An online octor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optinization The- ory and Applications, 153:688-708, 2012.
References
Bluvstein, D., Evered, S. J., Geim, A. A., Li, S. H., Zhou, H., Manovitz, T., Ebadi, S., Cain, M., Kalinowski, M., Hangleiter, D., Bonilla Ataides, J. P., Maskara, N., Cong, I. Gao, X., Sales Rodriguez, P., Karolyshyn, T., Se- meghini, G., Gullans, M. J., Greiner, M., Vuletié, V., and Lukin, M. D. Logical quantum processor based on reconfigurable atom arrays. Nature, 626(7997):58 65, December 2023. ISSN 1476-4687. s4 4415 026-777 URL http: http://xxxxdd.iioo doi.org/ 10. 001100///11555 023 06927-3.
Acharya et al., R. Suppressing quantum errors by scal- ing a surface code logical qubit. Nature, 614(7949); 676-681, February 2023. ISSN 1476-4687, doi: 10.1038/ $41586-022-05434-11 URL http: //dx doi. org/ 10. 10w. 011////11555502222202222
Altman, E. Constrained Markov decision processes. Rout- ledge, 2021.
doi: 10.1038/
An, 7.,, Song, H.-J., He, Q.-k,, and Zhou, D. L. Quan- tum optimal control of multilevel dissipative quantum systems with reinforcement learning. Phys. Rev. A, 103:012404, Jan 2021. doi: 10.1103/PhyseevA.000
Bradbury, J., Frostig. R., Hawkins, P., Johnson, J. Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. JAX: composable transformations of Puthon+NumPy programs, 2018. URL http://git hhu...hhhhh..... com/ gotg/e/oogeeeeeeoooo
012404. URL https:/ /t////1mm /iinkii....////// aps. org/doi/10. 1103/PhysRevA. 103. 012404.
Arute, F., Arya, K., Babbush, R., Bacon, D., Bardin, J. C., Barends, R., Biswas, R., Boixo, S., Brandao, F. G., Buell, D. A., et al. Quantum supremacy using a programmable superconducting processor. Nature, 574(7779):05-5500, 2019. doi: 10. 10.1038/44186-001 1-9666666-616665555
Breuer, H.-P. and Petruccione, F. The theory of open quan- tum systems. Oxford University Press, London, England, June 2002.
Brif, C., Chakrabarti, R., and Rabitz, H. Con- trol of quantum phenomena: past, present and fu- ture. New Journal of Physics, 12(7):075008, July 2010. ISSN 1367-2630. doi: 10.1088/1367-263001 7/075008. URL http://dss doi. wtt.:oo.g/....... 1367-2630/12/7/01535
Balasubramanian, G., Neumann, P., Twitchen, D., Markham, M.. Kolesov, R., Mizuochi, N., Isoya, J., Achard, J., Beck, J., Tissler, J., Jacques, V., Hemmer, P. R., Jelezko, F., and Wrachtrup. J. Ultralong spin coherence time in isotopically engineered diamond. Nature Materials, 8(5):383-37,, 2009. doi: 10. m0.1033// URL fttps::/don wogg//.....o.. 11. 1038/ mma..2400
Brown, J., Sgroi, P., Giannelli, L., Paraoanu, G. S., Paladino, E.,, Falci, G., Paternostro, M., and Ferraro,
10

---

Reinforcement Learning for Quantum Control under Physical Constraints
A. Reinforcement leaming-ennanced protocols for
Economou, S. E., Climente, J. I., Badolato, A., Bracker, A. S., Gammon, D., and Doty, M. Scalable qubit archi- tecture based on holes in quantum dot molecules. Phys. Rev. B, 86:085319, Aug 2012. doi: 10.1103/PhysRev. 86.085319. URL https::/p//.oo.c.. www.liinn..................m.... arg..org//o... 10. 1ww.1110yy/ee....... 86. 085319.
herent population-transfer in three-level quantum sys- tems. New Journal of Physics, 23()):093035, Septem- ber 2021. ISSN 1367-2630.
doi: 10.1088/1367-26300 article/10. 1088 ott../pp.p...eee......r.../
ac2393. URL
Brown, J., Paternostro, M., and Ferraro, A. Optimal quantum control via genetic algorithms for quantum state engineering driven-resonator mediated networks. Quantum Science and Technology. 8(2):025004, April 2023. ISSN 2058-9565, doi: 10.1088/2058-9555)
Egger, D., Werninghaus, M., Ganzhorn, M., Salis, G., Fuhrer, A., Müller, P. and Filipp. S. Pulsed reset protocol for fixed-frequency superconducting qubits. Phys. Rev. Appl., 10:044030, Oct 2018a. doi: 10.1103phhysRevApppied 10.044030. 1103/
URL
acb212. URL httt:::///oossciencee iop.orgi Article/10.1088/2200
https: aps PhysRevApplied. 10 044030.
Bukov, M., Day, A. G., Sels, D., Weinberg. P., Polkovnikov, A., and Mehta, P. Reinforcement learning in differ- ent phases of quantum control. Physical Review X, 8 (3), September 2018. ISSN 21603308. doi: 10.1103/ SoyssRevx.80031110
Egger, D., Werninghaus, M., Ganzhorn, M., Salis, G., Fuhrer, A., Müller, P., and Filipp, S. Pulsed reset protocol for fixed-frequency superconducting qubits. Phys. Rev. Appl., 10:044030, Oct 2018b. doi: 10.1105/PhyyReevApppiid 10.044030. https://link aps /ttg////////////o/////////////g 1103/ PhysRevApplied. 10. 044030.
arXiv: 1705.00565 Publisher: American Physical Society.
URL
Burkard, G. Non-markovian qubit dynamics in the presence of 1/f noise. Phys. Rev. B. 79:125317,
Mar 2009. doi: 0..108//hysReev..99.2555 1253717. URL https://link aps. www.saappp..pppo......e /ooo/ooo/o1/0 1103/ PhyaRevB. 79.12311
Erst, J. O., Alvarez, J. R., Barrett, T. D., and Kuhn, A. Bursts of polarised single photons from atom-cavity sources. Journal of Physics Atomic, Molecular and Optical Physics, 56(20):205003, sep 2023. doi: 10. org/10.1088/
Butterworth, S. On the theory of filter amplifiers. Experi- mental Wireless and the Wireless Engineer, 7:536-541, 1930.
URL https: //tp://xxxo doi. 6455 acf9d2.
Ernst, J. O., Chatterjee, A., and Franzmeyer, T. RLAqeWpe: Reinforcement Learning for Quantum Control with Physical Constraints, January 2025. URL https:// github. htcchub.com/// jan- wwwp://aooe//aaalaaaaooeo- -
Caneva, T., Calarco, T., and Montangero, S. Chopped random-basis quantum optimization. Phys. Rev. A, 84:022326, Aug 2011. doi: 10.1105/hhySeeA..88 022326. URL httpt:ttt////tttoc http://innkk..../ http:/nn......../ooo/////.. orpps./////o/00... 1103/PhysRevA. 84 022326.
Fletcher, R. Practical Methods of Optimization. John Wiley & Sons, New York, NY, USA, second edition, 1987.
Chatterjee, A., Schwinger, J., and Gao, Y. Y. Enhanced qubit readout via reinforcement learning. Physical Review Ap- plied, 23(5), May 2025. ISSN 2331-7019. doi: 10.1 103/ physrevappledd.23.55557.. URL http: dx doi. org/10
Fowler, A. G., Mariantoni, M., Martinis, M., and Cle- land, A. N. Surface codes: Towards practical large-scale quantum computation. Physical Review A, 86(3):032324, 2012.
23. 054057.
Davies, E. B. Markovian master equations. Communica- tions in Mathematical Physics, 39(2):91-110, 1974. doi: 10. 10.1/////806088889.. URL /nttps::// //wii.d//////////////// .org/.r...... 1007/BE01608388
Fromonteil, C., Bluvstein, D., and Pichler, H. Proto- cols for rydberg entangling gates featuring robustness against quasistatic errors. PRX Quantum, 4(2), June 2023. ISSN 2691-3399. doi: 10.1103/prxquantum.. 4.020335. URL http: / /xxx. doi ort..org/111.......1 PRXQuant 4. 020335.
Dirr, G., Helmke, U., Kurniawan, I., and Schulte- Herbrüggen, T. Lie-semigroup structures for reachability and control of open quantum systems: kossakowski-linbblad generators form lie wedge to markovian channels. Reports Mathematical Physics, 64(1):93-121, 2009, ISSN 0034-4877. doi: http:://o..ay///0.00000034441199
Fujimoto, S., Hoof, H., and Meger, D. Addressing func- tion approximation error in actor-critic methods. In Pro- ceedings of the 35th International Conference on Ma- chine Learning (ICML), pp. 1587-1596. PMLR, 2018 URL https: http:///pooedieeeeeeee mlr. www.pppees/eee. /80 fujimotol 88 html.
URL httpes//www.cccnneedddee.... ccclann//artice///////0448888999

---

Reinforcement Learning for Quantum Control under Physical Constraints
Gautier, R., Genois, E., and Blais, A. Optimal control in large open quantum systems: the case of transmon readout and reset, 2024. aattps://arxivg 2oss/2403 14765.
ume 58, pp. 221-236. American Mathematical Society, 2002.
Guilmin, P., Gautier, R., Bocquet, A., and Genois, É. dynamiqs: an open-source python library for gpu-accelerated and differentiable simulation of quan- tum systems, 2024. URL dynamigs/ dypamai.....aa.a.....
Giannelli, L.,, Rajendran, J., Macri, N., Benenti, G., Montangero, S., Paladino, E., and Falci, G. Opti- mized state transfer systems of ultrastrongly cou- pled matter and radiation. Nuovo Cimento C, 45 (6):1-4, July 2022a. ISSN 03905551, 03905551, doi: 10. 10.15973/cc42202222111y, URL http: //axxvv. arXiv:2203.03364 [cond-
Hairer, E., Nørsett, S. P., and Wanner, Solving Ordinary Differential Equations Nonstiff Problems, volume 8. Springer, 1993.
org/abs/2203.000 03364. mat, physics:quant-ph).
Henderson, P., Islam, R., Bachman, P., Pineau, J., Pre- cup. D., and Meger, D. Deep reinforcement learn- ing that matters. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 32, 2018. URL https: / /ttpp:://arxiv..oo/ aaa/aaiivvvv.....///.a org/ abs 1709.065600
Giannelli, L., Sgroi, S., Brown, J., Paraoanu, G. S., Paternostro, M., Paladino, E., and Falci, G. A tutorial optimal control and reinforcement learn- ing methods for quantum technologies. Physics Letters A, 434:128054, 2022b. ISSN 0375-9601.
Hou, S. C., Khan, M. A., Yi, X. X., Dong, D., and Petersen, I. R. Optimal lyapunov-based quan- tum control for quantum systems. Phys. Rev. A, 86:022321, Aug 2012. doi: 10.1103/PhysRewA.886 022321. URL /tttpp://linkk wtt..ka....o//////10.... org/doi/10: 1103/PhysReval 86. 022321.
doi: https://www.scieenccciiccee com/ scciene/arricceee///////11111113
URL
Glaser, S. Boscain, U., Calarco, T., Koch, C. P., Köckenberger. W., Kosloff, R., Kuprov, I. Luy, B., Schirmer, S., Hatultee-Hrbbüügeeen T., Sugny, D., and Wilhelm, F. K. Training schrödinger's cat: quan- tum optimal control: Strategic report on current sta- tus, visions and goals for research in europe. The European Physical Journal D, 69(12):779 December 2015. ISSN 1434-6060, 1434-6079, doi: 10.1 100/p/// e2015-60464-1. URL ht:p://link... http:://iii.....iig...... springer. com/10. 1 140 / jetddp/d//eeee hetpd//22111 5-6044--1
Huang, S., Dossa, R. F.J., Ye, C., Braga, J., Chakraborty, D.,
Mchta, K., and Araújo, J. G. Cleanrl: High-quality single- file implementations of deep reinforcement learning algo- rithms. Journal of Machine Learning Research, 23(274): 1-18, 2022. URL http: http://mmlrrrr./r/e//./. v23/21-133 11221-114..hat.
Jandura, S. and Pupillo, G. Time-optimal two- and three-qubit gates for rydberg atoms. Quantum, 6: 712, May 2022. ISSN 2521-327X. doi: 10.22331/ q- g-2002-05-13-771 13-712. URL https: http:://ooo...gggg
Goerz, M. H., Halperin, E. J., Aytac, J. M., Koch, C. P., and Whaley, K. B. Robustness of high-fidelity ryd- berg gates with single-site addressability. Phys. Rev. A. 90:032329, Sep 2014. doi: 10. 10.1103/PhyRevvA.00. 032329. URL https: aps. wttps///g////...////////// 1103/PhysRevA, 90. 032329.
Jandura, S., Thompson, D., and Pupillo, G. Optimizing rydberg gates for logical-qubit performance. PRX Quan- tum, 4:020336. Jun 2023. doi: 10.1103/RRXQanttm..
020336. URL https: ht///innk... org/doi/10. 1103/PRXQuantun 4. 020336.
Goerz, M. H., Whaley, K. B., and Koch, C. P. Hybrid optimization schemes for quantum control. EPJ
Jozsa. R. Fidelity for mixed quantum states. Journal of Modern Optics, 41(12):2315 2322333, 1994. doi: 10.1080/ (95003499444552177 URL httph:t/ts//ttt httt:s::/doi..o//1... 1088/09503399145522
Quantum Technology. 2(1), September 2015. ISSN
2196-0763. doi: 10. 1001144000pp00440000000000000000 URL http://da doi. .eeeiiorrg/e wtt.10..114//pi.c.... 840507-015-0034-
Keysight. M3202a pxie arbitrary waveform generator.
Goerz, M. H., Basilewitsch, D., Gago-Encinas, F., Krauss, M. G., Horn, K. P., Reich, D. M., and Koch, C. P. Kro- tov: A Python implementation of Krotov's method for quantum optimal control. SciPost Phys., 7:80, 2019. doi: 10.21 10011466siiisssst1yy......8
mhz.html, 2024. Accessed: 2025-05-04.
Khalid, I. Weidner, C. A., Jonckheere, E. A., Schirmer, S. G., and Langbein, F. C. Sample- efficient model-based reinforcement learning for quan- tum control. Phys. Rev. Res., 5:043002, Oct
Gottesman, D. An introduction to quantum error correction. In Proceedings of Symposia in Applied Mathematics, vol-
12

---

Reinforcement Learning for Quantum Control under Physical Constraints
2023. doi: 10.1 1011003PPyyyseeRReeeecch.0000000 URL https://tink..pps. wwtp:iiww.....a...o/o//oooa/e/ 1103/ PhysRevResearchh 5.043022.
Krenn, M., Landgraf, J., Foesel, T., and Marquardt, F. Artificial intelligence and machine learning for quantum technologies.
Phys. Rev. A. 107: 010101, Jan 2023. doi: 10.1 10.1103/PhysRevA.107 010101. URL https:/ http:::://liik..ps.. aps. wwwpppppppp.oo...... 1103/hhyseev..o 107. 010101.
Khaneja, N., Reiss, T., Kehlet, C., Schulte- Herbrugenn, T., and Glaser, S. J. Optimal control of cou- pled spin dynamics: design of nmr pulse sequences by gradient ascent algorithms. Journal of Mag- netic Resonance, 172(2):296-305, 2005. ISSN 1090-7807. doi: http://bbiirrg/10...00//mmm220.: 004. URL htt.:attt://.w..eeeeeeeeeeeeeeee scc.aancc//rriiee////////0007000
Kuklinski, J. R., Gaubatz, U., Hioe, F. T., and Bergmann, K. Adiabatic population transfer in three-level sys- driven by delayed laser pulses. Phys. Rev. A. 40:6741-611 --744, Dec 1989. doi: 10.1103/PhysRev.. 40.6741. URL htthttt::/ 1103/PhysRevA, 6741.
Kidger, P. On neural differential equations, 2022. URL
Levine, H., Keesling, A., Semeghini, G., Omran, A., Wang, T.T., Ebadi, S., Bernien, H., Greiner, M., Vuletic, V., Pich- ler, H., and Lukin, M. D. Parallel implementation of high- fidelity multiqubit gates with neutral atoms. Phys. Rev. Lett. 125:170503, Oct 2019, doi: 10. 10.110//hhsseevee 123.170503. URL https: //tp::::///ikkk 1inkk. org. doi/ 10. 10.11133PPyyyeee ht//PhyysReiitt... 23. 170503.
https:/
arxiv. http//awwvv......a....//../../at abs /ww/////....2 2202. 02435.
Koch, C. P. Controlling open quantum systems: Tools, achievements, and limitations. Journal of Physics: Condensed Matter, 28(21):21301,, June 2016. ISSN 0953-8984, 1361-648X. doi: 10.1088/0953-8884428 21/213001. URL http: www.:iiww....gaaeooooooo wwtp:oo////ab.////// 04417. 16Xiv:1603.000 [quant-ph].
Li S., Fan, Y., Li, X., Ruan, X., Zhao, Q., Peng, Z., Wu, R.-B., Zhang, J., and Song, P. Robust quantum control using reinforcement learning from demonstration. upi Quantum Information, I1(1), July 2025. ISSN 2056-6387. doi: 10. 10.1038//411444405-000000 doi wwg/10....... 1 1033//11114--0-3000005333
Koch, C. P., Boscain, U., Calarco, T., Dir,, G., Fil- ipp, S., Glaser, S. J., Kosloff, R., Montangero, S., Schurte-n T., Sugny, D., and Wilhelm, F. K. Quantum optimal control in quantum technologies. Strategic report current status, visions and goals for research in Europe. EPJ Quantum Technology, 9()):19 December 2022. ISSN 2662-4400, 2196- 0763. doi: 10. 101144400PP000444000000000011030
URL htwp: dx.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y. Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In International Con- ference on Learning Representations (ICLR), 2016. URL https: arwia.iivvv.... http:/aawvvv.....e.aaa/ http:////////1199999900 02971.
URL http://pppee..ooaarcccccs////e/e epjet/
springeropen,com wwt.c//o///aa/.
1140/
Lu, C., Kuba, J., Letcher, A., Metz, L.., Schroeder de Witt, C., and Foerster, J. Discovered policy optimisation. Ad- vances Neural Information Processing Systems, 35: 16455-16468, 2022.
Koch, J., Yu, T. M., Gambetta, J., Houck, A. A., Schus- ter, D. L., Majer, J., Blais, A., Devoret, M. H., Girvin, S. M., and Schoelkopf, R. Charge-insensitive qubit design derived from the cooper pair box. Physical Re- view A, 76(4), October 2007. ISSN 1094-1622, doi: 10. 10.1103/phyysrcww.7664233 URL http: /ttttp://xxxx//// doi. org/10. httg/10.1103/hhysseee 76. 042319.
Lukin, M. D., Fleischhauer, M., Côté, R., Duan, L. M., Jaksch, D., Cirac, J. I., and Zoller, P. Dipole blockade and quantum information processing in mesoscopic atomic ensembles. Physical Review Letters, 87(3)033790,, 2001. doi: 10. 1ee3/PhysRee..ee..7...3333 87.099911
Koch, M., Sames, C., Kubanek, A., Apel, M., Balbach, M., Ourjoumtsev, A., Pinkse, P. W. H., and Rempe, G. Feedback cooling of single neutral atom. Phys. Rew. Lett., 105:17 Oct 2010. doi: 10.1 1000/hhyssevvLett 105. 173003. ht 10. 100. 10.1103/Physseet http///hhyssseeeeeeeeeeeeeeessee 105. 173003.
Magnard, P., Kurpiers, P., Royer, B., Walter, T., Besse, J.- C., Gasparinetti, S., Pechal, M., Heinsoo, J., Storz, S., Blais, A., and Wallraff, A. Fast and unconditional all- microwave reset of superconducting qubit. Phys. Rev. Lett., 121:060502, Aug 2018. doi: 10. 10110//hysRevLett. 121.060502. URL https: /twnkiinkk/// .ps.org/ / doi/ 10. www1103//hyeeeveeece
121 060502.
Král, P., Thanopulos, I., and Shapiro, M. Colloquium: Coherently controlled adiabatic passage. Rev. Mod. Phys., 79:53-77, Jan 2007. doi: 10.1103/ReMoodPyy.. 79.53. URL https:/ //t/:///nnk//////////// aps. hwps.org//o// wrt/doi/dd.o.oooooooooooo 1103 /////eeddPyyss....om 73.
Mahesh, T. S., Batra, P., and Ram, M. H. Quantum op- timal control: Practical aspects and diverse methods, June 2022. URL http: //t::///aiiivaae org/ / abs 2205. 15574. arXiv:2205.1 15574 [quant-ph].
13

---

Reinforcement Learning for Quantum Control under Physical Constraints
Metz, F. and Bukov, M. Self-correctngg quantum many- body control using reinforcement learning with ten- sor networks. Nature Machine Intelligence, 5(7):
Pagano, A., Müller, M. M., Calarco, T., Montangero, S., and Rembold, P. Role of bases in quantum optimal con- trol. Phys. Rev. A, 10:0026008, Dec 2024. doi: 10.1103/ 1rysReevA.10000600000 URL https: /tt///iinka.p..aa aps. org/doi /10 111.11/0//////.a... 11 110.0626600 026608
780-791, July 2023. ISSN 2522-5839. doi: 10.1038/
$42256-023-0068877 URL http:: wwt..ddooogg///ooooooooooooooo.. 10. 0011000//422555 023- 00687-5.
Paparelle, I., Moro, L., and Prati, E. Digitally stim- ulated raman passage by deep reinforeement learn- ing. Physics Letters A, 384(14):126666, May 2020. ISSN 03759601, doi: 10.1016/f.physleta.20000 126266. URL AtLtsp:://LLLnniicubbb.c. com comm://eraaaeeeaa//////711122220
Mohan, M., de Keijzer, R., and Kokkelmans, S. Ro- bust control and optimal rydberg states for neutral atom two-qubit gates. Physical Review Research, 5(3), July 2023. ISSN 2643-1564, doi: 10.1 00.1103/physteeresearcch 5.033052. URL http:: PhysRevResearch. 5333052..
doi wwt/10..1000.
Pelegrí, G., Daley, A. J., and Pritchard, J. D. High- fidelity multiqubit rydberg gates via two-photon adia- batic rapid passage. Quantum Science and Technol- ogy, 7(4):045020, August 2022. ISSN 2058-9565. doi: 0110088/2558-5555/aee
Mølmer, K., Castin, Y.. and Dalibard, J. Monte carlo wave- function method in quantum optics. J. Opt. Soc. Am. B, 10 (3):524-533 -38, Mar 1993. doi: 11564//0SSAB.0..000555 URL https: //pg: . optica wttp:/iiic..r//saaaaaaa../ abstract www.eefimmuumseeeeee
URL http: dx..do.....ooooooooooooo...o..od org 10. 1.....088/20555 95 9565/556566555 ac.23a.
10-3-524- 524.
Nam Nguyen, H., Motzoi, F., Metcalf, M., Birgitta Wha- ley, K. Bukov, M., and Schmitt, M. Reinforcement learning pulses for transmon qubit entangling gates. Ma- chine Learning: Science and Technology, 5(2):025066, June 2024. ISSN 2632-2153. doi: 10. 10.1088/2632-21553/
Porotti, R., Essig, A., Huard, B., and Marquardt, F. Deep reinforcement learning for quantum state prepa- ration with weak nonlinear measurements. Quantum, 6:747, June 2022. ISSN 2521-327X.
doi: 10.22331/ 9-2022-06-28-744 747. URL http: / htp///xx.oo wor..ooo........o 22331/q 2022 - 08-0000-2888888 2877...
ad4f4d. URL http:// /tx///xx//// doi. wrg.ooggg//.ooo..ooooooooo/ 110.1080/ 1088/ 2632- 2153/ad4f4d.
Puzzuoli, D., Wood, C. J., Egger, D. J., Rosand, B., and Ueda, Qiskit dynamics: python package for sim- ulating the time dynamics of quantum systems. Jour- nal of Open Source Software, 8(90):5853, 2023. doi: 10.21105//jss.058833, URL https: http:::/ddooo.rgggg/ 21105/ joss. 05853.
Nielsen, M. and Chuang, L.. Quantum Computation and Quantum Information: 10th Anniversary Edition. Cambridge University Press, 2010.
Niu, M. Y., Boixo, S., Smelyanskiy, V. N., and Neven, H. Universal quantum control through deep reinforcement learning. npj Quantum Information, 5(1), December 2019. ISSN 20566387. doi: 10. 11008//41154-001-014444 Pub-
Quetschlich, N., Burgholzer, L., and Wille, R. Compiler optimization for quantum computing using reinforcement learning. In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2023.
lisher: Nature Partner Journals.
Norambuena, A., Mattheakis, M., González, J., and Coto, R. thysics-informed neural networks for quantum con- trol, December 2023. URL /ttt:::///xivvai/// org/ 2bs/ 06287. arXiv:2206.0628) [quant-ph].
Reich, D. M., Ndong, M., and Koch, C. P. Monotoni- C. cally convergent optimization in quantum control using Krotov's 's method. The Journal of Chemical Physics, ISSN 0021-9606. doi: 10.1063/1.3691827, URL attpps://oii. org/10.
136(10):10410) 03 2012.
Olle, J., Zen, R., Puviani, M., and Marquardt, F. Simul- taneous discovery of quantum error correction codes and encoders with a noise-aware reinforcement learn- ing agent. npj Quantum Information, 10(1), De- doi: 10.1038/
Reuer, K., Landgraf, J., Fösel, T., O'Sullivan, J., Beltrán, L., Akin, A., Norris, G. J., Remm, A., Kerschbaum, M., Besse, J.-C., Marquardt, F., Wallraff, A., and Eichler, C. Realizing deep reinforcement learning agent for real- time quantum feedback. Nature Communications, 14 (I), November 2023. ISSN 2041-1723, doi: 10.1038/ -41467-003-429900 URL http: www..doo..rgg//.o 10. 02218/444447-7772220000
cember 2024. ISSN 2056-6387, 021534-024000920- URL http://dd //xx doi.org/ 10. N0.1038/s41552 224-00920-y.
Pagano, A.. Weber, S., Jaschke, D., Pfau, T. Mein- ert, F. Montangero, S., and Büchler, H. P. Error budgeting for a controlled-phase gate with strontium- 88 rydberg atoms. Phys. Rev. Res., 4:033019, Jul 2022. doi: 10.1103/PhyaReeeeeeeccchc. URL PlystevResearch.440333
Richards, A. University of oxford advanced research computing, 2015. URL https: / zenodo.org/ record/22558.
wwtp///////ooooooooooo//oo//oo/t 1103/
14

---

Reinforcement Learning for Quantum Control under Physical Constraints
Ristè, D., Bultink, C. C., Lehnert, K. W. and DiCarlo, L. Feedback control of solid-state qubit using high- fidelity projective measurement. Phys. Rev. Lett., 109: 240502, Dec 2012. doi: 10.1 10.110//hyseeeeettt.ooo 109. 240502. URL httpst/s: iik aps. www.aaa.p...oo......... doi /10.
Vasilev, G. S., Kuhn, A., and Vitanov, N. V. Optimum pulse
shapes for stimulated raman adiabatic passage. Phys.
Rev. A, 80:013417, Jul 2009, doi: 10.1 10.1103/PhysRev4.800
013417. URL https: / h///iin.aappp org/doi/11
1103 1103/PhysRevA, 80 013417..
1103/PhysRevLett. 249.2000..
Vitanov, N. V., Suominen, K.-A., and Shore, B. W. Cre- ation of coherent atomic superpositions by fractional stimulated raman adiabatic passage. Journal of Physics B: Atomic, Molecular and Optical Physics, 32(18): 4535-4546, September 1999. ISSN 1361-6455. doi: 10. 10880995--4055/2118832.. URL http: //xx doi.
Saffman, M., Beterov, I., Dalal, A., Páez, E. J., and Sanders, B. C. Symmetric rydberg controlled- gates with adiabatic pulses. Phys. Rev. A, 101: 062309, Jun 2020. doi: 10.1103/PhysRevA..00 062309. URL https:/ //////1/ /wwp//inknaaeee aps. org/doi/10. 1103/PhysReva, 101. 101.002300..0.0.....
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017.
Vitanov, N. V., Rangelov, A. A., Shore, B. W. and Bergmann, K. Stimulated raman adiabatic passage in physics, chemistry, and beyond. Rev. Mod. Phys., 89:015006, Mar 2017. doi: 10. MooD0//s 015006. URL https: // llikk.......//. laikiaass....................... .ess.org org/doi doi /10. 1103 Ta13//evModPhy... 89. 015006.
Schäfer, F., Kloc, M., Bruder, C., and Lörch, N. A differentiable programming method for quan- tum control. Machine Learning: Science and Technology, 1(3):035009, September 2020. ISSN 2632-2153. doi: 10. 00.1088/26332-255349900000 URL htt.:e.......rgg/...ceeeeeeeeeee 0188/2632-2153/bbb995
Wang. C., Li, X., Xu, H., Li, Z., Wang, J., Yang, Z., Mi, Z., Liang, X., Su, T., Yang. C., Wang, G., Wang, W. Li. Y., Chen, M., Li, C., Linghu, K., Han, J. Zhang, Y.. Feng, Y., Song. Y., Ma, T., Zhang, J., Wang, R., Zhao, P., Liu, W., Xue, G., Jin, Y., and Yu, H. To- wards practical quantum computers: transmon qubit with a lifetime approaching 0.5 milliseconds. upj Quantum Information, 8(1), Jamuary 2022. ISSN 2056-6387, doi: 10. 11118///415544022-0051111111 org/10 10 1038 / 00c///..100///1333----0000000000 021-0000---20000
Sivak, V. V., Eickbusch, A., Royer, B., Singh, S., Tsiout- sios, I. Ganjam, S., Miano, A., Brock, B. L., Ding, A. Z., Frunzio, L., Girvin, S. M., Schoelkopf, R. J., and Devoret, M. H. Real-time quantum error cor- rection beyond break-even. Nature, 616(7955):50-55, doi: 10.1038/ $415866003--05882-6 URL http:/ /xxx doi.org/ 10. 10.1038/s4110 023 05782-6.
URL http::///// doi.
March 2023. ISSN 1476-4687.
Zeytinoglu, S., Pechal, M., Berger, S., Abdumalikov, A. A., Wallraff, A., and Filipp. S. Microwave-inducee amplitude- and phase-tunable qubit-resonator coupling in circuit quantum electrodynamics. Phys. Rev. A,
Sun, Y. Off-resonant modulated driving gate proto- cols for two-photon ground-rydberg transition and finite rydberg blockade strength. Opt. Express, 31(2)3114-3121, Jan 2023. doi: 10.1364/OE.
91 91:043846, Apr 2015. doi: 10.1103/PhysRevA.99 043846. URL https: / ht///inkapppp www.ii........oooo...ooooooooooe org/doi/10; PL03/PhysRevA, 91. 043846.
480513. URL https://opgt wwwp:a.pttccco..o. org/ abstract wfw:/RI==ee 31-2--1112--1-1----
Zhang. X.-M,, Wei, Z. Asad, R., Yang, X.C., and Wang. X. When does reinforcement learning stand out in quantum control? A comparative study on state preparation. npj Quantum Information, 5(1): 85, October 2019. ISSN 2056-6387. doi: 10.1038/ $41534-019-0201-8. URL https:: http://wwtt.r.r....mmm// com/articles/ 111-11103------0--------
Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Robotica, 17(2):229-23 2335, 1999.
Terhal, B. M. Quantum error correction for quantum memo- ries. Reviews of Modern Physics, 87(2):307-346, April 2015. ISSN 1539-0756. doi: 10.1103/revmodphys.. 87.307. URL http:/ dxx doi. oro.0..rg/10.10000 NevModPhys.87..
Álvarez, J.-R., IJspeert, M., Barter, O., Yuen, B., Barrett, T. D., Stuart, D., Dilley, J., Holleczek, A., and Kuhn, A. How to administer an antidote to schrodinger's cat. Journal of Physics B: Atomic, Molecular and Optical Physics, 55(5):054001, March 2022. ISSN 1361-6455 doi:
Van Der Linde, S., De Kok, W., Bontekoe, T., and Feld, S. qgym: A gym for training and benchmarking rl-based quantum compilation. In 2023 IEEE International Con- ference on Quantum Computing and Engineering (QCE), volume 2. pp. 26-30. IEFF, 2023.
URL http: : /xx doi. aai.org/11 .ttp://1.1118000 100010181110 1011111111-1555515555555555111 00113361165555555577756
15

---

Reinforcement Learning for Quantum Control under Physical Constraints
Appendix
A.3. Proximal Policy Optimisation (PPO)
Here we present detailed explanations of the extended RL background, quantum dynamical systems simulated in the main paper, show auxiliary results and explain our imple- mentation in greater detail.
Proximal Policy Optimisation (PPO) (Schulman et al., 2017) is popular algorithm in modern RL, combining the benefits of policy gradient methods with stability improvements. PPO aims to optimise the policy by ensuring that updates do not deviate too much from the previous policy. This is achieved using a clipped objective function.
A. RL Background
The objective function in PPO defined as:
A.1. Bandit Setting in Reinforcement Learning
I CLP (0) = E. min (ra(0)A,, clip +)A,]] (7)
In the bandit setting, the RL. problem is simplified as there no state transition, only actions and rewards. Each action a E A, which are time-dependent quantum control signals A, 24 yields a reward from stationary probability distri- bution. The objective is to maximise the expected reward over sequence of actions.
where:
re(0) me Te [+r]
To (st) is the probability ratio under the new
and old policies.
Ag is an estimate of the advantage function at time-step
Formally, given a set of actions A. each action a E A has an unknown reward distribution with expected reward R(a).
is hyper-parameter that controls the clipping range.
The goal is to find the action a" that maximises the expected reward:
The clipping mechanism in the objective function ensures that the new policy does not deviate significantly from the old policy, thereby improving training stability and prevent- ing large, destabilising updates.
a" arg max E [R(a)]
(4)
This setting forms the basis for more complex RL problems.
PPO also incorporates entropy bonus to encourage ex- ploration and prevent premature convergence to suboptimal policies. The overall objective with the entropy bonus can be written as:
A.2. Extended Time Horizon in Multi-step RL
For multi-step RL, we consider an extended time horizon. In contrast to the bandit setting, each episode is divided into four sections, each of 1/4 the length of the total number of action samples comprising the control signal. The agent does not observe any information about the noise at time step / 0, with the observation O O. However, time steps 0.25, 0.5, and 0.75, the agent receives the value of mean noise μn sampled at the beginning of the episode. Formally, the observation function O is defined as:
L( E. [L c (2))
+2
(8)
where ei and 02 are coefficients, and El (s)) denotes the entropy of the policy at state 8;-
In summary, PPO effectively balances exploration and ex- ploitation while ensuring stable policy updates, making a robust choice for RL in quantum control tasks.
0
if ( = 0
B. Quantum Dynamics & Control
O,
PB if t = 0.25k for = 3,
Quantum dynamics describes the time evolution of quan- tum systems. system's state is represented by a quantum state, vector in a complex Hilbert space H. The most common representation is the state vec- for (v) E H. A pure quantum state is described by normalised vector (Nielsen & Chuang, 2010) (eb) vn)" where (w/4b) = 1. more
The agent's policy æ then uses this observation to decide the action at each time step, where 5, is the union of the state in the bandit setting 8t and the observation O which defines an action a, through conditional probability distribution P.
(Vi, 4/2, general representation is the density matrix p. which for a pure state is p = |ı5 (11, (Nielsen & Chuang. 2010), and ex- tends to classical mixtures of pure quantum states. The quan- tum state populations are defined as |w/ (i.e. the diagonal terms of p). Operators in quantum mechanics are unitary, making dynamics reversible. The unitary time evolution of (t)) is governed by the denendenn Schrödinger equation:
T5 a.) = P[,
In general extended time horizon RL, the agent must con- sider the long-term consequences of its actions. This is formalised through the discount factor y, which ensures that future rewards are appropriately weighted. Given that we have a fixed number of four steps we set the discount factor to zero
16

---

Reinforcement Learning for Quantum Control under Physical Constraints
(e2)
δ ih a (b( - -0)))(1000) (o)),
(9)
Ax
(e1)
where h is the reduced Planck constant, and H is the Hamil- tonian operator representing the system's 's total energy. Quan- tum control manipulates systems to achieve desired dynam- ics using time-dependent control fields, represented by the control Hamiltonian. The total Hamiltonian (t) of con- trolled system is (Giannelli et al., 2022b):
AA
(gr)
1
f2g
(t) Ho + Σ Lai(t)H,
(10)
191)
(92)
where Ho is the drift Hamiltonian, (t) are time-dependent control actions, and H; are control Hamiltonians. In open quantum systems, environmental interactions lead to non- unitary evolution, also sometimes described as non-coherent. The master equation (Davies, 1974; Dirr et al., 2009) cap- tures this evolution as:
Figure 6: Energy level diagram for four level A system with state le2), detuned positively by Ax from [ei), to which cross talk is suppressed. There is an additional state [gr) which does not partake in the unitary dynamics, but to which the excited states decay (cf. red dotted lines). This gives rise to a lower bound in attained population transfer fidelities. The laser couplings from Stokes and Pump laser are shown in blue.
Op(t) i 00 -H p() L(o()),
(11)
where [ř, p(() denotes matrix commutation, and L(p) de- scribes non- non-unitary evolution (c.g. spontaneous emission, dephasing, cavity decay, etc.). Fidelity is a common mea- sure of similarity between quantum states. For arbitrary density matrices p and o, the fidelity (Jozsa, 1994) reads:
O
De
Q2
L
Pt
O
A P
As
HA/h-
(13)
fg
AP
0
F(o,o) = -TTT V
(12)
ff
fg
AP + Ax
All Rabi frequencies Ωpys are real. Additionally we in- clude sink state to which spontaneous emission occurs which couples equally to both excited state with rate T/v2, this is realistic insofar as spontaneous emission can always occur to states outside the manifold of interest, but as we do not consider spontaneous emission to 91 or 92 we obtain lower bounds on any population transfer fidelities F. The Lindbladian operator reads; I f/v2 (9a) (esl-
where Tr is the trace. In this paper, we evaluate the fidelity between a target state Pdes and the final evolved state p(ty) to assess the effectiveness of the applied controls (t)..
C. Electronic A Systems
A very common system configuration in quantum infor- mation contains two ground-states [9i) 192). coupled by a common excited state (e1), as is required for the imple- mentation of many quantum population transfer protocols, such as Stimulated Raman Adiabatic Passage (STIRAP) (Vitanov et al., 2017). We also include additional ex- cited state lez), detuned positively by an amount Ax from lex) to show the effect of crosstalk due to a coupling to an undesired transition. This configuration is ubiquitous and arises naturally in colour centres, quantum dots or other elec- tronic quantum systems. An explicit energy level diagram provided in Fig. 6. Slyss denote the Rabi frequencies of the Pump and Stokes pulses respectively and p are the detuning of the Pump pulse from resonance as well as the two photon detuning respectively. The Hamiltonian HA used to model the unitary dynamics, defined in the basis ((91) 192) I lez). le2)), after an application of the rotating wave approximation reads:
The initial state is always fixed as (g1), and the target (final) state is typically chosen as 192). However, the protocol can also be extended to other target states, such as the super- position state |+) = / (191) + 1922), as discussed in Ref. (Vitanov et al., 1999). More generally, the method allows for arbitrary superposition angles 6, but in this work, we focus on the case = म, which corresponds to full popu- lation transfer to (g2). This scenario is widely studied and well-documented in the literature (Vitanov et al., 2017).
For the A system introduce an additional reward term which reads -w, (fe)) + (e2)). This assigns lower re- wards to non-coherent dynamics, since we seek coherent population transfer and speeds up the learning dynamics.
We showcase two particular reference pulses for different pulse areas in Fig. 3. Trade-offs between pulse areas and
17

---

Reinforcement Learning for Quantum Control under Physical Constraints
Pulse Aroae / MMI2"
work has been done to improve its robustness to qtasi-static errors (Jandura et al., 2023; Mohan et al., 2023), but this often comes at the expense of the overall gate duration (Fromonteil et al., 2023) which is undesirable. Using the physics of a two photon process (similar to the A system dynamics) we follow the Hamiltonian definition Hrz = Ho.2 + Hint,2 for a two-photon Rydberg gate given in (Sun, 2023) (where h.c. denotes the hermitian conjugate);
10F
M Mininal Pulse Area Denehmark
101
0.970
0.975
0.980
0:955
0.990
0.995
Fidelity
Figure 7: We sweep W.. the pulse area penalty weight (ef.
(3)) over a range of values [0, 0.1, 0.25, 22 and approach minimal pulse area with respect to Norambuena et al. (2023) (green dotted line) whilst achieving significantly higher fi- delities F= > 0.83 (ef. Norambuena et al. (2023)). There is clear trade off and lower pulse areas generally adversely affect fidelities.
Ho. S2p(t)
110) (c0 + st((t)
(c0) 1r0) + h.c.
h
2
(15)
+ +tp(t) (e0) (c0) + at(() (r0) (r0),
with time-dependent Rabi frequencies (p))), (t()), and values for the one photon detuning Ap and two-photon detuning Ag. The Hamiltonian terms for (01) follow anal- ogously from symmetry considerations by swapping the qubit states in Ho.2.
population transfer fidelity are shown in Fig. 7. It is show that we approach the lower pulse area limit described in Ref. (Norambuena et al., 2023). We also show robustness to time-dependent noise in Fig. 8.
The interaction Hamiltonian Hant, for the state |1, 1) con- sists of the atom light interaction as well as the dipole- dipole interaction akin to (14). A basis transformation sim- plifies the Hamiltonian, the new basis states read (ē) (lel) + \le))//2,11
D. Rydberg gates
(Ir1) + [lr))/v2 and (R) ((re) + ler))/ after the rotating wave approximation, and effectively neglecting lee), as we are in the regime where Ap >> As. H.m h can be expressed as:
We first consider Rydberg gate based on a single laser ex- citation which is near resonant with the ground-state qubit (1) and Rydberg level (r) transitions. Following the imple- mentation experimentally shown in (Levine et al., 2019) and the Hamiltonian definition given in (Pagano et al., 2022) the Hamiltonian for the one-photon Rydberg gate Hr, = Ho + Hin reads:
Hint. 2p((t)
lt((t
[11] (a)
+
(2) (F]+
h
V2
2
(t)()
(7) (R) + (ts(t)
[ (rrl+ + h.c. + (16)
Ho h
Σ
Ω 2
(r))
(11, + (1) AIr Am)
2
V2
pt (ē) (ā]+ + (2 p (t) + B) (m)) (mrl+
(14)
Hiat h
(Ap(t) + at) R (7 + (t (F) (F)
= Blr,r)((,
Parameters Psy A Ss/P; B are defined as (14). The Lind- bladian decay terms for the two photon Rydherg gate are described similarly as for the one photon Rydberg gate, They read; E, Fr ((99l)))))) il + (i,rl) + [,((r)) rl) + Σ, (e ((e, il + (i, el) + ( (e, e)). where T, de- scribes the decay rate of the Rydberg level and I' the decay of the excited level |e) where for typical atoms r. >> Tr..
Here 2(1) and (t) are real amplitudes and detunings of a Rydberg laser and B describes the dipole blockade strength. The Linbladian terms are described by the addition of sink state ør which imposes lower bound on fidelity since any population which spontaneously decays leaves the computational subspace, as for the A system. They read; E,r(()r) ((r, il + ri) + (r((g)) (r. rl), where T, describes the decay rate of the Rydberg level. Many optimisation protocols consider B - oo, since the Rydberg gate operates the regime << B which precludes cou- pling of both qubits to Ir). however we fix B to a finite but realistic value in the range of hundreds of MHz (Pagano et al., 2022; Pelegrí et al., 2022; Sun, 2023).
Akin to the system we introduce an additional reward term which reads w ((rr) + ((0) + (FF) + (rr) + (RR)). This assigns lower rewards non-coherent dynamics, since we seek coherent population transfer and speeds up the learning dynamics.
The fidelity FR is defined by the Bell state fidelity as is com- mon in optimisation protocols of the Rydberg gate (Saffman et al., 2020; Jandura et al., 2023):
One of the drawbacks of this implementation, as described in the main text however, is that it is not particularly robust in the face of signal imperfections and noise, such as atomic motion, laser intensity of frequency fluctuations. Some
FR =
1 1+ 16
Σ
E
- (g) 00
(17)
11
18

---

Reinforcement Learning for Quantum Control under Physical Constraints
0a = 0.1
d
10
10 3
A
10-5
011
0.1
01
0.1
10-7
1
On =
3
0n
0
200
400
600
800
1000
1200
1400
200
400
600
800
1000
1200
1400
RL Update Steps
Update Steps
Figure 8: Robustness of PPO for A-system randomly generated noise with من 0.1 MHz (left) and μ^ MHz (right) plotted logarithmic scale and averaged over multiple seeds. The solid lines show the average infidelity, while the shaded regions indicated one s.d. parallel environments. For small noise levels F > 0.999 as shown in the left plot for μs 0.1 MHz, but as it increases fidelities drop to just below 0.97. Robustness to larger levels of noise is addressed with multi-step feedback in Sec. 5.4.
where g is the simulated computational basis state. With- loss of generality, the optimisation here focusses on the case where 0g 0, except 01,1 x. this is particularly useful insofar does not require additional single qubit rotations (in comparison to general C'Z gate) and does not introduce any further complexity associated with addi- tional rotations. In practice, the computational complexity of simulating the Rydberg gate can be reduced by focus- ing on initialising the system in the coherent superposition state |+, +). The optimisation then targets both the popu- lations and the relative phases of the resulting superposed states, ensuring they match the desired angles 0 and main- tain equal amplitudes. This approach effectively reproduces the transformation expected from an ideal logical C-Z gate: (100) + [10) + 110) - [11)).
randomly generated initial states yields very similar fideli- ties those obtained from the definition given in 17 with the 0.25μs C-Z gate yielding F z 0.99957 = 0.0003 (ver- sus z 0.99958 as obtained by taking the Bell state fidelity). Moreover, we observe mean fidelity of F > 0.999 for time-depende unbiased amplitude noise of og = +1 MHz and frequency noise of as = +22 MHz over 10 different randomly generated noise samples (cf. Eq. 25) for further details.
Following remarks made in Ref. (Sun, 2023) we reiterate that show increased resilience noise and achieve fi- delities in excess of 0.999 even with significant levels of time-dependent noise, spontaneous emission (using realistic parameters for 87 "Rb (Sun, 2023) atom) and a finite block- ade strength of 500 MHz. Moreover, our control solution is robust in fidelity across a variety of smaller blockade strengths.
As described in the main text, we focus on the implemen- tation of a two-photon Rydberg gate. For this, we fix the detuning of the pump pulse to a constant value, since time- dependent frequency chirp offers no advantages in terms of achievable maximum fidelities, we merely optimise its constant value. Over duration of 0.5ju fix Ug maximum value of 40 and (2Ap) (the effective Rabi frequency) to a maximum value of 56.6 with a pump detun- ing of 2.5 GHz and obtain an optimal control signal which is shown in Fig. 10. shall be noted that the signals are differ- ent from results in the literature since we impose the realistic constraint of amplitudes to start and end at zero amplitude compared to (Sun, 2023). The optimal time-dependent con- trol signals for direct realisation of a C-Z gate are shown in Fig. 10. Simulated randomised benchmarking with 100
E. Transmon Qubit Reset
Methods for unconditional Transmon qubit reset with fixed- frequency devices involve the coupling of a Transmon to a resonator through which excitations decay quickly, One particular hardware efficient protocol based on cavity- assisted Raman transition utilising the drive-induced cou- pling between IfO] and [gl), where (sn) denotes the tensor product of Transmon in [s} and readout resonator mode in the Fock state [n). By driving the Transmon simultane- ously at the |e0) ++ ffO] transition and the (f0) ++ (gl) transition, we can form A system the dyynes-Cumming
19

---

Figure Comparison of quantum state dynamics for our RL solution (top) and Optimal Control solution (bottom). The RL solution exhibits clearer interpretation of the optimal state dynamics with smooth population of (g2) and low excited state population le:). The optimal control solutions achieves high final population of the state (92), but the time Reinforcement Learning for Quantum Control under Physical Constraints
ladder which can be used reset the Transmon through fast single photon emission. The Transmon reset Hamiltonian is given by
H =xalag'
go V25 + +
((t)((')) h.c.
(18)
(A(t) + +o(()))
RL Solution State Evolution
1.0
where a(at) the resonator lowering (raising) operator, ((9*) the Transmon lowering (raising) operator, X the Transmon-resonatoor dispersive shift, the Transmon anhar- monicity, the Transmon-resonator coupling rate, 5 the dif- ference in the Transmon and resonator resonant frequencies, li()) the Transmon drive amplitude, A(t) the Transmon drive detuning, and (t(() the drive-induced stark shift. As determined Zeytinoglu al. (2015), this stark shift is first order quadratic in the drive amplitude, ds - (C). For the Transmon mode consider three levels (9 f) coupled with two level resonator. We neglect self-Kerr terms in the resonator mode as we target single photon pop- ulations where such non-linearities are not significant.
State Population
0.8
91
0.6
(1}
92)
0.4
e2)
94)
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
OC Solution State Evolution
1.0
The Lindbladian for the Transmon reset simulation given by
0.8
91
0.6
(1))
8 ;= - (Hs +D[p) [D[p]
(19)
922)
0.4
(2))
with k. describing the resonator decay rate, and I' the Trans- mon decay rate.
1--)
0.2
We construct the Transmon reset environment to match the physical parameters in Egger et al. (201 with maximum drive amplitudes of 330 MHz, however with an additional small detuning control of up to 1000 kHz for frequency corrections. To represent bandwidth constraints, we add Gaussian convolution of duration 14ns to the amplitude and detuning defined (20). We use the same reward function as in previous environments with a calibrated max limit of 900, and we neglect the pulse area penalty.
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Time / uus
max-steps steps
We first optimise the reset for higher qubit lifetime of Ti 500 us, representing the Transmon lifetimes currently attainable in experiment. Optimal waveforms and corre- sponding Transmon populations are shown Fig. 4, where the RL Pulse can achieve fidelities of 0.9997 even with realistic bandwidth constraints. Notably, we find the RL agent consistently produces Gaussian-suuare like waveform for the drive amplitude, satisfying the high amplitude reset rate and optimising its smoothing. Novelty is observed in the time-dependent detuning, which first stays at constant frequency throughout the drive until at reset a quick shift is observed from negative to positive. This results in the overall waveform correcting dynamic stark-shifts induced
evolution of the quantum states do not offer such a clear interpretation of the optimal time dynamics of the system.
20

---

Reinforcement Learning for Quantum Control under Physical Constraints
Stokes Amplitude
Pump Amplitude
Rabi Frequency MHz
30
400
200
10
S2p
0.0
0.2
0.4
0.0
0.2
0.4
Frequency Detuning / MHz
Stokes Detuning
Pump Detuning
0
3600
Ap
-50
3500
100
3400
As
-150-
3300
0.0
0.2
0.4
0.0
0.2
0.4
Time / ps
Time / jas
Figure 10: We show optimal signals for two photon Rydberg gate directly realising a C-Z gate, with amplitudes (i.e. Rabi frequencies) for Stokes and Pump pulses in MHz shown in the top row. The effective maximum Rabi frequency of the pump pulse (22p./2 Ar) is x 20 match that of the Stokes pulse. Detunings of the Stokes and Pump pulse are shown in the bottom row. Note the symmetry of the Stokes detuning in time which shows semblance of reflection symmetry about its centre which ensures that relative phase is acquired between the basis states (cf. (17)) and their populations largely return to their initial values. This pulse yields fidelity of 0.99917 for 0.5ps duration and can be shortened to 0.25μs with all signals re-scaled by which yields a fidelity of 0.99958 since we are mainly Rydberg level lifetime limited, with a finite blockade strength of 500MHz. This pulse also shown to be robust across variety of noise levels in amplitude and detuning.
21

---

Reinforcement Learning for Quantum Control under Physical Constraints
Comparison of Baselines: Mean Infidelity
vs Update Steps
10
10-1
1
1
10-3
0
1000
2000
3000
4000
5000
6000
RL Update Steps
PPO
TD3
DDPG
Vanilla PPO
Vanilla TD3
Vanilla DDPG
Figure 11: Comparisons of different RL Algorithms for Two Photon Rydberg Environment. As in Fig. 2 We compare three model-free RL algorithms - PPO (Schulman et al., 2017), DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018). Each algorithm is run on the same Nvidia V100 GPU, using single set of optimised with common batch size (one s.d. shaded) for the same number of total steps. We compare standard (vanilla) versions of the algorithms with the reward function -= F, our constrained formulation, using the reward defined in Eq. 2 and incorporating step-size penalties (cf. Eq. 3). Vanilla PPO still performs reasonably, but it should be noted that it generates solutions which are extremely difficult to realise experimentally because of very high instantaneous bandwidth requirements, therefore, motivating a constrained formulation which also improves computational speed as shown in Fig. 1. PPO achieves the best overall performance, surpassing DDPG and TD3 by several orders of magnitude mean infidelity. Furthermore, within a 24 hour GPU runtime period we could not achievable any reasonable fidelities with the alternative RL algorithms.
per method,
by the drive amplitude fall time, allowing for near ideal reset fidelities.
with an additional Heaviside detuning profile as seen in Figure 4 which applies frequency shift during the ring- down of the amplitude pulse.
When reducing the Transmon lifetime to Ti 48 as used in prior experimental work, the RL agent produces a similar waveform that achieves 0.997 fidelity matching the ideal calibrated square evolution, and achieving higher results than a calibrated square pulse which gets 0.992 and the experimental results in Egger et al. (2018a) which achieved 0.983. The success in optimising over range of Transmon Ti lifetimes demonstrates that high fidelity unconditional reset can be achieved on current Noisy Intermediate Scale Quantum devices with advanced pulse control.
This pulse, which we dub Hcaviside-Corredd Gaussian Square (HCGS), directly corrects for a Hamiltonian which includes a drive-dependent stark-shift. Due to the finite ring- up time required for the amplitude, a negative frequency shift is applied to correct the positive amplitude-induced shark-shift. The negative frequency shift is applied through- out the reset until the ring-down. Before the ring-down of the square pulse, the Heaviside profile produces a positive detuning to correct for the negative amplitude-induced stark shift.
We further verify the RL solution quality in the context of more significant Gaussian-see -eaoohhing kernel of 25ns and qubit Ti 500gss, and find that it achieves high fidelities of 0.9995 while standard square calibrated waveform deterio- rates further to 0.9944 as errors arising from the uncorrected stark shifts become more significant.
We note that this profile behaves quite similarly to past protocols such DRAG where additional phase com- ponent can be added to correct for unwanted Hamiltonian terms in the system. To further account for frequency band- width limitations, i.e. finite rise times for the phase control, the Gaussian Square duration fg and the Heaviside switch time i can be at different points, with the Heaviside typi- cally occurring few nanoseconds earlier account for the amplitude-driven stark shift.
E.1. Heaviside Corrected Gaussian Square
For the f0) <> [g]) transition in the reset process, the RL agent consistently finds a Gaussian Square pulse for the drive amplitude which reminisces of prior works, however
Overall the HCGS reset pulse only requires 4 parameters, the amplitude Gy and duration to of the Gaussian Square,
22

---

Reinforcement Learning for Quantum Control under Physical Constraints
along with the detuning magnitude Ag and the Heaviside switch time 41- The calibration procedure for Gaussian square pulse parameters has been detailed in previous works (Magnard et al., 2018; Egger et al., 2018a). Therefore, to calibrate the HCGS reset for real-world implementation of RL-optimised waveforms, only an additional sweep over detuning magnitude and switch time is needed.
DDPG.
G.2. Hyperparameters for RL Algorithm Benchmark
Hyperparameter
PPO
TD3
DDPG
ACTIVATION
relu6
relu6 false
relu6
ANNEAL.LR
false
false
CLIP_EPS
0.2
F. Experimental Feasibility
ENT_COEF
0
GAZ LAMBDA
0.95
Although we focus on simulations of physical experiments we want to highlight below that it is physically feasible to implement our RL approach on real physical devices with fast gate times < 1us in feasible time frames.
CAMMA
0.99
0.99
0.99
LAYER_8I2E
256
256
256
LR / LR_ACTOR
0.0005
0.0003
0.0003
LR.CRITIC
0.0003
0.0003
The amount of measurements of RL steps required until convergence, specifically, for the
function of the amount
MAX_GRAD_ NNRM
0.5
MINIBATCH.SI2E
32
32
Lambda system and Transmon system it is Nupdates b where b is the batchsize. For the noise free Lambda and Transmon system, this number is: 5100 256 and 578 x 256. For the Rydberg system, the physical number of measure- ments is m Nupdates * (where m = which an overhead associated with the quantum state reconstruction), for the Rydberg environment this number is: 4 6800 256.
NUM ENVS
256
256
2.56
NUMMEIIBATCHES
8
NUM STEPS
NUMLUPDATES
30,000
30,000
30,000
UPDATE.EPOCHT
4
BATCH_SIZE
256
256
BUFFER SIZE
100,000
100,000
EXPLORATION.NITSS
0.15
0.15
In Baum et al. (2021) the authors show that they generate complete gate sets on real device with O(10)) measure- ments (i.e. individual steps) per gate, with gate times on the order of 100ns. The reported experimental runtime in (Baum et al., 2021) is on the order of hours, with API calls dominating this time overhead, showing the feasibility of scaling large measurement numbers in real devices.
NOISE.CLIP
0.5
POL. PRRQ
POLICY.NOISE
0.2
TAU
0.01
0.01
LEARNING INGSSARSSS
1,000
1000
YFLCORF
0.5
Table 2: Comparison of RL hyperparameters for comparison
To reduce the number of experimental measurements, one can use pre-trained policy from simulation which is then adapted to account for any possible mismatch between sim- ulation and physical device parameters. In Ref. et al. (2025) the authors show that this reduces the number of on-device RL steps few thousand.
of PPO, TD3, and DDPG in Figs. 2 and 11.
We describe the RL
for our algorithm comparisons in Tab. 2. The vanilla algorithms use loss function which only contains linear fidelity term F. The constrained formulations use loss function as defined in Eq. 2 with appropriate smoothing arperpparaeetes described in Fig. 13 and max step size as inferred form Fig. 1.
G. Implementation Details
G.1. Training Implementation Details
G.3. Benchmarking of Simulation Speed
We leverage the Qiskit-Dynamicn Solver interface (Puzzuoli et al., 2023)² for constructing both Hamiltonians and col- lapse operators, enabling the simulation of open quantum systems through the dissipative master equation. We employ the Diffrax ODE solver (Kidger, 2022) for quantum system simulation, which utilise adaptive step-sizing techniques to efficiently integrate the first-order linear differential equa- tions, PureJAXRL for implementing PPO algorithms (Lu et al., 2022) and CleanRL (Huang et al., 2022) for TD3 and
Benchmarking absolute compute times across different hard- ware platforms, such as CPUs and GPUs, is challenging due to both systematic and random variations, even within the same architecture. Nevertheless, Fig. 12 highlights the ad- vantages of GPU parallelisation for quantum simulations. We observe up to two-order-of-magniiun improvement in speed per environment step, showcasing the significant performance benefits of running parallelised quantum simu- lations on GPUs, despite potential variability in the absolute timings.
"Similar functionality is available in Dynamigs with compara- ble performance characteristics (Guilmin et al., 2024).
Moreover, we compare the runtime of the different algo-
23

---

Reinforcement Learning for Quantum Control under Physical Constraints
A System Simulation
Rydberg Two-Photon Simulation
Transmon System Simulation
10
Noise
10P
Niiior
102
Noiss
OU Nons
OU Noiso
OU Nooe
Time per Eav Eavinoneecl Ins
Nuivn Frov (CPU Timing)
Time per Eav Arronmnet Ims
Neiar Froe (CPC Timing)
Noiss Free (CPU Timing)
OU Nosso (CPU Taming)
OD Noise (CPU Timing?
107
(CPL Tisiug
10P
Time tas
10
107
10'
30a
10P
10-3
1P
10-3
10000
20000
30000
10000
20000
30000
0
10000
20000
30000
Size
Batch Size
Barch Sine
We compare the time per environment step for Qiskit Dynamics simulation across multiple environments (A system, two photon Rydberg gate and Transmon) under noise-free and Orstein Uhlenbeck noise conditions. The left panel shows the A system simulation timings, while the right panel illustrates the Rydberg two- two-photon simulation timings on a V-100 Nvidia GPU where we parallelise the simulation of several environments with different random actions and fixed number of ODE solver steps= 4096. The solid lines represent the simulation times obtained with GPU, while the dashed and dotted horizontal lines indicate the corresponding CPU timings (Apple Silicon MI) for Qiskit (noise-free and O-U noise, respectively). Simulation time per environment is plotted on logarithmic scale and in the best case we get up to about two orders of magnitude improvement in simulation time per environment in a larger batch by moving to a GPU.
Figure 12:
Algorithm
Time to Conv. (s)
G.4. Signal Processing & Analysis
Our PPO
239.1
The reinforcement learning (RL) agent samples actions from the interval [-1, 1]. For the Rabi frequencies Slpys, these actions are rescaled to the interval (0, 1] ensure all ampli- tudes remain positive and real. Phase variations are already accounted for through the optimisation of Ap/5. No analo- gous rescaling is applied to the detunings A,
Reinforce RL (Brown et al., 2021) OC (BFGS) (Giannelli et al., 2022b)
2284.8
274.08
Krotov (Goerz et al., 2019)
3997
Table 3: Comparison of convergence times for different control algorithms which are benchmarked in Tab. 1 in the main text. Most of the algorithms we benchmarked did not run on GPU so we provide a CPU benchmark here (Mac M1 2020). We use a batch-size of 16 for Reinforce and our PPO implementation. We note that our implementation is largely faster due to the use of just in time compilation and automatic differentiation (Bradbury et al., 2018), but some speed up also comes from using a step size constraint on the ODE solver.
Each action whether an amplitude Ω or a detuning A., denoted a, is subsequently normalised by its respective maximum value, Qmax or Amax.
To encourage discovery of experimentally realistic pulse shapes, we apply additional smoothing and rescaling op- erations. The simulation timescale is fixed: 1 ps for the A system, 0.5 μs for the Rydberg atom, and 0.2 μs for the Transmon. All control signals are expressed in units of MHz. The parameters A and 22pys are discretised into 50 time steps for the A system and Rydberg atom, and 100 time steps for the Transmon. This discretisation provides balance between control flexibility and computational efficiency. The actions a; are smoothed with Gaussian convolution (a Gt) = f (t))((tT]) dr, where the Gaussian function G(t) is defined as:
rithms benchmarked in Tab. 1 in the main text in Tab. 3 on CPU and note that our implementation is faster. The RL algorithms implemented in Fig. 2 all ran on the same GPU and exhibit very similar compute times, hence we did not include an explicit timing comparison (the same holds for Fig. 11). However, we note that on the same GPU our constrained PPO formulation achieves > 0.99 fidelity in 39s compared to 4945s for DDPG and 4517s for TD3.
12 2 12
G(t) = N exp
(20)
where t, defines the standard deviation and its value corre- sponds to the strength of the convolution filter. An ablation over this is provided Fig. 13. This ensures that the
24

---

Reinforcement Learning for Quantum Control under Physical Constraints
A-System
Two Photon Rydberg
Transoo
10a
10
100
4-
10
10-1
10
10
10-2
-2-3
10-
10-3
10
0.001 0.01 0.1 10-1
10-1
0.001
0.01
0.1
0.001 0.01
Figure 13: Ablation over smoothness penalty coefficients WA wp and filter standard deviation l, for three different environments. Choice of smoothing parameters important for learning policies with low mean infidelity 1 1- (averaged over 64 parallel environments). For some systems, like the A system higher filter s.d. leads lower infidelity, whilst for other like the Transmon higher smoothing penalties lead to lower infidelities. The relationship to experimental feasibility with limited bandwidth electronics is analysed in Fig. 14.
Mean Inst.
Frequency Change
Max Inst.
Frequency Change
28
5
225
24
200
175
MHz
150
3-
+ 3-
125
Mean
Max
12
100
8
2-
4
50
1
25
0.001 0.01
0.1
0.001
0.01
0.1
UAA 2012
WA
Figure 14: Using the optimised control signals for the A-system obtained from the smoothing hyper-parameter ablation study (Fig. 13), analyse the instantaneous frequency change between successive output samples. This quantity is defined as dsample where = Qit) cos 2TT Jo A(T)dr), and the signal S(t) is digitally sampled at IGSa/s, corresponding to ofate-of-the-arr arbitrary waveform generator (AWG) capabilities (Keysight, 2024). We plot the mean (left) and maximum (right) instantaneous frequency changes across consecutive samples. Our results show that lower smoothing penalties and smaller kernel standard deviations lead to significantly larger maximum and mean instantaneous frequency variations. Instantaneous frequency changes of several hundred MHz pose serious challenges for experimental realisation, as changes exceeding 200MHz between samples are likely to induce severe signal distortions and fall outside the operational bandwidths of typical digital-to- -oatall--- converters (DACs), amplifiers, and other components in the signal chain. In contrast, signals with maximal instantaneous frequency changes of 100MHz are far more compatible with standard experimental hardware. Thus, addition to improving computational efficiency, the application of smoothing constraints is critical for ensuring that the optimised control signals are easily experimentally realisable. Similar results are found for the other physical systems but they further presented in favour of brevity.
25

---

Reinforcement Learning for Quantum Control under Physical Constraints
generated time-dependent control signals are smooth and give rise dynamics which can be solved in fixed number of time-steps, particularly at the beginning of the learning process when signals are randomly initialised. Pulse ampli- tude ends are always fixed at zero to ensure experimental viability with finite rise time effects, as signals cannot in- stantaneously start at non-zero amplitudes. Additionally, we use cubic spline interpolation (or linear interpolation for the Transmon) between action samples which is efficient for use with adaptive step size solver used for solving the master equation in different environments.
do not attain signals which exhibit low enough smoothness. The importance of generating smooth functions is three-foldd fold. Firstly smoother waveforms are easier to experimentally implement with electronics with limited instantaneous band- width, as well as finite modulator rise times, and they are less vulnerable to signal chain delay or timing issues. This is shown in Fig. 14, Secondly, they are more interpretable in terms of the time evolution of the different quantum states. This is shown in Fig. 9. Thirdly, increased smoothness sig- nificantly speeds up the adaptive step size solver time which is particularly advantageous when working with limited computational resources or larger quantum systems.
The pulse smoothness is defined in terms of different pulse smoothness functions. The first smoothing function is con- structed by calculating the second derivative of A(t):
Choosing the right smoothness penalty in the construction of the reward function is important as can determine the learning speed and the extent to which realistic and inter- pretable controls are generated. We find, that a low-pass filter approach with the right cut-off frequency generally works well and provides the fastest learning of smooth sig- nals as shown in Fig. 15. Other simpler smoothness func- tions such as the 41 or L2 norm are not considered because they were less well adapted for finding smooth signals that solved the quantum dynamics problems with finite number of maximal adaptive solver steps.
/-²
1
A dt2
Sd (alr)a
dt.
(21)
An alternative smoothing function is defined in terms of the difference in output to that generated by low pass Butterworth filter (Butterworth, 1930). This requires an expression of the filtered action which is the convolution of Att) with the impulse response h(t) of the Butterworth filter:
Picking the right hyper- hyper- parameters for the Gaussian convo- lution filter standard deviation t, defined in (20), as well as the right smoothing penalties wA and wi (cf. (3)) is cru- cial to ensure the optimal trade-off between smooth signal discovery to facilitate parallel optimisation, improved inter- pretability and discovery of high fidelity solutions. Overly strong signal smoothing smoothing penalties result in the optimiser focussing largely signal smoothness over fidelity of the quantum control task which is the primary objective. This is shown clearly in Fig. 13, where the A system benefits from higher strict smoothing in form of a larger Gaussian kernel and higher weak smoothing in form of a larger pulse smoothness penalty, compared to the two photon Rydberg gate.
Anterr (tuteet)== = ( A)(t)) = [[" h(t h(t h((=))A(T) dr (22)
Calculating the difference with respect to the unfiltered signal, we get an expression for the low-pass smoothness with respect to cut-off frequency wmaz and the filter order Norder:
Sip(a(t), Rorder Norders Wmax) /.' (antee(tt) (t) a(t) dt. (23)
It shall be noted that since all signals are discretised, the inte- grals decompose into discrete sums. The reference smooth- ness for an action is given by (B((t)) where B(t] is the Blackman window comprised of samples where n also defines the number of signal samples corresponding to P2 or A.
A final objective which competes with the fidelity, are the pulse areas A(22, and implicitly the pulse duration. Qmaa is limited physically by laser, RF or microwave power. Ad- ditionally, minimising pulse area is important for reducing the pulse energy and turn the amount of heat introduced into the system, particularly for those quantum systems op- erating at cryogenie temperatures. Generally faster pulse sequences increase the clock cycles of a particular quantum operation which is desirable, but secondary to their fidelity, so implementing optimal control for some maximal ampli- tude 4maz but with a minimal pulse area is considered in the example of a system. The baseline pulse area (cf. (3)), which is particularly relevant for the results shown in Fig. 7 and Fig. 3 is computed by comparing the generated pulse area = I (2()) to the area of Blackman window
0.42 - cos
cos
2mn N-1
+
B[n]
0.08 cos
4nn
osns E
N-1
0,
otherwise.
(24)
This choice is made as it is designed to have minimal spec- tral leakage, which means it suppresses high-frequent com- ponents effectively and mimics the smoothness of the sig- nals that we are looking for. Penalising pulse smoothness is required because even after applying a convolution filter, we
26

---

Reinforcement Learning for Quantum Control under Physical Constraints
Varions Smoothing Functions for Multilevel Lambda System
Liaee SDet
32 Panilel Eavos
Sike Liss
L.Phoo1 Pase
RL. Updates
Figure 15: Comparison of different smoothing functions for mean infidelity (1- F)) across 32 parallel environments with different seeds plotted against the number of RI. Updates for multi-level Lambda system. The legend corresponds to the type of smoothness penalty used where the ordering of the labels describes the amplitude and detuning smoothness functions respectively. One can observe that particularly for the amplitudes S2 using low pass filter (LPass ef. (23)) instead of second derivative penalty (SDer ef. (21)) allows for significantly sped up learning and also higher mean fidelity. For this ablation, the smoothness penalties wn = wA are fixed to 0.001.
Ap defined over the same timescale.
G.S. Noise Model
We use an Omstein-Uhlennheck noise model defined with standard deviation a and mean which defines time- dependent noise in time t:
4 - Vo +
(25)
where a defines the characteristic time scale of the noise fluctuations and (t) is random Gaussian noise at time with standard deviation of and mean of O.
27


Reviews:
[{"review_id": "EqMp6WoN8i", "reviewer": "Reviewer_sD3o", "summary": "The paper proposes a physics-constrained reinforcement learning algorithm to explore physically realizable pulses for quantum control tasks. The constraint on the pulses ensures smooth transitions and low energies, which result in noise robust pulses that may achieve higher fidelities.\n\nComprehensive experiments are conducted on three quantum control tasks and three quantum architectures, demonstrating the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer sD3o\nWe appreciate your positive feedback on our work and are glad that our research in AI for quantum science was well received.\n\n### Weaknesses\n> \"The proposed method requires precise calibration of the target system, and after each calibration the RL algorithm needs to be rerun to obtain high-quality pulses. I wonder, when integrated in a realistic system where calibrations are conducted routinely, how will the proposed RL algorithm perform (in the sense of run time and stability) in real devices.\"\n\nA1.1: We thank the reviewer for this insightful question regarding the handling of system parameter changes after re-calibration. For small drifts the inherent robustness of our learned policy (achieved via training in noisy environments) can avoid requiring retraining. For large deviations in system parameters, our method can be adapted by training an RL policy that conditions on the current system parameters. This policy is trained by sampling environments with varying system parameters, enabling it to adapt to novel system parameters when deployed.\n\nWe would like to point out an additional scenario that we observed in our experiments, where the learned policy can be attributed to symbolic equations (Appendix Section E.1), as is the case for the learned transmon reset pulse. \nOnly requiring four parameters to define, the learned pulse can be easily tuned in a physical experiment and does not require re-training.\n\n> \"The experiments in the paper are conducted with numerical simulations. For a technique targeting quantum control, it is essential to validate the method on real quantum devices... hard for the authors to access the limited hardware resource, while I still want to know if the proposed method can be applied to accessible commercial quantum devices, i.e., IBM’s devices.\"\n\nA1.2: Unfortunately, IBM does no longer allow pulse-level access to their devices. For future work, we are investigating alternative hardware providers which offer pulse level device access, including Rigetti [1] and IQM [2] to verify our solutions. Furthermore, we want to highlight that prior work found good agreement between numerical simulations of and experiments with transmon qubits [3,4].\n\n> \"Following point 2, the constraint design in the framework is not convincingly justified in real device experiments. In this sense, the paper lacks an important discussion on the criteria for selecting these constraints...\"\n\nA1.3: Thank you for this remark. We refer to the introduced method as \"physics-constrained\", as all constraints are derived from real, physical limitations.\n\nFirst, we constrain signal bandwidth (smoothness), and signal area, reflecting the limited instantaneous bandwidth of electronics and signal components in experiments, and limitations in available signal power and duration. We implement these constraints via the reward function (Eq.2), similar to the Lagrange Multiplier technique introduced in [5]. \n\nSecond, we constrain the policy to solutions that can be simulated within a predefined number of maximum solver steps. This constraint incorporates priors about the physical solution time scales into the algorithm (see lines 209-13), which incentivises adiabatic quantum state dynamics, yielding robust and interpretable solutions. Additionally, this constraint incentivises smoothness (requiring lower bandwidth) of signals, as smooth signals generally require fewer solver steps. Furthermore, the constraint on maximum solver steps significantly reduces computational demand.\n\nWe will include this explanation in Sec. 4.1 in our revised manuscript to make it clearer for the reader. We will also add a figure in Appendix Sec. F.3 which shows how smoothing constraints translate to real signal bandwidth limitations using an FFT analysis.\n\n>  \"Since the RL algorithm relies on a full simulation of the system, it is hard in its current form to scale up, i.e. to more than 10 qubits.\"\n\nA1.4: We agree with this observation. However, one could replace a full system simulation with an ML based emulator, or direct physical device sampling [6]. Given such improvements, our proposed method could scale to larger systems. We also want to highlight that control of smaller dimensional systems is an important and relevant research direction for advancing quantum technologies.\n\n> Minor comment\n\nThanks for spotting this, the third term intentionally has the same structure but there is a typo $S(\\Omega_i)$ should read $S(\\Delta_i)$.\n\n### References:\n\n[1] Rigetti Computing. (2025). Pulses and waveforms. pyQuil 4.16.1. https://pyquil-docs.rigetti.com/en/stable/quilt_waveforms.html\n\n[2] IQM. (2025). IQM Pulla 6.15. https://docs.meetiqm.com/iqm-pulla/\n\n[3] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.\n\n[4] Willsch, Dennis. arXiv preprint arXiv:2008.13490 (2020).\n\n[5] Bhatnagar, S., Lakshmanan, K. J Optim Theory Appl 153, 688–708 (2012).\n\n[6] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "There is no theoretical claim in the paper.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The paper proposes a novel RL framework to generate more robust quantum control pulses, which is a hard task for quantum control and little discussed in the literature of quantum + AI.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Pros:\n\n1.\tThe framework considers a hard balance in quantum control theory: how to find robust pulses that drive the quantum system to a desired state with high precision. This is tackled by an easy but clever combination of constraint design and RL methods to achieve high-robustness and high-fidelity at the same time. \n\n2.\tThe experiments include many interesting cases with multiple architectures, demonstrating the general feasibility of the proposed method. The resulting pulses look impressive in the sense of smoothness and pulse duration, both are essential in the real experiments.\n\n3.\tThe training environment includes a simulation of the Lindbladian of the quantum system, which is precise but demanding computation. The framework makes optimization on the engineering side to maximize the parallelism to speed up the training procedure. \n\nCons:\n\n1.\tThe proposed method requires precise calibration of the target system, and after each calibration the RL algorithm needs to be rerun to obtain high-quality pulses. I wonder, when integrated in a realistic system where calibrations are conducted routinely, how will the proposed RL algorithm perform (in the sense of run time and stability) in real devices.\n\n2.\tThe experiments in the paper are conducted with numerical simulations. For a technique targeting quantum control, it is essential to validate the method on real quantum devices. I understand that it might be hard for the authors to access the limited hardware resource, while I still want to know if the proposed method can be applied to accessible commercial quantum devices, i.e., IBM’s devices. \n\n3.\tFollowing point 2, the constraint design in the framework is not convincingly justified in real device experiments. In this sense, the paper lacks an important discussion on the criteria for selecting these constraints. Though, I do not doubt that the included constraints represent important features of robust pulses. \n\n4.\tSince the RL algorithm relies on a full simulation of the system, it is hard in its current form to scale up, i.e. to more than 10 qubits. \n\n\n\nThe above weaknesses are a bit nitpicking and may be out of scope of the current paper. Overall, I enjoyed reading the paper and like its idea. It is well-executed research in the direction of AI for quantum science.", "other_comments_or_suggestions": "Minor comments:\nThe second and third term in (2) has the same structure, which I believe is a typo.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kEcUgbz3pT", "reviewer": "Reviewer_Vwkd", "summary": "This paper explores the application of reinforcement learning (RL) for quantum control, introducing constraints aimed at improving learning efficiency. The authors present a rigorous approach to adapting RL for quantum applications and provide detailed reasoning behind the necessary modifications. The study focuses on a specific use case within quantum control, integrating a tailored set of constraints and optimizations to enhance RL performance.\n\n### Update after rebuttal\n\nThe authors have provided a detailed and constructive response that addresses all previously raised concerns. They clarified the distinction between their pulse-level quantum control setting and prior circuit-level RL work, appropriately situating their contributions within the literature. Code has been made anonymously available and demonstrates compatibility with standard RL libraries. Additionally, they conducted new experiments benchmarking PPO against DDPG and TD3, showing clear advantages in both performance and sample efficiency. The authors also elaborated on their notion of “interpretable quantum state dynamics,” adding further clarity. In light of these clarifications and additions, I updated the score accordingly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer vWkD\n\nThank you for recognising our rigorous adaptation of RL for quantum control and well reasoned introduction of constraints.\n\n### Weaknesses\n\n> \"Missing discussion of prior RL-based quantum computing research\". [...] \"... Prior work such as [1-4] should be discussed...\"\n\nA2.1: We thank the reviewer for highlighting these works; we will discuss them in the related work of the updated manuscript. We remark that these works do not address the same problem setting but instead focus on circuit-level quantum *compilation* (in a space of logical quantum gates) which is fundamentally different from the pulse-level *control* problem (in a space of physical control signals) that our work addresses. Hence, these works do not constitute approaches that we could benchmark against. We mention in the conclusion that expanding our pulse level control to multiple sequential pulses (i.e. a circuit) would be an interesting extension.\n\n> \"No code provided, reducing reproducibility and practical applicability.\" [...] \"The absence of released code raises concerns about reproducibility and the ability to benchmark against alternative methods.\"\"\n\nA2.2: We acknowledge the reviewers' concern and are committed to reproducibility. To allow for a better review process, we have anonymously published the code under https://anonymous.4open.science/r/RL4qcWpc/README.md. We will release all code publicly upon acceptance, as stated in Section 5.0.1.\n\n> \"The paper would benefit from additional benchmarking against existing RL-based quantum approaches...\"\n\nA2.3: As pointed out in A2.1 above, the highlighted related works [1-4] do not address the same problem setting as our work. \n\nWe are confident that our work includes empirical comparisons to all relevant baselines and politely ask the reviewer to specify if there were any missing comparisons.\n\n> \"It is unclear whether the implementation supports standard RL frameworks, which would facilitate broader testing.\"\n\nA2.4: Our implementation supports any RL framework, as can be seen in the published code. To emphasise this, and to further motivate the choice of PPO, we have now conducted an ablation study on replacing PPO by DDPG [5] or TD3 [6]. Our implementation of PPO achieves fidelities of over 0.999 for the experimental setup described in Sec. 5.1, whereas both DDPG and TD3 achieve maximum fidelities of 0.995 and 0.994 and take 100x longer to reach >0.99 fidelity on the same GPU. See A4.1 in response to K5by for more details.\n\n> \"Details on the specific RL algorithm employed and justification for its suitability to this problem are lacking.\"\n\nA2.5: Thank you for this feedback, we have added motivation for using PPO, an on-policy algorithm chosen for stability (less hyperparameter sensitivity) over DDPG [5] or TD3 [6]. Furthermore, we show that in our quantum control setting, PPO significantly outperforms alternatives, see A2.4 and A4.1 for more details.\n\n> \"The paper is heavily skewed towards quantum-specific adaptations, limiting its ML relevance for ICML\".\n\nA2.6: We believe that the paper's focus on physics-constrained RL for Quantum Control exactly fits the scope of ICML defined in the call for papers. Specifically, ICML calls for “Application-Driven Machine Learning [...] driven by needs of end-users in applications [...] such as physical sciences.” Also, see A4.1 for an ML specific benchmark.\n\n### Questions\n> \"What do the authors mean by \"interpretable quantum state dynamics\" in this context?\"\n\nA2.7: It refers to the clear, interpretable time evolution of a quantum system under optimised control (e.g. minimal excited state population in a Lambda system), unlike complex, fast dynamics induced by many unphysical optimal control solutions. A plot contrasting these will be added to the revised manuscript.\n\n> \"Do the authors plan to release their implementation... will it support standard RL libraries for benchmarking?\"\n\nA2.8: As noted in A2.2, the code is now anonymously available and supports standard RL libraries, shown via ablations with DDPG [5] and TD3 [6] vs. PPO (see A2.4).\n\n### Concluding remarks\nWe kindly ask the reviewer if we were able to address their concerns by (1) clarifying the difference to [1-4], (2) making our code available, (3) demonstrating adaptability through comparisons with other RL algorithms. In light of these clarifications and additional results, we politely ask the reviewer to consider updating their score, or to point out further concerns.\n\n### References:\n\n[1] van der Linde et al., 2023: RL-based quantum compilation benchmarking\n\n[2] Altmann et al., 2024: Challenges of RL in quantum circuit design\n\n[3] Kölle et al., 2024: RL environment for quantum circuit synthesis\n\n[4] Rietsch et al., 2024: RL for unitary synthesis of Clifford+T circuits\n\n[5] Lillicrap et al. 2016, 'Continuous Control with Deep Reinforcement Learning', ICLR 2016\n\n[6] Fujimoto et al. 2018, 'Addressing Function Approximation Error in Actor-Critic Methods', ICML 2018", "claims_and_evidence": "The authors make several claims regarding the effectiveness of their constrained RL approach for quantum control. While the paper provides strong theoretical and empirical reasoning for these adaptations, there are notable gaps in supporting evidence:\n- The paper does not sufficiently discuss prior work on RL applications in quantum computing.\n- Details on the specific RL algorithm employed and justification for its suitability to this problem are lacking.\n- The absence of released code raises concerns about reproducibility and the ability to benchmark against alternative methods.", "methods_and_evaluation_criteria": "The proposed methods appear well-suited for the specific quantum control application, with clear justifications for introduced constraints. However, broader ML contributions are limited, as the study primarily focuses on improving RL’s applicability to quantum systems rather than advancing RL methodologies themselves. The evaluation could be strengthened by benchmarking against standard RL methods for quantum applications, incorporating comparisons to prior approaches referenced in the literature.", "theoretical_claims": "The paper does not primarily focus on new theoretical advancements in RL or quantum computing. However, the mathematical formulations appear sound.", "experimental_designs_or_analyses": "The experimental setup is well-explained for its intended quantum control application. However:\n- It is unclear whether the implementation supports standard RL frameworks, which would facilitate broader testing.\n- The paper would benefit from additional benchmarking against existing RL-based quantum approaches to contextualize performance improvements.", "relation_to_broader_scientific_literature": "While the paper provides a rigorous adaptation of RL for quantum control, it does not sufficiently situate its contributions within the broader field of RL for quantum applications. Prior work such as [1-4] should be discussed to highlight differences and advancements beyond existing methods.", "essential_references_not_discussed": "e.g., the following work on RL for Quantum Computing could have been considered: \n- van der Linde et al., 2023: RL-based quantum compilation benchmarking\n- Altmann et al., 2024: Challenges of RL in quantum circuit design\n- Kölle et al., 2024: RL environment for quantum circuit synthesis\n- Rietsch et al., 2024: RL for unitary synthesis of Clifford+T circuits", "other_strengths_and_weaknesses": "Strengths:\n- A rigorous and well-explained adaptation of RL for quantum control.\n- Provides strong reasoning behind constraints and modifications.\n- Addresses a specific and well-motivated quantum control problem.\n\nWeaknesses:\n- The paper is heavily skewed towards quantum-specific adaptations, limiting its ML relevance for ICML.\n- Missing discussion of prior RL-based quantum computing research.\n- No code provided, reducing reproducibility and practical applicability.", "other_comments_or_suggestions": "If targeting ICML, consider emphasizing broader ML contributions beyond quantum-specific constraints.", "questions_for_authors": "What do the authors mean by \"interpretable quantum state dynamics\" in this context?\n\nDo the authors plan to release their implementation, and if so, will it support standard RL libraries for benchmarking?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fMyyMfiU4T", "reviewer": "Reviewer_8yQF", "summary": "This paper introduces a RL approach for quantum control under physical constraints, aiming to improve the fidelity and robustness of quantum control tasks in real-world scenarios. \n\nMain Findings and Results:\n1). The proposed physics-constrained RL algorithm achieves high-fidelity quantum control solutions across three different quantum systems. The fidelities exceed 0.999 across all tested systems, demonstrating superior performance compared to previous methods.\n2). The method shows significant robustness to time-dependent perturbations and experimental imperfections. \n3). By constraining the solution space to exclude control signals that induce overly fast quantum state dynamics, the algorithm improves computational scalability.\n\nMain Algorithmic/Conceptual Ideas:\n1). Physics-Constrained RL: The core idea is to incorporate physical constraints (e.g., maximum number of simulation steps)  directly into the RL framework. \n2). Reward Shaping: The reward function is designed to incentivize high fidelity while penalizing solutions with large pulse areas and non-smooth control signals, which helps in discovering control policies that are easier to implement experimentally and less prone to errors due to signal imperfections.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 8yQf\n\nWe thank the reviewer for recognising our integration of physical constraints into RL, which achieves high-fidelity, noise-resilient solutions.\n\n### Weaknesses\n> \"The term \"physics-constrained\" is used throughout the paper but is not adequately defined or justified.\"\n\nA3.1: Thank you for raising this point, we refer to A1.3 in the response to sD3o and provide additional detail in A3.7. \n\n> \"The computational efficiency claims are not convincingly demonstrated. \"\n\nA3.2: We acknowledge the lack of timing comparison in our work and provide it here for the results in Table 2 (Sec. 5.1) (with one additional comparison) for the noise-free λ System Optimisation. On a Mac M1 2020 CPU, our highly parallelisable implementation achieves up to 30x faster runtimes for a single run, while also yielding the highest fidelity.\n\nWe further show speedups over the baseline time by up to two orders of magnitudes with GPU based parallelisation in Appendix Fig. 11.\n\n|Algorithm|Time to convergence| Mean Fidelity|\n|-|-|-|\n|Our RL Algo|239.1 s| 0.999|\n|Reinforce RL Algo [T.1]| 2284.8 s|0.93|\n|OC (BFGS) [T.2]|274.08 s|0.89\n|OC Krotov [T.3]|7216 s|0.99|\n\n\n### Questions\n\n>How does the proposed method scale to larger quantum systems with more qubits or higher-dimensional state spaces?\n\nA3.3 Thank you for bringing this up, see response A1.4 to sD3o.\n\n>How does the proposed RL approach compare with other existing RL methods for quantum control tasks? [...] \"the results are not compared to a wide range of existing methods, especially the existing RL methods for quantum control.\" [...] \"For example, [2], [3]\"\n\nA3.4: [2] uses discrete actions, unsuitable for our real-world quantum control problems requiring continuous actions. Hence we cannot benchmark against it, but have added it to our related work section.\n\nUnfortunately, [3] does not provide an open source implementation and the paper does not provide sufficient detail to allow reimplementation, hence, we are unable to benchmark against it.\n\nWe want to highlight that we already included [3] in our related work section. We now include a more comprehensive discussion, highlighting its computationally less efficient approach of learning control signal parameters individually at each signal time step.\n\nWe are confident that our work includes empirical comparisons to all relevant baselines, and we kindly ask the reviewer to specify if any comparisons were missing. We now also include benchmarks against DDPG and TD3 and refer the reviewer to A4.1 in response to K5by.\n\n>How to verify the performance of the proposed approach on real physical systems?\n\nA3.5 Most quantum systems have well established theoretical models. Given the system parameters, learned pulses can be directly applied to real devices; many prior works show good agreement between simulation and experiment [4,5,6].\n\nIn future work, we are planning to extend our experiments to real world devices. \nUnfortunately, IBM has discontinued pulse-level access, so we are pursuing Rigetti [7] as an alternative.\n\n>How well does the proposed RL method generalise to other types of quantum systems?\n\nA3.6 Our model-free approach generalises to any quantum system that can be simulated. Our method can also be extended to black box sampling from real physical devices; the feasibility of this has been shown in [1]. \n\n>How sensitive is the method to the choice of physical constraints such as maximum solver steps? How to choose proper physical constraints?\n\nA3.7 Our ablation study (App. Fig. 12 and 13) shows that larger smoothing kernel standard deviations and smoothing penalties boost fidelity while keeping signals within the bandwidth limits (typically a few hundred MHz) of standard electronics. In the revised manuscript, we will expand on App. Sec. F.3 to explain how we choose physical smoothing constraints that are consistent with electronic bandwidth limitations.\n\nLimiting solver steps improves compute time without affecting optimal dynamics if steps are sufficient. We set N_max based on the adiabatic condition, determined by the maximal effective Rabi frequency (N_max ≳ 1/Ω_eff) and increase it until infidelity significantly drops, as outlined in lines 206–214.\n\n### Concluding remarks\nWe kindly ask the reviewer if we’ve addressed their concerns by (1) adding RL benchmarks and distinguishing our work from existing RL work, (2) including the requested timing comparison, and (3) explaining physics-driven constraints and their effects. With these updates and new results, we politely ask the reviewer to reassess their score or highlight remaining issues.\n\n### References:\n[1] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.\n\n[2] Niu et al 2019, npj Quantum Information\n\n[3] Ma et al. 2023, IEEE TNNLs\n\n[4] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.\n\n[5] Willsch, Dennis. arXiv:2008.13490 (2020).\n\n[6] Zhang, X. L. et. al. Physical Review A, 85(4), 042310.\n\n[7] Rigetti Computing. (2025). pyQuil 4.16.1.", "claims_and_evidence": "The claims made in the submission are not well-supported by clear and convincing evidence. Specifically, the notion of \"physics-constrained\" is not sufficiently developed or explained. This lack of clarity undermines the overall persuasiveness of the paper.\n\n(1) The term \"physics-constrained\" is used throughout the paper but is not adequately defined or justified. The authors mention incorporating physical constraints into the RL algorithm but fail to provide a detailed explanation of what these constraints entail and how they are specifically applied. Without a clear understanding of the constraints, it is difficult to assess their impact on the results and whether they truly enhance the robustness and efficiency of the quantum control solutions.\n\n(2) While the authors present numerical simulations for three quantum systems, the results are not compared to a wide range of existing methods, especially the existing RL methods for quantum control. \n\n(3) The computational efficiency claims are not convincingly demonstrated. The authors assert that their method improves computational efficiency by enabling parallel optimization, but they do not provide detailed comparisons with other methods in terms of computational resources and time. Without such comparisons, it is challenging to evaluate the actual benefits of their approach in terms of scalability and practical applicability.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria need more clarity, justification, and comprehensive validation to be considered appropriate for quantum control tasks. For example, the \"physics-constrained\" approach is vaguely defined, making it not clear to see how it specifically addresses quantum control challenges. The reward function's penalties lack clear justification, questioning their effectiveness in guiding the RL agent. Computational efficiency claims are unconvincing due to the lack of detailed benchmarking against other existing methods.", "theoretical_claims": "Not applicable. The claims made are largely based on the results of simulations and the design of the algorithm, rather than on formal mathematical proofs.", "experimental_designs_or_analyses": "Not exactly. The experimental design and analysis in the paper are based on numerical simulations, which are appropriate for the initial validation of the proposed method.", "relation_to_broader_scientific_literature": "The paper's key contributions are well-aligned with the broader scientific literature on quantum control and RL.", "essential_references_not_discussed": "There are several related works that are essential for understanding the context and significance of the key contributions, but are not currently discussed in the paper. For example, Universal quantum control through deep reinforcement learning (npj Quantum\nInf. 2019), Curriculum-based deep reinforcement learning for quantum control (IEEE TNNLS 2023).", "other_strengths_and_weaknesses": "Strengths：\n（1）The integration of physical constraints into the RL framework is a novel approach that enhances the practicality and robustness of quantum control solutions.\n（2）It is a significant advancement of achieving high fidelity and robustness to noise.\n\nWeaknesses:\n(1) The paper relies solely on simulations without experimental validation. How to assess its practical applicability?\n(2) More comprehensive benchmarking against existing methods especially the existing RL methods for quantum control would strengthen the paper's claims and demonstrate its advantages.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How does the proposed method scale to larger quantum systems with more qubits or higher-dimensional state spaces? \n2. How does the proposed RL approach compare with other existing RL methods for quantum control tasks?\n3. How to verify the performance of the proposed approach on real physical systems?\n4. How well does the proposed RL method generalize to other types of quantum systems?\n5. How sensitive is the method to the choice of physical constraints such as maximum solver steps? How to choose proper physical constraints?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HlegdzqFwK", "reviewer": "Reviewer_K5by", "summary": "The authors address the problem of optimal quantum control using reinforcement learning (RL). Specifically, they define an RL framework that incorporates several real-world physical constraints to enhance performance.\nFirst, they limit the agent’s possible actions to those that require a small number of simulation steps. This, they argue, has two key advantages: \n(i) It improves training efficiency, as the fidelity component of the reward can be computed more efficiently. \n(ii) It biases the agent toward more physically meaningful solutions due to a connection with adiabatic theory.\nSecond, they employ reward shaping to encourage the agent to generate smoother control signals, achieving similar benefits as (i) and (ii) above.\nFinally, they benchmark their approach on several experimental settings and demonstrate strong performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer K5by\n\nThank you for recognising our incorporation of physical constraints, which significantly advance the state of the art, as a valuable contribution.\n\n### Weaknesses\n\n> \"contributions are somewhat limited, as the approach builds on a standard RL framework for quantum optimal control, incorporating two novel techniques (limiting the agent’s action space and shaping the rewards).\"\n\nA4.1 While we agree that our work lies within applied machine learning, our implementation of physics-informed constraints enables a highly-parallelisable learning algorithm for arbitrary quantum systems, achieving state-of-the-art fidelities. This facilitates easy applications of RL to pulse-controlled quantum systems, aiding in the discovery of optimal and robust solutions, significantly surpassing other RL and non-RL methods.\n\nTo highlight our contribution, we have conducted an ablation study in the noise-free Lambda system explained in Sec. 5.1. We replace PPO with two alternative algorithms, DDPG [1] and TD3 [2]. Additionally, we compare to a vanilla version of each algorithm, where the reward function includes only a linear fidelity term (as done in many previous works, e.g. [3]). \n\nThe ablation results (see Table below) highlight the efficiency and effectiveness of our approach in achieving high-fidelity solutions. Notably, none of the vanilla algorithms achieve a fidelity above 0.99. Our PPO implementation significantly outperforms both DDPG and TD3, reaching fidelity >0.99 in less than 1% of the time. In our revised manuscript, we will include this baseline comparison along with a plot illustrating the learning dynamics.\n\n|Algorithm|Time to reach mean fidelity >0.99 (Nvidia P100-GPU)| Mean Fidelity (over 256 batches) after convergence|\n|-|-|-|\n|Constrained PPO (Ours)|39 s| 0.9997|\n|Vanilla PPO|never|0.989|\n|Constrained DDPG |4945 s|0.995\n|Vanilla DDPG |never|0.985|\n|Constrained TD3 |4517 s|0.992\n|Vanilla TD3 |never|0.625|\n\n\n> \"Due to data (interaction) driven nature of the approach this method will require huge numbers of uses of a quantum device and/or unrealistically strong simulators.\"\n\nA4.2 While our method requires a large number of quantum device interactions, it has been demonstrated in [4] that this is feasible in real world systems, given their fast gate times.\n\nWe also want to highlight that most quantum systems have extremely well understood theoretical models, once the model parameters are correctly identified, our method can be applied to real physical devices. This is demonstrated in prior work, where good agreement between experimental data and simulation results is shown [5, 6, 7].\n\n> I am not able to assess how impressive the experimental results are due to my limited familiarity with the field\n\nSee A4.1 for RL baselines. We also wish to highlight that we outperform alternative optimisation methods on the respective problem settings in Sec. 5 with superior fidelities and noise robustness. Moreover, the methods we introduce for pulse-control of physical systems can find application in various fields of physical science such as in Optimising Chemical Reactions [9] and High Energy Physics [8]. \n\n### References:\n\n[1] Lillicrap et al. 2016, 'Continuous Control with Deep Reinforcement Learning', ICLR 2016\n\n[2] Fujimoto et al. 2018, 'Addressing Function Approximation Error in Actor-Critic Methods', ICML 2018\n\n[3] Bukov, Marin, et al. Physical Review X 8.3 (2018): 031086.\n\n[4] Baum, Yuval, et al. PRX Quantum, vol. 2, no. 4, 2021, p. 040324.\n\n[5] Magnard, P., et. al. Physical Review Letters, 121(6), 060502.\n\n[6] Willsch, Dennis. arXiv preprint arXiv:2008.13490 (2020).\n\n[7] Zhang, X. L. et.al. Physical Review A, 85(4), 042310.\n\n[8] Capuano F. et. al. arXiv preprint arXiv:2503.00499 (2025).\n\n[9] Zhou, Z. et. al. ACS Central Science (Vol. 3, Issue 12, pp. 1337–1344). American Chemical Society (ACS) (2017).", "claims_and_evidence": "Yes, the claims are clearly stated, and the provided evidence is sufficiently convincing.", "methods_and_evaluation_criteria": "The experiments appear to be reasonable benchmarks for the problem. However, as I am not an expert in optimal quantum control, I cannot accurately assess the difficulty of these benchmarks or whether the achieved performance represents the state of the art. Nonetheless, they do beat rather involved schemes by a significant margin.", "theoretical_claims": "There are no critical theoretical claims in the paper that I was unable to verify.", "experimental_designs_or_analyses": "The experiments appear to be reasonable benchmarks for the problem. However, as I am not an expert in optimal quantum control, I cannot accurately assess the difficulty of these benchmarks or whether the achieved performance represents the state of the art.", "relation_to_broader_scientific_literature": "The questions are central to control, and extensively investigated; the work is well connected and the connections are listed.\nThe authors highlight the novelty of their work in two key aspects: (i) restricting the RL agent’s action space and (ii) defining a novel reward function. Through these mechanisms, they effectively incorporate physical constraints into the RL framework.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "From a technical standpoint, the contributions are somewhat limited, as the approach builds on a standard RL framework for quantum optimal control, incorporating two novel techniques (limiting the agent’s action space and shaping the rewards). However, if these modifications lead to significantly better experimental results compared to the state of the art, this would still constitute a valuable contribution. Unfortunately, I am not able to assess how impressive the experimental results are due to my limited familiarity with the field.\nA second concern is also honestly raised by the authors in the Limitations and Impact statement. Due to data (interaction) driven nature of the approach this method will require huge numbers of uses of a quantum device and/or unrealistically strong simulators. \nIt seems the only way this would really work would be in the far future, or given access to very good models of the system.\nHowever the latter should be impossible by virtue of the hardness of the simulation of quantum systems.\nSo it is not entirely clear what the more immediate impact of this line of approaches will be. \nThere would clearly be value in pushing ML/RL machinery, but I don't think the strenghts of this contribution are there. We do learn interesting facts about the relevance of rescticting to some simpler-to-simulate actions in the search space, but it is not clear to me to what extent this necessarily implies we will be far from globally optimal solutions.", "other_comments_or_suggestions": "None aside from issues mentioned in Other strenghths and weaknesses.", "questions_for_authors": "None aside from issues mentioned in Other strenghths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The rebuttal addressed the concerns from the reviewers and provided comprehensive feedback. It is useful for assessing the paper's contributions. Most reviewers recommend acceptance after discussion, and the ACs concur. The final version should include all reviewer comments, suggestions, and additional discussion from the rebuttal.
Decision: Accept (poster)