Model-free reinforcement learning with noisy actions for automated experimental control in optics
Authors: Lea Richtmann
Viktoria-S. Schmiesing
Dennis Wilken
Jan Heine
Aaron D Tranter
Avishek Anand
Tobias J. Osborne
Michèle Heurs
Venue: Accepted by TMLR ()

Abstract: Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training on simulations. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC), Truncated Quantile Critics (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. 
We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.

Published in Transactions on Machine Learning Research (07/2025)
Model-free reinforcement learning with noisy actions for automated experimental control in optics
Lea Richtmann" Institute for Gravitational Physics Leibniz Universily Hannover
Viktoria-S. Schmiesing
viktoria www.://sw...eeeeiigii.i..iiiiiee
Institute for Theoretioal Physics
Lesbniz University Hannover
Dennis Wilken
dennis. wilkenaaeiii www.-aiiii.ii--.aneee....d.
Institute for Gravitational Physics
Leibniz University Hannover
Jan Heine
Institate of Photonics
Lesbniz University Hannover
Aaron Tranter
ceron.tranter&anm.ccmm
Quantum Science and Technology
Australian National University
Avishck Anand
avishek. oaanekkk..nadd@@dde..........
Software Technology Department
Delft University of Technology
Tobias J. Osborne
Institute for Theoretical Physics
Leibniz Universily Hannover
Michèle Heurs
michele. hewrs www.srraae..nnnaaaaaaaaaeea
Instibute for Cravitational Physics Leibniz University Hannover
Reviewed on OpenReview: ht tp // en es ew. nt r dA
Abstract
Setting up and controlling optical systems is often challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses, or phases of light makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-frec RL approach that trains directly the experiment without pre-training on simulations. By utilizing the sample-effieiett algorithms Soft. Actor-Critic (SAC), Truncated Quantile Crities (TQC), or CrossQ, our agents learn to couple with 90% efficiency. A human expert reaches this efficiency, but the RL agents are quicker. In particular, the CrossQ agent outperforms the other agents in coupling speed while requiring only half the training time. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.
1 Introduction
In experimental physics, we work with complex and sensitive setups. Working in an optics lab means adjusting numerous mirrors, lenses, and other optical elements while optimizing complex parameters. Two of the
* Both first authors contributed equally

---

Published in Transactions on Machine Learning Research (07/2025)
main challenges are precision and the number of degrees of freedom. Often, tasks have to be repeated frequently. One example of such a task is coupling laser beams into optical fibers, used in many physics labs (Addanki et al. 2018 Lu et al. 2019 Larsen et al. 2019 Xavier & Lima 2020). It can be a laborious and timo-consuming task, especially in experiments with many fibers. Automating tasks like this can, therefore, free up domain expertise for more challenging lasks. Most of these repeated tasks have a very clear goal and can either be described as alignment or control problems. Alignment means the correct steering of a laser beam through an optical setup. Control refers maintaining a dynamic experiment at a desired position using feedback loops. While fiber coupling is primarily an alignment problem, correcting for drift can be considered control.
Automation of alignment and control tasks is a classic use case of reinforcement learning (RL) (Sutton & Vermuccio et al., 2022 RL has seen considerable success in recent years, both
Barto 2018
Krenn et al. 2023
in general Siiver et al. 2017
2018 Mnih et al. 2015; Vinyals et al. 2019; Berner et al.] 2019
Fawzi et al.
2022
Ruiz et al.J 2024) and specifically in robotics (James ct al.) 2019 Andrychowicz et al. 2020
Kumar et al.
2023 Sontakke et al.) 2023 Zhao et al.
2020). However, due to many RL. algorithms relying on a huge amount.
of data, at least in environments with continuous action spaces, most of these were performed in simulated
or toy environments (Akkaya et al.
2019: Ranawecra & Mahmoud 2023). Comparatively few experiments
were done in real-world environments DullccArnold et al. 2019 2021
Ding & Dong 2020). With the recent advance of more sample-efficient algorithms for contimuous action spaces, like Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2019), Twin Delayed Policy Gradient (TD3) (Fujimoto et al. 2018), Soft
Actor- -Criii (SAC) (Haarnoja et al. 2018), Truncated Quantile Critics (TQC) (Kuznetsov et al. 2020 and
CrossQ (Bhatt et al. 2024) directly training in an experiment has become more feasible. However, we still face several challenges, such partial obscrvability, time-consuming training, and noise, when applying RL to real-world setups (Dulac-Arnold et al. 2019 2021).
In this work, we demonstrate how an RL agent successfully learns to couple light into an optical fiber, reaching efficiencies equivalent to those of a human export. We sct up an experiment for fiber coupling on an optical table, motorizing the mirrors that guide the laser beain into the fiber. Our goal is to reach a specific coupling efficiency, which is the fraction of light entering the fiber. We have not included the absolute motor positions in the observation to train our agent to improve the coupling efficiency for any type of misalignment. This means that the partial observability increases, but the agent generalizes to setup perturbations.
The primary issue we encountered was the lack of precision in the motors, For instance, returning to a position was only possible with a considerable and umpredictable offset, which leads to noise in the actions, a special type of stochasticity of the environment. In contrast to adding artificial noise to the actions for exploration (Chiappa et al. 2023 Fbberhadd et al., 2022 Hollenstein et al., 2022: Zhang & Hoof 2021 Plappert let al. 2018), in our case, the noisy actions are inherent to the system. To solve this problem without full analysis and modeling of the behavior, we let our agent train directly on the experiment with the standard BaseleBas (Raffin et al., 2021 implementations of SAC, TQC, and CrossQ. To reset training episode, we could not reliably move to an absolute position but implemented a reset procedure that mainly relied on relative movement steps.
Despite the noisy actions and partial observability of the environment, the agent learns to reliably couple to an efficiency of 2 90% + 2% starting from a low power over the course of two days using CrossQ or SAC and nearly four days when trained with TQC. If we only need smaller effieiency, e-g., 87% + 2%, the training only takes twenty hours and can be performed in two nights, not taking away experimenting time. For comparison, the maximum coupling efficiency observed by the experimenter was 92%, and the one reached by the agent was 93%. We find that tuning the environment parameters thoroughly is crucial to reducing training time, which is of high priority for real-world RL applications.
Our successful training is a first step towards further applications. First, our experiment shows that laser beam alignment using RL is generally possible. A transfer to other scenarios, such as interference optimization alignment of laser field to an optical resonator, is straightforward and
of two beams at beam splitter requires only change of sensor (Sicgman 1986). Secondly, with training directly on the experiment, we show example of applying RL to control tasks without having to model the experiment in detail beforehand.
2

---

Published in Transactions on Machine Learning Research (07/2025)
This is particularly important for more complicated experiments, such as those in quantum and atomic optics, where it may be disproportionate or impossible to simulate the exact dynamics and noise.
2 Related work
Reinforcement learning is widely used in robotics (Singh et al. 2022). In many world robotics environments, the challenge of noisy observations is apparent, while there is less focus on noisy or inaccurate actions (hao et al. 2020). Still, in sim-to-real transfer studies, different forms of actuator imprecision have been shown to significantly affect policy transferability, but agents can still learn in environments with backlash (Golemo ct al. 2018). We identified backlash as a key challenge for our agent, rather than the widely studied problem of noisy observations. As the robotics field is highly diverse and direct comparisons between experiments are generally difficult, we focus our discussion of related work on the subfield of optical experiments.
The application of RL to optical systems encompasses a wide range of topics, including optical networks Kiran 2007: /Surrez-Varela et al.)
et al.
2019
Chen et al. 2019b a Luo et al.] 2019: Troia et al.] 2019: Li et al.
2020
Natalino & Monti 2020 Yu et al. 2023 adaptive optics (e et al.
2019
Nousiainen et al. 2021
Durech et al.] [022 Landman el al. 2021 Nousiainen el al. 2022a b Pou el al.
2022: Nousiainen el al.
2024), and optical nanostructure, thin films and optical layers (Saeedian et al.
2019
Jiang et al. 2020 Wang
ct al. 2021 Wankerl ct al. 2021 h. More related to our problem is work that studies how RL can be used to align and control tabletop optical experiments with lasers. In this category, some works are realized merely on simulation. Examples include studying mode-locked lasers (Sun et al. 2020). combining laser beams (Tünnermann & Shirakawa, 2021), and stacking laser pulses (Abuduweili et al. 2021 Abudweiili & Liu 2022). Other works include investigating how RL performs in actual experiment. Most agents in these studies are, however, not trained in the experiment but in simulation. Examples include aligning an optical interferometer (Sorokin et al.) 2020; Mukund et al. 2023). operating optical tweezers (Pracger let al. 2021), ; combining laser beams (Shpakovych et al. 2021) and operating pulsed lasers (Kuprikov et al, 2022]. It is rare that the agent is trained directly on the experiment (Ding & Dong 2020). One example is a study combining pulsed laser beams (Tünnermann & Shirakawa 2019). Here, one actuator performs the actions, and the output is a scalar, the power. The training time is about 4 hours; simulations show that this would quickly go up to 1-2 days if more than two beams should be combined. Another example is the generation of white light continuum (Valensise et al. 2021). Thereby, both the states and the actions are given by absolute positions of three actuators and moving to those positions, respectively. This gives their environment a relatively high observability for a real-world task. The authors claim to have obtained a successfully trained agent in 20 minutes. We deal with a higher (4) dimensional action space than both of these works. RL was also used to optimize the output power of an X-ray source (Mareev et al. 2023 For this, a single actuator was discretely controlled based on a scalar signal. As a side project, the paper looks simplified approach to fiber coupling using only two degrees of freedom and working with a discrete action space of size 4 employing DQNs (Muih et al., 2013). The work does not present the achieved coupling efficiencies. In contrast, we work with continuous action spaces and control all four degrees of freedom necessary for general beam alignment required for optimal fiber coupling. While training on the experiment can be difficult for many reasons (see Section 3), it can be the last resort in cases where creating a model that accurately represents the noise and dynamics of the system is very time-consuming, if not infeasible. Using too-inaccurate model, however, would make it impossible to cross the simulation-reality gap (Salvato et al. 2021 We therefore decided to study the little-explored field of in-situ training.
3
Fiber coupling
3.1
Experimental setup
To efficientl couple laser light into an optical fiber, we need a specific setup. Our goal is to reach a certain coupling efficiency, which is the fraction of light entering the fiber. To achieve this, the light has to enter the fiber at a specific angle and precise spot. The coupling efficiency depends on how accurately both are matched. To fulfill both constraints, we need two degrees of freedom in each axis, horizontal (x) and vertical Knigge et al. 2021 b. These are the same requirements as for any her beam alignment.
(v) (Senior 2009:
3

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Lenses
Laser beam in
Hand steering mirror
Laser beam in
Reference power meter
MI
Reference power meter
Motorized mirror
Controllers for actuators
Output power meter
Motorized mirror 2
Agent
Fiber
Not part of this experiment
M
(c)
dead-zone size factuator steps
Hand steering
10
mirror
Fiber collimator
10a
Output power meter
Fiber
10
1x
2x
actuator
Figure 1: Panel (a) and (b) show a conceptual scheme and lab setup of the fiber coupling experiment. Panel (c) shows the dead-zone characterization the four motorized mirror mount axes on log scale axis. Dead-zone means movement steps performed by the actuators that do result in a change in power. Appendix gives detailed description of the characterization.
task. This means that two mirrors, each tiltable in and are sufficient to acquire an arbitrary beam alignment. Furthermore, the laser beam must have the correct size, which is achieved by placing lenses the correct position before the light enters the fiber. To simplify the setup, we decided to motorize only the mirrors but not the lenses. In addition to the motorized mirrors, we have two mirrors that can be steered with hand-tuneable knobs. This makes it easier for humans couple light into the fiber and can be used to introduce perturbations to the setup. We measure the power at the output of the fiber with power meter (Thorlabs PM160). With the help of a reference power measurement, we can track power fluctuations and determine the coupling efficiency or normalized output power with an error of 2%. Our experiment is depicted in Figure 1 and further details are given in Appendix A
The four actuators moving the mirrors are stepper motors (Thorlabs ZFS 13). They are attached to the mirror mounts, each tilting the respective mirror in one axis. To understand the special constraints of our problem, move all actuators to position where have maximal coupling. Then, while holding the other three actuators fixed, scan the relevant movement range with one actuator. The power dependence each motorized degree of freedom looks Gaussian. Fitting it with Gaussian, we obtained standard deviations in the range of 10 - 2 x 10a actuator steps (see Appendix C for more details).
3.2 RL challenges
When RL to fiber couple in our lab, face several challenges. The training is timo-consuming environment step takes about one second. Furthermore, due to laser safety and possible equipment damage, we have to restrict the movement range of our actuators. The two challenges that are most crucial for shaping our environment are partial obscrvability and motor improcision.
Partial observability We work with a strongly underdetermined, only partially observable experiment. To describe the state of the experiment perfectly, we would need lot of information not available to us,
4

---

Published in Transactions on Machine Learning Research (07/2025)
e.g., the exact angle, position, and size of the incoming laser beam, well as the exact position of all of the mirrors and lenses. Even if we could get all of this information at the time of training, environmental drift, such as temperature, would require careful calibration to occur frequently. To make the agent robust against drift, we do not use the actuator positions as part of the observation. Even if we did, they would be very inaccurate due lo motor imprecision (see below). Instead, we solely rely on the power at the output of the fiber and the previous actions as our observation. This makes our environment partially observable and underdetermined due to four mirror positions leading to one output. Also, the signal very scarce, as when the motors leave certain movement range, no power at all gets coupled into the fiber, which means the agent does not get any feedback. Hence, the environment is reset when falling below a certain power.
Motor imprecision Our main challenge is based on the complex relationship between the expected movement of the used motors and their actual movement, which we call noisy actions. When we report actuator steps here, these are the steps the controller intended to move the motor. There is no feedback, o.g., an encoder, to ensure the intended position is reached. The imprecision includes backlash of the mechanical system, step loss, non-orthogonal degrees of freedom, i.e. the æ and y-axis are not independent from each other, and other errors. This leads to noise in the action. To understand its severity, Figure (c) shows the number of steps each actuator moves without any change of power, called dead-zone. Although all mirror axes are motorized with the same motor, gearbox and linear actuator, different dead-zone sizes are observed. A more detailed explanation of the imprecision and its characterization can be found in Appendix B The variety in the motor imprecision makes the action noise distribution hard to describe. On top of that, this affects our reset method (see Section 4h.
4 Our method
We cannot write down Markov state (see e.g. Sutton & Barto (2018) for an introduction) for our system. Therefore, treat it as an unknown episodic partially observable Markov decision process (POMDP, see e.g. Graesser & Keng (2220)). ). When sampling from it, we get stochastic process 01, T1, 02, 02, r2...., Or, where oz, dg and It are observations, actions and rewards at the discretized time t, and 7 is the episode length limitited by the maximal episode length T, ie. < T. See Tables I and 2 for environment hyperparameters.
To test out various RL algorithms and investigate differently designed enviromments before training on the actual experiment, we use virtual testbed. This is created by fitting the power depending on the position of each individual actuator with Gaussians. By multiplying them, we get. an approximate map from all four actuator positions to the power. We then set the amplitude to the highest power we observed until that point, which is 0.92. Using this, we create simplified virtual environment, not including noise. Although it is a strong simplification, it helped us get various insights much quicker than in the experiment. These numerical results can be found in Appendix
Actions We treat our action space as the 4-dimensional continuous action space [-1,1]x4, At time t, we can decompose the action a as a = (amir, Omly, 4m2z, Amzyy where cach component belongs to a different actuator. For example, amir belongs to the actuator that tilts mirror 1 in the horizontal (x) direction. Each of these actions is then multiplied by the maximum allowed action actuator steps Omax, rounded to the next integer, and sent to the different controllers. Using the virtual testbed, we find that maximum actions of approximately 6max 6- 100 are optimal (see Appendix C.6). However, in the experimental environment, the actuators have no feedback loop, so, potentially, they move significantly less due to their imprecision. This makes our actions noisy, which adds to the stochasticity in the environment. As discussed in Appendix C.8 this especially complicates the task for high powers.
Observations As we are dealing with a partially observable system, for successful training, we need to take great care in defining our observations. The only thing we observe in our environment is the coupling efficiency or normalized power, denoted P E [0, 1]. For example, Pe denotes the normalized power at time t. As usual in POMDPs (Tünnermann & Shirakawa 2019 [oookiin et al., (020)), we include a history of length e N in the observation. This would lead to an observation like o = (Pi-n...., Pt-1,P)) Pt) at time step t. It is common in RT experiments to observe the environment before and after an action, not during this
5

---

Published in Transactions on Machine Learning Research (07/2025)
Table I: Environment parameters for the main experiments where m'nn - 0.1 is the lowest power at which the training starts, Pran the power at which the agent fails and the reset is called, Pgoat is the power the agent should learn to reach, T is the maximal episode length in time steps, and the history length used in the observation.
Parameter
Pinin
Prail
Pgool
Omax
T n
Value
0.2
0.05
[0.85, 0.9]
6 10
30 4
action. However, this is not the only information available to us: In principle, continuously while the actuators are moving, i.e. record (Pr. Ptt./mes..., Pr+1) during action d, where my is the number of powers measured during that time, In the virtual testbed, we noticed that it is beneficial to use some of this information in our observation (see Appendix C.5). In particular, we take the average power Pave,t = -i Pt-1+ 1f-1t//////aa + 1) the maximal power observed Pmaxx = max,==.....m. Pt-1+i/m, and its relative position in the list of powers Zmax, = (arg max;=0.....n. Pt-1+s/m, + 1) into account. In addition, it is helpful to use the performed actions as part of the observation. This leaves us with the observation
record the power almost
= -(PP-1, 0k-1; Pave, P Pmax,es Fmax, k ( Pi).
i.e., the observation includes a history of the power before taking an action, the action, the average and maximum power and its relative position during the action, and the power afterward. Using the virtual 4 are optimal when using TQC (see Appendix C.5
testbed, we find that history lengths of approximately We deliberately do not take the absolute actuator positions as part of our observation. The main reason is that we want to make our agent robust to experimental alignment changes such as drift. When this happens, the absolute positions where the maximum power is reached will change. An agent trained with the absolute motor positions being part of the observation is not able to handle such situations (see Appendix C.5).
Episode and Resets We reset the environment either after a chosen maximum episode length t= T E N, when the agent reached its goal (i.e., P. > PRoat), or the agent failed (i.e., P. < Pran). While training with TQC on the virtual testbed, we find that it helps with reaching higher goals like Pgoal = 0.9 if we implement instance of curriculum learning (Narvekar et al. 2020) by starting with lower goal powers and raising the goal power during training, especially starting from Pstart, goal 0.85 (sec Appendix C.4).
A common way of resetting at the start of an episode is to move to random position within defined range. However, our actuators do not present the required precision for this. We, therefore, need a different way to reset. We nevertheless define neutral positions given in motor steps. These are positions where we had high power when we started our training. When we return to the neutral positions during training, depending on the original actuator position, the power varies between no power and high power.
The reset procedure depends on the last power value, and we want the power after the reset to be higher than Pmin- We distinguish between cases. If, during the reset procedure, the condition of different case applies, we jump to the corresponding case:
1. P. > Pmin: we choose a random power between Pmin + 0.1 and P Pooal and do random steps until the power drops below the chosen power value.
2. 0.09- < Pt- < Puin: we first reverse the last action. As long as the power is still under Pmin, we move the actuators one after the other in random order in the direction which the power increases. If the power decreases, we change the actuator's direction of movement. We repeat this process until we reach P. > Pmin- 3. Pt- < 0.09 every ten episodes: We first move to the neutral positions. From there, we do random steps until P > 0.09, and then follow the procedure of Case or 2 depending on the power.
The values of 0.09 and ten episodes were determined empirically by observing the algorithm performing on the experiment. Before starting the episode, we always perform some random steps to randomize the process more. Our reset procedure introduces a small dependence between successive episodes. Full independence was not possible in this experiment due to the motors' inherent inacouracies. In contrast, a fully random-based reset procedure is applicable while training solely on the virtual testbed. We compared this with our custom reset method on the virtual best and found major effect the training performance (see Appendix C.3
6

---

Published in Transactions on Machine Learning Research (07/2025)
Table 2: Reward hyperparameters for the main experiments
Parameter
A
A5
Ag
0t
a
0g
8f1
8,2
3gt
3g2
Value 10
100
100
0.9
0.5
0.5
5
5
5
1
Rewards We design our reward with the purpose of making the agent reach the goal as quickly as possible.
The agent gets a low negative reward when failing (P: < Piat), a high reward when reaching the goal (Pr > Pgoat); and else small reward every step depending on the power. We define the reward function depending on the current power Pt, the time step i, the goal power PR Pgoal minimal power Pmins fail power Pfall and the episode length T as
r2 = -R(,, 1, T, Prail P Pgoul, Pmni))
Ay (1 - aa) exp (-3 ay (-3,2 Prail P() Ag (1 11--ay ary) exp (-8 ag exp (Bg.2 P. A T ((1 a.) exp (P, PRoaI)) ) + a, (Pr- Pmin))
if Pt < P
if > P Pgoal
Paoul
else
where 3. B/1, f2, 1r1.3.1..... 292,992,4,00 Ag, A,: Ag E R, ag, E (0, 1). See Table 2 for their values in the main experiment, which were tuned in the virtual testbed described in Appendix C.T The return should never be higher when staying just below the goal than when actually reaching the goal. In the same way, it should always be better to stay just above the failing threshold than to fail. We enforce this by choosing the amplitudes according to Ag 2 A,. Each of the rewards consists of two terms. The a's are used to weigh their importance relative to one another. The failing reward consists of a term punishing it less when the agent fails later in the episode and a term punishing it less when the power with which the agent fails is close to the failing threshold. The two terms in the goal reward ensure that the agent is rewarded more for reaching the goal quickly and for reaching it with a higher power. The step reward contains both exponential and linear part to ensure that the agent clearly notes change to higher powers for low and high values. As the reward depends on the goal power, dividing it by the maximum possible return for that given goal power. The maximum possible return is given by the return when reaching the maximum possible power in the first step.
normalize the return in the shown plots by
Algorithms The contimuous action space limits our choice of algorithm. Our main environment parameter searches on the virtual testbed were performed with TQC. Subscquently, we also tested CrossQ, TD3, SAC, DDPG, PPO, and Advantage Actor-Critic (A2C). Of these, PPO and A2C performed worst. As they both do not use a replay buffer, this was expected. They are closely followed by DDPG. CrossQ, SAC, TD3, and TQC performed much better. SAC performed almost always slightly better than TD3. TQC has small drop compared to SAC in the middle of training that gets larger with rising goal powers. With rising goal powers CrossQ shows an ad advantage over the other algorithms. See Appendix C.7 for discussion. Thus, in the main experiment, we tested CrossQ, TQC, and SAC as algorithms. We used the algorithms from StableBaselines3 (Raffin et al. 2021 with standard hyperparameters further discussed in Appendix
Comparability of the virtual testbed and the real-world experiment The experiment has two main differences from the virtual testbed. Firstly, the actions are highly noisy due to the motors' inaccuracies. Secondly, as described Section 3 the two elements of the action belonging to one axis are coupled. However, in the virtual testbed, to keep the design simple, we uncoupled these actions. The interaction of the different actuators to achieve high coupling efficiencies is therefore easier in the virtual testbed, and for the agent, it is harder to learn working policy in the real experiment. The ablation studies the observation done on the virtual testbed (sce Appendix C.5) are generally transferable to the real-world experiment: as the level of partial observability is similar in both environments and the dimension of the act space is the same the studies of the history length are fully transferable (see Figure a)). The observation design (see Figure I0 b) is also transfcrable, as we have low noise in the power sensors, and saving these values is the same in both environments. Given that the power is the only environmental value utilized in the reward, and considering its low noise, the reward design is likewise transferable.
7

---

Published in Transactions on Machine Learning Research (07/2025)
For both the main experiment and virtual testbed, we used a gymnasium environment (Brockman et al. 2016) and the parameters provided in Tables [ and 2 if not specified differently. Our strategy is that the agent learns to deal with the noisy actions directly in the experiment. We are especially interested in investigating this in-situ learning of noise as, in our arca, we are often dealing with noise sources that cannot be modeled accurately, for example, when dealing with quantum states of light. In these cases, the only solution will be to learn to handle the noise through direct interaction with the experiment.
5 Experimental results in the optics lab
The agent was trained in our lab on components detailed in Section 3.1 Our training speed is limited by the time the actuators take to move, leading to each environment step taking about a second. We train each agent until the return starts to converge as depicted in Figure 2 (a)-(b). For Pgoal S 0.87, this took around 20 hours or 4- 10t steps. That we can train successfully in only 4- 10a steps is a result of the environment shaping discussed in Appendix C If we set a very high goal power (Pgoal = 0.9), the training takes longer: for CrossQ and SAC about 1.2- TOS time steps, which corresponds to 2 days, and for TQC 2 10 time steps, which sums up to nearly 4 days of training.
Using TQC, SAC, and CrossQ, we performed two training runs per algorithm on the experiment with goal power of Pgoal 0.85. In Figure 2 (a), we can see that in the beginning, the return rises quicker for SAC and CrossQ but is slightly outperformed by TQC later on. However, these differences are small.
When we choose Pgmal = 0.9, as shown in Figure 2 (b), we need significantly more training steps. Because of that, as discussed before, we also investigated pre-training using TQC. Although we found pre-training on the virtual testbed with an added noise model helpful (see Appendix D). we focus on in-situ training, as we want to find strategies that can work on experiments where noise model is hard or impossible to oblain. Thus, we implemented pre-training on lower goals directly on the experiment using TQC (light blue), i... started the training with the lower goal power PRoas RRoal = 0.85, increased it in small increments to Pgeal = 0.9 over the course of the first 109 training steps and left it at that for the next 10y steps. The normalized return of the agent first pre-trained on lower powers always drops after changing to a higher goal power, as it first needs to learn to handle the conditions of the changed environment. We can also see that the normalized return reaches lower values the higher the goal gets. This is expected as the task gets harder each time. For TQC, the normalized return for the agent pre-trained on lower goals reaches higher values than the one starting from scratch (groen). Additionally, we can use intermediate agents to align the experiment to lower powers.
The return for SAC (dark pink) reaches return values similar to those of TQC without pre-training. Both are surpassed by CrossQ (brown) and the pre-trained TQC agent, which are again similar. The training time is shorter for SAC and CrossQ than for the respective TQC agents. We conclude that in our experiment curriculum learning was helpful for such a high goal when using TQC (see also Stooke el al. (2021) while this is not needed when employing CrossQ.
To understand the help for our everyday lab work, we tested different agents in fiber coupling (marked with star in Figure 2 (a) and (b)). The test results are shown in Figure 2 (c)-(e). Each of the RL agents was tested a hundred times. We measure the power over time. Tf the agent does not reach the goal during an episode, we reset the environment, and the agent tries from there. One episode was at most 40s, and the longest attempt took around 350s. Figure 2 (c) shows how the averaged coupled power develops in the first s0 Generally, the power rises quicker the first seconds for agents that try to reach only Pgoal = 0.85. Interestingly, even though CrossQ in total faster in achieving the fiber coupling goal, for the first few seconds, the pre-trained TQC agent has the steepest power rise of all the agents trained with Proat 0.9.
Figure (e) shows the number of steps it took for the agents to reach their goal after their last reset, i.e., in the successful episode. In most cases, this corresponds to the first episode, and the agents need no additional resets. More precisely, the empirical probability of successful coupling in the first episode ranges from 95% (SAC) to 98% (Cross()) for Paoal = 0.85 and 83% (TQC without pre-trainn training) to 90% (TQC pre-trained) for P PRoal = 0.9. Of the agents trained with P PRoal = 0.85, the CrossQ and TQC agents behave very similarly, taking only five steps at most in 75% of the cases. The corresponding SAC agent is only half as fast and performs similarly to the TQC agent trained with P Pgoaa= 0.87. For P Pgoal 0.9, the CrossQ agent clearly
8

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(8)
(b)
1.0
1.0
normalized retur (smoothed)
normalized return (smoothed)
0.8
0.6
0.6
Pow 0.85, TQC
0.4
0.4
Pand 0.85, SAC
0.2
0.2
Prout 0.85,
CrossQ
0.0
Pansl 0.87,
-0.2
TQC
Pand 0.9, TQC
training steps (rounded 500)
training steps
(c)
(d)
Pros TQC. pre-trained
1.0
350
time to reach P=0) 0.9 [soconds]
300
0.8
Pansl 0.9,
250
SAC
power
0.6
200
CrossQ
Pane
150
0.4
100
0.0
15
20
25
Cross
SAC
TQC
TQC.
human
time [scconds]
pre trained
(e)
steps to reach Poot in episode
30
25
20
15
10
Pgod = 0.85 CrossQ)
Paou = 0.85,
Paod = 0.85, TQC
Pynui 0.87 'QQ
Post = 0.9 CrosQQ
Paos = 0.9
Pane = 0.9 TQC
Paod 0.9 TQC pre: trained
Figure Experimental results. (a) and (b) show the normalized return plotted against training steps for different algorithms and goal powers. In (a), we use TQC (bluc). SAC (pink), and CrossQ (orange) with Poos Pgoal 0.85 and TQC with Pgoal 0.87 (yellow). Of the agents trained on Peoal 0.9, shown in (b), one TQC agent, shown in light blue, was first pre-trained on lower, over time increasing, goal powers (0.85, 0.875, 0.89, 0.9). The others, ie., TQC (green), SAC (dark pink), and CrossQ (brown), were trained diroctly with Pgost 0.9. (c)-(c) show our results when testing the agents marked with a star in (a) and (b). For testing, we reset and let the agents fiber couple, which is repeated 100 times for the RL agent and 25 times for the human expert. If the RT agents do not reach their goal within 30 steps, we reset the environment (still measuring the time), and the agent continues from there. This repeated until the agent reaches its goal. (c) shows the power plotted against the time for the first 30 The error band is clipped to the power range (0, 1). (d) shows the time each agent trained to P Pgaal = 0.9 and a human expert took to reach that goal. (c) compares the number of environment steps it took for cach of the RL agents tested to reach their goal after their last. reset. (a)-(c) show the (smoothed) mean with 2a error bands created by smoothing and/or multiple runs. The error bands in (c) additionally include a power measurement error of 2%.

---

Published in Transactions on Machine Learning Research (07/2025)
Table 3: Reset by hand vs, automatic reset. We tested the agent (Pgoal = 0.9, pre-trained on lower goals as described in Figure 2 (b)) using two types of resets: using the reset procedure described in Section 4 (automatic reset using the motors) or resetting by tilting the hand-steering mirrors, modeling arbitrary experimental perturbation, 29 times. We compare the mean of the power at the start Po and end of the episode Pr, the empirical probability p(goal] of reaching the goal in the first episode, and mean of the number of environment steps needed to reach the goal ggaal.
Po
Pr
plgoal]
Eoaal
Reset by hand
0.384
0.914
0.86
8.33
Automatic reset
0.465
0.910
0.9
8.22
outperforms the other agents, taking in 75% of the cases at most seven steps. Of the remaining Pgoat = 0.9 agents, the pre-trained TQC agent performs best.
In Figure 2 (d), we show the total time it took each agent to reach goal of Pgoal 0.9. The results are similar to the results from (c) in that CrossQ is fastest, followed by TQC with pre-training, TQC without pre-training, and lastly SAC. For comparison, we also tested a human expert 25 times on how long they would take to couple the fiber to Pgosa = 0.9 using the hand steering mirrors. This, however, is not a fair comparison. The RL agents can change all four degroes of freedom at once. The experimenter, on the other hand, has access to more information, e.g., the continuous power measurement while moving an actuator, and does not have to deal with the imprecision in the actuators, which means they can easily go back to observed maximun. Despite this, can see that the RL agents generally faster, but all besides the CrossQ agent take longer in few cases, where the agent needs several episodes to get to the goal. The CrossQ agent reaches the goal faster than the human expert in 100% of the cases. In conclusion, we show that by using RL, we can consistently couple light into an optical fiber to high efficiencies, despite the noisy actions.
So far, we have shown that our agent can couple light into the fiber after reset using the motors that it has access to but not after general misalignment or drift in the setup. To show that the agent can also compensate for misalignment in other parts of the experiment, we performed the resets by manually misaligning the hand steering mirrors, e.g., tilting them until we were in a coupling regime with low power. Next, we called the agent for realignment. Table 3 shows that the results using this alternative reset method are very similar to the ones with automatic reset. Whether such a misalignment happened at the hand steering mirrors or another element not accessible to the agent, such as, for example, a drift in the position of the fiber collimator, is equivalent in terms of difficulty. Hence, our agent can also be used to control for arbitrary drifts at undetermined location. For this, we can use the almost continuous mcasurement of the power at the output of the fiber and call the agent to set it back to the desired coupling efficiency whenever it drops below a certain value.
6 Summary and Outlook
We have shown that our model-free RL agents successfully learn to couple laser light into an optical fiber, reaching the same efficiencies as a human expert while generally being faster. We find that sample-eficcient algorithms that use a replay buffer, such as CrossQ,, SAC, and TQC, are a must to overcome the challenge of otherwise not managcable training time. The use of CrossQ has a clear advantage. It halves the training time with respect to TQC. Additionally, the trained agent reaches the goal faster than the agents trained with the other two algorithms. Furthermore, our study suggests that curriculum learning can help to achieve more difficult goals.
As we train directly on the experiment, the agent learns to handle the specific noise present, and we can avoid creating an accurate simulation of the task. This makes our method suitable for set where it is impossible to model the noise accurately. Partial observability can be dealt with by carefully tuning the observations.
A central result is that the agent learns to deal with the imprecision of the mirror steering motors. Using a classic algorithm such gradient descent. would fail with these motors as their imprecision deters us from
10

---

Published in Transactions on Machine Learning Research (07/2025)
experimentally determining gradient and then going back to the starting position. One way to handle such imprecision is using motors with internal leedback loops. Using RI., we can avoid this, which helps to simplify the design of motors and experimental setups. Generally, automation gives us the possibility of remote-controlling experiments, which can be especially useful in experimental areas that are difficult to access, such as in vacuum lanks, in clean room lacilities, or, in extreme cases, in underground labs or in space.
In our experiment, we used four motors cover all degrees of freedom and demonstrated the general case of laser beam alignment. Reducing the number of actuators lowers the possible coupling efficiency. In contrast, increasing the number of mirror actuators offers no physical advantage. More complex setups can be divided into parts with multiple mirror-mirroo-scnson blocks.
Further exploration could include investigations into how the agents perform for other transverse optical modes of light, including those with multiple local maxima. Secondly, we start our RL training procedure under low-power conditions. This raises the question of how, or with how many additional sensors, we could generalize this to the case of starting with no power. Furthermore, to reduce training time, investigating the performance of bosel- or hybrid algorithms and other methods to increase sample efficiency is highly relevant. In addition, it might be interesting to explore replacing the use of history our observation by PID-inspired RL (Char & Schncider) 2023) and to investigate whether pre-training on expert demonstrations (Vecerik et al. 2018 Martínez et al.) 2017 Ramírez et al. 2022) as well as a modified procedure of ave-training on the virtual testbed (see Appendix D.3) D.3)
could speed up the learning process.
However, in this study we focus on showcasing direct training on the real experiment.
Our experiments are first step towards the extensive use of RL in our quantum optics laboratory. Optical experiments typically require various control loops to stabilize the experimental degrees of freedom against perturbations. These locks siguificantly increase the complexity of the experiments. For example, maintaining the length of optical cavities to achieve resonant light field enhancement requires complex components such as phase modulators (Drever et al., 1983). homodyne detectors (Hansch & Couillaud 1980: Heurs
let al. 2009), or split detectors (Shaddock et al. 1999: Chabbra et al.) 2021). RL offers the potential for streemlined control loops that rely solely on the measurementt of power in the reflection and transmission of the resonator. This could enable novel control strategies, such as phase control of squeezed vacuum states. These states are characterized by unique quantum noise properties but are otherwise dark. Consequently, phase control lypically requires auxiliary laser field (Vahlbruch et al. 2006) or int roducess unwanted phase noise (McKenzie et al., 2005). RL has the potential to provide a noise-free solution without the need for additional laser fields, which is particularly relevant for large-scale on-chip squeezing experiments in the field of quantum information (Masada et al. 2015).
In conclusion, we show that a common optics task like beam alignment can be solved with standard model-free RL algorithms. For the machine learning community, this demonstrates their versatility. Their availability lowers the barrier for the optics community to use them in other experiments. We showcase how these RL algorithms can be directly applied in the lab, circumventing the need for accurate experimental modeling.
Broader Impact Statement
Our work has no broader societal impact. However, if used correctly it could make work in labs less ressfuu. Also, it can help to increase health situations: leaning over optical tables in uncomfortable positions for many hours to make fine adjustments gives many rescarchers back pain. Using machine learning for automation can lower this hazard. However, we would like to point out that using these methods could also accelerate scientific work even more and thus make it more stressful if the freed-up time is used to do other work quicker. In order to have a positive impact we therefore have to evaluate how we use these techniques sensibly.
Data availability The Python code used for obtaining the results presented here is available al. https: //epbaabbcco//iicoorriimmmmeaeaa Biiiil__Feee__eeee Cibpppplliii
Acknowledgments
We thank Amir Abolfazli, Timko Dubielzig, Ludwig Krinner, Aditya Mohan, and Kilian Stahl for helpful discussions, Pratik Chakraborty and Manuel Schimanski for help in the lab, and Nived Johny, Jonas Junker,
11

---

Published in Transactions on Machine Learning Research (07/2025)
Arindam Saha, and Bernd Schulte for both. LR thanks Mika Nina Seibicke and Alina Jankowski for lab sitting and emotional support.
This work was supported by the Quantum Valley Lower Saxony (QVLS), the Deutsche Forschungsgemeinsccaatt (DFG, German Research Foundation) through SFB 1227 (DQ-mat), Germany's Excellence Strategy EXC-2123 QuantumProntiers 390837967, and Excellence Strategy FXC-2122 PhoenixD 390833453, the BMBF projects ATIQ and QuBRA, the BMWK project ProvideQ, and the Australian Research Council Centre of Excellence for Quantum Computation and Communication Technology (project mumber CE170100012).
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Πya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning heterogeneous systems, 2015. URL https://wwat wwwsorttwteeroll ow.org/ Licensed under Apache License, Version 2.15.1.
Abulikemu Abuduweili and Changliu Liu. An optical controlling environment and reinforcement learning benchmarks. 2022. URL httpp:://arxiv.orr/bbbs///
12114
Abulikemu Abuduweili, Jie Wang, Bowei Yang, Aimin Wang, and Zhigang Zhang. Reinforcement learning based robust control algorithms for coherent pulse stacking. Optics Express, 29(16):2606 26081, 2021.
URL https://opg. optica. org/ httg/e/fuulteee http:/ulllttt.t..ctrriiieeeeaeee sffm?uri=oe- 29- 161--266660111453322
Satish Addanki, T.S. Amiri, and P. Yupapin. Review of optical fibers-introduction and applications in fiber lasers. Results in Physics, 10:743 750, 2018. ISSN 2211-3797. doi: https:://6o.org//10..0
28. http:p/wwww...ccocccc......o.o.p
OpenAl: Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019. URL httttttt///axiii/ http://axxvv..raa//////// 1910.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learming dexterous hand manipulation. The International Journal of Robolics Research, 39(1): 20, 2020. URL https;
//journass.. www.ssa..eeee............ http:///f/fu111111117777////226/
OpenAl: Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Tlya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning, 2019. URL
https: http///axvvvv.. wwweiiv.orr/bbb////eeee 1etp:/991..0000 06680
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforeement learning for greater sample efficioncy and simplicity. International Conference on Learning Representationss 2024. URL lhttps://arxis http:::://xxvvoo//aaaaa abs/ 1902.05605
OpenAI: Greg Brockman, Vicki Cheung, Ludwig Pottersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016. URL https://arxxv.orr///baa MIT License, Version: gymnasium 0.29.1.
Licensed under The
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Namisha Chabbra, Andrew R. Wade, Emily Rose Rees, Andrew J. Sutton, Alberto Stochino, Robert L. Ward, Daniel A. Shaddock, and Kirk McKenzie. High stability laser looking to an optical cavity using tilt locking. Opt. Lett., 46(13):3199 3202, Jul 2021. doi: 40.1364/0L.427615. URL stract.
lan Char and Jeff Schneider. PID-inspired inductive biases for deep reinforcement learning in partially observable control tasks. In Advances in Newral Information Processing Systems, volume 36, 2023. URL httpp:://proceeddnng/ neurips. cc/ httpe///ppppp//////aaaaaaaaaaaaa 226b0-Abstract-Conitn Conf-renceeeeeeeeeee .tml
Xiaoliang Chen, Baojia Li, Roberto Proietti, Hongbo Lu, Zuqing Zhu, and SJ Ben Yoo. DeepRMSA: A deep reinforcement learning framework for routing, modulation and spectrum assignment elastic optical networks. Journal of Lightwave Technology. 37(16):4155-4111 2019a. URL httpss:/opg. httpp:://ppp.ppcia...oomo optica.
1t/abstract. wfttruriijitttt---S111 37-1111111----11555
Xiaoliang Chen, Roberto Proictti, and SJ Ben Yoo. Building autonomic clastic optical networks with deep reinforcement learning. IEEE Communications Magazine, 57(10):20 26, 2019b. URL https:
Alberto Silvio Chiappa, Alessandro Marin Vargas, Ann Zixiang Huang, and Alexander Mathis. Latent exploration for reinforcement learning. In Advances in Newral Information Processing Systems, volume 36, 2023. URL https::/pproceedinngs neurips ptt.rppp.ccpppppe....e//aapppppp 11448bbef6667-PPPae---oofeeeeee
7. URL hct.ap/......ooooo////ffaaaaaaaa
Zihan Ding and Hao Dong. Challenges of Reinforcement Learning, pp. 249-272 Springge,, 2020. ISBN 978-981-
15-4095-0. doi: 10. 11.1007797--99-115--
Ronald WP Drever, John L Hall, Frank V Kowalski, James Hough, GM Ford, AJ Munley, and Hywel Ward. Laser phase and frequency stabilization using an optical resonator. Applied Physics B, 31:97 105, 1983.
URL httpp:///ink: spr springer.... wwt.///tttte/////.o/...o
Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester, Challenges of real-world reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97. PMLR, 2019. URL
https://arxiv.. org/ abs/ abs////4.//1/////////// 12901
Gabriel Dulac-Arnold, Nir Levine, Daniel J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: Definitiins, benchmarks and analysis, volume 110, pp. 2419 2468. Springer, 9 2021. doi: 10. 110107775109444222-000011111111- springer. www.articie//o..o..mom. /tt.://rrccce///..0//////1000000
URL
https: https://iink.springge...
Eduard Durech, William Newberry, Jonas Franke, and Marinko V Sarunic. Wavefront sensor-less adaptive optics using deep reinforcement learning. Biomedical optics express, 12(9):5423 5438, 2021. URL https:
//opg.optica. wwtggoopcca..rrbbbeoooooooo fulltext. www.uri=moooeeee. ------222221--4555900000
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In Deep Reinforcement Learning Workshop NewrIPS 2022, 2022. URL httts::/ net:/foruuni---iiooo00o1SS
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J. R. Ruiz, Julian Schrittwieser, Grzegorz Swirszez, David Silver, Demis Hassabis, and Pushmeet Kohli. Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 610((930):47-555 2022. ISSN 1476-4687, doi: 000////11566-22000000 URL http:p////o11..rr//0000/////////
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods, 2018. URL httpt://atspa.......// wttpa//wwww.rrr////////..00//
13

---

Published in Transactions on Machine Learning Research (07/2025)
Florian Golemo, Adrien Ali Taiga, Aaron Courville, and Pierre- Pverre-Yves Oudeyer. Sim-to-real transfer with neural-augmented robot simulation. In Aude Billard, Anca. Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 517-828. PMLR, 29-31 Oct 2018. URL htttps://proceedingg mml.nngsss........mmm www.sesss..prr.eeeoeee..oeeeeeee vv// html
htt////////goolo// golemo18a
Laura Graesser and Wah L. Keng. Foundations of Deep Reinforcement Learning Theory and Practice in Python. Addison Wesley Data and Analytics Series. Pearson Education, Inc., 1st edition, 2020.
Hernán E. Greeco, Matthieu C. Dartiailh, Gregor Thalhammer- Thurner, Torsten Bronger, and Florian Baner. Pyvisa: the python instrumentation package. Journal of Open Source Software, 8(84):5304, 2023. doi: 10.21105/joss.00 05304. URL https://ddat http:///ooo.orgg/////o 21100.211///ss 05304 Licensed under The MIT License, Version 1.14.1.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the 35th International Conference on Machine Learning, volume 80. PMLR, 2018. URL https::/prroceedings. mlr. htw.e//gsmmrr.rre.////eee press/v80.
haarnoja18b.htmn
T.W. Hansch and B. Couillaud. Laser frequency stabilization by polarization spectroscopy of reflecting reference cavity. Optics commmumications, 35(3):441 444, 1980. URL
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature 585(7)25): 357 362, September 2020. doi: 10.1038/20 / 1 158--020-2649-2. URL https:///oii org/ 10. 1000000///441866000 20-2649-2 Licensed under modified BSD license, Version 1.26.2.
M. Heurs, I. R. Petersen, M. R. James, and E. H. Huntington. Homodyne locking of a squeezer. Opt. Lett., (4(16)22455224) Aug 2009. doi: 10. N0.1364/01.34.0222 URL httpss//opg..ptiieee wwtp://ppc.....ooooooooooooooooe org.oo//bo/orracc./ocooooooooo/a
cfm fimmURI=01-- 30-::o---4--111 1462
Jakob Hollenstein, Sayantan Auddy, Matteo Saveriano, Erwan Renaudo, and Justus Piater. Action noise in off-policy deep reinforcement learning: Impact on exploration and performance. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL httpphttppss.. /tttpp:///ppnnevieee wtt.://ooooooomii/111...........
John D. Hunter. Matplotiib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90 95, 2007. doi: 10.1109/MCSE.200 MCSS.2C07.2255555 URL attps://ieeexppore.iiee ieee. wttpeeerrr//ocoooo////52//22 Licensed under a BSD style license, Version 3.5.2.
Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Trpan, Julian Tbarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis. Sim-to-real via sim-to-sim: Data- effaceef robotic grasping via paudomizaed-to--aaoniie. adaptation networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition,, pp. 12627-12633 2019. URL Sim-To- Real via Sim-To-Sin Data-Efficient Robotic Gra
thecvf
com/ content
sping via
Zhang JianAo. safe-exit. URL http://tt//////ooooo///////////r
alledbhhettlegprggraeextts Licensed under MIT License, Version 0.1.2.
Anqing Jiang, Yoshic Osamu, and Liangyao Chen. Multilayer optical thin film design with deep Q-lcarning.
Scientific reports, 10(1):12780,
Hu Ke, Bing Xu, Zhenxing Xu, Lianghua Wen, Ping Yang, Shmai Wang, and Lizhi Dong. Self-learning control for wavefront sensorless adaptive optics system through deep reinforcement learning. Optik, 178:785-793, 2019.
14

---

Published in Transactions on Machine Learning Research (07/2025)
Keysight. Keysight connection URL http:///ww...aa.tt......///////a sp htw.:-iirrrrrsssssaaeeoooooooaaa software, Revision 2023 Update 1.
html Licensed as Commercial computer
Y. V. Kiran, T. Venkatesh, and C. Siva Ram Murthy. A reinforcement learning framework for path selection and wavelength selection in optical burst switched networks. 1EEE Journal on Selected Areas in Communications, 25(9):18 20,18822,2220 2007. URL
ieee. http:///.....eeeee.e../e///////s
Anja Knigge, Mats Rahmel, and Christian Knothe. Fiber Coupling to Polariaation-Mciitaiinnng Fibers and Collimation. Schäfter Kirchhoff GmbH, 2021. URL http://wwwuaaauur..//////o/ueeas
FibercouplingMhe2.
Mario Krenn, Jonas Landgraf, Thomas Foescl, and Florian Marquardt. Artificial intelligence and machine learning for quantum lechnologies. Physical Review A, 107, 2023. doi: 10. 10.1103/ 10.1103/Physlee Rew A.107.010101 URL https::/journaatt aps. org/pra/ wttp//assttatttooooe http://.....0////yeeeereee.. 107. 010101
Vikash Kumar, Rutav Shah, Vincent Moens Gaoyue Zhou, Vittorio Caggiano, Jay Vakil, Abhishek Gupta, and Aravind Rajeswaran. Robohive: A unified framework for robot learning. In Advances in Newral Information
Processing Systems, volume 36, 2023. URL httppss//proeeeiinngss pepppppcceppppppppprrrreee.. files/paper/
h27/:///22aacccccc1///////022662 haa.aaaeessaaaadeeeeeenaree.... html
Evgeny Kuprikov, Alexey Kokhanovskiy, Kirill Serebrennikov, and Sergey Turitsyn. Deep reinforcement
learning for self-tuning laser source of dissipative solitons. Scientific Reports, 12(1):7185, 2022. URL
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of contimuous distributional quantile critics. In Proceedings of the 37th International Conference Lachinel.aarningg volume 119. PMLR, 2020. URL https://proceedings .wl.ggsss....... ww..ppres/vv 19/ 1///uueetttov////uuzzett/v////,
Rico Landman, Sebastiaan Y Halfert, Vikram M Radhakrishnan, and Christoph U Keller. Self-oplimizing adaptive optics control with reinforcement learning for high-contrast imaging. Journal of Astronomical
Telescopes, Instruments, and Systems, 7(3):039002 039002, 2021. URL http:pp//www.ppppeiiiaaaaaallbla
ry. org.grg...iiannaooooaaacacooooar otgpr//eeeeattttteetuttoooooonnn
ssue oawp://0wDSSaiiiiiigggggeeeeeeea higa:/0..111711.1177....33....
Mikkel V. Larsen, Xueshi Guo, Casper R. Breum, Jonas S. Neergaard-Nielsen, and Ulrik L. Andersen.
Deterministic generation a two-dimensional cluster state. Science, (3466))369-380 2019. URL
https://www.cciene/ wwt.a/iiin.......oooo/u/o///o//. http:///w.....1111/////eeeeeeee/ aay4344
Xinge Li, Xiaoya Hu, Rongging Zhang, and Liuqing Yang. Routing protocol design for underwater optical wireless sensor networks: A multiagent reinforcement learning approach. IEEE Internet of Things Journal, 7(10):9805-9811 0818, 2020. URL hateps://eeeexplore leee http://astraatttoccmmtt////////t
Google Decpmind: Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Hoess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra, 2019. URL
02971
Ping Lu, Nageswara Lalam, Mudabbir Badar, Bo Liu, Benjamin T Chorpening, Michael P Buric, and Paul R Ohodnicki. Distributed optical fiber sensing: Review and perspective. Applied Physics Reviews, 6(4), 2019. URL https://pubs. aip. org ottpaa///pp/aarrccc/////////00/3
Xiao Luo, Chen Shi, Liqian Wang, Xue Chen, Yang Li, and Tao Yang. Leveraging double-agent-io based deep reinforcement learning to global optimization of clastic optical networks with enhanced survivability. Optics express, (786)77777 2019. URL https://opg..... optica. wttpiia...g/////tulteettrtt cfm?uri=oe- 27--6-8896 id=406937
-
15

---

Published in Transactions on Machine Learning Research (07/2025)
Evgenii Mareev, Alena Garmatina, Timur Semenov, Nika Asharchuk, Vladimir Rovenko, and Irina Dyachkova. Self-adjusting optical systems based on reinforcement learning. Photomics, 10(10), 2023. ISSN 2304-6732.
doi:
URL http://wwwdp..cc/////00000/-3330
David Martinez, Guillem Alenya, and Carme Torras. Relational reinforcement learning with guided demon- strations. Artificial Intelligence, 247:295 312, 2017.
Genta Masada, Kazunori Miyata, Alberto Politi, Toshikazu Hashimoto, Jeremy L O'brien, and Akira Furusawa. Contimuous- entanglement on a chip. Nature Photonics, 9(5):316-319, -19, 2015. URL https://ww.nntureecce http://ww.aatuu...et com
h01p://ppooooo......... 2015.
Kirk McKenzie, Eugeniy E Mikhailov, Keisuke Goda, Ping Koy Lam, Nicolai Grosse, Malcolm B Gray, Nergis Mavalvala, and David E McClelland. Quantum noise locking. Journal of Optics B: Quantum and Semiclassical Optics, 7(10):8421, sep 2005. doi:
URL https:
7/dx. fexx.doi.org 110..0100//4444466677/1/111
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https://arxiv: org/ org/abs/
5602
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland,, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Toannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015. 10.1038/naturel /ntuura 4236. URL https:///doi
ISSN 1476-4687. doi: 10.1038
10. 100////atanee13366666
Nikhil Mukund, James Lough, Aparna Bisht, Holger Wittel, Nadji, Christoph Affeldt, Fabio Bergamin, Mare Brinkmann, Volker Kringel, Harald Lück, et al. Neural sensing and control in a kilometer-scale gravitational-wann observatory. Physical Review Applied, 20(6):064041, 2023. URL https://journals.. ps. wttp:/prappleee/ssssttttt//////t 1tt.///tyyyeepppppee......... 20. 264044
Sanmit Narvekar, Bei Peng, Mattco Lconetti, Jivko Sinapov, Matthow Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-56 50, 2020.
Carlos Natalino and Paolo Monti. The optical RL-gym: An open-source toolkit for applying reinforcement learning in optical networks. In 22nd International Conference on Transparent Optical Networks (ICTON), pp. 1-5. IEEE, 2020. URL https::/ieeexpporeee leee. wtt.://rrr...ee..o/bbbbbbbbaaaaa wew.e//.g/.ssstaac//ooooott/////
Mattias Nordin and Per-Olof Gutman. Controlling mechanical systems with backlash survey, Automatica, 38(10):1633 2002.
80200047X
J. Nousiainen, C. Rajani, M. Kasper, T. Helin, S. Y. Haffert, C. Vérinaud, J. R. Males, K. Van Gorkom, L. M. Close, J. D. Long, A. D. Hedglen, O. Guyon, L. Schatz, M. Kautz, J. Lumbres, A. Rodack, J. M. Knight, and K. Miller. Toward on-sky adaptive optics control using reinforcement Bearning-mode based policy optimization for adaptive optics. Astronomy & Astrophysics, 664:A71, 2022a. URL hthpst 1/doi.org/10.10800 101.000001110000--360000000
Jalo Nousiainen, Chang Rajani, Markus Kasper, and Tapio Helin. Adaptive optics control using model-based reinforcement learning. Optics Express, 29(10))15327 15344, 2021. URL
fe/uu
Jalo Nousiainen, Byron Engler, Markus Kasper, Tapio Helin, Cédric T. Heritier, and Chang Rajani. Advances in model- based reinforcement learning for adaptive optics control. In Adoptive Optics Systems VIII, volume 12185, pp. 582-891. SPIE, 2022b. URL hatt:s/s/ww...gglgggaaaaaaaaa/// gs- osso0--spps//22255//1222////a int.://///...aaa/aaoaaoeee-e---e
piica-coattro//O.1111//1 http:oo1o1//111111111111 2630317 2630317...777777....._... short#
16

---

Published in Transactions on Machine Learning Research (07/2025)
Jalo Nousiainen, Byron Engler, Markus Kasper, Chang Rajani, Tapio Helin, Cédric T. Heritier, Sascha P. Quanz, and Adrian M. Glauser. Laboratory experiments of model-based reinforcement learning for adaptive optics control. Journal of Astronomical Telescopes, Instruments, and Systems, 10(01), February 2024. ISSN 1..
2329-4124. doi: 1 11..77.1.jaiss.00 10.1.00.00...11000.. URL https://doi.org/// 1117/1... JATIS. 11.1....1000001 019001
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference Learning Representations, 2018. URL
B. Pou, F. Ferreira, E. Quinones, D. Gratadour, and M. Martin. Adaptive optics control with multi-agent model-free reinforcement learning. Opt. Express, 30(2):2991 3015, Jan 2022. doi: 0..136//0E.440000 URL https://opg.optice
/ttg/////bstraatt.cff/....... 30-2---991
Matthew Praeger, Yunhui Xie, James A Grant-Jacob, Robert W Eason, and Ben Mills. Playing optical tweezers with deep reinforcement learning: in virtual, physical and augmented environments. Machine Learning: Science and Technology, 2(3):035024, 2021. URL httts://iopscieecce iop.org/atticceeee
1088/2632-288 21w3/abffff////eeeeeee
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable- baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021. URL http://jmlr Version 2.3.0.
/tt.att:////oorrroo////aaeaet/// ort//paperss/22220-1144 html Licensed under The MIT License,
Jorge Ramírez, Wen Yu, and Adolfo Perrusquía. Model-free reinforcement learning from expert demonstrations: a survey, Artificial Intelligence Review, 55)3333213-3 12211, 2022.
Mahesh Ranaweera and Qusay H. Mahmoud. Bridging the reality gap between virtual and physical environ- ments through reinforcement learning. IEEE Access, 11:19914-19927, 2023. doi:
49572. URL https://issexpplore leee. wtt.:i..eee..aareeree/eo///eeeee wttp//bsstattccaccuer///00000000
Francisco J. R. Ruiz, Tuomas Laakkonen, Johannes Bausch, Matej Balog, Mohammadamin Barckatain, Francisco J. H. Heras, Alexander Novikov, Nathan Fitzpatrick, Bernardino Romera-Paredes, John van de Wetering, Alhussein Fawzi, Konstantinos Meichanetzidis, and Pushmeet Kohli. Quantum circuit optimiza- tion with alphatensor, 2024. URL http://///axxxv.o//a/a//////
14396
Iman Sajedian, Trevon Badloc, and Junsuk Rho. Optimisation colour generation from diclectric nanostructures using reinforcement learning. Optics express, 27(4)55874 5883, 2019. URL https
//opg.optica.orggee.e wttp:/iaa..gg///oftltteettteeeee http://ullleett..cccuurr---ee www.:rriooo-222---550505 074-0---777----88444---55113313
Frica Salvato, Gianfranco Fenu, Fric Medvet, and Felice Andrea Pellegrino. Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access, 9:153171 153187, 2021. doi: 10.1109/ AAC/SS 2021..3122221111666 URL httep://ieeexploree ieteepeoreeeeeeeeeeeeeeeeeeeeeee otgp//bsstracc//oocmeet/// 9606868
John M. Senior. Optical fiber communiictionss. Prentice Hall, 3 edition, 2009. ISBN 97801303266 Previous ed.: 1992.
D. A. Shaddock, M. B. Gray, and D. F. McClelland. Frequency locking a laser to an optical cavity by use of spatial mode interference. Opt. Lett., 24(21):1499-1101, Nov 1999. doi: 0..134//OL.24.004490 URL httpp://opg
Alexey Shkarin. pylablib, 2024. http:p/:/gggaaa....ertttttooooos htw.:/dooccc..o////aeeeee Licensed under Creative Commons Attribution 4.0 International, Version 1.4.2.
Maksym Shpakovych, Geoffrey Maulion, Vincent Kermene, Alexandre Boju, Paul Armand, Agnès Desfarges- Berthelemot, and Alain Barthčlemy. Experimental phase control of a 100 laser beam array with quasi- reinforcement learning of a neural network in an error reduction loop. Optics Express, 12307-12377-3 2021. URI. http:://ooo..ptiiaa.rrr/////////
17

---

Published in Transactions on Machine Learning Research (07 (025))
Anthony E. Siegman. Lasers. University science books, 1986.
David Silver, Thomas Hubert, Julian Schritlwieser, loannis Anlonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 2017. ISSN 1476-4687. doi: 10. 1e.100//natuuee222
URL https://doi.orr htt.//i..oggg/// 1001111002//aaa222277700 http:///ooo...or/o.o.00//uuu22t0
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140 1144, December 2018. doi: 7/doi.org/10 11w.1111111///ceeeecc aar6404
URL https:
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: comprehensive survey. Artificial Intelligence Review, 55(2):945-995 990, 2022. URL https://link springer
Sumodh Sontakke, Jesse Zhang, Séb Arnold, Karl Pertsch, Erdem Bı yık, Dorsa Sadigh, Chelsea Finn, and Laurent Itti. Roboclip: One demonstration is enough to learn robot policies. In Advances in Newral Information Processing Systems, volume 36, 2023. URL https::/prooeeeding.. newriddp nwwrips cc/paper_fi fil
Dmitry Sorokin, Alexander Ulanov, Ekaterina Sazhina, and Alexander Lvovsky. Interferobot: aligning an optical interferometer by a reinforcement learning agent. In Advances in Neural Information Processing Systems, volume 33, 2020. URL https://proceedings www.eedddinnnnn.uipipeeeee neurips wttp:r/w...cappppp...aea///////p http////ps//ppp//2////1///// 9la544999ccbtttfff5474444111 1atp:aaaa611e---pee....
Adam Stooke, Anuj Mahajan, Catarina Barros, Charlic Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. 2021. URL
https://arziv.org www./r/g////eeeooo////..//..eeae abs/ 2107 12808
José Suárez-Varela, Albert Mestres, Junlin Yu, Li Kuang, Haoyu Feng, Albert Cabellos- Aparicio, and Pere Barlet-Ross Routing in optical transport networks with deep reinforcement learning. Journal of Optical Communications and Networking, 11(11):547 558, 2019. URL https:://ieeexplore ieee. org/abstract
/document/8847548
Chang Sun, Eurika Kaiser, Steven L. Brunton, and Nathan Kutz. Deep reinforcement learning for optical systems: A case study of mode-locked lasers. Mochine Learning: Science and Technology, 1, 12 2020. ISSN 26322153. doi: 10.
abb6dd.. URL http::://hoostann......oorrrraar 10t.://...........0///////////
2-2153/abb6a6/meta
Richard. S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. MIT press, second edition edition, 2018
Thorlabs. Thorlabs kinesis. URL
wwwttttoollss...oo 285. Licensed by Thorlabs, All Rights Reserved, Version 1.14.44.
Sebastian Troia, Rodolfo Alvizu, and Guido Maier. Reinforcement learning for service function chain
reconfiguration in NFV-SDN metro-core optical networks. IEEE Access, 7:167944-167957, 2019. URL
https://ieeexplore ieee. wtwp::/rreeeeeeeerrrrrarreeeeeee http://bbtraact/ddmunt//////e/
Henrik Tinnermann and Akira Shirakawa. Deep reinforcement learning for coherent beam combining applications. Oplics erpress, (7(17):24233-2422 2019. URL htttps://op.. optica. org/oe//fulltext.cc ott.:///g...pppc...r.rooooolott. fm? 2rru=Fee-22-11772222224444-66
Henrik Tünnermann and Akira Shirakawa. Deep reinforcement learning for tiled aperture beam combining simulated environment. Journal of Physics: Photonics, 3(1):015004, 2021. URL hatpp://iopscieecee
op.org/articlee/0..oom 10t.//0001.8882/1155 htt.a//cc..eee..oooo0///5555555- wtt.://///555-7777////dd/2/////
abcc833mmtmm
18

---

Published in Transactions on Machine Learning Research (07/2025)
Henning Vahlbruch, Simon Chelkowski, Boris Hage, Alexander Franzen, Karsten Danzmann, and Roman Schnabel. Coherent control of vacuum squeesing in the gravitational-wave detection band. Phys. Rev. Lelt., 97:011101, Jul 2006. doi:
URL
1103/P
SettevLett.97.011
Carlo M. Valensise, Alessandro Giuseppi, Giulio Ccrullo, and Dario Polli. Deep reinforcement learning control of white-light continuum generation. Oplicn, 8(2):239 242, 2021. URL https://opg..ppptttptet opt.ippp.pppp......p........ org/optic
c/
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards, 2018. URL https///arxx..orr/aab///// 008
Federico Vernuccio, Arianna Bresci, Valeria Cimini, Alessandro Giuseppi, Giulio Cerullo, Dario Polli, and Carlo Michele Valensise. Artificial intelligence in classical and quantum photonics. Laser and Photonics Reviews, 16, 5 2022. ISSN 18638899. doi: 10. http://O...0000000000000 URL https://omnineeiibaaryy. wile y. /ttmm//o//ful1/.10002///222111// 202100399
Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaäl Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Painc, Caglar Gulechre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agent reinforcement learning. Nature, 575:350 354, 2019. ISSN 1476-4687. doi: 10.1038//41158--019- 1724-2. URL https://doi.org 10. 000000//158801111177770
Haozhu Wang, Zeyu Zheng, Chengang Ji, and L Jay Guo. Automated multi-layer optical design via deep reinforcement learning. Machine Learning: Science and Technology, 2(2))025013, 2021. URL
https://Lopsciencce wttp://eennn...oo.oarrra.aeaaaae wttpp//.raaaattttoo.ooo
Heribert Wankerl, Maike T, Stern, Ali Mahdavi, Christoph Fichler, and Elmar W Lang. Parameterized reinforcement learning for optical system optimization. Journal of Physics D: Applied Physics, 54(30): 305104, 2021. URL
146188//1661663///////1 6wt///44///aaa///ee/eeeee abfddb/meta.
Michael L. Waskom. Scaborn: Statistical data visualization. Journal of Open Source Software, 6(60):3021, 2021. doi: 2221100//oss00022 URL htt.htttppp//. /tt////i.orr/11111////////////// 03021 Licensed under BSD 3-Clause "New" or 'Revised' License, Version 0.11.2.
Guilherme B Xavier and Gustavo Lima. Quantum information processing with space-division multippexin optical fibres. Communications Physics, 3(1):9, 2020. http://p/wwww...ateer././//////a F019-0269-7
Xiao Yu, Jia Yc, Lianshan Yan, Tao Zhou, Peng Li, Xihma Zou, Wei Pan, and Jianping Yao. Roal-time adaptive optical self-interference cancellation for in-band full-duplex transmission using SARSA (A) reinforcement learning. Optics Eapress, (111) 2023. URL httpp:::/ooo.gptttaaa.oo//////aa risee-31-811314011c5228
Vijie Zhang and Herke Van Hoof. Deep coherent exploration for continuous control. In Proceedings of the 38th International Conference on Machine Learning, volume 139, PMLR, 2021. URL https //proceedings... mlr pr....pprras//111//a//gg111 html
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In 2020 JEFF Symposium Series on Compulational Intelligence (SSCT), pp. 737 744, 2020. doi: 10.1100/SCC44333322200..888 ct/ ct/document/9308455
URL https::/heeexploreeeeeee ieee. ieeg..ooreeeeeeeeeeeee.eee/ owgeee.rrg/bbttrr. abstra
19

---

Published in Transactions on Machine Learning Research (07 2205))
A Additional details about the experimental setup
Our setup includes the following components: We use 1064 nm laser (Mephisto, Coherent), single mode iatrrization-maittanninn
fiber, and a -Kiahfer fiber collimator (60FC-SF-4-M8-08) at the input side of the fiber. The measurements of laser power are done with power meters (Thorlabs PM160, measurement. error 1%). In front of the experiment, we place a partially reflecting mirror to mcasure a fraction of the laser light with an additional power meter for power reference. The measurement used pause training in the event of laser failure and to track power fluctuations to determine the maximum power level. In this way, we can determine the coupling efficiency with an error of 2%. The input power is set to 1.00(1) mW.
For training on the experiment, we used an NVIDIA GeForce RTX 4070 GPU. In addition to the usual 2007), we used PyLabLib, Thorlabs
packages (Harris et al. 2020 Abadi et al. 2015 Waskom 2021 Hunter
Kinesis, VyVisa, Keysight Connection Expert and safe-exit (Shkarin Keysight JianAoh for communication with the experiment.
2024 Thorlabs:
Grecco et al. 2023
Due to safety constraints, we have to limit our state space and, in consequence, clip our actions if the actuator positions would otherwise move out of a certain range because it is unacceptable for the laser beam to wander around the room. Also, in the first test run, the action size was chosen so poorly that the mirror mounts were damaged. So, both laser sa safety and equipment damage are hazards that we need to consider.
B Explanation of the imprecision in the mirror steering motors and its characterization
Backlash is a phenomenon that is present when load is not directly connected to motor, such in geared mechanical systems (Nordin & Gutman (002). Dependent on the exact geometry of the system, i.e, mechanical tolerances, amount of gears, etc., it may resemble hysteresis between the expected and actual position or a dead-zone, where moving the actuator has no effect on the actual position whenever the rotational direction is reversed. is thus hard to model and predict a priori. Control has to be implemented based on the specific system and its use. These control schemes include hysteresis models, dead-zone models, and PI control. However, additional sensors are needed to get accurate feedback if multiple actuators are used. Step loss results from the difference in static and dynamic torque of motor. The motor steps result in a linear actuation, which changes the tilt of the mirror mount. Different tilt angles lead to different static loading of the motor and gearbox. This may lead to the initial step(s) being lost, as the motor can not deliver the starting torque, resulting in a partial step. Without feedback from, e.g., an encoder, this leads to a difference between the expected and actual position. Lastly, the non-orthogonal degrees of freedom are a result of the kinematic mirror mounts used and their mounting. Usually, this error is small for well-designed kinematic mirror mounts.
We performed dead-zone characterization. The core idea is to initiate mumber of movements, i.e., generate movement history, after which a maximum dead-zone is expected. This can simply be an initial long movement in one direction followed by a direction reverse. The long movement ensures a nearly linear behavior between the expected and actual position, as backlash is overcome in the mechanical system. The backlash after a change in rotational direction should, therefore, be large. Additionally, the movement history is similar between repetitions, enabling their comparison. Starting from a position with high coupling, we moved one actuator far out, then back to high coupling. From there, we reversed the movement and counted the steps the actuator had to move before the measured power changed by more than the power measurement error. Repeating this process 100 times yielded the distribution of the dead-zone size, shown in Figure I (c) for the four motors. This data helps us understand the uncertainty of the mirror mount movements. As no continuous feedback is employed, characterization of other positioning crrors is not possible in our setup.
C Environment and agent tuning on virtual testbed
We used the data of scanning each motor individually through the coupling peak to create a virtual testbed. Each dataset was normalized and fitted with a Gaussian; all of them were then multiplied. The highest coupling efficiency we had measured up to this point was 0.92; therefore, we use this as the amplitude. The
20

---

Published in Transactions on Machine Learning Research (07/2025)
following function, based on the fit values in motor steps, describes our virtual testbed:
1
Fmml
5470785
2
yml 55733944 2
P(umm Dmnl, Tmnz Umz) 0.92exp
+
2
11994
19145
Jmm2
5461786
2
1/n 5178016 2
12769
17885
We use this testbod to gain insights into the environment hyperparameters, observations, and algorithms to use in the following order.
First, we optimized the hyperparameters of the reward (a's, I'', A's). Next, we went to the parameters of the environment that appear in the reward, i.e, the goal power fgoa) and episode length T. The usual figure of merit is the normalized return in dependence on the training step. However, this depends reward, and the reward depends on both of these sets of parameters. Therefore, it is not possible to use the return as a figure of merit for these parameters. Instead, we trained TQC agent for a total of 10 timesteps. We tested the agent every 10 timesteps for 100 episodes, noting the probability of failure, the probability of reaching the goal, and the average power at the end of each episode. Our main figure of merit was the probability of reaching the goal after a training time the range of 10at to 4 10 time steps. Still, we also took the probability of failure and the average power at the end of each episode into account. We show the second one here only when used it for our decision.
the
were able to use the normalized return to compare other After fixing the first two sets of parameters, environment parameters, such as the length of the history in the observation and the maximal action, and different algorithms. All studies in the virtual lestbed were performed al least 5 times and, except for the algorithm tests, used TQC as the algorithm as this was the algorithm most used in the experiment. If not stated otherwise and if these were not the parameters being changed, we used the parameters in Tables] and 2
C.1 Reward Hyperparameters
We want a high probability of reaching the goal after the least amount of training time, so we shape the reward function accordingly. For tuning its hyperparaneters, we chose Plail 0.2, Puin 0.4, Pgeal = 0.8, T 20, 0g 0.5, a 0.9, A, 10, in contrast to Tables [ and 2 if those parameters were not the ones being changed. We tested the tuples of reward parameters given in Table 4 For each parameter we tested a number of different values and also checked the dependence of the variables on each other. After evaluation, we decided on the parameters in Table 2 The subscript s always refers to the step reward, f to the fail reward, and to the goal reward. The other parameters were Prail = 0.2, Pmin = 0.4, Paoal = 0.8, T = 20, 02 0.5, af = 0.9, A₁- 10 or given Tables] [ and 2
Table 4: test table tuming parameters
Parameter (g),
(a,, Ba)
(11) (0.1,0.5.0..
(ag. B,1, (0.1,0.5, 0.9}
Value
{10, 100, 1000}²
(0.1,0.5, 0.9} x x(1,5,10) 10)
x(1,5]2
x x(1,5]2
Prefactors First, we tested different prefactors A, and Ag. The results are shown in Figure 3 Looking at the probability of reaching the goal, Ar 100 seems to be the best value. For training steps over 3 104, we can see that Ag 1000 seems to be better than the other two values, before it seems that A, 100 is doing better. However, if we look at the probability of failure, we can see that using Ag 100, the probability of failure falls more quickly than if we are using A, 1000. Because resets after failure take a lot of time for this kind of Pmmi and Prail, we want the failure probability to be as low as possible and go with A, = 100.
Step Reward Second, we are looking at the step reward and optimizing for a, and 3,, Figure shows the probability of reaching the goal. We deem 3, and ag = 0.9 be the best parameters, although there is not very strong difference.
21

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
Ay
Ay = 100
Ay
0.8
jona]]
04
1000
0.0
training step 10
step in 10
training step in 10*
(a) Probability of reaching the goal in dependence of the training step.
Ay
Ay =
1.0
0.8
0.6
04
100
1000
0.0
10
10
Iraining stop in 10
trining step in 10
training Stop in 10*
(b) Probability failing in dependence of the training step.
Figure 3: Results for prefactor tuning: Probability of reaching the goal or failing for different prefactors the reward using TQC and Pian the parameters in Tables and
0.2, Pmin = 0.4, PRont = 0.8, = 20, a, = 0.5, af = 0.9 and otherwise The plots show the mean with 2o error bars created by multiple runs.
B
B 5
10
1.0
0.8
Planal
0.6
0.4
0.9
0.2
0.0
training step 10
training 101
training
10"
Figure 4: Results for tuning the parameters of the step reward: Probability reaching the goal for different
0,, 3, in the reward with TQC, Ptai 0.2, Pmin 0.4, Proal = 0.8, 20, a = 0.9, A, 10 and all other parameters as in Tables I andd 2
The plots show the mean with 2a error bars created by multiple runs.
22

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
Pæ B22
Ba 5. B2
1.0
0.8
Pignal]
0.4
0.2
0.0
Be 1. 1.
BA 5. 5.
0.1
0.8
Pigoat|
0.6
0.2
0.0
2
5
10
2
6
9 10
training step 10
training step 10
Figure 5: Results for tuning the parameters of the goal reward: Probability of reaching the goal for different
ag. Sy1. in the reward with TQC, Ptall 0.2, Pmin 0.4, Pgoal 0.8, T 20, a₂ 0.5, af 0.9,
A = 10 and all other parameters as in Tables 1 and 2
The plots show the mean with 2a error bars created
by multiple runs.
Goal Reward Third, we are looking at the goal reward and optimizing for agg, 891 and 8g2. Figure 5 shows the probability of reaching the goal. Here, there is a stronger difference between the different parameters. We are going with 0g = 0.5, 811 = 5 and B32 = 1. The other possibility would be 0g = 0.1, 8g1 = 1 and 8y2 = 5, which is worse in training steps 1 10a - 10 but better in training steps 4 10 our focus on the earlier phases of training and also do not want to emphasize the power with which the goal reached that much over the time in which it reached, which is why go with the first choice of parameters.
10a. However, we put
Fail Reward Lastly, are looking at the fail reward and optimizing for a 8/1 and 312. Figure 6 shows the probability of reaching the goal. Here, the choice is again not that clear, but deem By1 B32 5 and ay 0.5 to be the best choice of parameters.
C.2 Episode Length
We want find good trade-off between reaching the goal quickly and reaching it reliably. Using the same parameters for reward shaping, tested different episode lengths, in particular, T == 5, 10,...,,,, 50. The results are shown in Figure 7 As expected, the longer the episode, the higher the probability of reaching the goal (or failing). For some of these (i. e. T = 20, 30, 35, 40, we also varied the maximum allowed actuator steps per environment step 4max (i. e. doing simulations with amax = [2- 100 100) to see if it had an effect on this, which we could not confirm. However, we also have to take into account that longer episodes will take more time in the experiment. This is why we select T 30, as there is very big difference between this and T 30.
23

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
By 1. Byz
Bf1 = 5.
0.8
Pggnal]
0.6
0.4
0.2
0.0
By 1, 1y =5
Bn = 5. B12 =5
0.1
1.0
0.5
Pigoal]
0.6
0.4
0.2
0.0
2
6
9
10
5
8
10
training step 10
training step 104
Figure 6: Results for tuning the parameters of the fail reward: Probability of reaching the goal for different ay, 8y1,882 in the reward with TQC, Pian 0.2, Puuin 0.4, Pgoal = 0.8, T = 20, a, = 0.5, A, 10 and all other parameters as Tables
and
The plots show the mean with 2a error bars created by multiple runs.
Resas 2000
Oenas 6000
Oaz 10800
0.8
history length
P/poo]]
0.6
15
0.4
30
45
0.0
training step
training step
training step
Figure 7: The probability of reaching the goal in dependence of the training step for different maximum
episode lengths T and maximum actions Qmax with TQC, Pral 0.2, Pmin = 0.4, Pgoal 0.8, T 20, 0, 0.5, a = 0.9, A, = and all other parameters Tables and The plots show the mean with 2o error bars created by multiple runs.
24

---

Published in Transactions on Machine Learning Research (07/202) 2025)
(a)
(b)
0.8
normalized (smoothed)
0.8
0.6
0.6
0.4
B
0.4
0.2
0.2
0.0
0.0
-0.2
A,
40000
80000
start
after training
training steps (rounded to 500)
reset method
Figure 8 Comparison of different reset methods. The methods are the following: Reset as described the main paper (for testing the start and end of training), Reset described in the main paper, but first, go to neutral positions in every episode, C Reset by choosing random positions for all four actuators in interval of width 4.2 - 10 around the neutral positions. We use the parameters from Tables and TQC and Pgout = 0.85. (a) shows box plot of the starting powers. (b) shows the normalized return in dependence on time. The mean is shown with 2a error bands created by smoothing and multiple runs.
C.3 Reset Methods
In the virtual testbed, we compare the following reset methods:
Reset as described in the main paper (for testing at the start and end training).
B Reset as described in the main paper, but first, go to neutral positions in every episode.
C Reset by choosing random positions for all four actuators in an interval of width 4.2 10 around the neutral positions.
We used the parameters from Tables and 2 and Pgoal 0.85. The results can be seen in Figure In Panel (a), see the starting power for the different reset methods. Using method C, the starting distribution of powers is very different from the other reset methods. The median is similar, but the standard deviation is much higher. This led us to compare methods A and B additionaly. In method B, the median is slightly higher and independent of our policy. For method A, the distribution depends on the model used, and the median and 75th quantile are slightly higher and more comparable to the one of B after 10 training steps than the start. Because of this, the return for method the middle, as can be seen in Panel (b). We would have expected a higher impact from the reset method. However, the differences independent of each other, we do not gain artificial benefit from it.
slightly higher than for method A, especially
quite small. In conclusion, even though our method makes episodes not fully
C.4 Goal
The goal power is fully our choice. First, we check at what point the return starts to converge for which goal power. Therefore, look at the normalized return in dependence on the training steps for different goal powers. This is shown in Figure g (a). We can see that the higher the goal power, the lower the normalized return after convergence, and the later the return converges. We can see that this point happens siguificantly later for high goal powers. Also, for high goal powers like Pgoal = 0.9, the distance to the last curve bigger than, for example, for Pgoal 0.86.
Because of this, we wondered if it made sense to pre-train on lower goal powers. We tested this by starting with goal powers Pstart, goal 0.5, 0.7, 0.8, 0.85, 0.875 and raising it to 0.9 over the course of 105 training steps
25

---

Published in Transactions on Machine Learning Research (07/202) 2025)
1.0
Pigool| for =T after 10 trainingg steps
0.7
goal function
linear
0.6
nurmalized retum (smoothed)
0.8
step
0.5
0.6
0.4
0.4
0.3
0.2
Pooul
0.2
0.8
0.88
0.0
0.89
0.1
0.86
0.9
-0.
0.87
0.91
100000
200000
300000
400000
500000
0.6
0.7 Pgool.
0.8
training steps (rounded 1000)
Figure 9: (a) shows the normalized return in dependence of the training step for different goal powers. (b) shows the probability of reaching the goal power Pgoal = 0.9 after 10 training steps in dependence of Peoal, start- Hereby, the goal on which the model is trained either riscs in a lincar (orange) or step (bluc) function of the training step from Pgoal, start to Pgnal = 0.9 over the course of 10 training steps. Both panels show the mean with 2o error bands created by multiple runs and, in (a), smoothing.
either linearly, i.e,, raising it slightly in every training step, or in staircase way, i.e. raising it more every 10 training steps. Figure shows the probability of reaching the goal Pgoal = 0.9 after 10 training steps in dependence of the starting goal power Pstart, goal for the two different manners of raising the goal power. We can see that it can be helpful to raise the goal power steps, especially starting from Pstart, goal = 0.85.
C.5 Observation
We tested history lengths of n 1,...,6 6 and the maximum sensible length T 30 with Pgoal 0.85. The results can be found in Figure 10 (a). Depending on the training step, n = 3, 4 lead to the highest return. We went with n = 4 as this was higher around 2 10 to 5 10 training steps.
We lested removing Pave or Pmaxx and Zmax from the observation influences the performance. Figure 10 (b) shows that TQC performed worse on any of these combinations compared to the full observation presented above. However, leaving out Pave had much smaller impact than leaving out P Pmax and mmax: which makes the latter very important for us.
Additionally, we tested how agents perform in an environment that includes the absolute position of the actuators in the observation compared to one that does not. The normalized return against the training step can be found for both configurations, using the parameters Tables D and TQC, and Pgoal = 0.85, in Figure 10 (c). As expected, the agent learns faster and reaches a higher normalized return if those absolute positions are included. However, the main application for our agent is recouple the light into the fiber after other parts of the experiment have been misaligned. That means, that the optimal positions, i..., the means #4 of the underlying Gaussian, change, and the agent still has to be able to reach the goal. Hence, we tested the agent 100 times cach in environments in which A = 0, 4 mcans of the underlying Gaussian μt were shifted by Ho;, i.e, 14 = #4 + o; Figure 10 (d) shows the probability of reaching the goal against k. If no shifts occur, the agent with the absolute position as part of the observation performs slightly better than the one without. However, this quickly changes as more shifts are applicd. The agent with absolute position performs much worse any shifts appear. On the other hand, the agent not observing the absolute position performs well independent of shifts.
26

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(a)
(b)
bistory length
observation
0.8
without Pave R
incl. Praes-Ana:
normalized return (Smoothed)
3
normalized return (smoothedd
incl. Pare
0.6
0.6
incl. Pare.Paure Penuee
0.4
0.2
0.0
-0.2
20000
100000
20000
80000
training steps (rounded 500)
training steps (rounded 500)
(c)
(d)
1.0
normalized return (smoothed)
0.8
0.8
probability of reaching
0.6
absolute position
not of observation part observation
0.4
0.2
0.0
absolute position part observation
-0.2
part of observation
0
20000
40000
60000
80000
100000
training steps
number of shifts
Figure 10: Comparison of different observations and maximum actions. (a)-(c) show the normalized return dependence of the training step using the parameters in Tables D and P TQC, and Pgoal 0.85 for different history lengths in (a), leaving out different parts of the observation in (b), and with without including the absolute positions in the observation in (c). The models with and without the absolute positions were then tested 100 times each in an environment in which &o = ..., 4 means of the underlying Gaussian μ4, were shifted by Ho,, i.., 14 = + o. (d) shows the probability of reaching the goal against the number of shifts k. The plots show the mean with 20 error hands/bars created by multiple runs and, in (a)-(c), smoothing.
27

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(b)
1.0
max. action Amax [10*]
2
7
0.8
relum (smoothhe)
0.6
relur (smoothed)
0.6
6
10
0.4
0.4
0.2
0.2
0.0
-02
20000
40000
60000
80000
100000
0
20000
40000
60000
80000
100000
training steps (rounded 500)
triningg (rounded to 500)
(c)
(d)
1.00
0.75
0.8
return (smoothed)
return (smoothed)
0.6
SAC
0.50
TQC
TD3
0.25
DDPG
PPO
0.00
A2C
0.0
-0.2
CrosQQ
-0.2
-0.50
300000 training steps (rounded to 500)
training steps (rounded 500)
Comparison of different maximum actions and algorithms. The plots show the normalized return against the training step for different maximun actions 0max and TQC in (a) or for seven different algorithms: TQC, SAC, TD3, PPO, DDPG, A2C, and CrossQ in (b)-(d). (b) shows this for Peoal = 0.8, (c)
Figure 11:
for Pamal = 0.85, and (b) for Pgnal = 0.9. The other parameters are chosen as in Tables and The plots show the mean with 2a error bands created by multiple runs and smoothing.
C.6 Action
We tested different maximal actions 0max = 10 103,5 103 i0,, 10 with PRoal 0.85, TQC, and the other parameters as in Tables and 2 see which yields the highest return. The results are shown in Figure 11 (a). Maximum actions between 4 103 and 8 10 generally performed best (4 10 performed best out of them). Because of the imprecision of the motors, also did some tests in the lab, which is why we selected 0max 6 103 for our experiments. This is approximately half of the standard deviation of the Gaussian in x- direction.
C.7 Algorithms
We tested seven different algorithms with their standard hyperparameters in StableBaselines3 with the parameters in Tables and for PRoal = 0.8, 0.85, 0.9 for either 10 (for Pgoal 0.8, 0.85) steps or 5 10 (for Pgoal = 0.9) training steps. The results are shown in Figure 11 Panel (b)-(d). We can see that in the first 10a steps, A2C and PPO always perform worst, and DDPG is next in line. However in Figure 11 (d), we can see that for Pgoal = 0.9 PPO catches up to DDPG around training step 1.5 10 CrossQ, SAC, TQC, and TD3 perform much better than these three. TD3 is nearly always worse than SAC. TQC always has a drop in the middle region but catches up SAC in the end. CrossQ outperforms the other algorithms for Pgoal 0.9, while for lower goals, its behavior is similar to the other well-performing algorithms. Overall,
28

---

Published in Transactions on Machine Learning Research (07 2025)
(a)
pool
0.85
(b)
goal
0.9
1.0
normalized maalzzdd return (smoothed)
normalized (smoothedd
0.8
noise factor
0.6
0.1
0.4
2
3
0.2
0.2
0.0
0.0
50000
100000
150000
200000
100000
200000
300000
400000
500000
training steps (rounded to 500)
training steps (rounded to 500)
Figure 12: Comparison of different noise levels. The plots show the normalized return against the training step
for different goal powers goal 0.85, 0.9 and noise factors N = 0,.... 3 using TQC. The other parameters are chosen as in Tables and Both panels show the mean with 2a error bands created by multiple runs and smoothing.
CrossQ, followed by SAC, seems to be the best algorithm for this task when used on the virtual testbed. TQC suffers from a drop in performance in the middle of training compared to SAC. In contrast to that, in our physical experiments, TQC slightly outperforms SAC for PRoal = 0.85.
C.8 Effect of noise on the learning process
We use the characterization of the dead-zone in the actuators to derive a simple noise model: Each time the agent performs an action, its size is reduced by a value randomly sampled from the dead-zone characterization multiplied by a noise factor N. For a noise factor of N O, there is no noise, and the results are similar to the ones discussed in the virtual testbed section up this point. A noise factor of . = should make present in the experiment. In comparison, noise
the noise level of the virtual testbed comparable to the factors of N > 2 correspond to higher noise levels than in the experiment. For Pgoal = 0.85, 0.9 and noise factors of N = 0, 00.,,.. 3, the normalized return is shown in Figure [2, As expected, the return for higher noise levels generally smaller than the one for no noise, but for each of the presented noise levels, the agents are still able to learn. For Pgoal = 0.85, the graphs for N = 0, are very similar and only for the higher noise factors (W 2, 3) the learning curve clearly differs. That means that for a moderate goal power, the noise in the experiment does not affect the agent much. However, this is different for PRoal 0.9, the graph for N = 1 is grouped with the ones for = 3. Hence, the agents' learning curves are significantly impacted by the noise level found in the experiment.
D Other experimental runs in the optics lab
D.1 Different goal powers
We run experiments with Pgoal 0.85, 0.86, 0.87, 0.88, 0.9 using TQC. The normalized return is shown in Figure 13 (a). We can see that, just like in the virtual testbed, the training needs longer to converge the higher the goal. It. interesting that there a big gap between the agents with Pgoal = 0.87 and Pgoal 0.88. Please note that we performed these training runs (except for Pgont = 0.85) only once and draw our conclusions from there.
D.2 Replay buffer
We already discussed in the main paper that it can make sense to pre-train agents on lower goals. However, did not discuss what do with the replay buffer when changing the goal power. Here, we test if it would
29

---

Published in Transactions on Machine Learning Research (07/2025)
be better to keep delete it when changing to the next higher goal power. We perform two training runs on the experiment starting with P goal = 0.85, raising it to P = 0.875 after 3.8 10 training steps, then raising it to Pgeal = 0.89 after approximately 6.35 - 10 training steps, and then raising it to ggoal = 0.9 after 9.8 . 10a training steps. In the first, we delete the replay buffer after changing our goal to Pgoal = 0.875 and Pgoal = 0.89 (yellow, discussed in main paper); the second, we do not (green). Both runs are shown Figure 13 (b). In the yellow ones we see more drops after each rise in goal power, but normalized return is slightly higher in the end. Overall, the results are quite similar.
D.3 Pre-training on virtual testbed
Furthermore, we want to know if pre-training on the virtual testbed helps with the experiment's training times. We tested both an agent pre-trained on the virtual testbed without noise and on a version of the virtual testbed with noise. As noise, we sample random values for each of the actuators from the dead-zone characterization in Figure | (c) and reduce the absolute value of the action by these sampled values. Figure 14 shows the results next to the agent pre-trained on the experiment with lower goal powers. Panel (a) shows the normalized return plotted against training steps. Overall, the agent pre-trained on the virtual testbed without noise is more stable but not significantly better or faster in training. The agent pre-trained on the virtual testbed with noise reaches higher returns and is faster than the other two. Panel (b) shows test results (time needed to fiber couple to Pgoal 0.9) for the three agents and two agents trained only in the virtual lestbed, either with or without noise. The agent pre-trained on the virtual testbed with noise performed slightly better than the other two pre-trained agents, which showed no significant difference between them. The two agents only trained in the virtual testbed are significantly slower, the one trained with noise being slightly faster than the other. However, they are not as slow that it could not be useful: If the time for coupling is not relevant, might be enough to learn the very simple virtual testbed (even without noise).
E
Algorithm Hyperparameters
We use the default hyperparameters in StableBasclines3 (incl. contrib), Version 2.3.0 RRffin et al. 2021 or the way they appear in their tutorials. For completeness, we list them here and print the ones that are not default but used in the tutorial in bold.
TQC learning rate: 0.0003, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, no added action noise, update target network every 1 step, mumber of quantiles to drop per net: 2, number of critics networks: 2, number of quantiles for critic: 25
SAC learning rate: 0.0003, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, no added action noise, update target network every 1 step
CrossQ learning rate: 0.0003, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, discount factor: 0.99, update model every step, do gradient step after each rollout, no added action noise
TD3 learning rate: 0.001, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, action noise: up.ones actions), policy and target network updated every 2 steps, standard deviation of smoothing noise on larget policy: 0.2, clip absolute value of larget policy smoothing noise al: 0.5
actions), sigma=0.1 x
DDPG learning rate: 0.001, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft
update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, action noise: NormalActiolNooiseemmm---ppa np.ones( (number nnuonernnumber
actions), sigma=0.1
actions)
30

---

Published in Transactions on Machine Learning Research (07/2025)
PPO learningrate: 0.0003, number of steps between updates: 2048, batch size: 64, mumber of epochs when optimixing surrogate loss: 10, discount factor: 0.99, factor for trade-off between bias vs. variance for GAE: 0.95, clip range: 0.2, normalize advantage, entropy coefficient: 0.0, value function coefficient for loss calculation: 0.5, maximum norm for gradient clipping: 0.5
A2C learning rate: 0.0007, number of steps between updates: 5, discount factor: 0.99, factor for tradc-off between bias vs. variance for GAE: 1.0, entropy coellicient: 0.0, value function coefficient for loss calculation: 0.5, maximum norm for gradient clipping: 0.5, RMSProp epsilon: 1e-05, use RMSprop
31

---

Published in Transactions on Machine Learning Research (07 (07/202) 2025)
(a)
(b)
1.0
normalized return (smoothed)
gnai
normalized return (smoothed)
0.85
0.8
0.8
0.86
mm
0.6
0.87
0.6
start
0.88
with replay
0.4
0.4
without replay
0.2
0.2
0.0
0.0
-02
0.2
0
20000
40000
60000
50000
100000
150000
200000
timesteps
timesteps
Figure 13: Both panels show the normalized return plotted against the training step. (a) shows the training from the start for different goal powers. In (b), Pgual is raised in steps at cach black vertical line from 0.85 over 0.875 and 0.89 to 0.9. For training one of the models (yellow), delete the replay buffer after the first two black lines; for the other (green), we do not. Both panels show the mean with 2a error bands created by smoothing.
(a)
(b)
1.0
350
300
normalized return (smoothed)
time reach P 0.9 [seconds]
pretrained lower goals
pretrained on virtual testhed
200
only trained on vittual
150
peetrained testbed with noise only trained virtual with noise
100
0.0
50
-0.2
50000
100000
150000
200000
timesteps
Pre-training on virtual testbed. (a) shows the normalized return plotted against time for three agents: one is trained directly on the experiment with successively higher goal powers (blue), the other two are already pre-trained for 10 training steps on the virtual testbed either without noise (orange) or with noise (red). We used Pgoal 0.9, TQC, and the parameters in Tables and 2 (b) shows how long the agents marked with a star in (a) (green and pink are both only trained on the virtual testbed, without with noise, respectively) need to couple to P Pgeal = 0.9 the experiment. (a) shows the mean with 2a error bands created by smoothing.
Figure 14:
32


Reviews:
[{"review_id": "Pkf2RVhje5", "reviewer": "Reviewer_6WCJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the motivation for this line of work is quite strong as modeling environments accurately is indeed a major challenge. I am not familiar with the precise details of this setting, so I will refrain from commenting. It seems to me that there is much more that could be investigated in terms of experimental details, but the paper covers many bases.\n\nIn particular, I will highlight that I found the application of curriculum learning to be creative and quite persuasive in demonstrating its effectiveness.\n\nThe parametric form of the reward function still seems somewhat arbitrary, although the authors have put much effort into justifying it.\n\nOverall, I believe this paper contains interesting points. It is difficult for me to compare it against prior work, which presents me from recommending it with full confidence (especially given my lack of familiarity). Nonetheless, I commend the authors for their effort.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates a suite of reinforcement learning methods for experimental control in optics. They perform their experiments in-situ using live actuators, and demonstrate that their method can outperform human experts in this domain. Some effort is applied to optimize the reward function and MDP set-up, and curriculum learning is used in order to improve performance on particularly difficult objectives.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "If it is particularly easy, I would recommend extension to other set-ups (more actuators, different ambient conditions), but as this is difficult to perform for a journal revision, I will not require any such adjustments.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mV320U97A8", "reviewer": "Reviewer_sR4c", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. The paper presents interesting experimental results applying RL to a laser light coupling task, addressing practical challenges including partial observability and motor imprecision through careful design of the observation space and goal curriculum.\n2. The reasons behind design choices (observation space, goal curriculum) are explained in detail and justified through ablation studies.\n3. The comparison against human experts demonstrates the potential of RL systems for automating optics experiments.\n\n## Weaknesses:\n1. **Scope concern**: This paper does not propose modifications to existing RL algorithms or provide in-depth insights into RL methodology. It primarily uses established algorithms (SAC and TQC) as tools without advancing machine learning research, which, in my opinion, falls outside TMLR's scope as a machine learning research venue.\n2. The evaluation is limited to a single task, which raises questions about the generalizability of the conclusions and design choices to other domains. This narrow focus further suggests the work may be better suited for a journal focused on experimental optics rather than TMLR.\n3. The paper overlooks more recent advances in off-policy RL for continuous action spaces, such as CrossQ [1] and BRO [2], which could offer higher sample efficiency than SAC and TQC.\n4. Since sample efficiency is a major challenge for real-world applications, it's unclear why only model-free methods were considered when model-based RL approaches such as TD-MPCs [3, 4] typically demonstrate stronger performance with limited data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an application of model-free reinforcement learning algorithms (SAC and TQC) to a laser light coupling task in a real experimental setting. By comparing the performance of RL agents against human experts, the authors demonstrate the potential of using RL to automate optics experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. Expanding the evaluation to include at least one additional real-world experiment that differs significantly from the current one to demonstrate generalizability of the design choices.\n\n2. Testing more recent sample-efficient algorithms such as CrossQ [1], BRO [2], and TD-MPCs [3, 4] to strengthen the experimental comparison.\n\n3. Consider submitting to venues more focused on applications of AI in experimental physics or optics, where the practical contributions would be more aligned with the journal's scope.\n\n## References:\n[1] Bhatt, Aditya, et al. \"CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicity.\" *The Twelfth International Conference on Learning Representations*.\n\n[2] Nauman, Michal, et al. \"Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control.\" *Advances in Neural Information Processing Systems* 37 (2025): 113038-113071.\n\n[3] Hansen, Nicklas A., Hao Su, and Xiaolong Wang. \"Temporal Difference Learning for Model Predictive Control.\" *International Conference on Machine Learning*. PMLR, 2022.\n\n[4] Hansen, Nicklas, Hao Su, and Xiaolong Wang. \"TD-MPC2: Scalable, Robust World Models for Continuous Control.\" *The Twelfth International Conference on Learning Representations*.", "broader_impact_concerns": "There are no ethical concerns regarding the broader impact of this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "n6d2GODYfD", "reviewer": "Reviewer_4MWG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths  \n1. Practical Relevance: By addressing an important problem in automating optical experiments, the work has the potential to reduce manual tuning and improve experimental efficiency in optical research.\n2. Real-World Experimentation: Training directly on the experimental setup provides robust evidence of the practical viability of RL in complex optical systems.\n3. Detailed Analysis: The study offers comprehensive comparisons between different RL algorithms and thoroughly explores the benefits of curriculum learning, enhancing experimental validation.\n4. Virtual Testbed Utilization:: It is beneficial to leverage a virtual testbed to inform real-world training strategies, enabling fine-tuning of parameters before lab deployment.\n\nWeaknesses\n1. Comparison to Classic Noisy Environments: It is beneficial to clearly differentiate the optical environment in this study from typical noisy RL environment settings, such as those in robotics (e.g., Mujoco).\n2. Training Time and Sample Efficiency: Although the results are promising, the long training duration (up to 4 days for high goal power) raises practical concerns. It is better to conduct discussions about potential strategies —such as improved exploration techniques, hybrid approaches, or advanced sim-to-real methods—to reduce training time, and by reducing real-world training time.\n3. Real vs. Virtual Environment: It is beneficial to further analyze the transferability of ablation studies from the virtual testbed to real-world experiments. Discussion is needed on how reward and observation designs in virtual testbeds can be transferred to real-world experiments. In addition, it is beneficial to discuss similarities/differences between virtual testbed and real experiment.\n4. Reset Procedure Complexity: It would be beneficial to discuss the reset mechanisms in more detail. For instance, does the reset mechanism apply only during real-world training, or does it also extend to virtual training and controlling on real-world systems? A more thorough exploration of reset strategies, including alternative resetting methods, is beneficial.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the challenging problem of coupling laser light into an optical fiber within a complex, noisy, and partially observable environment. The authors demonstrate that reinforcement learning (RL) can overcome these challenges by training an agent directly on the physical setup, eliminating the need for an accurate simulation. They evaluate several RL algorithms and show that the experimental results achieve performance comparable to a human expert. Additionally, the paper investigates the benefits of curriculum learning and pretraining on a virtual testbed to enhance training efficiency, highlighting the potential of these techniques for real-world optical control applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Discuss the differences between the optical environment in this study and classic noisy RL environments (e.g., robotics control in Mujoco).\n2. Expand the discussion on training time and sample efficiency, including strategies to reduce training duration (such as sim-to-real approaches), and evaluate the impact of virtual pretraining.\n3.  Discuss the similarities and differences between virtual and real-world environments, focusing on the transferability of reward and observation designs.\n4.  Provide a more detailed analysis of the reset strategy, including its implications for both training and real-world control.\n\nPlease check more details in the [Weaknesses]  section above.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)