Model-free reinforcement learning with noisy actions for automated experimental control in optics

## Abstract

Abstract Experimental control involves a lot of manual effort with non-trivial decisions for precise adjustments. Here, we study the automatic experimental alignment for coupling laser light into an optical fiber using reinforcement learning (RL). We face several real-world challenges, such as time-consuming training, partial observability, and noisy actions due to imprecision in the mirror steering motors.
We show that we can overcome these challenges: To save time, we use a virtual testbed to tune our environment for dealing with partial observability and use relatively sample-efficient model-free RL algorithms like Soft Actor-Critic (SAC) or Truncated Quantile Critics (TQC). Furthermore, by fully training on the experiment, the agent learns directly to handle the noise present.
In our extensive experimentation, we show that we are able to achieve 90 % percent 90 90\% 90 % coupling, showcasing the effectiveness of our proposed approaches. We reach this efficiency, which is comparable to that of a human expert, without additional feedback loops despite the motors’ inaccuracies. Our result is an example of the readiness of RL for real-world tasks. We consider RL a promising tool for reducing the workload in labs.

## 1 Introduction

In experimental physics, we work with complex and sensitive setups. Working in an optics lab means adjusting numerous mirrors, lenses, and other optical elements while optimizing complex parameters. Two of the main challenges are precision and the number of degrees of freedom. Often, tasks have to be repeated frequently. One example of such a task is coupling laser beams into optical fibers, used in many physics labs [, , , ]. It can be a laborious and time-consuming task, especially in experiments with many fibers. Automating tasks like this can, therefore, improve the accuracy and repeatability of experiments and free up domain expertise for more challenging tasks. Most of these repeated tasks have a very clear goal and can either be described as alignment or control problems. Alignment means the correct steering of a laser beam through an optical setup. Control refers to maintaining a dynamic experiment at a desired position using feedback loops. While fiber coupling is primarily an alignment problem, correcting for drift can be considered control.

Automation of alignment and control tasks is a classic use case of reinforcement learning (RL) [, , ]. RL has seen considerable success in recent years, both in general [, , , , , , ] and specifically in robotics [, , , , ]. However, due to many RL algorithms relying on a huge amount of data, at least in environments with continuous action spaces, most of these were performed in simulated or toy environments [, ]. Only very few were done in real-world environments [, , ]. With the recent advance of more sample-efficient algorithms for continuous action spaces, like Deep Deterministic Policy Gradient (DDPG) [], Twin Delayed Policy Gradient (TD3) [], Soft Actor-Critic (SAC) [], and Truncated Quantile Critics (TQC) [], directly training in an experiment has become more feasible. However, we still face several challenges, such as partial observability, time-consuming training, and noise, to mention only a few, when applying RL to real-world setups [, ]. By learning to overcome these challenges, RL has great potential to simplify work in laboratories, particularly optics labs.

In this work, we demonstrate how an RL agent successfully learns to couple light into an optical fiber, reaching efficiencies comparable to those of a human expert. To our knowledge, this has not been achieved before. We set up an experiment for fiber coupling on an optical table, motorizing the mirrors that guide the laser beam into the fiber. Our goal here is to reach a specific coupling efficiency, which is the fraction of light entering the fiber. At first, we tried learning by directly interacting with the experiment using Proximal Policy Optimization (PPO) [] and the typical reset method of resetting the actuator positions to a random interval. However, even after training for two weeks, the return stayed at its minimum value. The main problems we face are partial observability, time-consuming training, and imprecise actuators. To tackle them, we used a very simple virtual testbed, not including noise, to adapt our environment and make it easier for agents to deal with its partial observability. This also offered the ability to test different algorithms in a short time. The motor’s inaccuracies have two major consequences: First, the actions become very noisy. Note here that, in contrast to adding artificial noise to the actions for exploration [, , , , ], in our case, the noisy actions are inherent to the system. We deal with that by directly training in the experiment. Therefore, we do not need an accurate noise model. Second, we are not able to reset in a typical way, so we adopted a unique reset method detailed in Section 4. In that way, we were able to train directly in the experiment with the standard StableBaselines3 [] implementations of SAC and TQC.

Despite the noisy actions, the agent learns to reliably couple to an efficiency of $\geq 90\%\pm 2\%$ starting from a low power over the course of nearly four days. If we only need a smaller efficiency, e.g., $87\%\pm 2\%$, the training only takes twenty hours and can be performed in two nights, not taking away experimenting time. For comparison, the maximum coupling efficiency observed by the experimenter was $92\%$, and the one reached by the agent was $93\%$. We find that tuning the training parameters thoroughly is crucial to reducing training time, which is of high priority for real-world RL applications.

Our study shows how RL can simplify experimental tasks by removing the need for closed-loop control for the motors and enabling remote control. We anticipate that our training results can be transferred to other alignment and control tasks, especially on systems which are not suited for conventional control schemes.
Furthermore, training on the experiment enables us to use schemes based on our results on other black-box systems. This removes the need to create an accurate model of the experiment and shortens implementation time. Especially for more complicated experiments, e.g., within quantum and atom optics, where a simulation of the exact dynamics might not be possible, this is imperative.

## 2 Related work

The application of RL to optical systems ranges through a wide range of topics including optical networks [, , , , , , , , ], adaptive optics [, , , , , ], optical nanostructure, thin films and optical layers [, , , ]. More related to our problem is work that studies how RL can be used to align and control tabletop optical experiments with lasers. In this category, some works are realized merely on simulation. Examples include studying mode-locked lasers [], combining laser beams [], and stacking laser pulses [, ]. Other works include studying how RL performs in an actual experiment. Most of these studies, however, do not train on the experiment but on simulation. Examples include aligning an optical interferometer [, ], operating optical tweezers [], combining laser beams [] and operating pulsed lasers [].
It is rare that the agent is trained directly on the experiment []. One example is a study combining pulsed laser beams []. One actuator performs the actions, and the output is a scalar, the power. The training time is about 4 hours; simulations show that this would quickly go up to 1-2 days if more than two beams should be combined. Another example is the generation of a white light continuum []. The actions are performed here with three actuators, and the observation is an intensity distribution. The authors claim to obtain successful training within 20 minutes. We deal with a higher (4) dimensional action space than both of these works. While training on the experiment can be difficult for many reasons (see Section 3), it can be the last resort in cases where creating a model that accurately represents the noise and dynamics of the system is very time-consuming, if not infeasible. Using a too-inaccurate model, however, would make it impossible to cross the reality gap []. We therefore decided to study this little-explored field.

## 3 Fiber coupling

### 3.1 Experimental setup

To efficiently couple laser light into an optical fiber, we need a specific setup. Our goal is to reach a certain coupling efficiency, which is the fraction of light entering the fiber. To achieve this, the light has to enter the fiber at a specific angle and precise spot. The coupling efficiency depends on how accurately both are matched. To fulfill both constraints, we need two degrees of freedom in each axis, horizontal ($x$) and vertical ($y$). This means we need two mirrors, each tiltable in $x$ and $y$. Furthermore, the laser beam must have the correct size, which is achieved by placing lenses in the correct position before the light enters the fiber. To simplify the setup, we decided to motorize only the mirrors but not the lenses. In addition to the motorized mirrors, we have two mirrors that can be steered with hand-tuneable knobs. This makes it easier for humans to couple light into the fiber. We measure the power at the output of the fiber with a power meter (Thorlabs PM160). With the help of a reference measurement, we can determine the coupling efficiency or normalized output power with an error of $2\%$. Our experiment is depicted in Figure 1, and further details are given in Appendix A.

Figure: Figure 1: Panel (a) and (b) show a conceptual scheme and lab setup of the fiber coupling experiment. Panel (c) shows the dead-zone characterization of the four motorized mirror mount axes on a log scale axis. Dead-zone means movement steps performed by the actuators that do not result in a change in power. Appendix B gives a detailed description of the characterization.
Refer to caption: extracted/5613251/pictures/schematicAndLabPictureAndDeadZone.png

The four actuators moving the mirrors are stepper motors (Thorlabs ZFS 13). They are attached to the mirror mounts, each tilting the respective mirror in one axis. To understand the special constraints of our problem, we move all actuators to a position where we have maximal coupling. Then, while holding the other three actuators fixed, we scan the relevant movement range with one actuator. The power dependence on each motorized degree of freedom looks Gaussian. Fitting it with a Gaussian, we obtained the standard deviations in Table 1. We can see that the coupling peaks are wider in the vertical axis ($y$) than in the horizontal ($x$).

**Table 1: The power as a function of the position of each actuator was fitted by a Gaussian. Its standard deviation in the number of actuator steps (incl. fitting error) is presented here.**
| Actuator | 1x | 2x | 1y | 2y |
| --- | --- | --- | --- | --- |
| Standard Deviation | $11994\pm 34$ | $12769\pm 37$ | $19145\pm 33$ | $17885\pm 39$ |

### 3.2 RL challenges

When we use RL to fiber couple in our lab, we face several challenges. The training is time-consuming as one actuation step takes about 1 second. Furthermore, due to laser safety and possible equipment damage, we have to restrict the movement range of our actuators. The two challenges that are most crucial for shaping our environment are partial observability and motor imprecision.

#### Partial observability

We work with a strongly underdetermined, only partially observable experiment. To describe the state of the experiment perfectly, we would need a lot of information not available to us, e.g., the exact angle, position, and size of the incoming laser beam, as well as the exact position of all of the mirrors and lenses. Even if we could get all of this information at the time of training, environmental drift, such as temperature, would require careful calibration to occur frequently. To make the agent robust against drift, we do not use the actuator positions as part of the observation. Even if we did, they would be very inaccurate due to motor imprecision (see below). Instead, we solely rely on the power at the output of the fiber and the previous actions as our observation. This makes our environment highly partially observable and underdetermined due to four mirror positions leading to one output. Also, the signal is very scarce, as when the motors leave a certain movement range, no power at all gets coupled into the fiber, which means we do not get any feedback (this is why we reset when falling below a certain power).

#### Motor imprecision

Our main challenge is founded in the complex relationship between the expected movement of the used motors and their actual movement, which we call noisy actions. They include backlash of the mechanical system, step loss, non-orthogonal degrees of freedom, i.e., the $x$ and $y$-axis are not independent from each other, and other errors. This leads to noise in the action. To understand its severity, we performed a dead-zone characterization shown in Figure 1 (c). We call dead-zone the actuation steps performed by the motors that have no effect on the system. Although all mirror axes are motorized with the same motor, gearbox, and linear actuator, different dead-zone sizes are observed. A more detailed explanation of the imprecision and its characterization can be found in Appendix B. The variety in the motor imprecision makes the action noise distribution unclear. Therefore, the agent has to learn to deal with the noisy actions in the experiment. Creating an accurate noise model, however, would be even less feasible. On top of that, this affects our reset method (see Section 4).

## 4 Our method

We cannot write down a Markov state (see e.g. [] for an introduction) for our system. Therefore, we treat it as an unknown episodic partially observable Markov decision process (POMDP, see e.g. []). Sampling from it, we get a stochastic process $o_{1},a_{1},r_{1},o_{2},a_{2},r_{2},...,o_{\tau}$, where $o_{t},a_{t}$ and $r_{t}$ are observations, actions and rewards at the discretized time $t$, and $\tau$ is the episode length limitited by the maximal episode length $T$, i.e. $\tau\leq T$. See Tables 2 and 3 for environment hyperparameters.

#### Actions

We treat our action space as the 4-dimensional continuous action space $[-1,1]^{\times 4}$. At time $t$, we can decompose the action $a_{t}$ as $a_{t}=(a_{\mathrm{m}1x},a_{\mathrm{m}1y},a_{\mathrm{m}2x},a_{\mathrm{m}2y})$ where each component belongs to a different actuator. For example, $a_{\mathrm{m}1x}$ belongs to the actuator that tilts mirror $1$ in the horizontal ($x$) direction. Each of these actions is then multiplied by the maximum allowed action in actuator steps $a_{\mathrm{max}}$, rounded to the next integer, and sent to the different actuators. However, in the experimental environment, the actuators have no feedback loop, so, potentially, they move significantly less due to their imprecision. This makes our actions noisy.

**Table 2: Environment parameters for main experiments where $P_{\text{min}}-0.1$ is the lowest power where the training starts, $P_{\text{fail}}$ is the power where the agent fails and the reset is called, $P_{\text{goal}}$ is the power the agent should learn to reach, $T$ is the maximal episode length in time steps, and $n$ is the history length used in the observation.**
| Parameter | $P_{\text{min}}$ | $P_{\text{fail}}$ | $P_{\text{goal}}$ | $a_{\text{max}}$ | $T$ | $n$ |
| --- | --- | --- | --- | --- | --- | --- |
| Value | $0.2$ | $0.05$ | $[0.8,0.9]$ | $6\cdot 10^{3}$ | $30$ | $4$ |

#### Observations

As we are dealing with a partially observable system, for successful training, we need to take great care in defining our observations. The only thing we observe in our environment is the coupling efficiency or normalized power, denoted $P\in[0,1]$. For example, $P_{t}$ denotes the normalized power at time $t$. As usual in POMDPs, we include a history of length $n\in\mathbb{N}$ in the observation (see e.g. [, ]). This would lead to an observation like $o_{t}=(P_{t-n},...,P_{t-1},P_{t})$ at time step $t$. It is common in RL experiments to observe the environment before and after an action, not during this action. However, this is not the only information available to us: In principle, we can record the power almost continuously while the actuators are moving, i.e., we can record $(P_{t},P_{t+1/m_{t}},...,P_{t+1})$ during action $a_{t}$ where $m_{t}$ is the number of powers measured during that time. In our experiment, we noticed that it is beneficial to use some of this information in our observation (see Section 5). In particular, we take the average power $P_{\mathrm{ave},t}=\sum_{i=0}^{m_{t}}P_{t-1+i/m_{t}}/(m_{t}+1)$ the maximal power observed $P_{\mathrm{max},t}=\max_{i=0,...,m_{t}}P_{t-1+i/m_{t}}$ and its relative position in the list of powers $x_{\mathrm{max},t}=(\mathrm{arg}\,\max_{i=0,...,m_{t}}P_{t-1+i/m_{t}})/(m_{t}+1)$ into account. In addition, we want the performed actions to be part of the observation. This leaves us with the observation

$$ $o_{t}=\left(\left(P_{k-1},a_{k-1},P_{\mathrm{ave},k},P_{\mathrm{max},k},x_{ \mathrm{max},k}\right)_{k=t-n,...,t},P_{t}\right),$ $$

i.e., the observation includes a history of the power before taking an action, the action, the average and maximum power and its relative position during the action, and the power afterward.

#### Episode and Resets

We reset the environment either after a chosen maximum episode length $t=T\in\mathbb{N}$, when the agent reached its goal (i.e., $P_{t}>P_{\text{goal}}$), or the agent failed (i.e., $P_{t}<P_{\text{fail}}$). A common way of resetting at the start of an episode is to move to a random position within a defined range. However, our actuators do not present the required precision for this. We, therefore, need a different way to reset. We nevertheless define neutral positions given in motor steps. These are positions where we had high power when we started our training. When we return to the neutral positions during training, depending on the original actuator position, the power varies between no power and high power.
The reset procedure depends on the last power value, and we want the power after the reset to be higher than $P_{\text{min}}$. We distinguish between 3 cases. If, during the reset procedure, the condition of a different case applies, we jump to the corresponding case:
1. $P_{t}\geq P_{\mathrm{min}}$: we choose a random power between $P_{\mathrm{min}}+0.1$ and $P_{\mathrm{goal}}$ and do random steps until we decrease the power below the chosen power value.
2. $0.09<P_{t}<P_{\mathrm{min}}$: we first reverse the last action. As long as the power is still under $P_{\mathrm{min}}$, we move the actuators one after the other in random order in the direction in which the power increases. If the power decreases, we change the actuator’s direction of movement. We repeat this process until we reach $P_{t}\geq P_{\mathrm{min}}$.
3. $P_{t}<0.09$ or every ten episodes: We first move to the neutral positions. From there, we do random steps until $P\geq 0.09$, and then follow the procedure of Case 1 or 2 depending on the power. The values of $0.09$ and ten episodes were determined empirically by observing the algorithm performing on the experiment.
Before starting the episode, we always perform some random steps to randomize the process more. Our reset procedure introduces a small dependence between successive episodes. However, this did not affect the training performance in the virtual testbed much (see. Appendix C). Furthermore, due to the motors’ inaccuracies, full independence was not possible in this experiment.

#### Rewards

We design our reward with the purpose of making the agent reach the goal as quickly as possible.
The agent gets a low negative reward when failing ($P_{t}<P_{\text{fail}}$), a high reward when reaching the goal ($P_{t}>P_{\text{goal}}$), and else a small reward every step depending on the power. We define the reward function depending on the current power $P_{t}$, the time step $t$, the goal power $P_{\text{goal}}$, minimal power $P_{\text{min}}$, fail power $P_{\text{fail}}$ and the episode length $T$ as

$$ $\displaystyle r_{t}=$ $\displaystyle R(P_{t},t,T,P_{\text{fail}},P_{\text{goal}},P_{\text{min}})$ $\displaystyle=$ $\displaystyle\left\{\begin{array}[]{ll}-A_{f}\left((1-\alpha_{f})\exp{(-\beta_ {f,1}\frac{t}{T})}+\alpha_{f}\exp{(-\beta_{f,2}\frac{P_{t}}{P_{\text{fail}}})} \right)&\text{if }P_{t}<P_{\text{fail}}\\ A_{g}\left((1-\alpha_{g})\exp{(-\beta_{g,1}\frac{t}{T})}+\alpha_{g}\exp{(\beta _{g,2}\frac{P_{t}}{P_{\mathrm{goal}}})}\right)&\text{if }P_{t}>P_{\text{goal}} \\ \frac{A_{s}}{T}\left((1-\alpha_{s})\exp{(\beta_{s}(P_{t}-P_{\mathrm{goal}}))}+ \alpha_{s}\left(P_{t}-P_{\mathrm{min}}\right)\right)&\text{else }\end{array}\right.$ $$

where $\beta_{s},\beta_{f1},\beta_{f2},\beta_{g1},\beta_{g2},A_{s},A_{f},A_{g}\in
\mathbb{R}$, $\alpha_{s},\alpha_{f},\alpha_{g}\in(0,1)$(see Table 3 for their values in the main experiment). The return should never be higher when staying just below the goal than when actually reaching the goal. In the same way, it should always be better to stay just above the failing threshold than to fail. We enforce this by choosing the amplitudes according to $A_{f},A_{g}\geq A_{s}$. Each of the rewards consists of two terms. The $\alpha$’s are used to weigh their importance relative to one another. The failing reward consists of a term punishing it less when the agent fails later in the episode and a term punishing it less when the power with which the agent fails is close to the failing threshold. The two terms in the goal reward ensure that the agent is rewarded more for reaching the goal quickly and for reaching it with a higher power. The step reward contains both an exponential and a linear part to ensure that the agent clearly notes a change to higher powers for low and high values.

**Table 3: Reward hyperparameters for main experiments**
| Parameter | $A_{s}$ | $A_{f}$ | $A_{g}$ | $\alpha_{s}$ | $\alpha_{f}$ | $\alpha_{g}$ | $\beta_{s}$ | $\beta_{f1}$ | $\beta_{f2}$ | $\beta_{g1}$ | $\beta_{g2}$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Value | $10$ | $100$ | $100$ | $0.9$ | $0.5$ | $0.5$ | $5$ | $5$ | $5$ | $5$ | $1$ |

#### Virtual testbed

We create a virtual testbed to test out various RL algorithms and investigate differently designed environments before training on the actual experiment. In Section 3, we fitted the power depending on the position of each individual actuator with Gaussians. By multiplying them, we get a map from all four actuator positions to the power. We then set the amplitude to the highest power we observed until that point, which is $0.92$.
We use this to create a simplified virtual environment. It does not include any noise, as the acquired data is not suitable for creating an accurate noise model. Accordingly, the virtual testbed is a strong simplification of the actual experiment. Nevertheless, it helped us get various insights on what to do in the real experiment.

For both the main experiment and the experiments in the virtual testbed, we wrote a gymnasium environment [] and used the algorithms from StableBaselines3 [] with standard hyperparameters further discussed in Appendix E. For the main experiments, we used TQC and SAC as algorithms and the parameters provided in Tables 2 and 3.

## 5 Experimental results

### 5.1 Virtual testbed

We used the virtual testbed to optimize for all parameters described in Section 4. This led to the results in Tables 2 and 3, which we use throughout this section. All training runs on the virtual environment presented in this paper (including the appendix, not including preliminary runs) took us around 6 days in total on an NVIDIA A100 GPU. An individual run took $20$ minutes per $10^{5}$ training steps. We highlight the most interesting insights here, the other results can be found in Appendix C. All experiments on the virtual testbed were performed $5$ times. For all figures, we show the mean with $2\sigma$ error bands, calculated using ewm.std() from pandas [, ] for smoothed functions and the standard deviation of several runs in the case of multiple runs.

#### Observation

We tested if removing $P_{\text{ave}}$ or $P_{\text{max}}$ and $x_{\text{max}}$ from the observation influences the performance. Figure 2 (a) shows that TQC performed worse on any of these combinations compared to the full observation presented above. However, leaving out $P_{\text{ave}}$ had a much smaller impact than leaving out $P_{\text{max}}$ and $x_{\text{max}}$, which makes the latter very important for us.

Figure: Figure 2: Virtual testbed results for the return plotted against the training step. In Panel (a), different observation variants are compared to each other using $P_{\text{goal}}=0.85$. In Panel (b), different algorithms are compared to each other (see Appendix E for their hyperparameters) using $P_{\text{goal}}=0.8$.
Refer to caption: x1.png

#### Algorithms

We investigated the performance of different RL algorithms. The continuous action space limits our choice of algorithm. We trained TQC, TD3, SAC, DDPG, PPO and Advantage Actor-Critic (A2C) [] in their standard StableBaselines3 [] implementations as discussed in Appendix E with $P_{\text{goal}}=0.8,0.85,0.9$ for $10^{5}$ time steps. For $P_{\text{goal}}=0.8$, this can be seen in Figure 2 (b). For this number of training steps, as expected, PPO and A2C performed worst as they both do not use a replay buffer. They are closely followed by DDPG. SAC, TD3 and TQC performed much better with a similar performance. SAC performed almost always better than TD3. TQC performed mostly better than TD3 and comparable to SAC but has a small drop around $2\cdot 10^{4}-5\cdot 10^{4}$ time steps.

#### Goal Power

The higher the goal power, the longer it takes for the training to converge. Because of this, we checked if it helps to start with lower goal powers and raise the goal power during training. For $P_{\text{goal}}=0.9$, we found that it can help to pre-train on lower goal powers and raise the power in steps, especially starting from $P_{\text{start, goal}}=0.85$, see Figure 10 in Appendix C.

### 5.2 Optics lab

After shaping the environment and agents on the virtual testbed, we performed training runs in the optics lab. For the training on the experiment, we used an NVIDIA GeForce RTX 4070 GPU. In addition to the usual packages [, , , ], we used PyLabLib, Thorlabs Kinesis, PyVisa, Keysight Connection Expert and safe-exit [, , , , ] for communication with the experiment. In addition to the standard deviations discussed above, we include the relative power measurement error of $2\%$ in the error bands in the figures here. Our training speed is limited by the time the actuators take to move, leading to each time step taking about a second. All training runs in the lab appearing in the paper (including the appendix, not including preliminary runs) took around $20$ days in total. We let each training run until its return starts to converge. For $P_{\text{goal}}\leq 0.87$, this took around 20 hours or $4\cdot 10^{4}$ steps. That we can train successfully in only $4\cdot 10^{4}$ steps is a result of our environment shaping discussed earlier in this section and in Appendix C. If we set a very high goal power ($P_{\text{goal}}=0.9$), the training takes much longer ($2\cdot 10^{5}$ time steps, which sums up to nearly 4 days of training).

Figure: Figure 3: Experimental results. Panels (a) and (b) show the return plotted against training steps for different agents. We use TQC agents except for the yellow curve, where we use SAC. Of the agents trained on $P_{\text{goal}}=0.9$ (shown in (b))) one agent was first pre-trained on lower, over time increasing, goal powers. Panels (c)-(e) show our results when testing the agents marked with a star in (a) and (b). For testing, we reset and let the agents fiber couple (this is tried $100$ times for the RL agent and $25$ times for the human expert). If the RL agents do not reach their goal within 30 steps, we reset the environment (still measuring the time), and the agent continues from there. This is repeated until the agent reaches its goal. Panel (c) shows the power plotted against the time for the first $35\text{\,}\mathrm{s}$. The error band is clipped to the power range $[0,1]$. Panel (d) compares the number of environment steps it took for each of the RL agents tested to reach their goal after their last reset. Panel (e) shows the time each agent trained to $P_{\text{goal}}=0.9$ and a human expert took to reach that goal.
Refer to caption: x2.png

In the different training runs, we changed algorithms and goal powers. Up until a goal power of $P_{\text{goal}}=0.87$, training runs start to converge at around $4\cdot 10^{4}$ steps. For higher goal powers, e.g., $P_{\text{goal}}=0.9$, this is not the case anymore. This is shown in Figure 3 (a). As the performance of SAC and TQC was comparable on the virtual testbed, we performed two training runs per algorithm on the experiment with a goal power of $P_{\text{goal}}=0.85$. We can see that in the beginning, the return rises quicker for SAC but is slightly outperformed by TQC later on. This is why we chose TQC for all other experiments presented here.

When we choose $P_{\text{goal}}=0.9$, we need significantly more training steps ($\geq 2\cdot 10^{5}$). Because of that and the results we got from the virtual testbed, we also tested pre-training on lower goals on the experiment, i.e., we started the training with a low goal power $P_{\text{goal}}=0.85$ and increased it in small increments to $P_{\text{goal}}=0.9$ over the course of the first $10^{5}$ training steps and left it at that for the next $10^{5}$ steps. These training runs are shown in Figure 3 (b).
The return of the agent first pre-trained on lower powers always drops after changing to a higher goal power, as it first needs to learn to handle the conditions of the changed environment. We can also see that the return reaches lower values the higher the goal gets. This is expected as the task gets harder each time. The return for the model pre-trained on lower goals reaches higher values than the one starting from scratch. Additionally, we can use intermediate models to align the experiment to lower powers. We conclude that pre-training on lower goals was helpful for such a high goal in our experiment (see also []).
In contrast to that, we found it not helpful to pre-train on the virtual testbed. However, one still might use the virtual testbed to train an agent that can couple light into the fiber less efficiently (see Appendix D).

To understand the help for our everyday lab work, we tested a few of our agents in the lab (marked with a star in Figure 3 (a) and (b)). The test results are shown in Figure 3 (c)-(e). Each of the RL agents was tested a hundred times, which took a total of $20-30$ minutes. We measure the power over time. If the agent does not reach the goal during an episode, we reset the environment, and the agent tries from there. One episode was at most $40\text{\,}\mathrm{s}$, and the longest attempt took around $350\text{\,}\mathrm{s}$. Panel (c) shows the power plotted against time for the four tested agents. The agent trained with $P_{\text{goal}}=0.85$ stays on top of both the agent trained with $P_{\text{goal}}=0.87$ and one of the agents trained with $P_{\text{goal}}=0.9$ for some time. This is due to the first agent reaching its goal faster than the other ones. Panel (d) shows the number of steps it took for the same four agents to reach their goal after their last reset. As expected, the agent with the lowest goal $P_{\text{goal}}=0.85$ is the fastest in fiber coupling. Furthermore, of the agents trained with $P_{\text{goal}}=0.9$, the one pre-trained on lower goal powers reaches this high goal faster. Interestingly, although the agent trained on $P_{\text{goal}}=0.87$ has to reach a lower goal than the pre-trained one with $P_{\text{goal}}=0.9$, the former is not faster than the latter.
For comparison, we also tested a human expert $25$ times on how long they would take to couple the fiber to $P_{\text{goal}}=0.9$ using the hand steering mirrors. This, however, is not a fair comparison. The RL agents can change all four degrees of freedom at once. The experimenter, on the other hand, has access to more information, e.g., the continuous power measurement while moving an actuator, and does not have to deal with the imprecision in the actuators, which means they can easily go back to an observed maximum. Despite this, we can see in Panel (e) that the RL agents are generally faster but take longer in a few cases, where the agent needs several episodes to get to the goal. Our hypothesis is that this is due to our episodes not being fully independent of each other. In conclusion, we show that by using RL, we can consistently couple light into an optical fiber to high efficiencies. Our results suggest that pre-training on lower goals can be helpful.

Another key result is that by using RL, we can overcome the imprecision of our motors.
Using a classic algorithm such as gradient descent would fail with these motors as the dead-zone deters us from experimentally determining a gradient and then going back to the starting position. Another way of dealing with the motors’ imprecision would be to implement a feedback loop. This, however, makes their design much more complicated. Using RL, we can circumvent this: no additional feedback loop is needed as the agent learns to handle the imprecision.

The method described here can also be used to control for drift, i.e., bringing the alignment back to its ideal point after it was lost due to changes in the lab (e.g., thermal alignment drifts, changes in other parts of the optical table, etc.). For this, we can use the continuous measurement of the power at the output of the fiber and call the agent to set it back to the desired coupling efficiency whenever it drops below a certain value.

## 6 Summary and Outlook

We have shown that our model-free RL agent successfully learns to couple laser light into an optical fiber, reaching the same efficiencies as a human expert while generally being faster. We used a virtual testbed to tune the parameters of the environment and select convenient RL algorithms before realizing the actual training on the experiment. We find that sample-efficient algorithms that use a replay buffer, such as SAC and TQC, are a must to overcome the challenge of otherwise not manageable training time. Partial observability can be dealt with by carefully tuning the observations. As we train directly on the experiment, the agent learns to handle the specific noise present, and we can avoid creating an accurate simulation of the task. This makes it easy to use on other setups. Our study suggests that pre-training to lower goals first can help achieve more difficult goals.

A central result is that the agent learns to deal with the imprecision of the mirror steering motors. Their movements are too inaccurate to use classic optimization techniques like gradient descent. One way to handle such imprecision is by implementing additional feedback loops. Using RL, we can avoid this, which helps to simplify the design of motors and experimental setups. Generally, automation gives us the possibility of remote-controlling experiments, which can be especially useful in experimental areas that are difficult to access, such as in vacuum tanks, in clean room facilities, or, in extreme cases, in underground labs or in space.

Further exploration could include investigations on how the agents perform for other modes of light, including multiple local maxima, and investigating the performance of model-based or hybrid algorithms. We start our RL training procedure under conditions where there is low power. This raises the question how, or with how many additional sensors, we could generalize this to the case of starting with no power. Furthermore, it might be interesting to explore replacing the use of a history as our observation by PID-inspired RL [].

Our experiments are a first step towards the extensive use of RL in our quantum optics laboratory. Optical experiments typically require various control loops to stabilize the experimental degrees of freedom against perturbations. These locks significantly increase the complexity of the experiments. For example, maintaining the length of optical cavities to achieve resonant light field enhancement requires complex components such as phase modulators [], homodyne detectors [, ], or split detectors [, ]. RL offers the potential for streamlined control loops that rely solely on the measurement of power in the reflection and transmission of the resonator. This could enable novel control strategies, such as phase control of squeezed vacuum states. These states are characterized by unique quantum noise properties but are otherwise dark. Consequently, phase control typically requires an auxiliary laser field [] or introduces unwanted phase noise []. RL has the potential to provide a noise-free solution without the need for additional laser fields, which is particularly relevant for large-scale on-chip squeezing experiments in the field of quantum information [].

#### Limitations

As with any experimental study, our findings have limitations. Due to the high training times on the experiment, we did not systematically perform several training runs with the same agent to compare them but only did so on the virtual testbed which does not include noise. Furthermore, our episodes are not fully independent of each other and can have self-driving effects.

#### Data availability

The Python code used for obtaining the results presented here is available at [https://github.com/ViktoriaSchmiesing/RL_Fiber_Coupling](https://github.com/ViktoriaSchmiesing/RL_Fiber_Coupling).

## Acknowledgments and Disclosure of Funding

We thank Amir Abolfazli, Timko Dubielzig, Ludwig Krinner, and Kilian Stahl for helpful discussions, Pratik Chakraborty and Manuel Schimanski for help in the lab, and Nived Johny, Jonas Junker, Arindam Saha and Bernd Schulte for both of them. LR thanks Mika Nina Seibicke and Alina Jankowski for lab sitting and emotional support.
This work was supported by the Quantum Valley Lower Saxony (QVLS), the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) through SFB 1227 (DQ-mat), Germany’s Excellence Strategy EXC-2123 QuantumFrontiers 390837967, and Excellence Strategy EXC-2122 PhoenixD 390833453, the BMBF projects ATIQ and QuBRA, the BMWK project ProvideQ, and the Australian Research Council Centre of Excellence for Quantum Computation and Communication Technology (project number CE170100012).

## Appendix A Additional details about the experimental setup

Our setup includes the following components: We use a $1064\text{\,}\mathrm{nm}$ laser (Mephisto, Coherent), a single mode polarization-maintaining fiber, and a Schäfter+Kirchhoff fiber collimator (60FC-SF-4-M8-08) at the input side of the fiber. The measurements of laser power are done with power meters (Thorlabs PM160, measurement error $1\%$). In front of the experiment, we place a partially reflecting mirror to measure a fraction of the laser light with an additional power meter for power reference. The measurement is used to pause training in the event of a laser failure and to track power fluctuations to determine the maximum power level. In this way, we can determine the coupling efficiency with an error of $2\%$. The input power is set to $1.00(1)\text{\,}\mathrm{mW}$.

Due to safety constraints, we have to limit our state space and, in consequence, clip our actions if the actuator positions would otherwise move out of a certain range because it is unacceptable for the laser beam to wander around the room. Also, in the first test run, the action size was chosen so poorly that the mirror mounts were damaged. So, both laser safety and equipment damage are hazards that we need to consider.

## Appendix B Explanation of the imprecision in the mirror steering motors and its characterization

Backlash is a phenomenon that is present when a load is not directly connected to a motor, such as in geared mechanical systems []. Dependent on the exact geometry of the system, i.e., mechanical tolerances, amount of gears, etc., it may resemble hysteresis between the expected and actual position or a dead-zone, where moving the actuator has no effect on the actual position whenever the rotational direction is reversed. It is thus hard to model and predict a priori. Control has to be implemented based on the specific system and its use. These control schemes include hysteresis models, dead-zone models, and PI control. However, additional sensors are needed to get accurate feedback if multiple actuators are used. Step loss results from the difference in static and dynamic torque of a motor. The motor steps result in a linear actuation, which changes the tilt of the mirror mount. Different tilt angles lead to different static loading of the motor and gearbox. This may lead to the initial step(s) being lost, as the motor can not deliver the starting torque, resulting in a partial step. Without feedback from, e.g., an encoder, this leads to a difference between the expected and actual position. Lastly, the non-orthogonal degrees of freedom are a result of the kinematic mirror mounts used and their mounting. Usually, this error is small for well-designed kinematic mirror mounts.

We performed a dead-zone characterization. The core idea is to initiate a number of movements, i.e., generate a movement history, after which a maximum dead-zone is expected. This can simply be an initial long movement in one direction followed by a direction reverse. The long movement ensures a nearly linear behavior between the expected and actual position, as backlash is overcome in the mechanical system. The backlash after a change in rotational direction should, therefore, be large. Additionally, the movement history is similar between repetitions, enabling their comparison.
Starting from a position with high coupling, we moved one actuator far out, then back to high coupling. From there, we reversed the movement and counted the steps the actuator had to move before the measured power changed by more than the power measurement error. Repeating this process $100$ times yielded the distribution of the dead-zone size, shown in Figure 1 (c) for the four motors. This data helps us understand the uncertainty of the mirror mount movements. As no continuous feedback is employed, characterization of other positioning errors is not possible in our setup.

## Appendix C Environment and agent tuning on virtual testbed

We used the data of scanning each motor individually through the coupling peak to create a virtual testbed. Each dataset was normalized and fitted with a Gaussian; all of them were then multiplied. The highest coupling efficiency we had measured up to this point was $0.92$; therefore, we use this as the amplitude. The following function, based on the fit values in motor steps, describes our virtual testbed:

$$ $\displaystyle P(x_{m1},y_{m1},x_{m2},y_{m2})=0.92\mathrm{exp}\Bigg{(}-\frac{1} {2}\Bigg{(}$ $\displaystyle\left(\frac{x_{m1}-5470785}{11994}\right)^{2}+\left(\frac{y_{m1}- 5573194}{19145}\right)^{2}$ $\displaystyle+\left(\frac{x_{m2}-5461786}{12769}\right)^{2}+\left(\frac{y_{m2} -5178016}{17885}\right)^{2}\Bigg{)}\Bigg{)}$ $$

We use this testbed to gain insights into the environment hyperparameters, observations, and algorithms to use in the following order.
First, we optimized the hyperparameters of the reward ($\alpha$’s, $\beta$’s,$A$’s). Next, we went to the parameters of the environment that appear in the reward, i.e., the goal power $P_{\text{goal}}$ and episode length $T$. The usual figure of merit is the return in dependence on the training step. However, this depends on the reward, and the reward depends on both of these sets of parameters. Therefore, it is not possible to use the return as a figure of merit for these parameters. Instead, we trained a TQC agent for a total of $10^{5}$ timesteps. We tested the agent every $10^{4}$ timesteps for $100$ episodes, noting the probability of failure, the probability of reaching the goal, and the average power at the end of each episode. Our main figure of merit was the probability of reaching the goal after a training time in the range of $10^{4}$ to $4\cdot 10^{4}$ time steps. Still, we also took the probability of failure and the average power at the end of each episode into account. We show the second one here only when we used it for our decision.
After fixing the first two sets of parameters, we were able to use the return to compare other environment parameters, such as the length of the history in the observation and the maximal action, and different algorithms. All studies in the virtual testbed were performed at least $5$ times and, except for the algorithm tests, used TQC as the algorithm as this was the algorithm most used in the experiment. If not stated otherwise and if these were not the parameters being changed, we used the parameters in Tables 2 and 3.

### C.1 Reward Hyperparameters

We want a high probability of reaching the goal after the least amount of training time, so we shape the reward function accordingly. For tuning its hyperparameters, we chose $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{s}=0.5,\ \alpha_{f}=0.9$, $A_{f}=10$, in contrast to Tables 2 and 3, if those parameters were not the ones being changed. We tested the tuples of reward parameters given in Table 4. For each parameter we tested a number of different values and also checked the dependence of the variables on each other. After evaluation, we decided on the parameters in Table 3. The subscript $s$ always refers to the step reward, $f$ to the fail reward, and $g$ to the goal reward. The other parameters were $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{s}=0.5,\ \alpha_{f}=0.9$, $A_{f}=10$ or given in Tables 2 and 3.

**Table 4: test table tuning parameters**
| Parameter | $(A_{f},A_{g})$ | $(\alpha_{s},\beta_{s})$ | $(\alpha_{f},\beta_{f1},\beta_{f2})$ | $(\alpha_{g},\beta_{g1},\beta_{g2})$ |
| --- | --- | --- | --- | --- |
| Value | $\{10,100,1000\}^{2}$ | $\{0.1,0.5,0.9\}$ | $\{0.1,0.5,0.9\}$ | $\{0.1,0.5,0.9\}$ |
|  |  | $\times\{1,5,10\}$ | $\times\{1,5\}^{2}$ | $\times\{1,5\}^{2}$ |

#### Prefactors

First, we tested different prefactors $A_{f}$ and $A_{g}$. The results are shown in Figure 4.

Figure: (a) Probability of reaching the goal in dependence of the training step.
Refer to caption: x3.png

Looking at the probability of reaching the goal, $A_{f}=100$ seems to be the best value. For training steps over $3\cdot 10^{4}$, we can see that $A_{g}=1000$ seems to be better than the other two values, before it seems that $A_{g}=100$ is doing better. However, if we look at the probability of failure, we can see that using $A_{g}=100$, the probability of failure falls more quickly than if we are using $A_{g}=1000$. Because resets after failure take a lot of time for this kind of $P_{\text{min}}$ and $P_{\text{fail}}$, we want the failure probability to be as low as possible and go with $A_{g}=100$.

#### Step Reward

Second, we are looking at the step reward and optimizing for $\alpha_{s}$ and $\beta_{s}$. Figure 5 shows the probability of reaching the goal.

Figure: Figure 5: Results for tuning the parameters of the step reward: Probability of reaching the goal for different $\alpha_{s},\beta_{s}$ in the reward with TQC, $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{f}=0.9$, $A_{f}=10$ and all other parameters as in Tables 2 and 3.
Refer to caption: x5.png

We deem $\beta_{s}=5$ and $\alpha_{s}=0.9$ to be the best parameters, although there is not a very strong difference.

#### Goal Reward

Third, we are looking at the goal reward and optimizing for $\alpha_{g},\beta_{g1}$ and $\beta_{g2}$. Figure 6 shows the probability of reaching the goal.

Figure: Figure 6: Results for tuning the parameters of the goal reward: Probability of reaching the goal for different $\alpha_{g},\beta_{g1},\beta_{g2}$ in the reward with TQC, $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{s}=0.5,\ \alpha_{f}=0.9$, $A_{f}=10$ and all other parameters as in Tables 2 and 3.
Refer to caption: x6.png

Here, there is a stronger difference between the different parameters. We are going with $\alpha_{g}=0.5,\beta_{g1}=5$ and $\beta_{g2}=1$. The other possibility would be $\alpha_{g}=0.1,\beta_{g1}=1$ and $\beta_{g2}=5$, which is worse in training steps $1\cdot 10^{4}-2\cdot 10^{4}$ but better in training steps $4\cdot 10^{4}-5\cdot 10^{4}$. However, we put our focus on the earlier phases of training and also do not want to emphasize the power with which the goal was reached that much over the time in which it was reached, which is why we go with the first choice of parameters.

#### Fail Reward

Lastly, we are looking at the fail reward and optimizing for $\alpha_{f},\beta_{f1}$ and $\beta_{f2}$. Figure 7 shows the probability of reaching the goal.

Figure: Figure 7: Results for tuning the parameters of the fail reward: Probability of reaching the goal for different $\alpha_{f},\beta_{f1},\beta_{f2}$ in the reward with TQC, $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{s}=0.5$, $A_{f}=10$ and all other parameters as in Tables 2 and 3.
Refer to caption: x7.png

Here, the choice is again not that clear, but we deem $\beta_{f1}=\beta_{f2}=5$ and $\alpha_{f}=0.5$ to be the best choice of parameters.

### C.2 Episode Length

We want to find a good trade-off between reaching the goal quickly and reaching it reliably. Using the same parameters as for reward shaping, we tested different episode lengths, in particular, $T=5,10,...,50$. The results are shown in Figure 8.

Figure: Figure 8: The probability of reaching the goal in dependence of the training step for different maximum episode lengths $T$ and maximum actions $a_{\text{max}}$ with TQC, $P_{\text{fail}}=0.2,\ P_{\text{min}}=0.4,\ P_{\text{goal}}=0.8,\ T=20$, $\alpha_{s}=0.5,\ \alpha_{f}=0.9$, $A_{f}=10$ and all other parameters as in Tables 2 and 3.
Refer to caption: x8.png

As expected, the longer the episode, the higher the probability of reaching the goal (or failing). For some of these (i. e. $T=20,30,35,40$, we also varied the maximum allowed actuator steps per environment step $a_{\text{max}}$ (i. e. doing simulations with $a_{\text{max}}=[2\cdot 10^{3},10^{4}]$ to see if it had an effect on this, which we could not confirm.
However, we also have to take into account that longer episodes will take more time in the experiment. This is why we select $T=30$, as there is not a very big difference between this and $T>30$.

### C.3 Reset Methods

In the virtual testbed, we compare the following reset methods:

- A
Reset as described in the main paper (for testing at the start and end of training).
- B
Reset as described in the main paper, but first, go to neutral positions in every episode.
- C
Reset by choosing random positions for all four actuators in an interval of width $4.2\cdot 10^{4}$ around the neutral positions.

We used the parameters from Tables 2 and 3 and $P_{\text{goal}}=0.85$. The results can be seen in Figure 9.

Figure: Figure 9: Comparison of different reset methods. The methods are the following: A – Reset as described in the main paper (for testing at the start and end of training), B – Reset as described in the main paper, but first, go to neutral positions in every episode, C – Reset by choosing random positions for all four actuators in an interval of width $4.2\cdot 10^{4}$ around the neutral positions. We use the parameters from Tables 2 and 3, TQC and $P_{\text{goal}}=0.85$. Panel (a) shows the starting powers. Panel (b) shows the return in dependence on time.
Refer to caption: x9.png

In Panel (a), we can see the starting power for the different reset methods. Using method C, the starting distribution of powers is very different from the other reset methods. The median is similar, but the standard deviation is much higher. This led us to compare methods A and B additionaly. In method B, the median is slightly higher and independent of our policy. For method A, the distribution depends on the model used, and the median and $75^{\text{th}}$ quantile are slightly higher and more comparable to the one of B after $10^{5}$ training steps than at the start. Because of this, the return for method B is slightly higher than for method A, especially in the middle, as can be seen in panel (b). We would have expected a higher impact from the reset method. However, the differences are quite small. In conclusion, even though our method makes our episodes not fully independent of each other, we do not gain an artificial benefit from it.

### C.4 Goal

The goal power is fully our choice. First, we check at what point the return starts to converge for which goal power. Therefore, we look at the return in dependence on the training steps for different goal powers. This is shown in Figure 10 (a).

Figure: Figure 10: Panel (a) shows the return in dependence of the training step for different goal powers. Panel (b) shows the probability of reaching the goal power $P_{\text{goal}}=0.9$ after $10^{5}$ training steps in dependence of $P_{\text{goal, start}}$. Hereby, the goal on which the model is trained either rises in a linear (orange) or step (blue) function of the training step from $P_{\text{goal, start}}$ to $P_{\text{goal}}=0.9$ over the course of $10^{5}$ training steps.
Refer to caption: x10.png

We can see that the higher the goal power, the lower the return after convergence, and the later the return converges. We can see that this point happens significantly later for high goal powers. Also, for high goal powers like $P_{\text{goal}}=0.9$, the distance to the last curve is bigger than, for example, for $P_{\text{goal}}=0.86$.

Because of this, we wondered if it made sense to pre-train on lower goal powers. We tested this by starting with goal powers $P_{\text{start, goal}}=0.5,0.7,0.8,0.85,0.875$ and raising it to $0.9$ over the course of $10^{5}$ training steps either linearly, i.e., raising it slightly in every training step, or in a staircase way, i.e. raising it more every $10^{4}$ training steps. Figure 10 shows the probability of reaching the goal $P_{\text{goal}}=0.9$ after $10^{5}$ training steps in dependence of the starting goal power $P_{\text{start, goal}}$ for the two different manners of raising the goal power. We can see that it can be helpful to raise the goal power in steps, especially starting from $P_{\text{start, goal}}=0.85$.

### C.5 Observation

We tested history lengths of $n=1,...,6$ with $P_{\text{goal}}=0.85$. The results can be found in Figure 11 (a).

Figure: Figure 11: Comparison of different history lengths and maximum actions. The return in dependence of the training step using the parameters in Tables 2 and 3 and $P_{\text{goal}}=0.85$ for different history lengths $n$ (Panel (a)) and maximum actions $a_{\text{max}}$ (Panel (b)).
Refer to caption: x11.png

Depending on the training step, $n=3,4$ lead to the highest return. We went with $n=4$ as this was higher around $2\cdot 10^{4}$ to $5\cdot 10^{4}$ training steps. We also tried to leave $P_{\text{ave}}$ or $P_{\text{max}}$ and $x_{\text{max}}$ out of the observation. The results are shown in Figure 2 (a) and are discussed in the main paper.

### C.6 Action

We tested different maximal actions $a_{\text{max}}=2\cdot 10^{3},4\cdot 10^{3},5\cdot 10^{3},...,10^{4}$ with $P_{\text{goal}}=0.85$ to see which yields the highest return. Maximum actions between $4\cdot 10^{3}$ and $8\cdot 10^{3}$ generally performed best ($4\cdot 10^{3}$ performed best out of them). Because of the imprecision of the motors, we also did some tests in the lab, which is why we selected $a_{\text{max}}=6\cdot 10^{3}$ for our experiments. This is approximately half of the standard deviation of the Gaussian in $x-$ direction.

### C.7 Algorithms

We tested six different algorithms with their standard hyperparameters in StableBaselines3 with the parameters in Tables 2 and 3 for $P_{\text{goal}}=0.8,0.85,0.9$ for either $10^{5}$ (for $P_{\text{goal}}=0.8,0.85$) steps or $5\cdot 10^{5}$ (for $P_{\text{goal}}=0.9$) training steps. The results are shown in Figure 2 (b) and 12.

Figure: Figure 12: Comparison of different algorithms. The plots show the return in dependence of the time for six different algorithms: TQC, SAC, TD3, PPO, DDPG and A2C. Panel (a) shows this for $P_{\text{goal}}=0.8$ and Panel (b) for $P_{\text{goal}}=0.9$.
Refer to caption: x12.png

We can see that in the first $10^{5}$ steps, A2C and PPO always perform worst, and DDPG is next in line. However in Figure 12 Panel (b), we can see that for $P_{\text{goal}}=0.9$ PPO catches up to DDPG around training step $1.5\cdot 10^{5}$. SAC, TQC, and TD3 perform much better than these three. TD3 is nearly always worse than SAC. TQC always has a drop in the middle region but catches up to SAC in the end. Overall, SAC seems to be the best algorithm for this task when used on the virtual testbed. In contrast to that, in our physical experiments, TQC slightly outperforms SAC for $P_{\text{goal}}=0.85$.

## Appendix D Other experimental runs in the optics lab

### D.1 Different goal powers

We run experiments with $P_{\text{goal}}=0.85,0.86,0.87,0.88,0.9$ using TQC. The return is shown in Figure 13 (a). We can see that, just like in the virtual testbed, the training needs longer to converge the higher the goal. It is interesting that there is a big gap between the agents with $P_{\text{goal}}=0.87$ and $P_{\text{goal}}=0.88$. Please note that we performed these training runs (except for $P_{\text{goal}}=0.85$ only once and draw our conclusions from there.

Figure: Figure 13: Both panels show the return plotted against the training step. Panel (a) shows the training from the start for different goal powers. In Panel (b), $P_{\text{goal}}$ is raised in steps at each black vertical line from $0.85$ over $0.875$ and $0.89$ to $0.9$. For training one of the models (yellow), we delete the replay buffer after the first two black lines; for the other (green), we do not.
Refer to caption: x13.png

### D.2 Replay buffer

We already discussed in the main paper that it can make sense to pre-train agents on lower goals. However, we did not discuss what we do with the replay buffer when changing the goal power. Here, we test if it would be better to keep or delete it when changing to the next higher goal power. We perform two training runs on the experiment starting with $P_{\text{goal}}=0.85$, raising it to $P_{\text{goal}}=0.875$ after $3.8\cdot 10^{4}$ training steps, then raising it to $P_{\text{goal}}=0.89$ after approximately $6.35\cdot 10^{4}$ training steps, and then raising it to $P_{\text{goal}}=0.9$ after $9.8\cdot 10^{4}$ training steps. In the first, we delete the replay buffer after changing our goal to $P_{\text{goal}}=0.875$ and $P_{\text{goal}}=0.89$ (yellow, discussed in main paper); in the second, we do not (green). Both runs are shown in Figure 13 (b). In the yellow ones we see more drops after each rise in goal power, but its return is slightly higher in the end. Overall, the results are quite similar.

### D.3 Pre-training on virtual testbed

Furthermore, we want to know if pretraining on the virtual testbed helps with the experiment’s training times.Fig 14 shows the results next to the agent pre-trained on the experiment with lower goal powers. Panel (a) shows the return plotted against training steps. Overall, the agent pre-trained on the virtual testbed is more stable but not significantly better or faster in training. We can not see an advantage in taking the time to build a virtual testbed for the purpose of pre-training here. Panel (b) shows test results (time needed to fiber couple to $P_{\text{goal}}=0.9$) for both agents and a third one that is only trained on the virtual testbed. There is no significant difference for both agents trained in the experiment (however they were pre-trained). The agent only trained in the virtual testbed is significantly slower. However, it is not as slow that it could not be useful: If the time for coupling is not relevant, it might be enough to learn on the very simple virtual testbed.

Figure: Figure 14: Pre-training on virtual testbed. Panel (a) shows the return plotted against time for two agents: one is trained directly on the experiment with successively higher goal powers (blue), and the other is already pre-trained for $5\cdot 10^{5}$ training steps on the virtual testbed. Panel (b) shows how long both agents and a third (an agent only trained on the virtual testbed) need to couple to $P_{\text{goal}}=0.9$ on the experiment.
Refer to caption: x14.png

## Appendix E Algorithm Hyperparameters

We use the default hyperparameters in StableBaselines3 (incl. contrib), Version 2.3.0 [] or the way they appear in their tutorials. For completeness, we list them here and print the ones that are not default but used in the tutorial in bold.

- TQC
learning rate: 0.0003, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, no added action noise, update target network every 1 step, number of quantiles to drop per net: 2, number of critics networks: 2, number of quantiles for critic: 25
- SAC
learning rate: 0.0003, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, no added action noise, update target network every 1 step,
- TD3
learning rate: 0.001, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, action noise: NormalActionNoise(mean=np.zeros(number actions), sigma=0.1 $\times$ np.ones(number actions), policy and target network updated every 2 steps, standard deviation of smoothing noise on target policy: 0.2, clip absolute value of target policy smoothing noise at: 0.5
- DDPG
learning rate: 0.001, replay buffer size: 1000000, learning starts after 100 steps, batch size: 256, soft update coefficient: 0.005, discount factor: 0.99, update model every step, do 1 gradient step after each rollout, action noise: NormalActionNoise(mean=np.zeros(number actions), sigma=0.1 $\times$ np.ones(number actions)
- PPO
learningrate: 0.0003, number of steps between updates: 2048, batch size: 64, number of epochs when optimizing surrogate loss: 10, discount factor: 0.99, factor for trade-off between bias vs. variance for GAE: 0.95, clip range: 0.2, normalize advantage, entropy coefficient: 0.0, value function coefficient for loss calculation: 0.5, maximum norm for gradient clipping: 0.5
- A2C
learning rate: 0.0007, number of steps between updates: 5, discount factor: 0.99, factor for trade-off between bias vs. variance for GAE: 1.0, entropy coefficient: 0.0, value function coefficient for loss calculation: 0.5, maximum norm for gradient clipping: 0.5, RMSProp epsilon: 1e-05, use RMSprop

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Regarding Figure 2, my understanding is that the reward function differs for different goal power levels. In this case, plotting episode returns in the same figure is not the most appropriate. Is there a way to define a normalized or common metric?
- Prior work in RL has shown that a few human demonstrations can help speed up RL, even overcoming the sparse reward problem. I wonder if using a few human demos can accelerate learning and maybe even remove the need for the dense reward function.
soundness: 4
strengths: - The authors have rigorously designed the experiments, performing method selection in simulation and experimentation on hardware.
- The presentation is very clear. The paper covers the problem of interest, potential challenges, design choices, experiment designs, and results in great detail.
- The main experiment results are convincing: the RL agent can overcome action stochasticity and perform from the limited observations to solve the optical fiber coupling problem above the human level.
summary: This paper presents an application of deep reinforcement learning algorithms to a physical task of optical fiber coupling. As designed by the authors, the RL agent controls two motorized mirrors, each with two degrees of freedom. The task is to move the mirrors to align the laser beam for the highest output power.

The control problem emphasizes practicality. Concretely, the authors employ a POMDP setting where the observations only consist of fields related to the measurable power output and historical actions. This design choice ensures the policy wouldn't rely on mirror position readings (subject to drift) and other hard-to-measure metrics. As a result, a policy is learnable directly on real hardware. The paper also details the experiment design, including the reward function, resetting scheme, and curriculum learning.

Experiment results show that the proposed RL framework derives a control policy that performs the optical fiber coupling task faster than a human expert.
weaknesses: - It is nice to see RL applied to a physical control system and have utilities for research in another domain. However, the control problem solved in the work is relatively simple, especially when the robotics community has trained RL policies on higher degrees of freedom systems, even from image observations.
- I sense there's not much representation-learning in this project. I think the insights will be more relevant if the authors can scale the experiment up to higher-dimensional observations. For example, is there a way to have a visual observation of laser paths as input to the policy?

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: Could transfer learning from pre-trained models or virtual simulations speed up the training process?

How resilient is the RL agent to modifications in the optical configuration, such as slight misalignments or shifts in the location of the optical fiber?

Could the method be applied to tasks with more complicated noise distributions or higher dimensional control requirements?

Given the lengthy training procedures, is the agent's performance sustainable in terms of wear on mechanical components?
soundness: 4
strengths: Originality: In optical systems, where simulation is difficult due to noise and system complexity, this research uses model-free reinforcement learning in an area where traditional models fall short.

The RL agent's ability to attain high coupling efficiencies directly on physical hardware, matching or surpassing human performance, is demonstrated by rigorous experimental results.

Significance: The effective use of RL in this context may encourage the development of comparable methods in other physical and optical disciplines where intricate, noisy situations defy conventional modeling.

Clarity: Detailed experimental methods improve reproducibility, and the report clearly conveys its approach and the justification for algorithm selections.
summary: The coupling of laser light into optical fibers is a difficult task in optics labs because of its high precision and sensitivity requirements. This work investigates the use of model-free reinforcement learning (RL) to automate and optimize this process. The work shows that RL can effectively handle motor imprecision and experimental noise without the need for a comprehensive simulation model. By employing the Truncated Quantile Critics (TQC) and Soft Actor-Critic (SAC) algorithms, the authors get coupling efficiencies above 90%, which is on par with human experts. By demonstrating RL's promise in automated optical alignment tasks, this work opens the door for its application in intricate optical experiments where comprehensive noise modeling or conventional control techniques are impracticable.
weaknesses: Limited Generalization: Although the RL agent performs admirably, it is designed especially for this particular experimental configuration. The significance of the paper would be enhanced by greater generalizability or adaptability to different optical settings.

Training Time: The practical implementation of this strategy in time-sensitive settings may be limited by the time needed to reach high efficiency, which can take up to four days.

Dependency on Particular Hardware: The configuration is dependent on motorized mirrors and particular power measurements, which may restrict its direct application to other labs without comparable apparatus. Investigating different sensing techniques may increase flexibility.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 3
questions: - Why evaluate only TQC in fig 2e and not SAC also?
- What is the interpretation of fig 2d? Why is the number of steps of the last episode important?
soundness: 2
strengths: - good exposition of the problem being solved and the associated challenges
- extensive experiments demonstrating the feasibility of the application
- detailed description of the experimental setup. Without a background in optics experimentation I got the feeling that I could reproduce the experiments to some degree if required to. The paper contains thorough descriptions of design choices regarding the actions pace, processing of observations, formulation of the reward function, reset procedure, etc.
summary: An applied contribution proposing the use of off-policy continuous action space RL algorithms (SAC, TQC, TD3, etc.) for controlling four stepper motors tilting two mirrors in optics experiments. The goal is to automate the alignment of a laser beam for coupling to an optical fiber.

In this setup the main algorithms tested appear to perform more or less the same, with the only difference being made by the value of the target power.
weaknesses: The only reserve I have is **whether ICLR is a good venue for this submission**. This contribution demonstrates that off-the-shelf RL algorithms (literally StableBaseline implementations) can be used for automating parts of the calibration and setup of optics experiments in the presence of noisy actuators.

This in itself is not surprising nor novel. DRL algorithms have been shown to perform control in real-life experiments with noisy actuators, partial observability and, in addition to this work, noisy observations, multiple times and even for higher-dimensional problems. In its current form the paper presents no findings that can be of general interest for the broader RL community.

It would probably be an excellent contribution to a more applied venue.

**Minor:**
- figures presenting multiple algorithms and power goals in the same pane can be confusing / difficult to follow (eg.: fig 2a-b).

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. I am aware of the experiments on pretraining the policy on a virtual environment and transferring it to the real setup. But that may be too challenging because of the gap between simulation and reality. However, what is the issue with pretraining the policy on expert demonstrations (from humans, or classic controllers) from the actual setup and fine tuning online on the actual setup? This can be a very reasonable approach if deploying on the actual system is expensive. 
2. How does the performance of imitation learning on expert data compare to RL?
3. Are noisy actions anything beyond stochasticity in the transition dynamics? What was done specifically regarding noisy actions?
4. What do shaded regions in the figures represent? 
5. How many seeds have been used in the experiments?
soundness: 1
strengths: 1. The paper presents a real-world deployment of reinforcement learning, which I find valuable given the challenges of implementing RL on hardware.

2. The experimental setup is thoroughly detailed, offering insights for practitioners and researchers looking to deploy RL in a similar application.
summary: The paper discusses using RL for automated control in optical experiments, specifically for coupling laser light into optical fibers. The authors demonstrate that an RL agent, trained directly on the experiment without pre-training, can achieve coupling efficiencies comparable to human experts. The authors use standard SAC and TQC.
weaknesses: 1. My main concern is that this paper reads more like a technical report on applying standard RL algorithms to a specific problem. The application itself does not appear to be novel, and given the lack of additional contributions beyond using off-the-shelf RL techniques, I believe this paper is not relevant to ICLR. I would recommend the authors consider submitting to venues focused on optics, sensors, or robotics.

2. The authors highlight challenges in applying RL to real-world problems due to poor sample efficiency and argue that their environment is particularly difficult due to partial observability and noisy actions. These are valid challenges that could inspire algorithmic innovations to tackle them. However, the authors simply use existing model-free RL algorithms without any modifications. If sample efficiency is a concern, model-based methods would be more appropriate, or the authors can consider pretraining the policy with expert data. Similarly, if partial observability is an issue, methods designed for POMDPs or architectures with memory should be considered. Lastly, the impact of noisy actions is not clear to me, even though it is in the tilte of the paper. In my opinion, noisy actions is nothing but stochasticity in the transition dynamics, and RL algorithms are by definition applicable to stochastic transition dynamic. Given the fact that standard RL algorithms have worked in this case, the authors should either downplay these challenges or address them with appropriate methods and more thorough investigations.

3. I identified some inaccurate statements in the paper:
* L43: “Only very few [RL applications] were done in real-world environments.” This is misleading as there are hundreds of papers on RL applied to real-world robotics, beyond simulations. Additionally, RL has been applied to complex real-world tasks like fusion reactor control [1], balloon navigation [2], and data center cooling [3].
* L474: “Our experiments are the first steps towards RL in optics.” While I am not an optics expert, several papers have already applied RL in this field, some of which are cited by the authors, though others are missing [4, 5, 6]. The primary distinction in this work seems to be the direct training of the RL agent on hardware, which, albeit challenging, is not considered a novel contribution. In fact, as stated in Point 2 above, perhaps offline pretraining of the policy on expert data would be an even more suitable direction.

4. The main body of the paper includes many experimental details that are unnecessary for readers unless they intend to replicate the exact setup. This makes the paper harder to follow and obscures key information. I suggest moving these details to the appendix. Additionally, the paper contains repetitive statements, with the same points being reiterated in slightly different ways (e.g., strengths of RL algorithms, challenges in applying RL, etc.) without adding new insights. Reducing these repetitions would improve clarity.

[1] Tracey, Brendan D., et al. "Towards practical reinforcement learning for tokamak magnetic control." Fusion Engineering and Design 200 (2024).

[2] Bellemare, Marc G., et al. "Autonomous navigation of stratospheric balloons using reinforcement learning." Nature 588.7836 (2020).

[3] Lazic, Nevena, et al. "Data center cooling using model-predictive control." Advances in Neural Information Processing Systems 31 (2018).

[4] Pou, B., et al. "Adaptive optics control with multi-agent model-free reinforcement learning." Optics express 30.2 (2022)

[5] Nousiainen, Jalo, et al. "Laboratory experiments of model-based reinforcement learning for adaptive optics control." Journal of Astronomical Telescopes, Instruments, and Systems 10.1 (2024).

[6] Mareev, Evgenii, et al. "Self-Adjusting Optical Systems Based on Reinforcement Learning." Photonics. Vol. 10. No. 10. MDPI, 2023.

Meta-review: This paper proposes an application of model-free RL, using soft actor-critic and truncated quantile critics, to automate laser alignment tasks in optical systems. This is primarily an application paper on a novel domain, at least for the ICLR community. They train their agent on real hardware from scratch and compare performance to human experts in coupling laser light into optical fibers. 

It is commendable and impressive to see a custom hardware implementation built from the ground up and applied to reinforcement learning. This problem is particularly challenging because of partial observability and imprecision in the hardware system. These challenges are their primary motivation for using an RL setup. The application area is certainly novel for the ICLR community and the experimental rigor is detailed. The trained agent also handles noisy actions and motor imprecisions and should have high relevance to optics researchers.

However, one of the key concerns was limited novelty for the ML community. The paper does not propose a new algorithm or learning paradigm, which is fine for an empirical paper but the setup is not shown to generalize which is typical for application papers. This is harder to do due to hardware limitations and therefore inherently more time consuming. Perhaps the bigger issue is lack of direct relevance to ICLR. I personally think this is a very interesting application and we should be building world models with model-free and model-based methods for novel domains. However, the bar for such an application paper should be to demonstrate strong baselines with clear metrics, or show generalization to perturbations / different setups, or solve a previously unsolved applied problem. So although this experimental work can and will have implications for optics researcher, it lacks broader contribution to the ML community. So I encourage the authors to continue this work and address the issues raised during the review process.

Decision: Reject