Model-free reinforcement learning with noisy actions for automated experimental control in optics

Authors: Lea Richtmann, Viktoria-S. Schmiesing, Dennis Wilken, Jan Heine, Aaron D Tranter, Avishek Anand, Tobias J. Osborne, Michèle Heurs
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: reinforcement learning, model-free, experimental control, optics, fiber coupling

Abstract:
Setting up and controlling optical systems is often a challenging and tedious task. The high number of degrees of freedom to control mirrors, lenses or phases makes automatic control challenging, especially when the complexity of the system cannot be adequately modeled due to noise or non-linearities. Here, we show that reinforcement learning (RL) can overcome these challenges when coupling laser light into an optical fiber, using a model-free RL approach that trains directly on the experiment without pre-training. By utilizing the sample-efficient algorithms Soft Actor-Critic (SAC) or Truncated Quantile Critics (TQC), our agent learns to couple with 90% efficiency, comparable to the human expert. We demonstrate that direct training on an experiment can replace extensive system modeling. Our result exemplifies RL's potential to tackle problems in optics, paving the way for more complex applications where full noise modeling is not feasible.

Decision: reject
Reviewer scores: 5, 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1729383843960

thread_type:
official_review

<thread_object>
creator: reviewer_axyw

<content>
summary:
The paper discusses using RL for automated control in optical experiments, specifically for coupling laser light into optical fibers. The authors demonstrate that an RL agent, trained directly on the experiment without pre-training, can achieve coupling efficiencies comparable to human experts. The authors use standard SAC and TQC.

strengths:
1. The paper presents a real-world deployment of reinforcement learning, which I find valuable given the challenges of implementing RL on hardware.

2. The experimental setup is thoroughly detailed, offering insights for practitioners and researchers looking to deploy RL in a similar application.

weaknesses:
1. My main concern is that this paper reads more like a technical report on applying standard RL algorithms to a specific problem. The application itself does not appear to be novel, and given the lack of additional contributions beyond using off-the-shelf RL techniques, I believe this paper is not relevant to ICLR. I would recommend the authors consider submitting to venues focused on optics, sensors, or robotics.

2. The authors highlight challenges in applying RL to real-world problems due to poor sample efficiency and argue that their environment is particularly difficult due to partial observability and noisy actions. These are valid challenges that could inspire algorithmic innovations to tackle them. However, the authors simply use existing model-free RL algorithms without any modifications. If sample efficiency is a concern, model-based methods would be more appropriate, or the authors can consider pretraining the policy with expert data. Similarly, if partial observability is an issue, methods designed for POMDPs or architectures with memory should be considered. Lastly, the impact of noisy actions is not clear to me, even though it is in the tilte of the paper. In my opinion, noisy actions is nothing but stochasticity in the transition dynamics, and RL algorithms are by definition applicable to stochastic transition dynamic. Given the fact that standard RL algorithms have worked in this case, the authors should either downplay these challenges or address them with appropriate methods and more thorough investigations.

3. I identified some inaccurate statements in the paper:
* L43: “Only very few [RL applications] were done in real-world environments.” This is misleading as there are hundreds of papers on RL applied to real-world robotics, beyond simulations. Additionally, RL has been applied to complex real-world tasks like fusion reactor control [1], balloon navigation [2], and data center cooling [3].
* L474: “Our experiments are the first steps towards RL in optics.” While I am not an optics expert, several papers have already applied RL in this field, some of which are cited by the authors, though others are missing [4, 5, 6]. The primary distinction in this work seems to be the direct training of the RL agent on hardware, which, albeit challenging, is not considered a novel contribution. In fact, as stated in Point 2 above, perhaps offline pretraining of the policy on expert data would be an even more suitable direction.

4. The main body of the paper includes many experimental details that are unnecessary for readers unless they intend to replicate the exact setup. This makes the paper harder to follow and obscures key information. I suggest moving these details to the appendix. Additionally, the paper contains repetitive statements, with the same points being reiterated in slightly different ways (e.g., strengths of RL algorithms, challenges in applying RL, etc.) without adding new insights. Reducing these repetitions would improve clarity.

[1] Tracey, Brendan D., et al. "Towards practical reinforcement learning for tokamak magnetic control." Fusion Engineering and Design 200 (2024).

[2] Bellemare, Marc G., et al. "Autonomous navigation of stratospheric balloons using reinforcement learning." Nature 588.7836 (2020).

[3] Lazic, Nevena, et al. "Data center cooling using model-predictive control." Advances in Neural Information Processing Systems 31 (2018).

[4] Pou, B., et al. "Adaptive optics control with multi-agent model-free reinforcement learning." Optics express 30.2 (2022)

[5] Nousiainen, Jalo, et al. "Laboratory experiments of model-based reinforcement learning for adaptive optics control." Journal of Astronomical Telescopes, Instruments, and Systems 10.1 (2024).

[6] Mareev, Evgenii, et al. "Self-Adjusting Optical Systems Based on Reinforcement Learning." Photonics. Vol. 10. No. 10. MDPI, 2023.

questions:
1. I am aware of the experiments on pretraining the policy on a virtual environment and transferring it to the real setup. But that may be too challenging because of the gap between simulation and reality. However, what is the issue with pretraining the policy on expert demonstrations (from humans, or classic controllers) from the actual setup and fine tuning online on the actual setup? This can be a very reasonable approach if deploying on the actual system is expensive. 
2. How does the performance of imitation learning on expert data compare to RL?
3. Are noisy actions anything beyond stochasticity in the transition dynamics? What was done specifically regarding noisy actions?
4. What do shaded regions in the figures represent? 
5. How many seeds have been used in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730575347485

thread_type:
official_review

<thread_object>
creator: reviewer_nh2k

<content>
summary:
An applied contribution proposing the use of off-policy continuous action space RL algorithms (SAC, TQC, TD3, etc.) for controlling four stepper motors tilting two mirrors in optics experiments. The goal is to automate the alignment of a laser beam for coupling to an optical fiber.

In this setup the main algorithms tested appear to perform more or less the same, with the only difference being made by the value of the target power.

strengths:
- good exposition of the problem being solved and the associated challenges
- extensive experiments demonstrating the feasibility of the application
- detailed description of the experimental setup. Without a background in optics experimentation I got the feeling that I could reproduce the experiments to some degree if required to. The paper contains thorough descriptions of design choices regarding the actions pace, processing of observations, formulation of the reward function, reset procedure, etc.

weaknesses:
The only reserve I have is **whether ICLR is a good venue for this submission**. This contribution demonstrates that off-the-shelf RL algorithms (literally StableBaseline implementations) can be used for automating parts of the calibration and setup of optics experiments in the presence of noisy actuators.

This in itself is not surprising nor novel. DRL algorithms have been shown to perform control in real-life experiments with noisy actuators, partial observability and, in addition to this work, noisy observations, multiple times and even for higher-dimensional problems. In its current form the paper presents no findings that can be of general interest for the broader RL community.

It would probably be an excellent contribution to a more applied venue.

**Minor:**
- figures presenting multiple algorithms and power goals in the same pane can be confusing / difficult to follow (eg.: fig 2a-b).

questions:
- Why evaluate only TQC in fig 2e and not SAC also?
- What is the interpretation of fig 2d? Why is the number of steps of the last episode important?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731099630258

thread_type:
official_review

<thread_object>
creator: reviewer_obmd

<content>
summary:
The coupling of laser light into optical fibers is a difficult task in optics labs because of its high precision and sensitivity requirements. This work investigates the use of model-free reinforcement learning (RL) to automate and optimize this process. The work shows that RL can effectively handle motor imprecision and experimental noise without the need for a comprehensive simulation model. By employing the Truncated Quantile Critics (TQC) and Soft Actor-Critic (SAC) algorithms, the authors get coupling efficiencies above 90%, which is on par with human experts. By demonstrating RL's promise in automated optical alignment tasks, this work opens the door for its application in intricate optical experiments where comprehensive noise modeling or conventional control techniques are impracticable.

strengths:
Originality: In optical systems, where simulation is difficult due to noise and system complexity, this research uses model-free reinforcement learning in an area where traditional models fall short.

The RL agent's ability to attain high coupling efficiencies directly on physical hardware, matching or surpassing human performance, is demonstrated by rigorous experimental results.

Significance: The effective use of RL in this context may encourage the development of comparable methods in other physical and optical disciplines where intricate, noisy situations defy conventional modeling.

Clarity: Detailed experimental methods improve reproducibility, and the report clearly conveys its approach and the justification for algorithm selections.

weaknesses:
Limited Generalization: Although the RL agent performs admirably, it is designed especially for this particular experimental configuration. The significance of the paper would be enhanced by greater generalizability or adaptability to different optical settings.

Training Time: The practical implementation of this strategy in time-sensitive settings may be limited by the time needed to reach high efficiency, which can take up to four days.

Dependency on Particular Hardware: The configuration is dependent on motorized mirrors and particular power measurements, which may restrict its direct application to other labs without comparable apparatus. Investigating different sensing techniques may increase flexibility.

questions:
Could transfer learning from pre-trained models or virtual simulations speed up the training process?

How resilient is the RL agent to modifications in the optical configuration, such as slight misalignments or shifts in the location of the optical fiber?

Could the method be applied to tasks with more complicated noise distributions or higher dimensional control requirements?

Given the lengthy training procedures, is the agent's performance sustainable in terms of wear on mechanical components?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731135022196

thread_type:
official_review

<thread_object>
creator: reviewer_k3xd

<content>
summary:
This paper presents an application of deep reinforcement learning algorithms to a physical task of optical fiber coupling. As designed by the authors, the RL agent controls two motorized mirrors, each with two degrees of freedom. The task is to move the mirrors to align the laser beam for the highest output power.

The control problem emphasizes practicality. Concretely, the authors employ a POMDP setting where the observations only consist of fields related to the measurable power output and historical actions. This design choice ensures the policy wouldn't rely on mirror position readings (subject to drift) and other hard-to-measure metrics. As a result, a policy is learnable directly on real hardware. The paper also details the experiment design, including the reward function, resetting scheme, and curriculum learning.

Experiment results show that the proposed RL framework derives a control policy that performs the optical fiber coupling task faster than a human expert.

strengths:
- The authors have rigorously designed the experiments, performing method selection in simulation and experimentation on hardware.
- The presentation is very clear. The paper covers the problem of interest, potential challenges, design choices, experiment designs, and results in great detail.
- The main experiment results are convincing: the RL agent can overcome action stochasticity and perform from the limited observations to solve the optical fiber coupling problem above the human level.

weaknesses:
- It is nice to see RL applied to a physical control system and have utilities for research in another domain. However, the control problem solved in the work is relatively simple, especially when the robotics community has trained RL policies on higher degrees of freedom systems, even from image observations.
- I sense there's not much representation-learning in this project. I think the insights will be more relevant if the authors can scale the experiment up to higher-dimensional observations. For example, is there a way to have a visual observation of laser paths as input to the policy?

questions:
- Regarding Figure 2, my understanding is that the reward function differs for different goal power levels. In this case, plotting episode returns in the same figure is not the most appropriate. Is there a way to define a normalized or common metric?
- Prior work in RL has shown that a few human demonstrations can help speed up RL, even overcoming the sparse reward problem. I wonder if using a few human demos can accelerate learning and maybe even remove the need for the dense reward function.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732790812024

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Changes to the paper

comment:
Thank you for your comments. We made the following changes throughout the paper based on your reviews. We also performed additional experiments in the virtual testbed studying the impact of different noise levels, which can be found in the newly added Appendix C.8.

Logs of the changes:

All Figures: The information that the shaded areas are 2 $\sigma$ error bands/bars was included in the caption of the figures instead of the text (formerly L366-369). (Reviewer AXyW, Q4)

All Figures: normalized return throughout the paper by dividing the return by its maximum possible value for each goal power to make it comparable across goal powers (Reviewer K3XD, Q1; Reviewer nH2K W4)

L43-44: “Only very few [RL applications] were done in real-world environments.” → “Comparatively few experiments were done in real-world environments.” (Reviewer AXyW, W3)

L58: added that noisy actions are a special type of stochasticity (Reviewer AXyW, Q3, W2)

L85:  added more citations for adaptive optics applications (Reviewer AXyW, W3)

L99-105: added citation and explanation of X-ray experiment and comparison to our work (Reviewer AXyW, W3)

L146-147: Changed the sentence formerly in L144 to make clear that two mirrors each tiltable in two axes are sufficient for any beam alignment, added citation (Reviewer oBMD, W1)

L220-222: added that noisy actions are a special type of stochasticity, and refer to Appendix C.8 for more information on the influence on the learning behaviour  (Reviewer AXyW, Q3, W2)

L313-316: added explanation of normalized return (Reviewer K3XD, Q1; Reviewer nH2K W4)

L373: added to explanation of Figure 2 (d) that the episode after the last reset is the succesful episode (Reviewer nH2K, Q2)

L458-460: added the following into the description of using the agent for correction of slight misalignment: A misalignment at the hand steering mirrors is equivalent to a slight shift in the location of the fiber collimator (or any other element not accessible to the agent) (Reviewer oBMD, Q2)

L495-496: added that future work should include investigations on whether pretraining on expert demonstrations could speed up the learning process (Reviewer oBMD, Q1; Reviewer K3XD, Q2)

L511-515: added additional last paragraph to Summary and Outlook to clarify the aim of the paper

L930-932: details on graphics card and used packaged moved from L364 to Appendix A (Reviewer AXyW, W4)

L1382-1397, L1440-1457: added Appendix C.8  with experiments for different levels of noise to explain its influence on the agent (Reviewer AXyW, Q3, W2, Reviewer oBMD, Q3)

The supplementary material was updated accordingly (noise experiments, normalized return).
</content>
</thread_object>

</thread>
<thread>
creation date:
1733232314632

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Clarifications

comment:
We want to thank all reviewers again for their respectful feedback. In this general response, we would like to clarify two aspects:

We were asked several times to provide additional experimental validation. Conducting experiments in an optics lab can be very time-consuming and financially costly. It is, therefore, not feasible to quickly conduct more experiments (please note that also the email about the extension of the rebuttal period stated that no additional significant experiments should be requested).

The use of RL is still very novel in the laser optics community, with very few examples of successful applications to actual experiments in the lab. However, we are sure that RL provides great potential for this community but is often not easily accessible. The main aim of our work is to show that common optics tasks can be solved and streamlined using existing RL techniques, avoiding the need to dive deep into algorithm development or build sophisticated simulations. Several reviewer’s comments point out ways to possibly improve/optimize our approach and we fully agree (as written in our Summary and Outlook section). However, as the application of RL in this domain is novel, we aimed to demonstrate that it is possible to use RL for a common optics task rather than showing the best and most general way.
 
Thanks again for your time.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an application of model-free RL, using soft actor-critic and truncated quantile critics, to automate laser alignment tasks in optical systems. This is primarily an application paper on a novel domain, at least for the ICLR community. They train their agent on real hardware from scratch and compare performance to human experts in coupling laser light into optical fibers. 

It is commendable and impressive to see a custom hardware implementation built from the ground up and applied to reinforcement learning. This problem is particularly challenging because of partial observability and imprecision in the hardware system. These challenges are their primary motivation for using an RL setup. The application area is certainly novel for the ICLR community and the experimental rigor is detailed. The trained agent also handles noisy actions and motor imprecisions and should have high relevance to optics researchers.

However, one of the key concerns was limited novelty for the ML community. The paper does not propose a new algorithm or learning paradigm, which is fine for an empirical paper but the setup is not shown to generalize which is typical for application papers. This is harder to do due to hardware limitations and therefore inherently more time consuming. Perhaps the bigger issue is lack of direct relevance to ICLR. I personally think this is a very interesting application and we should be building world models with model-free and model-based methods for novel domains. However, the bar for such an application paper should be to demonstrate strong baselines with clear metrics, or show generalization to perturbations / different setups, or solve a previously unsolved applied problem. So although this experimental work can and will have implications for optics researcher, it lacks broader contribution to the ML community. So I encourage the authors to continue this work and address the issues raised during the review process.