An Optical Control Environment for  Benchmarking Reinforcement Learning Algorithms
Authors: ABULIKEMU ABUDUWEILI
Changliu Liu
Venue: Accepted by TMLR ()

Abstract: Deep reinforcement learning has the potential to address various scientific problems. In this paper, we implement an optics simulation environment for reinforcement learning based controllers. The environment captures the essence of nonconvexity, nonlinearity, and time-dependent noise inherent in optical systems, offering a more realistic setting. 
Subsequently, we provide the benchmark results of several reinforcement learning algorithms on the proposed simulation environment. The experimental findings demonstrate the superiority of off-policy reinforcement learning approaches over traditional control algorithms in navigating the intricacies of complex optical control environments.

Published in Transactions on Machine Learning Research (10/2023)
An Optical Control Environment for Benchmarking Reinforcement Learning Algorithms
Abulikemu Abuduweili
abulkken
Robotics Institute, Cornegie Mellon University
Changliu Liu
@andrea cmaa. edu
Robotics Institute, Carnegic Mellon University
Reviewed on OpenReview: Attps: // openreview. net/ formno id= 61TK2U9B96
Abstract
Deep reinforcement learning has the potential to address varions scientific problems. In this paper, we implement an optics simulation environment for reinforcement learning based controllers. The environment captures the essence of nonconvexity, nonlinearity, and time- dependent noise inherent in optical systems, offering more realistic setting. Subsequently, provide the benchmark results of several reinforcement learning algorithms on the pro- posed simulation environment. The experimental findings demonstrate the superiority of off-policy reinforcement learning approaches over traditional control algorithms in navigat- ing the intricacies of complex optical control environments.
1 Introduction
In recent years, deep reinforcement learning (RL) has been used to solve challenging problems in various fields (Sutton & Barto 2018), including self-driving car (Bansal et al. 2018 and robot control (Zhang et al. 2015). Among all applications, deep RL made significant progress playing games on superhuman level (Mnih et al. 2013 Silver et al. 2014 2016 Vinyals al., 2017). Beyond playing games, deep RL has the potential to strongly impact the traditional control and automation tasks in the natural science, such control problems in chemistry (Dressler el al. 2018), biology (Izawa et al.; 2004), quantum physics (Bukov et al. 2018), optics and photonics (Genty et al. 2020).
In optics and photonics, there are particular potentials for RL methods to drive the next generation of optical laser technologies (Genty et al.] 2020). That is not only because there are increasing demands for adaptive control and automation (of tuning and control) for optical systems (Baumeister al. 2018). but also because many phenomena in optics are nonlinear and multidimensional (Shen 1984), with noisc- sensitive dynamics that are extremely challenging to model using conventional approaches. RL methods are able to control multidimensional environments with nonlinear function approximation (Dai et al. 2018). Thus, exploring RL controllers becomes increasingly promising in optics and photonics as well as in scientific rescarch, medicine, and other industries (Genty et al.) 2020 Fermann & Hart] 2013).
In the field of optics and photonics, Stochastic Parallel Gradient Descent (SPGD) algorithm with a PID
controller has traditionally been employed tackle control problems (Cauwenberghs 1993 Zhou al. 2009; Abuduweili et al. 2020a). These problems typically involve adjusting system parameters, such the delay line of mirrors, with the objective of maximizing reward, such optical pulse energy. SPGD is a specific case of the stochastic error descent method (Cauwenberghs) 1993 Dembo & Kailath 1990). which operates based on a model-free distributed learning mechanism. The algorithm updales the parameters by perturbing each individual parameter vector, resulting in a decrease in error or an increase in reward. However, the applicability of SPGD is limited convex or near-convex problems, while many control problems in optics exhibit non-convex characteristics. As a result, SPGD struggles to find the global optimum of an optics control system unless the initial state of the system is in close proximity to the global optimum. Traditionally,

---

Published in Transactions on Machine Learning Research (10/2023)
experts would manually tune the initial state of the optical system, followed by the use of SPGD-PID to control the adjusted system. Nevertheless, acquiring such expert knowledge becomes increasingly challenging as system complexity grows.
To enable efficient control and automation in optical systems, researchers have introduced deep reinforcement
learning (RI.) techniques (Tünnermann & Shirakawa 2019: Sun et al., 2020; Abuduweili et al. 2020b 2021)
Previous studies predominantly focused on implementing Deep Q-Network (DQN) (Mnih et al. 2013 and
Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015 in simple optical control systems, aiming
to achieve comparable performance to traditional SPGD-PID controllers (Tinnermann & Shirakawa 2019;
Valensisc et al., 2021). However, there is a lack of research evaluating a broader range of RL algorithms in more complex optical control environments. The exploration and evaluation of RL algorithms in real- world optical systems pose significant challenges due to the high cost and the need for experienced experts to implement multiple optical systems with various configurations. Even for a simple optical system, substantial efforts and resources are required to instrument and implement RL, algorithms effectively.
Simulation has been widely utilized in the fields of robotics and autonomous driving since the early stages of research (Ponerleau 1998 Bellemare et al. 2013). As the interest and application of tarring-based robotics continue to grow, the role of simulation becomes increasingly crucial in driving research advancements. We believe that simulation holds equal importance in evaluating RT. algorithms for optical control. However, to the best of our knowledge, there is currently no open-source RL environment available for optical control simulation.
In this paper, we present OPS (Optical Pulse Stacking), an open and scalable simulator designed for con- trolling typical optical systems. The underlying physics of OPS aligns with various optical applications, including coherent optical inference (Wetzstein et al. (020) and linear optical sampling (Dorrer et al. 2003 which find applications in precise measurement, industrial manufacturing, and scientific research. A typi- cal optical pulse stacking system involves the direct and symmetrical stacking of input pulses to multiply their energy, resulting in stacked output pulses (Tünnermann & Shirakawa| 2017 Stark et al.) 2017 As- trauskas et al. 2017: Vang et al. 2020). By introducing the OPS optical control simulation environment, our objective is to encourage exploration of RL applications in optical control tasks and further investigate RL controllers in natural sciences. We utilize OPS to evaluate several important RL algorithms, including Twin Delayed Deep Deterministic Policy Gradient (TD3) (Fujimolo et al. 2018). Soft Actor- (SAC) [Haarnoja et al., 2018), and Proximal Policy Optimization (PPO) (Schulman et al., 2017). Our findings indicate that in a simple optical control environment (nearly convex), the traditional SPGD-PID controller performs admirably. However, in complex environments (non-convex optimization). SPGD-PID falls short, and RI-trained policies outperform SPGD-PID. Following the reporting of these RT, algorithm results, we discuss the potential and challenges associated with RL algorithms in real-world optical systems. By provid- ing the OPS simulation environment and conducting RL algorithm experiments, we aim to facilitate rescarch on RL applications in optics, benefiting both the machine learning and optics communities. The code of the paper is available at https: h httpss///gttab..ccm/lllleeee www.eeeeeeesseeeeeoeeeeeeeeeeeee cement-eeeeeeeeeee Ptw.eeetttteeerroooooooeeeeetaaa
2
Simulation environment
2.1 Physics of the simulation
The optical pulse stacking (OPS), also known as pulse combination, system employs a recursive approach to stack optical pulses in the time domain. The dynamics of the OPS are similar to the recurrent neural networks (RNN) or Wavenet architecture (Ord et al., 2016). We illustrate the dynamics of the OPS in RNN style as shown in Fig. 1 Tn the OPS system, the input consists of periodic pulse train Q with repetition period of T. Assuming the basic function of the first pulse at time step t is denoted as E E(t) (a complex function), the subsequent pulses can be described as E, = TT), Eg = E(t + 2T), and so on. The OPS system recursively imposes time delays to earlier pulses in consecutive pairs. For instance, in the first stage of OPS, a time-delay controller imposes the delay a. on pulse 1 to allow it to combine (overlap)
The periodic pulse train typically emitted by lasers, where each laser pulse's wave function is nearly identical except for the time delay.
2

---

Published in Transactions on Machine Learning Research (10/2023)
Bou
3rd stage stacking
Ej E1234
B5
2 stage stacking
E12
T2
E3,4
72
Egg
1 stage stacking
E
Eg
Eg
E
Eg
E
E
Eg
Figure 1: Mustration of the principle of optical pulse stacking. Only 3-stage pulse stacking was plotted for simplicity.
with pulse 2. With the appropriate time delay, pulse can be stacked with the next pulse, E2, resulting in the stacked pulses F,, Ft( +(L+m) + F(t + TT). Similarly, pulse can be stacked with pulse 4, creating 3a, Elt + 2T + TI) + E(t 37),, and so forth. In the second stage of OPS, an additional time delay, is imposed on E, to allow it to stack with Es,4, resulting in E1,23,4. This stacking process continues in each subsequent stage of the OPS controller, multiplying the pulse energy by a factor of 2 by stacking 2 pulses, where N time delays (TI, T2, .., TN) are required for control and stabilization. Additional details about the optical pulse stacking are shown in appendix A.]
0.8
combinad putsa energy
0.8
combined puice
0.6
0.4
0,2
2nd stage time delay
0.2
1st Stage time de
V
20
20
time delly
(a) Oue stage OPS (1-d)
(b) Two stage OPS (2-d)
Figure 2: Function of the (a) 1-stage OPS: pulse energy (I w.r.t. delay line Ti- (b) 2-stage OPS: pulse
energy Pa(Ti, 72) w.r.l. delay lines (21,
2.2 Control objective and noise
The objective of controlling an OPS system is to maximize the energy the final stacked (output) pulse by adjusting the time delays. We denote the vector of time delays as = [Ti, ... Ty), and E (t; r) represents the final stacked pulse under the time delay configuration т. For N-stage OPS system, the energy of the final stacked pulse, denoted (T(T), is computed the integral of the norm of Font (l: over
3

---

Published in Transactions on Machine Learning Research (10/2023)
time. In mathematical terms, we express it as (T (t; ddtt. To formulate the objective function for controlling an N-stage OPS system, we aim to find the optimal set of time delays r' that maximizes (T(T). The objective function can be defined as follows:
arg max (T( = arg max TN (N T2, TN)
(1)
When ignoring noise, the objective function of the final pulse energy, PN, with respect to the time delays,
can be derived based on optical coherence. Figure 2(a) depicts the pulse energy function (T in a 1-stage
OPS system, showing the relationship between pulse energy and the time delay тΙ- Similarly, Fig. 2(b) displays the function surface of P2(TI, 72) in a 2-stage OPS system, illustrating how pulse energy varies with the first and second stage time delays (TI, T2). As evident from the figures, the control objective of the OPS system is nonlinear and non-convex, even when noise is disregarded. This inherent complexity arises due to factors such as optical periodicity and the nonlinearily of coherent interference. Consequently, achieving the global optimum or better local optima becomes challenging task for any control algorithms, particularly when starting from a random initial state.
However, in practical scenarios, noise cannot be ignored, and the OPS system is highly sensitive to noise. This sensitivity primarily due to the pulse wavelength being on the order of micrometers (1um 10-6m). Environmental noise, including vibrations of optical devices and atmospherie temperature drift, can easily cause shifts in the time delays, resulting in changes to the output pulses. As result, the objective function in real-world applications is much more complex than what is depicted in Fig. 2 especially for higher- stage OPS systems with higher dimensions. Consequently, achieving the control objective becomes even more challenging in the presence of unknown initial states and unpredictable noise in such noise-sensitive complex systems (Genty et al.) 2020). Therefore, model-based controllers face significant difficulties in implementation. In this paper, we primarily focus on model-free reinforcement learning approaches to address these challenges.
In this simulation, incorporate two types of noise: fast noise arising from device vibrations and slow noise caused by temperature drift. The fast noise is modeled as a zero-mean Gaussian random noise with variance σ², following the simulation noise approach outlined in (Tünnermann & Shirakawa 2019). On the other hand, the slow noise f4 accounts for temperature drift. and is represented as a. piecewise iss linear function [Ivanova et al. 2021). To capture the combined effect of these two noise sources, we define the overall noise e as a random process. Specifically, we can express it as follows:
E [ Mr, VAR[e]] o²
(2)
2.3 Reinforcement learning environment
Interactions with RL agent. An RL agent interacts with the OPS environment in discrete time steps, as shown in Fig. 3 At each time step t, the RL agent receives the current state of the OPS environment, denoted as s,. Based on this state, the agent selects action a, to be applied to the environment. The action could involve adjusting the time delays in the OPS system. Once the action is chosen, it is transmitted to the OPS environment, which then processes the action and transitions to new state St-1. The OPS environment provides feedback to the RL agent in the form of a reward r, The reward serves as a measure of how well the OPS system is achieving the objective of maximizing the final stacked pulse energy. The RL agent utilizes the experience tuple (s,, ae 8t41:Te) to learn and update its policy π T(a 8) over time. The goal of the agent is to learn a policy that maximizes the expected cumulative reward over the interaction with the OPS environment.
State space. The state space of the OPS system is a continuous and multidimennioonll vector space. The state value at time step t, denoted as st, corresponds to the pulse amplitude measurement of the final stacked pulse, given by 8t = EEoe T-]- Therefore, 8t provides a time-domain representation of the final stacked pulse, offering direct insight into the control performance. In practical implementations, the pulse amplitude is typically detected using a photo-detector and subsequently converted into digital time-scrics signals. In our simulation, we have incorporated real-time rendering of the pulse amplitude to facilitate the monitoring of the control process.
4

---

Published in Transactions on Machine Learning Research (10/2023)
E,3,,44
OPS Environment
E
E3.4
RL agent
Photo detection
state
reward r
2 stackad pulses
1 stacked pulses
puises
stage time delay
1st stage time delay Ty
action
Figure 3: Illustration of the interaction between RL agent and OPS environment. Only 2-stage OPS was plotted for simplicity.
Action space. The action space of an N-stage OPS environment is a continuous and N-dimensional vector space. At each time step t, the action a₂ corresponds to an additive time delay value Ar(t) for the N-stage OPS environment: at = (t(t)=
(t).. The OPS environment applies the additive time delay value
att) transition to the next state.
Reward. As mentioned in section 2.2 the objective of the OPS controller is maximizing the final stacked pulse energy (T( In our simulation, use the normalized final pulse energy as the reward value. The reward at each time step is defined as:
(T - Pmnaa) )2 (Pmin P mmxx )
(3)
where Pmax is the maximum pulse energy at the global optimum, and Puni is the minimum pulsc energy. The maximum reward 0 achieved when P(r) = Pmaz (peak position of Fig, 2(b)
State transition function. The environmental noise has direct impacts on the delay lines, including the vibration and temperature-indueed shift noise of the delay line devices. Therefore, in the state transition process, the actual applied delay line value Tr + 1) is a combination of the action at and the noise et. Specifically, can be expressed as:
(4) After selecting the pulses, the real-time delay Freal 1) is imposed them using delay line devices, which introduce additional time delays for the pulses. The state transition process is governed by the combination of the current state, the action taken, and the noise present. The specific form of the state transition follows the principles of coherent light interference (Saleh & Teich 2019).
Treat + Treal(!) + a2 + Et-
Let f be an interference observation
function. Then the state transition can be written as:
8++1 [Treal (1 + 1)) (Treaa + + +e) S( (8t) ar ex)
It is important to note that the slow-changing noise term E = μ4 follows a piccewise linear function that changes slowly over time. During episodic training for RI. agents, lr can be treated
[[t]=
constant value within episode. However, the value of de may vary from one episode to the next. In this case, assuming that pe changes very slowly, the OPS control process can be modeled as a Markov decision process (MDP).
Mode
Initial state
Noise
Objective
from optics_ems import OPS_em env env OPS_ ops_een(staag-5, mode- "nedium") env.reset() False while action saa.attioo_spccessmmleee] observation, reward, done, info env.step(actton) env.render()
near the optimum
time-independent:
Easy
dus 0
convex
time independen::
Medium
random
non-convex
time-dependent:
Hard
random
dua 0
non-convex
d
Table 1: Different difficulty modes on OPS.
Figure 4: Example code of the OPS environment.
5

---

Published in Transactions on Machine Learning Research (10/2023)
Control difficulty of the environment. We have implemented the OPS environment support arbitrary stages of pulse stacking (N E 1,..)...). As the number of stages increases, the control task becomes more challenging, In addition to the customizable number of stages, we have also introduced three difficulty modes (casy, medium, and hard) for each stage of OPS, as outlined in table [ The difficulty mode is determined by the initial stale of the system and the distribution of noise. These difficully modes allow for different levels of complexity and challenge in the control task, providing flexibility for evaluating and training control algorithms in various scenarios.
Easy mode. In the easy mode of the OPS environment, the initial state of the system is set to be near the global optimum. This configuration is often encountered in traditional optics control problems where experts fine-tune the initial state to facilitate casier control. As depicted in Fig. 5(a) provide an example of the initial state for the easy mode in a 3-stage OPS environment. The proximity of the initial state to the global optimum allows the control objective in the easy mode to be considered convex.
Medium mode. In the medium mode of the OPS environment, the initial state of the system is randomly determined, illustrated in Fig. 5(b) This random initialization introduces non-convexity into the control problem, making it more challenging to solve. However, in the medium mode, the noise present in the system is dime-independent We model this noise as Gaussian distribution, where e, follows a normal distribution (H,,o). This setting aligns with classical reinforcement lcarning and typical Markov Decision Process (MDP) settings, where the noise distribution remains the same throughout each episode.
Hard mode. In the hard mode of the OPS enviromment, similar to the medium mode, the initial state of the system is randomly determined. However, in contrast to the medium mode, the behavior of the noise in the hard mode is more complex. The mean value of the noise distribution f4 becomes a time-dependent variable that slowly changes over time. This time-dependent noise introduces additional challenges and complexity to the control problem. In this case, the control problem deviates from typical Markov Decision Process (MDP) setting. By incorporating the noise term into the state definition & [8c: ee): effectively convert the control process into Partially Observable Markov Decision Process (POMDP). The hard mode is designed to mimic real-world settings more closely. In practical applications, when deploying the trained model in a testing environment, we often encounter temperature drift, which causes the noise distribution of the testing environment to differ from the training environment. Therefore, the hard mode simulates the realistic scenario where the noise distribution is not stationary and may vary over time, making the control problem more challenging and closer real-world conditions.
1.0
Indenally Inommliee
0.6
0.6
8.6
0.4
3.4
9.4
0.0
-50
Time
(a) Initial state: easy mode
(b) Initial state: medium /hard mode
(c) Target optimal state
Figure 5: Rendering examples of the (a) initial state for easy mode, (b) initial state for medium or hard mode, (c) global optimal target state in 2-stage OPS. In the initial state of the easy mode is closer to the target state. The initial state of medium or hard mode random.
API & sample usage. The simulation in this study is based on the Nonlinear-Optical-MMddin (Hult 2007,, which provides the optical and physical fundamentals for the OPS environment. To facilitate in- legration and compatibility with existing frameworks and tools, the OPS environment is designed to be compatible with the widely used OpenAl Gym API (Brockman et al. 2016). To demonstrate the usage of the OPS environment, we provide an example code snippet Fig.
Features of the OPS environment. We summarize the key features of the OPS environment. as follows:

---

Published in Transactions on Machine Learning Research (10/2023)
Open-source optical control environment: To the best of our knowledge, this is the first open-sourced RL environment, for optical control problems. The use of open-source licenses enables researchers to inspect the underlying code and modify the environment if necessary to test new research ideas.
Scalable and hifficulty-ajjustable scientific environment: Unlike many RL environments that are casy to solve, our OPS environment allows flexible adjustment of difficulty. The dimension of the action space can easily scale with the stage number N. Choosing a larger N with the hard mode makes controlling the environment more challenging. Effective solutions to hard scientific control problems can have a broad impact on various scientific control problems (Genty et al. 2020 Fermann & Hartl 2013).
Realistic noise: In the hard mode of the OPS environment, we model the noise distribution as a time- dependent function. This mirrors a real-world scenario which noise-distribution shift occurs wet the testing and training environments. Such realistic noise modeling particularly relevant for noise- sensitive systems (Ivanoaa et al. 2021) and increases the stochasticity of the environment.
Extendable state and struct ural information: When μt changes very slowly, the OPS control process can be formulated an MDP. In cases where 14 undergoes substantial variations and the noise e varies with time, the inclusion of the noise term within the state definition as á, = [st]es] seamlessly transforms the control process into a POMDP. Furthermore, we can explore the structural information or physical constraints from the function of the OPS (see Fig. 2 and incorporate it with RL controllers.
3 Experiments
3.1
Experimental setup
We present benchmark results for various control algorithms, including a traditional control algorithm and three state-of-thes thearttt--artt-e reinforcement learning algorithms:
SPGD (Stochastic Parallel Gradient Descent) algorithm with PID controller: SPGD is a widely used approach for controlling optical systems (Cauwenberghs 1993 Zhou al., 2009). In SPGD, the objective gradient estimation is achieved by applying random perturbation to the time delay value, denoted ôr. The update formula is r(++1) = r(t) (T(() + (T where is the update step-size. The output of the SPGD algorithm is then sent to a PID controller to control the system. In this work, we refer to the SPGD-PID controller as the SPGD controller.
PPO (Proximal Policy Optimization) is an on-policy reinforcement learning algorithm (Schllman et al, 2017). It efficiently updates its policy within a trust region by penalizing KI. divergence or clipping the objective function.
SAC (Soft Actor-Critic) is off-policy reinforcement learning algorithm (Haarnoja et al., 2018 It learns two Q-functions and utilizes entropy regularization, where the policy is trained to maximize trade-off between expected return and entropy.
TD3 (Twin Delayed Deep Deterministic policy gradient) is an off-policy reinforcement learning algorithm (Fjiimooo et al. 2018). learns two Q-functions and uses the smaller the two Q-values to form the targets in the loss functions. Additionally, TD3 adds noise the target action for exploration.
We used the algorithms implemented stable- baselines-3 (Raffin et al. 2019). The training procedure for an RL agent consists of multiple episodes, and each episode consists of 200 steps. Across each of the experimental configurations, we executed the experiments with 20 different random seeds and present the results accompanied by their respective standard deviations. A detailed explanation of RL algorithms
and their associated hyperparameters is presented in appendices B.1 and B.2
3.2 Results on controlling 5-stage OPS
In this section, we present the results obtained for the 5-stage OPS system, which involves stacking 32 pulses, We evaluate all four algorithms in three difficulty modes: easy, medium, and hard. It is important note that SPGD is a training-free method, as relies on a fixed policy. Therefore, we only evaluate the testing performance of SPGD. For the RL algorithms (PPO, TD3, and SAC), we assess both the training convergence and the testing performance of the trained policy.
7

---

Published in Transactions on Machine Learning Research (10/2023)
The training curves, depicting the reward per step during training iterations, shown in Fig. 6(a) for the easy mode, Fig. 6(b) for the medium mode, and Fig. [[c] for the hard mode. From these plots, we observe that TD3 and SAC exhibit similar performance, which is consistently higher than that of PPO across all three difficulty modes. Notably, in the hard mode of the environment, the convergence speed of the algorithms slows down. Moreover, the final convergence value decreases as the difficulty of the environment increases. For instance, in the easy mode, TD3 converges to a reward value of -0.05 within 150,000 steps, while it takes 200,000 steps to converge reward value of -0.15 in the hard mode.
5.0
0.0
5.
-6.2
-0.2
103
101
7D3
SAC
54c
1AC
1P0
-5.8
55030 135000 200020
300300
50300 150000 350300 200000 330003
35003 100000 35:000 202000 330300
(a) Training curve: easy mode
(b) Training curve: medium mode
(c) Training curve: hard mode
Figure 6:
Training curve for SAC, TD3, and PPO 5-stage OPS environment for (a) easy mode, (b)
medium mode, and (c) hard mode. The dashed region shows the area within the standard deviation.
2.0
10
1.
g
6.6
208
805
TD3
7D3
TD3
SAC
36C
SBC
PP0
E02
710
Ranations
(a) Testing curve: easy mode
(b) Testing curve: medium mode
(c) Testing curve: hard mode
Figure 7: Evaluation of the stacked pulse power Pv (normalized) of different policies in the testing envi- ronment for (a) easy mode, (b) mediun mode, and (c) hard mode.
Following the training of RL agents, we proceeded to evaluate the performance of the trained policies in the testing environment. The final pulse energy Pv achieved under different iterations depicted in Fig. 7(a) for the easy mode, Fig. 7(b) for the medium mode, and Fig. 7(c] for the hard mode. We observed that SPGD performs admirably in the easy mode, where the control problem is close to convex. However, its performance deteriorates significantly in the medium and hard modes, which non-convex control problems. This disparity arises because RL controllers capable of learning better policies through exploration in non- convex settings. Furthermore, the off-policy RI. algorithms (TD3 and SAC) outperform the on-policy RI algorithm (PPO) in our simulation environment. Across all methods, the testing performance in the hard mode is lower than that the medium mode, despite both being non-convex control problems. This discrepancy can be attributed to the more complex and realistic noise present in the hard mode, which slows down the convergence rate and roduces the final pulsc energy the testing environment.
The final pulse energy Pw is reported for both the training and testing environments, detailing the perfor- mance of the trained policy presented in table It's noteworthy that the training and testing environments for the casy and medium modes exhibit similarity, akin to the classical Atari setup, with performance dis- crepancies primarily arising from inherent randomness. However, the hard mode introduces distinct noise characteristics between the training and testing environments due to gradual temperature drift. This dis- crepancy results in a performance gap between the two domains. As depicted, in the easy mode, SPGD significantly outperforms other algorithms, as indicated by Welch's t-test with p 0.05 (Welch] 1947). For the intricate medium and hard modes, TD3 and SAC exhibit superior performance compared to PPO and
8

---

Published in Transactions on Machine Learning Research (10/2023)
SPGD, with statistical significance according to Welch's t-test. Notably, there exist no substantial differ- ences between TD3 and SAC. The fundamental differentiation between SAC, TD3, and PPO lies in their underlying methodologies. SAC and TD3 operate as off-cc approaches, whereas PPO functions as an on-policy method. This can potentially be attributed to PPO's susceptibility to becoming ensmared in local optima due to limited exploration and a tendency to emphasize recent data. On the other hand, SAC and TD3 harness historical experiences, drawn from replay buffer with a capacity of 10000, thus avoiding undue reliance on only the most recent data. Our observations align harmoniously with insights derived from a benchmark study (Yu et al., 2020). As elucidated in (Yu et al. 2020), SAC achieves successful task resolu- tion across the board, whereas PPO demonstrates adeptness in most tasks while encountering challenges in some cases. The experimental results of different stage OPS environments and discussions can be found in appendices B.3 and B.5
Table 2: The performance of the implemented algorithms is evaluated based on the Final Pulse Energy PN (mean + standard deviation) in 5-stage OPS environments. The results of the best-performing algorithm on each task, as well as all algorithms that have performances that are not statistically significantly different (Welch's (-lest with p < 0.05), are highlighted in boldsace.
Mode
Evaluation environment
SPGD
PPO
SAC
TD3
easy
training
0.9919 0.0121
0.8184 + 0.0884
0.9410 + 0.0572
0.9552 + 0.0483
testing
0.9909 0.0125
0.8067 0.0661
0.9403 + 0.0531
0.9572 = 0.0501
medium
training
0.6155 + 0.0271
0.7591 + 0.0728
0.9145 + 0.0727
0.9353 = 0.0451
testing
0.6178 + 0.0263
0.7519 + 0.0623
0.9098 1 0.0838
0.9285 1 0.0437
hard
training
0.5321 0.0352
0.6977 0.0732
0.8739 0.0775
0.8724
0.0680
testing
0.5132 + 0.0348
0.6526 + 0.0687
0.8371 +: 0.0804
0.8488 + 0.0615
3.3 Run-time Analysis
Our experiments were conducted on Ubuntu 18.04 system, with an Nvidia RTX 2080 Ti (12 GB) GPU, Intel Core i9-7900x processors, and 64 GB memory. We present the time usage of distinct RL algorithms within a 5-stage OPS environment under hard mode, as delincated in table 3 In the table, "TPS' signifies time usage per step, measured in seconds. The "convergence step' designates the iteration count at which an algorithm attains its peak value, beyond which further iterations yield minimal alterations. During training, the time per step encompasses simulation TPS (denoting OPS simulation time per step) and optimization TPS (reffecting the time spent on gradient descent optimization per step). For instance, TD3 involves a simulation cost of 0.0086s per step and an optimization of 0.0175s per step. With approximately 212018 steps to convergence, the total training duration amounts to about 5456s. Notably, there exists negligible disparity in training times across various algorithms. In the post-training inference phase, the inference TPS represents the time taken for action calculation per step. For instance, TD3's inference cost. is 0.0036s per step, with around 58 steps to achieve high pulse energy, leading to a total control time of approximately 0.71 s. As evident, TD3 and SAC distinctly outperform PPO in terms of inference time efficiency, showcasing remarkable superiority. The GPU memory usage of all algorithms is about 1000 MB.
Table 3: Time usage (simulation, optimization, inference time) of the RL algorithms on 5-stage OPS environment with hard mode. The time usage per step (TPS) is quantified in seconds. The algorithm with the optimal time efficiency, as well as algorithms that are not significantly different-performer (Welch's t-test with p < 0.05), are highlighted in boldface.
Simulation
Training
Inference convergence step
Time
simulation TPS
optimization TPS
convergence step
total time
inference TPS
total time
PPO
0.0086
0.0274
142666 + 25390
5141 + 979
0.0057
78 # 10
1.13 # 0.21
SAC
0.0086
0.0199
168844 = 21889
4833 + 740
0.0046
10
0.89 0.18
TD3
0.0086
0.0175
212018 + 28484
5456 + 804
0.0036
58 7
0.71 0.12
3.4
Comparison of the different settings of OPS environment
In this section, we investigate the impact of different modes (easy, medium, hard) and stage numbers (N) in N-stage OPS environment. We evaluate the trained TD3 and SAC policies, as well as SPGD, on different
9

---

Published in Transactions on Machine Learning Research (10/2023)
300000
TD3ghard
SACghard
TD3@medium
Testing puise energy
0.9
Convergence step
SAC@medium
TD3ghard
200000
eD3@easy
harghard
SAD@easy
0.7
SPGD@hard
150000
TD3@medium
SPGD@medium
100000
0.5
TD3@easy
eaygeasy
50000
SPGD@easy
2
Stage number QPS
Stage number
(a) Testing curve (final return Pm)
(b) Training convergence step
Figure 8: (a) Final return PN of different stage OPS on testing environment controlled with TD3 or SAC. (b) Convergence steps for the training of TD3 and SAC on different stage OPS environments. The dashed region shows the area within the standard deviation.
testing enviromments with varying stage numbers. Figure 8(a) illustrates the final return PN in relation to the stage number N in the N-stage OPS environment, comparing the performance of different algorithms across different modes. From the figure, draw the following conclusions: (1) For the easy mode, SPGD outperforms SAC and TD3, and all methods achieve almost-optimal performance regardless of the stage number N. (2) Across both the medium and hard modes, the performance of off-policy RL methods (SAC and TD3) is comparable, and RL methods are better than SPGD. (3) As the stage number increases in the hard and medium modes, the performance SPGD drops significantly. Moreover, for N 2 4, RL methods (SAC and TD3) outperform SPGD significantly (under Welch's t-test). Figure [(b) illustrates the training convergence steps for different stage OPS. It be observed that the stage number increases, the number of steps required for training convergence also increases significantly.
-0.1
Energy
Training Reward
0.7
0.6
-0.4
0.5
-0.5
2-stage
3 3-stage
2 2-t3gelTD3
2-stage(SPGD)
0.7
4-stage
Tt
TD
3-stape(SPGD 4-stage/5PGr 5.stage(5PGD)
5.stage
0.8
50000
100000 150000 200000 250000 300000
0.1
100 125
175
200
Iterations
Iterations
(a) Training curve: TD3 with hard mode
(b) Testing curve: TD3 with hard mode
Figure 9: Comparison of the results on hard mode N-stage OPS environment with TD3 algorithms. (a) shows the training curve; (b) shows the evaluation of TD3 and SPGD in the testing environment. The dashed region shows the area within the standard deviation.
To provide clearer illustration of the impact of stage number N in the OPS environment, present the training and testing curves for TD3 and SPGD on the hard mode. Figure [(a] displays the training curve of TD3 on different N-stage OPS systems. It can be observed that as the stage number increases, the training convergence becomes slower. Figure 9(b) showcases the testing curve of SPGD and TD3 on different N- stage OPS systems. From the figure, we can draw the following obsorvations: (1) With an increase stage number, the final return Py becomes smaller for both TD3 and SPGD, indicating decrease performance the system becomes more complex. (2) TD3 consistently outperforms SPGD for stage numbers N 2 4.
10

---

Published in Transactions on Machine Learning Research (10/2023)
This suggests that RL methods like TD3 are more effective in handling the control challenges posed by OPS systems with a larger number of stages.
3.5
Transferring trained policy between different modes
5.9
0.
8.5
03
hardehard hesomedim
medun-turt medumo-erdin
hard
5.
harcowayy
60
medumcestt
estyomed
53
25
20
75
200
115
150
273
250
61
23
30
150
150
372 200
51 22
50
300
125
150
275
200
(a) hard trained
(b) medium trained
(c) trained
Figure 10: Demonstration of the transfer performance of the trained policy on (a) hard mode training environment; (b) medium mode training environment; (c) easy mode training environment. The dashed region shows the area within the standard deviation.
The major difference between the simulation and real-world environments is the different noise levels. In order to investigate the transferability of trained policies between different noise levels in the OPS envi- ronment, we conducted a simulated experiment. Our simulation enviromment incorporates different noise levels depending on the difficulty mode: "easy", 'medium', and 'hard'. We trained policies on the 'hard" mode environment and tested their performance on 'hard', "medium"; and "easy" mode environments. The transfer results are presented in Fig. [0(a] As can be observed, the trained policy can be successfully transferred to both "medium" and "easy" mode enviromments, achieving high performance in terms of pulse energy. Figure I0(b) and Figure [0(c) depict the transfer results of policies trained on the 'medium' and 'casy" mode environments, respectively. It can be seen that when policies trained on casier environments are transferred to a harder enviromment, their performance drops. On the contrary, policies trained in harder environments can be effectively applied to easier environments. Based on these findings, training policies in harder simulation environments that introduce more noise and uncertainty can be more useful. This approach allows us to explore and develop fast and robust control algorithms that can then be deployed on real-world physical systems.
4 Discussion
4.1
Real-world environment and simulation
Deploying RL algorithms real-world optics systems poses several challenges, including the need for signal conversion, time delays, and manual tuning of optical devices. In our simulation system, have the advantage of faster control steps and simplified initial alignment. In real-world optics systems, the optical signal needs to be converted to an electrical analog signal using photo-detector (PD), which is then further converted to a digital signal using an analog-to-digital converter (ADC). Furthermore, the delay line device or controller introduces supplementary time overhead to the control step, typically spanning from 0.1 to second. Nonetheless, within our simulation system, both simulation and control steps can be performed in less than 0.01 soconds, as demonstrated in table 3 This facilitates expedited training and evaluation processes. Furthermore, in real-world OPS systems, manual buning of the optical devices is required when the optical beams misaligned, which can be time- consuming process taking several hours even days, In contrast, in our simulation system, we can casily reset the environment to achieve the initial alignment, simplifying the setup and reducing the time required for system preparation.
Previous endeavors to directly train RL algorithms to real-world OPS systems have encountered obstacles, inchuding slow training real environment, and unstable and non-optimal convergence RI. algorithms. Consoquently, the sim2real approach, which involves training and evaluating RL algorithms in simulation environments before deploying them real-world systems, has garnered considerable interest. Our objective
11

---

Published in Transactions on Machine Learning Research (10/2023)
is to conduct comprehensive research on RL algorithms within the simulation environment and subsequently leverage the sim2real approach to transition these algorithms into real-world applications.
The validity of our simulation is partially confirmed in (Tünnermann & Shirakawa, 2019 Yang et al. 2020
The experiments detailed in (Tinnermann & Shirakawa, 2019) align with the 1-stage OPS environment described within our paper. In their study, the authors conducted both simulations and real experiments to assess RL algorithms and demonstrated their usefulness in the context of their paper. They also emphasized the value of using simulations for training due to the time and effort required for real-world RL training. Although OPS simulation and experimental settings more intricate than those in (Tinnermann & Shirakawa 2019), the underlying physics remains consistent, bolstering the reliability and accuracy of our simulation environment.
4.2 RL controllers and different simulation modes
The experimental results demonstrate that off-policy RL algorithuns (TD3 and SAC) outperform traditional SPGD controllers in larger N-stage OPS systems, particularly in the challenging hard mode. In the simu- lation, the casy mode corresponds to the traditional control approach in which experts manually fine-tune the OPS system to achieve a state close to the global optimum (representing the real-world "easy' mode) before employing SPGD for control. However, the future of optical control lies in automation. The hard mode in the simulation reflects more realistic scenario where direct control of the OPS system is performed without initial expert tuning. In this context, RL controllers exhibit significant promise for optical systems. This motivates our focus developing OPS simulation environments, emphasizing the need for fast training and noise-robust RL algorithms capable of handling non-stationary noise and non-convex control objectives. Additionally, exploring the nonconvex and periodic nature of OPS objectives holds potential benefits for real-world RL applications, incorporating valuable structural information into the control tasks in optics.
4.3 Limitations
In our simulation, we model vibration-induced fast noise as Gaussian noise. However, it's important to acknowledge that vibrations may not consistently conform to a Gaussian distribution. The characteristics and features of noise generated by vibrations can vary based on range factors. For instance, vibration properties exhibit non-symmetry or involve non-linear effects, the resulting noise profile might deviate from Gaussian attributes. Guided by the Central Limit Theorem, we consider Gaussian noise to be a suitable approximation for representing fast noise in our simulation. As a result, the principal disparity between real-world experiments and simulations often originates from divergent noise properties. In forthcoming research, we aim to delve into sim2real transfer and formulate more realistic gap, taking into account these muancod noise charactcristics.
5 Conclusion
In this paper, we present OPS, an open-source simulator for controlling pulse stacking systems using RL algorithms. To the best of our knowledge, this is the first publicly available RL environment specifically tailored for optical control problems. We conducted evaluations of SAC, TD3, and PPO within our proposed simulation environment. The experimental results clcarly demonstrate that off off-policy RL methods outper- form traditional SPGD-PID controllers by a substantial margin, especially in challenging environments. By offering an optical control simulation environment and providing RL benchmarks, aim is to encourage the exploration and application of RL techniques in optical control tasks, as well as to facilitate further advancements of RL, controllers in the field of natural sciences.
12

---

Published in Transactions on Machine Learning Research (10/2023)
References
Abulikemu Abuduweili, Bowei Yang, and Zhigang Zhang, Modified stochastic gradient algorithms for con- trolling coherent pulse stacking. In Conference on Lasers and Slectro-Optic. pp. STh4P.1, 2020a.
Abulikemu Abuduweili, Bowei Yang, and Zhigang Zhang. Control of delay lines with reinforcement learning for coherent pulse stacking. In Conference Lasers and Opectro-Optics pp. JW2F.33, 2020b.
Abulikemuu Abudwweili, Jie Wang, Bowei Yang, Aimin Wang, and Zhigang Zhang. Reinforcement learning based robust control algorithms for coherent pulse stacking. Opt. Express, 29(16) 17860662-2081,,, Aug 2021.
Alekh Agarwal, Sham M. Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22 (98):1 76, 2021.
Ignas Astrauskas, Edgar Kaksis, Tobias Flöry, Giedrius Andriukaitis, Audrius Pugžlys, Andrius Baltuška, John Ruppe, Siyun Chen, Almantas Galvanauskas, and Tadas Balčiunas. High-energy pulse stacking via regenerative pulse-burst amplification. Optics letters, 42(11) 220112220112244 2017.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. arXiv preprint 18:2: 2018.
Thomas Baumeister, Steven L Brunton, and Nathan Kutz. Deep learning and model predictive control for self-tuning mode-locked lasers. JOSA B, 35(3):617-620 2018.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-25 279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiu: 01066.015000 2016.
Marin Bukov, Alexandre GR Day, Dries Sels, Phillip Weinberg, Anatoli Polkovnikov, and Pankaj Mehta. Reinforcement learning in different phases of quantum control. Physical Review X, 8(3):031086, 2018.
Gert Cauwenberghs. A fast stochastic error-descent algorithm for supervised learning and optimization. Advances in neural information processing systems, 5:244 251, 1993.
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao Hc, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Learning, pp. 1125 1134. PMLR, 2018
Amir Dembo and Thomas Kailath. Model-free distributed learning. IEEE Transaclions on Neural Networks, 1(1):58-70) 70, 1990.
C Dorrer, DC Kilper, HR Stuart, G Raybon, and MG Raymer. Linear optical sampling. IEEE Photonics Technology Letters, 15(12):1746-1778, 2003.
Oliver J Dressler, Philip D Howes, Jaebum Choo, and Andrew J deMello. Reinforcement learning for dynamic microfluidic control. ACS omega, 1008:400111 2018.
Simon S Du, Yuping Luo, Ruosong Wang, and Hanrui Zhang. Provably efficient g-learning with function approximation via distribution shift error checking oracle. arXiv preprint arXiv:1906.00 06332,, 2019.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine leorning, pp. 1339-1338. PMLR, 2016.
Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint Xie: 12901, 2019.
Martin E Fermann and Ingmar Hartl. Ultrafast. fibre lasers. Nature photonics, 7(11):868 874, 2013.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80, pp. 1596.1966. PMLR, 2018.
Goëry Genty, Lauri Salmela, John M Dudley, Daniel Brunner, Alexey Kokhanovskiy, Sergei Kobtsev, and Sergei K Turitsyn. Machine learning and applications in ultrafast photonics. Nature Photonics, pp. 1-11, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. Tn Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1861 1870. PMLR, 10-15 Jul 2018.
Johan Hult. A fourth-order runge-kutta in the interaction picture method for simulating supercontinuum generation in optical fibers. J. Lightwove Technol., 25(1))3777 3775, Dec 2007.
Yoanna M Ivanova, Hannah Pallubinsky, Rick Kramer, and Wouter van Marken Lichtenbelt. The influence of a moderate temperature drift on thermal physiology and perception. Physiology & Behavior, 229:1 13257, 2021.
Jun Izawa, Toshiyuki Kondo, and Koji Ito. Biological arm motion through reinforcement learning. Biological cybernetics, 91(1):10-2) 2004.
Michael Laskin, Denis Yarats, Hao Lin, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprinl ar arXiv:21 10222.0..000 15191, 2021.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arxiv preprint W0X001150900071,, 2015.
Sobhan Miryoosefi, Kianté Brantley, Hal Daumé III, Miroslav Dudík, and Robert Schapire. Reinforcement learning with convex constraints. arXie preprinl or 1xXxv:1906.09333 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Joannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXic preprint 3xXiv:1312.5602 2013.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukeuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.08 160:009999 2016.
D Pomerleau. An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems; Morgan Kaufmann Publishers Inc.: Burlinglon, MA, USA, 1998.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. 2019.
Stable baselines3.
A Ray, J Achiam, and D Amodei. Benchmarking safe exploration in deep reinforcement learning, arxiv. arXiv preprint XxX:::1910001100, 7, 2019.
John F Ready, Industrial applications of lasers. Elsevier, 1997.
Bahaa EA Saleh and Malvin Carl Teich. Fundamentals of photonics. john Wiley & sons, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXio preprint arxi: 707.06347 2017.
Yuen-Ron Shen. The principles nonlinear optics. New York, 1984.
Anthony F. Siegman. Lasers. University science books, 1986.
14

---

Published in Transactions on Machine Learning Research (10/2023)
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic policy gradient algorithms. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, 387 395, PMLR, 2014. URL http:://rroeeddings mlr. www.spreessse.... v32/ silver14.html
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam,, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, (29(778)):484444 489, 2016.
Henning Stark, Michael Müller, Marco Kienel, Arno Klenke, Jens Limpert, and Andreas Tünnermann. Electro-optically controlled divided-pulse amplification. Optics express, 25(12):13494-13503, 2017.
Chang Sun, Eurika Kaiser, Steven L Brunton, and J Nathan Kutz. Deep reinforcement learning for optical systems: A case study of mode-locked lasers. Machine Learning: Science and Technology. 1(4):045013, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Doepmind control suitc. arXiv preprint arXiv:1801.0068 2018.
Henrik Tünnermann and Akira Shirakawa. Delay line coherent pulse stacking, Opt. Lett., 42(23))4829-4882. Dec 2017.
Henrik Tünnermann and Akira Shirakawa. Deep reinforcement learning for coherent beam combining ap- plications. Opt. Express, 27(17):24223 (0(1)):22223-22230, Aug 2019.
Carlo M Valensise, Alessandro Giuseppi, Giulio Cerullo, and Dario Polli. Deep reinforcement learning control of white-light continuum generation. Optica, 8(2):239-244) 2021.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, et al. Starcraft ii: A new challenge for reinforcement learning. arXiv preprint arvi:: 708. 782, 2017.
Hao-nan Wang, Ning Liu, Yi-yun Zhang, Da-wei Feng, Feng Huang, Dong-sheng Li, and Yi-ming Zhang, Deep reinforcement learning; a survey. Prontiers of Information Technology & Electronic Engineering, 21 (12):1726- 1744, 2020.
Bernard L Welch. The generalization of 'student' 'sopbole when several different population varlances are involved. Biometrika, 34(1-2):28-3) 35, 1947.
Gordon Wetzstein, Aydogan Ozcan, Sylvain Gigan, Shanhui Fan, Dirk Englund, Marin Soljačió, Cornelia Denz, David AB Miller, and Demetri Psaltis. Inference in artificial intelligence with deep optics and photonics. Nature, 588(7836):39 47, 2020.
Bowei Yang, Guanyu Liu, Abuduweili Abulikemu, Yan Wang, Aimin Wang, and Zhigang Zhang. Coherent stacking of 128 pulses from a ghz repetition rate femtosecond yb:fiber laser. In Conference on Lasers and Electro-Optien pp. JW2F 28, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning, In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Uperoft, and Peter Corke. Towards vision-based deep reinforcement learning for robotic motion control. arXiu preprint arXis:1511.0379. 2015.
Pu Zhou, Zejin Liu, Xiaolin Wang, Yanxing Ma, Haotong Ma, Xiaojun Xu, and Shaofeng Guo. Coherent beam combining of fiber amplifiers using stochastic parallel gradient descent algorithm and its application. IEFE Journal of Selected Topics in Quantum Flectronics, 15(2):248 256, 2009.
15

---

Published in Transactions on Machine Learning Research (10/2023)
A Related Works
A.1 Optical Pulse Stacking
High-cnergy lasers find extensive application in scientific rescarch, medical procodures, and various industries [Fermann& & Hart] 2013). Nonetheless, the pulse energy of individual lasers often falls short for applications such as large-scale industrial material processing (Ready 1997). This limitation arises due to the non- lincarities and losses inherent in laser resonators (Siegman 1986. Overcoming this challenge necessitates innovative approaches to achieve higher energies without being constrained by the limitations of a single laser pulse. Optical (coherent) pulse stacking has emerged & Shirakawa, 2017). offering promising avenue for scaling pulse energy. The nonlinearity and periodic- ity inherent to light interference, underlie not only OPS but also other optical control challenges. These techniques are integral to precision measurements, industrial manufacturing, and scientific investigations. Our simulation, which mirrors these optical control complexities, represents a significant and representative environment for optical control experimentation.
solution to this predicament (Tünnermann)
Foltter
Photo Oatecres
Jine
delay Inee
Figure 11: Real-world optical pulse stacking system. maximum pulse energy.
controller adjusts time delay values to achieve
The actual OPS system is depicted in Fig. 11 A controller is responsible for determining the value of each time delay т by measuring the final stacked pulse using a photodetector. Subsequently, these time delay values are transmitted to each delay line device in order to adjust the positions of the pulses. It should be noted that in Fig. I1 the controller is connected to the electric signal line of the 1st, 2nd, and 3rd delay line devices located the bottom. Conducting real-world OPS control experiments can be both costly and time-consuming due to the complexities involved in the process. To visually illustrate the stacking procedure for combining two pulses, please refer to Supplemental Video 7
A.2 Reinforcement Learning Benchmarks
In tandem with the remarkable achievements of Deep Reinforcement Learning (RL), plethora of Rein- forcement Learning Benchmarks have been explored. These include well-known platforms like the Atari Arcade Learning Environment (Bellemare et al. 2013), the OpenAl gym (Brockman et al.) 2016), and the DeepMind Control (DMC) Suite (Tassa et al. 2018). Beyond these general benchmarks, the advancement of Reinforcement Learning across diverse applications has spurred the development of specialized bench- marks. Noteworthy examples include Rllab (Duan et al., 2016),, tailored for continuous control challenges, and Meta-World (Yu et al. 2020). serving as benchmark for Meta Reinforcement Learning. SafetyGym (Ray et al., 2019) evaluates the ability of RT agents to accomplish tasks while adhering safety constraints, while URLB (Laskin et al. (2221) assesses the performance of Unsupervised Reinforcement Learning, While these existing benchmarks aptly cater to control problems in the fields of Robotics and Machine Learning, a notable gap exists in the availability of benchmark and repository of user-friendly baseline algorithms for scientific control problems or optical control tasks. This significant gap is our primary motivation for propelling progress in the realm of RL benchmarks for optical control problems through OPS.
Pulse- htw.ee/eeeaaaaeeeeeeeeeaaaaaaaaa
gif
16

---

Published in Transactions on Machine Learning Research (10/2023)
B Additional details of Experiments
B.1 RL algorithms
Reinforcement Learning (RL) is a field of machine learning focused how intelligent agents should make decisions in an environment to maximize cumulative rewards (Sutton & Barto 2018). In our OPS envi- ronment, RL algorithms learn a policy that aims to maximize the output pulse power by controlling time delay, as detailed in section 2.3] For this study, we conducted a thorough evaluation of three RL algorithms: Proximal Policy Optimization (PPO), Twin Delayed Deep Deterministic Policy Gradients (TD3), and Soft Actor- Critic (SAC). We briefly illustrate each algorithm as follows.
Proximal Policy Optimization (PPO) is popular policy gradient algorithm used in reinforcement learning (Schulman et al. 2017). PPO belongs to the class of on-policy algorithms,, meaning it optimizes the policy by collecting data through interactions with the environment in real-time. The key idea behind PPO is to ensure that the policy updates are not too large, preventing significant policy changes that might lead to instability or catastrophic forgetting. Specifically, PPO uses a clipped surrogate objective function to update the policy. The objective aims to maximize the expected reward while also introducing constraint that prevents the policy update from deviating too far from the current policy.
Soft Actor-Critic (SAC) is one of the commonly used deep reinforcement learning algorithms intro- duced by (Haarnoja et al. 2018). SAC is an off-policy algorithm, which means it can learn from past experiences (replay buffer) and docs not rely on the most recent data like on-policy algorithms. Off-policy learning enables SAC to efficiently use data and improve sample efficiency. One the key features of SAC is the emphasis on entropy maximization. The policy is not only optimized to maximize the expected reward but also to maximize the entropy of the policy. By maximizing the entropy, SAC encourages exploration and avoids premature convergence to suboptimal policies. Another key feature is the soft value function. SAC uses soft value functions instead of the typical Q-functions used in other algorithms. These soft Q-functions estimate the expected return, but they are 'softened" by adding an entropy term.
Twin Delayed Deep Deterministic policy gradient (TD3) is a powerful deep reinforcement learning algorithm designed for solving tasks with continuous action spaces (Fujimoto et al., 2018). Similar to SAC, TD3 is an off-policy algorithm, which means it could efficiently use data and improve sample efficiency. TD3 follows the actor-critic framework. It consists of two neural networks: an actor-network and two critic networks. The actor-network learns the policy, mapping states to continuous actions, while the critic networks estimate the Q-values (expected return) for different state-action pairs. The actor network learns a deterministic policy, allowing for easier optimization in contimuous action spaces. TD3 employs two critic networks to reduce overestimation biases commonly encountered Q-learning algorithms.
For further information on RL algorithms, readers can refer to the comprehensive survey paper by (Wang
ct al. 2020
B.2
Experimental setting
To optimize the performance of RL algorithms, we conducted a grid search to identify optimal hyperparam- eters. Each set of hyperparameters was thoroughly evaluated using different random seeds. The search process involved training on the 5-stage OPS environment with medium difficulty. Detailed information re- garding the hyperparameter ranges and the selected values for TD3, SAC, and PPO can be found in tables 4
to 6 While we didn't explicitly mention the specific hyperparameter values except in tables 4 maintained the default settings provided by the stable baseline3 library.
6 we
Model size stands as a pivotal hyperparamcter, bearing significance not only on evaluation performance but also exerting influence on training duration. Table 7 presents a juxtaposition of diverse model sizes for the policy network within the TD3 algorithm and the 5-stage OPS environment. In this context, [64,64] signifies fully connected network boasting two layers, cach with a hidden dimension of 64. Converscly, [256,256] represents a fully connected network encompassing three layers, with hidden dimension of 256. Evidently, the network architecture of (256,256) emerges as the most proficient, excelling both in the assessment of
17

---

Published in Transactions on Machine Learning Research (10/2023)
Table 4: TD3: ranges used during the hyperparameter search and the final selected values.
Hyperparameter
Range
Best-selected
Size of the replay bufler
10000,0000,,00000000
10000
Step of collect transition before training
{100, 1000, 10000}
1000
Unroll Length/n-ss
{1,10, 100}
100
Training epochs per update
{1,10, 100)
100
Discount factor (7)
(0.98, 0.99, 0.999}
0.98
Noise type
('uormal', 'ornstein-uhlenbeek' None]
'normal'
Noise standard value
{0.1, 0.3, 0.5, 0.7, 0.9}
0.7
Learning rate
{0.0001, 0.0003, 0.001,0.00,00001 {2, 3}
0.0003
Policy network hidden layer
2
Policy network hidden dimension Optimizer
(64, 128, 256}
256
Adam
Adam
Table SAC: ranges used during the hyperparameter search and the final selected values.
Hyperparameter
Range
Best-selected
Size of the replay buffer
[1000,10000,100000
10000
Step collect transition before training
{100, 1000, 10000)
1000
Unroll dength/n-stepp
{1,10, 100}
1
Training epochs per update
{1,10, 100)
1
Discount factor (7)
{0.98, 0.99, 0.999]
0.99
Generalized State Dependent Exploration (gSDE)
(True, False)
True
Soft update coefficient for 'Polyak update" (T)
{0.002,0.005, 0.01, 0.02]
0.002
Learning rate
{0.0001, 0.0003, 0.001,0.003,0.01/
0.0003
Policy network hidden layer Policy network hidden dimension Optimizer
2
2
{64, 128, 256}
256
Adam
Adam
Table 6: PPO: ranges used during the hyperparameter search and the final selected values.
Hyperparameter
Range
Best-sclected
Unroll Length/n-step
25,5550,,,,,,
512
Training epochs per update
[1,5,10}
10
Clipping range
(0.1,0.2,0.4)
0.2
Discount factor (7)
(0.98, 0.99, 0.999)
0.99
Entropy Coefficient.
(0, 0.001, 0.01, 0.1)
0.001
GAE (A)
(0.90, 0.95, 0.98, 0.99)
0.95
Value function coefficient Learning rate
(0.11.0..005.0077....) (0.0001, 0.0003, 0.001,0.003,0.01)
0.5
0.0003
Gradient norm clipping
{0.1, 0.5, 1.0, 5.0) [2, 3}
0.5
Policy network hidden layer
2
Policy network hidden dimension Optimizer
(64, 128, 256}
256
Adam
Adam
18

---

Published in Transactions on Machine Learning Research (10/2023)
stacked pulse energy (Evaluation performance) and time efficiency (Training Time Usage). Consequently, opt for the (256,256] architecture our policy network configuration.
Comparative evaluation of various models within the TD3 algorithm and 5-stage OPS environment, with emphasis on performance, memory usage, and time costs.
Table 7:
Model size
Evaluation performance
Memory Usage Training Time Usage
[64, 64]
0.8931 = 0.0873
933 MB
83 min
[256, 256]
0.9285+0.0043 0.9243 #0.0399
989 MB
2 90 min
[256, 256, 256]
1030 MB
2 117 min
B.3 Results on controlling 4-stage and 6-stage OPS environments
We present the training testing iterations) for the 4-stage OPS environment in Fig, 12 and for the 6-stage OPS environment in Fig. 13 From the figures, it is evident that TD3 and SAC outperform PPO in terms of performance. Moreover, comparing Fig. 12 (4-stage) to Fig. 13 (6-stage), it can be observed that as the stage number increases, the training convergence becomes slower and the final return PN bocomes smaller, particularly in the medium and hard difficulty modes.
(training reward iterations) and testing curves (final pulse energy Px
-33
-0.3
-3
-0.4
-24
-25
.05
-55
-1.
53000 100300 352000 200002 230003 300020
50562 100000 159000 235002 255020 300030
55000 100200 152003 202000 250032 330300
Rterotions
kerstions
Reralions
(a) Easy mode: train
(b) Medium mode: train
(c) Hard mode: train
0.5
0.5
ca
3.5
3.3
0.4
1.4
54
0.3
903
700
602
MO
6C
5AC
100
25
50
35
160
125
350
175
285
25
93
140
125
163
175 203
25
93
75
160
19%
150
175
760
(d) Easy mode: testing
(e) Medium mode: testing
Hard mode: testing
Figure 12: 4-stage OPS experiments. Training reward was plotted for (a) easy mode, (b) medium mode, and (c) hard mode. Evaluation of the stacked pulse power P1 (normalized) of the testing environment plotted for (d) casy mode, (c) medium mode, and (f) hard mode.
B.4 Rendering the controlling results OPS environment
depicts the pulse trains on a 5-stage hard mode OPS system controlled by TD3, starting from a random initial state. It is evident from the figure that the TD3 algorithm is capable of attaining maximum power within 40 iterations. For more comprehensive visualization, please refer to supplemental video 2
Figure 14
http:////stttctcc.oooo/ooeeeette Pulse- htt.s//aiiiccciiiaaaaaaaaaaaaaaa
19

---

Published in Transactions on Machine Learning Research (10/2023)
0.0
Traning
703
TD3
703
36C
5AC
SAC
300000
Marations
Auralions
Ranation
(a) Easy mode: train curve
(b) Medium mode: train curve
(c) Hard mode: train curve
1.8
1.0
10
B
0.4.
9.7
57
3.6
26
0.3
0.3
53
022
5AC
Iterotions
Berations
Eoration
(d) Easy mode: testing curve
(e) Medium mode: testing curve
(f) Hard mode: testing curve
Figure 13: 6-stage OPS experiments. Training reward was plotted for (a) easy mode, (b) medium mode, and (c) hard mode. Evaluation of the stacked pulse power Pg (normalized) of the testing environment was plotted for (d) easy mode, (e) medium mode, and (f) hard mode.
0.4
0.0
202
230
0.0
300
0.0
-200
-200
71mo
(a) Initial state
(b) After 10 iterations
(c) After 20 iterations
1.0
trerstonn22 35
1.0
1.0
bemelon-:22
0.8
0.0
0.6
0.6
0.0
0.4
0.4
0.4
0.2
0.0
-230
-105
202
200
300
0.300
-200
-300
300
250
300
04
-200
-300
300
250
335
(d) After iterations
(e) After 40 itcrations
After 50 iterations
Figure 14: Demonstration of the controlling 5-stage OPS hard mode testing environment by TD3 algorithm after training. (a): initial state of pulses; (b) pulse stale after 10 control iterations; (c) pulse state after 20 control iterations; (d) pulse state after 30 control iterations; (e) pulse state after 40 control iterations; (e) pulse state after 50 control iterations.
20

---

Published in Transactions on Machine Learning Research (10/2023)
B.5 Discussion about experimental results
In our experiments, we observed significant performance advantage of SAC and TD3 over PPO. The primary distinction between SAC, TD3, and PPO lies in their underlying approach SAC and TD3 are off- policy methods, while PPO is an on-policy method. The fundamental characteristic of on-policy algorithms is that both the data used for learning and policy improvement originate from the same policy being updated. This implies that the agent relies heavily on recent experiences to adjust. its policy (Sutton & Barto 2018). In the context of our OPS environment, where numerous local optimal points exist (as depicted in Figure 2); PPO can become trapped in local optima due to its limited exploration and lendency to focus on recent data, limiting its ability to discover globally optimal policies. Conversely, SAC and TD3 can leverage historical experiences (e.g., from a replay buffer of size 10000) and are not constrained to relying solely on the most recent data. This typically results in more efficient data utilization and potentially better convergence towards optimal solutions compared to on-policy methods. Our observations align with findings from a benchmark study (Yu et al., 2020). As demonstrated in Figures 6 and 8 of (Yu et al. 2020), SAC successfully solves all tasks presented in the paper, while PPO manages to solve most tasks but falls short in cerlain cases. We acknowledge that PPO might exhibit. better performance in specific scenarios. However, considering the results presented in both our paper and (Yu et al., 2020 it appears that SAC may serve more promising starting point for addressing control problems using RL algorithms.
B.6 Feedback delay
In control systems, the presence of feedback loops invariably introduces delays due to finite sensing speeds, signal processing, computation of control inputs, and actuation. Specifically, let Treat()) denote the time delay (or state) of a real OPS system at time step t. Consider that activities such as photo-detection, data acquisition, analog-to-digital conversion, and action computation require a time interval 8. Consequently, a feedback delay of å emerges within our OPS control system. When we execute action system's time delay (state) might deviate from Treal(t) to (t) + D(6) due to the feedback delay. Here, D(d) accounts for an additive tilyy value attributed to the feedback delay D(-). The the next step's time-delay can be expressed as:
a (t)) the
Treat 1)= + az + D(5).
(6)
In our simulation, we replicate the feedback delay (or hardwareedepeeddent delay) by injecting Tree-running noise, including elements like slow noise and fast noise, into the delay lines. This process is elucidated in ( which bears a resemblance to the reprosentation in 66. This noise effectively emulates the sequence of events seen in weal-world experiments, encompassing photo-detection, data preprocessing, and the compulation of actions.
The proposed OPS environment offers versatile noise levels, allowing for the configuration of distinct feedback delays and associated system free-running noise. In this context, we present an exploration involving the training of RL algorithms within a low feedback delay system, followed by the transfer of the learned policy to system characterized by a higher feedback delay. The training noise within the 5-stage hard-mode OPS system aligns with the feedback delay ö and noise e, D(6) exemplified by 8 = 10 ms. Subsequently, the policy trained under these conditions is evaluated within systems featuring elevated feedback levels k- and noise Czew D(k d). where & takes on values such as 1, 20, 40, 60, 80, and 100. As depicted in Fig. 15 it is evident that, regardless of the specific feedback delay, the trained policy consistently allains convergence within 70 steps, thereby ensuring a cumulative pulse power exceeding 0.7 (a.u.). However, it is noteworthy that an increase in feedback delay corresponds to a notable decline in performance, particularly concerning pulse energy.
C Potential Impact and future work
Our simulation environment offers significant benefits in tackling challenging and realistic reinforcement learning problems. Real-world reinforcement learning problems are often highly challenging due to factors such as high-dimensionality of control, noisy behaviors, and distribution shift (Dulac-Arold et al. 2019: Agarwal et al. 2021 Du et al. 2019). By selecting a large N-stage number with the hard mode in our
21

---

Published in Transactions on Machine Learning Research (10/2023)
0.8
0.7
Stacked Pulse
0.6
delay:16
delay:205
delay:408
0.3
delay:605
de1
0.2
delay:1006
0.1
25
50
75
100
125
150
175
200
lterations
Figure 15: Subsequent the training TD3 policy within a low feedback delay OPS system characterized by 8, we then evaluate the policy's performance within systems characterized by higher feedback delays k 8.
simulation environment, OPS environment exhibits distinct noise distribution in the lesting enviromment compared to the training enviromment, which mirrors the challenges encountered in real-world reinforcement learning problems.
create high-dimensional and difficult control sccnarios. The hard mode of the
In our simulation, we have access to the objective function the OPS (ignoring noise), which provides valuable structural information and physical constraints. This enables us to explore additional information about the OPS function and incorporate into RI. algorithms. Rather than focusing on generic noncon- vex problems, many real-world scenarios involve specifie nonconvex control problems with known objective functions physical constraints (Miryoosefi et al. 2019). Exploring the nonconvex and periodic nature of the OPS objective can greatly benefit real-world RT problems that encompass structural information.
Similar to our OPS control system, optical control problems in general are influenced by the nonlinearity and periodicity of light interactions. This includes applications such as coherent optical interference (Wetzstein let al. 2020 and linear optical sampling (Dorrer et al.) 2003), which find utility in precise measurement, industrial manufacturing, and scientific rescarch. We consider our simulation environment be an important and representative optical control environment. Furthermore, RL methods have the polential to drive advancements in optical laser technologies and the next generation of scientific control technologies (Genty
ct al. 2020
22


Reviews:
[{"review_id": "ylR7GnPenH", "reviewer": "Reviewer_cc9b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The optical pulse stacking problem is presented clearly and the performance evaluations of the RL methods in the simulation environment seem rigorous. Further, the proposed simulation environment allows for scalable benchmarking, in particular, due to the adaptable dimension of search space and different noise settings. As the authors claim their results will attract readers from the ML and optics community, however, the RL methods are presented very briefly making it hard to understand their difference for people from the optics community. Additionally, the manuscript focuses on accuracy as the main metric. In the automation of experimental setups, also other metrics such as execution time, required resources, etc. of the algorithms are important. Finally, the authors claim that their simulation environment is close to real-world experiments, however, besides showing the experimental setup in the appendix, there is no direct comparison of the behavior of the simulation environment and the real experiment. Accordingly, this claim seems not well justified and it is unclear how the model trained in the simulation environment would perform in the real-world experiment. \n\n# Strengths\n- the manuscript raises important points regarding the automation of the adjustment of optical setups\n- the authors propose a scalable open-source environment to evaluate and compare different RL methods on an optical control problem\n- in the simulation environment, the problem can be decomposed into different difficulty levels, this might help to analyze and evaluate different algorithms\n\n# Weaknesses\n- there is no comparison of the simulation environment towards the real experiment, it would have been great to see how the simulation-based trained policies perform in controlling the real-world experiment and how close the simulations are to the experiment\n- the manuscript does not explain well the used methods, for people from the optics community that might not be fully confirmed with the ML methods this might be hard to follow\n- the authors focus mainly on performance as the metric and do not provide how long the training of the different methods takes (e.g. on a standard CPU/GPU). However, this might be an important consideration for people that have to solve real-world problems where time is always an important resource \n- after evaluating the methods the authors do not discuss why the methods lead to different performances. Can they identify certain features of the algorithm that seem to be important for solving the tasks?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The manuscript introduces a control problem arising in optical setups that are based on the goal of stacking several optical pulses generated by a laser. The overall goal of the task is to optimize the total energy of the stacked pulse by controlling different delay lengths in a stacking stage. This pulse stacking problem exhibits various complexities, including slow time-dependent noise and fast-varying noise, multiple local minima, and a non-convex error function. Further, scaling the number of to-be-stacked pulsed effectively increases the search space, the number of local minima and accordingly the difficulty. After introducing the problem, the authors propose an open-source simulation environment that emulates the optical experiment allowing for fast iteration and benchmarking of different control algorithms. In the following, different algorithms are evaluated in this environment. The authors hereby focus on four methods known from Reinforcement Learning (RL) that are evaluated in this simulation environment to solve the task, namely SPGD, PPO, SAC, and TD3. Furthermore, different versions of the task are presented that are generated by considering different varying noise conditions, initialization, and the number of stacked pulses considered, resulting in tasks of varying difficulty levels. The difficulty levels claimed are verified by evaluating the different algorithms and finding clear increasing errors and longer convergence times towards more difficult problems. Thereby, the results demonstrate that the easiest task, where the system is already close to a minimum, is effectively solved by SPGD, while tasks with more complex noise conditions require more advanced methods. Among the tested approaches, TD3 emerges as the most promising method for solving the pulse stacking task effectively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Main Requests\n- add justification/validation that the simulation environment is close to the experimental setup\n- move the chosen parameters/model sizes as presented in the appendix toward the main text and discuss the influence of model sizes\n- add the time it takes to train/evaluate the different RL methods \n- add a discussion on potential reasons why some of the methods work better than others\n\n# Minor Requests\n- remove doubled text in between equations (4) and (5) on page 5 + remove typos in the manuscript", "broader_impact_concerns": "There are no concerns on the ethical implications of this work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UE7Kt4zMhP", "reviewer": "Reviewer_Pc1f", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Caveat:\n\nI'm approaching this work as an RL practitioner with a Physics undergraduate degree. As such my understanding of the optical system that is targeted by this simulation environment may be limited and incomplete and so, too, my knowledge of the relevant optics literature.\n\n# Strengths:\n- Simple and clear writing.\n- Straightforward application of RL to a novel application domain.\n\n# Weaknesses:\n- Motivation for the optical stacking is missing. Why do people want to stack these signals in practice?\n- Surprising failure of PPO in the easy setting in Fig 6(a). A good discussion of why would be of interest to the RL community.\n- Discussion of limitations missing. While there is configurable noise in the state transition, is this the only source of added challenge in reality? A discussion of the ways the environment still falls short of realism (and could be improved in future) is missing. One question in particular that came to mind immediately is whether it is realistic to assume the algorithm has access to the next state instantaneously (without hardware-dependent delay?) or whether the noise in Equation (4) is enough to incorporate this systematic delay.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a simulated RL environment for the application of controlling the timing of several light pulses to maximize the energy of the resulting pulse. They also provide initial benchmarking results of some of the most popular RL algorithms (TD3, SAC, PPO) on the environment in various configuration; these are also compared to the non-RL state-of-the-art stochastic parallel gradient descent (SPGD). Results suggest that RL algorithms may have improvements to offer the chosen application domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Also see the weaknesses in the section above.\n\n# Additional changes:\n- The two paragraphs between Equations (4) and (5) are near duplicates.\n- \"If higher accuracy is required, one can include the noise term in the state [...], which transforms the control process into a POMDP.\" Can you elaborate on this? Why would this provide greater accuracy? Would the same algorithms still be appropriate then?\n- Can you discuss the actual wall time of these experiments? How practical is it to run these experiments for 300k steps? Would a practitioner train a policy in simulation and load a frozen policy into a physical system? If not, is it possible to give realtime state and reward feedback at low enough latency? ", "broader_impact_concerns": "Not knowing what pulse stacking is typically used for, I can't assess whether the application is of broader concern. Yet another reason to add a few motivating examples to the introduction.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xY6geUV7qw", "reviewer": "Reviewer_C8oN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Main Argument\n\n**Recommendation**: ~Reject~ Accept\n\nOverall, I believe that the OPS environment could be beneficial to many researchers in the areas of reinforcement learning and optical control and could be a significant contribution to these fields. Unfortunately, the experiments provided in the paper do not fully substantiate the usefulness of the OPS environment. \n\nMy main concern is that a number of the claims made in the _Experiments_ section and appendix are not fully supported by statistically significant data. If more random seeds were used, then the claims could potentially be supported by statistically significant data.\n\nIn section 3.2, multiple claims are made which are not supported by statistically significant data. The caption for the relevant figure, Figure 6(a), states that the dashed (shaded) regions are standard deviations (standard errors). With normality assumptions, a 95% confidence interval would be roughly double the size of this region, and the confidence intervals would likely overlap for all algorithms in these plots. Likely, this normality assumption is too strong and the confidence intervals would be even larger than this. Furthermore, when considering confidence intervals for a specific algorithm across plots, the confidence intervals would also likely overlap (for example, considering the confidence intervals of PPO in both Figure 6(a) and Figure 6(c)). Because of this, a number of claims are not supported by this data:\n\n1. TD3 and SAC perform better than PPO\n2. The final, converged-to value decreases as the difficulty increases\n\nFurthermore, in section 3.2 the paper makes the claim that in the hard environment, the convergence speed slows down. I am not fully convinced of this statement based on Figure 6. For example, TD3 and SAC both seem to converge around 150,000 steps for the medium and hard modes. The claim could likely be supported by increasing the statistical significance of the data shown (by using more runs) or by using a more measured claim. For example \"compared to the easy mode environment, the convergence speed is slower in the hard mode environment for all algorithms\". How was convergence speed measured here? Perhaps my evaluation of convergence is incorrect.\n\nSimilar claims to those above are also made in Appendix B, where similar experiments are run for the 4-stage and 6-stage OPS environment. Again, the data does not appear to be statistically significant and support the claims:\n\n1. TD3 and SAC outperform PPO\n2. As the stage number increases, the final return P_N becomes smaller\n\nIn Section 3.3, the claim that the rate of convergence decreases as the stage number of the OPS environment increases does not appear to be supported by statistically significant data in Figure 9(a). I could not find in the paper an explanation of the lines and shaded regions in Figure 9(a), but I am assuming the lines are the mean performance and the shaded regions are standard errors across the 10 runs. Can the authors confirm if this is the case? If this is the case, then as above, the relevant confidence intervals would be significantly overlapping. \n\nIn Section 3.4, how many runs are used for these transfer experiments? I am assuming only a single run was used (since no shaded regions are plotted). If this is the case, why was only one run used? Is one run enough to verify the hypotheses made in this section? Perhaps there is some part of the environment I am misunderstanding which warrants the use of a single run here, or perhaps more runs actually were used and my assumption is wrong. Or maybe this sections is simply for a proof-of-concept, showing that it is _possible_ to train on one version of the environment and do fairly well on the other versions of the environment. If this is the case, it might be good to clarify.\n\nSome experiments do not provide sufficient experimental details. For SAC, was the entropy scale hyperparameter swept? If so, what values were swept, and which value was selected? If not, what value was used? Why were episodes cut off at 200 steps? Episode cutoffs are known to affect learning dynamics, with longer episodes typically being more difficult [1]. How was convergence measured?\n\nAnother note is that these reinforcement learning algorithms may not be fully tuned. The hyperparameters were chosen using 3 random seeds. It is well known that these algorithms exhibit large variation across random seeds [2]. It could be the case that due to this randomness, incorrect hyperparameters were chosen. If the results in the paper are to serve as a benchmark of performance against which future researchers can compare their algorithms, then likely using more runs when tuning these hyperparameters would be useful.\n\nFinally, is hard mode actually more difficult than medium mode of the OPS environment? In many of the figures, the performance of each algorithm on medium mode is comparable to its performance on hard mode (and typically not statistically significant). Then in Figure 10 (a,b), transferring from training on medium mode to testing on hard mode results in similar final performance as training and testing on hard mode. Can the difficulty of hard mode be adjusted by adjusting the level of noise? How similar to a real optical stacking controller is hard mode (this might not even be possible to check)? Maybe this question would be answered by having more experimental runs, which would better show the difference in performance of each algorithm on medium and hard mode.\n\nIn summary, OPS could be a significant contribution to the fields of reinforcement learning and optical control. Unfortunately, the utility of the environment comes into question due to the lack of statistically significant data to support the claims made by the paper. I would be happy to change my recommendation to _accept_ if more runs were used such that the **experimental claims** which I noted above were supported by statistically significant data. Clearly, upon publication, the results in this paper will be used to benchmark new algorithms against, and so it is crucial that the results presented here be statistically significant and of high confidence.\n\n# Small Things\n\n**These did not affect the scoring of the paper**\n\n- Algorithm references are not in parentheses in Section 3.1\n- “We implement the RL algorithms using stable-baseline-3“ → “We used the algorithms implemented in stable-baselines-3”\n- Y axis label in Figure 8(b) is cut off\n- Section 4.1: “We In real-world optics systems...” → “In real-world optics systems...” \n\n# References\n\n[1] Andrew Patterson, Samuel Neumann, Martha White, Adam White. Empirical Design in Reinforcement Learning. 2023.\n\n[2] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. Deep Reinforcement Learning that Matters. 2018.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "# Summary of Contributions\n\nThe Paper _An Optical Control Environment for Benchmarking Reinforcement Learning Algorithms_ introduces a new environment for reinforcement learning called _Optical Pulse Stacking_ (OPS). OPS is a scalable simulator of optical control systems and will be released as open-source code. Currently, there are no open-source implementations of optical control simulated reinforcement learning environments, and OPS could help advance optical control through automation using reinforcement learning algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Critical Changes\n\nSee the _Main Argument_ above. \n\nI would be happy to increase my recommendation to an _accept_ if more runs were used as outlined in the final paragraph of the _Main Argument_ section above.\n\n# Changes which would simply strengthen the work\n\nSee _Small Things_ above.", "broader_impact_concerns": "# Broader Impact and Ethical Concerns\n\nNo ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new reinforcement learning environment that is motivated by an application in Optics (Optical Pulse Stacking). We had expert reviewers from both RL and the Optics communities. They have raised several issues in their initial reviews, including the statistical significance of some of the results.
The authors diligently revised the paper to address them. The reviewers are now all in favour of accepting the paper (Two Accepts and one Leaning Accept). They note that the paper might be of interest to both ML and optics communities. I recommend acceptance of this paper.
Decision: Accept (poster)