Aligning an optical interferometer with beam divergence control and continuous action space
Authors: Stepan Makarenko
Dmitry Igorevich Sorokin
Alexander Ulanov
Alexander Lvovsky
Venue: CoRL2021 Poster (2021)
Keywords: sim-to-real
robotics
optical interferometer

Abstract: Reinforcement learning is finding its way to real-world problem application, transferring from simulated environments to physical setups. In this work, we implement vision-based alignment of an optical Mach-Zehnder interferometer with a confocal telescope in one arm, which controls the diameter and divergence of the corresponding beam.  We use a continuous action space; exponential scaling enables us to handle actions within a range of over two orders of magnitude. Our agent trains only in a simulated environment with domain randomizations. In an experimental evaluation, the agent significantly outperforms an existing solution and a human expert.

Aligning an optical interferometer with beam divergence control and continuous action space
Stepan Makarenko 122,, Dmiiryy Sorokin', Alexander Ulanov', and A. I. Lvovsky 1.3
Russian Quantum Center, Moscow, Russia
Moscow Institute of Physics and Technology, Russia 3 University of Oxford, United Kingdom
makarenko sd @ @phy@tech.ecttttt
Abstract: Reinforcement learning finding its way real-world problem appli- cation, transferring from simulated environments to physical setups. In this work, we implement vision-based alignment of an optical Mach-Zehnder interferometer with confocal telescope in one arm, which controls the diameter and divergence of the corresponding beam. We use continuous action space; exponential scaling enables us to handle actions within a range of over two orders of magnitude. Our agent trains only in a simulated environment with domain randomizations. an experimental evaluation, the agent significantly outperforms an existing solution and human expert.
Keywords: sim-to-real, robotics, optical interferometer
Introduction
Reinforcement learning (RL) demonstrates incredible success in simulated environments, surpassing human expert in Atari [1], chess, shogi and Go [2], well in more complicated games such as Dota2 [3] and StarCraft [4]. In robotics, reinforcement learning shows remarkable results in pushing [5], grasping [6] and stacking objects [7]. RL enables robots walk, overcoming different obstacles [8] and learn agile animal skills [9]. However, RL agents are not yet widely used in real robotics and can hardly be compared to humans in a physical environment.
The principal problems of real-world applications of RL are nonstationarity and stochasticity of the physical environment, complexity and time intensiveness of data acquisition well unsafety of training and evaluation of policies on real robots [10]. A common method to handle these challenges to train agent in high-fidelit simulation (source domain) of the real environment (target domain). An agent can then be transferred to a physical robot. The main limitation of this approach inevitable discrepancy between the source and target domains that lead performance loss after transfer. This limitation can be mitigated with the randomization of simulator parameters [5, 9, 11].
A promising application domain for smart robotics is experimental optics. Optical physicists work with installations consisting of hundreds or thousands clements. These setups need to be thor- oughly aligned before measurements can be conducted. Dependent the setup size, such a setup can require several hours of work by a group experts on a daily basis. The automation of this procedure could drastically enhance the productivity of research groups. Since the alignment is an iterative decision-nee maaiinnn-miingggg process with a dell-defined reward function, it natural to formulate and solve it in terms of reinforcement learning.
The alignment of large optical setup can typically be separated into modular tasks which can be completed in sequence. A common modular task is the alignment of a Mach-Zehnder interferometer (MZI), which consists in matching the optical modes of the electromagnetic waves in two paths so that they exhibit visibility of unity when interfering with each other. The robotization of MZI align- ment was studied by Sorokin et al. [12], where this problem was treated as a partially observable Markov decision process, with the observation being sequence of interference patterns observed
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

on camera as the relative phase of the two arms is being varied. They trained a discrete-action (VD D3QN [13]) agent, dubbed Interferobor, in a simulated environment with domain random- izations and successfully transferred the trained agent to a real setup. The trained agent achieved superhuman-les performance terms of speed and quality of alignment.
However, the task of Ref. [12] was limited to matching the spatial positions and directions of the two otherwise identical beams. It is much simpler than those alignment tasks that occur in practice, which additionally require matching the geometric sizes of the beams, their divergence, and, in the case of pulsed lasers, arrival time. These additional features of the experiment result in more complex interference patterns (a richer observation space in RL parlance), hence requiring a more sophisticated agent for their interpretation.
The main contribution of our work is to develop a new agent for interferometer alignment, which different RI. algorithm and features number of
addresses this challenge. Our solution is based important innovations. First, the action space was changed from discrete to continuous, which also changed the policy task from classification to regression. This enabled us to solve the issue arising in the discrete action space, consisting in the number of actions growing exponentially with the number of degrees of freedom [14]. However, it raises a new concern: the agent's actions towards the end of the alignment process need to be much finer than in the beginning; the action space spans over two orders of magnitude. We address this with our second innovation - exponential action rescaling, which allows the agent to effectively explore actions of different magnitudes. Third, by granting the agent a continuous action space, we run risk to leave the safe space of experimental parameters, which may even result in damaging the equipment, We solve this issue by introducing penalties that discourage the policy from approaching boundaries. Fourth, we enhance the set of domain randomizations that help our agent perform better in the sim-to-real transfer. Our resulting agent' significantly outperforms the original Interferobot and a human expert in terms of both time and quality of the alignment.
2 Related works
RL agents for robotics can be trained either on real-world data or in simulation. The advantage of the former is that the agent receives hands-on experience of the environment in which it will be tested. The shortcoming is that the acquisition of a large dataset required for training is a complicated and time-consuming process. This is especially the case when the observations are vision-based. For example, the training of a grasping agent in Kalashnikov et al. [6] required several weeks, in spite of parallelization across 7 identical robots. This complication can be addressed with the help of behavior cloning: Vecerik et al. [15] trained an agent solving the insertion task using raw visual images using dozens of expert successful and failed trajectories. Hands-on training is simplified for agents without visual observations. For example, Haarnoja et al. [8] developed a quadruped walking robot that can generalize to unseen terrains with the training requiring as little as two hours. Another approach for training robot locomotion was presented by Yang et al. [16], who collected 4.5 minutes worth of data from simple quadrupedal robot to model the robot's dynamics and used this model to real-time action planning.
An alternative approach is to train the agent in simulation and transfer it to real-world system after- wards. To eliminate the discrepancy between the real and simulated environments, Tobin et al. [11] proposed a simple technique called domain randomization: they trained an object detection model on simulated images with different textures, lights, object and camera positions, and demonstrated that such a model achieves high accuracy in the real world. Peng et al. [5] trained robotic hand to push an object using randomization of physical parameters such as friction, mass, damping, etc. Another example is the assion-based task of dexterous in-hand manipulation [17], which demonstrated the effectiveness of applying randomizations to both the physical parameters and image observations.
The automation of optical system alignment is constantly evolving. In 1987 Gabler et al. [18] demonstrated automated alignment of an optical fibre used an iterative algorithm that consequently finds the maximum of a photo-detector signal with respect to each of the three fibre movement axis. Fang and Savransky [19] considered automated alignment of system of two lenses with 8 degrees of freedom. Acquiring focal plane image, their algorithm performs principal component analysis and Kalman filtering to calculate proper control inputs.
'https:///gii http:/ttwwt.t.teeeeeaeeeaaaaaaaa
Makar http://LLiinneeeereeeeee alignment

---

Deep machine learning, particularly RL, has become increasingly popular in experimental optics. For example, Sun et al. [20] used an RL agent stabilize mode-locked laser by controlling wave- plates and polarizers. RI. algorithms are also routinely used to optimize optical communications [21], e.g. to route traffic in optical transport networks [22, 23].
3 Mach-Zehnder interferometer
piezo mirror
BS
camera
Laser
lens
lens 2
mirror
BS
mirror 2
Figure 1: Conceptual scheme of the Mach-7ehnder interferometer. Lens 2, mirror and BS are motorized optical elements controlled by an RI agent.
Interference is a physical phenomenon that results from coherent addition of amplitudes of two or more overlapping waves; the resulting amplitude depends on the relative phase of the component waves. Interferometers, which are among the main instruments of experimental optics, use interfer- ence precisely measure this phase difference.
In this paper we consider an MZI displayed in Fig. 1. A collimated (parallel) laser beam is divided by beam splitter (BS 1). The resulting beams propagate through different paths before being recombined, with the help of steering mirrors, by another beam splitter (BS 2) and viewed by a cam- era. One of the mirrors mounted on piezoelectric transducer vary the relative phase şp of the beams. One of the beam paths (the lower one in Fig. 1) contains system of two lenses (telescope). which control the beam divergence and size. We use motorized mounts for mirror 2 and BS 2: lens is mounted on motorized translation stage. Aligning the interferometer requires precise matching of the two beams terms of their transverse positions, directions. radii and divergences.
The images from the camera serve as observations for our RL agent. Examples of acquired images shown in Fig. 2. The curved shape of the interference fringes is consequence of different divergences of the two beams. We apply an asymmetric sawtooth voltage pattern to the piezo, akin to Ref. [12], to view the temporal dynamics of fringes. The amplitude of the piezo motion corresponds to a phase difference of about 2x. For a misaligned interferometer, the piezo motion causes transverse displacement of the fringes visible in Fig. 2 (b-d), which permit the agent to extract the information about the sign of the difference of the beams' angles. When fully aligned, the interference will appear single blinking spot [Fig. 2 (a)].
The complexity of alignment arises because changing the angular orientations of the mirrors simul- tancously affects the position and angle of the lower beam, while the lens movement changes the lower beam radius and divergence. Moreover, images received from the camera can be corrupted by noise, aberrations, ambient light and dust. The problem is additionally complicated by the inaceu- racy of mounts and positioners whose relative action noise about 4%..
The position-dependenn Eupper (ær, y) + Etower(r, 12] where Eupper/lower (x, y) is the amplitude of each beam. The quality metric of alignment, visibility, is defined as
intensity of the interference pattern can be written [(x, y, sp)
(hoo min (1oo(())) max (Teot (4)) + min (TP)
where (4)) = IS (ax, y, yp dady is total light intensity in the image plane with the phase difference between the arms. The intensity depends time because of periodic piezo movement

---

c
Figure 2:
Camera images from the Mach-Zehnder interferometer. (b-d) Example images acquired with misaligned interferometer. The images in each row correspond varying path length differ- ences of the interferometer arms.
that varies the phase difference between the two interferometer arms. The visibility lies in [0, 1] range, where V corresponds to perfectly aligned interferometer.
Background
We consider standard partially observable Markov decision process (POMDP) setting where the purpose of an agent is maximize its cumulative reward during the policy execution. The obser- vation of the environment at timestep / is defined O, E O and is sampled from the distribution ot (o,();), where 8t E is the state. The policy of our agent a deterministic function a, T(.)) yielding action, which is element of the action space A. Following the action, the agent receives a reward r, x AA- - and the next observation 0++1 ~ ( st41), where $141 is produced according reward sum (return) from timestep
latent transition distribution $c+1 F(st+1|st, ac). The discounted
each episode R. ET 2 where y E [0, 1] the discount factor and T is the horizon of the episode.
The agent's objective is to learn the policy =" that maximizes the expected return ( Errp(r(s)) Ro] where p(T[Tr) is the distritution of trajectories T = (oo, 4n, 01 01, 0T OT) produced by policy
n = arg max J(T).
Policy gradient. The policy æ defined by parameters (i.e. π = TQ), so the optimization (2) performed with respect to the parameters: 8" arg maxe J (TT)). A common approach such a problem is policy gradient [24], which iteratively improves the policy in terms of expected return via gradient ascent:
aJ(
0 0, +
(3)
a0
where a is the learning rate. The algorithms of this family vary by the method of approximating the unknown (TT)).

---

A common approach to continuous-action Markov decision processes is TD3 [25] which is an ex- tension of another popular algorithm DDPG [26]. This algorithm uses three neural networks: one (actor) for deterministic policy and two (crities) for evaluating the action-state values Q(si, az)- Using two critics allows the algorithm to suffer less from overestimating the Q-values; in addition, the algorithm smoothens the Q-functions by adding Gaussian noise to the target actions when up- dating the parameters of the critics. Gaussian noise is also added to each policy action during the training encourage exploration.
Domain randomization. To lower the performance losses associated with the transferring of the agent from a simulated to real-world environment, domain randomization is used. The agent is trained for set of tasks with different dynamics F(s //sim) R F F(sx|8(-11 01-1: /reaa) and sensor noise models (o St Msim) = (o /real) where Msim/roal are sets of environment parameters. Whereas we do not know the actual Preal of the real environment, varying Psim in a range containing preal helps to improve generalization and transfer quality.
Our method
In this work, we apply continuous control reinforcement learning methods to align the MZI. As mentioned above, the alignment is performed step by step and the actions depend on the interference patterns observed by the agent, so it can be naturally viewed as a POMDP.
The agent is trained in simulation and evaluated listed in Appendix A and the MZI simulator is described in Appendix B.
real interferometer. The training algorithm is
State, observation and action. The state of the environment is vector consisting of the transverse position (x. y). direction angle (az; Qty), radius Flower and divergence curvature p of the lower beam in the plane of the camera. The position and transverse direction of the upper beam are assumed to be zero. This state is fully determined by the angles of the mirror and the beam splitter and the position of the lens. The range of allowed states, listed in Table 1, is restricted by the requirement that the beam remains visible on the camera and the position of the lens stays within the travel range of the translation stage.
The observation is a set of 16 consecutive 64 64 images acquired by the camera during one period of the piezo mirror. The actions are five-dimensional vectors specifying the relative angular deflections of both mirrors along the x and y axes and the linear displacement to their current positions. Each action vector component lies in the [-1, range, where the values of 11 correspond to the maximum and minimum values as listed in Table 1. The absolute values of each action are restricted to the interval [2.5 10 10-3, 1]. because smaller actions do not produce observable changes in the interference pattern and fall within the uncertainty range of motorized mounts.
the lens with respect
Episode and reset.
To compare the performance of our agent with that of Sorokin et al. [12], we keep the episode length equal 100 actions. At the beginning of each episode, the beam in the lower path is misaligned by setting the mirror angles and lens position to random values within the allowed range.
Reward. Under the normal alignment procedure, the reward at each step is positive and consists of two terms:
R R= V - log(1 - V).
(4)
The first term the visibility, which, discussed above, is the primary metric for the interferometer alignment quality. The second term rewards high quality of the final alignment, which is critical for optical experiments; it tends to infinity for V 1 [12].
Element angle/ position mirror 2, x
mirror 2, 1.8 10
BS 2, x
BS 2,
lens 2
Amplitude of max deflection
2.6 10-3
1.3 10
0.9 10
7.5
Table I: Maximum deflection of each optical element. Mirrors angles are given in radians, lens positions in millimeters.
5

---

Simulation without phase noise
Simulation with phase noise
Experiment
Mean intensity
10.0
7.5
2.5
0.0
2
10
12
14
0
8 Frame
10
12
14
0
8
12
Frame
Frame
Figure 3: Effect of phase noise. The plots display the integrated intensity of the interference pattern as the path length difference is varied. The behavior becomes closer real after applying phase noise.
However, if the agent proposes action that takes one of the controls out of the boundaries defined in Table 1. the episode is terminated to avoid damage to the equipment. Additionally, the agent is penalized with the reward of P -0.04. This penalty is important during early stages of the training. when the reward (4) is normally close to zero. A small negative reward for unsafe actions will train the agent to be aware of the bounds, but will not discourage exploration. The specific value P was handpicked via experimentation. On the other hand, when the agent is well-trained, it normally receives significant positive reward for each step. At this stage, terminating episode has a major negative effect on the return, thereby strongly discouraging unsafe actions.
Domain randomization. To learn more generalized policy and facilitate sim-to-real transfer, we introduce several domain randomizations, which are used during the training. The only randomiza- tion that changes the dynamic of the environment F( it, at) is the beam radius randomization within 20% of the measured radius r = 0.71 mm. The randomization is applied at the beginning of each episode.
Additionally, we introduce the following randomizations of the observation (o) First, to ad- dress the noise of the camera detector, scattering, air fluctuations and dust effects, we add Gaussian noise each pixel acquired by the camera. The standard deviation of the noise is 20% with re- spect to the simulated intensity value. Second, we rearrange cyclically the images within the video sequence to account for the randomness of the camera trigger. In addition, we randomize the duty cycle (fraction of time spent the forward and backward passes) of the piezo. Third, we vary the beams' brightness by +-30% to model the variance of the camera exposure. All these randomizations applied to each time step (i.e. they are constant for the 16 frames acquired in each step).
The randomizations described above have been used in Sorokin et al. [12] and have proven their effectiveness. An additional randomization we introduce in this work addresses irregularities in the motion of the piezo mirror and the frame rate, as well as fluctuations in the air density in different arms of the interferometer, all of which lead to random variation of the optical path length difference between the two MZI arms. To simulate these effects, we add Gaussian noise with the standard deviation of 0.5 rad to this phase difference.
Action rescaling. As the alignment progresses, the agent's actions become smaller and more precise. The typical action magnitude decreases during an episode by about two orders (as illustrated the experimental Section 6 below). It therefore desirable that the exploration noise also decrease with the action magnitude. We satisfy this by setting the agent's neural network to output "raw action" value ag 1,, from which the actual action calculated according to
a (igg)( 100g|ao1-1
if |aol > 0.17
(5)
if |aol 0.17
This transformation produces rescaled actions with absolute values |a| E {0} [2.5 10 10-3, 1].
Algorithm and network architecture. We use the standard TD3 algorithm with handpicked hyperparameters to produce a deterministic policy, which yields the raw action ag as described above. For both critics and the actor, we use the VGG-16 [27] architecture, modified as follows. The number of convolutional layers in the encoder set to 8, followed by three MI.P layers, with no dropout. We have chosen this architecture because max pooling operations help reducing the

---

V V2 0.92
V 2 0.95
V V2 0.98
Human
93.9 (0%)
103.6 (0%)
129.6 (10%)
TD3 (our agent)
56.16 (0%)
75.06 (0%)
120.1 (4%)
Interferobot
98.7 (7.6%)
116.1 (7.6%)
156.4 (10.6%)
Table 2: Comparison with human expert and the original Interferobot. The time required to reach visibility thresholds of 0.92, 0.95 and 0.98 are shown, together with the percentage of episodes the threshold has not been reached (in parentheses).
overfitting and sensitivity to individual pixel noise. We use orthogonal initialization in all models as we found it to ensure faster convergence.
We train the agent with discount factor = 0.8; this relatively short reward sight inspires our agent to reach high visibility faster. Gaussian exploration noise added to the raw action dg with the standard deviation decreasing exponentially from 0.5 to 0.02 during the training. Although the variance of this noise is independent of the magnitude of ag, effect on the actual action proportional to its magnitude due to the exponential dependence (5). The total number of steps is 10 and the replay buffer size 10 Updates are performed every ten steps. The whole training on an NVidia RTX 2080 GPU takes about 26 hours.
6 Experimental evaluation
1.000
0.675
0.800
0.925
Vesbility
Visiblit
0.003
0.4
model Interferabot
0.875
0.802
0.833
0.800
model
Interferroot
Step
Agent
(a) Mean visibility per step
(b) Box plot of visibility
Figure 4: Comparison of Interferobot and our agent, demonstrating the advantage of the latter in convergence speed and the final alignment quality. As evident from the box plot (b), the 25-75 interquartile range for our agent is significantly narrower than that of Interferobot, indicating higher stability of the former. All results are averaged over episodes.
For experiments, build an MZI according to the scheme shown in Fig. 1. We continuous HeNe laser with wavelength À = 632 nm, Newport Picomotor mirror mounts, a Standa 8MT167- 25I.S linear translation stage, CMOS camera with 16 fps acquisition rate, and high-bandwidth photodetector to precisely measure the visibility. In addition to our agent, we also train and evalu- the original Interferobot [12] with the action space extended to include the lens movement. For each agent, we run 50 episodes of evaluation with 100 timesteps. Figure shows the evaluation re- sults. Our agent significantly outperforms Interferobot in terms of the alignment time, final visibility (averaged over the last 40 steps of each episode) and its variance.
Additionally, agent has been compared with human expert, who aligned the interferometer 10 times by manually turning the controls on the physical setup. As evidenced by Table 2, our agent surpasses the human both in the speed and quality of the alignment.
common trick used by human experts aligning optics is turn two mirrors steering the same beam by equal and opposite angles. This action moves the position of the beam without changing

---

120.0%
80.0%
Action norm
80.0%
40.0%
20.0%
10"
Step
Step
(a) Percentage of parallel actions.
(b) Norm of actions.
Figure 5: Interpreting the agent's policy. a) Percentage of actions that change the distance between beams while conserving the directions increases with the step number. b) Mean norm of actions decreases with the step number.
its direction. Fig. 5a shows that our agent learns this method and uses it extensively: by the end of the procedure, about 70% of the actions contain such parallel movements. Fig. 5b shows the action norm as a function of the step number. The descending trend means that at the beginning our agent uses large actions to obtain rough alignment and decreases the action size to fine-tune the interferometer.
Table 3 presents the results of an ablation study. A standard TD3 agent does not outperform the original Interferobot and achieves a mean visibility of V 0.83. Action rescaling and phase noise randomizations significantly boost the performance of agent, leading to visibility of V 0.98. notable that the phase noise randomization slightly decreases the mean visibility of the Interferobot but improves the standard deviation. This bias-variance trade-off can be due to the relative simplicity of the Interferobot model which cannot capture richer observations produced by this randomization.
Model
mean visibility for last 40 step
standard deviation
TD3 AR + PN
0.98
0.03
TD3 + AR
0.95
0.06
Interferobot
0.92
12
Interferobot PN
0.90
0.08
TD3
0.83
0.18
Table 3: Comparative evaluation with ablated agents and original Interferobot [12]. PN: phase
noise; AR: action rescaling.
7 Conclusion
We demonstrated a novel RI algorithm for the automatic alignment of an optical interferometer, which contains system of lenses in one of its optical paths. Such setup results richer obser- vation and action spaces and constiututes a better approximation of a general problem of achieving mode matching between two arbitrary Gaussian beams. Our reinforcement learning agent with continuous action space solves the alignment problem successfully, surpassing both a previous so- lution and human expert. Important innovations include exponential scaling the action space, an additional domain randomization that helps the agent generalize to a real setup after training in simulation as well discrete reward structure that trains the agent to avoid unsafe actions. Our solution will work for any Mach-Zehnder interferometer with 2 x 2 degrees of freedom (mirror angles) controlling the position and direction, and one additional degree of freedom (lens position) controlling the width and divergence of one of the beams.

---

8 Acknowledgments
We acknowledge support from Russian Science Foundation (19-71-10092).
References
[1] Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13Xi::3322.602,, 2013.
[2] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al. general reinforcement learning algorithm that masters chess, shogi, and go through self self-play. Science, 362(6419):1 140-1144-400 1144, 2018.
[3] C. Berner, G. Brockman, Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912: 066600, 2019.
[4] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):5003354 2019.
[5] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pages 3803-3810 8800 IEEE, 2018.
[6] D. Kalashnikov, A. Irpan, P. Pastor, J. Iharz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. Conference on Robot Learning, pages 651-673. PMLR, 2018.
[7] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine. Composable deep re- inforcement learning for robotic manipulation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6244-6251. IEEE, 2018.
[8] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning, arXiv preprint arXiv: /8/2. 103, 2018.
[9] X. B. Peng, E. Coumans, Zhang, T.-W. Lee, Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXiv:2004.00784 2020.
[10] G. Dulac-Amold, D. Mankowitz, and T. Hester. Challenges of real-world reinforcement learn- ing, arXiv preprint arXiv; Tww...9.....900 2019.
[11] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[12] D. Sorokin, A. Ulanov, E. Sazhina, and A. Lvovsky. Interferobot: aligning optical inter- ferometer by a reinforcement learning agent. In Advances in Neural Information Processing Systems, volume 33, pages 13238-13248, Curran Associates, Inc., 2020.
[13] Y. Huang, G. Wei, and Y. Wang. Vd d3qn: the variant of double deep q-learning network with dueling architecture. In 2018 37th Chinese Control Conference (CCC), pages 9130-9135. IEEE, 2018
[14] G. Auloc-Amold, R. Evans, van Hasselt, Sunehag, Lillicrap, J. Hunt, Mann, T. We- ber, T. Degris, and B. Coppin. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv: 151.07667, 2015.
[15] M. Vecerik, O. Sushkov, D. Barker, T. Rothörl, T. Hester, and J. Scholz. A practical approach to insertion with variable socket position using deep reinforcement learning. 2019 Interna- tional Conference on Robotics and Automation (ICRA), pages 754-760. IEEE, 2019.

---

[16] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, J. Tan, and V. Sindhwani. Data efficient rein- forcement learning for legged robots. Conference on Robot Learning, pages 1-10. PMLR, 2020.
[17] O. M. Andrychowicz. B. Baker, M. Chociej. R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20 2020.
[18] C. Gabler, K. Li, Hackwood, and G. Beni. An optical alignment robot system. In Integration and Packaging of Optoelectronic Devices, volume 703, pages 8-28. International Society for Optics and Photonics, 1987.
[19] Fang and D. Savransky. Automated alignment of recontigurable optical system using focal-plan sensing and kalman filtering. Applied optics, 55(22):3967-5977 2016.
[20] C. Sun, E. Kaiser, S. L. Brunton, and J. N. Kutz. Deep reinforcement learning for optical systems: A case study of mode-locked lasers. Machine Learning: Science and Technology, (4):045013, 2020.
[21] D. Wang and M. Zhang. Artificial intelligence in optical communications: from machine learning to deep learning. Frontiers in Communications and Networks, 2:9, 2021.
[22] J. Varela- A. Mestres, J. Yu, L. Kuang, H. Feng, A. Cabellos-Aparicie, and P. Barlet- Ros. Routing in optical transport networks with deep reinforcement learning. Journal of Optical Communications and Networking, 1 1(11):547-558, 2019.
[23] J. Li, D. Wang, M. Zhang, and S. Cui. Digital using deep reinforcement learning. Optics Letters, 45(16):4654 4657, 2020.
self-evolve optical transceiver
[24] R. S. Sutton, D. A. McAllester, S. P. Singh, Y. Mansour, al. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pages 1057-1063. Citeseer, 1999.
[25] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference Machine Learning, pages 1587-1596. PMLR, 2018.
[26] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.029) 2015.
[27] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. arXiv preprint arXiv: 1556, 2014.
10


Reviews:
[{"review_id": "cAFLe-VEf1B", "reviewer": "Reviewer_GxfU", "summary": "This paper proposes using reinforcement learning to train a robot for aligning an interferometer. They extend setting to a richer observation and action spaces that are closer to the real-world. To work with continuous action spaces, TD3 algorithm is used. Additional novel components include predicting the actions in the log space to handle the large range of actions, and adding phase noise for domain randomization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The experimental evaluation, particularly on the real system, is convincing, and shows that the proposed approach can outperform the baseline, and a human. Further, the method discovers parallel movement, which is interesting.\n\nWeaknesses:\n- The section on action rescaling (line 216) was a little confusing — specifically, it wasn't clear how this is related to the exploration noise. It seems like exploration noise is reduced by decaying the standard deviation of the Gaussian noise (line 230).\n- It might be helpful to have a discussion of whether the proposed system is specific to the setup considered, or whether it could generalize to other related systems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper extends prior work to a practical real-world system, and can potentially be used to implement reinforcement learning in other real-world systems.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "- Clarify how action rescaling is related to exploration noise.\n- Discuss how the method may / may not be used in other real-world applications.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "GUSD02BR77R", "reviewer": "Reviewer_rhsP", "summary": "The goal of this paper is to use reinforcement learning to solve a practical problem in optics. In particulars, the authors use RL to align an optical interferometer. The problem seems perfectly suited for RL, as there is a clear observation space, action space and reward function, and the problem can be simulated with relative ease. The authors follow a sim-to-real approach, training their agent entirely in simulation, and then transferring it to a real world system. To ensure effective sim-to-real transfer, the authors make use of domain randomization, in which they both randomize the dynamics (a little), and the observations (a lot). \n\nThe closest prior work is that of Interferobot [1], on which this work closely builds. The primary improvement over Interferobot is the use of a continuous action space and an exponential action scaling scheme that allows the agent to take actions across two orders of magnitude. The authors use TD3 as their RL algorithm. The paper re-uses some domain randomizations from prior work, and also introduces some new ones. \n\n[1] D. Sorokin, A. Ulanov, E. Sazhina, and A. Lvovsky. Interferobot: aligning an optical interferometer by a reinforcement learning agent. In Advances in Neural Information Processing Systems. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Practical solution to a real world problem. This is not something that you can say for most papers submitted to CoRL (as most robotics is limited to toy-settings, not real world ones). \n- Clear presentation \n- Outperforms prior work by a decent margin\n\nWeaknesses\n- Somewhat incremental taking into account the prior work (Interferobot) ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is a bit incremental taking into account prior work (as I mentioned above), which is why I am keeping my recommendation at a weak accept. \n\nAfter author response: I found the author response to be quite compelling, so I am upgrading my rating to a strong accept. While I am not an optics expert, the works appears to be tackling a much harder problem than the previous work, and while each of the change might be incremental by itself, there are quite a few such changes that come together to solve a real world problem, which makes this paper a strong candidate for acceptance to CoRL (in my view).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I'll leave some typos here: \n\nTypos\nLine 91: “becomes” → “has become”\nLine 105: “mororized” → “motorized” \nLine 107: “required” → “requires”\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "2W8Tezc-D6", "reviewer": "Reviewer_ENy6", "summary": "The paper presents an application of Deep RL to the problem of aligning an optical interferometer. In simulation, TD3 is used to learn a controller that is able to manipulate different parts of the interferometer such as mirror orientations and lens locations. Domain randomization is applied for succesful transfer from simulation to the actual physical system. The resulting policy is able to control the interferometer parts more precisely and quicker than a human expert and another deep RL baseline algorithm. Since the action space on the actual system can span several orders of magnitude, the method applies an exponential scaling to the actions after sampling actions from the interval [-1, 1].", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Application of a deep RL algorithm to a real world problem\n- It's interesting to see that the learned policy applies techniques also developed by human experts to solve the task (parallel actions)\n\nWeaknesses:\n- Very incremental", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The work is heavily inspired by [12] in terms of the problem to be solved and the methodology applied. The main differences seem to be the use of TD3 instead of double dueling DQN, action space scaling to handle the multiple orders of magnitude, and an additional randomized parameter of the domain. In my opinion, this is too incremental to recommend it for acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Although I'm quite sure the paper uses the same simulator as in [12], it is never actually introduced\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "0c8AX066cYM", "reviewer": "Reviewer_PgwD", "summary": "This paper is concerned with the problem of automatically aligning elements for optical setups. The proposed method is to train a policy to adjust the components positions based on the current interferometer output. The policy is trained in simulation and then deployed on a real setup. The paper's contributions are an extension to the prior simulated benchmark for this task and exponential action rescaling to perform coarse-to-fine adjustments. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clear in how it describes the problem setting and method. The experiments are well reported, and the human baseline shows the speed advantage of the RL agent. The introduction discusses installations that consist of \"hundreds or thousands of elements\", but the task presented only has 9. Will this method scale to larger settings? RL is notoriously bad at scaling to large action spaces without some sort of modularity or hierarchy. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this is an interesting paper with potential impact for optics practitioners, CoRL seems like the wrong venue for this work.\n\n------\nUpdate after author response.\nI agree the work is well done, and I am fine with it being accepted to CoRL. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "My concern with this paper is whether CoRL is an appropriate venue. The work is interesting and seems significant for the optics community, but I do not have the qualifications to judge whether the problem setting and results are realistic and useful. From an RL perspective, the method is sound but not particularly novel. I could see the action rescaling being applicable to other robotics problems (fine grained manipulation tasks) but this is a small part of the paper. Overall, I would prefer for this paper to be submitted to an optics venue. "}}]

Author rebuttal:

Meta-review: The reviewers didn't fully agree on the scores for the paper, but after the discussion with the authors, most of them felt that their points were adequately addressed in the rebuttal. I particularly encourage the authors to further clarify how this work is more than an incremental improvement over a baseline in [12] to address the remaining comments but I recommend this paper for acceptance at CoRL.

Decision: Accept (Poster)