Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper is concerned with the problem of automatically aligning elements for optical setups. The proposed method is to train a policy to adjust the components positions based on the current interferometer output. The policy is trained in simulation and then deployed on a real setup. The paper's contributions are an extension to the prior simulated benchmark for this task and exponential action rescaling to perform coarse-to-fine adjustments. 
strengths_and_weaknesses: The paper is clear in how it describes the problem setting and method. The experiments are well reported, and the human baseline shows the speed advantage of the RL agent. The introduction discusses installations that consist of "hundreds or thousands of elements", but the task presented only has 9. Will this method scale to larger settings? RL is notoriously bad at scaling to large action spaces without some sort of modularity or hierarchy. 
summary_of_recommendation: While this is an interesting paper with potential impact for optics practitioners, CoRL seems like the wrong venue for this work.

------
Update after author response.
I agree the work is well done, and I am fine with it being accepted to CoRL. 
issues: My concern with this paper is whether CoRL is an appropriate venue. The work is interesting and seems significant for the optics community, but I do not have the qualifications to judge whether the problem setting and results are realistic and useful. From an RL perspective, the method is sound but not particularly novel. I could see the action rescaling being applicable to other robotics problems (fine grained manipulation tasks) but this is a small part of the paper. Overall, I would prefer for this paper to be submitted to an optics venue.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes using reinforcement learning to train a robot for aligning an interferometer. They extend setting to a richer observation and action spaces that are closer to the real-world. To work with continuous action spaces, TD3 algorithm is used. Additional novel components include predicting the actions in the log space to handle the large range of actions, and adding phase noise for domain randomization.
strengths_and_weaknesses: Strengths:
- The experimental evaluation, particularly on the real system, is convincing, and shows that the proposed approach can outperform the baseline, and a human. Further, the method discovers parallel movement, which is interesting.

Weaknesses:
- The section on action rescaling (line 216) was a little confusing — specifically, it wasn't clear how this is related to the exploration noise. It seems like exploration noise is reduced by decaying the standard deviation of the Gaussian noise (line 230).
- It might be helpful to have a discussion of whether the proposed system is specific to the setup considered, or whether it could generalize to other related systems.
summary_of_recommendation: The paper extends prior work to a practical real-world system, and can potentially be used to implement reinforcement learning in other real-world systems.
issues: - Clarify how action rescaling is related to exploration noise.
- Discuss how the method may / may not be used in other real-world applications.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The goal of this paper is to use reinforcement learning to solve a practical problem in optics. In particulars, the authors use RL to align an optical interferometer. The problem seems perfectly suited for RL, as there is a clear observation space, action space and reward function, and the problem can be simulated with relative ease. The authors follow a sim-to-real approach, training their agent entirely in simulation, and then transferring it to a real world system. To ensure effective sim-to-real transfer, the authors make use of domain randomization, in which they both randomize the dynamics (a little), and the observations (a lot). 

The closest prior work is that of Interferobot [1], on which this work closely builds. The primary improvement over Interferobot is the use of a continuous action space and an exponential action scaling scheme that allows the agent to take actions across two orders of magnitude. The authors use TD3 as their RL algorithm. The paper re-uses some domain randomizations from prior work, and also introduces some new ones. 

[1] D. Sorokin, A. Ulanov, E. Sazhina, and A. Lvovsky. Interferobot: aligning an optical interferometer by a reinforcement learning agent. In Advances in Neural Information Processing Systems. 
strengths_and_weaknesses: Strengths
- Practical solution to a real world problem. This is not something that you can say for most papers submitted to CoRL (as most robotics is limited to toy-settings, not real world ones). 
- Clear presentation 
- Outperforms prior work by a decent margin

Weaknesses
- Somewhat incremental taking into account the prior work (Interferobot) 
summary_of_recommendation: The paper is a bit incremental taking into account prior work (as I mentioned above), which is why I am keeping my recommendation at a weak accept. 

After author response: I found the author response to be quite compelling, so I am upgrading my rating to a strong accept. While I am not an optics expert, the works appears to be tackling a much harder problem than the previous work, and while each of the change might be incremental by itself, there are quite a few such changes that come together to solve a real world problem, which makes this paper a strong candidate for acceptance to CoRL (in my view).
issues: I'll leave some typos here: 

Typos
Line 91: “becomes” → “has become”
Line 105: “mororized” → “motorized” 
Line 107: “required” → “requires”
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an application of Deep RL to the problem of aligning an optical interferometer. In simulation, TD3 is used to learn a controller that is able to manipulate different parts of the interferometer such as mirror orientations and lens locations. Domain randomization is applied for succesful transfer from simulation to the actual physical system. The resulting policy is able to control the interferometer parts more precisely and quicker than a human expert and another deep RL baseline algorithm. Since the action space on the actual system can span several orders of magnitude, the method applies an exponential scaling to the actions after sampling actions from the interval [-1, 1].
strengths_and_weaknesses: Strengths:
- Application of a deep RL algorithm to a real world problem
- It's interesting to see that the learned policy applies techniques also developed by human experts to solve the task (parallel actions)

Weaknesses:
- Very incremental
summary_of_recommendation: The work is heavily inspired by [12] in terms of the problem to be solved and the methodology applied. The main differences seem to be the use of TD3 instead of double dueling DQN, action space scaling to handle the multiple orders of magnitude, and an additional randomized parameter of the domain. In my opinion, this is too incremental to recommend it for acceptance.
issues: - Although I'm quite sure the paper uses the same simulator as in [12], it is never actually introduced
review_title: null

Meta-review: The reviewers didn't fully agree on the scores for the paper, but after the discussion with the authors, most of them felt that their points were adequately addressed in the rebuttal. I particularly encourage the authors to further clarify how this work is more than an incremental improvement over a baseline in [12] to address the remaining comments but I recommend this paper for acceptance at CoRL.

Decision: Accept (Poster)