## 3 Results and Methodology  
Reinforcement learning is a subfield of machine learning in which an algorithm gets feedback from the environment in the form of reward or punishment. According to this approach, the optimization problem becomes how to design an agent that acts optimally to get the highest long-term expected reward from the environment. Or alternatively, in the specific domain of optical phasing, given a diffraction image of an intersection of three adjacent segments, how to move piston actuators \(A\) and \(B\) in Figure 2 in order to get the maximum Strehl ratio of the wavefront at the intersection. In a reinforcement learning setup, it is helpful to model the problem as a Markov Decision Process (MDP) in which the following tuple of elements \(\{\mathcal{S},\mathcal{U},\mathcal{P},\mathcal{R},\gamma\}\) have to be defined. The set of states the agent can be at, \(\mathcal{S}\), are the diffraction images of the intersection of the segments taken at four different wavelengths. It is shown with dashed line in Figure 2. It is a \(24\times 24\) pixels image around the center of the intersection. It is assumed that tip-tilt values have been restored for each segment in a previous stage, hence only piston values remain. The actions, \(U\), are the set of all possible piston movements that can be commanded to segments \(A\) and \(B\). These are pairs of values with units of length. They are limited to a distance equivalent to \(\pm\lambda_{0}/2\), being \(\lambda_{0}\) the reference wavelength. Since only piston steps in the range \(\pm\lambda_{0}/2\) are considered, it is assumed that those action values should be enough to correct the piston misalignment completely. It might be the case that after movements have been applied to both \(A\) and \(B\) segments pistons, the final piston error among them might lie outside the \(\lambda_{0}\) limit. Using four wavelengths helps to distinguish states outside the ambiguity range. The element \(P\) is the transition probability. It is the probability of ending at a final state \(s_{t+1}\) conditioned on both, an initial state \(s_{t}\) and a certain action, \(u_{t}\). They are the stochastic rules that govern the physics of the environment, i.e., the probability distribution over states that can be reached from initial state when a certain action is taken. It corresponds to the dynamics of the system and it is implicitly learned by the algorithm through experience. The reward signal, \(R\), is obtained when taking action \(u_{t}\) at state \(s_{t}\). The sum of the maximum intensity values of the PSF images for each wavelength at the intersection is used as the reward. This value is proportional to the Strehl ratio. In order to produce the intersection PSF, a circular mask is first placed at the center of the junction of the three segments. That mask isolates a circular portion of the wavefront. The diameter of the mask is equivalent to \(0.2\,\mathrm{m}\) at the pupil. The reward is deterministic in the simulations. However, it can be considered stochastic in a general RL setup, as long as the expected long-term reward defines the agent final goal. The algorithm aims to maximize the expected long-term value of these reward outcomes. And last, the discount factor \(\gamma\) is used in a sequential task to indicate how valuable it is to achieve the rewards as soon as possible. In the one step MDP case, this hyperparameter is set to \begin{table}
\begin{tabular}{l l} \hline \hline
**Parameter** & **Value** \\ \hline Focal length of telescope & \(f=170\,\mathrm{m}\) \\ Defocus distance & \(d=9\,\mathrm{m}\) \\ Physical size of segment & \(h=0.95\,\mathrm{m}\) \\ Pupil scale at the detector & \(p=0.019\,\mathrm{m}/\mathrm{pixel}\) \\ Fried parameter & \(r_{0}(@500 nm)=0.2\,\mathrm{m}\) \\ Largest wavelength & \(\lambda_{0}=700\,\mathrm{nm}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Simulation parameters. zero. This means that the agent only cares about the immediate reward.