## 3 Results and Methodology
A one step MDP is shown diagrammatically in Figure 1.  
Figure 3 represents the phase of the wavefront centered at the intersection after the circular mask has been applied. It is interesting to notice that the diameter of the wavefront is only sampled by ten pixels in the simulation. On the right hand side of the same figure, the PSF image of that part of the wavefront is showed. In a physical setup, that can be achieved by placing a microlens array centered at each segment junction. The PSF is obtained for each of the four wavelengths. The mask would be only needed during training. The goal is to find the best possible policy \(\pi_{\theta}(u|s)\) such that the final expected reward is maximized. The policy gives a probability distribution over actions that the piston actuators can take from a given state. It can be represented with a three layer convolutional neural network where \(\theta\) is the set of parameters to be tuned. Each convolutional layer has 16 filters with weights and ReLU activation function. The sizes of the filters are \(7\times 7,5\times 5,3\times 3\) at each layer respectively. The depth of each filter matches the previous activation depth. A trainable bias parameter for each filter is also considered in the network. A fully connected layer is placed at the end to compute the final scores. The output of the network defines the mean of a bivariate probability normal distribution over actions. This mean is the action that is more likely to achieve the highest long-term reward from the current state, according to the agent experience. An action sampled from that distribution comprises two length components to be commanded to both pistons, \(A\) and \(B\). Sampling the action from the normal distribution rather than selecting the mean predicted by the network allows the agent to explore nearby actions that might end up being a better option than the prediction itself. Since the absolute piston positions are unknown, the actions represent the relative piston movement from initial to final state. The quantity to be optimized is called utility and it can be expressed mathematically by the following manner:  
\[\max_{\theta}U(\theta)=\max_{\theta}P(s_{t},u_{t};\theta)R(s_{t},u_{t}), \tag{1}\]  
where \(P(s_{t},u_{t};\theta)\) is the probability of the state and the action under a particular policy. And \(R(s_{t},u_{t})\) is the reward. The gradient of the utility of the policy parameters \(U(\theta)\) can be approximated with an empirical estimate for \(m\) samples [17]:  
\[\
abla_{\theta}U(\theta)\approx\frac{1}{m}\sum_{i=1}^{m}\
abla_{\theta}\log \pi_{\theta}(u_{t}^{(i)}|s_{t}^{(i)})\cdot R(s_{t}^{(i)},u_{t}^{(i)}) \tag{2}\]  
where \(m\) is the number of trials used in the estimation of the gradient. Figure 3: (**a**) Wavefront at the intersection once a circular mask has been applied. It contains a portion of the wavefront from each of the three segments. (**b**) PSF of the previous circular wavefront at the intersection. It is recorded at the focus plane behind the microlens. A Gaussian model is used to describe a stochastic policy over the continuous action space. The mean of the Gaussian is where the agent thinks that lies the action that is more likely to give the highest long-term expected reward from the current state. The variance of the Gaussian quantifies the uncertainty about that prediction. Since the random policy happens to be Gaussian, the form of the gradient of the log-probability turns out to be:  
\[\
abla_{\theta}\log\pi_{\theta}(u_{t}\mid s_{t})=\frac{u_{t}-\mu_{\theta}(s)}{ \sigma^{2}}\
abla_{\theta}\mu_{\theta}(s_{t}). \tag{3}\]  
The CNN returns two single scalar values for each diffraction image that it takes as an input. The mean of the bivariate distribution \(\mu_{\theta}\) is precisely the output of the convolutional network. The gradient can be computed with respect to its parameters \(\theta\) through backpropagation in the usual way. The variance of the distribution \(\sigma^{2}\) is fixed to a small value.