## 3 Results and Methodology
However, it can also be parameterized and learned from the experience. The action to be taken by the agent is sampled from that distribution. Selecting actions randomly allows the agent to explore new optimal actions while exploiting the current policy. Now, with the expression of the gradient, the optimum value of the parameters can be found with the gradient ascent. Adam algorithm, a variant of the latter, was used in the simulations [18]. On the other hand, a second convolutional network is used as a function approximator to represent the value function [19], \(V_{\phi}^{\pi}(s_{t})\). It takes as input the state i.e., the diffraction image of the intersection, and returns the expected reward from that state under the current policy \(\pi_{\theta}\). The set of parameters \(\phi\) that better approximate the value function are learned in a supervised manner from experience. The value function can be used as a state dependent baseline to reduce the variance of the algorithm [20]. Using the advantage estimator \(\hat{A}=r_{t}-V_{\phi}^{\pi}(s_{t})\) rather than simply the reward in the Equation (2) makes the learning process more stable. Using the baseline function makes the variance decrease without changing the expectation of the gradient. The capture range defines the interval of possible piston jump values that the agent is trained to detect and act upon. Capture ranges considered here are suitable for fine tuning the piston positions after a previous coarse piston alignment stage has been carried out. The intersection has two piston jumps to segments \(A\) and \(B\). Every combination of piston step values generates a distinctive diffraction pattern. The broader the capture range, the more patterns the agent is required to recognize to be able to perform the proper action. Algorithm 1 shows the complete learning sequence. The initial state is the diffraction image of an intersection with piston jumps from bottom left segment to segments \(A\) and \(B\). They can be any random value within the capture range, see Figure 2 for clarity. The policy network takes the diffraction image as input and predicts the mean of a Gaussian over the continuous action space. Then an action is sampled from that distribution in step 4. Next, state and reward are recorded once the action has been performed on the environment in step 5. The advantage is computed in step 6 and it quantifies how good or bad that reward is with respect to the average reward achieved on that state. The expected reward from a state \(s_{t}\) following the policy \(\pi_{\theta}\) is given by the value function. In order to be self  
of \([-\lambda_{0}/2,+\lambda_{0}/2]\). On the other hand, the dashed line shows the learning process for piston steps within the range \([-\lambda_{0}/4,+\lambda_{0}/4]\). ```
1:Initialize policy and baseline parameters: \(\theta,\phi\)
2:whiletrainingdo
3:\(s_{t}\)\(\triangleright\) Initialize state with random piston values
4:\(u_{t}\sim\pi_{\theta}(s_{t})\)\(\triangleright\) Sample next action from current policy
5:\(s_{t+1},r_{t}\gets u_{t}\)\(\triangleright\) Apply piston displacement and get reward from PSF
6:\(\hat{A}=r_{t}-V_{\phi}^{\pi}(s_{t})\)\(\triangleright\) Compute advantage between value function and immediate reward
7:\(\phi\leftarrow\min_{\phi}\left\|r_{t}+V_{\phi}^{\pi}(s_{t+1})-V_{\phi}^{\pi} (s_{t})\right\|^{2}\)\(\triangleright\) Update value function parameters
8:\(\theta\leftarrow\
abla_{\theta}\log\pi_{\theta}(u_{t}|s_{t})\hat{A}\)\(\triangleright\) Update policy with gradient
9:endwhile
```  
**Algorithm 1** Policy gradients with value function baseline  
It is interesting to visualize how accurately the network rectifies the piston misalignment during training. It is important to point out that the RL agent does not predict the piston mse error. It rather learns how to minimize it through the optimization of the PSF. In a real scenario, it would not be possible to know the ground truth piston misalignment in the initial state that gave rise to the diffraction image.