## 3 Wavefront Sensorless AO-RL Environment
### Reward Function  
There are two available choices for the reward function in the RL environment: the Strehl ratio and the combination of single-mode fiber total power with the Structural Similarity Index (SSIM).  
In the former, the reward function is calculated using the Strehl ratio of the optical system, defined as the ratio of the normalized peak intensity of the point spread function to the peak intensity of the ideal point spread function without aberrations. A high Strehl ratio implies a high degree of wavefront correction, where a focused beam of light resembles an Airy disk and is approximately proportional to the amount of light that can be coupled into a fiber [54]. It is considered an approximation because, for optimal coupling into an optical fiber, a focused beam should resemble a Gaussian profile. As proposed by [PERSON] [55], the Strehl ratio \(r_{1}\) of systems with a circular pupil is expressed in terms of the variance of the phase aberration across the pupil  
\[r_{1}=e^{-\sigma_{\Phi}^{2}} \tag{1}\]  
where \(\sigma_{\Phi}^{2}\) represents the variance of the phase aberration.  
Although the Strehl ratio is commonly used in AO, its computation is not ideal for the applications in this work since it requires a point spread function and a camera, whereas our system explicitly avoids reliance on a camera (specifically an InGaAs camera), in favor of a cheaper, compact, and faster solution. The calculation of the Strehl ratio requires sufficiently long exposures for speckle variation and there are alternative methods that utilize fewer  
Figure 4: Focal plane profile, (**left**) continuous, (**right**) discretized.  
array sizes. Additionally, the Strehl ratio is not an ideal metric for coupling into a single-mode fiber, as the correlation between the single-mode fiber-coupling efficiency and the Strehl ratio is reduced at high Strehl ratios, restricting its applicability as reward function to lower values. At higher Strehl ratios, it is limited by the mode mismatch between a focused flat top beam and the Gaussian mode in a single-mode fiber [54; 56]. Therefore, a different reward function has been added to the RL environment. This additional reward combines the total power of single-mode fiber and the Structural Similarity Index (SSIM). Although the total power of a single-mode fiber is a crucial indicator of its ability to concentrate power, due to its very small diameter, it is not particularly useful for exploration as it misses a significant portion of observations. Therefore, relying solely on the total power of a single-mode fiber as a reward function is inadequate. To address this, we combine it with SSIM as a criterion to measure the similarity between two arrays on a photodetector, like the Strehl ratio. The first array serves as a reference, which represents the power distribution when there is no wavefront distortion, and the second array represents the current power distribution. In other words, single-mode fiber total power measures the performance of the exit of the single-mode fiber, while SSIM measures the performance of the entrance of the single-mode fiber. This reward, named \(r_{2}\), is formally defined by  
\[r_{2}=\beta P_{SMF}+(1-\beta)\frac{(2\mu_{curr}n_{ref}+c_{1})(2\sigma_{curr, ref}+c_{2})}{(\mu_{curr}^{2}+\mu_{ref}^{2}+c_{1})(\sigma_{curr}^{2}+\sigma_{ ref}^{2}+c_{2})} \tag{2}\]  
where \(\beta\) is a constant that controls the weighting between single-mode fiber total power (\(P_{SMF}\)) and SSIM. The SSIM equation involves the variables \((\mu_{ref},\sigma_{ref})\), which are the mean and variance of the power distribution of the reference array when there is no distortion, and \((\mu_{curr},\sigma_{curr})\) as the mean and variance of the current power distribution. \(c_{1}\) and \(c_{2}\) are two constants to stabilize the division when the denominator is small, avoiding numerical singularities.