## 5 Results
### PPO Algorithm in Semi-Dynamic Environment  
The experimental setup assigned for a quasi-static environment consists of a 64-actuator action space, a 2 \(\times\) 2 pixel observation space, and a Strehl ratio reward function. To extend  
Figure 10: Power distribution on a focal plane (**left**) before proposed AO, (**upper right**) after the implementation of random actions through the PPO algorithm in the initial episodes, (**middle right**) after the utilization of the Shack–Hartmann wavefront sensor, (**lower right**) after the application of the PPO algorithm following a sequence of episodes.  
and validate our algorithmic framework for dynamic environments that better approximate real-world scenarios, such as the Earth's turbulent atmosphere, it is necessary to refine the RL environment to address specific challenges posed by non-static conditions.  
Before transitioning to the dynamic environment, it is essential to adjust the initial weights of the agents to facilitate more effective and efficient learning. To accomplish this, we have employed a semi-dynamic environment, as explained in Section 3.1. This semi-dynamic environment employs the reward function \(r_{2}\) in (2), replacing the Strehl ratio in (1) based on the discussion in Section 3.4.  
In the conducted experiment, each iteration involves 100 episodes of a randomly selected quasi-static environment, with each episode consisting of 20 time steps. The results of our experiment on a semi-dynamic environment using a quasi-static configuration with the PPO algorithm are presented in Figure 11 (red). This shows that the configuration used for a quasi-static environment is insufficient for learning a policy within a semi-dynamic environment, reaching a mean value of 20% coupling efficiency. Based on this poor performance, we hypothesize that partial observability and the curse of dimensionality can be significant factors. Specifically, the observation space is limited to a low dimension of \(2\times 2\) pixels, which may result in inadequate information gathered from the environment, leading to partial observability. Additionally, the utilization of a high-dimensional action space can cause an exponential rise in computational effort and lead to the curse of dimensionality [60]. To address these challenges within our existing setup, we reduced the action space dimension by employing the first six modes (second radial orders) of Zernike polynomials and increased the observation-space dimension to a \(5\times 5\) pixel photodetector (config 2). Furthermore, considering the time dependency, we applied the frame stacking technique [59] using a sequence of three frames, obtaining the results in Figure 11 (blue). Hyperparameter optimization has been performed for this experiment.  
By increasing the dimension of the observation space to a \(5\times 5\) pixel photodetector and reducing the action space to a first six modes (second radial orders) of Zernike polynomials, we can achieve a total power after single-mode fiber-coupling efficiency of 36%. In the semi-dynamic environment, it is noticeable that the Shack-Hartmann reaches a mean coupling efficiency of 48%. Part of the discrepancy could be attributed to the fact that the Shack-Hartmann wavefront sensor with 12 lenslets across the a  
Figure 11: Comparison between two configurations in atmospheric turbulence of \(D/r_{0}=3.33\) in semi-dynamic environment. Config. 1: observation space of \(2\times 2\) pixels, action space of 64 actuators. Config. 2: observation space of \(5\times 5\) pixels, action space of first 6 modes (2 nd radial orders) of Zernike polynomials. The reward function is Equation (2).  
128 \(\times\) 128 pixels camera for adjustment, in contrast to our use of a 5 \(\times\) 5 pixel photodetector for calculating the reward value.