## 4 Methodology and Algorithms Training
### Soft Actor-Critic (SAC)  
SAC is an off-policy actor-critic algorithm based on a maximum entropy RL framework. It is particularly useful in complex and stochastic environments, such as wavefront sensorless AO systems. SAC uses a deep neural network to approximate the actor and critics. The actor component of the algorithm is tasked with maximizing the expected return while promoting exploration through random actions rather than becoming trapped in suboptimal policies. On the other hand, the critic component is responsible for estimating the Q-function of a given state-action pair. The Q-function provides feedback for improving the policy by adjusting the actions that the agent takes in each state to maximize the expected sum of future rewards [35].  
SAC has shown promising results in various domains. However, one major drawback is its sensitivity to the choice of temperature \(\alpha_{t}\) and intuitively selected target entropy parameters. These parameters play a crucial role in the algorithm's performance and their selection can significantly affect the outcome. To address this, Reference [57] proposed a method of automatic gradient-based temperature tuning by matching the expected entropy \(\log\pi_{t}^{*}(a_{t}|s_{t};\alpha_{t})\) to a target entropy value \(\tilde{\mathcal{H}}\) at time \(t\):  
\[\alpha_{t}^{*}=\arg\min_{\alpha_{t}}\mathbb{E}_{a_{t}\sim\pi_{t}^{*}}\big{[}- \alpha_{t}\log\pi_{t}^{*}(a_{t}|s_{t};\alpha_{t})-\alpha_{t}\tilde{\mathcal{H }}\big{]}, \tag{3}\]  
where the temperature \(\alpha_{t}\) controls the stochasticity of the optimal policy, and \(a_{t}\) and \(s_{t}\) are the action and the state at time \(t\), respectively.  
In our preliminary assessment, we evaluate the SAC using three different temperature settings: fixed, learned, and semi-learned temperatures. Under the fixed temperature condition, we achieved the best reward by setting the constant value \(\alpha_{t}=0.4\). This resulted in a mean reward of 52.63% with a standard deviation of 14.64% at the end of the training process. For the learned temperature condition, we optimized it using a learning rate \(\alpha_{t}\)-\(lr=10^{-1}\) within the learning rates ranging from \(10^{-6}\) to \(5\times 10^{-1}\). The semi-learned temperature condition was also optimized using the same learning rate, along with a minimum value of \(\alpha_{t}=0.4\), resulting in improved performance compared to the entire range of learning rates and minimum \(\alpha_{t}\) values between 0 and 1. Figure 5 illustrates our results, showing that the fixed and semi-learned temperature settings show faster learning rates compared to the purely learned setting, and that semi-learned converges to the best policy. As a result, we have chosen to use the semi-learned temperature configuration in all subsequent experiments.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Hyperparameter** & **SAC** & **DDPG** & **PPO** \\ \hline Buffer size & 128 & 256 & - \\ Actor—\(lr\) & \(5\times 10^{-4}\) & \(5\times 10^{-5}\) & \(1\times 10^{-2}\) \\ \(\text{Critic--}lr\) & \(1\times 10^{-2}\) & \(1\times 10^{-2}\) & \(5\times 10^{-6}\) \\ Actor—hidden dim. & 150 & 250 & 150 \\ \(\text{Critic--}\)hidden dim. & 80 & 65 & 50 \\ \(\text{Clipping }c\) & - & - & 0.35 \\ \(\text{Temp. }\alpha_{t}\text{-}lr\) & \(1\times 10^{-1}\) & - & - \\ \(\text{Temp. }\alpha_{t}\text{-}\text{min limit}\) & 0.4 & - & - \\ \(\text{No episodes per iteration}\) & 1 & 2 & 2 \\ \(\text{No updates per iteration}\) & 20 & 20 & 20 \\ \(\text{Polyak }(\rho)\) & 0.99 & 0.99 & - \\ \(\text{Discount }(\gamma)\) & 0.95 & 0.95 & 0.95 \\ \(\text{Reward scaling}\) & No & Mean–std & No \\ \(\text{learned }\alpha_{t}\) & Semi & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hyperparameters and corresponding values in quasi-static environment.