## 4 Approach
### Actor-Critic Framework and Proximal Policy Optimization (PPO) Algorithm  
This study introduces a DRL algorithm based on the PPO method under the actor-critic architecture [51], aiming to learn spatio-temporal scaling strategies for the \(\mathbf{B}\)-matrix in a VarDA system. The VarDA system is formulated as an interactive environment, where an agent learns optimal actions (scaling strategies) through trial-and-error interactions. The actor-critic framework jointly learns a stochastic policy (actor) and a state value function (critic), thereby improving decision quality and training efficiency. Specifically, we adopt policy gradient methods to optimize the policy directly instead of indirectly deriving it from an approximated value function [52]. The objective is to maximize the expected cumulative reward by updating the policy via gradient ascent, as defined by the following expression:  
\[\
abla_{\theta}J(\pi_{\theta})=\underset{\tau\sim\pi_{\theta}(\tau)}{\mathbb{ E}}\Big{[}\sum_{t=0}^{T}\
abla_{\theta}log\pi_{\theta}(a_{t}\mid s_{t})R(\tau) \Big{]} \tag{20}\]  
where \(\theta\) denotes the parameters of the policy network. In the customized environment VarDA W-ST, a trajectory \(\tau=s_{1},a_{1},s_{2},a_{2},\ldots,s_{T},a_{T}\) (as illustrated in Figure 2) denotes a sequence of states and actions executed by the agent under policy \(\pi_{\theta}\). The probability of generating a trajectory under the current policy is denoted by \(\pi_{\theta}(\tau)\), and \(\pi_{\theta}(a_{t}\mid s_{t})\)represents the probability of selecting action \(a_{t}\) in state \(s_{t}\). The cumulative reward associated with the trajectory is defined as \(R(\tau)=\sum_{t=0}^{T}r(s_{t},a_{t})\).  
**Why Choose PPO?** In this study, we adopt policy gradient methods to optimize the policy directly, offering a substantial improvement in computational efficiency over traditional approaches that first approximate the value function before deriving the policy indirectly. Modern policy gradient algorithms can be broadly classified into two categories: on-policy methods, such as Advantage Actor-Critic (A2C), Trust Region Policy Optimization (TRPO), and PPO; and off-policy methods, including Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), and Soft Actor-Critic (SAC) [53]. Off-policy methods, such as DDPG leverage an experience replay buffer to enhance sample efficiency by reusing past interactions. However, they are highly sensitive to hyperparameter tuning, such as exploration noise and the soft target update rate. Poorly tuned hyperparameters can result in unstable training dynamics and even policy collapse [52]. In contrast, PPO improves training stability through the incorporation of importance sampling and a clipping mechanism. Specifically, PPO utilizes trajectory data collected from an earlier policy to compute the probability ratio between new and old policies. By applying a clipping constraint, PPO prevents excessively large policy updates, thereby mitigating instability and enhancing robustness [54]. Additionally, PPO integrates generalized advantage estimation (GAE) and reward normalization, reducing sensitivity to hyperparameter variations and ensuring adaptability across diverse tasks [53; 55; 56]. Given these advantages, we adopt PPO as the policy optimization method, achieving a balance between training stability and computational efficiency while minimizing the need for extensive hyperparameter tuning and ultimately improving the reproducibility of our experiments.