## 4 Approach
### Implementation Details in the DRL-AST Method  
As illustrated in Figure 3, the overall architecture is composed of the following core components:  
* **GRU-Based Encoder (Common Module)**: a shared gated recurrent unit (GRU) encoder [57] is used to capture temporal dependencies in the input state sequence. At each timestep \(t\), it updates the hidden state \(h_{t}\) based on the previous hidden state \(h_{t-1}\). The resulting representation is fed into both the actor and critic networks. This shared structure helps extract temporal features and improves training stability. * **Actor (Policy Network)**: the actor network receives the encoded state \(h_{t}\) and passes it through fully connected layers with ReLU activation [58] and LayerNorm normalization [59]. The fully connected layers perform linear mappings from state features to  
Figure 2: Trajectory of states and actions within the VarDA W-ST environment under an actor-critic policy. the action space. ReLU introduces nonlinearity, and LayerNorm stabilizes training. The final layer outputs the mean and log standard deviation of a diagonal Gaussian distribution with dimension \(CK\). An action \(a_{t}=\mathbf{W}=[w_{1},w_{2},\ldots,w_{CK}]\) is then sampled from this distribution to produce the adaptive scaling vector. * **Critic (Value Network)**: the critic receives the encoded state \(h_{t}\) and a similar architecture with the actor. It outputs a scalar value \(V_{H}(s_{t})\), representing the expected cumulative reward from state \(s_{t}\). This value is used as a baseline to estimate the advantage \(\hat{A}_{t}\), which is essential for guiding policy improvement during training. * **Replay Buffer**: following each interaction, the agent receives the next state \(s_{t+1}\) and a scalar reward \(r_{t}=R(s_{t},a_{t})\), which reflects the improvement in assimilation quality resulting from taking action \(a_{t}\) in state \(s_{t}\). The experience tuple \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\) is temporarily stored in a replay buffer. During training, the agent samples mini-batches of these collected experiences to update the policy and value networks. This strategy improves sample efficiency and reduces the effects of temporal correlation in sequential data, thereby enhancing training stability. The actor network is updated by maximizing a clipped surrogate objective defined by the PPO algorithm. This formulation constrains the magnitude of policy updates, ensuring training stability. The surrogate loss is given by the following:  
\[\mathcal{L}^{clip}(\theta)=\sum_{(s_{t},a_{t})}\min\bigl{(}\text{rate}_{t}( \theta)\cdot\hat{A}_{t},\ \text{clip}(\text{rate}_{t}(\theta),1-\zeta,1+\zeta)\cdot\hat{A}_{t}\bigr{)} \tag{21}\]  
where \(rate_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta}(a_{t}|s_{t})}\) is the probability ratio between the current (\(\pi_{\theta}\)) and previous (\(\pi_{\theta^{t}}\)) policies for the action \(a_{t}\) taken in state \(s_{t}\). The advantage estimate \(\hat{A}_{t}\) quantifies how much better an action is compared to the critic's expected return and is derived using the GAE method. The clipping function restricts policy updates to a trust region defined by \(\zeta\), which is typically set to 0.2, preventing large deviations in policy updates. The critic network, parameterized by \(\mu\), is trained to minimize the discrepancy between the predicted value \(V_{\mu}(s_{t})\) and the target return \(\hat{R}_{t}=\hat{A}_{t}+V_{\mu}(s_{t})\). The advantage is computed as a weighted sum of temporal-difference (TD) errors [51], where \(\lambda\) is a decay factor controlling the bias-variance trade-off:  
\[\hat{A}_{t}=\delta_{t}+(\gamma A)\delta_{t+1}+(\gamma A)^{2}\delta_{t+2}+\cdots \tag{22}\]  
where \(\delta_{t}\) denotes the TD error, which quantifies the deviation between the predicted value and the bootstrapped return from the next state. Figure 3: The integrated architecture of the actor–critic network and the PPO-based training process in our VarDA DRL-AST method.