## 3 Methodology
### Detailed Framework Architecture  
The state encoder in our framework utilizes a transformer-based architecture with an embedding dimension of 256, featuring 8 attention heads across 4 transformer layers. We applied a dropout rate of 0.1 and layer normalization after each attention layer to prevent overfitting and to stabilize training. The dual-stream architecture processes two distinct types of actions: immediate actions for rapid threat response (e.g., blocking malicious IPs and terminating suspicious processes) and strategic actions for long-term defense (e.g., security policy updates and system-wide patches). Figure 1 illustrates how these action streams are processed through their respective networks. The immediate response network \(\pi_{I}\) employs ReLU activation functions with batch normalization after each fully connected layer to maintain optimal data flow. The strategic response network \(\pi_{S}\) combines LSTM processing for temporal pattern recognition with ReLU-activated fully connected layers to capture both the immediate threat indicators and long-term attack patterns. This dual-stream design enables the system to simultaneously handle urgent security incidents while maintaining a strategic defense posture. ```
0: Training dataset \(\mathcal{D}\), learning rates \(\alpha_{1},\alpha_{2}\), and replay buffer size \(N\)
0: Trained models \(\pi_{I}\), \(\pi_{S}\), and state encoder \(f_{\theta}\)
1: Initialize state encoder \(f_{\theta}\), immediate policy \(\pi_{I}\), and strategic policy \(\pi_{S}\)
2: Initialize replay buffer \(\mathcal{B}\leftarrow\{\}\)
3: Initialize target networks \(\pi^{\prime}_{I},\pi^{\prime}_{S}\) with \(\pi_{I},\pi_{S}\)
4:for episode \(=1\) to \(M\)do
5: Initialize episode buffer \(\mathcal{E}\leftarrow\{\}\)
6: Sample initial state \(s_{0}\) from \(\mathcal{D}\)
7:for\(t=0\) to \(T-1\)do
8:\(h_{t}\gets f_{\theta}(s_{t})\) [Encode current state]
9:\(a_{t}\leftarrow\text{AttentionScore}(h_{t},\mathcal{E})\) [Temporal attention]
10:\(s^{aug}_{t}\leftarrow\text{TemporalAggregation}(h_{t},\alpha_{t})\)
11:\(a_{t}\leftarrow\pi_{I}(s^{aug}_{t})\) [Immediate action]
12:\(a_{S}\leftarrow\pi_{S}(s^{aug}_{1:t})\) [Strategic action]
13:\(\beta_{1},\beta_{2}\leftarrow\text{AdaptiveWeighting}(s^{aug}_{t})\)
14:\(a_{t}\leftarrow\text{CombineActions}(a_{t},a_{S},\beta_{1},\beta_{2})\)
15: Execute \(a_{t}\), observe \(r_{t}\), \(s_{t+1}\)
16: Store \((s_{t},a_{t},r_{t},s_{t+1})\) in \(\mathcal{E}\)
17: Update \(\mathcal{B}\) with \(\mathcal{E}\)
18:if\(|\mathcal{B}|\geq\text{batch}_{\_size}\)then
19: Sample mini-batch \(\mathcal{M}\) from \(\mathcal{B}\)
20: Calculate TD error for \(\pi_{I}\):
21:\(\delta_{I}\gets r+\gamma\max_{\alpha^{\prime}}\pi^{\prime}_{I}(s_{t+1},a ^{\prime})-\pi_{I}(s_{t},a_{t})\)
22: Calculate strategic value for \(\pi_{S}\):
23:\(\delta_{S}\leftarrow\sum_{k=0}^{H}\gamma^{k}r_{t+k}-\pi_{S}(s_{1:t},a_{t})\)
24: Update \(\pi_{I}\) with \(\
abla_{\theta_{I}}(\delta^{2}_{I})\)
25: Update \(\pi_{S}\) with \(\
abla_{\theta_{S}}(\delta^{2}_{S})\)
26: Update state encoder:
27:\(L_{state}\leftarrow\text{StateConsistencyLoss}(f_{\theta})\)
28:\(L_{temporal}\leftarrow\text{TemporalCoherenceLoss}(f_{\theta})\)
29: Update \(f_{\theta}\) with \(\
abla_{\theta}(L_{state}+L_{temporal})\)
30:endif
31: Periodically update target networks \(\pi^{\prime}_{I},\pi^{\prime}_{S}\)
32:endfor
```  
**Algorithm 1** ARCS Training Procedure  
As for the implementation details, we designed the immediate response network \(\pi_{I}\) with a three-layer architecture. The network processes the state representation through sequential transformations:  
\[ \begin{split} h_{1}&=\text{ReLU}(W_{1}s_{t}+b_{1 })\\ h_{2}&=\text{ReLU}(W_{2}h_{1}+b_{2})\\ Q_{I}&=W_{3}h_{2}+b_{3},\end{split}  \tag{12}\]  
where the layers are dimensioned as [256; 128; 64], respectively, with batch normalization applied after each layer to maintain stable training dynamics. The strategic response network \(\pi_{S}\) incorporates temporal dependencies through an LSTM-based architecture:  
\[