## 3 Materials and Methods
### Model Architectures
#### 3.3.1 Deep Q-Network
```
Initialize experience replay buffer \(D\) to capacity \(N\) Initialize Q-network \(Q(s_{t},a_{t};\theta)\) with random weights \(\theta\) Initialize a target network \(\tilde{Q}\big{(}s_{t},a_{t};\tilde{\theta}\big{)}\) where \(\tilde{\theta}\ =\ \theta\) Set action space \(a\) = [0, 1, 2, 3, 4, 5] for six possible class labels For episode = 1, \(M\) do: Initialize state \(s_{1}\) from first observation of CFG features For step \(t\ =\ 1\) to \(T\) do: With probability \(\varepsilon\) select a random action \(a_{t}\)  or select action \(a_{t}\ =\ arg\ max_{a}Q(s_{t},a_{t};\theta)\) Execute action \(a_{t}\) on the environment (i.e., predict vulnerability type) Observe reward \(r_{t}\) and next state \(s_{t+1}\) Store state transition \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\) in replay buffer \(D\) Randomly sample minibatch of transition tuple \(\langle s_{t},a_{t},r_{t},s_{t+1}\rangle\) from \(D\) Set \(y_{i}\ =\ \ r_{i}+\gamma\max_{a^{\prime}}\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! actions that lead to lower return, until it arrives at the optimal policy. Under a normal policy gradient, it keeps new and old policies close in parameter space. However, since even minor variations in parameter space can result in significant performance differences, a single bad step can collapse the policy performance. This risk makes large step sizes particularly dangerous when applying vanilla policy gradients, thereby hurting the sample efficiency. To address this challenge, an improved PPO algorithm, firstly introduced by [PERSON] et al. [39] from OpenAI [40], refines policies by taking the largest step possible to enhance performance while adhering to a constraint that ensures the new policy remains sufficiently similar to the previous one. This paper implemented the clip version of PPO, which uses specialized clipping (positive or negative advantage) instead of KL-divergence in the objective function to remove incentives for the new policy to get far from the old policy. Let \(\pi_{\theta}\) denote a policy with parameter \(\theta\), and \(J(\pi_{\theta})\) denotes the expected finite-horizon undiscounted return of the policy. The gradient of \(J(\pi_{\theta})\) can be defined as:

\[\
abla_{\theta}J(\pi_{\theta})\ =\ \underset{\tau\sim\pi_{\theta}}{\sum} \Bigg{[}\underset{t=0}{\sum}\ \
abla_{\theta}log\pi_{\theta}(a_{t}|s_{t})A^{\pi_{\theta}}(s_{t},a_{t})\Bigg{]}, \tag{3}\]

where \(\tau\) is a trajectory and \(A^{\pi_{\theta}}\) is the advantage function for the current policy. PPO-clip updates policies via:

\[\theta_{k+1}\ =\ arg\underset{\theta}{\max}\underset{s,a\sim\pi_{\theta_{k}}}{ \underset{s,a\sim\pi_{\theta_{k}}}{\sum}}\left[L(s,a,\theta_{k},\ \theta)\right], \tag{4}\]

which takes multiple steps of minibatch size to maximize the objective. Here \(L\) is given by:

\[L(s,a,\theta_{k},\ \theta)\ =\ \min\bigg{(}\frac{\pi_{\theta}(a|s)}{\pi_{ \theta_{k}}(a|s)}A^{\pi_{\theta_{k}}}(s,a),\ g\big{(}\epsilon,\ A^{\pi_{ \theta_{k}}}(s,a)\big{)}\bigg{)}, \tag{5}\]

where

\[g(\epsilon,\ A)\ =\ \begin{cases}(1+\epsilon)A,\ \ A\geq 0\\ (1-\epsilon)A,\ \ A<0\end{cases}  \tag{6}\]

in which \(\epsilon\) represents a (small) hyperparameter that dictates the permissible deviation of the new policy from the old. To derive meaningful insights from this clipping setup, let us focus on a specific state-action pair \((s,a)\) and think of cases. * Advantage is positive: Suppose the advantage for that state-action pair is positive; its contribution to the objective function diminishes to: \[L(s,a,\theta_{k},\ \theta)\ =\ \min\bigg{(}\frac{\pi_{\theta}(a|s)}{\pi_{ \theta_{k}}(a|s)},\ (1+\epsilon)\bigg{)}A^{\pi_{\theta_{k}}}(s,a).\] (7) Since the advantage is positive, the objective will increase if the action becomes more likely, i.e., if \(\pi_{\theta}(a|s)\) increases. However, the min function in this expression imposes a cap on how much the objective can rise.