# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### WFQ Continual-DRL Algorithm
Figure 6 illustrates the steps of the adaptive algorithm of WFQ continual-DRL. Figure 4.5 The structure of the WFQ continual-DRL algorithm. Figure 4.6 The diagram illustrates the steps of the WFQ continual-DRL algorithm. The details of the WFQ continual-DRL algorithm are given in Algorithm 1. ```
1:Initialization: Initialize the parameters of network, policy \(\pi\), critics networks \(Q_{\theta 1},Q_{\theta 2}\), and target critics \(\overline{Q}_{\theta 1},\overline{Q}_{\theta 2}\), i.e., \(\emptyset\), \(\theta_{1}\), \(\theta_{2}\), respectively. Initialize Fisher Information Matrix \(F_{\theta}\) and Optimal Parameters \(\theta^{*}\), \(\emptyset^{*}\). 2:For each episode do: Initialize environment and initialize state \(s_{0}\). Initialize episode reward to 0. 3:For each environment step do: Select action: \(\mathbf{a}_{t}\sim\pi_{\mathcal{O}}(\mathbf{a}_{t}|\mathbf{s}_{t})\). Execute action in environment and observe next state \((\mathbf{s}_{t+1})\), reward \((\mathbf{r}_{t+1})\), and done signal. Store transition: \((\mathbf{s}_{t},\mathbf{a}_{t},\mathbf{r}_{t+1},\mathbf{s}_{t+1})\) in replay buffer. Store parameter \(\theta^{*}\), and compute Fisher Information Matrix \(\mathbf{F}_{\theta}\) every \(X\) episode. 4:Update Parameters: Randomly sample minibatch of m transitions \((\mathbf{s}_{k},\mathbf{a}_{k},\mathbf{r}_{k+1},\mathbf{s}_{k+1})\) from replay buffer. Compute target Q-value using the minimum of two target Q-networks: \(\mathbf{y}_{k}=\mathbf{\tau}_{k+1}+\gamma(1-\mathbf{a}_{k})\left[min_{\theta}\overline{Q} _{\theta_{i}}(\mathbf{s}_{k+1},\mathbf{a}_{k+1}),\overline{Q}_{\theta_{i}}(\mathbf{s}_{k +1},\mathbf{a}_{k+1})\right]-\mathbf{a}log\pi_{\varphi}(\mathbf{a}_{k+1}\mid\mathbf{s}_{k+1})\) Update Critic networks \(Q_{\theta 1},Q_{\theta 2}\) minimizing loss: \(J_{Q}=\frac{1}{m}\sum\limits_{k=1}^{m}\frac{1}{2}[\mathbf{z}_{k}-Q_{\theta}(\mathbf{s }_{1}\mathbf{a}_{k})]^{2}+\sum\limits_{t}\frac{\Delta E_{\text{WC}}}{2}\mathbf{F}_{i} \Big{(}\theta_{i}-\theta_{i}^{*}\Big{)}^{2}\) Where \(E_{\text{WC}}\) is the EWC regularization strength. Update Actor network \(\pi_{\mathcal{O}}\) by minimizing the loss: \(J_{\pi}=\frac{1}{m}\sum\limits_{k=1}^{m}\left[\mathbf{a}log\pi_{\varphi}(\mathbf{a}_{ k}\mid\mathbf{s}_{k})-Q_{\theta}((\mathbf{s}_{k}\mathbf{1}\mathbf{a}_{k}))\right]+\sum\limits_{t} \frac{\Delta E_{\text{WC}}}{2}\mathbf{F}_{i}(\mathbf{\varphi}_{i}-\mathbf{\varphi}_{i}^{ *})^{2}\) Perform gradient updates: \(\mathbf{\phi}\leftarrow\mathbf{\phi}-\eta_{\pi}\
abla_{\mathbf{\phi}}\mathbf{\eta}_{\pi}\). \(\theta_{1}\leftarrow\theta_{1}-\eta_{\mathcal{O}_{1}}\
abla_{\theta_{i}}J_{Q}\). \(\theta_{2}\leftarrow\theta_{2}-\eta_{\mathcal{O}_{2}}\
abla_{\theta_{i}}J_{Q}\). Soft update target networks: \(\mathbf{\bar{\theta}}_{1}\leftarrow\mathbf{\tau}\theta_{1}-(1-\mathbf{\tau})\mathbf{\bar{ \theta}}_{1}\). \(\mathbf{\bar{\theta}}_{2}\leftarrow\mathbf{\tau}\theta_{2}-(1-\mathbf{\tau})\mathbf{\bar{\theta}}_ {2}\). 5:Compute Fisher Information Matrix (FIM) Every X Episodes: Compute gradient-based Fisher Information Matrix using log probabilities for the actor and critic losses: \(\mathbf{F}_{\theta}=\mathbb{E}\Big{[}\Big{(}\frac{dlog\pi_{\mathcal{O}}(\mathbf{a}_{t })}{\theta_{\theta}}\Big{)}^{2}\Big{]}\) Store optimal parameters \(\theta^{*}\) and Fisher Information Matrix \(\mathbf{F}_{\theta}\). ```  
**Algorithm 1** The WFQ continual-DRL Algorithm (integration SAC with EWC)