# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### WFQ Continual-DRL Algorithm
In this approach, the EWC method is added to preserve performance on previous tasks during continual learning. For a batch of sampled transitions \((s_{\mathbf{t}},a_{\mathbf{t}},r_{t+1},s_{t+1})\), the target value is computed as:  
\[y_{k}=r_{k+1}+\gamma(1-d_{k})\big{[}min_{\theta}\overline{Q}_{\theta_{1}}(s_{k +1},a_{k+1}),\overline{Q}_{\theta_{2}}(s_{k+1},a_{k+1})\big{]}-\alpha log\ \pi_{\phi}(a_{k+1}\ |\ s_{k+1}) \tag{9}\]  
where \(r_{k+1}\) is a reward function, \(\gamma\) is discount factor, \(min_{\theta}\overline{Q}_{\theta_{1}}(s_{k+1},a_{k+1}),\overline{Q}_{\theta_ {2}}(s_{k+1},a_{k+1})]\) represent minimum of two target Q-networks, the algorithm reduces the risk of overestimation by selecting the smaller value from the two targets. The \(\alpha log\pi_{\phi}(a_{k+1}\ |\ s_{k+1})\) function, \(\alpha\) is a temperature coefficient that controls the trade off between exploration and exploitation. \(log\pi_{\phi}(a_{k+1}\ |\ s_{k+1})\) is the probability of taking the next action from the next state observed. For two critic networks, the critic loss function along with computing an EWC regularization term is the given by:  
\[J_{Q}=\frac{1}{m}\sum_{k=1}^{m}\frac{1}{2}\ [y_{k}-Q_{\theta}(\,s_{k}|a_{k})]^{2 }+\sum_{i}\frac{\lambda_{EWC}}{2}\ F_{i}(\theta_{i}-\theta_{i}{}^{*})^{2} \tag{10}\]  
where \(y_{k}\) is target value, \(Q_{\theta}(s_{k}\,|\,a_{k})\) represents the predication value, which estimates future reward action from current state \(s_{\mathbf{t}}\). \(\lambda_{EWC}\) is a hyperparameter that controls the strength of penalty. A higher value indicates that the actor is more constrained to preserve close to previous knowledge. \(F_{i}(\theta_{i}-\theta_{i}{}^{*})^{2}\) function to compute how much the current critic network parameters deviate from previously learned parameters, using FIM to take important weight. The updated actor network is represented as follows:  
\[J_{\pi}\ =\frac{1}{m}\sum_{k=1}^{m}\ \big{[}\alpha log\pi_{\phi}(a_{k}\ |\ s_{k})-Q_{\theta}(\,(\,s_{k}|a_{k}))\big{]}+\ \sum_{i}\frac{\lambda_{EWC}}{2}\ F_{i}(\phi_{i}-\phi_{i}{}^{*})^{2} \tag{11}\]  
where \(\alpha log\pi_{\phi}(a_{k}\ |\ s_{k})\) represents the entropy term and actor sampled actions from probability distribution leading to high Q values (expected reward), where \(Q_{\theta}(s_{k}\,|\,a_{k})\) represents the predication value, which estimates future reward. \(\sum\limits_{i}\frac{\lambda_{EWC}}{2}\ F_{i}(\phi_{i}-\phi_{i}{}^{*})^{2}\) is a regulation term to compute how much the current critic network parameters deviate from previously learned parameters, and FIM means to take important weight. To perform gradient updates for actor and critic networks:  
\[\phi\leftarrow\phi-\eta_{\pi}\
abla_{\phi}\ J_{\pi} \tag{12}\]  
Figure 4.0  
Figure 4.1  
Slowly update the parameters of target Q networks to prevent instability caused by rapidly changing targets using a soft update:  
Figure 4.2  
Figure 4.3The structure of the WFQ continual-DRL algorithm as illustrated in Figure 4.4  
The FIM is computed after finishing training on a task before starting on another task, including computing and storing the FIM. It is estimated by computing the squared gradients of the loss for the policy and critic networks with respect to their parameters, averaged over samples from the replay buffer size, where \(\theta_{i}^{*}\) is the optimal parameter set after learning task A for future EWC loss computation. In our paper, the FIM was computed every 30 episodes, and then the parameters \((\theta_{i}^{*},\mathcal{O}_{i}^{*})\) and FIM were stored until the next 30 episodes, in an algorithm of SAC with EWC. This interval was chosen based on empirical test observe improve stability and adaptability; updating too often restricts the agent from learn new information, while updating too rarely weakens the term of regularization. In the WFQ continual-DRL framework, the 30-episode interval provides an effective tradeoff between preserving previous knowledge and learning new tasks.