# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### Continual Reinforcement Learning Framework for WFQ
In addition, the packet loss \(l_{i}\) for \(i,2,3,\ldots,n,q_{i}\), which includes the number of packets lost for queue length represented as a vector, where every packet loss results in a penalty \(c_{i}\) represented as a scalar. We also assume that each packet in \(q_{i}\) represents the current length of \(q_{i}\) a vector number of packets generates a holding cost \(h_{i}\) per unit of time, \(h_{i}\) the holding cost per packet in \(q_{i}\) representing a scalar equivalent to imposing a penalty for its delay. A counted continual DRL scheme includes  
Figure 2: Interaction between the WFQ continual-DRL algorithm and the multi-queueing system. a mechanism to evaluate how well past learning is transferred to the new requirements, thus confirming the adaptation of requirements and optimal dynamic bandwidth allocation. Each element in the vector of queue length \(q_{i}\) is defined as the number of packets currently held in that queue. The packet loss is also described as a vector, where each element corresponds to the number of packets lost in the queue. In this paper, we incorporate EWC into RL to prevent catastrophic forgetting when switching between diverse tasks, thereby ensuring that the agent retains knowledge of previous tasks while optimizing the current task. By assessing which parameters are critical for previously learned tasks and constraining their updates during training on new tasks, this approach helps maintain the performance on earlier tasks. This enables knowledge retention and adaptation to new traffic situations without catastrophic forgetting, thereby improving generalization to previously unnoticed workloads. * EWC regulation: The loss function is modified by adding the EWC penalty to ensure stability when learning a new policy. The regularization term is added for each policy and critic losses, confirming that changes to parameters considered important for previous tasks are penalized. Thus, the final loss function employed in backpropagation is: \[\mathbf{\mathcal{L}_{total}}=\mathbf{\mathcal{L}_{RL}}+\sum\limits_{i}\frac{\lambda_{ EWC}}{2}\mathbf{F_{i}}(\theta_{i}-{\theta_{i}}^{*})^{2}\] (2) where \(\mathcal{L}_{RL}\) is the standard loss function, \(\sum\limits_{i}\frac{\lambda_{EWC}}{2}\). \(F_{i}(\theta_{i}-{\theta_{i}}^{*})^{2}\) is the term representing the EWC regulation function, \(\lambda_{EWC}\) reflects the significance of the old task compared to the new task, \(\theta_{i}^{*}\) is the optimal parameter from the previous task, and \(i\) tags each parameter. \(F_{i}\) is the Fisher Information Matrix (FIM), which measures the importance of each parameter. The aim of EWC is to maintain the network parameters relative to the learned parameters of both the first and second tasks. When the model is applied to a third task, this constraint can be enforced either with two separate penalties or by combining them into one, since the sum of two quadratic penalties is also a quadratic penalty [23]. * FIM: The FIM provides essential parameters for preserving primary tasks. It is an essential concept that reflects the amount of information an observable random variable carries about an unknown parameter upon which the probability of that random variable depends. It assesses the sensitivity of the network's performance in terms of parameter variations \(\theta_{i}^{*}\) and it quantifies the sensitivity of the model's parameters to modifications in the loss function, capturing the importance of each parameter for previously learned tasks. In practice, the FIM is estimated by computing the squared gradients of the loss for the policy and critic networks with respect to their parameters, averaged over samples from the replay buffer.