# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### WFQ Continual-DRL Algorithm
This process, known as reparameterization sampling of the action value, allows for effective policy optimization as follows [24]:  
\[a_{t}=f_{\varnothing}(\varepsilon_{t};s_{t})\qquad\quad\text{Where }\varepsilon\ \sim\ N\ (0,1) \tag{8}\]  
where \(a_{t}\) represents the action, \(s_{t}\) represents the current state, and \(\varepsilon\) is the noise vector from the Gaussian distribution. This function \(a_{t}=f_{\varnothing}(\varepsilon_{t};s_{t})\) means that internal noise addition allows the agent to produce stochastic actions necessary for exploration while trusting the training process is stable and efficient. The action reductions are set to be within the required range by applying the ReLU activation function, which is practical to eliminate negative values where only positive actions are valid, which is particularly useful in settings (bandwidth weights) for WFQ scheduling. The policy network effectively explores continuous action while a stochastic process balances between exploration and exploitation. The approach presented in this study combines SAC with EWC. The goal is to integrate the EWC regulation mechanism into SAC to maintain stability across states while preserving knowledge learned from the first task through training. As usual, the end of the task includes calculating and storing the FIM and the parameters \((\theta_{t}^{*},\mathcal{O}_{i}^{*})\). When reserving a new state, the training is initialized in that state. At the parameter update stage, the term representing the EWC loss function is added to the SAC losses. After completing the task, the Fisher matrix is updated, and the network parameters are saved. The combined losses that ensure the performance of the previous task are held constant while learning a new task, allowing the network to output the action [23]. In the WFQ continual-DRL algorithm, the significance of each model parameter is computed and evaluated. When the algorithm begins, the model is set to evaluation mode, and a dictionary is initialized to store Fisher information values for each parameter. Following this, a batch of experiences is sampled from the replay buffer, and the gradient is computed based on the loss function associated with either the actor or the critic network. In the actor network, the loss is defined as the mean of the predicted actions, whereas in the critic network, it is the mean squared error between the predicted Q-values and the target Q-values. After performing back-propagation, the Fisher information for each parameter is updated by averaging the squared gradients across the sampled batch. These values are then detached from the computation stored in a dictionary, to ensure that the learned values of the critical parameters are retained when the model is adapted to new tasks. The optimal parameter values are also stored to facilitate the EWC penalty calculations during training. Figure 4: Structure of the neural network integrating SAC with EWC for twelve queues. EWC characterizes itself from other continual learning methods, such as Gradient Episodic Memory (GEM) and Learning without Forgetting (LwF), by focusing on simplicity, memory efficiency, and biological probability. Unlike GEM, which requires high memory and computational costs, it stores data from previous tasks and performs config on previous outputs or parameters, resulting in additional complexity and storage requirements. In contrast, EWC introduces a simple regularization based on the Fisher Information Matrix (FIM) to preserve important weights without requiring the preservation of old data or models, which makes EWC efficient, lightweight, and perfect for continual learning in dynamic resource settings, such as DRL-based queuing and scheduling systems [31]. The process of choosing the optimal value for \(\lambda_{EWC}\) is conducted in two steps. In the first step, six distinct ranges are defined using the random function, and each one is explored independently.