# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### WFQ Continual-DRL Algorithm
EWC identifies important neural network weights from previous queue patterns and penalizes changes to these weights during training on new patterns, thereby decreasing catastrophic forgetting. This process helps preserve performance consistency across varying queue loads and quick convergence in the early training episodes. As a result, the continual-DRL agent adapts efficiently to changing traffic intensities while ensuring stable scheduling [23]. In this study, a simulation was conducted for twelve queues as an enhancement. The parameter tuning process of the neural network for integration of SAC-EWC into the WFQ continual-DRL algorithm involved expanding the number of queues to twelve, which increased the state as input to the neural network and the action space as output, making the scheduling problem more dynamic and complex. This adjustment demands a more capable model to allocate bandwidth effectively across a larger number of queues. However, increasing the number of neurons in the network to 128 enhances the model's ability to approximate complex policies, resulting in improved decision-making. When extended to twelve queues, the input defined as state will be increased and output representing action will increase. Increasing the number of neurons from 64 to 128 to enhance the model's learning allows the network to capture more complex patterns in the data training, which is mainly valuable in dynamic environments such as multi-queue systems. This adjustment aims to improve performance and reach higher prediction accuracy. This enables the model to capture more complex pattern in the data through training data and avoid underfitting. Underfitting occurs when the model is too simple to capture the underlying patterns in the training data, resulting in inadequate and poor learning. This results in increased errors and invariably low rewards, even during training. A larger replay buffer enables the agent to retain past experiences for a longer period, thereby enhancing training stability by providing a diverse range of samples during training. Additionally, increasing the batch size to 256 contributes to more stable updates by averaging gradients over a larger set of experiences, thereby reducing variance in training. These enhancements improve learning efficiency while providing excellent stability in dynamic bandwidth allocation scenarios. The structure of the neural network integrating SAC with EWC for twelve queues is shown in Figure 4.  
In the SAC structure of the critic network, the use of target critics stabilizes the learning process for both critic networks \(Q_{1}\) and \(Q_{2}\). The \(\pi\) network produces the \(\mu\) and \(\sigma\) of the distribution. In this approach, the action is displayed as a probability distribution (Gaussian distribution). The Gaussian distribution is represented as follows:  
\[f\left(a|\sigma,\ \mu\right)=\frac{1}{\sigma\sqrt{2\pi}}exp^{-\frac{1}{2}\left( \frac{a-\pi}{\sigma}\right)^{2}} \tag{7}\]  
where \(\mu\), \(\sigma\) represents the input to the Gaussian distribution formula predicted by the policy network. \(a\) is the output from a Gaussian distribution. Whenever the action stays in the area around the mean, the action will be better [30]. Figure 3: Structure of the neural network integrating SAC with EWC for eight queues. The action was sampled randomly from a probability distribution defined by \(\mu\), \(\sigma\), which is known as a stochastic action or stochastic policy. A random noise \(\varepsilon\) is generated to confirm exploration, the value of the noise \(\varepsilon\) is drawn from a random distribution represented by \(N(0,1)\). The output action was sampled from \(a=\mu+\sigma\varepsilon\), which internally is computed in Gaussian distribution.