# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### WFQ Continual-DRL Algorithm  
This study integrated continual learning techniques with a DRL framework for weighted fair queues within the proposed WFQ continual-DRL algorithm to dynamically allocate bandwidths. In this approach, an agent explores the dynamics of the queueingsystem through periodical, recursive interactions within the architecture of WFQ continual DRL. The agent gains new insights about the environment's surroundings and exploits them to obtain high rewards. The agent also retains and transfers history-learned policies to improve decision-making in dynamic traffic environments and accumulates knowledge over time. One challenge is that the huge buffer properties of the WFQ system and several queues contribute to the curse of dimensionality. To address this issue, the WFQ continual-DRL algorithm is employed, providing the control policy estimates as input to the neural network parameters. As input for neural networks, the system's state is transformed into a binary vector, thus eliminating the need for storage space. This enables us to receive normalized state information for various buffers, preventing a gradient explosion problem when computing policy gradients and accessing normalized queue lengths. This improvement enhances the numerical stability, interpretability, and learning efficiency of the system, where \(K_{i}\) a vary widely. This ensures that the transformed values are more fluent for the WFQ continual DRL algorithm. \(\sum_{t=1}^{n}\left\lceil log_{2}(K_{i}+1)\right\rceil\) is the dimension of s. The off-policy WFQ continual-DRL algorithm is also used to achieve high levels of data efficiency. In essence, the off-policy technique breaks the correlation between consecutive data using a replay buffer, and can, therefore, achieve more data usage efficiency by repeatedly utilizing historical data. After each investigation stage, transition data \((s_{t},a_{t},r_{t+1},s_{t+1})\) are gathered and saved in the replay buffer to be sampled later and used to modify the control strategy. In addition to the sampled updating policy strategy, the updated parameters of the neural network are used to compute the FIMs, to ensure that the most critical parameters from the past are preserved. Continual learning is used to overcome catastrophic forgetting. The SAC procedure is a sophisticated technique that optimizes the expected reward and policy entropy simultaneously. The efficiency of the investigation is improved by enhancing the entropy and optimizing the expected reward by maximizing the entropy of the policy. SAC enhances the efficiency of self-exploration and has been shown to achieve stable performance in continuous action spaces. Stable performance in a continuous action space has also been demonstrated. SAC involves employing a policy network \(\pi\), and a critic network Q to approximate a stochastic policy. The structure of the neural network integration of SAC with EWC for eight queues is illustrated in Figure 3.  
Two critic networks are utilized in the SAC algorithm to mitigate the overestimation bias commonly found in DRL. To evaluate the Q-values corresponding to state-action pairs, the critics, represented as \(Q_{1}\) and \(Q_{2}\), are accountable, while their target networks, \(\overline{Q}_{1}\) and \(\overline{Q}_{2}\), facilitate stable learning through soft updates. The control calculates the target Q-value by selecting the minimum estimate from the two critics to reduce the likelihood of overestimation during training. The critics are updated using the MSE, which is the loss between the predicted Q-values and the computed target [24]. In the proposed algorithm of the WFQ continual-DRL algorithm, the integration of continual learning, represented by EWC, with the DRL algorithm, particularly in the WFQ system, enhances the agent's ability to retain optimal scheduling knowledge across various traffic scenarios.