# Optimizing Weighted Fair Queuing with Deep Reinforcement Learning for Dynamic Bandwidth Allocation
## 4 Proposed Model
### Continual Reinforcement Learning Framework for WFQ  
The process allocates bandwidth weights in the WFQ system, which operates under a control policy that takes into consideration the system's dynamic nature. The interaction between the algorithm and multi-queue systems is shown in Figure 2.  
After observing the system's status, represented in the form of queue lengths, the agent decides to allocate bandwidth weights as an action. Until the next decision is taken, the queueing system will use the allocated bandwidth and send rewards back to the agent, allowing it to adjust its policy accordingly. This study focuses on dynamic control and slot dependence, where the decision time is t seconds. Since the state may vary significantly within a specific time slot, it is more practical to implement it in the engineering field, but this also makes it more challenging to determine the optimal policy. By integrating continual learning with DRL, the agent learns optimal policies, retains knowledge from past experiences, and adapts to evolving network conditions without catastrophic forgetting. This approach enables the system to efficiently generalize across different traffic patterns, thus reducing the need for comprehensive retraining. A continual RL-based agent can enhance bandwidth allocation over time by leveraging past knowledge while adapting to previously unnoticed traffic behaviors, leading to optimal stability, scalability, and long-term performance in dynamic networks. For a continuous state space and a continuous action space, the continual DRL situations are as follows:  
State: \(q_{i}\) is defined as the number of packets that are either serviced by the server \(i\) or are waiting in \(queue_{i}\). \(q_{i}=(q_{1},q_{2},\ldots,q_{n}),i=1,\ 2,\ldots,\ n\) also represents the state of the system. As it captures queue lengths, the continual DRL method is practical for use in telecommunications networks. If we assume \(q_{i}\leq K_{i}\), then each queue has a limited buffer. \(s=\{q=(q_{1},q_{2},\ldots,q_{n})\in N_{n}:q_{i}\leq K_{i}\}\) represents the state space. However, this is enhanced with historical states, thus allowing the model to be better generalized and to retain past traffic patterns. In the state of a multi-queue system, the input to the neural network is a vector representing the current queue lengths of the multi-queue queues as \(s=\{q=(q_{1},q_{2},\ldots,q_{n})\in N_{n}:q_{i}\leq K_{i}\}\). To confirm stable and efficient learning, each queue length was normalized before being input into the neural networks. The normalization approach involves logarithmic scaling \(\sum_{i=1}^{n}\left\lceil log_{2}(K_{i}+1)\right\rceil\), which corresponds to the current queue length of \(q_{i}\). This transformation reduces the impact of large values, ensuring the state input remains within a manageable range while preserving relative differences and stabilizing the training process. This normalized state vector is then provided into fully connected layers of the neural networks for policy and Critic network estimation, encouraging the agent to learn QoS-aware scheduling decisions effectively. Action: At the decision-making step, the bandwidth weights for each queue are allocated, and this defines the action \(a\). \(a\) represents the action space, where \(\{a=(w_{1},w_{2},\ldotsldots,w_{n})\in\mathbb{R}_{+}^{n}:\sum_{i=1}^{n}w_{i}=1\}\). In continual DRL, the model leverages history-learned policies, adapting to new traffic patterns without relearning from scratch and improving the efficiency of decision-making. Reward: In queueing systems, minimizing the average delay is equivalent to reducing the average queue length. It is assumed that for every packet in the queue, the cost per unit of time is equal to the cost of charging for the delay. The reward function is as follows:  
\[\mathbf{r}(\mathbf{q},\mathbf{a})=-\sum_{i=1}^{n}(\mathbf{h}_{i}\mathbf{q}_{i}+\mathbf{c}_{i}\mathbf{l}_{ i}) \tag{1}\]  
where the symbol \(l_{i}\) represents the number of lost packets in a given unit of time.