# On the Fairness of Internet Congestion Control over WiFi with Deep Reinforcement Learning
## 4 System Model of Proposed TCP-Switching Mechanism
### Model Interpretation
#### 4.1.1 Proposed DRL-Based CCA Switching  
Our DRL framework utilizes a Deep Q-Network (DQN) to dynamically select and switch CCAs based on real-time network conditions. DQN was chosen over other DRL algorithms, such as DDPG, Double DQN, and DeepSARSA due to its superior performance in fairness analysis, as highlighted by [53], and its effectiveness in handling large discrete action spaces, which is relevant for CCA selection [54]. Furthermore, [55] demonstrated the ability of DQN to manage complex decision-making tasks, further supporting its suitability for our research. State Space (S)  
The state space captures key network metrics influencing the performance of CCA, including throughput, RTT, loss rate, and jitter. These parameters allow the DQN to assess and adjust the CCA based on real-time conditions. Figure 2: A high-level view of our model design with different components and their interactions. Action Space (A)  
The action space consists of selecting between different CCAs, such as _Cubic_, _BBR_, or _PCC_, where each action represents a potential CCA switch. Reward Function (R)  
The reward function is designed to improve network performance by addressing disparities in metrics and promoting beneficial CCA switches. For simplicity, we use  
\[R(s_{t},a_{t})=\frac{1}{2}\log(\text{Throughput}(t))-\frac{1}{4}\log(\text{ Latency}(t))-\frac{1}{4}\log(\text{LossRate}(t)),\]  
but the constants can be tuned to encourage actions that emphasize throughput and reduce RTT and the loss rate differently. Q-Function (Q)  
The Q function estimates the expected cumulative reward for each pair of state actions, which guides the selection of the optimal CCA:  
\[Q(s_{t},a_{t})=\mathbb{E}\bigg{[}R(s_{t},a_{t})+\gamma\max_{a^{\prime}}Q(s_{t+ 1},a^{\prime})\bigg{]}\]  
with \(\gamma=0.95\) balancing immediate and future rewards. This function helps determine actions that maximize long-term network performance. DRL trains a neural network to solve problems to optimize our proposed approach, predicting and responding to congestion indicators, effectively using the available bandwidth, and switching to the optimal CCA. Due to current dependencies on user-space libraries, DRL implementations must run in the user space (not in the kernel). The proposed DRL-Based TCP Switching Algorithm is shown in Algorithm 1. Given time and resource constraints, network environment complexity, data preprocessing needs, and real-time modeling challenges, we implemented offline DRL as the sole method for switching CCAs in this paper. The process was as follows. _Data Collection:_ Historical network traffic data were gathered using tools like iperf3 and Wireshark, capturing metrics such as throughput, packet loss, RTT, and congestion levels. _Offline Training:_ The DRL model was trained offline with the collected data, learning to identify network behavior patterns and select suitable CCAs. Validation: The model was validated with separate datasets to ensure its effectiveness and generalization to unseen network conditions, fine-tuning parameters as needed. _Validation_ The DRL model was validated with separate datasets to ensure its effectiveness and generalization to unseen network conditions, fine-tuning parameters as needed. _Realtime Monitoring:_ The continuous monitoring of real-time network metrics was conducted to update network condition variables. These metrics were inputted into the trained DRL model to predict the optimal CCA for current network conditions. _Algorithm Selection Policy:_ A policy was developed based on the DRL model's decisions, determining CCA selection based on observed network conditions. _Simulation:_ The DRL-based congestion control mechanism was simulated in an offline environment, evaluating performance and the ability to mitigate unfairness. _Optimization:_ The DRL model and selection policy were refined based on simulation results and validation feedback, enhancing decision-making accuracy and robustness.