## 7 Conclusions and Future Work  
In this paper, we conducted a comprehensive analysis of various AQM algorithms, including our implementation of a preliminary architecture of L4S in the FreeBSD network stack. We conducted extensive research. For this purpose, we created two network scenarios, one with high bandwidth and the other with low bandwidth. We assessed their performance in a scenario where multiple TCP connections compete in the same bottleneck. Our findings revealed that L4S exhibits very low RT in low-bandwidth conditions, nearly a second faster than its competitors while maintaining a high level of throughput. Furthermore, we have developed an innovative hybrid A3C-L4S approach to enhance network performance, specifically for latency-sensitive applications. This approach leverages the multi-agent and asynchronous nature of A3C to enhance exploration and adaptability while minimizing resource consumption on the routers. It is a novel addition to network management, offering a unique solution to enhancing network performance. The A3C model can converge properly, learning the intricate relationship between the drop probability and the queuing delay, further enhancing the adaptability of our approach. The potential of leveraging the L4S architecture in FreeBSD with a deep-reinforcement learning model like A3C to optimize the gaps in our network infrastructure is immense. This combination can make our network more suitable for a specific type of latency-sensitive data. However, there is a need to optimize both the TCP and AQM sides of L4S to respond faster to real-time network traffic, in which A3C plays a significant role. We are also exploring the energy efficiency of the final system and how we can adjust the model parameters to consume fewer resources while maintaining an enhanced performance. In our future work, we aim to dedicate more time to fine-tuning the A3C model by training it on larger datasets. Additionally, we intend to deploy the model directly into the kernel and utilize an online learning approach for training. This would enable the model to continuously learn and adapt in real time. The communication between kernel space and user space would also introduce some delay, which would be negligible under typical  
Figure 38: Case 1: (ECN enabled) Congestion Window Scenario 2 with Bandwidth = 1 Mbps, Delay = 20 ms. (**a**) FQ-CoDel (_cvnd_); (**b**) FQ-PIE (_cvnd_); (**c**) L4S (_cvnd_). Figure 39: Case 2:(ECN disabled) Congestion Window Scenario 2 with Bandwidth = 1 Mbps, Delay = 20 ms. (**a**) FQ-CoDel (_cvnd_); (**b**) FQ-PIE (_cvnd_); (**c**) L4S (_cvnd_). conditions, but we have to explore the other scenarios, especially under high load. The perfect solution would be directly coding the machine learning algorithm in the kernel, which has difficulties. Moreover, we intend to explore using the UDP protocol in our research. UDP is well-suited for applications where low latency and high throughput are prioritized, and reliability is not critical. By leveraging UDP, we can cater to latency-sensitive applications that require efficient data transmission. Building on the prior research in [38], which focused on implementing an ML-based multi-path TCP, we see potential in collaborating with source control protocols like TCP and UDP, along with network devices such as routers and switches [38]. Exploring the interactions and optimizations between these components could lead to significant network performance and efficiency advancements. By combining the strengths of machine learning, protocols like TCP and UDP, and network devices, we can enhance the overall performance, reliability, and adaptability of future network systems. Conceptualization, J.K. and S.R.P. ; Methodology, J.K. and S.R.P. ; Software, J.K.; Validation, D.S. ; Formal analysis, D.S. ; Investigation, D.S. and S.R.P. ; Data curation, D.S. ; Writing--original draft, D.S. ; Writing--review & editing, J.K. and S.R.P. ; Visualization, D.S. ; Supervision, J.K. and S.R.P. ; Project administration, S.R.P. ; Funding acquisition, J.K. and S.R.P.