# Learning-Based Online QoE Optimization in Multi-Agent Video Streaming
## 2 Related Work  
**Multi-Agent Reinforcement Learning:** In the past, the Multi-agent Reinforcement Learning (MARL) technique [17] has been discussed for scenarios where all of the agents make decisions individually to achieve a global optimal. Existing works include coordinated reinforcement learning [18], which coordinates both the action selections and the parameter updates between users; sparse cooperative Q-learning [19], which allows agents to jointly solve a problem when the global coordination requirements are available; ref. [20], which uses the max-plus algorithm as the elimination algorithm of the coordination graph; ref. [21], which compares multiple known structural abstractions to improve the scalability;and ref. [22], which automatically expands an agent's state space when the convergence is lacking. Apart from the standard Q-learning [23] and policy gradient [24] algorithms, there is rich literature on meta-heuristic algorithms for reinforcement learning. In [25], an ant-colony optimization method for swarm reinforcement learning is provided, which improves empirically over the Q-learning-based methods by using parallel learning inspired by ant swarms. Building on biological inspired algorithms, ref. [26] provides a genetic algorithm to search for parameters for deep-reinforcement learning. In addition, the authors of [27] study a modification of ant colony optimization by considering \(\epsilon\)-greedy policies combined with Levy flight for random exploration in order to search for possible global optima. In addition, the authors of [28] consider a multi-period optimization using an ant-colony-optimization-inspired algorithm relaxation-induced neighborhood search algorithm for performing a search in large neighborhoods. Recently, along with the development of neural networks and deep learning, the deep-MARL [29] is proposed to resolve real-world problems with larger state spaces. With various aspects of deep-MARL researched, such as investigating the representational power of network architectures [30], applying deep-MARL with discrete-continuous hybrid action spaces [31], enhancing the experience selection mechanism [32], etc., real-world applications can be solved, including wireless sensor networks (WSN) routing [10], vehicle networks spectrum sharing [11], online ride-sourcing (driver-passenger paring) services [33; 34], video game playing [35], and linguist problems [29]. Comparing with existing work, our proposed solution in this paper focuses on optimizing inter-agent fairness objectives in reinforcement learning. **Video Streaming Optimization:** In order to improve the performance of data streaming, various techniques have been proposed. The mostly discussed method is Adaptive Bitrate (ABR) [36] streaming, which dynamically adjusts the streaming bitrate to reduce the stall time. The different algorithms include BBA [37], Bola [38], FastMPC [39], LBP [40], FastScan [36], and Pensieve [41]. In addition to ABR and bandwidth allocation considered in this paper, caching is also a popular technique to reduce the stall time and further improve QoE. Inspired by the LRU cache replacement policy, ref. [42] analyzes an alternative gLRU designed for video streaming application, and [PERSON] [7] proposes a Q-learning-based cache replacement policy to jointly optimize the hit ratio and stall time. Within an edge network environment, the placement of calculations will also affect the streaming performance, thus work [12] breaks the hierarchical service placement problem into sub-trees, and further solves it using Q-learning. For video streaming services still using Constant Bitrate (CBR) systems, ref. [43] proposed QUVE, which estimates the future network quality and controls video-encoding accordingly. The study in [44] considers maximizing QoE by optimizing the cache content in edge servers. This is different from our setup, where we consider caching chunks at client devices. Similar to us, ref. [45] also provides a bandwidth allocation strategy to maximize QoE.