## 3 Related Work
### Reinforcement Learning in Edge Computing
In particular, they analyze the impact of the limited vision of RL agents on achievable performance and they propose an extension algorithm to allow learning agents to query an upper-level agent with global knowledge scope. Paper [22] contributes to the online resource allocation problem in edge computing. The proposal combines online resource allocation with reinforcement learning in edge computing. The research objective is to provide social welfare and improve resource utilization. The authors define a dynamic resource allocation problem where users request resources from an edge computing server. The problem is addressed by a policy gradient algorithm. Performance is evaluated experimentally. Paper [23] deals with the design of an RL-based management solution for orchestrating resources at the network edge. Model-free optimization is used to make control decisions. In particular, the authors consider service migration for cost reduction, approached by a DQN algorithm. This algorithm is used to capture mobility regularities and to balance the VM and data migration costs. Mobile edge computing is also considered by [24]. The authors propose an RL-based optimization framework, to drive resource allocation in wireless applications. In particular, they consider Q-learning-based and deep reinforcement learning techniques. In [25], the authors investigate the mobile edge caching problem given limited storage capacity. They define a caching strategy based on multi-agent deep reinforcement learning. On the proposed approach, edge servers operate as agents and select files to be cached. Also, the the cold start problem is considered, handled by a heuristic approach. Some appreciable results are found through experiments, in terms of latency. Still on the subject of resource management, some proposals refer to the offloading of service requests when the computing and/or storage resources are insufficient to satisfy them. For example, paper [26] is a survey that focuses on the application of RL and DRL techniques in edge computing systems for computation offloading. The authors classifyresearch papers according to use cases, architectures, objectives, and algorithms. Paper [27] is a survey of RL-based scheduling schemes in mobile edge computing networks. In particular, it reviews and analyzes RL-based approaches to offloading scheduling, content caching, and communication. The applications considered include industrial automation, autonomous driving, robotics, VR, and AR. Paper [28] deals with the offloading operations in mobile edge computing due to hardware bottlenecks. The authors propose a deep reinforcement learning algorithm to control the offloading scheduler. The algorithm is based on the so-called double dueling deep Q-network (D3 QN). The experimental results show improved battery exploitation and drop rate in comparison with a baseline scheme. Paper [29] deals with applications in the internet of things (IoT), including virtual reality, mobile games, and wearable devices. Such applications are assumed to be deployed using edge computing. The authors analyze computation offloading mechanisms by formulating a stochastic game. They define a multi-agent reinforcement learning architecture to solve the stochastic game based on multi-agent Q-learning. Energy efficiency is demonstrated by experiments. Some proposals have taken into account, in regard to the deployment of applications at the edge, very latency-sensitive applications in highly dynamic scenarios. For example, paper [30] shows a path planning method for a UAV-mounted mobile edge computing system. The global objective of the research is to maximize the quality of experience of mobile users through deep Q-learning. The performance metrics considered are service rate, average risk, path length, and delay, determined by simulations. Paper [31] deals with a multi-UAV-assisted MEC system, where different UAVs cooperate to serve IoTDs.