# Intelligent End-Edge Computation Offloading Based on Lyapunov-Guided Deep Reinforcement Learning
## 1 Introduction
Nevertheless, the inherent dynamism and randomness of industrial wireless networks pose significant challenges in gathering comprehensive system information necessary for accurate modeling [6]. This becomes particularly problematic in large-scale industrial settings, where a vast amount of intricately coupled system data results in an unwieldy state space, significantly impeding the efficiency of reinforcement learning. To overcome this bottleneck, data-driven deep reinforcement learning (DRL), as an innovative scheme, opens up new avenues for the computation offloading problems. DRL skillfully combines deep learning and reinforcement learning, using deep neural networks (DNN) to directly map from the environment state to the optimal action in order to maximize the long-term reward. This process is done automatically through continuous interaction with the environment. This integration not only alleviates the computational complexity of the problem but also endows the approach with the capability to autonomously learn from past experiences, bypassing the need for manually labeled training data, thus greatly facilitating the feasibility of its real-time online application. This is significant for efficiently generating computation offloading strategies in dynamic environments. However, existing works have given limited consideration to the long-term network computation rate problem under long-term stability of the system. To this end, this paper proposes an intelligent computation offloading algorithm based on Lyapunov optimization and DRL for the end-edge task offloading problem under long-term stability constraints to maximize the long-term network computation rate. In the case of rapidly changing channel conditions and dynamic task arrivals, our algorithm can maximize the long-term network computation rate and guarantee long-term stability of the system. The main contributions of this paper are summarized as follows. * Firstly, we construct an edge computing-enhanced industrial wireless network, where each industrial terminal can choose an industrial base station for binary computation offloading. In this scenario, we formulate a network computation rate maximization problem while balancing constraints including offloading time, CPU frequency, energy consumption, transmit power, and data queue stability. * Secondly, to maintain system stability, we define two dynamic queues, which are task queue and virtual energy queue. Based on these two dynamic queues, we minimize the upper bound of the drift-penalty function by using Lyapunov optimization, and transform the long-term MINLP problem into deterministic sub-problems. As the Lyapunov optimization problem is still non-convex, we further solve the problem with DRL. * Thirdly, we employ asynchronous advantage actor-critic (A3C) and propose Lyapunov-guided A3C algorithm named LyA3C for solution. Experiment results show that the LyA3C algorithm can converge stably and effectively improve the long-term network computation rate by comparing it with A2C-based and TD3-based algorithms. The rest of this paper is organized as follows. Section 2 discusses related works. Section 3 describes the system model. Then, Section 4 presents the proposed LyA3C algorithm. Next, Section 5 evaluates LyA3C through extensive experiments. Finally, Section 6 concludes the whole work.