# Intelligent End-Edge Computation Offloading Based on Lyapunov-Guided Deep Reinforcement Learning
## 4 End-Edge Computation Offloading Based on Lyapunov-Guided DRL
### Problem Transformation by Lyapunov Optimization  
As the problem \(P_{1}\) is a MINLP problem, its direct solution process is complicated, the Lyapunov optimization theory is introduced to model the dynamic queue, and this strategy decomposes the MINLP problem into deterministic sub-problems through the construction of mathematical models under the premise of ensuring the stability of the system. In order to satisfy the average power consumption constraint \(C_{8}\) in \(P_{1}\), we introduce \(N\) virtual energy queues \(\{Y_{n,m}(t)\}_{N\times M}\) for industrial terminals. Let \(Y_{n,m}(1)=0\), the queue length can be updated as  
\[Y_{n,m}(t+1)=\max(Y_{n,m}(t)+e_{n,m}-\xi_{n},0), \tag{9}\]  
When the virtual energy queue is stabilized, the average energy consumption satisfies \(e_{n,m}\leq\xi_{n}\). After establishing dynamic queues, we define \(F(t)=\{Q(t),Y(t)\}\) as the total queue backlog, then we employ quadratic Lyapunov function  
\[L(F(t))=\frac{1}{2}\sum\limits_{n=1}^{N}Q_{n}(t)^{2}+\frac{1}{2}\sum\limits_{ n=1}^{N}\sum\limits_{m=1}^{M}Y_{n,m}(t)^{2}, \tag{10}\]  
to represent the stability of the queues. Correspondingly, the Lyapunov drift function of the total queue backlog is defined as  
\[\triangle\ L(F(t))=\mathbb{E}[(L(F(t+1))-L(F(t))|F(t)\}\,, \tag{11}\]  
which demonstrates the improvement from \(t\)-th time slot to \((t+1)\)-th time slot. That is to say, the Lyapunov drift can reflect the system dynamics between two continuous states. From the perspective of system stability, the smaller the Lyapunov drift, the more stable the system is. However, the Lyapunov drift still depends on the system information of the next time slot. Thus, we further derive the Lyapunov drift. According to [27], for nonnegative real numbers \(A\), \(B\), \(C\), and \(D\) satisfying \(A\leq\max(B-C,0)+D\), there should be \(A^{2}\leq B^{2}+C^{2}+D^{2}-2B(C-D)\), so we have  
\[ \begin{split} Q_{n}(t+1)^{2}&=[[Q_{n}(t)-D_{n}]+A_{ n}]^{2}\\ &\leq Q_{n}(t)^{2}+(D_{n})^{2}+(A_{n})^{2}+2Q_{n}(t)(A_{n}-D_{n}), \end{split}  \tag{12}\]  
In this way, we can drive the upper bound of the Lyapunov drift function for data queue as  
\[ \begin{split}\Delta L(Q_{n}(t))&=\mathbb{E}\bigg{[} \frac{1}{2}\sum\limits_{n=1}^{N}Q_{n}(t+1)^{2}-\frac{1}{2}\sum\limits_{n=1}^{ N}Q_{n}(t)^{2}|Q_{n}(t)\,\bigg{]}\\ &\leq\mathbb{E}\bigg{[}\sum\limits_{n=1}^{N}\Big{(}\frac{1}{2} \Big{(}(D_{n})^{2}+(A_{n})^{2}\Big{)}+Q_{n}(t)(A_{n}-D_{n})\Big{)}|Q_{n}(t) \,\bigg{]}\\ &=B_{1}+\mathbb{E}\bigg{[}\sum\limits_{n=1}^{N}\big{(}Q_{n}(t)(A _{n}-D_{n})\big{)}|Q_{n}(t)\,\bigg{]},\end{split}  \tag{13}\]  
where \(B_{1}=\sum\limits_{n=1}^{N}\frac{(D_{n})^{2}+(A_{n})^{2}}{2}\). At the same time, we can have the upper bound of the Lyapunov drift function for virtual energy queue as  
\[\Delta L(Y_{n,m}(t))\leq B_{2}+\mathbb{E}\left[\sum\limits_{n=1}^{N}\sum\limits_{ m=1}^{M}Y_{n,m}(t)(e_{n,m}-\xi_{n})|Y_{n,m}(t)\right], \tag{14}\]  
where \(B_{2}=\sum\limits_{n=1}^{N}\sum\limits_{m=1}^{M}\frac{(e_{n,m})^{2}+(\xi_{n}) ^{2}}{2}\). Through the above process, the upper bound of the Lyapunov drift function for total queue backlog is given by  
\[\Delta L(F(t))\leq B+\sum\limits_{n=1}^{N}Q_{n}(t)\mathbb{E}[(A_{n}-D_{n})|F(t )]+\sum\limits_{n=1}^{N}\sum\limits_{m=1}^{M}Y_{n,m}(t)\mathbb{E}[(e_{n,m}-\xi_ {n})|F(t)], \tag{15}\]  
where \(B=B_{1}+B_{2}\). At the same time, it is obvious that the upper bound of the Lyapunov drift function is no longer decided by the system information of the next time slot. Then, the Lyapunov drift-plus-penalty approach is used to maximize the network computation rate while satisfying queue stability, that is to say minimizing the drift-plus-penalty expression within each time slot, we have  
\[\Delta(F(t))\triangleq\Delta L(F(t))-V\sum\limits_{n=1}^{N}\mathbb{E}[\beta_{ n}z_{n,m}|F(t)], \tag{16}\]  
where \(V>0\) is penalty weight that controls the significance of system data and virtual energy queue backlog.