## 4 Deep Learning-Based MEC System Energy Saving Framework
### DQN Agent for MECS Sleep Decision  
\(Q\)-learning aims to maximize long-term cumulative rewards. To achieve the goal, \(Q\)-learning attempts to derive an optimal policy function (\(\pi^{*}\)) based on states that selects actions to maximize the average cumulative rewards by interacting with the environment through a sequence of actions, observed states, and incurred rewards. Specifically, \(Q\)-learning is represented by a tuple \(<\mathcal{S},\mathcal{A},\mathcal{R}>\), where \(\mathcal{S}\) is a set of possible states, \(\mathcal{A}\) represents a set of possible actions, and \(\mathcal{R}\) is a set of rewards. A state-action value function (generally known as \(Q\)-function) obtained when a \(Q\)-learning agent follows a policy \(\pi\) is defined as  
Figure 3: Block diagram of the STGCN workload predictor. \[Q^{\pi}(s,a)=\mathbb{E}[\sum_{i=t}^{T}\gamma^{i-t}r(i+1)|s(t)=s,\pi(s(t))=a], \tag{12}\]  
where \(s(t)\) is a state at time \(t\), \(\pi(s(t))\) represents the action taken by following the policy \(\pi\), the parameter \(\gamma\in[0,1]\) is a discount factor, and \(r(i+1)\in\mathcal{R}\) is a reward obtained after taking an action at a state \(s(i)\). Thus, \(\sum_{i=t}^{T}\gamma^{i-t}r(i+1)\) is the cumulative reward obtained after taking an action \(\pi(s(t))\) in a state \(s(t)\). If we define an optimal state-action value function as \(Q^{*}(s,a)=\max_{\pi}Q^{\pi}(s,a)\), it is known that an optimal policy (\(\pi^{*}\)) is obtained as follows [36]. \[\pi^{*}=\arg\max_{a\in\mathcal{A}}Q^{*}(s,a),\forall s\in\mathcal{S}. \tag{13}\]  
Our problem in (9) requires determining an optimal MECS sleep state vector at the beginning of each time slot. To resolve the problem by exploiting \(Q\)-learning, we define the following states, actions, and rewards. State: The energy consumption and the maximum service delay of a MEC system during a time slot \(t\) are determined by the workload distribution (\(Q_{i}(t)\)s) at the beginning of a time slot, the amount of workload newly offloaded to each MECS during a time slot (\(\lambda_{i}(t)\)s), and the capacity of MECSs (\(C_{i}\)s). In addition, \(\lambda_{i}(t)\) is influenced by \(\eta(t)\) and \(\bar{\lambda}_{i}(t)\). Since the DQN agent does not know \(\bar{\lambda}_{i}(t)\), it uses \(\bar{\lambda}_{i}(t)\) provided by the first module and estimates \(\lambda_{i}(t)\). Therefore, the state of the MEC system at the beginning of a time slot \(t\) is denoted by the vector. \[s(t)=[Q_{i}(t),\bar{\lambda}_{i}(t),C_{i}],\forall i\in\mathcal{M}. \tag{14}\]  
Action: Since the role of a DQN agent is to determine a MECS status vector, we specify the action of the agent as a vector. \[a(t)=\eta(t)=[\eta_{i}(t)],\forall i\in\mathcal{M}. \tag{15}\]  
Reward: The goal of the controller is to decide an optimal \(\eta(t)\) that minimizes the amount of energy consumed by the MEC system during a time slot \(t\) while preventing the maximum service delay provided by the system during the time slot from being greater than \(\delta_{th}\). Therefore, the desirable behavior of the agent is to minimize the energy consumption when \(\delta(t)<\delta_{th}\) and to reduce \(\delta(t)\) rapidly when it becomes larger than \(\delta_{th}\). To guide the agent to behave in such a manner, we define the following penalty function by normalizing both the average energy consumed by the MEC system and the maximum service delay provided by the system during a time slot. \[p_{e}(t) =\frac{\frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}}\eta_{i}(t) \xi(t)-\min_{i\in\mathcal{M}}(\zeta_{i,min})}{\sum_{i\in\mathcal{M}}(\zeta_{i, max})-\min_{i\in\mathcal{M}}(\zeta_{i,min})}\] \[p_{d}(t) =\delta(t)/\delta_{th},\]  
where \(\zeta_{i,min}\) represents the minimum amount of energy that a MECS \(i\) may consume when it is in an active mode, while \(\zeta_{i,max}\) represents the maximum energy that a MECS \(i\) can consume when it is in an active mode. Then, we configure the reward for the action taken at the beginning of a time slot \(t\) (i.e., a MECS status vector \(\eta(t)\)) as  
\[r(t+1)=\left\{