## 4 Deep Learning-Based MEC System Energy Saving Framework
### DQN Agent for MECS Sleep Decision
\begin{array}{ll}-p_{e}(t)&\text{, if }\ \delta(t)<\delta_{th} \\ -p_{d}(t)&\text{, if }\ \delta_{th}\leq\delta(t)<2\delta_{th}\\ -3&\text{otherwise}\end{array} \right. \tag{16}\]  
As we can see in Equation (13), finding an optimal policy becomes finding an optimal \(Q\)-function. To find \(Q^{*}(s,a)\), a \(Q\)-learning agent has to search the state-action space whose size is \(|\mathcal{S}|\times|\mathcal{A}|\). When \(\mathcal{M}\) is given, \(|\mathcal{A}|=2^{|\mathcal{M}|}-1\). However, since \(\bar{\lambda}_{i}(t)\in\mathbb{N}\cup\{0\}\), \(|\mathcal{S}|\) is enormous. Given the vast search space, it is unlikely that an agent can visit every possible state-action pair to assess \(Q\)-function. Therefore, we take the DQN approach, which can approximate \(Q^{*}(s,a)\) with great accuracy by using a deep neural network parameterized by \(\theta\). We denote the DQN network by \(\mathbb{Q}\). Figure 4 depicts the DQN agent model, which is composed of a 1D CNN (Convolutional Neural Network) and a fully connected network. The 1D CNN network reduces the input dimension from \(|s(t)|=3|\mathcal{M}|\) to \(m_{1}\) by extracting features from \(s(t)\). The fully connected network is composed of \((L)\) hidden layers. Each hidden layer \(l\in[1,L]\) is composed of \(n_{l}\) perceptrons equipped with the ReLU (Rectified Linear Unit) activation function. The output layer yields the estimated state-action value \(Q(s,a)\). Training the DQN agent is to find the parameter \(\theta\) that can approximate the relations between the input and the output of the model. Following the DQN training technique in [16], we employ a replay memory D and a target network \(\mathbb{q}^{\prime}\) to train the DQN agent that estimates \(Q^{*}(s,a)\) in a stable and efficient manner. The DQN agent generally accumulates experiences sequentially. However, these sequential data tend to have high correlations, which can cause instability when used to train the DQN network \(\mathbb{Q}\). Replay memory addresses this issue by randomly sampling the agent's experiences for training, thereby reducing these correlations and ensuring a more independent data distribution. The target network \(\mathbb{q}^{\prime}\) with parameters \(\theta^{\prime}\) is introduced to ensure stable learning of \(\mathbb{Q}\) by providing stable state-action value targets. The target network is structurally identical to the DQN network, and \(\theta^{\prime}\) is updated at every \(C\) training iteration by copying \(\theta\). In a state \(s(t)\), the DQN agent selects an action \(a(t)\) for the time slot \(t\) in the \(\epsilon\)-greedy manner. That is, \(a(t)\) is randomly selected with a probability \(\epsilon\). Otherwise, \(a(t)=\arg\max_{a\in\mathcal{A}}\mathcal{Q}(s(t),a;\theta)\). After playing \(a(t)\), the DQN agent receives a reward \(r(t+1)\), and the state is changed to \(s(t+1)\). The agent stores the experienced transition \(\
u(t)=\{s(t),a(t),r(t+1),s(t+1)\}\) to the replay memory. Every time the number of new experienced transitions becomes \(n_{e}\), we uniformly sample \(n_{b}\) transitions from D to form a minibatch \(\mathcal{B}=\{\
u_{1},\dots,\
u_{n_{b}}\}\). For each \(\
u_{i}=\{s_{i},a_{i},r_{i},s^{\prime}_{i}\}\in\mathcal{B}\), the target \(Q\)-value is calculated as \(y(v_{i})=r_{i}+\max_{a^{\prime}\in\mathcal{A}}Q^{\prime}(s^{\prime}_{i},a^{ \prime};\theta^{\prime})\), where \(Q^{\prime}(s^{\prime}_{i},a^{\prime};\theta^{\prime})\) is an output of the target network. Then, the estimation loss of the DQN network \(\mathbb{Q}\) for a sample \(\
u_{i}\) is given as \(\omega(v_{i})=(Q(s_{i},a_{i};\theta)-y(v_{i}))^{2}\). Thus, the loss function for the minibatch \(\mathcal{B}\) is given as  
\[\Omega_{\mathcal{B}}(\theta)=\frac{1}{n_{b}}\sum_{i=1}^{n_{b}}\omega(v_{i}). \tag{17}\]  
Therefore, the DQN network parameter at the beginning of the training with the \(k\)-th minibatch (denoted as \(\theta(k)\)) is updated by using the stochastic gradient descent to minimize \(\Omega_{\mathcal{B}}(\theta)\).