## 4 DQN-Based Priority/Demanding Resource Management
### DQN-Based Solutions  
Our work aims to provide flow execution by using the DQN algorithm to handle resource demand and prioritization on the MEC server. In fact, DQN is one of the most powerful tools for assisting network controls and adaptation of resource estimates. Moreover, DQN offers two stages of architecture (e.g., target network and online network). Figure 2 indicates the network conditions, which consist of several stages of the IIoT cluster that are divided into three categories of network determination resource tasks. The proposed method is conducted using DQN with an SDN/NFV controller for interaction and abstracts the resource utilization from MEC resources in computing the IIoT tasks. A strategy chooses the action obtained through long-term optimization, and DQN maximizesthe reward value through the selection of the optimal value. Controller gathers the state as in Equation (14) and selects action \(a(t)=\pi(s(t))\) to obtain the current reward \(r_{t}\), while \(s(t)\) is transferred to the next state. \(\pi\) is the specific policy. The interaction with the environment proceeds based on the updated state, aiming to maximize the reward value by maintaining the ongoing process. The cumulative discounted reward over the time interval is calculated using \(Q((s(t),a(t))\) as in the following equation:  
\[Q(s(t),a(t))=E\bigg{[}\sum_{t=1}^{T}\gamma^{t}r(t)\big{|}s(t),a(t)\bigg{]}, \gamma\in(0,1) \tag{17}\]  
From the equation, \(\gamma\in(0,1)\) is the discount factor. If \(\gamma=1\), the agent only considers the current rewards, whereas if \(\gamma\leq 1\), the agent considers the later rewards. \[Q(s(t),a(t))=Q(s(t),a(t))+\alpha(r(t)+\gamma maxQ(s(t+1),\;a(t+1))-Q(s(t),a(t))) \tag{18}\]  
When this value is used to indicate iteratively estimate the optimal Q-value for all the state and action pairs s(t), a(t), and the optimal policy \(\pi^{*}\):  
\[\pi^{*}=\underset{\pi}{argmax}Q(s(t),\;a(t)\,) \tag{19}\]  
\(\alpha\) is the learning rate, while the deep neural network (DNN) technique is leveraged. DQN is constructed into two structures, with one of the DNNs being the online network, which fits the value function Q, and another DNN being the target network, which is used to gain the target Q value. \(\varnothing\) and \(\varnothing^{-}\) are the weight of online and target on Q-network, respectively. \[Q^{*}=r\big{(}s(t),a(t)+\gamma maxQ(s(t+1),a(t+1));\varnothing^{-}\big{)} \tag{20}\]  
In addition, in the learning rate phase, the reward and state updates obtained from the iteration with an environment that is stored in the experience buffer in forms \(s(t),a(t),r(t),\)\(s(t+1)\). The parameters of DNN are updated by interactively minimizing the loss function as follows:  
\[\small\textit{LOSSFUNCTION}=E\Big{[}(Q^{*}-Q(s(t),a(t);\varnothing))^{2} \Big{]} \tag{21}\]  
Algorithm 1 depicts the hierarchical structure proposed for handling the resource priority and demanding satisfaction in terms of energy efficiency in the MEC server for controlling the IIoT tasks. On the other hand, the DQG-PD algorithm is leveraged for priority and demand resource allocation in the MEC server and employs DQN to determine  
Figure 2: DQN-based MEC for priority/demanding resource utilization. the optimal actions for each time slot. Initially, the algorithm sets up two Q-networks, an online network and a target network with random weights and initializes a replay memory to store previous experiences. In each episode, an initial state proceeds through multiple time steps, deciding between exploration and exploitation. During exploration, an action is selected randomly, while in exploitation, the action that maximizes the Q-value (predicted future reward) is chosen based on the online Q-network. The selected action is then executed, leading to a new state and reward stored in the replay memory. It periodically samples a mini-batch of experiences from this memory to update the online Q-network by performing gradient descent on the loss between predicted and target Q-values.