# A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field
## 3 Problem Definition and Proposed Algorithm  
In this section, we formulate the problem and describe its solution. We propose an algorithm to improve the system throughput and satisfaction degree associated with the quality of service (QoS) by reducing the total task execution delay and total energy consumption of devices in an IIoT environment, where devices and MECSs have limited computing capability and queue length for processing tasks. The objectives and constraints of this study can be summarized as follows:  
\[ \begin{split} O1\ :\ \min\limits_{O}\sum\limits_{c=1}^{C}\{(1- \alpha)E_{d}(t)+\alpha\cdot L_{d}(t)\}\\ s.t.\begin{cases}C1\ :\ o_{d}\in\{0,\ 1,\ 2\},&\forall o_{d}\in O\\ C2\ :\ L_{c}\leq L_{req},&\forall c\in C\\ C3\ :\ q_{d}\leq q_{d}^{max},&\forall d\in D\\ C4\ :\ q_{m}\leq q_{m}^{max},&\forall m\in M\end{cases}\end{split}  \tag{5}\]  
where \(\alpha\) is a factor that balances the energy consumption and task execution delay and \(O\) denotes the offloading strategy set of all tasks. Any \(o_{d}\) belonging to the set \(O\) means an offloading strategy of a task generated in the device \(d\) in a time slot, thus, it can assume a value between 0, 1, and 2, such as C1. At O1 in Equation (5), \(\{(1-\alpha)E_{d}(t)+\alpha\cdot L_{d}(t)\}\) indicates an offloading cost created by combining the energy consumption of a device and the task execution delay. Since the goal of the proposed method is to minimize the offloading cost, it is used as a reward function in a [PERSON] decision process when learning progresses. Through C1, observe that the problem is an integer problem. \(E_{d}(t)\) and \(L_{d}(t)\) represent the energy consumption and task execution delay for processing task \(c\), respectively. C2 demonstrates that the task execution delay must satisfy the delay constraints. Let \(q_{d}\) and \(q_{m}\) be the loads of device \(d\) and MECS \(m\), respectively, while \(q_{d}^{max}\) and \(q_{m}^{max}\) denote the maximum loads of device \(d\) and MECS \(m\), respectively. We assume an IIoT scenario that is characterized by a division of work and a job shop. According to such characteristics, cooperating devices close to each other are clustered. After clustering, a CH in each cluster is selected considering the working characteristics and computational capabilities of the devices in the cluster. It is assumed that the selected CH does not change. However, it is difficult to find the optimal offloading strategy in an environment where the states of MECSs and devices change dynamically over time. In a dynamic environment, it is not appropriate to use conventional heuristic methods owing to a high computational complexity. Thus, we use the Q-learning algorithm, a model-free reinforcement learning (RL) scheme that can be executed on a CH without requiring prior knowledge of the environment and high computational resources. Prior to executing Q-learning, each CH determines its serving MECS and D2D device that execute the new tasks in the cluster. Its serving MECS and D2D device are selected considering their respective workloads. The CH determines whether the task is offloaded; if so, the task is offloaded to the serving D2D device or MECS. This approach ensures that the system throughput is improved and delay constraints are satisfied via reducing task blocking and queuing delay. We define the subsequent Markov decision process (MDP) as follows:  
_Agent_: The CH(\(ch_{i}\)), \(\forall\ ch_{i}\in\{ch_{1},\ldots,ch_{n}\}\)  
_State_: \(s^{i}(t)=\big{(}c_{d}(t),q_{d}(t),\ q_{m}(t),\ q_{j}(t),l_{d}(t),l_{m}(t),l_{j} (t)\big{)}\), being the state of the created task at time slot \(t\) in cluster \(i\).