# A Cluster-Based Optimal Computation Offloading Decision Mechanism Using RL in the IIoT Field
###### Abstract  
In the Industrial Internet of Things (IIoT), various tasks are created dynamically because of the small quantity batch production. Hence, it is difficult to execute tasks only with devices that have limited battery lives and computation capabilities. To solve this problem, we adopted the mobile edge computing (MEC) paradigm. However, if there are numerous tasks to be processed on the MEC server (MECS), it may not be suitable to deal with all tasks in the server within a delay constraint owing to the limited computational capability and high network overhead. Therefore, among cooperative computing techniques, we focus on task offloading to nearby devices using device-to-device (D2D) communication. Consequently, we propose a method that determines the optimal offloading strategy in an MEC environment with D2D communication. We aim to minimize the energy consumption of the devices and task execution delay under certain delay constraints. To solve this problem, we adopt a Q-learning algorithm that is part of reinforcement learning (RL)  
when using this paradigm. Computation offloading includes full offloading, a method that sends all tasks to the MEC server (MECS) at once, and partial offloading, a method that partitions and sends tasks to the MECS. The former processes tasks locally or on the MECS, while the latter processes partitioned tasks simultaneously locally and on the MECS. There are two types of computation offloading that can be adopted in the execution step. One is single-hop computation offloading, which sends the task directly to the MECS or cloud data center. The other is multi-hop computation offloading, where tasks are sent to nearby devices and then to the MECS or cloud data center [6]. However, MECS also has limited computation capability, making it unsuitable for processing many tasks compared to the cloud. Furthermore, the more tasks processed in MECS, the greater the burden on the cellular network for task transfer between the device and MECS [7]. Therefore, the paradigm of cooperative computing has recently been proposed, in which device-to-device (D2D) communication has garnered increasing popularity as a prominent technology that addresses the shortcomings of MECS [8]. MEC paradigm is a technology that is being used to support the limited computation capability and battery life of devices, and computation-intensive or delay-sensitive tasks. Therefore, computational offloading technology is one of the essential elements in the MEC paradigm. In [9], offloading decision is defined as a resource allocation problem with a single objective function that minimizes task execution time. However, this study did not take into account the energy consumption and load conditions of MECSs, which are particularly important in the IIoT environment during offloading decisions. Selecting offloading nodes that minimize task execution time using reinforcement learning is studied in [10]. However, this study also has limitations that do not take into account energy consumption. In [11], energy consumption and task execution time are defined as offloading cost, with game theory used to minimize offloading cost and optimize offloading. However, if the offloading target is limited to MECS, there is a limit to minimizing offloading costs. If the MECS is not close to the device, energy consumption may increase for task transmission. In addition, if multiple tasks are simultaneously offloaded into a MECS, the load on the server sharply increases and task execution time increases, which may not satisfy the latency constraints. Thus, D2D communication is used to solve this problem in consideration of computing resources. The focus of [12] is on formulating the computation resource allocation problem that reduces offloading costs in the IIoT field. However, the study does not consider cooperative computing. Minimizing the energy consumption of devices while ensuring reliability and latency requirements is studied in [13].