# Load Balancing of Two-Sided Assembly Line Based on Deep Reinforcement Learning
## 4 Deep Reinforcement Learning Algorithm Based on DPPO and CNN (DPPO-CNN)
### Reward Function  
Reinforcement learning agents can achieve the optimal task assignment strategy \(p_{\theta}(a_{t}|s_{t})\) by maximizing cumulative rewards (\(r_{sum}\)) and then realize the optimization objectives. Sparse reward is adopted by the traditional deep reinforcement learning algorithm, i.e., rewards \(r_{1}\), \(r_{2}\),..., \(r_{n-1}\) are all equal to 0 until all tasks have been assigned. The environment gives feedback reward \(r_{n}\) to the agent, then, the accumulate reward \(r_{sum}=r_{n}\). This is an easy way to converge the algorithm because for many tasks, the agent can obtain positive samples with a certain probability by conducting random exploration in the environment, and the positive samples occupy a relatively significant proportion of the total samples at the early stage of learning. However, with the increase in task complexity, the probability of obtaining positive samples through random exploration becomes small, and the sparse feedback signals cannot indicate the exploration direction for the agent. The algorithm will be difficult to converge or the convergence speed will be very slow. To overcome this problem, it is necessary to add other reward items or punishment items to make the reward function become dense, and to guide the agent to explore the environment more efficiently [39]. In this paper, for objective \(LE\), if both the operation time of the tasks and the number of stations are determined, the smaller the \(ct\), the higher the \(LE\). For objective \(SI\), if the workloads of tasks which are assigned to a station are substantially equal to cycle time of that station, the difference between the \(ST_{\max}\) and the \(ST_{i}\) will be small, and so will the \(SI\). Similarly, for objective \(CSI\), if the completion time of a station is substantially equal, the value of \(CSI\) is small, which means the operation of the assembly line is stable and balanced. Therefore, the values of \(SI\) and \(CSI\) will decrease if the idle time of each station is reduced. There are two types of idle time that can be generated on a two-sided assembly line due to its parallel structure and the sequence-dependent relationship of its tasks, as shown in Figure 12: (1) End idle time--If the completion time of the current station will exceed the cycle time when the new task is assigned to the current station, the new task has to be assigned to the next station. In this case, the completion time of the current station will be less than the cycle time, and the time difference between the completion time of the current station and cycle time is the end idle time. As shown in Figure 12, on workstation 1, after task 5 is completed, task 4 needs to be assigned, but if the operation time of task 4 is 9, whether it is assigned to the left or right stations of workstation 1, the completion time will exceed cycle time 18; therefore, task 4 has to be assigned to workstation 2, and there has to be idle time at the end of the left and right stations of workstation 1. (2) Sequence-dependent idle time--If sequence-related tasks are assigned to the left and right stations of the same workstation, due to the absolute sequence constraint between them, this may lead to a situation in which some tasks may have to sit idle and wait, and this kind of idle time inside stations is called sequence-dependent idle time. As shown in Figure 12, on workstation 2, task 7, assigned to right station, can not be started from time 0 because its immediate precedent, task 4, is assigned to left station of the same workstation. Task 7 has to wait before task 4 is completed; therefore, there are nine time unitsof sequence-dependent idle time before task 7, which is caused by task 4.