# Load Balancing of Two-Sided Assembly Line Based on Deep Reinforcement Learning
## 4 Deep Reinforcement Learning Algorithm Based on DPPO and CNN (DPPO-CNN)  
### DPPO-CNN Agent  
The architecture of DPPO-CNN with distributed multiple processes is shown Figure 3; the main process and \(m\) subprocess are turned on simultaneously. The main process is responsible for network training and update (gradient calculation and update), while the subprocess is only responsible for data acquisition without gradient calculation. The main process includes the experience pool and the main Actor-Critic network, and each subprocess includes the child Actor-Critic network. The Actor network is used to make task assignment decisions \(a_{t}\) based on the environmental status \(s_{t}\) of the TAL, while the Critic network is used to evaluate the quality of task allocation decisions \(a_{t}\). Actor-Critic network structures are shown in Figures 4 and 5. (1) Initialize the main Actor-Critic network of the main process and send its parameters to each subprocess through an orbit. (2) The child Actor-Critic network of the subprocess loads the main Actor-Critic network and then interacts with the environment and transmits interactive trajectory (state matrix \(s\), task assignment decisions \(a\), reward function \(r\)) to the main process through an orbit. (3) The main process stores the interaction experience of all subprocesses in the experience pool. When the amount of experience stored in the experience pool exceeds the capacity of the experience pool, it is packaged as a training set to train the main Actor-Critic network. (4) Transfer the updated main Actor-Critic network to each subprocess again and go back to (1). Figure 4: Actor network. Figure 3: The architecture of DPPO-CNN. The Actor network is the strategy network, which approximates the optimal task assignment strategy \(p_{\theta}(a_{t}|s_{t})\) by using the neural network with parameter \(\theta\). The network structure is shown in Figure 4, including a two-layer convolutional network and a three-layer fully connected network. The dimensional matrix \(M\times N\) is the input of the network at time \(t\) corresponding to the environmental status \(s_{t}\). \(M\) is the number of feature vectors, and \(N\) is the number of total assembly tasks. After the matrix is input, two layers of convolution operations are carried out on it. The convolution Kernel size is as follows: Kernel = (1, 3). The feature vector obtained after convolution is flattened and input into the three fully connected layers. The number of nodes in the first two layers is 256, and the number of nodes in the last layer is \(N\). Then, it is normalized by the SoftMax function and outputs \(p_{\theta}(a_{t}|s_{t})\), which is the probability of output task to be assigned \(a_{t}\) when the Actor policy network is in the status \(s_{t}\). The Critic network is evaluation network, which approximates the optimal strategy evaluation value \(v_{\Psi}(s_{t}|a_{t})\) by using the neural network with parameter \(\Psi\). The network structure is shown in Figure 5. In this paper, the first several layers of the Critic network structure are the same as that of the Actor network structure, but the last layer is the linear regression layer; that is, the SoftMax layer in the Actor network is replaced with \(v_{\Psi}(s_{t}|a_{t})=f(h(t);\Psi)=\omega*h(t)+b\), where \(v_{\Psi}(s_{t}|a_{t})\) is the output of the Critic network at time \(t\) and \(h(t)\) is the output of the previous fully connected layer. \(\Psi\) is the parameter of the internal unit node of the network, including the weight \(\omega\) and the bias item \(b\). In each subprocess, the learning process is shown in Figure 6. The agent observes environmental status \(s_{t}\) of task assignment, makes the selection decision, and outputs the task to be assigned \(a_{t}\) and updates the status and gives back the reward \(r_{t}\) after the assignment of task \(a_{t}\). The agent can solve the problem through continuous interaction with the environment.