# A Novel Hybrid Path Planning Method Based on Q-Learning and Neural Network for Robot Arm
## 2 Method Description
### Angle-Finding by Trained Neural Network (Passive)
The number of hidden layers and their neurons was selected empirically by trial and error during the training. It also has three inputs, including indexes of the row and column of the current cell and selected action. It has six outputs because the robot arm used in this article is model IRB 1600 that is a 6 DOF type. The activation functions of the input and output neurons are the ReLU (Rectified Linear Unit) function because this function is most widely simply used without any complexity. Additionally, the activation function of the hidden neurons is the sigmoid function. \[\sigma_{ReLU}(x)=\max\{0,x\}, \tag{3}\]  
\[\sigma_{sigmoid}(x)=\frac{1}{1+e^{-x}}, \tag{4}\]  
After designing the overall structure of the neural network, we collect data set to train it with respect to the designed structure. In order to collect the data set, we place the end-effector of the robot arm in different cells of the gridded workspace. In each cell, the vicinity cells such as upper, lower, left, and right cells are considered as input that we use their indexes as current cell indexes. In fact, they are four different data that by executing up, down, left, and right actions that they potentially can reach the central cell in which the end-effector is placed. Next, when the end-effector is in the central cell, the joints angles of the robot arm are measured as the output of those four data. In fact, by placing the end-effector in each cell, four data can be generated which their inputs are different, but their outputs are the same. The end-effector can reach the central cell from those four vicinity cells by setting the robot arm to measured joints angles. Since the inputs are cell indexes and numerical forms of actions, they are integers, but the outputs, which are joints angles, are float types because a joint angle is a float value. In this regard, in order to use in neural network training, we collect 128 data points which around 70% of which are used in the learning mood, and the rest are used in testing and validation mood. Since we chose an eight-by-four gridded workspace, that amount of data points are enough to train our neural network well. Obviously, the higher the resolution, the more data is needed. Since we have a total of 32 cells (eight by four grid), and each cell can potentially move in 4 directions, the number of all possible moves is 128 (32 \(\times\) 4). That means we use all possible data in the data set. Therefore, we may claim that our data set is effective because it is rich enough to cover all motions from all cells. We want to highlight that when the robot is in the cells that are on the border of the grid also has four possible actions. The only difference is that when it gets an action to leave the grid, it stays in that border cell and does not move. After data acquisition, we switch to train the neural network to use it in the angle finding phase. In order to train the neural network, we first initialize the weights to small random numbers. Next, for each training sample, we compute the output value. Then, by means of the gradient descent algorithm, we update the weights via the backpropagation method. This process repeats again and again so that the weights converge to their optimum values and the error ranges are within the allowable range. Figure 1(a) shows the structure of the neural network, and Figure 1(b) shows the data acquisition sample through vicinity cells. The neural network that we use in this part is a multi-layer perceptron with a single hidden layer neural network (MLP-single hidden layer) with ReLU and sigmoid activation function. This type of neural network was first introduced by [PERSON] [15]. In our hybrid method, we train the neural network only once, and after training, we use in the angle-finding process. In other words, this time-consuming training could be a single process before path planning in the experimental environment, while using the trained neural network takes only a few seconds and could be used every time during path planning in factories.