# A Novel Hybrid Path Planning Method Based on Q-Learning and Neural Network for Robot Arm
## 2 Method Description
### Action-Finding by Q-Learning (Active)
This depends on several factors, including the arm's dimensions, number of joints. However, in this method, the 2D workspace is a fraction of the whole workspace active in the production line, which is highly common in factories. Where robot arms are fixed and placed next to a conveyor belt and do assembly process or manufacturing process. In Figure 1, a scheme of the 2D workspace with three main cells, including the start in yellow, target in green, the obstacle in red, and direction of actions, are shown from the top view. As Figure 1 shows, in this paper, we choose four actions, including up, down, left, and right as action space. Although it is possible to increase the number of actions, the computational cost will increase as well. Additionally, the resolution, which is actually the size of end-effector motion in each step, can be increased but at the cost of a higher computational load. It is also possible to decrease it but at the risk of finding no path. Indeed, there is a trade-off in the granularity of discretization or resolution: too fine will increase the search space exponentially, result in greater computational cost and slowness, and too coarse may result in failing to find a path even if there exists one or finding inaccurate results. In the results section, we will show the resolution effect on slowness by providing the diagram of Q-learning, neural network, and total running time versus the number of cells. As the number of cells increases, the time of running Q-learning code increases too. According to the application of the robot arm, one should determine that what resolution and speed are suitable. In this article, according to the required resolution and speed, we use an eight-by-four gridded 2D workspace that means there are thirty-two cells (states) in which the end-effector can be placed. The cells in which the start, target, and obstacle are located are considered the start, target, and obstacle cells, respectively. Then, the best actions from the start cell to the target cell are found by the Q-learning algorithm under reward and punishment conditions. We use Q-learning as the RL algorithm because, according to [PERSON]'s results [14], to get a maximum accumulated reward in windy grid world and maze problems, the Q-learning algorithm is better than the SARSA and other algorithms. In other words, since our problem is similar to [PERSON]'s problem, we use Q-learning as the RL algorithm in the action-finding phase. Q-learning is an off-policy RL algorithm that seeks to find the best action to take given the current state. It is considered off-policy because the Q-learning function learns from actions that are outside the current policy, like taking random actions, and therefore a policy is not needed. More specifically, Q-learning seeks to learn a policy that maximizes the total reward. In this method, the reward and punishment are defined as follows: the agent receives a reward +50 if it reaches the target cell. The agent gets a punishment of \(-100\) if it reaches the obstacle cell. Finally, all other actions result in \(-1\) punishment. Given these roles, the agent tries to reach the target to earn and collect the maximum reward. Additionally, the agent tries to avoid obstacles and other useless actions to collect maximum rewards. It can be said that the Q-learning algorithm starts searching the optimal sequence of actions from the start cell and then reaches the target cell while avoiding the obstacle cell.