# A Novel Hybrid Path Planning Method Based on Q-Learning and Neural Network for Robot Arm
## 2 Method Description
### Action-Finding by Q-Learning (Active)
We can write the state space and action space in the following ways. \[S=\{cell_{1},\ \ cell_{2},\ \ cell_{3},\ \ \ldots\,\ cell_{n\times m}\} \tag{1}\]  
where \(n\) and \(m\) are the world width and world length, respectively. In fact, the more resolution it needs, the more cells we should provide. The state space has only \(n\times m\) members. \[A=\{Up,\ Down,\ Left,\ Right,\ \ldots,\ Action_{k}\}, \tag{2}\]  
The action space has \(k\) members, including simple movements of up, down, left, and right. Since the state space and action space are smaller and more straightforward, interacting with the environment can be modeled in programming software easily without using simulators, which are basically complex software. This strategy can significantly reduce the running time of code and improve complexity. Before starting the Q-learning algorithm to find the optimal path, we must identify three positions: the position of start, the position of the target, and the obstacle's position. Like in real applications, we place the position of start, target, and obstacle randomly. Therefore, to detect the positions, an image processing algorithm is used. We use a picture taken by the camera from the top view, which is cropped from the 2D workspace, as input of the image processing section. Indeed, by a computer vision algorithm, five different colors, including yellow, which represents the start point; red, which represents the obstacle; green, which represents the target; black, which represents grid borders; and white, which represents background color, are detected via their RGB code. Then, it is ready to use the KNN algorithm for further analysis. KNN is one of the simplest classification algorithms available for supervised learning. This algorithm falls under the supervised learning category and is used for classification (most commonly) and regression. It is a versatile algorithm also used for imputing missing values and resampling datasets. As the name (K Nearest Neighbor) suggests, it considers K Nearest Neighbors (data points) to predict the class or continuous value for the new data point [13]. By means of the KNN algorithm, we classify pixels of the images by different colors via RGB code of colors to differentiate between the start cell in yellow, the obstacle cell in red, the target cell in green, and the rest of the cells in the background color. The idea is to search for the closest match of the RGB code of colors of pixels. So, since there are five main colors, including yellow, red, green, black, and white, we use an optimum number for the K. In this section, the image pixels are swept so that the coordinate of the center of each particular color is obtained. Thus, in order to increase the speed of sweeping, a low-resolution image can be used. Simply put, the KNN algorithm classifies unknown pixel points by finding the most common class among the K-closest examples. We defined a distance metric or similarity function by pixels indexes of the image to apply the KNN classification. This simple algorithm is used to avoid complexity. In other words, we use this algorithm because the KNN classifier is by far the simplest machine learning and image classification algorithm. In fact, it is so simple, and it does not need to be trained. However, this algorithm has a few limitations: since KNN finds coordinates of pixels with similar colors, start, obstacle, and target cells have to have different colors. Additionally, we can only use one obstacle unless we use different colors for obstacles. In fact, although the KNN is simple and faster than other algorithms, it is not able to distinguish objects by their shape, but it can detect them by their colors. All in all, KNN image processing distinguishes the start, target, and obstacle from other areas, and then it obtains their coordinates of the center of the area by analyzing the picture taken from the top view of the 2D workspace. It should be noted that a workspace of a robot arm is the set of all positions that it can reach.