Online path planning of cooperative mobile robots in unknown environments using improved Q‐Learning and adaptive artificial potential field

The Journal of EngineeringVolume 2023, Issue 2 e12231 ORIGINAL RESEARCHOpen Access Online path planning of cooperative mobile robots in unknown environments using improved Q-Learning and adaptive artificial potential field Melika Ataollahi, Melika Ataollahi School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran Contribution: Formal analysis, ​Investigation, Methodology, Software, Writing - original draftSearch for more papers by this authorMohammad Farrokhi, Corresponding Author Mohammad Farrokhi [email protected] School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran Center of Excellence for Modelling and Control of Complex Systems, Iran University of Science and Technology, Tehran, Iran Correspondence Mohammad Farrokhi, Narmak, Farjam St., School of Electrical Engineering, Iran University of Science and Technology, Tehran 16846-13114, IRAN. Email: [email protected] Contribution: Formal analysis, Methodology, Supervision, Validation, Writing - review & editingSearch for more papers by this author Melika Ataollahi, Melika Ataollahi School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran Contribution: Formal analysis, ​Investigation, Methodology, Software, Writing - original draftSearch for more papers by this authorMohammad Farrokhi, Corresponding Author Mohammad Farrokhi [email protected] School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran Center of Excellence for Modelling and Control of Complex Systems, Iran University of Science and Technology, Tehran, Iran Correspondence Mohammad Farrokhi, Narmak, Farjam St., School of Electrical Engineering, Iran University of Science and Technology, Tehran 16846-13114, IRAN. Email: [email protected] Contribution: Formal analysis, Methodology, Supervision, Validation, Writing - review & editingSearch for more papers by this author First published: 09 February 2023 https://doi.org/10.1049/tje2.12231AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Abstract Online path planning in unknown environments with obstacles is a challenging issue in mobile robot systems. Cooperative mobile robots have received much attention in recent years due to their robustness and ability to perform more complex tasks and operations faster. In this article, by unifying the Dyna Q-learning and a novel concept (introduced as a feature matrix) an improved Q-learning is proposed that accelerates the learning process even with poor choice of parameters. In addition, to overcoming the local minima, an adaptive artificial potential field method is developed, so that two parameters of the new concept named the virtual obstacle are controlled using the massachusetts institute of technology (MIT) method to navigate the mobile robot in the proper direction. Then, with proper utilizing of the improved Q-learning and adaptive artificial potential field method, the path planning is performed online effectively while target tracking and collision avoidance are guaranteed. Finally, the performance of the proposed method is tested on decentralized cooperative mobile robots. The simulation results showed optimal path planning in terms of the distance in an unknown environment and collision avoidance with the obstacles. In addition, getting out of the local minima (i.e. immobility) is guaranteed in all situations. 1 INTRODUCTION One of the most fundamental issues in mobile robot systems is the path planning and navigation. If the environment is static or known a priori, the path can be planned offline [1, 2]. However, in many practical applications the environment is unknown or dynamic; hence, it is necessary to develop the path online according to the local information [1, 3]. Nevertheless, in the online path planning, the calculations should be minimized to plan the path in real time. One of the remarkable methods in path planning is the machine learning-based methods [4]. The Q-Learning (QL) has been used widely to design the path [5-7]. Deep Learning (DL) and QL were also used for classification [8, 9] and target-object detection [10, 11]. In this regard, the current and future challenges are reviewed in [12]. In [13], the time complexity related to the dimensions of the Q-table is reduced because unlike Classical Q-Learning (CQL), where the Q-table data is updated until convergence, in this method the Q-table data is updated only once. In [14], by defining state variables based on space allocation, the convergence of the learning process was improved. To improve the CQL structure, other methods have been employed to develop the learning process. In [6], the Flower-Pollination Algorithm is used to better initialize the Q-table at the beginning of the learning process. The DL has also been employed to improve the CQL by function approximation in [15] and [16]. In Deep Q-Learning (DQL), continuous state space is considered. To increase the convergence rate of DQL, Asynchronous Advantage Actor-Critic (A3C) has been employed in [17]. In [18], Deep Reinforcement Learning (DRL) is utilized when the dimensions of the Q-table increase due to the large amount of sensory data. Classical algorithms are based on optimization methods. The Artificial Potential Field (APF) is the most widely used scheme. This algorithm navigates the mobile robot to the target without colliding with the obstacles and shortest path [19, 20]. However, the most important problem of the APF is the local minima [21]. To solve this problem, the weights of the attractive and repulsive forces can be adjusted [22, 23]. In [24], the definition of attractive and repulsive potential functions was modified. In [25], using interaction of the potential field with optimization variables, motion controllers continuously communicate with each other to maintain communication integrity in multiple mobile robots. To design the path using neural networks, the quality of the training samples must be improved. In this regard, the APF method was used in [26] to improve the quality of the input data to the neural network. In [27], the CQL was employed for collision avoidance, and the APF method was used for target tracking. In [28], to plan the path for Autonomous Underwater Vehicles (AUV) the APF method was used to adjust the CQL parameters; and in [29] a fuzzy system was used to avoid collisions. Some related articles and their aims are shown in Table 1. TABLE 1. Related papers. Ref. Year Main purpose [28] [29] 2018 2021 - Utilize the APF to find the shortest path and target tracking. - Collision avoidance by QL. [30] 2019 - Utilize the APF method to find the shortest path. - Guaranteed collision avoidance by the combination of QL and Fuzzy systems. [31] 2020 - Introduction the APF with the concept of black hole to correct the potential field in the face of local-stable-point. - The energy field of the APF is considered as the input of the DQL, predicting the Q value. - Applied to an unknown and dynamic environment by considering the limited field of view. [32] 2017 - Initialization of the Q-table by the APF implementation to update the Q-table to increase convergence speed and quality before training phase. [33] 2022 [34] 2021 - Utilize the real-time information as input to DQL. - Utilize the APF to improve the action space and reward function of the DQN to solve the sparse reward. [35] 2022 - Deep deterministic policy gradient (DDPG) to adjust APF functions to help allocate power between different energy sources. [36] 2021 - Define implementation and parameters based on the behaviour of the APF method. - Utilize the Deep Deterministic Policy Gradient (DDPG) reinforcement-learning algorithm to optimize the policy. - No local minimum occurrence [37] 2012 - Defining a look-up table based on an evaluation function affected by potential forces and reinforcement rewards. - Utilize the APF method to find and fast convergence. - Utilize the reinforcement learning to ensure consistent behaviours and convergence. - Applied to an unknown and dynamic environment by considering the limited field of view. In examining the mobile robots group, it is necessary to look over the existing concepts and terms. The structure of the robots group is the most comprehensive part, which can be divided into three categories according to Table 2. TABLE 2. Structures of multiple mobile robots. Type Characteristics Centralized -There is a leader and at least one follower. -The requirement for a strong communication platform, based on the central processing unit (leader). Decentralized -The independence of each member to make decisions. -No need for a robust communication platform. -Relative communication exchange (based on definition and need) between group members. Hierarchical -Division of group members into several subgroups. -The need for a strong communication platform based on the local central authority (the leader of each subgroup). -Changing the leader of each subgroup based on the change of position of the members. Another point is how to communicate between group members, which is generally divided into three categories according to Table 3. TABLE 3. Communications of multiple mobile robots. Type Characteristics Explicit - Full knowledge of the decision and sensor information of other group members. Virtual -Identifying a specific and defined trace or sign left by other members. Action Detection -Identifying changes in the environment (changing the position of fixed obstacles, etc.), -Only in known environments. The third concept is the interaction of mobile robots, which significantly determines the assignment of tasks and the behaviour planning of each group member. Table 4 shows the types of interactions in the group of mobile robots. TABLE 4. Interactions of multiple mobile robots. Type Characteristics Collective - Completely independent decision-making by each member - No requirement for a communication platform - One similar target for all group members. Collaborative - Independent decision making by each member after receiving the necessary information from other members, - Requisition for a communication platform (exchange of required information: location, desired target, etc.), - the existence of several targets, - Assigning several mobile robots to one of the goals and executing operations simultaneously. Cooperative - Independent decision-making by each member after receiving the necessary information from other members - Communication platform request (exchange of required information such as location, intended purpose etc.) - the existence of multiple goals, - Assigning several mobile robots to one of the goals, - Implementation of operations for collaboration and assistance, not necessarily the simultaneous utilize of robots on a specific target (possibility of complementary actions by several mobile robots). Coordinative - Independent decision making by each member after receiving the necessary information from other members - Communication platform request (exchange of required information such as location, intended purpose, etc.) - the existence of multiple goals, - assigning a mobile robot to one of the targets, - Not disrupting the performance of other members. In this paper, an online path planning for Cooperative mobile robots in unknown environments with static and dynamic obstacles is proposed. To this end, an improved CQL (as the Dyna-Feature QL) is proposed to increase the convergence rate of the algorithm. In addition, an Adaptive Artificial Potential Field (A-APF) method is employed to overcome local minima. The distributed learning is performed online for a decentralized group of mobile robots. Using the proposed method, the best action in every state is found faster even by poor choice of parameters and function approximation. The advantages and novelties can be stated as follows: Adaptively adjusting parameters of a new concept named virtual repulsive force in the proposed A-APF method to overcome the problem of local minima Significant increase in the convergence rate of the learning process by improvement of the CQL and development of by the A-APF method as a supervisor Guaranteed Collision free path and target reaching by considering the A-APF method as a supervisor Online path planning in an unknown and cluttered environment with static and dynamic obstacles Planning an optimal and safe path using local information Smoothing the planned path by considering a limited set of allowable actions and reducing energy in the mobile robots by applying two fuzzy systems The cooperation of the mobile robot group in finding the main goal among possible situations (such as applications such as rescue etc.) In some applications, employing cooperative mobile robots increases the quality of performing tasks. As explained in Tables 2 and 4, despite independent performance of each group member, they can help each other to achieve their goals. This cooperation leads to reduction of energy consumption, less exploration time in the environment, and faster achievement of goals. Moreover, many applications in the real world are in unknown and dynamic environments such that a detailed map of the environment and the number and position of moving obstacles are not available. Therefore, by assuming that a well-known environment exists a priori, sever limitations are created for applying the proposed method in real conditions. In this regard, an unknown environment with static and dynamic obstacles is considered. Hence, the path must be planned online and in real time. This paper is organized as follows. Section 2 presents the mobile robot model and definition of the path planning. The proposed method is explained in Section 3. In Section 4, mathematical analysis of the proposed method is given. Simulating results are presented in Section 5. Conclusions are drawn in Section 6. 2 PROBLEM FORMATION Robot model A group of cooperative two-wheeled mobile robots is considered. The dynamic model of the ith mobile robot is described as [38]: p ̇ r , i = J ( p r , i ) ω i M ω ̇ i + C ( p ̇ r , i ) ω i + D ω i = τ i $$\begin{equation} \def\eqcellsep{&}\begin{array}{l} {{{{\dot{\bf p}}}}}_{{\rm{r}},\,i} = {{\bf J}}({{{\bf p}}}_{{\rm{r}},\,i}){{{\bm \omega }}}_i\\[8pt] {{\bf M}}{{{{\dot{\bm \omega }}}}}_i + {{\bf C}}({{{{\dot{\bf p}}}}}_{{\rm{r}},\,i}){{{\bm \omega }}}_i + {{\bf D}}{{{\bm \omega }}}_i = {{{\bm \tau }}}_i \end{array} \end{equation}$$ (1)where p r , i = [ x r , i , y r , i , ϕ r , i ] T ${{{\bf p}}}_{{\rm{r}},\,i} = {[{x}_{{\rm{r}},\,i},{y}_{{\rm{r}},\,i},{\phi }_{{\rm{r}},\,i}]}^T$ denotes the position and orientation of the ith mobile robot, ω i = [ ω R , i , ω L , i ] T ${{{\bm \omega }}}_i = {[{\omega }_{{\rm{R}},\,i},{\omega }_{{\rm{L}},\,i}]}^T$ denotes the angular velocities of the left and right wheels, τ i = [ τ R , i , τ L , i ] T ${{{\bm \tau }}}_i = {[{\tau }_{{\rm{R}},\,i},{\tau }_{{\rm{L}},\,i}]}^T$ represents the control torques applied to the wheels, and M is the symmetric and positive definite inertia matrix, C ( p ̇ r , i ) ${{\bf C}}({{{\dot{\bf p}}}}_{{\rm{r}},\,i})$ is the centripetal and Coriolis matrix, and D denotes the surface friction. Moreover, i = 1 , … , I $i = 1,\ldots,I$ represents the ith mobile robot and I is the total number of members in a group. The relationship between the angular ( ω R , i , ω L , i ) $({\omega }_{\,\,{\rm{R}},\,i},{\omega }_{\,\,{\rm{L}},\,i})$ and linear velocity ( v i ${v}_i$ ) of the wheels is: [ v i , ω i ] T = B − 1 [ ω R , i , ω L , i ] T B = 1 r i 1 b i 1 − b i $$\begin{equation} \def\eqcellsep{&}\begin{array}{@{}*{3}{c}@{}} {{{[{v}_i,{\omega }_i]}}^T = {{{\bf B}}}^{ - 1}{{[{\omega }_{{\rm{R,}}\,i},{\omega }_{{\rm{L}},\,i}]}}^T}&{{{\bf B}} = \dfrac{1}{{{r}_i}}\left[ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} 1&{{b}_i}\\[5pt] 1&{ - {b}_i} \end{array} } \right]} \end{array} \end{equation}$$ (2)where r i ${r}_i$ is the radius of the wheel and b i ${b}_i$ is the half-width of the ith mobile robot. The kinematic equation of the robot can be expressed as: d d t x r , i y r , i ϕ r , i = cos ϕ r , i 0 sin ϕ r , i 0 0 1 v i ω i . $$\begin{equation}\frac{d}{{dt}}\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {{x}_{{\rm{r}},\,i}}\\[5pt] {{y}_{{\rm{r}},\,i}}\\[5pt] {{\phi }_{{\rm{r}},\,i}} \end{array} } \right] = \left[ { \def\eqcellsep{&}\begin{array}{@{}*{2}{c}@{}} {\cos {\phi }_{{\rm{r}},\,i}}&\quad 0\\[5pt] {\sin {\phi }_{{\rm{r}},\,i}}&\quad 0\\[5pt] 0&\quad 1 \end{array} } \right]\left[ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {{v}_i}\\[5pt] {{\omega }_i} \end{array} } \right].\end{equation}$$ (3) Path planner Path P of the ith robot is defined as P i = [ p 1 , i , … , p T , i ] T ${{{\bf P}}}_i = {[{{{\bf p}}}_{1,i},\ldots,{{{\bf p}}}_{{\rm{T}},i}]}^T$ , where p 1 , i = [ x 1 , i , y 1 , i ] T ${{{\bf p}}}_{1,i} = {[{x}_{1,i},{y}_{1,i}]}^T$ is starting position and p T , i = [ x T , i , y T , i ] T ${{{\bf p}}}_{{\rm{T}},i} = {[{x}_{{\rm{T}},i},{y}_{{\rm{T}},i}]}^T$ is the target position that the ith mobile robot must eventually reach. In literature, the path planning is examined from different criteria such as distance, time, and energy [39]. Here, the path is optimized using distance and energy criterion. The current position of the path planner unit for the ith robot is defined as p n , i ≔ [ x n , i , y n , i ] T ${{{\bf p}}}_{n{\rm{,}}\,i}\coloneq {[{x}_{n{\rm{,}}\,i},{y}_{n{\rm{,}}\,i}]}^T$ . Then, according to the local information and the current position of the ith robot at each step p r , i ( n ) = [ x r , i ( n ) , y r , i ( n ) ] T ${{{\bf p}}}_{{\rm{r}},\,i}(n) = {[{x}_{{\rm{r}},\,i}(n),{y}_{{\rm{r}},\,i}(n)]}^T$ , the next planned position is defined as p n + 1 , i = [ x n + 1 , i , y n + 1 , i ] T ${{{\bf p}}}_{n + 1,\,i} = {[{x}_{n + 1,\,i},{y}_{n + 1,\,i}]}^T$ . Note that n = 1 , … , N $n = 1,\ldots,N$ (where p N , i = p T , i ${{{\bf p}}}_{N,i} = {{{\bf p}}}_{T,i}$ ) indicates the path planning steps. To plan an online and optimal path in an unknown environment, the following assumptions are made: Assumption 1.The ith mobile robot in its current position can only detect obstacles within its detection area R Detection area , i ${R}_{{\rm{Detection}}\,{\rm{area}},\,i}$ . Assumption 2.Mobile robots can only detect and move in a direction with an interval of [ ϕ r , i ( n ) − 90 ∘ , ϕ r , i ( n ) + 90 ∘ ] $[{\phi }_{r,\,i}(n) - {90}^ \circ ,{\phi }_{r,\,i}(n) + {90}^ \circ ]$ . Assumption 3.Group members only share their current and next positions, and the main target's position (cooperating to find the main target faster). Assumption 4.Cartesian coordinate system with two dimensions is used to express the position of each robot. 3 PATH PLANNING The proposed path-planning unit generates a sequence of suitable positions (using suitable actions or directions of the robot) such that "target tracking" and "collision avoidance" are obtained at the same time. In addition, the path is optimal in terms of local information. Figure 1 shows different parts of the mobile robot motion. According to the problem definition, the output of the path-planning unit is obtained as a suitable action by Improved-QL and/or A-APF method. The path smoother unit reduces the energy consumption of the robot. Then, its output is transferred to the motion control to navigate the mobile robot. These parts will be explained in the following subsections in details. FIGURE 1Open in figure viewerPowerPoint Path planning unit for the ith mobile robot. Improved Q-Learning Here, we utilize the CQL structure as a part of path planning unit. Due to the decentralized architecture, a specific Q-table will be constructed for every robot. In the Q-table, the number of rows is l (allowable actions A i = { a 1 , … , a l } ${{\rm{A}}}_i = \{ {{a}_1,{\rm{ }} \ldots ,{\rm{ }}{a}_l} \}$ ) and the number of columns is m (possible states { S 1 , … , S m } ) $\{ {S}_1, \ldots ,{S}_m\} )$ . Hence, the dimensions of the Q-tables are m×l. The types of states are defined as: S i = safe state R Detection area , i ≤ ρ r _ obs , i non − safe state R Detection area , i > ρ r _ obs , i winning state ρ r _ T , i ≤ ε RL . T failure state ρ r _ obs , i ≤ ε RL . o $$\begin{equation}{S}_i = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{2}{l}@{}} {{\rm{safe state}}}&\quad {{R}_{{\rm{Detection area,}}i} \le {\rho }_{{\rm{r\_obs}},i}}\\[3pt] {{\rm{non - safe state}}}&\quad {{R}_{{\rm{Detection area,}}i} > {\rho }_{{\rm{r\_obs}},i}}\\[3pt] {{\rm{winning state}}}&\quad {{\rho }_{{\rm{r\_T}},i} \le {\varepsilon }_{{\rm{RL}}{\rm{.T}}}}\\[3pt] {{\rm{failure state}}}&\quad {{\rho }_{{\rm{r\_obs}},i} \le {\varepsilon }_{{\rm{RL}}{\rm{.o}}}} \end{array} } \right.\end{equation}$$ (4)where ρ r _ obs , i = min ∥ p r , i − o ∥ ${\rho }_{{\rm{r\_obs}},i} = \min \| {{{{\bf p}}}_{{\rm{r,}}i} - o} \|$ , in which o ∈ C $o \in \,\,C$ indicates the Euclidean distance from the ith mobile robot to the nearest obstacle, C = { o 1 , … , o J } $C = \{ {o}_1,\ldots,{o}_J\} $ is the set of J observed obstacles, ρ r _ T , i = ∥ p r , i − p T ∥ ${\rho }_{{\rm{r\_T}},i} = \| {{{{\bf p}}}_{{\rm{r,}}i} - {{{\bf p}}}_{\rm{T}}} \|$ indicates the Euclidean distance from the ith mobile robot to the target, εRL.T and εRL.o are the target area and the collision area, respectively. The value of the selected action aRL,i = an, i in state Sn, i is updated as: Q n , i ( S n , i , a n , i ) = Q n , i ( S n , i , a n , i ) + α i R n , i + γ i max a ′ Q n , i ( S n + 1 , i , a ′ ) − Q n , i ( S n , i , a n , i ) $$\begin{eqnarray} &&{Q}_{n,\,i}({S}_{n,\,i},{a}_{n,\,i})\, = \,{Q}_{n,\,i}({S}_{n,\,i},{a}_{n,\,i})\,\nonumber\\ &&\quad + \,{\alpha }_i\,\left[ {{R}_{n,i}\, + \,{\gamma }_i\,\mathop {\max }\limits_{a^{\prime}} \,{Q}_{n,\,i}({S}_{n + 1,\,i},a^{\prime})\, - \,{Q}_{n,\,i}({S}_{n,\,i},{a}_{n,\,i})} \right]\nonumber\\ \end{eqnarray}$$ (5)where 0 ≤ α i < 1 $0 \le \,{\alpha }_i < 1$ , 0 < γ i < 1 $0 < \,{\gamma }_i\, < 1$ and Sn+1, i denotes the next state by performing action an,i at state Sn,i. The main problems of Reinforcement Learning (RL) are (1) wrong decisions at the beginning of the learning process and (2) slow convergence. Slow convergence increases the number of primary collisions and primary wrong decisions cause collision with obstacles. When the number of actions and states of the Q-table increase, the problem get worse due to the following reasons: The Q-table is empty at the beginning of training. Undefined states in the Q-table occur. The states in the Q-table are independent. Deep Reinforcement Learning (DRL), as a remarkable and excellent approach, has been proposed in literature to solve Problems (2) and (3) [39]. However, Deep Learning (DL) is slow for the decision-making process and it should be used only when necessary [17]. Nowadays researchers are developing DL algorithms with significantly improved results. In this manuscript, a supervisor is proposed to overcome these three problems such that even if an undefined state in the Q-table occurs, it is still possible to select the actions that prevent collisions. The supervisor is an Adaptive Artificial Potential Field (A-APF) method, which will be explained in Section 3.2. Before performing a RL , i $a{\,}_{{\rm{RL}}\,,\,i}$ , its quality must be checked, and if a RL , i $a{\,}_{{\rm{RL}}\,,\,i}$ causes a collision, selected action will be corrected by the supervisor. The quality estimation of a RL , i $a{\,}_{{\rm{RL}}\,,\,i}$ is as follows: x Predicted , i ( n ) = x i ( n ) + σ i cos φ ̂ ( a RL , i ) y Predicted , i ( n ) = y i ( n ) + σ i sin φ ̂ ( a RL , i ) $$\begin{equation}\left\{ { \def\eqcellsep{&}\begin{array}{@{}*{1}{c}@{}} {{x}_{{\rm{Predicted,}}\,i}(n) = {x}_i(n) + {\sigma }_i\cos \left( {\hat{\varphi }(a{\,}_{{\rm{RL}}\,,\,i})} \right)}\\[6pt] {{y}_{{\rm{Predicted,}}\,i}(n) = {y}_i(n) + {\sigma }_i\sin \left( {\hat{\varphi }(a{\,}_{{\rm{RL}}\,,\,i})} \right)} \end{array} } \right.\end{equation}$$ (6)where p Predicted , i ( n ) ≔ [ x Predicted , i ( n ) , y Predicted , i ( n ) ] T ${{{\bf p}}}_{{\rm{Predicted,}}\,i}(n)\coloneq {[{x}_{{\rm{Predicted,}}\,i}(n),{y}_{{\rm{Predicted,}}\,i}(n)]}^T$ and σ i ${\sigma }_i$ are the predicted position and the movement step size of the ith robot when the action a RL , i $a{\,}_{{\rm{RL}}\,,\,i}$ is performed, respectively. Note that in defining the set of allowable actions in QL are defined from the beginning. Therefore, the relative direction to ( ϕ r , i ( n ) − 90 ∘ ) $(\phi {\,}_{{\rm{r}},\,i}(n) - {90}^ \circ )$ axis of action a RL , i ∈ A i $a{\,}_{{\rm{RL}}\,,\,i} \in {A}_i$ is defined as φ ( a RL , i ) ; $\varphi (a{\,}_{{\rm{RL}}\,,\,i});$ and according to Assumption 2, 0 ∘ ≤ φ ( a RL , i ) ≤ 180 ∘ ${0}^ \circ \le \varphi (a{\,}_{{\rm{RL}}\,,\,i}) \le {180}^ \circ $ . Moreover, φ ̂ ( a RL , i ) = ( ϕ r , i ( n ) − 90 ∘ ) + φ ( a RL , i ) $\hat{\varphi }(a{\,}_{{\rm{RL}}\,,\,i}) = (\phi {\,}_{{\rm{r}},\,i}(n) - {90}^ \circ ) + \varphi (a{\,}_{{\rm{RL}}\,,\,i})$ is the direction of action a RL , i $a{\,}_{{\rm{RL}}\,,\,i}$ from the x-axis. The Dyna-QL has been introduced to improve convergence of the CQL [31]. In this manuscript, we introduce a novel concept called the feature matrix (M) to increase the speed of convergence and solve all the aforementioned problems. Note that for defining the feature matrix, the state corresponding to each row of this matrix is known and completely defined according to the position of the obstacles around the mobile robot at the beginning. This matrix defines the relationship between a "non-safe states" defined as N S = { S ns , 1 ⋯ S ns , K } $NS = \{ {S}_{{\rm{ns,1}}} \cdots {S}_{{\rm{ns,}}K}\} $ and the "obstacles position", where K < m (K is the number of non-safe states in the Q-table and m is the possible states for the robot). Hence, matrix M shows the "Common Features" between non-safe states. In other words, it represents similarities of states in terms of obstacles position. Using M, the experience of current states (i.e. Sns,k (k = 1, …, K)) can be generalized to (NS-Sns,k) that has at least one common feature with Sns,k. The first step to define M, is the definition of non-safe states. When C ≠ ∅ $C \ne \emptyset $ , the relative direction of the robot to the obstacle is determined as follows: ω r _ o j = Forward ϕ r , i ( n ) − ϕ o j ( n ) < 30 ∘ Right − Forward 30 ∘ ≤ ϕ r , i ( n ) − ϕ o j ( n ) ≤ 60 ∘ Left − Forward − 60 ∘ ≤ ϕ r , i ( n ) − ϕ o j ( n ) ≤ − 30 ∘ Right ϕ r , i ( n ) − ϕ o j ( n ) ≥ 60 ∘ Left ϕ r , i ( n ) − ϕ o j ( n ) ≤ − 60 ∘ $$\begin{equation}{\omega }_{{\rm{r\_o}}_j} = \left\{ { \def\eqcellsep{&}\begin{array}{@{}*{2}{l}@{}} {{\rm{Forward}}}&{\left| {{\phi }_{{\rm{r}},\,i}(n) - {\phi }_{{{\rm{o}}}_j}(n)} \right| < {{30}}^ \circ }\\[6pt] {{\rm{Right - Forward}}}&{{{30}}^ \circ \le {\phi }_{{\rm{r}},\,i}(n) - {\phi }_{{{\rm{o}}}_j}(n) \le {{60}}^ \circ }\\[6pt] {{\rm{Left - Forward}}}&{ - {{60}}^ \circ \le {\phi }_{{\rm{r}},\,i}(n) - {\phi }_{{{\rm{o}}}_j}(n) \le - {{30}}^ \circ }\\[6pt] {{\rm{Right}}}&{{\phi }_{{\rm{r}},\,i}(n) - {\phi }_{{{\rm{o}}}_j}(n) \ge {{60}}^ \circ }\\[6pt] {{\rm{Left}}}&{{\phi }_{{\rm{r}},\,i}(n) - {\phi }_{{{\rm{o}}}_j}(n) \le - {{60}}^ \circ } \end{array} } \right.\end{equation}$$ (7) ϕ o j ( n ) = tan − 1 y o j − y r , i x o j − x r , i $$\begin{equation}{\phi }_{{{\rm{o}}}_j}(n) = {\tan }^{ - 1}\left( {\frac{{{y}_{{{\rm{o}}}_j} - {y}_{r,i}}}{{{x}_{{{\rm{o}}}_j} - {x}_{r,i}}}} \right)\end{equation}$$ (8)where ϕ o j ( n ) ${\phi }_{\,{{\rm{o}}}_j}(n)$ is the direction of the obstacle oj ( j = 1 , … , J $j = 1,\ldots,J$ from the x-axis at the nth step, and p o j = [ x o j , y o j ] T ${{{\bf p}}}_{\,{{\rm{o}}}_j} = {[{x}_{{{\rm{o}}}_j},{y}_{{{\rm{o}}}_j}]}^T$ is the position of the jth detect