# Reinforcement-Learning-Based Path Planning: A Reward Function Strategy
## 2 Proposed Methodology
### Reinforcement Learning and Q-Learning Algorithm
In this algorithm, the policy necessary to compute the function, \(V(s)\), from the value of the state is evaluated:  
\[V(s)\leftarrow\sum_{s^{\prime},r^{\prime}}p\big{(}s^{\prime},r|s,\pi(s)\big{)} \big{[}r+\gamma\text{V}(s^{\prime})\big{]} \tag{10}\]  
The improved policy is calculated using an anticipated view to replace the initial value of the policy, \(\pi(s)\):  
\[\pi(s)=arg\max_{u}\sum_{s^{\prime},r^{\prime}}p\big{(}s^{\prime},r|s,a\big{)} \big{[}r+\gamma\text{V}(s^{\prime})\big{]} \tag{11}\]  
Algorithm 1 shows a complete version of policy iteration [15]. ```
1: Initialize
2:\(V(s)\in\mathbb{R}\) and \(\pi(s)\in\mathcal{A}\) arbitrarily, for all states \(s\in\mathcal{S}\); \(V(final)=0\)
3: Policy Evaluation
4: Loop:
5:\(\
abla\gets 0\)
6:\(\text{Loor for each }s\in\mathcal{S}\):
7:\(v\gets V(s)\)
8:\(V(s)\leftarrow\sum\limits_{s^{\prime},r^{\prime}}p(s^{\prime},r|s,\pi(s))[r+ \gamma\text{V}(s^{\prime})]\)
9:Until \(\
abla<\theta\)
10:Policy Improvement
11:\(policy-stable=True\)
13:Loop for each \(s\in\mathcal{S}\):
14:previous action \(\leftarrow\pi(s)\)
15:\(\pi(s)=arg\max\limits_{s}\sum\limits_{s^{\prime},r^{\prime}}p(s^{\prime},r|s,a) [r+\gamma\text{V}(s^{\prime})]\)
16:
17: If the previous action \(\
eq\pi(s)\), then policy \(-stable=False\)
18: If policy \(-stable=True\), then, stop and return \(V\) and \(\pi\)
```  
**Algorithm 1**Policy Iteration  
Table 1 shows the hyperparameters used in the policy iteration algorithm. A high discount rate is considered for the agent to pursue long-term rewards. On the other hand, a low learning rate is considered to guarantee convergence, which implies a higher number of iterations. For practical purposes, the policy iteration technique will be referred to hereafter as RL to differentiate it from the Q-Learning (QL) and Deep Q-Learning (DQL) techniques. The algorithm was developed with Python 3.11.7 programming using an Intel Core i5-10300 processor with 4 cores up to 2.50 GHz, an NVIDIA GeForce GTX 1650 graphic card, 16 GB of RAM, and Windows 11. Q-Learning is a method based on values and policies, which uses a Q-table to update the action-value function at each step rather than the end of each episode. Each row contains a state-action value. Q represents the \"Quality\" of each action performed in that \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Episodes & 100 \\ Learning rate & \(1\times 10^{-6}\) \\ Average iteration & 550 \\ Discount rate, \(\gamma\) & 0.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Training the hyperparameters of the policy iteration algorithm. The architecture employed to approximate the Q function in the Deep Q-Learning algorithm is shown in Figure 6. It is a fully connected neural network with the following characteristics:  
* Input layer: two fully connected neurons, one for the \(x\) position and one for the \(y\) position. * Hidden layers: two fully connected layers with 64 neurons and the ReLU activation function. * Output layer: a fully connected dense layer, which has 64 neurons as the input to the last hidden layer. It produces a vector \((1,\ 8)\) that corresponds to the Q values for each possible action in an environment. Table 2 describes the hyperparameters selected for this study. Selecting a high discount rate allows the agent to pursue a greater reward in the long term. As the discount rate decreases, the agent will pursue short-term payoffs. A low learning rate translates into slow convergence, so a high number of episodes is necessary to guarantee convergence with a low learning rate. A high learning rate can result in network divergence. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Episodes & 1000 \\ Learning rate, \(\alpha\) & \(1\times 10^{-3}\) \\ Random probability starts & 1 \\ Random probability end & \(1\times 10^{-2}\) \\ Random probability decay & 0.995 \\ Target update frequency, \(\theta\) & 10 \\ Discount rate, \(\gamma\) & 0.99 \\ Memory size & 10,000 \\ \hline \hline \end{tabular}
\end{table}