# Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
## 3 Background
### Reinforcement Learning  
Reinforcement Learning (RL) is a goal-oriented approach that extracts successful actions in an area of concern during its training. This method allows a robot to make correct decisions for a task without human intervention. The RL consists of two main parts: an agent and the RL environment (Figure 2).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{2}{c}{**RL Algorithm**} & \multicolumn{2}{c}{**RL Environment**} & \multirow{2}{*}{**Map Representation**} \\ \cline{2-3} \cline{5-6}  & **Approximator** & **Agent** & **Input** & **Output** & **Reward** \\ \hline [PERSON]. [PERSON] & Support Vector & Policy Search & & & \\ Machine Policy & Dynamic & Laser sensor & Discrete action & Squared error & Occupancy map \\ et al. [35] & Learning & Programming & & & \\ \hline [PERSON] & & Deep & & & \\ et al. [36] & CNN & Q-network & RGB-D camera & Discrete actions & Keep moving is value 1, & Corridor environment, \\ et al. [36] & CNN & Q-network & RGB-D camera & & & \\ et al. [36] & CNN & Q-network & & & \\ et al. [36] & CNN & Q-network & & & \\ et al. [37] & Actor-critic & & & & & \\ et al. [37] & network & A3C agent & Laser sensor & Continuous actions & Intrinsic reward & \begin{tabular}{c} Simulated \\ environment, 3 maps \\ with different \\ floor plans \\ \end{tabular} \\ \hline [PERSON] & Actor-critic & & & & & \\ et al. [41] & network with & DDPG agent & Laser sensor, target points & \
\begin{tabular}{c} Continuous \\ actions: linear \\ and angular \\ velocities \\ \end{tabular} & & & & \\ et al. [42] & Gazebo & & & & \\ et al. [43] & Gazebo & & & & \\ et al. [44] & Graph neural & Deep & & & \\ et al. [42] & networks & Q-network & Laser sensor & Sensing action & Raw reward & Occupancy map \\ et al. [42] & FCQN with & Deep & & & \\ et al. [43] & FCQN with & Q-network & Partial map & Discrete action & Heuristic reward function & Occupancy map, ROS \\ et al. [43] & Actor-critic & & & & \\ et al. [44] & network & Fast Hybrid & & Continuous actions: linear &
\begin{tabular}{c} Goal reached is \\ value 20, \\ collision is \\ value of \(-20\) \\ \end{tabular} & Simulated and real \\ et al. [45] & Actor-critic & & & & \\ et al. [46] & Network & A3C agent & & & \\ et al. [47] & Actor-critic & & & & \\ et al. [48] & & & & \\ et al. [49] & & & & \\ et al. [48] & Actor-critic & & & & \\ et al. [49] & & & & \\ et al. [48] & & & & \\ et al. [49] & & & & \\ et al.  
The process of RL starts with the environment sending its initial observation (or so-called state) to the agent. According to its computation, the agent makes the action in response to this observation. By this, the action changes the environment, which can be good or bad. Then, the environment sends a new observation and a reward for the last action to the agent. It receives and updates its knowledge and then takes the next action based on the computational analysis. The process repeats in this manner until the environment gives the signal of the end of an episode.  
The agent can be seen as a computational controller. It contains a policy and a learning algorithm. The policy is a function approximator (deep neural network), which selects appropriate actions with regards to the observations from the RL environment. The learning algorithm component is to search an optimal policy by maximizing the cumulative reward. It continuously updates the policy parameters based on reward, actions, and observations.  
In this study, we applied the actor-critic agent belonging to the class of RL algorithms. There are several known varieties of actor-critic agents, which use either a deterministic actor or stochastic actor, with Q-value critic or a value critic. The difference among them is in the manner of how the data of an actor and critic are updated in the process.