# Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
## 4 The Proposed Occupancy-Reward-Driven Exploration
### Reinforcement Learning Environment of the Mobile-Robot Exploration  
The proposed RL environment, which is presented in Algorithm 1, has a class structure with property values and several certain functions [62]. It is presented in Algorithm 1. The constructor function is the main one, in which the action and observation specifications are defined with their maximum and minimum value ranges. The reset function is called every time the exploration is launched and when the episode is finished during training. In our custom RL environment, in lines 8-15, the reset function sets the map visualization to  
Figure 5: The MDP model of the mobile-robot exploration task.  
the initial uncertainties, sets the robot's initial position, resets the observation values, and activates the ROS interface for the laser scan data and velocity commands.  
```
1:classdef ExplorationRLEnv
2:propertiesmaxRange = 4.095
3:methods
4:functionconstructor
5:define observation \(O\) with lower and upper limit values
6:define action \(a\) with lower and upper limit values
7:end
8:function reset
9:initialize robotPose, observation
10:map = occupancyMap
11:enableROSInterface
12:isDone = false
13:isBumpedObs = false
14:reward = 0;
15:end
16:function [observation, reward, isDone] = step (constructor, action)
17:scanMsg = receive(scanSub)
18:scan = lidarScan(scanMsg)
19:observation = scan.Ranges
20:insertRay(map, robotPose, scan, maxRange)
21:velMsg.Linear.X = action(1)
22:velMsg.Angular.Z = action(2)
23:send (velPub, velMsg)
24:if the last 3 robotPose values are the same
25:isBumpedObs = true
26:end
27:ifisMapExplored = true \(||\) isBumpedObs = true
28:isDone = true
29:resetSimulation
30:clear('node')
31:else
32:isDone = false
33:end
34:if t is equal 1
35:exp = totalMapValues
36:else
37:exp = previousReward-totalMapValues
38:previousReward = totalMapValues
39:end
40:reward at time \(t\)
41:end
42:end
43:end
```  
**Algorithm 1**The RL environment for the mobile-robot exploration  
Another required function for the RL environment is the step function. The whole process of an episode is carried out in the step function. The action as the input parameter of the step function is taken from the actor-critic neural network of the DDPG agent at each iteration and passed to the robot as the commands of the linear and angular velocities by the ROS publisher node (lines 21-23). The observation as the output parameter receives the sensing laser ranges from the sensor and inserts the rays into the occupancy map in lines 17-20.  
Lines 24-26 show the robot's collision with obstacles in the mapping. The logic of these lines is such that if the values of the robot position are not changed in the last three iterations, the robot has hit an obstacle and cannot avoid it. In essence, the robot does not try to avoid the obstacle as usually occurs using classical algorithms. It only detects the collisions as a bad event that should not be repeated in the next episodes. As the simulation results will show in the next section, this straightforward logic satisfies and can operate correctly for the mobile-robot motion. If the experiment runs in real-world conditions, then a bumper sensor can be used on the robot to detect the collision with an obstacle.  
In lines 27-33, the _isDone_ as the output parameter is applied, which is a flag of the episode states. When the flag has true value, the current episode must be finished, and the exploration process should be aborted. This happens in two cases: a map is fully explored or the robot hits an obstacle.  
The reward function of lines 34-40 is discussed next in detail.