# Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
## 3 Background
### Actor-Critic Deep Deterministic Policy Gradient Algorithm  
The DDPG algorithm is a model-free, online, off-policy RL method. The DDPG agent uses a deterministic actor and Q-value critic. The DDPG agent searches for an optimal policy that maximizes the expected cumulative long-term reward. It can be applied only for an RL environment with continuous action spaces [59]. In the DDPG algorithm, the actor-critic architecture applies four function approximations: deterministic actor network, target actor network, critic network, and target critic network. Considering each separately, Figure 2(a) represents the actor architecture, in which the actor \(\mu(O,\ \theta^{n})\) directly maps the observations \(O_{i}\) to corresponding actions \(a_{i}\), which maximizes the long-term reward \(R\). In Figure 2(b), the critic \(Q(O,\ A,\ \theta^{Q})\) takes actions and observations and returns the corresponding expectation \(Q\) of long-term reward. The parameters \(\theta^{n}\) and \(\theta^{Q}\) are network weights. The general actor-critic architecture is represented in Figure 2(c), in which the RL environment passes the observation to the actor and critic. The actor determines the action and sends it to the critic. That is, the critic estimates the value of how much reward the agent will obtain from this situation. Combining the value with the reward \(R\) gives the estimated value of receiving the current observation and making the current action. Figure 2: RL system. The actor target network \(\mu^{\prime}\Big{(}O,\,\theta^{\mu^{\prime}}\Big{)}\) and critic target network \(Q^{\prime}\Big{(}O,\,A,\,\theta^{Q^{\prime}}\Big{)}\) are time\(-\)delayed copies of their original networks that slowly track the actor and critic networks. The main role of the target networks is to improve the stability in the learning by periodically saving the actor and critic parameters. The weight parameters \(\theta^{\mu^{\prime}}\) and \(\theta^{Q^{\prime}}\) of the actor and critic target networks are updated by the equations below for \(\tau\ll 1\):  
\[\theta^{Q^{\prime}} \leftarrow \tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}} \tag{1}\] \[\theta^{\mu^{\prime}} \leftarrow \tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}} \tag{2}\]  
The critic side of the DDPG algorithm updates the critic by minimizing the loss between target \(y\) and the original \(Q\) value of the critic network through the following equation:  
\[L=\frac{1}{M}\,\sum_{i}\Bigl{(}y_{i}-Q\Big{(}O_{i},\,a_{i},\theta^{Q}\Big{)} \Big{)}^{2} \tag{3}\]  
The target \(y\) is calculated using the Bellman equation:  
\[y_{i}=r_{i}+\gamma Q^{\prime}\Big{(}O_{i+1},\,\,\mu^{\prime}\Big{(}O_{i+1}, \theta^{\mu^{\prime}}\Big{)},\theta^{Q^{\prime}}\Big{)} \tag{4}\]  
where \(Q^{\prime}\) is the next \(Q\) value obtained from target networks, \(\gamma\) is the discount factor, and \(r\) is the reward at time \(i\). The actor side of the DDPG algorithm updates the actor parameters using the sampled policy gradient using the following equation:  
\[\
abla_{\theta^{\mu}}\,J\,\,\approx\,\,\frac{1}{M}\sum_{i}\
abla_{a}Q\Big{(}O _{i},\,A,\,\theta^{Q}\Big{)}\bigg{|}_{A=\mu(O_{i},\,\theta^{\mu})}\cdot\
abla_ {\theta^{\mu}}\mu(O_{i},\theta^{\mu}) \tag{5}\]  
Here, \(\
abla_{a}Q\big{(}O_{i},\,A,\,\theta^{Q}\big{)}\) is the gradient of the critic output with respect to the action computed by the actor network. The gradient of the actor output is \(\
abla_{\theta^{\mu}}\mu(O_{i},\theta^{\mu})\) with respect to the actor parameters [60]. In the discussion of the DDPG algorithm, which incorporates DQN [61], two trick techniques with data, the replay buffer and the minibatch, cannot be excluded. The replay buffer is like a data stack with 'last in--first out' principal operation. The experience tuples \((O_{i},\,a_{i},\,r_{i},O_{i+1})\) from the RL environment are added to the end of the buffer so that the oldest experience is pushed out. The replay buffer can have a large size, and the large size should be set.