# A Deep Reinforcement Learning Approach for Active SLAM
## 2 Problem Formulation  
### Traditional Approach  
The active SLAM problem is included within a wider mathematical framework, Partially Observable Markov Decision Processes (POMDP), that generalize such decision making processes under both action and observation uncertainties. A POMDP is formally defined as a 7-tuple \((\mathcal{S},\mathcal{A},\mathcal{T},\Omega,\mathcal{O},\mathcal{R},\gamma)\) consisting of the finite agent's state space \(\mathcal{S}\), a finite set of actions \(\mathcal{A}\), a transition function between states \(\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\Pi(\mathcal{S})\) where \(\Pi(\mathcal{S})\) is the probability distribution over \(\mathcal{S}\), a finite observation space \(\Omega\) and its conditional probabilities \(\mathcal{O}:\mathcal{S}\times\mathcal{A}\rightarrow\Pi(\Omega)\), a reward function \(\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) and the discount factor \(\gamma\) which allows to work with finite rewards even when planning has an infinite time horizon. Every time step, the agent selects an action to execute \(a_{t}\in\mathcal{A}\) based on the current policy \(\pi\), generating a transition from \(s_{t}\) to \(s_{t+1}\equiv s^{\prime}\), both contained in \(\mathcal{S}\), where a new observation \(o_{t}\in\Omega\) is made. The goal of the agent is to find the optimal policy \(\pi^{*}\) that maximizes a metric of the reward function, usually defined as the future expected discounted reward \(R_{t}\triangleq\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a_{t})\). Resolution of POMDP can be based on several methods, such as Monte Carlo, dynamic programming or temporal differences. Reinforcement Learning (RL) algorithms, based on the latter and Bellman equations, solve the problem by iterating either directly over the policy or over a value function. This function is just a mapping to real numbers from the value encoded in being at a certain state, or in a more complex fashion, from the value of being at a certain state and also executing a certain action according to \(\pi\); and are denoted as \(V(s)\) and \(Q(s,a)\), respectively. Both value- and policy-based trends allow a POMDP resolution when the process itself (i.e., \(\mathcal{T}\) and \(\mathcal{O}\)) is not known nor modelled (model-free). In any case, RL algorithms are just recurrences over the expected rewards, as shown in Algorithm 1 of Q-learning, or more specifically, in Equation (6)-the Q-function's Bellman update. The design of \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **Task** & & **Reward** & **Complex** & **Generalization** & **Partial** \\  & **Accomplished** & **Architecture** & **Design** & **Environment** & **Unknown env.** & **Observability** \\ \hline [PERSON] and [PERSON] [19] & Navigation & DQN & Extrinsic & ✓ & & \\ [PERSON] et al. [20] & Navigation & ASC & Extrinsic & ✓ & & \\ [PERSON] et al. [22] & Navigation & D3 QN & Extrinsic & ✓ & & \\ [PERSON] et al. [26] & Goal navigation & ASC & Intrinsic & ✓ & & \\ [PERSON] and [PERSON] [27] & Goal navigation & ASC & Intrinsic & ✓ & & \\ [PERSON] et al. [35] & Goal navigation & ADDTG & Intrinsic & ✓ & ✓ & \\ [PERSON] et al. [33] & Frontier selection & ASC & Intrinsic & ✓ & \\ [PERSON] et al. [34] & Frontier selection & ASC & Intrinsic & ✓ & ✓ & ✓ \\ [PERSON] et al. [31] & Active localization & A2C & Posterior belief & ✓ & ✓ & \\ [PERSON] et al. [30] & Active localization & ASC & Posterior belief & ✓ & ✓ & \\ [PERSON] et al. [29] & Exploration & DQN & Entropy & ✓ & ✓ \\ [PERSON] et al. [32] & Active SLAM & DQN+GNN & T-opt & & ✓ & ✓ \\ Ours & Active SLAM & D3 QN & D-opt & ✓ & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison with prior methods. the reward function is then extremely important in order to achieve the desired goal, as it occurs in optimization problems with cost functions.