# Fast 3D Semantic Mapping in Road Scenes +
## 3 Problem Formulation  
### Notation  
The target is to estimate the 3D semantic map \(\mathcal{M}\) comprising of a pose-graph of keyframes with semantic map from a monocular camera. The 3D map \(\mathcal{M}\) is reconstructed by the estimation of depth and poses, where each 3D point \(\mathbf{P}\) can be labeled as one of the solid semantic objects in the label space \(\mathcal{L}=\{l_{1},l_{2},\ldots,l_{k}\}\) like _Road_, _Building_, _Tree_, etc. We use \(\mathbf{X}=\{X_{1},X_{2},\ldots,X_{M}\}\) to denote the set of random variables corresponding to the 3D points \(\mathbf{P}_{i}:i\in\{1,\ldots,M\}\), where each variable \(X_{i}\in\mathbf{X}\) take a value \(l_{i}\) from the predefined label space \(\mathcal{L}\).  
Let \(I_{i}:\Omega_{i}\rightarrow\mathbb{R}^{3}\) symbolize an \(\mathrm{H}\times\mathrm{W}\) RGB image of an input sequence at the frame indexed by \(i\). Keyframes are extracted from the sequence in light of camera's pose \(\mathbf{T}_{i}^{j}\) at the \(j\)th frame with respect to the previous \(i\)th keyframe. We define the \(i\)th keyframe to be a tuple \(K_{i}=(I_{i},D_{i},V_{i},S_{i})\), where \(D_{i}:\Omega_{D_{i}}\rightarrow\mathbb{R}\) is the full-resolution inverse depth map associated with image \(I_{i}\), and \(V_{i}:\Omega_{V_{i}}\rightarrow\mathbb{R}\) is the variance associated with the inverse depth map. The inverse depth model is a better description for visual depth estimation, which assumes normally distributed [38]. The inverse depth map and the variance map are defined in the subset of pixels as \(\Omega_{D_{i}},\Omega_{V_{i}}\subset\Omega_{i}\), which means semi-dense, available  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Method** & **M** & **S/D** & **L** & **O** & **I** & **SDT** & **RT** \\ \hline [PERSON] et al. [37] & & & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) \\ [PERSON] et al. [35] & & \(\surd\) & & \(\surd\) & & & \\ [PERSON] et al. [9] & & \(\surd\) & & \(\surd\) & & & \(\surd\) \\ [PERSON] et al. [28] & \(\surd\) & & & \(\surd\) & & \(\surd\) & \\ [PERSON] et al. [30] & & \(\surd\) & & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) \\ [PERSON] et al. [8] & & \(\surd\) & & & & & \(\surd\) \\ [PERSON] et al. [10] & & \(\surd\) & & \(\surd\) & \(\surd\) & \(\surd\) \\ Ours & \(\surd\) & & & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison with some related work: \(\mathrm{M}=\mathrm{monocular}\) camera, \(\mathrm{S/D}=\mathrm{stereo/depth}\) camera, \(\mathrm{L}=\mathrm{lidar}\), \(\mathrm{O}=\mathrm{outdoor}\), \(\mathrm{I}=\mathrm{incremental}\), \(\mathrm{SDT}=\mathrm{sparse}\) data structures, \(\mathrm{RT}=\mathrm{real}\) time.  
for certain image regions of large intensity gradient. The symbol \(S_{i}:\Omega_{S_{i}}\rightarrow\mathbb{R}^{|\mathcal{L}|}\) represents the full-resolution semantic map with the probability of each object class.  
The keyframes are consecutively stacked in a pose-graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\), where \(\mathcal{V}=\{\mathcal{K}_{0},\dots,\mathcal{K}_{n}\}\) is the set of keyframes and \(\mathcal{E}=\{\mathbf{S}_{i}^{j}\in\text{Sim}(3):\mathcal{K}_{i},\mathcal{K}_ {j}\in\mathcal{V}\}\) is the set of constraint factors. Each \(\mathbf{S}_{i}^{j}=(\mathbf{T}_{i}^{j},s_{i}^{j})\) consists of a camera's pose \(\mathbf{T}_{i}^{j}=\binom{R\,t}{01}\) from the keyframe \(i\)th to the keyframe \(j\)th, and scale factor \(s_{i}^{j}>0\). In reference to world frame \(W\), regarded as the pose of the first keyframe \(\mathcal{K}_{0}\), the pose of the keyframe \(i\)th is denoted as \(\mathbf{T}_{W}^{i}\). For a sequence of keyframes (\(n\) keyframes), we get the \(n\)th keyframe's pose \(\mathbf{T}_{W}^{n}=\prod_{\mathbf{I}}^{n}\mathbf{T}_{k-1}^{k}\).