2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
Madrid, Spain, October 1-5, 2018

Fast and Accurate Semantic Mapping through Geometric-based
Incremental Segmentation
Yoshikatsu Nakajima1 , Keisuke Tateno2 , Federico Tombari2 and Hideo Saito1
Abstract— We propose an efficient and scalable method forincrementally building a dense, semantically annotated 3D map
in real-time. The proposed method assigns class probabilities toeach region, not each element (e.g., surfel and voxel), of the 3D
map which is built up through a robust SLAM framework andincrementally segmented with a geometric-based segmentation
method. Differently from all other approaches, our method
has a capability of running at over 30Hz while performing
all processing components, including SLAM, segmentation, 2D
recognition, and updating class probabilities of each segmentation label at every incoming frame, thanks to the high efficiency
that characterizes the computationally intensive stages of ourframework. By utilizing a specifically designed CNN to improve
the frame-wise segmentation result, we can also achieve highaccuracy. We validate our method on the NYUv2 dataset bycomparing with the state of the art in terms of accuracy andcomputational efficiency, and by means of an analysis in termsof time and space complexity.

I. INTRODUCTION
The task of incrementally building a semantically annotated 3D map is a challenging research topic for both
the robotics and computer vision communities. It has awide range of applications including autonomous grasping
and manipulation of objects, scene understanding, robotics
navigation and augmented reality. For this reason, a valuable
research effort is currently undergoing in literature with theaim of developing efficient systems that can scale up to
mobile/embedded architectures while being robust enoughto generalize to unseen environments.
Motivated by the recent developments of deep learning andConvolutional Neural Networks (CNNs) for 3D data, recent
methods have mostly focused on increasing the accuracyof the semantic segmentation map [1], [2], [3]. At thesame time, they still face the critical issue of yielding realtime performance, since such systems are built on a setof computationally demanding processing stages, including
3D reconstruction, camera pose estimation and CNN-based
semantic segmentation. This becomes even more relevantwith regards to embedded and mobile architectures that are
typically employed for the aforementioned applications ofrobotics navigation/grasping and augmented reality.
To achieve real-time performance, some of these methods
suggested to only extract semantic information on a subsetof the input frames. For example, the methods proposed byHermans et al. [4] and McCormac et al. (SemanticFusion)

Fig. 1: Our method achieves accurate semantic mapping(comparable to the state of the art [1], bottom row) while
being more efficient and scalable. It relies on the geometricsegmentation that takes into account semantic informationand thus providing meaningful segments than the method ofTateno et al. [6] (middle row).

[1] achieved, respectively, an output frame-rate of 4Hz and25.3Hz, by running semantic segmentation, respectively,
every 6 and every 10 frames. While such frame skipping
strategy can improve run-time performance, it limits theirrange of application, since it tends to bring in inaccuracies
under fast camera motions.
In this paper, we propose a novel incremental semantic
mapping approach that aims at overcoming such issues byyielding highly accurate semantic scene reconstruction (seebottom row of Fig. 1) in real-time. The framework relies on
effectively combining a reliable camera pose tracking (InfiniTAM v3 [5]), an incremental segmentation approach [6],and an efficient CNN-based semantic segmentation method.
In particular, the 3D map of the scene is built through thefast and robust surfel-based SLAM approach in [7], andgeometric segmentation labels are assigned to each surfel

1 Department of Science and Technology, Keio University, Kanagawa,(Japan) {nakajima, saito}@hvrl.ics.keio.ac.jp
2 Chair for Computer Aided Medical Procedures (CAMP), TU
Munich, Boltzmannstr. 3, 85748 Munich (Germany) {tateno,

tombari}@in.tum.de978-1-5386-8094-0/18/$31.00 ©2018 IEEE385

based on the approach of [6]. Class probabilities of eachlabel are updated through a specifically designed CNN.
We introduce a new probabilistic strategy to deal with oneof the most delicate stages, i.e. class probability assignment.
According to this strategy, and in contrast to conventional semantic mapping methods which assign class probabilities toeach surfel [4], [1], [2], we assign class probabilities to eachsegment. This reduces notably the time complexity since ateach new frame probability distributions need to be updated
for those segments which are visible on the image plane fromthe current camera pose, in contrast to conventional methods
which need to update such probabilities for all surfels on theimage plane. This strategy also reduces notably the spacecomplexity since probability distributions need to be stored
only at each segment rather than each surfel.
In return, the semantic information also improves thegeometric-based segmentation from [6]. By taking into account semantic information, it provides additional edges that
better represent the semantic structure of the scene, hence
allowing to obtain accurate segment regions (see middle rowof Fig. 1). Since smoothing of semantic labels is carried out
at the geometric fusion stage, this allows us to utilize a CNN
with a low resolution (i.e. 40 × 30) output, with a forward
pass requiring only 19ms on an off-the-shelf GPU (i.e., a
GeForce GTX 1080).
The overall framework is capable of working in realtime on off-the-shelf architectures, while the requiring a low
computational complexity with respect to state of the art. In
addition, differently from other methods such as [4], [1], [2],[3], [8], our approach does not require any post-processing
based on, e.g., Conditional Random Field, to refine the outputof the semantic mapping. We demonstrate the effectivenessand efficiency of our approach on a common benchmark, i.e.
the NYUv2 dataset [9], reporting comparable accuracy than
the state-of-the-art approaches while being notably faster andscaling better in terms of memory requirements. In addition,
we also report an analysis in terms of time and spacecomplexity of our method, demonstrating its advantages withrespect to conventional approaches.

even if the method computes the RDF once every 6 input
frames and the CRF once every 30 frames. SemanticFusion[1] employs the CNN model proposed by Noh et al. [11]for 2D semantic segmentation, a Bayesian framework for
2D-3D label transfer, and a CRF for 3D refinement. By usinga CNN to carry out semantic segmentation of each input
frame, the method can achieve a better runtime performance.
However, the CNN still requires 51.2ms and the Bayesian
update scheme requires a further 41.1ms, eventually runningat 25.3Hz by applying these stages once every 10 input
frames.
Other related works include [12], [13], [14] that aim atbuilding a semantic 3D map, although not incrementally.
[12] firstly builds a 3D map of a scene through RGB-D
SLAM framework, then assigns class probabilities to eachpoint of the 3D map by means of a Dense CRF. [13] exploits
relational information derived from the full-scene 3D map for
object labeling relying on a Markov-Random-Field (MRF)based model.
In addition, several methods for recognizing only a part ofthe 3D map without making a dense semantic 3D map havebeen proposed [15], [16], [17], [18]. SLAM++ [15] maps
indoor scenes at the level of semantically defined objects.
Bowman et al. [16] improved the RGB SLAM performancein terms of camera pose and scale estimation by utilizing not
only low-level geometric features such as points, lines, andplanes but also detected objects as landmarks.
B. 2D semantic segmentation
Several CNN models [19], [20], [11], [21] for semantic segmentation have been proposed, sometimes yielding
impressive results. To achieve a highly precise semantic
segmentation map, such methods aim at exploiting global
information and context to improve the features extracted
by the convolutional layers. In particular, Fully Convolutional Network (FCN) [19] proposed a skip architecturethat combines semantic information from a deep layer withappearance information from a shallow layer to performaccurate and detailed segmentation.

II. RELATED WORK
C. 3D geometric segmentation

A. Semantic mapping

On the other hand, 3D geometric segmentation algorithms
have been developed, to extract geometrically separated
segments from 3D data by unsupervised fashion. Real-time
segmentation for depth map has been investigated by theworks of Uckermann et. al. [22], [23], Pieropan et al. [24]and Abramov et al. [25]. As a consequence, in addition toframe-wise segmentation, [26], [6] has addressed the problem of real-time geometric segmentation for 3D point cloudor 3D mesh reconstructed via dense SLAM by incremental
approach.

Related work aimed at incrementally computing a semantic 3D map of the environment are mostly build on top of thefollowing three main stages: (i) frame-wise segmentation toestimate the per-pixel class probability of the input frame, (ii)
2D-3D label transfer to fuse the 2D semantic segmentation
labels to the 3D map; and, (iii) 3D refinement to denoise theclass probabilities of the 3D map [4], [1], [2], [3], [10], [8].
Notably, [4] employed Random Decision Forests (RDF), a
Bayesian framework and Conditional Random Field (CRF)
respectively to carry out the three above-mentioned stages.
Since the CRF works on each element of the 3D map
reconstructed via SLAM, it is effective in refining the semantic model and obtain high accuracy. Nevertheless, it iscomputationally heavy, as it requires 400 to 1800ms justfor the CRF stage, yielding a frame-rate of 3.9 to 4.6Hz

III. METHOD
Fig. 2 shows the flow diagram of our framework. The input
is represented by RGB and depth frames obtained from amoving RGB-D sensor, which are processed individually.
386

Fig. 2: Flow of the proposed framework. Efficient CNN-based semantic segmentation is exploited to refine the geometricedges on frame-wise segmentation, then it is efficiently fused in the SLAM-based model using the rendered viewpointaccording to the estimated camera pose.

Our method has four components: SLAM framework, 2D
semantic segmentation with a specifically designed CNN,
incrementally building a geometric 3D map, and updatingclass probabilities assigned to each segment of the geometric
3D map. Firstly, SLAM and semantic segmentation with
the CNN are performed simultaneously. In the segmentation
stage, the geometric edge map is generated from the current depth frame and improved with edges extracted fromthe semantic segmentation result toward the semantic-aware
representation. The geometric 3D map is updated through theedge map, and rendered to the current image plane. Finally,class probabilities assigned to each segmented region are
updated with the rendered segmentation map. The following
section describes these components in more detail.

input image. On the other hand, Low-Res Net employs the
ResNet architecture [28], which achieved higher accuracy
than the VGG model [29] in ImageNet [30], and employs
skip connections as done by FCN [19].
Towards the goal of achieving a fast forward pass, we do
not incorporate multi-layered upsampling and design it onlywith two deconvolution layers with two strides. Therefore,
given the input image It (u), u = (x, y) ⊂ Z2 , 0 ≤x < W, 0 ≤ y < H, Low-Res Net outputs a semantic
segmentation map in Fig. 2 (h) as a set of semantic classprobabilities, i.e.
S̃(v) = P(c|It )(1)
where v = (s, t) ⊂ Z2 , 0 ≤ s < W/8, 0 ≤ t < H/8.
Here, P(c) denotes a class probability, where P(c) ⊂ R, 0 ≤P(c) ≤ 1, c ⊂ Z, 0 ≤ c < N with N being the numberof categories. The symbol ˜ denotes instead hereinafter amap of size H/8 × W/8. In our implementation, H = 240,
W = 320, and the number of channels of the input image I
is 3 as in ResNet [28].

A. SLAM
To carry out SLAM in terms of camera pose estimationand fusion we employ the dense approach of InfiniTAM v3[5], relying on the efficient and scalable data representation
proposed by Keller et al. [7], which uses a set surfels sk tobuild the 3D map. As per this method, at the t-th incoming
RGB-D frames, the current camera pose Tt ∈ SE(3) isestimated through Iterative Closest Point [27] and RGBalignment. The new surfels generated from the current depth
map are fused into the 3D map by means of the estimatedcamera pose, and are used to refine the 3D coordinates andnormal associated to the existing surfels.

C. Segmentation
Our geometrical segmentation scheme is based on themethod proposed by Tateno et al. [6]. The method incrementally builds up a geometric 3D map, where a segmentation
label li is associated with each surfel sk , by properly
propagating and merging segments extracted from the current
depth map.
As a result, we obtain a binary geometric edge map B g
in Fig. 2 (c) from the input depth frame by comparing
neighboring normal angles and vertex distances and by
relying on a vertex and normal map as proposed in [6]. Here,B g (u) takes 1 if u is on an edge and 0 for otherwise. It is
important to point out that, while B g is stable since thoseedges are extracted geometrically, edges between objects that
do not present notable geometric characteristics (e.g., suchas with two nearby flat objects) can not be extracted.

B. CNN architecture
The details of the CNN architecture proposed in ourframework, Low-Res Net, are shown in Fig. 2 (g). The
architecture combines concepts from state-of-the-art CNN
models, i.e. Deep Residual Networks (ResNet) [28] andFCN [19]. Specifically, the original FCN architecture [19]utilizes the VGG model [29] to extract features and outputs
a semantic segmentation result at the same resolution of the387

Fig. 3: Example results of our segmentation improvement scheme.

Differently from the geometric segmentation from [6], we
introduce semantic information into the segments. First, we
generate a class map S̃ c , where each component S̃ c (v) has
a class category c, withS̃ c (v) = arg max S̃(v) = P(c|It ).

[8]. Conversely, we propose to assign class probabilities
to each segmentation label li associated to each region
constituting the geometric 3D map. With our approach, eachlabel li is assigned to a discrete probability distribution
P(c|I1...t ) and to a probability confidence Γ. P(c|I1...t )is initialized to 0 over all class probabilities and Γ is also
initialized to 0. Therefore, the space complexity for storingclass probabilities is O(N ·Nl ), where Nl denotes the numberof segmentation labels, in contrast to conventional methods[4], [1] which require O(N ·Ns ), where Ns is the number of
elements of the 3D map (e.g., the number of surfels). This is
an important difference in terms of scalability since typicallyNs  Nl . This also appears as a more natural approach,since it could be argued that humans recognize objects byassigning semantic labels in a region-wise manner rather thanelement-wise.
In order to fuse the output of the Low-Res CNN properlywith the 3D map, we update class probabilities assigned
to each segmentation label li using a confidence-based approach. Firstly, we render the updated geometric 3D map
onto the current image plane using the estimated camera
pose Tt and the 3D position x(k) associated with eachsurfel sk . The rendered segmentation map L(u), where
each component is associated to a segmentation label li ,is generated with L(π(Tt 1 x(k))) = li (k) by denoting thesegmentation label li of a surfel sk with li (k). Here, L(u)takes φ on the pixel u which is not filled with a label li .
Although the CNN-based semantic segmentation used in
our framework is fast, its output S̃ has a low resolution.
Using the rendered segmentation map L whose size is
H × W (i.e. the size of input image), detailed information
is introduced to S̃ to update the class probabilities of eachlabel li with the following update scheme.
First, a set Cv and a set Cv,li are defined as
Cv=(s,t) = u = (x, y) ⊂ Z2 |L(u) 6= φ∧(4)8s ≤ x < 8(s + 1) ∧ 8t ≤ y < 8(t + 1)

(2)

cAfter applying a median filter to S̃ c to remove isolated
points, we resize S̃ c to S c with a nearest neighbor interpolation. We would like to point out that the choice of such an
efficient interpolation approach over a higher quality resizing
such as bilinear interpolation is motivated by the fact that
contours of a CNN-based semantic segmentation map areoften imprecise, hence a better interpolation method would
not yield benefits in terms of accuracy. At the same time,noise in the segment contours is eventually averaged out
by the employed confidence-based label fusion approach.
Then, we generate a binary semantic edge map B s with thefollowing scheme:S c (x, y) 6= S c (x + 1, y) ∨1if
S c (x, y) 6= S c (x, y + 1) ∨B s (u = (x, y)) =S c (x, y) 6= S c (x + 1, y + 1)
0 otherwise(3)
The final binary semantic-aware edge map B, (d) in Fig. 2,
is obtained by applying a binary OR operator between B g
and B s .
In Fig. 3, the geometric edge map in (c) and the semanticaware edge map in (d) show the benefit of our segmentation
improvement scheme. Edges between objects which havepoor geometric characteristics (i.e., wall and picture in theupper row and desk and paper in the bottom row) aresuccessfully merged to the edge map.
Similar to [6], segments of the semantic-aware edge map
B are properly extracted by means of a connected component
algorithm and utilized for incrementally propagating andmerging into the geometric 3D map with the estimatedcamera pose Tt .
D. Probability fusion
Conventional methods assign class probabilities to eachelement that composes the 3D map [4], [1], [2], [3], [10],andCv,li = u ⊂ Cv |L(u) = li .
388

(5)

booksceiling

chair

floor

furnitureobjects

painting

sofa

table

tv

wall

windowclass avg.

pixel avg.

MethodHermans et al. [4]RGBD-SF [1]RGBD-SF-CRF [1]Eigen-SF [1]Eigen-SF-CRF [1]Li et al. [2]Ours-Geometric-Only
Ours

bed

TABLE I: Quantitative results for the NYUv2 dataset [9]. These results were captured immediately after processing theframe. All accuracy evaluations were performed at 320 × 240 resolution. We calculated these accuracies with the same
strategies as [1]. Ours-Geometric-Only denotes the method of building the geometric 3D map without our segmentation
improvement scheme.

68.4
61.7
62.0
47.8
48.3
64.9
83.7
83.7

45.4
58.5
58.4
50.8
51.5
34.6
6.4
15.6

83.4
43.4
43.3
79.0
79.0
72.0
32.0
24.4

41.9
58.4
59.5
73.3
74.7
67.5
52.8
56.7

91.5
92.6
92.7
90.5
90.8
90.5
83.1
83.3

37.1
63.7
64.4
62.8
63.5
65.0
73.5
76.1

8.6
59.1
58.3
46.7
46.9
17.2
40.0
52.5

35.8
66.4
65.8
64.5
63.6
67.3
4.3
40.8

28.5
47.3
48.7
45.8
46.5
59.3
75.3
77.7

27.7
34.0
34.3
46.0
45.9
41.3
56.6
53.0

38.4
33.9
34.3
70.7
71.5
60.0
53.1
57.3

71.8
86.0
86.3
88.5
89.4
85.1
75.0
75.3

46.1
60.5
62.3
55.2
55.6
57.0
50.2
64.4

48.0
58.9
59.2
63.2
63.6
60.3
52.8
58.5

54.3
67.5
67.9
69.3
69.9
70.3
66.9
70.7

updating class probabilities of the 3D map with a frame-wise
recognition.
IV. EXPERIMENTS
A. Dataset and implementation
We evaluate our system on the common NYUv2 dataset[9]. The dataset contains 206 test set video sequences,
however, for a fair comparison, we picked up 140 test
sequences having a frame-rate over 2Hz which is the same as[1]. Since our Low-Res CNN outputs semantic segmentation
with the size of W/8×H/8, we resized the ground truth Sgt
to S̃gt by filling S̃gt (v) with the label which mostly occupies
the area of Sgt (u) corresponding to S̃gt (v). After training
our Low-Res Net with the MS COCO dataset [31] for 10
epochs, we fine-tuned the network with the training dataset of
the NYUv2 dataset [9] for 50 epochs. These evaluations are
conducted on an Intel Core i7-5557U 3.1GHz CPU, GeForceGTX 1080 GPU, and 16GB RAM.

Fig. 4: Example definition of set Cv and Cv,li .

In words: Cv is a set of coordinates to which the labels are
assigned in the region of L(u) corresponding to S̃(v), while
Cv,li is a set of coordinates to which the label li is assigned(See Fig. 4).
When the set Uv of labels li which is included in theregion of L(u) corresponding to S̃(v) is defined as
Uv=(s,t) = li = L(x, y) ⊂ Z|8s ≤ x < 8(s + 1)∧(6)
8t ≤ y < 8(t + 1) ,B. Accuracy

the class probabilities P(c|I1...t ) and the probability confidence Γ of each element l ⊂ Uv are updated throughP(c|I1...t ) ←1 ΓP(c|I1...t 1 ) + γP(c|It )·Z
Γ+γ|Cv,l |Γ ← Γ + γ, γ =|Cv |In this section, we experimentally demonstrate the accuracy of our method by quantitatively comparing the accuracy
with other state-of-the-art methods through Table I. Additionally, Fig. 5 and Fig. 6 show qualitative results of our densesemantic mapping.
As shown in Table I, our method achieves 0.8% higher
average pixel accuracy compared to SemanticFusion [1] and0.4% higher average pixel accuracy compared to Li et al.
[2]. As it can be noted, our method is particularly capable ofoutperforming other semantic mapping methods for object
categories characterized by a big size. For the class bed,
there is a significant accuracy increase of 15.3% over thestate of the art; while, for the class furniture and sofa,
we achieve 11.1% and 18.4% improvement, respectively.
The reason why we achieve high accuracy especially onsuch categories is that our segmentation strongly relies ongeometric information, and geometric boundaries associatedto these categories (e.g., bed and wall and floor and furniture)
are often quite clear.

(7)

which is applied to all class probabilities. Here, the constant
Z is for normalizing class probabilities to a proper distribution. With this scheme, the weight of the probability whichcross over two or more segment regions (e.g., wall and object
in Fig. 4) is reduced. By applying the same strategy to all v
constituting S̃(v), we update class probabilities of all labels
included in the rendered segmentation map L(u).
Therefore, letting the size of S̃(v), H/8 × W/8 be H̃ ×
W̃ , the time complexity for updating class probabilities isO(H̃ W̃ (8 × 8 + |Uv |N )), which means calculating set Cv ,Cv,li , and Uv takes 8 × 8 and updating all class probabilities
N assigned to each label in Uv takes |Uv |N . Note thatconventional methods [4], [1], [3], [10] take O(HW N ) for389

Fig. 5: Qualitative results of our dense 3D semantic mapping on two scenes (left: bedroom 0112, right: dining room 0017).
See Table I for class colors.

TABLE II: Comparison of run-time performance. FQ denotes
the frequency to perform a recognition of the input frame andupdate class probabilities of the 3D map.

Fig. 6 shows the benefit of the segmentation improvement from the viewpoint of accuracy compared with “OursGeometric-Only”, where we build the geometric 3D map
without our segmentation improvement scheme. Particularlyin the upper three rows, the paintings and the window
on the wall, which are difficult to distinguish only withthe geometric-based segmentation, are also segmented andannotated correctly. The geometric 3D map in Fig. 5 also
shows the validity of the segmentation improvement especially on the above-mentioned regions. The example resultsof building a geometric 3D map with/without segmentation
improvement are in Fig. 3 (e) geometric 3D map of Tateno etal. [6] and (f) geometric 3D map of our method. We achieved
semantic-aware representation rather than the geometriconly incremental segmentation method [6]. This improved
segmentation scheme allows achieving higher accuracy interms of pixel average than state-of-the-art methods. As
shown in Table I, the accuracies of the class painting andwindow are significantly improved for 36.5% and 14.2%,
respectively, and 3.8% for overall categories between “Ours”
and “Ours-Geometric-Only”.
The lower two rows of Fig. 6 show failure cases. Since
our method mainly extracts edges from the vertex and normal
map obtained from the incoming depth image, it is difficult
to successfully segment distant objects where depth values
tend to be unstable (i.e., the third row of Fig. 6) and manage
scenes where many small objects are lined up where vertices
and normals are cluttered (i.e., the fourth row of Fig. 6). In
Table I, this is the same reason why the categories of small
objects such as book and objects score low accuracies. We
leave the exploration of improving these limitation to future

MethodHermans et al. [4]SemanticFusion [1]Yang et al. [3]Li et al. [2]Ours

3D map
Dense
Dense
Dense
Semi-Dense
Dense

FQ
every 6 frames
every 10 frames
every frame
every key-frame
every frameFPS
3.9 - 4.6 Hz
25.3 Hz
2 Hz
10 Hz
30.9 HzTABLE III: Average time spent on each processing stage.
processing for segmentation are in line 2-4 and processing
for recognition are in 5-7. Note that the processing with *and the processing with ** can be processed simultaneously.
Component
SLAM *
Generate a binary geometric edge map Bg *Segmentation improvement
Update the geometric 3D map
Low-Res CNN **Generate a rendered segmentation map L
Probability fusion
Total

Consumed time
8.13 ms
1.04 ms
0.39 ms
8.74 ms
19.32 ms
2.52 ms
1.37 ms
32.34 mswork.
C. Computational cost
In this section, we demonstrate the advantage of reducing
the computational complexity, i.e. one of the main contributions of this method. We quantitatively compare the run-time390

Fig. 6: Qualitative results for the NYUv2 dataset [9]. As with Table I, Ours-Geometric-Only denotes the method of building
the geometric 3D map without our segmentation improvement scheme. See Table I for class colors.

performance with state-of-the-art approaches through TableII.
As shown in Table II, we achieved real-time performance(i.e., over 30Hz) while performing all processing components
on every input frame. As analyzed in the last paragraph ofSec. III-D, the time complexity for updating class probabilities of the 3D map (i.e., Probability fusion) is O(H̃ W̃ (8 ×
8 + |Uv |N )). Considering the average number of |Uv | was
1.28 through the experiments, the average time complexity
O(H̃ W̃ (8 × 8 + |Uv |N )) turns into O(H̃ W̃ (8 × 8 + N )) =O(HW + H̃ W̃ N ) in contrast to the one of conventional
methods O(HW N ) [4], [1], [3], [10]. Therefore, as shownin Table III, updating class probabilities of the 3D map
only took 1.37ms on average, whereas SemanticFusion [1]spent 41.1ms for the processing. Furthermore, the processing
for 2D recognition (i.e., Low-Res CNN) only took 19.32ms
while maintaining high accuracy in the end, as mentioned in
Section. IV-B.

Lastly, we discuss about the results of reducing the spacecomplexity through Fig. 7. As shown there, the memory
usage of our method is significantly reduced compared tothe one of SemanticFusion [1] over all frames. The averagememory usage of our method is 0.08% of the one of SemanticFusion [1]. The reason for this significant improvement
is that, as mentioned in Sec. III-D, the space complexityof our method is O(N · Nl ) whereas SemanticFusion takesO(N · Ns ), where Nl and Ns were 1032 and 844260 in theend of the scene respectively.
V. CONCLUSION
In this paper, we proposed an efficient semantic mapping
approach by assigning class probabilities to each regionof the geometric 3D map which is incrementally built
up through a robust SLAM framework and a geometricbased incremental segmentation. Through our experiments,
we demonstrated that our approach notably compressed the391

Memory usage (Byte)[11] H. Noh, S. Hong, and B. Han, “Learning deconvolution network forsemantic segmentation,” in IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 1520–1528, 2015.
[12] S. Sengupta, E. Greveson, A. Shahrokni, and P. H. Torr, “Urban
3d semantic modelling using stereo vision,” in IEEE International
Conference on Robotics and Automation (ICRA), pp. 580–585, IEEE,
2013.
[13] H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, “Semanticlabeling of 3d point clouds for indoor scenes,” in Advances in neural
information processing systems, pp. 244–252, 2011.
[14] Z. Zhao and X. Chen, “Building 3d semantic maps for mobile robots
using rgb-d camera,” Intelligent Service Robotics, vol. 9, no. 4,
pp. 297–309, 2016.
[15] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, andA. J. Davison, “Slam++: Simultaneous localisation and mapping at thelevel of objects,” in IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 1352–1359, IEEE, 2013.
[16] S. L. Bowman, N. Atanasov, K. Daniilidis, and G. J. Pappas, “Probabilistic data association for semantic slam,” in IEEE International
Conference on Robotics and Automation (ICRA), pp. 1722–1729,
IEEE, 2017.
[17] D. Gálvez-López, M. Salas, J. D. Tardós, and J. Montiel, “Real-time
monocular object slam,” Robotics and Autonomous Systems, vol. 75,
pp. 435–449, 2016.
[18] N. Fioraio and L. Di Stefano, “Joint detection, tracking and mappingby semantic bundle adjustment,” in IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 1538–1545, IEEE, 2013.
[19] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), pp. 3431–3440, 2015.
[20] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” IEEE
transactions on pattern analysis and machine intelligence (TPAMI),
vol. 39, no. 12, pp. 2481–2495, 2017.
[21] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking
atrous convolution for semantic image segmentation,” arXiv preprint
arXiv:1706.05587, 2017.
[22] A. Uckermann, R. Haschke, and H. Ritter, “Realtime 3D segmentation
for human-robot interaction,” in 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013.
[23] A. Uckermann, C. Elbrechter, R. Haschke, and H. Ritter, “3D scene
segmentation for autonomous robot grasping,” in 2012 IEEE/RSJ
International Conference on Intelligent Robots and Systems (IROS),
oct 2012.
[24] A. Pieropan and H. Kjellstrom, “Unsupervised object exploration using
context,,” in The 23rd IEEE International Symposium on Robot andHuman Interactive Communication (RO-MAN), 2014.
[25] A. Abramov, K. Pauwels, J. Papon, F. Worgotter, and B. Dellen,“Depth-supported real-time video segmentation with the kinect,” inIEEE Workshop on Applications of Computer Vision (WACV), 2012.
[26] R. Finman, T. Whelan, M. Kaess, and J. J. Leonard, “Toward lifelong
object segmentation from change detection in dense RGB-D maps,”in 2013 European Conference on Mobile Robots, ECMR 2013 Conference Proceedings, pp. 178–185, 2013.
[27] K.-L. Low, “Linear least-squares optimization for point-to-plane icp
surface registration,” Chapel Hill, University of North Carolina, vol. 4,
2004.
[28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer
vision and pattern recognition (CVPR), pp. 770–778, 2016.
[29] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” arXiv preprint arXiv:1409.1556,
2014.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification
with deep convolutional neural networks,” in Advances in neural
information processing systems (NIPS), pp. 1097–1105, 2012.
[31] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays,
P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollar, “Microsoft coco:
Common objects in context,” arXiv preprint arXiv:1405.0312, 2014.

10

10

SemanticFusion [1]

9

10

Ours

8

10

7

10

6

10

5

10

4

10

3

10

0

50

100

150

200

250

300

350

400

Frame

Fig. 7: Comparison of memory usage for storing class probabilities with SemanticFusion [1] on the scene bedroom 0112
of the NYUv2 dataset [9].

computational complexity in terms of both of time and space
while achieving comparable accuracy against state-of-the-art
approaches without any post-processing to the semantic 3D
map. Furthermore, we confirmed that our strategy improved
the incremental segmentation framework beyond the geometric only to the semantic-aware representation.
ACKNOWLEDGMENT
This work was partially supported by JST CREST“Intelligent Information Processing Systems Creating CoExperience Knowledge and Wisdom with Human-Machine
Harmonious Collaboration.”
R EFERENCES[1] J. McCormac, A. Handa, A. Davison, and S. Leutenegger, “Semanticfusion: Dense 3d semantic mapping with convolutional neural networks,” in IEEE International Conference on Robotics and Automation(ICRA), pp. 4628–4635, IEEE, 2017.
[2] X. Li and R. Belaroussi, “Semi-dense 3d semantic mapping frommonocular slam,” arXiv preprint arXiv:1611.04144, 2016.
[3] S. Yang, Y. Huang, and S. Scherer, “Semantic 3d occupancy mapping
through efficient high order crfs,” IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS), 2017.
[4] A. Hermans, G. Floros, and B. Leibe, “Dense 3d semantic mapping ofindoor scenes from rgb-d images,” in IEEE International Conferenceon Robotics and Automation (ICRA), pp. 2631–2638, IEEE, 2014.
[5] V. A. Prisacariu, O. Kähler, S. Golodetz, M. Sapienza, T. Cavallari,P. H. Torr, and D. W. Murray, “InfiniTAM v3: A Framework for LargeScale 3D Reconstruction with Loop Closure,” ArXiv e-prints, 2017.
[6] K. Tateno, F. Tombari, and N. Navab, “Real-time and scalable incremental segmentation on dense slam,” in IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), pp. 4465–4472,
IEEE, 2015.
[7] M. Keller, D. Lefloch, M. Lambers, S. Izadi, T. Weyrich, and A. Kolb,“Real-time 3d reconstruction in dynamic scenes using point-based
fusion,” in International Conference on 3DTV-Conference, pp. 1–8,
IEEE, 2013.
[8] A. Kundu, Y. Li, F. Dellaert, F. Li, and J. M. Rehg, “Joint semantic
segmentation and 3d reconstruction from monocular video,” in European Conference on Computer Vision (ECCV), pp. 703–718, Springer,
2014.
[9] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in European Conference
on Computer Vision (ECCV), pp. 746–760, Springer, 2012.
[10] V. Vineet, O. Miksik, M. Lidegaard, M. Nießner, S. Golodetz, V. A.
Prisacariu, O. Kähler, D. W. Murray, S. Izadi, P. Pérez, et al.,“Incremental dense semantic stereo fusion for large-scale semantic
scene reconstruction,” in IEEE International Conference on Robotics
and Automation (ICRA), pp. 75–82, IEEE, 2015.

392