Eurographics Workshop on 3D Object Retrieval (2017)
I. Pratikakis, F. Dupont, and M. Ovsjanikov (Editors)

SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset
Binh-Son Hua 1
Quang-Trung Truong 1
Minh-Khoi Tran 1
Quang-Hieu Pham 1
4
5
5
Asako Kanezaki
Tang Lee
HungYueh Chiang
Winston Hsu 5
6
7
8
9
Bo Li
Yijuan Lu
Henry Johan
Shoki Tashiro
Masaki Aono 9
Minh-Triet Tran 10
Viet-Khoi Pham 10
Hai-Dang Nguyen 10,12
Vinh-Tiep Nguyen 10,11
Quang-Thang Tran 10
Thuyen V. Phan 10
Bao Truong 10
Minh N. Do 13
Anh-Duc Duong 11
2
3
1
Lap-Fai Yu
Duc Thanh Nguyen
Sai-Kit Yeung
1 Singapore University of Technology and Design

2 University of Massachusetts Boston

4 National Institute of Advanced Industrial Science and Technology (AIST), Japan
6 University of Southern Mississippi

7 Texas State University

10 University of Science, VNU-HCM, Vietnam

8 Fraunhofer IDM@NTU, Singapore

3 Deakin University

5 National Taiwan University
9 Toyohashi University of Technology, Japan

11 University of Information Technology, VNU-HCM, Vietnam

12 John von Neumann Institute, VNU-HCM, Vietnam

13 University of Illinois at Urbana-Champaign

Abstract
The goal of this track is to study and evaluate the performance of 3D object retrieval algorithms using RGB-D data. This is
inspired from the practical need to pair an object acquired from a consumer-grade depth camera to CAD models available inpublic datasets on the Internet. To support the study, we propose ObjectNN, a new dataset with well segmented and annotated
RGB-D objects from SceneNN [HPN∗ 16] and CAD models from ShapeNet [CFG∗ 15]. The evaluation results show that the
RGB-D to CAD retrieval problem, while being challenging to solve due to partial and noisy 3D reconstruction, can be addressed
to a good extent using deep learning techniques, particularly, convolutional neural networks trained by multi-view and 3D
geometry. The best method in this track scores 82% in accuracy.
Categories and Subject Descriptors (according to ACM CCS): I.4.8 [Computer Vision]: Scene Analysis—Object Recognition

1. Introduction
Off-the-shelf consumer depth cameras have become widely adopted
in computer graphics, computer vision, gaming and medical imaging applications. In the past few years, several methods and publicly
available scene and object datasets have been proposed to significantly improve the performance of scene understanding algorithms,
particularly, object retrieval in the 3D domain. SHREC is an annual
challenge held at the 3D Object Retrieval (3DOR) workshop that focuses on studying and benchmarking the state-of-the-art algorithms
in this area.
In this SHREC track paper, we aim to study and evaluate theperformance of 3D object retrieval algorithms using RGB-D datafrom consumer depth cameras. We focus on the problem of pairing
an RGB-D object captured in a real world environment to a virtual
CAD model created by 3D artists. Such pairing, when available,
could be used for applications such as semantic annotation, shapecompletion, and scene synthesis.
To support the study in this track, we propose ObjectNN, a newdataset for benchmarking RGB-D Object-to-CAD Model retrieval.
It contains well segmented and annotated RGB-D objects acquiredin real-world setting by SceneNN [HPN∗ 16] and richly annotatedc 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Figure 1: Examples of RGB-D objects (first row) and CAD models(second row) in ObjectNN. The task of this track is to retrieve aplausible CAD model given an RGB-D object. Most RGB-D objects
in 3D representation are partial and might contain some noise in thegeometry due to imperfect reconstruction, making the recognition
and retrieval task more challenging.
CAD models collected from the internet by ShapeNet [CFG∗ 15].
In total, ObjectNN contains 1667 RGB-D objects and 3308 CAD
models in 20 categories. Some example RGB-D objects from ourdataset are shown in Figure 1.
Participants are asked to run their retrieval algorithms on the proposed dataset and submit the retrieved CAD models for evaluation.
Based on the evaluation results, we provided an analysis on theperformance of several object retrieval algorithms.

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset

SHREC track∗

NIST [GDB 15]
DUTH [PSA∗ 16]
IUL [PPG∗ 16]
Ours

Query dataset

Target dataset

Categories

Attributes

60 RGB-D objects
383 range-scan models
200 RGB-D objects
1667 RGB-D objects

1200 CAD models
Similar to query dataset
Similar to query dataset
3308 CAD models

60
6
N.A.
20

Geometry only
Object from cultural heritage domain
Objects scanned in lab setting
Objects from real-world environments

Table 1: Relevant datasets in previous SHREC tracks. To serve the purpose of the contest in this track, we built a new object dataset called
ObjectNN upon SceneNN [HPN∗ 16], a public dataset with more than 100 densely annotated scene meshes acquired from the real world, andShapeNet [CFG∗ 15], a large-scale dataset of CAD models.
2. Dataset
The query dataset consists of 1667 objects extracted from morethan 100 3D reconstructed real-world indoor scenes in the SceneNN
dataset [HPN∗ 16]. Each object has four data entries: (1) color triangle mesh; (2) a set of RGB-D frames extracted from the scene that
contain the object; (3) a mask of the object in each frame; (4) andthe camera pose for each frame. All objects are carefully segmented
and annotated in both 3D and 2D by utilizing the user interactive
tool by Nguyen et al. [TNHYY16], which can annotate RGB-D
scenes densely at per-vertex and per-pixel level.
The target dataset has 3308 CAD models of indoor objects,
which are extracted from ShapeNetSem, a richly annotated subset of
ShapeNet [CFG∗ 15]. We adopt and modify the category definitionsin NYU Depth v2 dataset [SHKF12] to create 20 object categories
for common indoor objects such as table, chair, bookshelf, monitor,
lamp, pillow, etc.
Compared to the datasets of the previous SHREC tracks, ObjectNN is designed to follow the acquisition procedure in real world
settings more closely. In a general indoor scene, objects are often
organized or scattered in a scene depending on their usage andfunctionality. The acquisition of such objects is often performed byscanning the entire scene, which often results in cluttered and partial
object reconstructions. This is fundamentally different from objects
acquired with lab settings whose reconstructions are much more
complete. Table 1 shows a comparison of our dataset to those ofsome previous works. As can be seen, our dataset has a larger scalecompared to the existing datasets, and thus offers a good opportunity
to benchmark supervised learning techniques for object retrieval.
Ground Truth. As there is not a standard metric to measure thesimilarity of two shapes, particularly between an RGB-D object
and a CAD model, to establish the ground truth for evaluation,
we classify all objects manually using an interactive tool that cansimultaneously display an RGB-D object and a CAD model. Twohuman subjects are asked to pair an RGB-D object in the query
set to some CAD models in the target set by assigning them to apredefined category. The classification is based on shape and color
similarity of the objects in each pair. After classification, for each
RGB-D object, all CAD models belonging to the same category are
assumed to be its ground truth targets.
Beyond categories, we also classify objects more strictly into
subcategories by considering more object attributes. In total there
are 20 categories and 40 subcategories. In this very first contest, weonly consider 20 categories in our evaluation.

Availability. To support the training process of methods based onsupervised learning, we split ObjectNN into three subsets: training, validation and test, with ratio 50%, 25%, and 25% respectively. We then release the training and validation set to participants,
with ground truth categories available for both RGB-D objects and
CAD models. In this contest, all CAD models are placed in the
training set. The test set only contains RGB-D models and theircategories are not released to ensure a fair evaluation. Note that
unsupervised learning is still applicable to solve the retrieval problem, and it would become much more challenging if the groundtruth categorization of CAD models is not utilized. The dataset
and the ground truth annotation tool are available for download at
http://www.scenenn.net.

3. Overview
All methods used by the participants are summarized in Table 2.
For brevity, we refer to each method by a unique name of an author
from each group followed by the technique they used. In total we received nine registrations for participation and five result submissions
excluding a baseline submission from the organizers themselves, corresponding to a submission rate of 55.56%. Each group can propose
more than one approach to solve the problem, which corresponds toone or more runs to be evaluated.
In general, the proposed approaches belong to two main classesof techniques: deep learning using convolutional neural networks(Section 4–6) and feature matching using bag-of-words or shapedescriptors (Section 7–9). Interested readers could proceed to SecTeam

MethodTraining

DomainColor

Kanezaki
Tang
Tang
Tang
TruongTruongTran
Li
TashiroRotationNet (Sec. 4)
3DCNN (Sec. 5.1)
MVCNN (Sec. 5.1)
CDTNN (Sec. 5.2)
2D (Sec. 6.1)
3D (Sec. 6.2)
BoW-RoPS (Sec. 7)
SBR-VC (Sec. 8)
GOLD (Sec. 9)

Y
Y
Y
Y
Y
Y
N
N
N

View-based
Full 3D
View-based
View-based
View-based
Full 3D
Full 3D
View-based
View-based

N
N
Y
Y
Y
N
N
N
Y

Table 2: An overview of the techniques used by the participants,
most of which are based on supervised learning, especially thosebuilt atop view-based and 3D volume convolutional neural networks.
Unsupervised learning that performs feature matching using handcrafted descriptors can also be used to solve the problem.

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Datasettion 4–9 for more technical description of each method. We discuss
evaluation results in Section 10.

7UDLQLQJ
8QDOLJQHG
GDWDVHW$OLJQHGGDWDVHW
5RWDWLRQ1HW

4. RotationNet
Our method is based on a convolutional neural network (CNN)
named RotationNet [KMN16]. An overview of the training andinference process using RotationNet is illustrated in Fig. 2. Similarly
to multi-view CNN [SMKLM15], it takes multi-view images ofan object as input and estimates its object category. Our method
treats the pose labels as latent variables, which are optimized toself align in an unsupervised manner during the training using an
unaligned dataset. The code is available at https://github.
com/kanezaki/rotationnet.

4.1. Training
We assume that multi-view images of each training object instanceare observed from all the pre-defined viewpoints. Let M be thenumber of the pre-defined viewpoints and N denote the number oftarget object categories. A training sample consists of M images ofan object {xi }M
i=1 and its category label y ∈ {1, . . . , N}. We attacha view label variable vi ∈ {1, . . . , M} to each image xi and set it toj when the image is observed from the j-th viewpoint, i.e., vi ←j. In our method, only the category label y is given during thetraining whereas the view labels {vi } are unknown, namely, {vi } are
treated as latent variables that are optimized in the training process.
We introduce an “incorrect view” label and append it to the target(i)
M×(N+1)category labels. Letting Pi = {p j,k } ∈ R+
denote a matrix
composed of P(ŷi | xi , vi ) for all the M viewpoints and N + 1 classes,
the target value of Pi in the case that vi is correctly estimated isdefined as follows:(
1 ( j = vi and k = y), or ( j 6= vi and k = N + 1)(i)
p j,k =(1)0 (otherwise).
The parameters of RotationNet are iteratively updated via standard
back-propagation of M softmax losses, whereas {vi }M
i=1 are determined in a manner to maximize the probability that a training sample
belongs to its correct category y in each iteration.

,QIHUHQFHᶅ

5RWDWLRQ1HWᶆᶇ$SDUWLDOVHWRIPXOWLYLHZVYLHZSRLQW
HVWLPDWLRQ

ᶅ

ᶆ

ᶇ

 nWRLOHW|

Figure 2: Overview of training and inference using RotationNet. We
jointly learn the parameters of RotationNet and estimate the pose ofobject instances in the unaligned 3D object dataset. In the inference
phase, RotationNet takes a partial set of all the multi-view images ofan object as input and outputs its object category by rotation, where
the best pose is selected to maximize the object category likelihood.
Thresh. We rank categories by the classification scores for eachquery. If the score of a category is higher than a certain ratio of the
maximum score, we add the target models in the category to theretrieval set. We set the threshold to 0.1. In the case that the scoresof all the categories (except for the predicted category) are lower
than 10% of the score of the predicted category, we obtain exactly
the same retrieval set as “single” case.
1000. In order of decreasing classification scores, we add the target
models in the same category to the retrieval set for each query. We
stop adding a target model when the number of the models in the
retrieval set reaches to 1000.
In addition, we also perform the “Single” and “Thresh” retrieval
but classifying the objects to subcategories for a more rigorous
evaluation.
5. Multi-view and 3D Convolutional Neural Network

We place virtual cameras on the M = 20 vertices of a dodecahedron encompassing the object. There are three different patterns ofrotation from a certain view, because three edges are connected toeach vertex of a dodecahedron. Therefore, the number of candidates
for all the view labels {vi }M
i=1 is 60 (= 3M).

We propose two classes of methods to solve the problem: (1) Convolutional neural network (CNN) based techniques that include 3D
CNN, multi-view 2D CNN (MVCNN), and 2D-3D fused classification, and (2) Cross-domain triplet neural network. Before applying such techniques, we preprocess SceneNN (SNN) objects andShapeNetSem (SNS) objects as follows.

4.2. Retrieval
First, we train our RotationNet model using 3308 CAD models in
ShapeNetSem. Then we fine-tune the model using 830 objects in
SceneNN. We did not use colors nor textures. We train two different
RotationNet models, one to predict category labels and the other topredict subcategory labels. For each query, we construct a retrieval
set using five different approaches described below.

Voxelization. Voxelization is a prevailing 3D data representation
method. It normalizes object size to 1 × 1 × 1 space with a scaling
factor and captures occupancy of the scaled object in space with
64 × 64 × 64 resolution binary voxel grid, i.e., structural information
and real size measurement of an object is kept. We multiply the value
stored in every binary voxel with the scaling factor and augment
every object by rotating it 0, 90, 180, 270 degrees horizontally, as
the object may not be well aligned.

Single. We predict a single category label for each query and construct a retrieval set with all the target models in the same category.

View Rendering. We render each object in both SNN and SNSfrom 12 views that are distributed around the object every 30 de-

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset
64 × 64 × 64

5.2. Cross-Domain Triplet Neural Network
Category

3D CNN
3D Scaled
Voxel Gridfc6 fc7

5 3D conv layers

fc8 softmax

224 × 224
view 1view 2...

MVCNN

Category(12 CNNs)...

View-Poolingfc6 fc7

fc8 softmax

view 122D Views

We build a cross-domain triplet neural network (CDTNN) to learn
the similarity between SNN objects and SNS objects. A triplet network [SKP15] has 3 streams of CNNs: anchor, positive, and negative.
The anchor is for SNN views, and the positive and negative are for
SNS views. Each stream is an MVCNN without fully connected
layers, i.e. it has 12 streams of conv1 to pool5 layers of AlexNet
with one view-pooling layer, generating a feature map with size
9216. The 3 streams are sharing parameters.

5 2D conv layers

Figure 3: The network structures of the 3DCNN and MVCNN
for 3D object classification. The 3DCNN is modified based on
VoxNet [MS15]. The MVCNN [SMKLM15] uses AlexNet as thebase model.
grees and elevated 30 degrees from the ground. SNN objects are
rendered with color textures while SNS objects are not. All objects
are rendered with Phong shading on a black background.
Since some scenes and objects in SNN are upside down, for data
augmentation purpose, we additionally flip the SNN training objects
vertically to render views.
5.1. CNN-based Classification
Since a retrieved object in SNS is considered correct if it is fromthe same category as the query SNN object, and since the categories
of SNS objects are known, we can reduce the retrieval problem
into a classification problem of SNN objects. We sort the predictedprobabilities of the 20 categories in descending order. According to
the sorted list, we first push all SNN objects from the first category
into the results, then the second, and so on.
3D CNN. Since the problem is reduced to classification, we build a
3D CNN and train our network on scaled SNN voxel representations
to learn the structural similarity and predict the class of an SNN
object. Mirroring the success of AlexNet, we borrow the idea tomodel our 3D network with 8 layers including 5 convolutional
layers and 3 fully-connected layers. Each convolutional layer has
3 × 3 × 3 kernel and 1 × 1 × 1 stride. The first, the second and thefifth convolutional layers are followed with a max pooling layer.
Three fully-connected layers with output size 4096, 4096 and 20
respectively are attached after 5 convolutional layers. An illustrationof our network architecture is shown in Figure 3.
MVCNN. We utilize MVCNN [SMKLM15] to learn from theviews and predict the SNN objects categories. We adopt AlexNet as
the base model of MVCNN, and put the View-Pooling layer (maxpooling) after pool5 layer. Each of the 12 CNNs share parameters.
The parameters are pretrained with ImageNet.
2D-3D Fused Classification. 3DCNN learns objects structural andscaling information in real world, while MVCNN learns surface
texture and contour of objects. We fuse the 2D and 3D methods by
averaging the predicted category probabilities.

Since the views from SNN and SNS have different look, i.e. SNN
are real object while SNS are human made and with no texture in our
settings, we propose CDTNN by adding an “adaptation layer” after
the anchor stream. The adaptation layer is simply a fully-connected
layer with both input and output size 9216. The features generated
by each stream are used as the representations, and their distances
are squared Euclidean distances.
Our methods are implemented using TensorFlow. Our experiments are performed on a workstation with two NVIDIA K80 GPUs.
6. 2D-3D Retrieval using VGG and VoxNet
In this approach, the retrieval problem is solved in two steps: (1)determine the object category and (2) compare the features of theobject with those of the CAD models in the same category, anddetermine the final similarity ranking. In Step 1, we leverage stateof-the-art object recognition techniques in both the 2D and 3D
domain, i.e., convolutional neural networks. In Step 2, the features
predicted by the neural networks are extracted for comparison. An
overview is shown in Figure 4.
6.1. 2D Retrieval using VGG
In image classification, traditional techniques are based on handcrafted features which are recently shown to be suboptimal compared to features learned by deep neural networks. Inspired by theimpressive performance of recent pretrained networks, we employ
the popular VGG model [SZ14] to solve this problem.
We first fine tune this model by using the 2D images of the RGBD objects. As an object could appear in multiple RGB-D frames,
for each object we pick and assign to it a representative image. In
particular, we set a frame as representative when its object mask has
VGG-16 model
CAD modelfeature maps

RGB-D object
feature map

Similarity metric

ShapeNet

Desk

Color and mask

SceneNN

Figure 4: An overview of object retrieval using image data with
VGG. The pipeline for retrieval using 3D data is almost the same
but with VGG replaced by VoxNet.

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset

the most number of valid pixels. We also use this mask to filter theforeground pixels in the frame, which can then be used as input totrain the network. After fine tuning, we obtain an accuracy of 77.56%
on the validation set, which also agrees with the 74% accuracy on
the test set reported in Section 10.
Generally speaking, as we cast the current retrieval problem into
object classification, obtaining the category is enough for evaluation.
For each query object, we simply return all CAD models in thepredicted category.
For completeness, we further compute a distance score to rank
the similarity of a retrieved object to the query as follows. First,
we apply a forward pass to the rendered image of all CAD models,and extract the output of the third last layer of the VGG network
to obtain a feature vector with 4096 elements. During retrieval, thefeature vector of the query object is extracted and compared to thefeature vectors of all CAD models. We use L2-norm to compare thevectors, and then sort the distance in increasing order to obtain the
ranking for the retrieved objects.
6.2. 3D Retrieval using VoxNet
For 3D, we use VoxNet [MS15], a compact convolutional neuralnetwork for object recognition that has about 1.2 million parameters.
This network only contains two convolutional layers followed by
two fully connected layers, and does not use dropout and pooling.
We train this network from scratch using the provided object dataset.
We use a binary volume representation as the 3D CNN describedin Section 5. All objects are scaled and centered at the unit cube. We
achieved 50% validation accuracy by training with RGB-D objects
in the training set alone. We also achieved approximately the samelevel of accuracy when the network is furthered trained with CAD
models. To support rotation invariance, we augment the data with128 rotation directions and retrain the work with augmented data.
This results in accuracy improvement to 52.93%.
It is interesting to see that the results by 2D recognition outperforms that by 3D recognition. This could be explained by the factthat our 3D volumes do not consider object scale and color information. Also, the 2D convolutional network might be more well trained
thanks to more data availability.
7. Bag-of-Words Framework with RoPS
The requirement of this challenge is to product a ranklist of target
objects in ShapeNet corresponding to a test object in SceneNN.
However, because query objects can be partially constructed withnoise and holes, while target objects are ideally modelled, we decide
not to directly evaluate the similarity between a test object with atarget object. Our approach is to determine the possible categories
of a query object, then retrieve all target objects (in ShapeNet) in
those categories to produce the result rank list.
Our objective is to verify whether our method purely based on
3D object retrieval can be used to classify a query object with highaccuracy. Thus, we do not train machine-learning-based classifiers
to recognize the category of a query object. Instead, we determine
the category of an object based on the the corresponding retrieved
ranklist in SceneNN (c.f. Fig 5).
c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Query
model

FeatureExtraction
RandomSampling,
RoPS

Top-K Voting

20-dimensionalscore vector………
Top-K retrieved models
from SceneNN

Soft Assignment
tf,idf weighting

1- or 2-best
categories

Retrieval in SceneNN
Inverted Index Tree
L1 measure

Quantization,
Feature Encoding
FeatureDescriptors

Retrieval in ShapeNet

Retrieved ranklist from ShapeNet
Histogram of
3D visual words

Figure 5: Bag-of-Words framework for 3D object retrieval.
Our method for 3D object retrieval is based on Bag-of-Words(BoW) scheme for visual object retrieval [SZ03, NNT∗ 15]. We use
our BoW framework for 3D object retrieval [PSA∗ 16] with 3D
feature RoPS [GSB∗ 13] to retrieve RSceneNN (q), a rank list of top K
similar labeled objects in SceneNN corresponding to a query object
q. Then we use simple voting scheme to determine the top 1 and
top 2 categories of q from RSceneNN (q). Finally, we create the result
ranklist RShapeNet (q) by adding target objects in ShapeNet in thosecategories. All selected target objects in the same category have thesame similarity with a query object q.
First, all labeled and unlabeled 3D objects in SceneNN are normalized to fit in a unit cube. As shapes have irregular mesh, we
subdivide the mesh to reduce significant difference in the densityof vertices between different parts in a shape. We uniformly takerandom samples of 5% ≤ pSampling ≤ 20% in each object and use
RoPS [GSB∗ 13] as the feature to describe the characteristics of themesh clipped in a sphere with supporting radius r = 0.1.
We train a codebook with the size relatively equal to 10% of
the total number of features in the corpus, using Approximate KMeans. In our experiments, our codebook has the size of 100,000 and
150,000. We use soft-assignment [PCI∗ 08] with 3 nearest neighbors
to reduce quantization error and L1, asymmetric distance measurement [ZJS13], to evaluate the dissimilarity of each pair of objects.
We experiment with four runs with the following configurations:
Run 1 The category of an object is based on Top K = 1 in the rank
list with pSampling = 10%, codebook size = 150,000.
Run 2 The two best categories of an object is based on Top K = 4
in the rank list with pSampling = 20%, codebook size = 100,000.
Run 3 The category of an object is based on Top K = 1 in the rank
list with pSampling = 20%, codebook size = 150,000.
Run 4 The two best categories of an object is based on Top K = 7
in the rank list with pSampling = 20%, codebook size = 100,000.
In our implementation, codebook training is performed on a 2.4
GHz Intel Xeon CPU E5-2620 v3 with 64 GB RAM. The 3D feature
extraction and description are executed on a 2 GHz Intel Xeon
CPU E5-2620 with 1 GB RAM. The retrieval process with feature
quantization and similarity matrix output is performed on another
2.2 GHz Intel Xeon CPU E5-2660 with 12 GB RAM. The averagetime to calculate features of a model is 1–2 seconds and it takes on
average 0.1 second to compare a test object against the training set.
Experimental results show that our method, without using 2D/3D
deep learning features and color information, can retrieve with highaccuracy 3D objects based on their partial 3D appearance. This is a

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset

ShapeNet

Figure 7: Overview of the view-based shape feature method.
images [PPTP10]. Then we extract SURF [BETVG08] local features from each image and encode them into a feature vector with
Gaussian of Local Descriptors (GOLD) [SGMC15], which consistsof a mean and a covariance component.
Figure 6: Overview of the SBR-VC algorithm: the first row is for theprecomputation whereas the second row is for the retrieval stage.
promising step for our future enhanced system in which we integrate
2D views and deep learning features to boost the performance of
our system.

8. View Clustering and Parallel Shape Context Matching
We adapt the sketch-based 3D model retrieval algorithm based onview clustering (SBR-VC) [LLJF16, LLJ13] to solve the problem
in this track. Figure 6 illustrates an overview of the algorithm. In
this approach, an RGB-D object is first aligned by Continuous
Principle Component Analysis (CPCA) [Vra04]. Then, the front
view is regarded as a representative view of the RGB-D object andalso used to extract an outline image, similar as a sketch in theoriginal algorithm SBR-VC, for RGB-D object and CAD model
pair comparison. The remaining steps are similar to the original
SBR-VC steps in [LLJF16, LLJ13].
Particularly, the SBR-VC algorithm retrieves CAD models given
RGB-D queries in four steps, as below.
Step 1 Viewpoint entropy-based adaptive view clustering. It is tocluster a set of sample views of each CPCA-aligned target CAD
model into an appropriate number of representative views according
to its visual complexity, which is defined as the viewpoint entropy
distribution of its sample views.
Step 2 Feature view generation. Outline feature views for the RGBD queries and the 3D CAD models are generated. The RGB-D casehas been explained above. In the 3D CAD model case, silhouette
views are first rendered followed by outline feature extraction.
Step 3 Relative shape context computation. Rotation-invariant relative shape context features [BMP02] are extracted to represent
outline feature views for both the query and target 3D objects.
Step 4 Online retrieval. The relative shape context matching is performed in parallel between the RGB-D outline feature view andseveral representative views of a target CAD model and the minimum matching cost is chosen as the RGB-D-to-CAD distance. The
models are ranked accordingly based on the distances.

9. View-based Shape Feature
We tackle this problem by using view-based shape features. The
overall steps are shown in Figure 7. First, we render each target
object (CAD models) from multiple views to obtain face orientation

On the other hand, we render each query object (RGB-D object)from a single view under the assumption that each query object
can be captured in a representative
view. We calculate this view v
with v = ∑ p∈Q a p n p /k ∑ p∈Q a p n p k where p is a triangle in thequery object Q, a p is area of the triangle, and n p is the triangle
surface normal. Similar to the process for CAD models, we generatea face orientation image from each query. After obtaining the faceorientation image, we extract SURF local features and encode themwith GOLD. For retrieval, GOLD features extracted from target
objects and query objects are then power and L2 normalized. We
compute dissimilarity d between query object Q and target object
M as d = min kfQ − fi k where fQ is GOLD feature vector of Q andifi is the i-th feature vector of M.
In our experiment, we use image resolution 300 × 300, 66 view
points based on the sub-divided octahedron, and 128 dimension
SURF feature. This results in GOLD vector of 8384 dimension.
10. Evaluation
For each query model, each participant submits a ranked list where
retrieved models are sorted by similarity with the query model. Eachranked list is evaluated based on the ground truth category and subcategories. We use the following measures: Precision-Recall curve,
Mean Average Precision (mAP), and Normalized Discounted Cumulative Gain (NDCG). Please refer to our homepage for additional
evaluation results with F-Score, Nearest Neighbor First-Tier (Tier1)
and Second-Tier (Tier2).
To compute the scores, we only consider the first K retrieved objects, where K is the number of objects in the ground truth class. We
found that this evaluation strategy gives consistent scores for bothtechniques in supervised and unsupervised categories. A precisionrecall plot for all methods is shown in Figure 8, and the evaluation
results with all metrics are shown in Table 3. Note that for supervised methods, a query can simply return all objects in the predictedcategory, resulting in extreme scores (either 0 or 1) in all metrics.
After averaging, the scores across methods become mostly similar.
This does not occur in the case of unsupervised learning.
In general, it can be seen that deep learning methods (Kanezaki,Tang, and Truong) outperform techniques that apply feature matching directly between an RGB-D object and a CAD model (Tashiro
and Li) by a significant margin. This performance gap could be
explained by the fact that the ground truth category of the CAD
models are provided. This is a strong hint that supervised methods
can utilize to turn the object retrieval problem into object classification, which can be solved using the state-of-the-art deep learningc 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset
1.0

Kanezaki-Single
Kanezaki-Thresh
Kanezaki-1000
Tang-3DCNN
Tang-MVCNN
Tang-Fuse
Tang-CDTNN
Truong-2D
Truong-3D
Tran-BoW-RoPS1
Tran-BoW-RoPS2
Tran-BoW-RoPS3
Tran-BoW-RoPS4
Li
TashiroPrecision

0.8
0.6
0.4
0.2
0.0
0.0

0.2

0.4

Recall

0.6

0.8

1.0

Figure 8: Precision-recall plots of all methods submitted by participants. While not yet perfect, most methods based on convolutional
neural networks perform effectively, either using multi-view or 3D
shape data. Feature matching techniques have moderately low accuracy, partly due to the high variance in shape geometry andappearance in each category of our dataset. (Best view on screenwith color)
techniques, i.e., convolutional neural networks. The bag-of-words
approach by Tran, while not based on deep learning, also exploits
the ground truth categorization hint to achieve good accuracy. In contrast, unsupervised learning techniques not making use of the groundtruth categories and simply relying on feature matching between
RGB-D objects and CAD models do not work very well, becauseeven the RGB-D objects and CAD models in a same category canvary greatly in both shape and appearance.
Among supervised methods, convolutional neural networks
trained by object images are competitive, if not better, than thosetrained by 3D volumes. For example, Kanezaki-single, -thresh, and-1000 variant top our evaluation chart, and outperform several other3D convolutional networks like Tang-3DCNN and Truong-3D. Despite that, it is worth noting that Tang-3DCNN is very competitive,
and so as two other view-based methods like Tang-MVCNN and
Truong-2D. Tang-3DCNN outperforms Truong-3D by a large gap
even though both use a binary volume representation and convolutional neural network for classification. A possible reason is that
Truong-3D used a more compact network and a lower resolution
volume than Tang-3DCNN’s, which results in lower discriminative
power.
From the unsupervised learning evaluation results, it can be seenthat the method by Tashiro is slightly more effective than Li’s. However, there are still potential for both methods to improve. For example, a better categorization where objects in the same category hasless shape and appearance variation could lead to better results.
Figure 9 shows the precision score plots for each category of
all methods. It can be seen that the maximum precisions in mostcategories are over 75%. Some categories, such as chair, display,
keyboard, cup, or bed, have precisions very close to 100%. Therealso exist some ambiguous categories with mean and median precisions below 50%, for example, box, machine, and printer, whose
maximum precisions are just slightly above 50%. This suggeststhat more discriminative training data might be required to reduce
ambiguity.
c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

Team

Run

Precision

RecallmAP

NDCG

KanezakiKanezakiKanezaki
Tang
Tang
Tang
Tang
TruongTruong
Tran
Tran
Tran
Tran
Li
TashiroSingleThresh
1000
3DCNN
MVCNN
Fuse
CDTNN
2D
3D
BoW-RoPS1
BoW-RoPS2
BoW-RoPS3
BoW-RoPS4
SBR-VC
GOLD

0.792
0.793
0.820
0.769
0.727
0.759
0.672
0.740
0.487
0.703
0.690
0.691
0.689
0.105
0.141

0.792
0.799
0.820
0.769
0.727
0.759
0.672
0.740
0.487
0.703
0.690
0.691
0.689
0.320
0.472

0.792
0.794
0.833
0.749
0.710
0.746
0.649
0.740
0.487
0.703
0.676
0.691
0.675
0.062
0.149

0.792
0.796
0.805
0.774
0.735
0.763
0.714
0.740
0.487
0.703
0.695
0.691
0.692
0.476
0.552

Table 3: Evaluation results on the test set. Kanezaki’s methods
top the scoreboard. Other methods based on convolutional neuralnetworks also perform well.
11. Conclusions
In this SHREC track, we proposed a new dataset to evaluate 3D
object retrieval algorithms. The query is an RGB-D object and thetarget is a CAD model. By evaluating various techniques submitted
by the participants, we found that for this problem, convolutional
neural networks with multi-view representation are currently among
the most effective, followed by those with 3D volume representation. Techniques based on unsupervised learning appear to be lesscompetitive in our benchmark.
There are a few promising future research directions. First, thedataset can be extended to include more objects and categories, e.g.,
those from ScanNet [DCS∗ 17] which is recently available on arXiv.
In addition, stricter categorization in the ground truth would be
helpful to benchmark the retrieval more rigorously. Second, object
fragments could be added to pose an even more challenging trackfor object detection: each fragment can contain multiple objects
and background structure such as floor and walls. Third, the current
ground truth categorization for CAD models can be removed tochallenge approaches based on supervised learning.
Acknowledgement
We acknowledge the support of the SUTD Digital Manufacturing
and Design (DManD) Centre which is supported by the National
Research Foundation (NRF) of Singapore. Bo Li is partially supported by the University of Southern Mississippi Faculty Startup
Funds Award. The research done by Henry Johan is supported by
the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centres in Singapore Funding
Initiative. The research conducted by Lap-Fai Yu is supported by
the National Science Foundation under award number 1565978.
References[BETVG08] BAY H., E SS A., T UYTELAARS T., VAN G OOL L.: Speededup robust features (surf). Computer Vision and Image Understanding 110,
3 (2008). 6

Hua et al. / SHREC’17: RGB-D to CAD Retrieval with ObjectNN Dataset

Figure 9: Precision plots of all methods for each category in the dataset. Mean, median, and standard deviation of the precision score percategory are provided at the bottom rows. As can be seen, most categories can be well recognized with maximum accuracy over 75%. The
mean and median statistics also suggest that more training data might be necessary to improve the accuracy for a few challenging categories
such as box, machine and printer. (Best view on screen with color)[BMP02] B ELONGIE S., M ALIK J., P UZICHA J.: Shape matching andobject recognition using shape contexts. IEEE Transactions on PatternAnalysis and Machine Intelligence 24, 4 (2002). 6[CFG∗ 15] C HANG A. X., F UNKHOUSER T., G UIBAS L., H ANRAHAN P.,
H UANG Q., L I Z., S AVARESE S., S AVVA M., S ONG S., S U H., X IAO
J., Y I L., Y U F.: ShapeNet: An Information-Rich 3D Model Repository.
arXiv:1512.03012 (2015). 1, 2
[DCS∗ 17] DAI A., C HANG A. X., S AVVA M., H ALBER M.,
F UNKHOUSER T., N IESSNER M.: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. arXiv:1702.04405 (2017). 7[GDB∗ 15]G ODIL A., D UTAGACI H., B USTOS B., C HOI S., D ONG S.,
F URUYA T., L I H., L INK N., M ORIYAMA A., M ERUANE R., O HBUCHI
R., PAULUS D., S CHRECK T., S EIB V., S IPIRAN I., Y IN H., Z HANG
C.: Range scans based 3d shape retrieval. In Eurographics Workshop on
3D Object Retrieval (3DOR) (2015). 2[GSB∗ 13] G UO Y., S OHEL F. A., B ENNAMOUN M., L U M., WAN J.:
Rotational projection statistics for 3D local surface description and object
recognition. International Journal of Computer Vision 105, 1 (2013). 5[HPN∗ 16]

H UA B.-S., P HAM Q.-H., N GUYEN D. T., T RAN M.-K., Y U
L.-F., Y EUNG S.-K.: Scenenn: A scene meshes dataset with annotations.
In International Conference on 3D Vision (3DV) (2016). 1, 2

[KMN16] K ANEZAKI A., M ATSUSHITA Y., N ISHIDA Y.: Rotationnet:
Joint learning of object classification and viewpoint estimation using
unaligned 3d object dataset. arXiv:1603.06208 (2016). 3[LLJ13] L I B., L U Y., J OHAN H.: Sketch-based 3D model retrieval by
viewpoint entropy-based adaptive view clustering. In Eurographics Workshop on 3D Object Retrieval (3DOR) (2013). 6[LLJF16] L I B., L U Y., J OHAN H., FARES R.: Sketch-based 3D model
retrieval utilizing adaptive view clustering and semantic information.
Multimedia Tools and Applications (2016). 6[MS15] M ATURANA D., S CHERER S.: Voxnet: A 3d convolutional neuralnetwork for real-time object recognition. In IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS) (2015). 4, 5
[NNT∗ 15] N GUYEN V., N GO T. D., T RAN M., L E D., D UONG D. A.: A
combination of spatial pyramid and inverted index for large-scale image
retrieval. International Journal of Multimedia Data Engineering andManagement 6, 2 (2015). 5[PCI∗ 08]

P HILBIN J., C HUM O., I SARD M., S IVIC J., Z ISSERMAN A.:
Lost in quantization: Improving particular object retrieval in large scale
image databases. In CVPR (2008). 5[PPG∗ 16]

PASCOAL P. B., P ROENÃĞA P., G ASPAR F., D IAS M. S., F ER REIRA A., TATSUMA A., AONO M., L OGOGLU K. B., K ALKAN S.,
T EMIZEL A., L I B., J OHAN H., L U Y., S EIB V., L INK N., PAULUS

D.: Shape Retrieval of Low-Cost RGB-D Captures. In Eurographics
Workshop on 3D Object Retrieval (3DOR) (2016). 2[PPTP10] PAPADAKIS P., P RATIKAKIS I., T HEOHARIS T., P ERANTONIS
S.: Panorama: A 3d shape descriptor based on panoramic views for
unsupervised 3D object retrieval. International Journal of Computer
Vision 89, 2 (2010). 6[PSA∗ 16]

P RATIKAKIS I., S AVELONAS M., A RNAOUTOGLOU F., I OAN NAKIS G., KOUTSOUDIS A., T HEOHARIS T., T RAN M.-T., N GUYEN
V.-T., P HAM V.-K., N GUYEN H.-D., L E H.-A., T RAN B.-H., T O
Q., T RUONG M.-B., P HAN T., N GUYEN M.-D., T HAN T.-A., M AC
K.-N., D O M., D UONG A.-D., F URUYA T., O HBUCHI R., AONO M.,
TASHIRO S., P ICKUP D., S UN X., ROSIN P., M ARTIN R.: Partial ShapeQueries for 3D Object Retrieval. In Eurographics Workshop on 3D Object
Retrieval (3DOR) (2016). 2, 5
[SGMC15] S ERRA G., G RANA C., M ANFREDI M., C UCCHIARA R.:
Gold: Gaussians of local descriptors for image representation. Computer
Vision and Image Understanding 134 (2015). 6[SHKF12] S ILBERMAN N., H OIEM D., KOHLI P., F ERGUS R.: Indoor
segmentation and support inference from rgbd images. In ECCV (2012).
2[SKP15] S CHROFF F., K ALENICHENKO D., P HILBIN J.: Facenet: A unified embedding for face recognition and clustering. In CVPR (2015).
4[SMKLM15] S U H., M AJI S., K ALOGERAKIS E., L EARNED -M ILLER
E. G.: Multi-view convolutional neural networks for 3D shape recognition. In ICCV (2015). 3, 4
[SZ03] S IVIC J., Z ISSERMAN A.: Video google: A text retrieval approachto object matching in videos. In ICCV (2003). 5[SZ14] S IMONYAN K., Z ISSERMAN A.: Very deep convolutional networks
for large-scale image recognition. arXiv:1409.1556 (2014). 4[TNHYY16] T HANH N GUYEN D., H UA B.-S., Y U L.-F., Y EUNG S.-K.:
A robust 3d-2d interactive tool for scene segmentation and annotation.
arXiv:1610.05883 (2016). 2[Vra04] V RANIC D.: 3D Model Retrieval. PhD thesis, University ofLeipzig, 2004. 6[ZJS13] Z HU C., J EGOU H., S ATOH S.: Query-adaptive asymmetricaldissimilarities for visual object retrieval. In ICCV (2013). 5

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.