Multimedia Tools and Applications (2019) 78:35157–35178
https://doi.org/10.1007/s11042-019-08078-y

Feature set aggregator: unsupervised representation
learning of sets for their comparison
Takahiko Furuya 1 & Ryutarou Ohbuchi 1
Received: 10 August 2018 / Revised: 5 June 2019 / Accepted: 2 August 2019 /Published online: 20 August 2019
# Springer Science+Business Media, LLC, part of Springer Nature 2019

Abstract
Unsupervised representation learning of unlabeled multimedia data is important yet challenging problem for their indexing, clustering, and retrieval. There have been many attempts tolearn representation from a collection of unlabeled 2D images. In contrast, however, lessattention has been paid to unsupervised representation learning for unordered sets of highdimensional feature vectors, which are often used to describe multimedia data. One such
example is set of local visual features to describe a 2D image. This paper proposes a novel
algorithm called Feature Set Aggregator (FSA) for accurate and efficient comparison amongsets of high-dimensional features. FSA learns representation, or embedding, of unorderedfeature sets via optimization using a combination of two training objectives, that are, setreconstruction and set embedding, carefully designed for set-to-set comparison. Experimental
evaluation under three multimedia information retrieval scenarios using 3D shapes, 2D
images, and text documents demonstrates efficacy as well as generality of the proposed
algorithm.
Keywords Unsupervised learning . Feature aggregation . Set-to-set comparison . Multimedia
information retrieval . Autoencoder . Neural network

1 Introduction
Recent advances in deep learning have achieved great successes in many computer visiontasks. The successes, especially for 2D images, are attributed in large part to availability oflabels required to effectively train deep neural networks. However, in practice, many multimedia data collections are often left unlabeled due to high cost of manually annotating large* Takahiko Furuya
takahikof@yamanashi.ac.jp
Ryutarou Ohbuchi
ohbuchi@yamanashi.ac.jp

1

University of Yamanashi, 4-3-11 Takeda, Kofu-shi, Yamanashi-ken 400-8511, Japan

35158Multimedia Tools and Applications (2019) 78:35157–35178

number of data items. Motivated by this situation, unsupervised representation learning, whichattempts to obtain meaningful features from unlabeled dataset, has been gaining muchattention for indexing, clustering, or retrieval of multimedia data without labels.
A multimedia datum, that is, a 2D image, a 3D shape, or a text document, can often be
represented by an unordered set of high-dimensional feature vectors. For example, a 2D image
is often represented as a set of local visual features (e.g., SIFT [40]). A 3D shape is represented
as a set of local geometric features (e.g., [12]) or 3D point set (e.g., [6]). Also, Bag-of-Words
model treats a document as a set of words where each word is described by a high-dimensional
vector (e.g., [62]). Such unordered sets that ignore spatial relations among parts of the datum
are robust against variation of context, for example, varied poses of articulated objects or
swapped phrases in a sentence.
This paper addresses the problem of Set-to-Set Comparison (SSC) that measures distancesamong unlabeled data, each of which is described by a set of high-dimensional feature vectors.
Achieving accuracy and efficiency for SSC under unsupervised setting is quite challenging.
Earth Mover’s distance [56] or Hausdorff distance [22] have been conventionally employed
for SSC. However, these distances are sensitive to outlier features and suffer from high cost offinding element-to-element correspondences between two sets.
To improve accuracy and efficiency of SSC, feature aggregation algorithms including Bagof-Features [8] and its variants (e.g., [54, 66]) have been proposed. Feature aggregation is
performed by encoding each feature within the set and then pooling the encoded features toform a single feature vector that succinctly describes the set. Most of the feature aggregation
algorithms optimize feature encoding via codebook learning. However, the aggregated features, typically generated by simple sum of the encoded features, are not necessarily optimized
for SSC. In addition, aggregated features are often very high-dimensional (e.g., more than 10 k
dimensions [54]) and thus are not suitable for efficient multimedia indexing, clustering, andretrieval.
In this paper, towards more accurate and efficient SSC, we propose a novel unsupervised
representation learning algorithm dedicated to multimedia data described by unordered featuresets. A neural network, called Feature Set Aggregator (FSA), aggregates an input feature set to
a compact representation for effective SSC. Training of the FSA is more data-driven than theprevious SSC algorithms ([8, 54, 66]). To effectively train the FSA, we design two training
objectives for SSC. The first objective is set reconstruction using angular chamfer distancewhich aims to learn accurate representations of “whitened” input feature sets. The setreconstruction loss differs from vector reconstruction loss used by standard autoencoders[16]. The second objective is set embedding using triplets on family of sets to learn “smooth”
embedded feature space appropriate for SSC. Different from the common label-based triplet
sampling (e.g., [23]), our approach creates diverse triplets from an unlabeled collection offeature sets.
Efficacy and generality of the FSA are demonstrated via experiments under three multimedia information retrieval scenarios, that are, 3D shape retrieval, 2D image retrieval, and text
document retrieval. In majority of the retrieval experiments, the FSA significantly outperforms
the conventional SSC algorithms including state-of-the-art feature aggregation algorithms.
Contribution of this paper can be summarized as follows;&

We propose a novel unsupervised representation learning algorithm for comparison amongsets, where each set consists of unordered features vectors. The algorithm, called Feature

Multimedia Tools and Applications (2019) 78:35157–35178

&

35159

Set Aggregator, uses a pair of training objectives designed for accurate reconstruction of
set and accurate embedding of aggregated features.
We extensively evaluate the proposed algorithm under three multimedia information
retrieval scenarios to demonstrate accuracy, efficiency, and generality of the algorithm.

2 Related work
2.1 Supervised deep learning for unordered Sets
Motivated in part by successes in 2D image recognition, a variety of Deep Neural Network(DNN) architectures for diverse data representation, including those for unordered sets, havebeen proposed. Zaheer et al. [74] proposed a DNN called DeepSets that processes sets of highdimensional features. To obtain invariance against permutation of elements in a set, DeepSets
performs max-pooling over mid-level features that appear at an intermediate layer of the DNN
to form a global feature of the set. Application of DeepSets includes 3D point set classification,
set anomaly detection, and others. The DNN proposed by Furuya and Ohbuchi [14] aggregatesa set of local geometric features to describe a 3D shape for 3D model retrieval. PointNet [6]and its variant PointNet++ [55], both by Qi et al., enables end-to-end learning of 3D point sets
for their classification and semantic segmentation. For the task of person re-identification, Liuet al. [39] proposed a DNN that handles sets comprising 2D images of identical person. In thefield of 2D image recognition, Arandjelović et al. [3] and Lin et al. [36] proposed feature
aggregation modules called NetVLAD and NeXtVLAD, respectively. These modules treat afeature map activated in a hidden layer of Convolutional Neural Network (CNN) as anunordered set of visual features. The set of visual features is aggregated to a single feature
vector for subsequent classification.
Note, however, that all the DNNs described above are trained by using a large collection oflabeled data. On the other hand, our approach is unsupervised. That is, representations ofunordered feature sets are learned from multimedia data without having labels.

2.2 Unsupervised representation learning for unordered Sets
Learning meaningful feature representation from unlabeled data is challenging since directsupervision can’t be used. Thus, surrogate tasks are generally adopted to train DNNs without
supervision by labels. Autoencoder [16] and its variants (e.g., [46, 63]) learn feature representation via reconstruction task. An autoencoder consists of a pair of an encoder and adecoder. The encoder transforms an input to its latent representation, or code, and the decoder
reconstructs the input from the code. Other surrogate tasks to learn visual representation of 2D
images include, for example, solving jigsaw puzzle [49], sorting video sequence [30], tracking
objects in video [65], inpainting images [53], and generating images [18].
In contrast to unsupervised representation learning for 2D images, there has been little study
on representation learning using unlabeled sets of high-dimensional features. Recently,
Achlioptas et al. [2] and Yang et al. [73] proposed autoencoders for unordered sets of 3D
points to learn 3D shape feature representation. Compared to the autoencoders for 3D point set,
our approach is more general. That is, the FSA can handle unordered sets of feature vectors
having arbitrary number of dimensions including but not limited to 3.

35160Multimedia Tools and Applications (2019) 78:35157–35178

Meanwhile, unsupervised feature aggregation is widely adopted to obtain a representation
for inter-set comparison of an unordered set of features. Bag-of-Features [8], Localityconstrained Linear coding [66], Fisher vector coding [54], and Vector of Locally Aggregated
Descriptors [25] are popular feature aggregation algorithms. They learn a codebook containing
representative features, or codewords. By using the codebook, each input feature in an input
set is encoded into one of the codewords. The encoding is performed by using first-order
statistics [8, 66] or higher-order statistics [25, 54] in the input feature space. The encodedfeatures are pooled by summing to form an aggregated feature per set for classification or SSC.
More recently, feature aggregation algorithms aiming at more accurate feature encoding [13,38] or codebook-free aggregation [37, 67] have been proposed. However, these feature
aggregation algorithms are not necessarily optimized for SSC. In addition, these methods
often yield very high-dimensional aggregated features. In this study, we introduce a neuralnetwork to learn representations of feature sets for accurate and efficient SSC than the previous
approaches.
In addition to multimedia indexing and retrieval, use cases of aggregated features that
describe unordered sets include biometric identification. For example, matching fingerprint
images or palmprint images [43] involves comparison among unordered sets of keypoints
detected on skin images. In order to secure biometric verification system and protect privacy ofusers, skin images are often encoded to their feature representations. To achieve accurate andfast verification, previous studies adopt techniques such as feature selection [33], featurecompression [41], or feature fusion [32, 34, 35, 42]. Although we don’t conduct experiments
on fingerprint/palmprint recognition in this paper, our proposed learning algorithm for unordered sets would be applicable to such biometric identification problems that require comparison of unordered sets.

3 Proposed algorithm
3.1 Overview of feature set aggregator
A Feature Set Aggregator (FSA) is a shallow neural network for learning accurate and compact
representations of unordered sets of high-dimensional features. Figure 1 shows a processing
pipeline of the FSA. We use 3D shapes as the datatype for the explanation here, but thearguments hold for other multimedia data as 2D images and text documents. The FSA is anasymmetric tandem of an encoder and a decoder. The encoder encodes and aggregates a set of(pre-whitened) features via three processing steps. Each input feature is first transformed into asparse code. The set of sparse-code is then pooled by sum to form a single feature vector per
input set. Finally, the pooled feature is embedded, via metric learning, into a feature spacesuitable for SSC.
The FSA is trained by using a combination of two training objectives. The first objective,
set reconstruction, attempts to learn accurate representations of sets. We propose to use angularchamfer distance as the loss function for reconstruction of sets of whitened input features. The
second objective, set embedding, is designed to learn accurate distance metric among sets. To
train FSA using these objectives, the algorithm generates, from unlabeled training data, a large
number of triplets by generating family of sets from input sets.
During the inference, the encoder part only of the FSA is utilized to obtain an embeddedfeature of an input set. That is, the decoder part of the FSA is not used for feature aggregation

Multimedia Tools and Applications (2019) 78:35157–35178

35161

Fig. 1 Proposed Feature Set Aggregator (FSA) effectively and efficiently compares sets of high-dimensionalfeatures in the embedded feature space. Embedding is optimized by using the two training objectives, i.e., setreconstruction and set embedding, designed for set-to-set comparison

after the training. SSC is performed by computing Euclidean distances among the embeddedfeatures produced by the encoder. Note that, although we evaluate the FSA using multimedia
information retrieval scenario, the FSA would be applicable to other tasks such as clusteringthat requires inter-set distance/ similarity computation.

3.2 Architecture of feature set aggregator
Encoder. The encoder function f transforms a set SI = {x1, ..., xn}∈ℝdf of n feature vectors
having df dimensions into a de-dimensional feature vector. For example, these numbers would
be n = 512, df = 64 and de = 256. For simplicity, we assume the cardinality n of each set is equalamong all the sets contained in a dataset. As shown in Fig. 1, the encoder comprises threeblocks, that are, the sparse coding block, the average pooling block, and the sparse embedding
block. The sparse coding block has an architecture similar to an encoder part of k-sparseautoencoder [46]. That is, each input feature is transformed into a dc-dimensional vector via asingle fully-connected (FC) layer activated by ReLU [17] function. Then, the vector of
activation values is sparsified by keeping the kc largest activations while setting the rest to
zero. Formally, each input feature xi is sparsely encoded to hi by using the following equation.
hi ¼ top kðReLUðxi Wsc þ bsc ÞÞ

ð1Þ

In Eq. 1, Wsc is a linear projection matrix having size df × dc and bsc is a dc-dimensional bias
vector. Wsc and bsc are tuned during the training. The function “top_k” chooses the kc largest
values from the vector activated by ReLU. We use dc = 2048 and kc = 5 throughout theexperiments unless otherwise stated.
The average pooling block aggregates the n sparse vectors to a single dc-dimensional vector
per set by using the following equation.

35162Multimedia Tools and Applications (2019) 78:35157–35178

hpooled ¼

1 n
∑ hi
n i¼1

ð2Þ

Pooling operation provides for invariance against permutation of elements in the set andreduces both spatial and temporal costs for comparison among sets. We found that, inpreliminary experiments, average-pooling performed better than max-pooling.
The sparse embedding block refines the pooled feature hpooled to a more compact andaccurate feature. Similar to the sparse coding block, we use the following equation to obtain dedimensional vector f.



ð3Þ
f ¼ top k ReLU hpooled Wse þ bse
In Eq. 3, Wse is a matrix having size of dc × de and bse is a de-dimensional vector. The function“top_k” in Eq. 3 extracts the largest ke activations from the vector produced by ReLU. We set
the number of non-zero elements ke to roughly one tenth of de. For example, when we use de =256, ke is set to 25. Finally, the sparse vector f is normalized by its Euclidean norm to form anembedded feature of the input set.
Injecting sparseness into the encoder is motivated by the success of feature aggregation
algorithms based on sparse coding (e.g., [13, 66]). Sparse coding is expected to enhancesaliency of features. Positive impact of sparseness adopted by FSA on SSC accuracy will bedemonstrated in the experimental section.
Decoder. The decoder reconstructs the input feature set from its embedding f produced by
the encoder. We use an FC layer without activation function for decoding. As shown in Eq. 4,
the de-dimensional embedded feature f is linearly transformed to an ndf-dimensional vector r.
Wd is a matrix having size of de × ndf and bd is a ndf-dimensional bias vector. The vector r isthen reshaped to a matrix having size n × df that represents the reconstructed set of n features
having df dimensions.
r ¼ f Wd þ bd

ð4ÞPreprocessing. Prior to be passed onto the FSA, each input feature is normalized by using
PCA-whitening [24] for fast and effective training. PCA-whitening reduces dimensionality offeatures and spheres their distribution to have zero-mean, unit-variance, and no correlation
among the dimensions. Equation 5 computes the normalized input xnormalized from the input
feature x, where μ(x) and σ(x) are mean and standard deviation of x, respectively. P is a PCA
projection matrix learned by 250,000 input features randomly collected from the training
dataset. Matrix P of size dorig × df reduces dimensionality of input feature x from dorig to df. The
value dorig depends on the input feature while df is set manually. In this study, df is set at 64 formost cases in the experiments.
xnormalized ¼x−μðxÞ
PσðxÞ

ð5Þ

3.3 Effective training of feature set aggregator
The FSA is trained by using a combination of two objective functions, that are, set reconstruction loss LR and set embedding loss LE described below. Overall loss for the FSA is a

Multimedia Tools and Applications (2019) 78:35157–35178

35163

weighted sum of the two losses; L = LR + αLE. The hyper-parameter α balances the two terms.
Impact of α on retrieval accuracy will be evaluated in the experiments.

Set reconstruction using angular chamfer distance We require a reconstruction loss thatsatisfies the following two criteria; (1) suitable for comparing sets of whitened features thathave sphered distribution, and (2) invariant against permutation of elements both in input sets
and reconstructed output sets. To meet these criteria, we propose to use angular chamferdistance shown in Eq. 6 as the set reconstruction loss LR.



!
N
1
xy
1
xy
LR ¼ ∑∑ min 1−ð6Þ
þ ∑ min 1−
n x∈S I y∈S O
kxkkykn y∈S O x∈S I
kxkkyki¼1
In Eq. 6, SI and SO indicate the input and the output feature set of the FSA, respectively. n is
cardinality of the sets and N is the number of sets used for training. While standard chamferdistance usually adopts Euclidean distance to find a pair of elements that are closest across two
sets (e.g., [2, 73]), we employ Cosine distance (i.e., angle) as it is a better metric for comparing
whitened high-dimensional features having sphered distribution. The experiments will demonstrate validity of choosing the angular chamfer distance.

Set embedding using triplets on family of sets For accurate SSC, sets consisting of similarfeatures should be close to each other, while sets having dissimilar features should be distantfrom each other, in the embedded feature space. Such embedded feature space is obtained via
training by using triplets. We use a triplet (Sa, Sp, Sn), in which the anchor set Sa is considered
to be more similar to the positive set Sp than the negative set Sn.
To collect diverse and a large number of such triplets from unlabeled training dataset, we
propose a novel triplet generation algorithm based on family of sets (see Fig. 2). Specifically,
we first randomly pick two different sets, i.e., SA and SB, from the training dataset. Assume thatboth SA and SB have n elements, and importantly, the pair of sets is at least partially distinct. We
then generate a family of sets out of the union SA ∪ SB to be used as either Sa, Sp, or Sn. That is,a generated set S ⊂ SA ∪ SB consists of rn elements randomly sampled from SA and (1 − r)nelements randomly sampled from SB where the ratio r is randomly determined from a uniform
distribution in the range [0, 1]. From SA ∪ SB, three member sets of a triplet, Sa, Sp, and Sn, arecreated by using three different ratios ra, rp, and rn such that ra < rp < rn. We form numerous
such triplets (Sa, Sp, Sn) as training samples for set embedding.

Fig. 2 Low-cost triplet generation using family of sets. We use n = 10, ra = 0.1, rp = 0.4, rn = 0.8 for description
in the figure

35164Multimedia Tools and Applications (2019) 78:35157–35178

Intuitively, the relation among the sets in the triplet, i.e., “Sa is more similar to Sp than Sn.”would hold since Sa is likely to share more elements with Sp than Sn. Our triplet generation
algorithm enables us to collect numerous and diverse triplets at very low cost from theunlabeled training dataset. We use the triplet hinge loss [57] shown in Eq. 7 as the embedding
loss LE, where f (·) denotes the encoder function of the FSA. Note that output of the function fcorresponds to the embedded feature vector f described in Section 3.2. Minimizing LE leads
FSA to form the desired embedded feature space where Sa lies closer to Sp than Sn.
 

N
 2
LE ¼ ∑ max 0;  f ðS a Þ−f S p 2 −k f ðS a Þ−f ðS n Þk22 þ 1

ð7Þ

i¼1

Optimization We employ stochastic gradient descent algorithm with mini batch to train the
FSA. Specifically, each parameter of the FSA is iteratively updated by using the following
equation.
wi ←wi −η

∂L
∂wi

ð8Þ

In Eq. 8, L = LR + αLE is the overall objective function and wi indicates each element ofprojection matrices and bias vectors of FSA, i.e., Wsc, bsc, Wse, bse, Wd, and bd. η is a learning
rate. Details on optimization will be described in Section 4.1.

4 Experiments and results
4.1 Experimental setup
We verify effectiveness of the proposed FSA under 3D shape retrieval, 2D image retrieval, anddocument retrieval scenarios where each datum is described by an unlabeled feature set.

Benchmark datasets We use nine datasets. Table 1, Table 2, and Table 3 summarize statisticsof the datasets. For 3D shape retrieval, we use ModelNet10, ModelNet40 [69] andShapeNetCore55 [5] that consist of polygonal 3D models. We use the training/testing data
split provided by [5, 69]. For 2D image retrieval, we use Caltech101 [10], Flowers102 [48],and Animals with Attributes 2 (AwA2) [71] datasets. Although the images of AwA2 haveanimal class labels and attribute labels, we use only animal class labels as groundtruth forevaluation. For document retrieval, we use News20 [29], Amazon4 [4], and DBpedia14 [31]datasets. News20 includes text messages posted to newsgroups. Amazon4 contains productreviews. DBpedia14 comprises documents that summarize Wikipedia articles. We use only thetesting set of DBpedia having 70,000 documents for our experiments. For Caltech101,
Flowers102, AwA2, News20, Amazon4, and DBpedia14, we randomly split the whole dataset
so that each category contains nearly equal number of samples between the training set and thetesting set.
For each dataset, the training set is used to train the FSA, while the testing set is used forevaluation. We use Mean Average Precision (MAP) [%] as a quantitative measure of retrieval
accuracy. For each query of category c, retrieval targets of the same category c are treated as

35165Multimedia Tools and Applications (2019) 78:35157–35178
Table 1 Benchmark datasets for 3D shape retrieval
datasets

ModelNet10

ModelNet40

ShapeNetCore55# of training data
# of testing data
# of categories
examples ofcategory

3991
9843
35,764908
2468
10,265104055
bathtub, bed, chair, table,
airplane, chair, plant, car, knife, guitar, table, lamp, sofa,
monitor
piano
clock

correct while those belonging to categories other than c are treated as incorrect. Since trainingof the FSA is affected by randomness in its initialization and triplet generation steps, we run
every experiment three times and their averaged accuracies are reported. We do not reportdeviations of accuracy since they are within 1% in MAP for most cases.

Features We use multiple (i.e., two to four) features per retrieval scenario to make theevaluation convincing. Most of the features described below have dimensionality of about
100 to 500. The preprocessing by using PCA-whitening compresses them down to df = 64
dimensions.
For 3D shapes, we employ RoPS [20], POD [12], and raw oriented points (OP) as local
geometric features. A 3D polygon model is first converted into a set of 2048 oriented points byusing the algorithm by Ohbuchi et al. [50]. We then extract a set of either 512 RoPS or 512
POD features from the oriented point set to describe shape of the 3D model by the set offeatures. We also use a set of randomly subsampled 1024 OPs to describe the 3D model where
each OP is represented by a 6-dimensional vector (i.e., 3 dimensions for point position and theother 3 dimensions for point orientation). As an exception, we don’t reduce the dimensionalityof OPs at the preprocessing stage for it already has low dimensionality.
For 2D images, we use Dense SIFT (DSIFT) and convolutional neural network activation(CNNA) as local visual features. From each image, we extract a set of 1024 SIFT [40] features
sampled at random positions and at random scales. For CNNA, we use three CNNs, that are,
AlexNet [27], VGG19 [60], and ResNet50 [21] trained by using ImageNet corpus [9]. Each
CNNA feature is denoted by “CNNA-Alex”, “CNNA-Vgg19”, and “CNNA-Res50” depending on the CNN used for local feature extraction. CNNA-Alex is extracted as follows. We first
randomly crop an image to obtain 20 patches having size 224 × 224 pixels each. Each patch is
fed into AlexNet to obtain the pool5 feature map having 6 × 6 spatial resolution. We partition
the feature map into 36 feature vectors. As a result, we obtain a set of 20 × 36 = 720 CNNAAlex features per image. Similarly, for CNNA-Vgg19 (or CNNA-Res50), we feed 5 patches
randomly cropped per image into the CNN. We then obtain the block4_pool feature map of
VGG19 (or the activation_48 feature map of ResNet50) having 14 × 14 spatial resolution.
Table 2 Benchmark datasets for 2D image retrieval
datasets

Caltech101# of training data
# of testing data
# of categories
examples of category

4367
4310
101
ant, ferry, crocodile,umbrella, mandolin

Flowers102

AwA2
4119
4070
102

lotus, rose,
marigold, sunflower,
hibiscus

18,67018,65250antelope, dalmatian,hippopotamus, wolf

35166

Multimedia Tools and Applications (2019) 78:35157–35178

Table 3 Benchmark datasets for document retrieval
datasets

News20

Amazon4# of training data
5652
4000
# of testing data
5662
4000
# of categories
20
4
examples ofbaseball, motorcycle, gun,book, DVD, electronics,
category
religion, space
kitchen

DBpedia14
35,00035,00014
company, animal, film, artist,building

Thus, we obtain 5 × 14 × 14 = 980 CNNA-Vgg19 features (or 980 CNNA-Res50 features) per
image.
For text documents, we use word vectors learned by using Latent Semantic Analysis (LSA)[28] or Word2Vec (W2V) [47] algorithms. After removing stop words, each document is
described by a set of 200 words randomly sampled from the document. Each word istransformed into a word vector by using the trained model either of LSA [19] or W2V [68].

Competitors We compare the FSA against 13 unsupervised SSC algorithms roughly grouped
into four approaches described below.
&
&

&

&

No aggregation. Chamfer distance (CD) [15] performs SSC by all-pair matching offeatures between two sets. We also tried SSC using Earth Mover’s distance, but we found
it was infeasible since computing bijections of feature vectors was temporally inefficient.
Aggregation using first-order statistics. Each feature in a set is encoded based on
similarities, or weights, to one or more codewords. The encoded features are pooled, bysumming, to form an aggregated feature per set. We adopt Bag-of-Features (BF) [8],Locality-constrained Linear coding (LL) [66], and Database-adaptive k-Sparse
Autoencoder (DkSA) [13] as competitors. Additionally, we evaluate sum pooling of raw
input features, denoted by “sum”. Without elaborate encoding, this probably is the simplestof feature aggregation methods.
Aggregation using higher-order statistics. A set of features is aggregated by using
higher-order statistics, such as covariance, of features distributed around codewords. We
use Vector of Locally Aggregated Descriptors (VLAD) [25], Fisher Vector coding (FV)[54], Super Vector coding (SV) [70], Fine-residual VLAD (FVLAD) [38], Second-orderaggregation with Matrix Power Normalization (SMPN) [37], and Grassmann Pooling (GP)[67] as competitors.
Aggregation using neural network. NetVLAD [3] and NeXtVLAD [36] are feature
aggregation modules which can be incorporated in neural networks having various
architectures. To verify effectiveness of feature aggregation by FSA, we replace theencoder part of FSA described in Section 3.2 with NetVLAD or NeXtVLAD modules.
Note that NetVLAD and NeXtVLAD do not explicitly constrain embedded features to besparse during their training. This way, we can evaluate the effect sparseness has on
accuracy. For a fair comparison, we use the same decoder and loss function for NetVLAD,
NeXtVLAD, as well as for FSA.

Implementation details of FSA To train the FSA, we use Adam optimizer [26] with initial
learning rate η = 0.001 to minimize the overall objective function L described in Section 3.3.

35167

Multimedia Tools and Applications (2019) 78:35157–35178
Table 4 Values of hyper-parameters for FSA used in the experiments
hyper-parameters

3D shape retrieval

2D image retrievaland document retrieval(dc, kc)(de, ke)α(2048, 5)(2048, 200) or (256, 25)
5 for RoPS and POD feature0.5 for OP feature(2048, 5)(2048, 200) or (256, 25)
1Each mini-batch contains 8 triplets (i.e., 24 sets) that are created on-the-fly during training.
Training is iterated for 100 epochs. We used TensorFlow r1.7 [1] to implement the FSA.
Table 4 summarizes values of the hyper-parameters for the FSA. We used the values inTable 4 unless otherwise noted. As we will show in Section 4.4, the balancing parameter α has
an impact on retrieval accuracy. Therefore, in the case of the 3D shape retrieval scenario, we
searched for an optimal value of α for each input feature. On the other hand, for 2D image anddocument retrieval, we fixed α to 1. This is because searching for optimal values of α for all
the pairs of input features and datasets in these scenarios is computationally too expensive.

4.2 Comparison with existing unsupervised SSC algorithms
Accuracy Table 5, Table 6, and Table 7 compare retrieval accuracies of 3D shapes, 2D images,
and documents, respectively. Each table shows MAP scores for every combination of the input
features and the 14 SSC algorithms including the proposed FSA. BF, LL, DkSA, NetVLAD,
and NeXtVLAD produce aggregated features having 2048 dimensions, while VLAD, FV, SV,
and FVLAD produce aggregated features with nearly 8000 dimensions. For the FSA, we use
two embedded feature dimensions, i.e., de = 2048 and 256, denoted by “FSA-2048” and “FSA256”, respectively, in the tables. We do not report accuracies of CD for the AwA2 dataset andthe DBpedia14 dataset. This is because chamfer distance when applied to such a large dataset
had excessively high temporal cost.
Table 5 3D shape retrieval accuracy (MAP [%])
Algorithms

CD [15]sum
BF [8]LL [66]
DkSA [13]VLAD [25]FVLAD [38]
FV [54]
SV [70]SMPN [37]GP [67]NetVLAD [3]
NeXtVLAD [36]FSA-2048 (proposed)
FSA-256 (proposed)

ModelNet10

ModelNet40

ShapeNetCore55

RoPS

POD

OP

RoPS

POD

OP

RoPS

POD

OP

50.4
38.0
54.1
54.0
59.6
50.4
50.2
51.9
52.7
55.4
55.2
58.1
60.3
70.0
69.4

50.3
42.6
57.0
55.6
61.2
53.7
53.8
54.1
54.1
58.6
57.3
60.4
61.4
71.6
71.4

51.5
19.4
48.6
50.8
53.6
50.1
47.9
47.9
47.0
39.2
31.5
51.3
53.2
68.1
67.9

39.0
27.1
38.0
40.7
44.3
37.4
38.5
39.8
40.1
42.7
42.1
41.8
42.3
52.3
51.6

38.4
30.5
40.6
42.7
48.4
41.8
41.5
42.6
42.5
46.9
45.9
45.1
44.8
56.6
55.5

40.8
10.4
38.1
43.4
38.4
36.7
35.0
37.3
34.5
23.1
15.8
34.7
36.7
44.5
43.8

35.5
26.5
36.6
37.8
42.8
34.7
35.3
36.7
37.4
41.2
41.0
42.0
40.9
51.5
50.6

35.7
31.3
40.3
41.7
47.9
40.2
40.0
43.7
41.6
46.7
45.2
43.3
43.6
55.0
53.8

41.6
10.9
41.1
45.4
38.1
39.0
35.6
39.2
33.3
34.5
25.8
40.9
44.2
53.9
53.5

CD [15]
sum
BF [8]LL [66]
DkSA [13]VLAD [25]FVLAD [38]
FV [54]
SV [70]SMPN [37]GP [67]NetVLAD [3]
NeXtVLAD [36]FSA-2048(proposed)
FSA-256 (proposed)

algorithms

13.8
41.7
37.8
38.8
43.1
40.7
36.6
39.6
40.9
44.3
42.1
42.9
44.2
47.4

45.0

11.5
12.9
15.3
18.0
20.5
22.4
23.1
22.4
21.7
22.5
20.4
21.3
21.7
27.9

26.5

27.0
65.0
42.4
43.6
62.0
57.3
56.5
55.8
51.9
51.2
49.4
49.6
47.4
60.0
53.9

8.4
35.9
33.2
35.9
40.3
42.1
43.1
40.1
42.8
40.3
42.6
40.9
41.0
43.8

38.5

11.8

4.5
5.2
9.2
8.8
10.0
10.8
12.0
11.1
11.2
11.8
9.1
13.3
12.8
15.2
37.5

5.7
27.1
28.6
32.9
35.1
37.9
37.5
36.1
36.3
37.2
33.6
32.3
32.9
40.6

DSIFT CNNAAlex

CNNARes50

DSIFT CNNAAlex

CNNAVgg19

Flowers102

Caltech101

Table 6 2D image retrieval accuracy (MAP [%])

52.5

3.6
35.1
28.4
37.7
43.2
50.5
51.0
47.9
51.0
46.4
45.1
47.7
45.6
58.7

CNNAVgg19

46.6

22.2
47.7
41.0
47.0
53.1
55.4
54.5
49.6
50.3
48.3
46.4
47.7
51.1
59.4

CNNARes50

5.1

10.4

17.3

–
13.9
12.6
12.2
14.8
17.2
18.0
16.8
17.2
16.0
15.5
12.6
11.3
18.3

–
3.6
4.2
4.3
4.4
5.4
5.4
5.4
5.3
4.9
4.4
4.4
4.2
5.1

–
11.0
10.8
10.7
11.6
12.4
11.8
10.7
10.8
12.3
11.2
10.6
11.3
10.7

CNNAVgg19

DSIFT CNNAAlex

AwA2

58.3

–
54.3
26.1
27.8
45.2
44.0
39.1
36.5
35.3
37.1
37.3
41.5
40.5
58.4

CNNARes50

35168
Multimedia Tools and Applications (2019) 78:35157–35178

35169Multimedia Tools and Applications (2019) 78:35157–35178
Table 7 Document retrieval accuracy (MAP [%])
algorithms

CD [15]sum
BF [8]LL [66]
DkSA [13]VLAD [25]FVLAD [38]
FV [54]
SV [70]SMPN [37]GP [67]NetVLAD [3]
NeXtVLAD [36]FSA-2048 (proposed)
FSA-256 (proposed)

News20

Amazon4

DBpedia14

LSA

W2V

LSA

W2V

LSA

W2V

8.1
12.6
13.0
12.9
14.3
14.0
8.7
11.4
12.8
21.7
17.1
10.7
12.9
23.5
18.9

9.7
16.4
11.1
11.9
12.6
12.9
8.6
11.1
12.7
16.8
14.3
20.1
20.0
29.0
28.0

42.0
51.6
47.6
48.3
47.0
37.4
35.1
44.8
48.7
61.2
59.8
50.0
52.4
61.4
61.5

44.7
55.9
45.5
46.5
42.9
36.3
36.4
37.3
37.2
55.8
55.8
53.2
55.1
64.8
65.3

–
57.0
43.5
50.0
37.9
28.9
32.2
36.1
35.2
56.7
41.5
58.9
57.2
66.3
64.2

–
60.6
42.3
46.9
34.6
41.3
39.0
39.4
40.9
56.4
35.7
65.8
56.7
69.9
66.3

The results in Table 5, Table 6, and Table 7 demonstrate high accuracy as well as highgenerality of the FSA. While not true in all the cases, FSA-2048 significantly outperforms theother SSC algorithms for diverse combination of benchmark datasets and input features. The
proposed neural network architecture and its training objectives of the FSA appears to learn
better representation of unordered feature sets for SSC. In many cases, more compact FSA-256
still yields accuracies superior or comparable to its competitors. On the other hand, the existing
feature aggregation algorithms show limited generality compared to the FSA. For example,
VLAD and FVLAD perform well in 2D image retrieval, but their accuracies are lower thanthose of the simple sum aggregation in the most cases of text document retrieval.
Comparison of FSA-2048 with NetVLAD and NeXtVLAD verifies effectiveness of theembedding function of FSA. FSA-2048 consistently outperforms NetVLAD and NeXtVLAD
for all the datasets in Table 5, Table 6, and Table 7. The dimensionality of these three features
are identical at 2048. As we will show in Section 4.4, introducing sparseness to feature coding
and feature embedding of FSA has positive impact on retrieval accuracy.

(a) sum,
MAP = 38.0 %(b) DkSA,
MAP = 59.6 %(c) FSA (proposed),
MAP = 69.4 %Fig. 3 Visualization of feature spaces formed by aggregated RoPS features. Each colored dot represents anembedded feature of a 3D shape belonging to one of the ten object categories

35170(a) sum,
MAP = 55.9 %Multimedia Tools and Applications (2019) 78:35157–35178(b) DkSA,
MAP = 42.9 %(c) FSA (proposed),
MAP = 65.3 %Fig. 4 Visualization of feature spaces formed by aggregated W2V features. Each colored dot represents anembedded feature of a text document belonging to one of the four topics

Visualization We visualize feature spaces produced by three feature aggregation algorithms,
i.e., sum of feature vectors, DkSA, and the proposed FSA. We use t-SNE algorithm [45] forvisualization. Figure 3 visualizes feature spaces formed by aggregated RoPS features of test-set
3D shapes contained in the ModelNet10 dataset. Similarly, Fig. 4 compares feature spaces
formed by aggregated W2V features of test-set text documents in the Amazon4 dataset. We
can observe that, especially in the case of 3D shape shown in Fig. 3, the FSA yields anembedded feature space where categories are better separated than feature spaces produced by
the sum aggregation and the DkSA aggregation.
Temporal complexity Table 8 compares temporal complexity of SSC algorithms tocompute a distance between two feature sets. Recall that each feature set contains n
vectors having df dimensions. Chamfer distance (CD) requires high computational costfor distance computation since it does not aggregate features. Sum aggregation is
temporally very efficient since it aggregates features by simple summing. VLAD
requires O(m·n·df) for aggregation, whose computation is dominated by searchingcodewords closest to each of n input features. m is number of the codewords for
VLAD. FSA requires O(dc·(n·df + de)) for aggregation since it needs O(n·df·dc), O(n·dc),
and O(dc·de) for sparse coding, average pooling, and sparse embedding, respectively.
For feature aggregation, FSA has the highest complexity in Table 5 since dc is typicallylarger than m (e.g., dc = 2048 and m = 128). On the other hand, for distance computation, FSA
is more efficient than CD and VLAD. Dimensionality of aggregated feature de of FSA issignificantly smaller than that of VLAD.
Table 9 shows computation time per query on the ModelNet40 test set including
2468 3D models. We used a PC having an Intel Core i7 6700 CPU, a GeForce GTX
1080 Ti GPU, and 64GB DRAM for the timing. In Table 9, the columns named“feature extraction” and “feature aggregation” are processing times for extracting 512
RoPS features from the query 3D model and aggregating these RoPS features of theTable 8 Temporal complexity for set-to-set comparison
SSC algorithms

feature aggregation

distance computation

CD
sum
VLAD
FSA

–
O (n·df)
O (m·n·df)
O (dc·(n·df + de))O (n2·df)
O (df)
O (m·df)
O (de)

35171

Multimedia Tools and Applications (2019) 78:35157–35178
Table 9 Computation time [s] per query
SSC algorithms

feature extraction

feature aggregation

distance computation

total

CD
sum
VLAD
FSA

0.053
0.053
0.053
0.053

–
0.00060.00350.0059

3.216
0.002
0.006
0.001

3.269
0.056
0.063
0.060

We used n = 512, df = 64, m = 128, dc = 2048, de = 64

query, respectively. The column “distance computation” means time for comparing theaggregated feature of the query against aggregated features of the 2467 retrieval targets.
Note that codes of the SSC algorithms used for the timing were accelerated by different
hardware. For example, our implementation of FSA employs the GPU for acceleration
while VLAD utilizes multiple cores of the CPU.
Although direct comparison is difficult for the reason described above, Table 9 indicatesthat the three algorithms that employ feature aggregation based approach, that are, sum,
VLAD, and FSA, are more efficient than the algorithm without aggregation, that is, CD.
Retrieval time of FSA is comparable to that of sum aggregation, which pools features very
quickly. In the feature aggregation based approaches, computation time are dominated by
RoPS feature extraction. In the case of CD, the cost of distance computation is dominant, inwhich element-to-element comparison of all the features between pairs of sets must be
performed.
Since the proposed FSA is based on neural network, its training cost is much higher thanthe existing feature aggregation algorithms. VLAD took about 20 s for codebook learning with
m = 128 while FSA took about 4 h of training for the ModelNet40 dataset. Note, however, thattraining of FSA is preprocessing. It is required only once per dataset before querying thedataset.

Table 10 3D shape retrieval accuracy (MAP [%])
algorithms

supervision bylabels?

ModelNet10

ModelNet40

D2 [51]SPRH [64]
LFD [7]
SPH [52]
PANORAMA [58]SV-DSIFT [11]LL-MO1SIFT [11]DkSA-POD [13]FSA-DSIFT (proposed)
FSA-MO1SIFT (proposed)
FSA-POD (proposed)
3D ShapeNets [69]
MVCNN [61]
DeepPano [59]DLAN [14]PANORAMA-ENN [58]

No
No
No
No
No
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes

28.6
41.2
49.8
44.1
60.3
60.2
63.1
61.2
73.3
73.1
71.6
68.3
–
84.2
90.6
93.3

19.1
32.9
40.9
33.3
46.1
49.5
48.0
48.4
58.8
60.3
56.6
49.2
79.5
76.8
85.0
86.3

35172Multimedia Tools and Applications (2019) 78:35157–35178

Table 11 Comparison of set reconstruction loss (MAP [%] for ModelNet10)
set-to-set distance

Earth Mover’s distance
Hausdorff distancechamfer distanceelement-to-element distance
Euclidean distance

Manhattan distance

Cosine distance

61.1
52.4
60.2

61.3
52.4
60.0

63.8
58.5
66.6

4.3 Comparison with existing 3D shape retrieval algorithms
The experiments under the 3D shape retrieval scenario using ModelNet10 and ModelNet40
follow the evaluation protocol provided by Wu et al. [69]. Therefore, we can compare accuracyof our 3D shape matching algorithm using the FSA against accuracies of existing algorithms
for 3D shape retrieval.
Table 10 compares MAP scores for the ModelNet10 and the ModelNet40 datasets. As
competitors, we use eight algorithms that compare unlabeled 3D shapes without any supervision. Among the eight algorithms, SV-DSIFT [11], LL-MO1SIFT [11], and DkSA-POD [13]are state-of-the-art algorithms for comparing unlabeled 3D shapes. They extract a set of
handcrafted feature vectors called DSIFT, MO1SIFT, or POD, respectively. The set of features
is aggregated per 3D shape by using feature aggregation algorithms, i.e., SV, LL, or DkSA. We
replace these feature aggregation algorithms used by these algorithms with the proposed FSA.
In addition, we also list, in Table 10, retrieval accuracies of five deep neural network-based 3D
shape matching algorithms that rely on supervised training using object category labels.
In Table 10, those algorithms using FSA outperform the other eight unsupervised algorithms. Feature aggregation using the FSA significantly boosts MAP scores, by about 10% for
most cases, compared to such state of the art feature aggregation algorithms as SV, LL, and
DkSA. As expected, deep neural networks trained with supervision yield higher retrieval
accuracies than the algorithms without supervision. Interestingly, however, the FSA-basedalgorithms outperform the 3D ShapeNets. This result would partially supports our argument,that is, the FSA learns accurate representation of feature sets without relying on labels.

Fig. 5 Retrieval accuracy plottedagainst blending parameter α ofthe loss function (ModelNet10)

RoPS

POD

OP

75

MAP [%]

70
65
60
55
50
0

0.01 0.05 0.1 0.5 1
5 10
for objective function L

50

35173

Multimedia Tools and Applications (2019) 78:35157–35178
Fig. 6 Retrieval accuracy plottedagainst aggregated feature
dimension (ModelNet10)

MAP [%]

LL
DkSA

VLAD
FSA

FV

80
70
60
50
40
30
20
10
0
2

8
32
128 512 2,048 8,192
aggregated feature dimensions

4.4 In-depth evaluation of feature set aggregator
This section explores effect design parameter has on the proposed algorithm. We use theModelNet10 dataset along with the RoPS feature for the experiments described below.

Set reconstruction loss We compare nine set reconstruction losses. We employ three set-toset distances, i.e., Earth Mover’s distance, Hausdorff distance, and chamfer distance for LR.
Besides, for each set-to-set distance, we vary distance metric among set elements. That is, we
use Euclidean, Manhattan, and Cosine distances as element-to-element distances. We disable
set embedding loss LE by fixing the hyper-parameter α to 0 in the experiment. The results
shown in Table 11 indicates that choice of set reconstruction loss has a significant impact onretrieval accuracy. Chamfer distance combined with Cosine distance, in other words, angularchamfer distance, performs the best among the nine loss functions. Angle would be an
appropriate metric to compare the whitened input features having sphered distribution.
Set embedding loss We investigate effectiveness of the set embedding loss LE by varying thehyper-parameter α from 0 to 50. Figure 5 plots MAP scores against the different values of α.
In the figure, α = 0 means the FSA is trained solely by set reconstruction loss LR. Mixing LR
and LE by using values of α > 0, for example, α = 0.5, improves retrieval accuracy for all thethree 3D shape features we experimented with. The proposed triplet generation algorithm
using family of sets has a positive impact on learning better embedding feature space for SSC.
In Fig. 5, peaks of retrieval accuracy appear at different values of α depending on the feature
Fig. 7 Retrieval accuracy plottedagainst sparseness kc of the sparse
coding block (ModelNet10)

RoPS

POD

OP

75

MAP [%]

70
65
60
55
50
0 10 20 30 40 50 60 70 80 90 100
kc for sparse coding block35174Multimedia Tools and Applications (2019) 78:35157–35178

Fig. 8 Retrieval accuracy plottedagainst sparseness ke of the sparseembedding block (ModelNet10)

RoPS

OP

POD

75

MAP [%]

70
65
60
55
50
0

50

100

150

200

ke for sparse embedding block(and also probably on the dataset). This implies that manually searching an appropriate valueof α is required to obtain the best result when using the proposed FSA algorithm. Note,however, that the FSA is not very sensitive to the value of α. The FSA consistently produced
good MAP scores (over 60%) for the wide range of α we experimented with.

Embedded feature dimension Figure 6 plots MAP scores against the number of dimensionsof aggregated features. For the FSA, aggregated feature dimension corresponds to dimensionality de of the embedded feature space. Accuracies of the conventional feature aggregation
algorithms, i.e., LL and DkSA, drop significantly if dimensionality of features becomes low. In
contrast, the FSA keeps high MAP value (nearly 60%) even at dimensionality as small as 8.
Sparseness in encoder Figure 7 and Fig. 8 show an impact of sparsity in the encoder part of
the FSA on retrieval accuracy. Figure 7 plots accuracies against the number of non-zero
activations kc of the sparse coding block. We fixed the other hyper-parameters dc, de, and ke to2048, 256, and 25, respectively. For all the features in Fig. 7, accuracies have peaks at aroundkc = 5. Increasing kc, i.e., having the coding block produce denser activations, lowers retrieval
accuracy. Sparseness in the coding block enhances saliency of each encoded input features.
Meanwhile, Fig. 8 plots accuracy against ke, i.e., the number of non-zero activations in thesparse embedding block, while keeping the number of non-zero activations kc at 5. The plots ofaccuracy have weak peaks around ke = 25 or ke = 50. Although not so evident as in the case ofsparse coding block, sparseness of the embedding block also helps the FSA learn better feature
representation of sets.
Ablation study Table 12 demonstrates importance of whitening input features, sparseness in
encoder, and objective function for training. In the table, “No” for the “whitening input
features” column indicates that input sets of RoPS features are fed into the FSA as-is without
Table 12 Ablation study of FSA (MAP [%] for ModelNet10)whitening input features

sparseness in encoder

objective function

MAP [%]

Yes
Yes
Yes
Yes
No

Yes
Yes
Yes
No
No

LR + αLELE only
LR only
LR only
LR only

69.4
66.7
66.9
59.7
55.4

Multimedia Tools and Applications (2019) 78:35157–35178

35175

whitening. “No” for the “sparseness in encoder” column means sparsification using top-k
activations is omitted at the sparse encoding layer and the sparse embedding layer of the FSA.
The results validate the design of FSA algorithm that aims at accurate SSC.

5 Conclusion and future work
In indexing, clustering, or retrieval of multimedia data, each datum is often described by anunordered set of high-dimensional feature vectors. To perform these operations effectively andefficiently, Set-to-Set Comparison (SSC) among such unordered sets is very important.
Aiming at accurate and efficient SSC, this paper proposed an unsupervised representation
learning algorithm called Feature Set Aggregator (FSA). To learn effective embedding of
unlabeled sets in an unsupervised manner, we designed two training objectives for FSA, thatare, set reconstruction using angular chamfer distance and set embedding using triplets
generated on family of sets. Extensive evaluation under three multimedia information retrieval
scenarios using 3D shapes, 2D images, and text documents demonstrated efficacy andgenerality of the FSA.
As future work, we will seek better neural network architecture and better training objective
than the current FSA. For example, the current FSA with only a few layers is much shallower
than recent neural networks used, for example, for image recognition. We need to evaluate
influence of increasing the number of layers of FSA on SSC accuracy. Also, we plan toevaluate the FSA under scenarios other than multimedia information retrieval, e.g., clustering,
to further verify the quality and generality of feature representation of sets learned by the FSA.
Possible evaluation scenarios include more specific tasks such as fingerprint recognition or
palmprint recognition (e.g., [43]). Recognizing fingerprints or palmprints often requires set-toset comparison where each set consists of keypoints extracted from a skin image. Furthermore,
we envision extending the FSA to the problem of cross-modal set-to-set comparison. Learningcommon representation of sets describing different types of multimedia data would be valuable
for cross-modal information retrieval (e.g., [72]) or more challenging cross-modal multimedia
understanding problem [44].

References
1. Abadi M et al (2016) TensorFlow: a system for large-scale machine learning. Proc. OSDI 2016:265–283
2. Achlioptas P, Diamanti O, Mitliagkas I, Guibas L (2017) Learning Representations and Generative Models
for 3D Point Clouds, arXiv preprint, arXiv:1707.02392
3. Arandjelović R, Gronat P, Torii A, Pajdla T, Sivic J (2018) NetVLAD: CNN architecture for weakly
supervised place recognition. TPAMI 40(6):1437–1451
4. Blitzer J, Dredze M, Pereira F (2007) Biographies, Bollywood, boom-boxes and blenders: domain
adaptation for sentiment classification. Proc. ACL 2007:440–447
5. Chang AX et al. (2015) ShapeNet: An Information-Rich 3D Model Repository, arXiv:1512.03012
6. Charles RQ, Su H, Kaichun M, Guibas LJ (2017) PointNet: deep learning on point Sets for 3D classificationand segmentation. Proc. CVPR 2017:77–85
7. Chen DY, Tian XP, Te Shen Y, Ouhyoung M (2003) On visual similarity based 3D model retrieval. Comput
Graph Forum 22(3):223–232
8. Csurka G, Dance CR, Fan L, Willamowski J, Bray C (2004) Visual categorization with bags of Keypoints.
Proc. ECCV 2004 workshop on statistical learning in computer vision: 59–74

35176

Multimedia Tools and Applications (2019) 78:35157–35178

9. Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database.
Proc CVPR 2009:248–255
10. Fei-Fei L, Fergus R, Perona P (2004) Learning generative visual models from few training examples: anincremental Bayesian approach tested on 101 object categories. Proc CVPR workshop 2004:59–70
11. Furuya T, Ohbuchi R (2014) Fusing multiple features for shape-based 3D model retrieval, Proc British
Machine Vision Conference (BMVC)
12. Furuya T, Ohbuchi R (2015, 2015) Diffusion-on-manifold aggregation of local features for shape-based 3Dmodel retrieval. Proc. ICMR:171–178
13. Furuya T, Ohbuchi R (2016) Accurate aggregation of local features by using K-sparse autoencoder for 3Dmodel retrieval. Proc. ICMR 2016:293–297
14. Furuya T, Ohbuchi R (2016) Deep aggregation of local 3D geometric features for 3D model retrieval. Proc
BMVC 2016:121.1–121.12
15. Gavrila DM, Philomin V (1999) Real-time object detection for “smart” vehicles. Proc. ICCV 1999:87–93
16. Geoffrey E, Hinton RRS (2006) Reducing the dimensionality of data with neural networks. Science313(5786):504–507
17. Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. Proc AISTATS 2011:315–323
18. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2016)
Generative adversarial nets. Proc NIPS 2016:2672–2680
19. Günther F English LSA space, https://sites.google.com/site/fritzgntr/home
20. Guo Y, Sohel F, Bennamoun M, Lu M, Wan J (2013) Rotational projection statistics for 3D local surface
description and object recognition. IJCV 105(1):63–86
21. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proc CVPR 2016:770–
778
22. Henrikson J (1999) Completeness and total boundedness of the Hausdorff metric, MIT Undergraduate
Journal of Mathematics: 69–80
23. Hoffer E, Ailon N (2015) Deep metric learning using triplet network. Proc. ICLR 2015 workshop
24. Hyvärinen A, Hurri J, Hoyer PO (2009) Natural image statistics: a probabilistic approach to earlycomputational vision. Springer, Verlag
25. Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image
representation. Proc CVPR 2010:3304–3311
26. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. Proc. ICLR 2015
27. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neuralnetworks. Proc. NIPS 2012: 1097–1105.
28. Landauer TK, Foltz PW, Laham D (1998) An introduction to latent semantic analysis. Discourse Process25(2–3):259–284
29. Lang K (1995) Newsweeder: Learning to filter netnews. Proc. ICML 1995:331–339
30. Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, Ming-Hsuan Yang, Unsupervised representation learningby sorting sequences, Proc. ICCV 2017, pp. 667–676, 2017.
31. Lehmann J et al (2015) DBpedia–a large-scale, multilingual knowledge base extracted from Wikipedia.
Semantic Web 6(2):167–195
32. Leng L, Zhang J (2013) Palmhash code vs. palmphasor code. Neurocomputing 108:1–12
33. Leng L, Zhang J, Xu J, Khan MK, Alghathbar K (2010) Dynamic weighted discrimination power analysis
in DCT domain for face and palmprint recognition. Proc ICTC 2010:467–471
34. Leng L, Li M, Leng L, Teoh ABJ (2013) Conjugate 2DPalmHash code for secure palm-print-vein
verification. Proc CISP 3:1705–1710
35. Leng L, Li M, Kim C, Bi X (2017) Dual-source discrimination power analysis for multi-instance contactless
palmprint recognition. MTAP 76(1):333–354
36. Lin R, Xiao J, Fan J (2018) NeXtVLAD: An efficient neural network to aggregate frame-level features forlarge-scale video classification, Proc. ECCV 2018 workshops: 206–218
37. Lin T-Y, Maji S, Koniusz P (2018) Second-order democratic aggregation. Proc. ECCV 2018:639–656
38. Liu Z, Wang S, Tian Q (2016) Fine-residual VLAD for image retrieval. Neurocomputing 173(3):1183–1191
39. Liu Y, Yan J, Ouyang W (2017) Quality aware network for set to set recognition. Proc. CVPR 2017:4694–
4703
40. Lowe DG (2004) Distinctive image features from scale-invariant Keypoints. IJCV 60(2):91–110
41. Lu L, Zhang J, Gao C (2011) Muhammad Khurram khan, Khaled Alghathbar, two-directional twodimensional random projection and its variations for face and palmprint recognition. Proc ICCSA 2011:
458–458
42. Lu L, Zhang J, Gao C (2011) Muhammad Khurram khan, ping Bai, two dimensional PalmPhasor enhancedby multi-orientation score level fusion. Proc STA 2011:122–129

Multimedia Tools and Applications (2019) 78:35157–35178

35177

43. Lu L, Beng A, Teoh J (2015) Alignment-free row-co-occurrence cancelable palmprint. Fuzzy Vault Pattern
Recogn 48(7):2290–2303
44. Lu H, Li Y, Chen M, Kim H, Serikawa S (2018) Brain intelligence: go beyond artificial intelligence. Mobile
Netw Appl 23(2):368–375
45. van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605
46. Makhzani A, Frey B (2014) k-sparse autoencoders, Proc. ICLR 2014
47. Mikolov T, Sutskever I, Chen K, Corrado G, Dean J (2013) Distributed representations of words andphrases and their compositionality. Proc. NIPS 2013:3111–3119
48. Nilsback M-E, Zisserman A (2008) Automated flower classification over a large number of classes. Proc
ICVGIP 2008:722–729
49. Noroozi M, Favaro P (2016) Unsupervised learning of visual representations by solving jigsaw puzzles.
Proc. ECCV 2016:69–84
50. Ohbuchi R, Minamitani T, Takei T (2005) Shape-similarity search of 3D models by using enhanced shapefunctions. IJCAT 23(2):70–85
51. Osada R, Funkhouser T, Chazelle B, Dobkin D (2002) Shape distributions. ACM Trans Graph (TOG) 21(4):807–832
52. Papadakis P, Pratikakis I, Perantonis S, Theoharis T (2007) Efficient 3D shape matching and retrieval using
a concrete radialized spherical projection representation. Pattern Recogn 40(9):2437–2452
53. Pathak D, Krähenbühl P, Donahue J, Darrell T, Efros AA (2016) Context encoders: feature learning by
Inpainting. Proc CVPR 2016:2536–2544
54. Perronnin F, Sánchez J, Mensink T (2010) Improving the fisher kernel for large-scale image classification.
Proc. ECCV 2010, part IV: 143–156
55. Qi CR, Yi L, Su H, Guibas LJ (2017) PointNet++: deep hierarchical feature learning on point Sets in a
metric space, Proc. NIPS 2017: 5105–5114
56. Rubner Y, Tomasi C, Guibas LJ (1998) A metric for distributions with applications to image databases. Proc
ICCV 1998:59–66
57. Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition andclustering. Proc CVPR 2015:815–823
58. Sfikas K (2018) Ioannis Pratikakis, Theoharis Theoharisa, ensemble of PANORAMA-based convolutional
neural networks for 3D model classification and retrieval. Comput Graph 71:208–218
59. Shi B, Bai S, Zhou Z, Bai X (2015) DeepPano: deep panoramic representation for 3-D shape recognition.
Signal Process Lett 22(12):2339–2343
60. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition.
Proc. ICLR 2015:1–14
61. Su H, Maji S, Kalogerakis E, Learned-Miller E (2015) Multi-view convolutional neural networks for 3D
shape recognition. Proc. ICCV
62. Thorsten Joachims A (1997) Probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. Proc ICML 1997:143–151
63. Vincent P, Larochelle H, Lajoie I, Bengio Y, Manzagol P-A (2010) Stacked Denoising autoencoders:learning useful representations in a deep network with a local Denoising criterion. J Mach Learn Res 11:3371–3408
64. Wahl E, Hillenbrand U, Hirzinger G (2003) Surflet-pair-relation histograms: a statistical 3D-shape representation for rapid classification. Proc Fourth Int Conf 3D Digit Imag Model (3DIM) 2003:474–481
65. Wang X, Gupta A (2015) Unsupervised learning of visual representations using videos. Proc. ICCV 2015:
2794–2802
66. Wang J, Yang J, Yu K, Lv F, Huang T, Gong Y (2010) Locality-constrained linear coding for image
classification. Proc. CVPR 2010:3360–3367
67. Wei X, Zhang Y, Gong Y, Zhang J, Zheng N (2018) Grassmann pooling as compact homogeneous bilinear
pooling for fine-grained visual classification. Proc ECCV 2018:365–380
68. Word2vec. https://code.google.com/archive/p/word2vec
69. Wu Z, Song S, Khosla A, Yu F, Zhang L, Tang X, Xiao J (2015) 3D ShapeNets: a deep representation forvolumetric shapes. Proc. CVPR 2015:1912–1920
70. Xi Z, Kai Y, Zhang T, Huang TS (2010) Image classification using super-vector coding of local image
descriptors. Proc ECCV 2010:141–154
71. Xian Y, Lampert CH, Schiele B, Akata Z (2018) Zero-shot learning - a comprehensive evaluation of thegood, the bad and the ugly. TPAMI 40(8)72. Xu X, Song J, Lu H, Yang Y, Shen F, Huang Z (2018) Modal-adversarial semantic learning network forextendable cross-modal retrieval. Proc ICMR 2018:46–54
73. Yang Y, Feng C, Shen Y, Tian D (2017) FoldingNet: Interpretable Unsupervised Learning on 3D Point
Clouds, arXiv preprint, arXiv:1712.07262

35178Multimedia Tools and Applications (2019) 78:35157–35178

74. Zaheer M, Kottur S, Ravanbakhsh S, Poczos B, Salakhutdinov RR, Smola AJ (2017) Deep sets, Proc. NIPS
2017: 3394–3404.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps andinstitutional affiliations.

Takahiko Furuya received his B. Eng., M. Eng., and Ph. D. in engineering degrees from University of
Yamanashi in 2008, 2010, and 2015, respectively. He worked for Nisca corporation from 2010 to 2013. He
was granted JSPS Research Fellowships for Young Scientists (DC2) in 2015. In 2015, he joined, as an assistantprofessor, the Department of Computer Science, University of Yamanashi. His research interests include multimedia information retrieval and machine learning.

Ryutarou Ohbuchi received the B. Eng from the Department of Electrical and Electronic Engineering, Sophia
University, Tokyo, Japan. He received M. Eng. from the Department of Computer Science, University of Electrocommunications, Tokyo, Japan. He joined IBM Japan Science Institute in 1984. He received his Ph. D in
Computer Science from the University of North Carolina at Chapel Hill in 1994. From 1994 to 1999, he worked
as a researcher at IBM Research, Tokyo. In 1999, he joined, as an associate professor, the Department ofComputer Science, University of Yamanashi, Yamanashi. Japan. He is a full professor since 2007. In the past, he
has worked on graphics display hardware design, augmented reality display for medical applications,photorealistic rendering, digital watermarking of 3D shape models, and others. His recent research interests
include multimedia data analysis, retrieval, and mining, especially of 3D shape models. Prof. Ohbuchi is amember of ACM, IEEE Computer Society, IPSJ, and IIEEJ.