1

Pattern Recognition Letters

Joint Object Recognition and Pose Estimation using
Multiple-anchor Triplet Learning of Canonical Plane
Shunsuke Yonedaa,, Kouki Uenoa , Go Irieb , Masashi Nishiyamaa , Yoshio Iwaiaa Graduate School of Engineering, Tottori University, 101 Minami 4-chome, Koyama-cho, Tottori, 680-8550, Japanb NTT Corporation, 1 Morinosato-wakamiya 3-chome, Atsugi, 243-0198, Japan

ABSTRACT
Accurate object recognition and pose estimation models are essential for practical applications ofrobot arms, such as picking products on a shelf. Training such a model often requires a large-scale
dataset with qualified labels for both object classes and pose parameters, and collecting accurate poselabels is particularly costly. A recent paper [28] proposed a triplet learning framework for joint object
recognition and pose estimation without explicit pose labels by learning a spatial transformer network
to estimate the pose difference of an input image from an anchor image depicting the same object
in a reference pose. However, our analysis suggests that the pose estimation accuracy is severely
degraded for input images with large pose differences. To address this problem, we propose a newlearning approach called multiple-anchor triplet learning. The basic idea is to give dense reference
poses by preparing multiple anchors so that there is at least one anchor image having a small pose
difference to the input image. Our multiple-anchor triplet learning is an extension of the standard
single-anchor triplet learning to the multiple-anchor case. Inspired by the idea of multiple instancelearning, we introduce a selection layer that automatically chooses the best anchor for each input
image and allows the network to be trained end-to-end to minimize triplet-based losses. Experimentswith three benchmark datasets in product picking scenarios demonstrate that our method significantly
outperforms existing methods in both object recognition and pose estimation accuracy.

1. Introduction
Real-world applications of robot arms, e.g., picking a producton a shelf, often require accurate object recognition and pose
estimation. Many vision-based techniques have been developed
to equip these functions targeting to realistic use cases such as
warehouse automation [11, 17].
Following the success of deep learning in various image processing tasks, it has also become mainstream in both object
recognition and pose estimation. Given the correlation between
the two tasks, recent methods have been designed to perform
both tasks simultaneously [27, 4]. These existing methods perform well when a sufficient amount of training data associatedwith accurate object class and pose parameter labels are available. However, collecting such a large-scale dataset with qualified labels is generally costly. It is even more so for pose pa-

e-mail: nishiyama@tottori-u.ac.jp (Masashi Nishiyama)

rameters, due to the nature of the need for fine numerical labels.
In this work, we aim to develop an efficient approach to learning a network for joint object recognition and pose estimation
using a training dataset with no explicit pose parameter labels
but with only object class labels.
A recent method [28] proposed a joint object recognitionand pose estimation model that can be learned without explicit
pose parameter labels, by combining the ideas of triplet learning [29, 24] and spatial transformer networks (STNs) [15]. The
triplet learning and STN are typically used to improve object
recognition accuracy. Triplet learning minimizes the feature
distance between an anchor image (called anchor) and a positive image (called positive) of the same class as the anchor,
and maximizes that between the anchor and a negative image(called negative) of a different class. Meanwhile, the originaluse of STN [15] is to apply a geometric transformation (e.g.,
2D affine transformation) to the input image purely to improverecognition accuracy. Instead, [28] proposes integrating both2
into a unified framework to estimate the relative pose difference
from the anchor to the positive. Considering that many products
handled by the robot arm are locally planar (e.g., boxed foods),[28] trains the STN to match the appearance of a “dominant
plane” of the object (called canonical plane) of the anchor andthat of the positive within the triplet learning framework.
This elegant idea allows for pose estimation of nearly planar objects without labeling the pose parameters, and can also
improve the accuracy of object recognition. However, our analysis (given later in Sec. 3) found that the training of the STN
could fall to a poor local minimum depending on the choice of
the anchor, resulting in severe degradation of pose estimation
accuracy, especially for images with large pose differences.
In this paper, we propose a multiple-anchor triplet learning
method to address this problem. Unlike the standard triple
learning framework that uses only a single anchor, our approachallows us to have more than one anchor per triplet, giving
denser reference poses to prevent pose estimation from being
trapped in poor local minima. Moreover, we introduce a selection layer that automatically selects the best anchor from among
candidate anchors to enable end-to-end learning with multipleanchor triplet data. Experimental results with three publicly
available datasets for object recognition in product picking scenarios show that our method reduces the pose estimation error
while improving object recognition accuracy.
2. Related work
2.1. Object Recognition
Existing methods for object recognition can be categorized
into a model-based approach and a data-driven approach. The
model-based approach uses a classification model (classifier)to categorize an input image into a set of known object classes.
The classifier is trained to minimize some loss function, e.g.,
softmax cross-entropy between prediction and the ground truthclass label. A convolutional neural network such as ResNet [14]is typically used as the classifier. The model-based approachprovides highly accurate classification as long as the possible
object classes are prespecified. Utilizing this advantage of themodel-based approach has been the mainstream for more advanced tasks such as object detection [19, 21].
The data-driven approach aims at learning distance metrics ina feature space and solving the classification task in the nearest
neighbor manner. A feature extraction network is trained sothat it minimizes the intra-class distances and maximizes theinter-class distances. Modern methods use convolutional neuralnetworks for feature extraction, such as the siamese network [5,1] and triplet network [29, 24]. More sophisticated versions
have also been proposed in recent years [10, 31].
In this work, we consider the data-driven approach for object recognition. In a product sorting task in a warehouse, novel
classes of objects can be frequently added to the gallery. Insuch a case, the model-based approach requires frequent retraining of the model, which is often time consuming, whereas
the data-driven approach circumvents this. Our network is designed to extract a deep feature for an input image and is trainedin the triplet learning framework. Moreover, the network is also

trained so that it can estimate the pose of the object in the image
by leveraging the output of the intermediate STN module.
2.2. Pose Estimation
Existing methods for pose estimation can be categorized intoa keypoint-based approach and a learning-based approach. The
keypoint-based approach is designed to extract discriminative
three-dimensional points and match their local features in apoint cloud space [8, 23]. The existing methods perform wellwhen a sufficient number of points representing the shape of theobject have been acquired. However, it is known that the performance decreases for objects with less discriminative shapes.
The learning-based approach is typically designed to matchthe appearance of the object in an RGB image space, which
has attracted much attention recently [27, 4, 6]. Accurate pose
estimation is possible when a large number of training samples with pose parameter labels are available. However, assigning accurate full three-dimensional pose parameters to each object is costly [4, 6]. Assigning ground truth three-dimensionalbounding boxes to each object is still hard work [27].
Some recent studies have explored weakly supervised approaches to reduce the labeling cost, which is the most relevant type of methods to ours. The majority of the methods perform joint object recognition and pose estimation [16, 26, 18].
Kanezaki et al. [16] assumed that the object pose is capturedby multiple views of objects and estimate the object category
and pose using multiple images in the test stage. Sundermeyer
et al. [26] and Li et al. [18] used three-dimensional models ofobjects for training to learn their shapes.
Unlike the existing methods, we instead focus on accomplishing this task without making such assumptions. We aim
to estimate the parameters of linear transformations of posechanges. The network can be trained using only RGB images
without requiring explicit pose labels. Furthermore, our model
can estimate the object class and pose together from a single
RGB image, which can be applicable to diverse robot arm systems in practical scenarios. As mentioned earlier, our method isbased on [28]. However, [28] has a drawback that the pose estimation accuracy is severely reduced for images with large posevariations. In this paper, we address this problem by proposing
a novel multiple-anchor triplet learning method.
3. Preliminary and Analysis
Before explaining the proposed method, we first introduce [28] which is the background framework of the proposed
method. We also show our analysis to reveal that the pose estimation accuracy by [28] is degraded for an input image with alarge pose difference from the anchor.
3.1. Overview of [28]The overview of the network proposed in [28] is shown in
Fig. 1(a). It mainly consists of two major trainable parts: thefeature extraction network c( ) and the STN module s( ). As
in the triplet learning framework [29, 24], it has three streams
for the anchor, positive, and negative, and the parameters of c( )and s( ) are shared over all the three streams. Note that the STN

3
STN modules(𝑰𝑝 )

Feature extraction

𝐇
Positive

Transformed positivec(s(𝑰𝑝 ))×

𝑰𝑝

FeatureAppearance loss𝐿𝑎c(𝑰𝑎∗ )

Anchor𝐿𝑡

𝑰𝑎

Features(𝑰𝑛 )

𝐇
Negative

Transformed negative

Triplet lossc(s(𝑰𝑛 ))×𝑰𝑛Feature(a) Architecture of [28]STN modules(𝑰𝑝 )

Feature extraction

𝐇
Positive

𝑰𝑝

Transformed positivec(s(𝑰𝑝 ))×
FeatureMultiple-anchor version of
appearance loss
Multiple anchors

,…,
𝑖=1

𝑰𝑎𝑖

𝑖=𝑁

𝐿𝒜

c(𝑰𝑎∗ )

Selection layer

𝐿𝑡

𝑰𝑎∗ = argmin 𝐿𝑎 𝑰𝑎𝑖 , 𝑰𝑝
𝑰𝑎
𝑖 ∈𝒜

𝑰𝑎∗

FeatureTriplet losss(𝑰𝑛 )

𝐇
Negative

𝑰𝑛

Transformed negativec(s(𝑰𝑛 ))×
Feature(b) Architecture of ours
Fig. 1: Overview of the network architectures of (a) [28] and (b) ours. s( ) synthesizes an image by removing pose changes using the STN in terms of the homographytransformation. c( ) extracts a feature using a convolutional neural network. Unlike (a) [28], (b) ours uses the selection layer to handle multiple anchors.

module s( ) is only set in the positive and negative streams, rightafter the input layer.
Suppose a collection of training triplets are available, where
each consists of an anchor Ia , positive I p , and negative In . I p
contains the same object as the anchor Ia (but with a differentpose), while In does a different object. Every image is assumed
to be pre-processed to contain only a single foreground object,
due to the fact that background models are often available inwarehouse automation as described in [32], and thus can readily
be removed and segmented by a standard foreground segmentation method. Given such a dataset, c( ) and s( ) are trainedby minimizing the distance between the features of the anchorc(Ia ) and the “transformed” positive c(s(I p )) (by the STN module), while maximizing that between the features of the anchorc(Ia ) and the transformed negative c(s(In )). The resulting features are expected to be consistent with the object classes.
At the same time, the STN module s( ) is trained to estimate
the relative pose of the positive I p from the anchor Ia . Typicalobjects considered in warehouse automation scenarios can bewell represented by a small number of “canonical planes,” andan image of such an object is likely to capture at least one ofthese planes. Based on this idea, [28] first converts the full 3D
pose estimation problem into a problem of 2D transformation

estimation of the canonical plane, which can be directly handled by the STN module. Suppose an anchor depicts an object
in a reference pose with respect to a canonical plane, and thesame anchor is used for all the triplets for the same plane of thesame object. In this case, the pose of the object in the positive
can be predicted by estimating a 2D homography transformation matrix Ĥ from the anchor to the positive. The following
appearance loss La is introduced to enforce the positive to bealways transformed to the anchor by the STN module s( ).
La (Ia , I p ) = ||Ia − s(I p )||1 .

(1)

The loss returns a small value when Ia and I p are similar after
the transformation s( ) is applied. The minimization can be performed together with the feature learning in the triplet learning
framework, without explicit pose parameter labels.
3.2. Analysis
We found that the pose estimation accuracy by thismethod [28] decreases rapidly when the pose difference between the positive I p and the anchor Ia is large. This is becausechanges in appearance due to rotation often involve a certain
level of ambiguity, leading to poor local minima for training

4
0.07

4.3. Selection Layer

𝐿𝑎

Our loss function Eq. 2 can be decomposed into two processes as:I∗a = argmin La (Iai , I p ).
(3)

0

-180

-135

-90

-45

0

45

90

135

180

Iia ∈A

In-plane rotation of input image (deg)Fig. 2: Local minima of La with respect to in-plane rotations. Other than the
global optimum at 0◦ , La has multiple poor local minima at ±180◦ .

the STN module s( ) using the appearance loss La . To be specific, we show in Fig. 2 the value of La when various in-planerotations are applied to a book image. As can be seen in the example, when the image is rotated by 180◦ , the resulting image
looks highly similar to its original (i.e., 0◦ ). That suggests that,
if the in-plane rotation angle of a positive I p from the anchor Ia
is outside of [−90, 90] degrees, the appearance loss minimization is likely to be trapped at the points displaced by ±180◦ .
Consequently, the pose estimation result may often deteriorate
for an input image with a large pose difference from the anchor
Ia . To tackle this issue, we propose a multiple-anchor extensionof the standard single-anchor triplet learning.

The idea behind our approach is to create a situation in which
there is at least one anchor having a close angle for any positiveby preparing multiple anchors with different rotation angles.
4.1. Multiple-anchor Triplet for Pose Estimation
Our triplet tuple consists of a set of N anchors denoted byN
A = {Iai }i=1
, one positive I p , and one negative In . The set of theanchors A is prepared to represent the same canonical plane ofan object from different rotation angles. While the simplest option is to use N images taken from N random viewpoints as A,
we adopted a more systematic and effective method. Specifically, we set up the camera so that its optical axis is perpendicular to the floor and place the object parallel to the floor surface.
The optical axis of the camera is set so that it passes through theobject’s center of gravity. We then acquire N images for eachcanonical plane at a regular interval. As discussed in Sec. 5.7,
this method gives better object recognition and pose estimation
performance than the random acquisition case.
4.2. Appearance Loss for Multiple-anchor Case
Given the “bag of anchors” A, inspired by the idea of multiple instance learning [2], we define our multiple-anchor versionof the appearance loss LA as:
Ii ∈A

Ii ∈A

(4)

This means that once the best anchor I∗a in terms of the original(single-anchor version of) appearance loss La is chosen from
A according to Eq. 3, the multiple-anchor version of the appearance loss LA (A, I p ) turns into the original appearance loss
La (Ia∗ , I p ).
Based on this fact, our multiple-anchor triplet learning can be
implemented by just introducing a selection layer into the original triplet network. Our network architecture overall can beillustrated as in Fig. 1(b). It is the same as the original network
Fig. 1(a) except for the selection layer that determines the bestanchor according to Eq. 3. For training of the network, the gradient is computed by using only the Ia∗ chosen by the selection
layer, and the STN module can also be updated based on thehomography transformation matrix Ĥ from the selected anchorIa∗ to the given positive I p .
4.4. Training

4. Multiple-anchor Triplet Learning

LA (A, I p ) = min
La (Iai , I p ) = min||Iai − s(I p )||1 .
aa

LA (A, I p ) = La (Ia∗ , I p ).

(2)

By this, only the anchor image closest to the positive contributes to the final loss value, which mitigates the ambiguity
over the angles.

Our network is trained to minimize the total loss function thatcan be expressed as
L(IA , I p , In ) = λLA (A, I p ) + Lt (Ia∗ , I p , In )(5)

= λLa (Ia∗ , I p ) + Lt (Ia∗ , I p , In ),(6)

where Lt is the standard triplet loss term which is for learningfeatures for object recognition, and λ is a hyperparameter. The
triplet loss Lt is defined as:
Lt (Ia∗ , I p , In ) = max(||c(Ia∗ ) − c(s(I p ))||22 −||c(Ia∗ ) − c(s(In ))||22 + m, 0),(7)where m is the margin. Note that the triplet loss term is computed only for the best anchor Ia∗ selected by the selection layer,
hence the standard form [29, 24] can be directly used. Lt returns
a smaller value when the features of the selected anchor Ia∗ andI p are closer as well as those of Ia∗ and In are farther.
4.5. Inference
Following [28], our method performs object recognition ina nearest neighbor classification manner. For each object, westore the set of N anchor images A of its canonical planes andtheir features in a database as the reference set. Given an input(query) image Iq , our method first extracts its feature using thetrained network, i.e., the anchor branch in Fig. 1(b), as c(s(Iq )).
Object recognition is done by finding the closest anchor image Ia∗ from the database in terms of the feature distance, i.e.,||c(Ia∗ ) − c(s(Iq ))||, and assigning the object class of the selectedanchor image Ia∗ . The homography transformation matrix Ĥ
output by the STN module gives the estimated pose of the input
Iq from the chosen anchor Ia∗ .

54.6. Extension to Other Types of Losses
Our multiple-anchor idea can be extended to several othertypes of loss functions. Here, we consider extending the noise
contrastive estimation (NCE) loss to its multiple-anchor version. Many studies [9, 25, 30] have reported the effectivenessof NCE loss in various tasks, e.g., self-supervised learning andmetric learning. Suppose given {I p , Ia∗ , In1 , . . . , InM } with I p as
the positive, Ia∗ as the anchor chosen by the selection layer frommultiple anchors, and Inj ( j = 1, . . . , M) as the negatives, thetotal loss function is expressed as:(a) ARC dataset(b) YCB datasetL(I p , Ia∗ , In1 , . . . , InM ) = λLa (I p , Ia∗ )+LN (I p , Ia∗ , In1 , . . . , InM ), (8)where La is the selected appearance loss of Eq. 4, λ is a hyperparameter, and LN is the NCE loss. LN is defined as:
LN (I p , Ia∗ , In1 , . . . , InM )




exp(sim(I p , Ia∗ )/τ) ,
= −E logP
p n
exp(sim(I p , Ia∗ )/τ) + M
j=1 exp(sim(I , I j )/τ)(9)
where sim( ) is a cosine similarity function and τ is a temperature parameter. In our experiments in Sec. 5.10, we also evaluate the multiple-anchor NCE loss.
5. Experiments
5.1. Datasets
We evaluated object recognition accuracy and pose estimation error of our method by using three major datasets for product picking in warehouse scenarios: ARC dataset [3], YCB
dataset [7], and APC dataset [22]. We used RGB images rendered from the object models available in these datasets so that
we can have the ground truth pose parameters for evaluatingpose estimation performance.
ARC dataset. Fig. 3(a) shows the examples of the images contained in ARC dataset. It contains 34 distinct objects: 19 areplanar, nine are rectangular, and six are cylindrical. Following [28], the number of canonical planes are set to 2, 6, and 10,
respectively. The resulting number of the anchor sets are 152,
i.e., 19 × 2 + 9 × 6 + 6 × 10.
YCB dataset. Fig. 3(b) shows the examples of the images included in YCB dataset. The objects in YCB dataset have relatively complicated shapes, making it more challenging than
ARC dataset. It contains 34 objects: seven are planar, 11 arerectangular, and 16 are cylindrical. We prepare the same number of the canonical planes for each object type as ARC dataset,leading to 240 sets of the anchors, i.e., 7 × 2 + 11 × 6 + 16 × 10.
APC dataset. Fig. 3(c) shows the examples of the images included in APC dataset. It contains 17 objects: six are planar, eight are rectangular, and three are cylindrical. We prepare the same number of the canonical planes for each object
type as ARC dataset, leading to 90 sets of the anchors, i.e.,
6 × 2 + 8 × 6 + 3 × 10.
We generated positives for triplet learning by applying a random homography matrix to the same object as in the anchor,(c) APC dataset
Fig. 3: Object images from ARC dataset, YCB dataset, and APC dataset usedin our experiments. The color frame indicates the type of the object shape(Orange: planar, Red: rectangular, Blue: cylindrical).

and negatives by applying another random homography matrix
to different objects. The parameters of the homography matrix are sampled in the range of [−25, 25] pixels for translation, [−180, 180] degrees for rotation, [0.8, 1.2] for scale, and[−0.5, 0.5] for the remaining parameters. We generated a set ofanchors for each canonical plane by rotating it by 360/N degrees (We explain N in Sec. 4.1). For training, we used 20, 000
triplets for ARC dataset, 25, 000 for YCB dataset, and 10, 000
for APC dataset. The test sets were prepared so that they were
disjoint from the training sets. Specifically, we stored 152 × N
anchors (i.e., reference images) for ARC dataset, 240 × N for
YCB dataset, and 90 × N for APC dataset in the reference
database. We randomly generated 1, 000 test input samples foreach dataset in the same manner as the training samples. The
size of each image was fixed to 100 × 100.
5.2. Experimental Setup
Implementation Details. Our feature extraction network is
consisted of two convolution layers, one max pooling layer,
and one global max pooling layer. The output feature is 128dimensional. In the STN, we used a regressor with two convolution layers, two max pooling layers, and two fully connected
layers. We trained the network using stochastic gradient descent with a learning rate of 0.01 and a momentum parameterof 0.9 for 100 epochs.
Performance Measures. We used the correct match rate toevaluate the object recognition accuracy and the Frobenius
norm between the estimated homography matrix and the groundtruth homography matrix to evaluate the pose estimation error.
Note that we computed the pose estimation error if and only if
the object classes of the test input samples were correctly recognized as those of the reference images. We report the average
performance over the nine runs, each with the different trainingtesting splits.

6
2.0

0.5

1
2
4
6
9
Number of anchor images N

0.0

Transformed Input
[19]
Ours

Anchor(reference)

1.0

2.0
Pose Err.
12469
Number of anchor images N

0.0(c) Rec. Acc. on YCB

Fig. 5: Qualitative examples. In each row, from left to right, a test input image,
test input images transformed by the STN module trained by [28] and ours, andthe corresponding anchor (reference) image are shown, respectively.
1
2
4
6
9
Number of anchor images N(d) Pose Err. on YCB

multiple-anchor idea plays an important role in correctly estimating the relative pose changes.

2.0

5.4. Comparison with Triplet Learning [29] and STN Classifier [15]Pose Err.

Rec. Acc.

1.0

1
2
4
6
9
Number of anchor images N(b) Pose Err. on ARC

Rec. Acc.

(a) Rec. Acc. on ARC

0.5

Test Input

Pose Err.

Rec. Acc.

1.0

0.5

0.0
1
2
4
6
9
Number of anchor images N(e) Rec. Acc. on APC

1
2
4
6
9
Number of anchor images N(f) Pose Err. on APC

Fig. 4: Performance for different number of anchors N. (a) and (b) Results on
ARC dataset. (c) and (d) Results on YCB dataset. (e) and (f) Results on APC
datast.

5.3. Comparison with [28]We first demonstrate the effectiveness of our multiple-anchor
triplet learning by comparing with [28] based on the singleanchor triplet learning. We consistently set the balancing hyperparameter λ = 10 in Eq. 5 as suggested in [28]. The results areshown in Table 1(a) and (f). We see that our method (f) outperformed [28] (a) in both recognition accuracy and pose estimation error with significant margins on all the datasets. It proved
the strong advantage of our multiple-anchor triple learning inboth object recognition and pose estimation. For further analysis, we evaluated the performance of our method while changing the number of anchors N for N = 1, 2, 4, 6, 9. Note thatN = 1 corresponds to [28]. The results are shown in Fig. 4. The
results suggest that the performance of multiple-anchor cases(N > 1) tend to outperform the single-anchor version (N = 1),
which further emphasizes the effectiveness of our idea of using multiple anchors. We found that the larger N does not always improve performance. This may be because the number oftraining samples used per anchor may be reduced as the numberof anchors grows.
Fig. 5 shows qualitative examples. We see that the images
transformed by [28] have different appearances as the corresponding anchors. Meanwhile, those obtained by using our
method are close to the anchors, even for objects with relatively
complex contours (as the example in the second row) and input images with large rotation angles from the anchors (as theexample in the third row). This clearly demonstrates that our

We compared the performance of our method with that of
the standard triplet learning method (TL) [29]. We only evaluated the object recognition accuracy because the standard triplet
network was unable to estimate object poses. The results areshown in Table 1(b) and (f). Comparing the standard triplet
learning method (b) with our method (f), we see that our network using the multiple-anchor triplet learning is superior to
the simple triplet network.
Next, we compared the performance of our method with thatof the original STN [15]. For a fair comparison, the existing
STN is constructed to have the same configuration as our network. That is, it consists of one STN module, two convolution
layers, two max pooling layers, and two fully connected layers.
Instead of triplet-based learning, the baseline network is trainedwith the cross-entropy loss. The results are shown in Table 1(c)and (f). Comparing the original STN (c) with our method (f),our approach of embedding an STN module in multiple-anchor
triplet learning is superior to the original STN.
5.5. Evaluation of Object Recognition using Local Descriptors
Object recognition using SIFT+RANSAC and SuperPoint+RANSAC, i.e., image registration using RANSAC [13]with SIFT [20] local descriptors and SuperPoint [12] local descriptors, has been a strong approach to joint object recognition and pose estimation. We thus compare our method withthem to analyze the superiority of our method. For SIFT feature extraction, we set the sigma of the Gaussian to 1.6, thecontrast threshold to 0.004, and the edge threshold to 1, 000.
For SuperPoint feature extraction, we set the radius of nonmaximum suppression to 4, the threshold of keypoints to 0.005,
and the maximum number of keypoints to 1024. The results areshown in Table 1(d) and (e). Our method (f) outperforms both
SIFT+RANSAC (d) and SuperPoint+RANSAC (e) in terms ofobject recognition accuracy while achieving comparable performance in pose estimation.

7

(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)
(j)

Table 1: Performance of our method and the existing methods.
ARC
YCB
Rec. Acc.
Pose Err.
Rec. Acc.
Pose Err.
[28] (Single anchor, N = 1)
0.84±0.01 1.86±0.01 0.72±0.01 1.88±0.01
0.87±0.01n/a
0.78±0.01
n/a
TL [29]STN [15]0.88±0.02 2.80±0.16 0.82±0.02 6.13±0.01
SIFT+RANSAC [20, 13]0.82±0.02 0.63±0.05 0.80±0.03 0.64±0.19
SuperPoint+RANSAC [12, 13]0.83±0.01 0.66±0.03 0.82±0.01 0.64±0.03
Ours (Multiple anchors, λ = 10, N = 4)0.93±0.01 0.66±0.01 0.84±0.01 0.60±0.01
Ours (Multiple anchors, λ = 0, N = 4)0.88±0.01 1.70±0.02 0.78±0.01 1.75±0.03
0.87±0.01 1.64±0.01 0.74±0.01 1.65±0.01
Ours (Multiple anchors, Random position, λ = 10, N = 4)
Ours (Multiple anchors, No STN module in negative, λ = 10, N = 4) 0.93±0.01 0.67±0.03 0.83±0.01 0.63±0.01
0.94±0.01 0.67±0.02 0.89±0.01 0.61±0.02
Ours (Multiple anchors, NCE loss, λ = 10, N = 4)

(a)

(b)

(c)

Fig. 6: (a) Planar objects, (b) non-planar objects, and (c) examples of nonplanar objects from different views.
Table 2: Effect of planar objects and non-planar objects in YCB dataset.
Rec. Acc.
Pose Err.
(a) Ours (Planar)
0.91±0.02 0.57±0.03
0.78±0.02 1.86±0.04(b) [28] (Planar)(c) Ours (Non-planar) 0.81±0.02 0.68±0.05(d) [28] (Non-planar)
0.64±0.01 1.89±0.03

5.6. Performance without Appearance Loss
To further analyze the effectiveness of the appearance loss
using multiple anchors Eq. 5, we evaluate the case when we set
λ = 0. The results are shown in Table 1(g). Comparing themethod without appearance loss (g) with the method with it (f),we confirm that our multiple-anchor version of the appearance
loss contributes to both increasing object recognition accuracy
and reducing the pose estimation error.
5.7. Comparison of Anchor Acquisition Methods
As described in Sec. 4.1, our method acquires anchor imagesof canonical planes in the fixed camera setting. To analyze theimpact of the way to collect the anchors, we evaluated the performance with anchors acquired from random camera positions.
The results when using the anchors from random position are
shown in Table 1(h). Comparing with the results when using
the fixed positions in (f), we confirm that the acquisition of representative canonical planes is effective in terms of improving
the performance of object recognition and pose estimation.
5.8. Investigation of Effect of Non-planar Objects
We evaluated the performance of our method for planar objects and non-planar objects contained in YCB dataset. The
planar objects mean simple shapes, i.e., planar, rectangular,
and cylindrical, while non-planar objects mean complex shapes.
Specifically, non-planar objects consist of not only canonicalplanes but also curved surfaces, projections, and concavities.

APC
Rec. Acc.
Pose Err.
0.84±0.01 1.90±0.01
0.88±0.01
n/a
0.92±0.04 2.68±0.53
0.87±0.01 0.50±0.02
0.90±0.01 0.51±0.03
0.93±0.01 0.46±0.01
0.89±0.01 1.68±0.04
0.91±0.01 1.59±0.01
0.93±0.01 0.53±0.01
0.97±0.01 0.52±0.02

We divided YCB dataset into 11 planar objects in Fig. 6(a) and23 non-planar objects in (b). Examples of non-planar objects
from different views are shown in (c). We used the same configuration of Table 1(f) except the datasets. We compared theperformance between our method and [28].
The results for planar objects and non-planar objects in YCB
dataset are shown in Table 2. For planar objects, our method (a)obtained higher performance than (b). For non-planar objects,
our method (c) also obtained higher performance than (d). We
believe that our multiple-anchor triple learning has the advantage for both planar objects and non-planar objects. However,the performance for the non-planar objects (c) is lower thanthat for planar objects (a). We consider that this is the limitation of our method because the homography estimation usedin the STN module cannot sufficiently represent the complexityof the shape of non-planar objects. We need to develop further
a method for pose estimation of complex shaped objects.
5.9. Performance without STN Module in Negative Stream
We evaluated the case when the STN module is removed
from the negative stream of Fig. 1(b). We used the same configuration of Table 1(f) except the use of the STN module ofthe negative stream. The results without the STN module in thenegative stream are shown in Table 1(i) and the results with itin (f). We see that the pose estimation error of (f) was slightly
improved compared with the one of (i) in all the datasets. We
consider that the STN module in the negative stream is meaningful to reducing the pose estimation error.
5.10. Performance with Multiple-anchor NCE Loss
We evaluated the performance of our method when using themultiple-anchor NCE loss LN instead of the triplet loss Lt . We
set M = 64, λ = 10, and τ = 0.07. The same structures ofthe positive stream, the anchor stream, and the negative stream
illustrated in Fig. 1(b) were used. The number of anchors in theselection layer was N = 4.
The results using NCE loss LN are shown in Table 1(j). Comparing with the results of Triplet loss Lt (f), we see that theNCE loss increased the object recognition accuracy in all thedatasets. However, the NCE loss did not contribute to reducing the pose estimation error in APC dataset. We consider that
NCE loss is valuable in improving object recognition accuracy.

8
6. Conclusions
We proposed a multiple-anchor triplet learning method forjoint object recognition and pose estimation without explicit
pose parameter labels. Aiming at avoiding the problem offalling into poor local minima during pose estimation, ourmethod uses multiple anchors to enable finer pose estimation.
We showed that the proposed multiple-anchor triplet learning
can be readily implemented by introducing a simple selection
layer that determines the closest anchor to the positive image.
We demonstrated that our method outperformed several existing methods with significant performance gain.
In future work, we will further evaluate our method on
datasets of objects with various shapes and complex backgrounds. We are planning to expand the appearance loss term tometrics other than the L1 norm and develop a spatial transformmodule that can handle more transformations than just planar
homography transformations.
References[1] Ahmed, E., Jones, M., Marks, T.K., 2015. An improved deep learning
architecture for person re-identification, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3908–
3916.
[2] Andrews, S., Tsochantaridis, I., Hofmann, T., 2002. Support vector machines for multiple-instance learning, in: Proceedings of Advances in
Neural Information Processing Systems (NeurIPS), pp. 561–568.
[3] Araki, R., Yamashita, T., Fujiyoshi, H., 2018. ARC2017 RGB-D dataset
for object detection and segmentation, in: Proceedings of Late Breaking
Results Poster on International Conference on Robotics and Automation.
[4] Balntas, V., Doumanoglou, A., Sahin, C., Sock, J., Kouskouridas, R.,
Kim, T., 2017. Pose guided RGBD feature learning for 3D object pose
estimation, in: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 3876–3884.
[5] Bromley, J., Guyon, I., LeCun, Y., Sackinger, E., Shah, R., 1993. Signature verification using a ”siamese” time delay neural network, in:Proceedings of Advances in Neural Information Processing Systems(NeurIPS), pp. 737–744.
[6] Bui, M., Zakharov, S., Albarqouni, S., Ilic, S., Navab, N., 2018. Whenregression meets manifold learning for object recognition and pose estimation, in: Proceedings of IEEE International Conference on Robotics
and Automation (ICRA), pp. 1–7.
[7] Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., Dollar, A.M.,
2015. The YCB object and model set: Towards common benchmarks for
manipulation research, in: Proceedings of International Conference on
Advanced Robotics (ICAR), pp. 510–517.
[8] Chen, D.Y., Tian, X.P., Shen, Y.T., Ouhyoung, M., 2003. On Visual
Similarity Based 3D Model Retrieval. International Journal of Computer
Graphics Forum , 223–232.
[9] Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020. A simple framework for contrastive learning of visual representations, in: Proceedings of
Machine Learning Research (PMLR), pp. 1597–1607.
[10] Chen, W., Chen, X., Zhang, J., Huang, K., 2017. Beyond triplet loss:
A deep quadruplet network for person re-identification. Proceedings ofIEEE Conference on Computer Vision and Pattern Recognition (CVPR) ,1320–1329.
[11] Correll, N., Bekris, K.E., Berenson, D., Brock, O., Causo, A., Hauser,K., Okada, K., Rodriguez, A., Romano, J.M., Wurman, P.R., 2018. Analysis and observations from the first Amazon Picking Challenge. IEEE
Transactions on Automation Science and Engineering 15, 172–188.
[12] DeTone, D., Malisiewicz, T., Rabinovich, A., 2018. Superpoint: Selfsupervised interest point detection and description, in: 2018 IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops(CVPRW), pp. 337–33712.
[13] Fischler, M., Bolles, R., 1981. Random sample consensus: A paradigm
for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24, 381–395.

[14] He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 770–778.
[15] Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K., 2015.
Spatial transformer networks, in: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 2017–2025.
[16] Kanezaki, A., Matsushita, Y., Nishida, Y., 2018. Rotationnet: Joint object
categorization and pose estimation using multiviews from unsupervised
viewpoints, in: Proceedings of IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pp. 5010–5019.
[17] Leitner, J., Tow, A.W., Sunderhauf, N., Dean, J.E., Durham, J.W., Cooper,M., Eich, M., Lehnert, C., Mangels, R., McCool, C., Kujala, P.T., Nicholson, L., Pham, T., Sergeant, J., Wu, L., Zhang, F., Upcroft, B., Corke,P., 2017. The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research, in: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 4705–4712.
[18] Li, Y., Wang, G., Ji, X., Xiang, Y., Fox, D., 2018. Deepim: Deep iterative
matching for 6D pose estimation, in: Proceedings of European Conference on Computer Vision (ECCV), pp. 695–711.
[19] Liu, W., Anguelov, D., Erhan, D., 2016. Ssd: Single shot multibox
detector, in: Proceedings of European Conference on Computer Vision(ECCV), pp. 21–37.
[20] Lowe, D.G., 2004. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision (IJCV) 60, 91–110.
[21] Redmon, J., Farhadi, A., 2017. Yolo9000: Better, faster, stronger, in: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6517–6525.
[22] Rennie, C., Shome, R., Bekris, K.E., Souza, A.F.D., 2016. A dataset for
improved RGBD-based object detection and pose estimation for warehouse pick-and-place. IEEE Robotics and Automation Letters 1, 1179–
1185.
[23] Rusu, R.B., Bradski, G., Thibaux, R., Hsu, J., 2010. Fast 3d recognition and pose using the viewpoint feature histogram, in: Proceedings ofInternational Conference on Intelligent Robots and Systems (IROS), pp.
2155–2162.
[24] Schroff, F., Kalenichenko, D., Philbin, J., 2015. FaceNet: A unified embedding for face recognition and clustering, in: Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
815–823.
[25] Sohn, K., 2016. Improved deep metric learning with multi-class n-pairloss objective, in: Proceedings of Advances in Neural Information Processing Systems (NeurIPS), pp. 1857–1865.
[26] Sundermeyer, M., Marton, Z.C., Durner, M., Brucker, M., Triebel, R.,
2018. Implicit 3d orientation learning for 6D object detection from rgb
images, in: Proceedings of European Conference on Computer Vision(ECCV), pp. 712–729.
[27] Tekin, B., Sinha, S., Fua, P., 2018. Real-time seamless single shot 6D
object pose prediction, in: Proceedings of IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), pp. 292–301.
[28] Ueno, K., Irie, G., Nishiyama, M., Iwai, Y., 2019. Weakly supervised
triplet learning of canonical plane transformation for joint object recognition and pose estimation, in: Proceedings of International Conference on
Image Processing (ICIP), pp. 2476–2480.
[29] Wang, J., Song, Y., Leung, T., Rosenberg, C., Wang, Philbin, J., Chen, B.,
Wu, Y., 2014. Learning fine-grained image similarity with deep ranking,in: Proceedings of IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), pp. 1386–1393.
[30] Wu, Z., Xiong, Y., Yu, S., Lin, D., 2018. Unsupervised feature learning via non-parametric instance discrimination, in: Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
3733–3742.
[31] Yu, B., Liu, T., Gong, M., Ding, C., Tao, D., 2018. Correcting the triplet
selection bias for triplet loss, in: Proceedings of The European Conference on Computer Vision (ECCV), pp. 71–86.
[32] Zeng, A., Song, S., Yu, K.T., Donlon, E., Hogan, F.R., Bauza, M., Ma, D.,
Taylor, O., Liu, M., Romo, E., Fazeli, N., Alet, F., Dafle, N.C., Holladay,R., Morona, I., Nair, P.Q., Green, D., Taylor, I., Liu, W., Funkhouser, T.,
Rodriguez, A., 2019. Robotic pick-and-place of novel objects in clutter
with multi-affordance grasping and cross-domain image matching. International Journal of Robotics Research , 1–16.