WEAKLY SUPERVISED TRIPLET LEARNING OF CANONICAL PLANE
TRANSFORMATION FOR JOINT OBJECT RECOGNITION AND POSE ESTIMATION
Kouki Ueno1 , Go Irie2 , Masashi Nishiyama1 , Yoshio Iwai1
1

Graduate School of Sustainability Science, Tottori University, Japan
2
NTT Corporation, Japan
ABSTRACT

We propose a method for jointly performing object recognition and pose estimation using training samples of canonicalplanes to reduce the effort of data label supervision. Collecting a sufficient number of training samples is important to realizing high performance. However, labeling pose parameters
is time consuming. We thus train our network model using
only object class labels without explicitly labeling pose parameters. To recognize objects and estimate their poses, wedesign a network with a spatial transformer in a contrastive
learning manner such that the canonical plane of an object isalways transformed to a certain pose and the features are consistent with those of the object class. Experiments show that
our method has improved accuracy in object recognition andlower error in pose estimation compared with simply using
triplet learning or a spatial transformer network on a publicly
available dataset.
Index Terms— Weak supervision, Canonical plane,
Triplet learning, Spatial transformer networks
1. INTRODUCTION
There is a demand for robotic arm systems [1, 2] that are automatically able to pick and stow various objects in a warehouse. Grasping an object with a robotic arm requires twotypes of computer vision technique, namely object recognition, and pose estimation. In the field of the automatic warehouse, there are already many applications that adopt object
recognition techniques [3, 4]. Existing methods [5, 6] that
perform object recognition and pose estimation simultaneously have recently attracted attention. The existing methods perform well when many training samples with object
class labels and pose parameter labels are collected. However, labeling pose parameters to manage training samples
is time consuming and subjective. Weakly supervised learning methods that jointly perform object recognition and pose
estimation are thus expected. Recent papers [7, 8] tackled
the development of the weakly supervised learning method.
Kanezaki et al. [7] assumed that object poses are categorized
into several viewpoints and multiple images are available in
the test process. Sundermeyer et al. [8] assumed that three-dimensional models of objects are available for training samples. We instead focus on achieving the goal without making
such assumptions.
Before we consider a weakly supervised learning method,
we discuss major existing methods of object recognition
using triplet learning [9] and spatial transformer networks(STNs) [10]. As described in [11, 12, 13, 14], triplet learning
has achieved high accuracy of object recognition in various
applications. Triplet learning is used to extract invariant deep
features when there is a variation in appearance due to posechanges. When training a network model, the triplet loss
function decreases the distance between an anchor image(simply referred to as an anchor) and positive image (simply referred to as a positive), while increasing the distance
between the anchor and negative. In contrast to triplet learning, the STN infers parameters of spatial transformation toincrease the accuracy of object recognition without the use ofpose parameter labels for training. After applying a spatialtransformation to deep feature maps using the inferred theparameters, the STN increases the tolerance to pose changes
in the recognition task.
The present paper designs a novel network model by embedding an STN module in triplet learning to jointly perform
object recognition and pose estimation. We do not use poseparameter labels to train our network model. We perform
weakly supervised learning for object recognition and pose
estimation using only RGB images with object class labels.
A straightforward idea would be to first train the network including an STN module with a standard triple learning, extract deep features for object recognition, and infer the poseparameters using the STN for pose estimation. However, sucha simple combination does not work well for the following
reasons.
• An STN is often implemented to alleviate the variationof pose changes through planar spatial transformations.
However, we cannot simply use an STN module because the objects have three-dimensional shapes.
• The anchor of triplet learning dynamically affects theaccuracy of pose estimation and object recognition. A
criterion for selecting anchors that are beneficial to both
has not been investigated.

Positive #&s(#& )c(s(#& ))!"$c(#$ )

Anchor #Negative #'s(#' )(a)!,c(s(#' ))(b)Fig. 1. Our network model using triplet learning of canonicalplane transformation. Function s( ) synthesizes an image byremoving pose changes using the STN. Function c( ) generates a feature map from an image using a convolutional neuralnetwork and vectorizes the feature map.

Fig. 2. Examples of observed canonical planes.
PlanerRectangular• The triplet loss function is designed to increase the accuracy of object recognition. It does not explicitly reduce the error in pose estimation.

CylinderApproximation

We exploit the following strategy to overcome these issues.
• We consider how to use an STN module while maintaining a planar spatial transformation for our task. We
introduce canonical planes of a three-dimensional object. Generally, an object is approximately represented
by a large number of planes. We assume that an object
comprises a very small number of canonical planes.
• We believe that a good anchor in triplet learning forboth object recognition and pose estimation is an image that well describes the cues of an object. To select anchor images, we acquire representative canonicalplanes that are less affected by pose changes.
• We add a loss term to smoothly connect from the STN
to triplet loss. Pose estimation using an STN is geometrically possible when the anchor and positive appear similar after applying the STN. Our method thus
computes the L1 loss between the anchor and positive
for a triplet loss function.
In experiments, our method improved the accuracy of object recognition from 90% to 95% compared with an existing triplet learning method while reducing the error of pose
estimation from 1.1 to 0.4 compared with an existing STNmethod on the Rutgers APC RGB-D dataset.
2. TRIPLET LEARNING OF CANONICAL PLANE
TRANSFORMATION
2.1. Overview
Figure 1 illustrates our network model of canonical planetransformation. In the figure, I p represents the image of thepositive for triplet learning, I a the image of the anchor, andI n the image of the negative. The triplet of the positive,Fig. 3. Examples of three categories of object shape.
anchor, and negative corresponds to canonical planes of objects. The positive belongs to the same object as the anchor
while the negative belongs to a different object. We set theL1 loss between I a and s(I p ) to evaluate the closeness of
the anchor and positive when applying the STN. Our method
computes the triplet loss using vectorized feature maps of theanchor, positive, and negative. Note that our method shares
the same weights of c( ) and s( ) for the anchor, positive,and negative streams. In the test process, we prepare target
samples of representative canonical planes. Given a query
sample, our method infers relative spatial parameters of posechange from the query sample to a target sample. Our method
also extracts deep features for object recognition. Canonicalplanes, anchors, and the L1 loss are respectively described inSections 2.2, 2.3, and 2.4.
2.2. Canonical planes
Objects in a warehouse are often packed into a textured box
or bag so as to prevent their deformation. We assume thatthe objects consist of a plurality of canonical planes. Whenan object is placed on a floor, the pose of the object does notcontinue to change. The pose settles into a static state, asshown in Figure 2(a). The object pose is thus not free to shift
in all directions, and there is a bias to a specific direction.
In other words, one surface is observed mainly for each object, such as in Figure 2(b). We term the surface a canonicalplane. The number of canonical planes depends on the shapeCameraYellow : Planar, Red : Rectangular, Blue : Cylinder

Optical axisofcamera
Floor

Object(a)(b)

Fig. 4. Setting for acquiring anchor images.
Fig. 5. Objects used in our experiments.
of the object as illustrated in Figure 3. We assume that twocanonical planes are observed for a planar object, six canonical planes are observed for a rectangular object, and 10 canonical planes are observed for a cylinder approximated as an octagonal prism. Using canonical planes, our method estimatesthe object pose through a simple STN module implemented
for the planar object.
2.3. Anchor for triplet learning
To improve the accuracy of object recognition and reduce theerror of pose estimation, we acquire an anchor image in which
a representative canonical plane well expresses a cue of an object. We collect representative canonical planes in the setting
as illustrated in Figure 4(a). We set up the camera so that its
optical axis is perpendicular to the floor. We place the object parallel to the floor surface. We let the optical axis of thecamera pass through the object center of gravity. Figure 4(b)shows examples of anchor images for representative canonical planes.
2.4. L1 loss for pose estimation
To jointly perform object recognition and pose estimation, weuse a loss function for our network model expressed as
L = Lt + Ls ,(1)where Lt is the triplet loss term for object recognition and Ls
is the L1 loss term for pose estimation. To evaluate the distance between the appearances of canonical planes in images,
we define Ls as
Ls = ||I as(I p )||1 .

(2)

The equation returns a small value when taking similar appearances between the image of the anchor and the image ofthe positive after applying the STN. After learning, the STN
tries to linearly transform the query image of an arbitrary object pose so as to match the anchor image, where the resulting
transformation parameters inferred by the STN gives the poseparameters of the object.

To evaluate the distance between vectorized feature maps
in triplet learning, we define Lt as
Lt = max(||c(I a )||c(I a )c(s(I p ))||22c(s(I n ))||22 + m, 0),(3)where m is the margin. The equation returns a small value
when the anchor and positive feature vectors of the same object are close while the anchor and negative feature vectorsof different objects are not close. Object recognition can be
performed through nearest-neighbor classification.
3. EXPERIMENTS
3.1. Dataset
To evaluate the accuracy of object recognition and the error ofpose estimation, we used three-dimensional mesh models of
17 objects included in the Rutgers APC RGB-D dataset [15].
Figure 5 shows examples of the objects. We generated positive samples of triplet learning by applying an affine transformation to the same object as in the anchor, and negative
samples by applying an affine transformation to different objects. We randomly set the parameters of affine transformation in the range of translation of [ 50, 50] pixels, range ofrotation of [ 60, 60] degrees, and range of scale of [0.8, 1.2].
We used anchor images as target samples in the test process.
We randomly generated query samples in the same manner as
the generation of training samples. We used 10,000 training
triplet samples, 1000 query samples, and 90 target samples.
The size of each sample was 100 ⇥ 100 pixels. We repeatedthe procedure nine times.
In triplet learning, we computed 128-dimensional feature vectors using two convolutional layers, one max poolinglayer, and one global max pooling layer. In the STN, we useda regressor having two convolutional layers, two max poolinglayers, and two fully connected layers.
We used the correct-match rate to evaluate the accuracyof object recognition. We used the Frobenius norm between
the inferred affine matrix and labeled affine matrix to evaluate
the error in pose estimation.

Accuracy ofobject recognition [%]Error ofpose estimationrole in correctly estimating relative spatial parameters and removing the effects of pose changes.

100

3.3. Comparisons with existing methods
800101!

1010010100

1.00!

Fig. 6. Accuracy of object recognition and the error in pose
estimation for a varying parameter .

QuerySpatial transformation
Baseline Our method

TargetWe compared the performance of our method with that of an
existing triplet learning method [9]. We used a triplet network by removing the STN s( ) and L1 loss term Ls from ournetwork. We only evaluated the accuracy of object recognition because the triplet network was unable to estimate object
poses. The average accuracy achieved using the triplet network is 90.2 ± 1.5 whereas the accuracy of our method is95.1 ± 1.1. We confirmed that our network combining triplet
learning with an STN using the L1 loss is superior to the simple triplet network.
We next compared the performance of our method withthat of an existing STN method [10]. We combined a recognition module with the STN module used in our network.
The recognition module has two convolutional layers, twomax pooling layers, and two fully connected layers. We usedcross-entropy loss in the existing method. We evaluated theaccuracy of object recognition and the error in pose estimation. The average accuracy and error when using the existing
STN method are 92.2 ± 4.4 and 1.09 ± 0.44 whereas the accuracy and error when using our method are 95.1 ± 1.1 and
0.44 ± 0.02. We confirmed that our method embedding anSTN module in triplet learning is superior to the existing STN
method.
3.4. Comparison with the random selection of anchors

Fig. 7. Examples of query samples after removing posechanges.
3.2. Basic performance
We evaluated the performance of our method while changing
the parameter in Equation (1). We set =0, 1, 10, 100.
Note that = 0 refers to the baseline method where the STN
is simply set in triplet learning without the use of Ls .
Figure 6 shows the average matching rate of object recognition and the average Frobenius norm of pose estimation.
We see that the performances for = 1, 10 were superior
to the performance for = 0. We confirmed that Ls for ourmethod works well in terms of increasing the accuracy of object recognition and reducing the error in pose estimation. We
obtained high performance when = 10.
Figure 7 shows examples of the target samples, and thequery samples after passing through the STN module. We seethat the samples modified using the baseline method have almost the same appearances as the query images. Meanwhile,
we see that the images obtained using our method are close
to the target samples. We believe that Ls plays an important

As described in Section 2.3, our method selects anchor images by acquiring representative canonical planes. We here
compare the performance of our method with that of the random selection of anchors. The average accuracy and error
when using the random selection are 89.1 ± 1.1 and 0.82 ±0.02 whereas the average accuracy and error when using ourmethod are 95.1 ± 1.1 and 0.44 ± 0.02. We confirmed thatthe acquisition of representative canonical planes is effective
in terms of improving the performance of object recognitionand pose estimation.
4. CONCLUSIONSWe proposed a weakly supervised learning method to jointly
recognize objects and estimate their poses through the triplet
learning of canonical plane transformation. We demonstrated
that our method outperforms the single use of triplet learningor the STN even though we did not provide pose parameterlabels.
In future work, we will further evaluate our method on
datasets of objects having various shapes. We will also explore the use of homography transformation instead of affine
transformation to treat pose changes that are more complex.

5. REFERENCES[1] P. Baker and Z. Halim, “An exploration of warehouseautomation implementations: cost, service and flexibility issues,” Supply Chain Management: An International Journal, vol. 12, no. 2, pp. 129–138, 2007.
[2] N. Correll, K. E. Bekris, D. Berenson, O. Brock,
A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M.
Romano, and P. R. Wurman, “Analysis and observations from the first amazon picking challenge,” IEEETransactions on Automation Science and Engineering,
vol. 15, no. 1, pp. 172–188, 2018.
[3] W. Liu, D. Anguelov, and D. Erhan, “Ssd: Single shotmultibox detector,” in Proceedings of European Conference on Computer Vision, 2016, pp. 21–37.
[4] J. Redmon and A. Farhadi, “Yolo9000: Better, faster,stronger,” in Proceedings of IEEE Conference onComputer Vision and Pattern Recognition, 2017, pp.
6517–6525.
[5] B. Tekin, S.N. Sinha, and P. Fua, “Real-time seamlesssingle shot 6d object pose prediction,” in Proceedingsof IEEE Conference on Computer Vision and Pattern
Recognition, 2018, pp. 292–301.
[6] V. Balntas, A. Doumanoglou, C. Sahin, J. Sock,
R. Kouskouridas, and T. Kim, “Pose guided rgbd featurelearning for 3d object pose estimation,” in Proceedingsof IEEE International Conference on Computer Vision,
2017, pp. 3876–3884.
[7] A. Kanezaki, Y. Matsushita, and Y. Nishida, “Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints,” in Proceedings of IEEE Conference on Computer Vision andPattern Recognition, 2018.
[8] M. Sundermeyer, Z. C. Marton, M. Durner, M. Brucker,and R. Triebel, “Implicit 3d orientation learning for6d object detection from rgb images,” in Proceedingsof European Conference on Computer Vision, 2018, pp.
712–729.
[9] J. Wang, Y. Song, T. Leung, C. Rosenberg, Wang,J. Philbin, B. Chen, and Y. Wu, “Learning fine-grained
image similarity with deep ranking,” in Proceedingsof IEEE Conference on Computer Vision and PatternRecognition, 2014, pp. 1386–1393.
[10] M. Jaderberg, K. Simonyan, A. Zisserman, andK. Kavukcuoglu, “Spatial transformer networks,” inProceedings of the 28th International Conference on
Neural Information Processing Systems, 2015, vol. 2,pp. 2017–2025.

[11] D. Cheng, Y. Gong, S. Zhou, J. Wang, and N. Zheng,“Person re-identification by multi-channel parts-based
cnn with improved triplet loss function,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1335–1344.
[12] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, andJ. Sivic, “Netvlad: Cnn architecture for weakly supervised place recognition,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition,
2016, pp. 1437–1451.
[13] A. Gordo, J. Almazán, J. Revaud, and D. Larlus, “Endto-end learning of deep visual representations for imageretrieval,” International Journal of Computer Vision,
vol. 124, no. 2, pp. 237–254, 2017.
[14] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A
unified embedding for face recognition and clustering,”
in Proceedings of IEEE Conference on Computer Vision
and Pattern Recognition, 2015, pp. 815–823.
[15] C. Rennie, R. Shome, K. E. Bekris, and A. F. D. Souza,“A dataset for improved rgbd-based object detection andpose estimation for warehouse pick-and-place,” IEEE
Robotics and Automation Letters, vol. 1, pp. 1179–1185,
2016.