Semi-supervised Pose Estimation with Geometric Latent Representations
Authors: Luis A. Perez Rey
Dmitri Jarnikov
Mike Holenderski
Venue:  (2020)
Keywords: Semi-supervised learning
pose estimation
angle estimation
variational autoencoders

Abstract: Pose estimation is the task of finding the orientation of an  object within an image with respect to a fixed frame of reference. Current classification and regression approaches to the task require large quantities of labelled data for their purposes. The amount of labelled data for pose estimation is relatively limited. With this in mind, we propose the use of Conditional Variational Autoencoders (CVAEs) \cite{Kingma2014a} with circular latent representations to estimate the corresponding 2D rotations of an object. The method is capable of training with datasets that have an arbitrary amount of labelled images providing relatively similar performance for cases in which 10-20% of the labels for images is missing. 

Under review as a conference paper at ICLR 2019
SEMI-SUPERVISED POSE ESTIMATION WITH GEOMETRIC LATENT REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Pose estimation is the task of finding the orientation of an object within image with respect to a fixed frame of reference. Current classification and regression ap- proaches the task require large quantities of labelled data for their purposes. The amount of labelled data for pose estimation is relatively limited. With this in mind, we propose the use of Conditional Variational Autoencoders (CVAEs) Kingma et al. (2014) with circular latent representations to estimate the corresponding 2D rotations of an object. The method is capable of training with datasets that have an arbitrary amount of labelled images providing relatively similar performance for cases which 10-20% of the labels for images is missing.
INTRODUCTION
Pose estimation the computer vision task in which the rotation of an object is estimated with respect to a certain frame of reference. The estimation of the position and the pose of an object within an image an important challenge of computer vision and can be applied to wide range of areas such as car navigation (Braun et al. 2016), augmented reality (Marchand et al., 2016) and robot interaction (Xiang al. 2018).
Several methods based on the use of convolutional neural networks have been proposed high per- formance. These approaches are divided into two: classification- where the possible rota- tions that can be predicted are discretized into predefined sets with different coarseness levels and regression- based where a continuous prediction is provided.
Most of them are supervised methods which require large datasets with accurately labelled images and with a sufficiently good coverage of all possible rotations each object instance. One of such datasets obtained from natural images and manually annotating objects in images is the Pascal3D+ dataset (Xiang et al. 2014) with about 22 thousand images from twelve categories. Compared to other datasets for other computer vision tasks such as classification (with millions of images) Li et al. (2009) the example data is scarce and the labelling process might be cumbersome and prone errors. Thus, the need for a method train with limited amount of labelled data.
To solve this problem there have been dataset generation approaches proposed to create new training labels and images (Su et al. 2015 Varol et al. 2017 Johnson-Roberson et al. (007)) from 3D models and inserting them in natural scenery. Other approaches are focused on reducing the supervision to auxiliary tasks indirectly related the pose estimation such as in Esteves et al, (2018b) [Prokudin et al. (2018) addresses the problem of label quality by including an uncertainty estimation in the rotation predictions.
The work presented in this paper proposes a semi-supervised probabilistic method for pose esti- mation that incorporates the geometry of rotations into the latent variable representations. The method is capable of using both labelled and unlabelled images from object to improve the rota- tion regressions by performing the auxiliary task of reconstruction from the autoencoders. We have also investigated the effects of having variable amount of labelled data in controlled setting and variable coverage of the rotations.
In the general setting of object pose estimation object is located in the scene of interest and an image has been taken of it. The object has a certain rotation with respect to the a fixed frame of reference. The goal is to estimate the rotation that the object has undergone from the available

---

Under review as a conference paper at ICLR 2019
image. In particular interested in the case were might have dataset in which the objects might be partially labelled.
More formally, consider that have training dataset consisting of L labelled images with their corresponding rotation representations (i))) in which 2; E X and fy E R correspond to the (-th image and rotation, respectively, and unlabelled images (I=)u=1- How can we obtain model for estimating the pose of object based on image, taking advantage of both datasets? How does such a model perform with respect to the amount of supervision provided and the angle coverage?
2 RELATED WORK
Current approaches for rotation estimation using neural networks can be divided into classification and regression. Classification methods discretize the possible rotations orientations of an object
(Su et al., 2015 Li et al. 2019b] Tulsiani & Malik (015) while regression methods provide more approach to the continuous nature of rotations (Mahendran et al., 2018a Liao et al. 2019 Prokudin et al. 2018).
Since rotations have a particular topology, their structure needs to be taken into account for both classification and regression tasks. Mahendran et al. (2018a) studies the performance of regression methods by using different rotation representations in a supervised setting. [iao et al. (2019) focus on the task of regression by representing rotations in n-spheres for different pose-related tasks.
Prokudin et al. (2018).
In Esteves al. 2018b equivariant representations used which capture the group structure of rotations. Their method is trained with the aid of spherical convolutional neural networks (Cohen al. (2018) based on an auxiliary classification task, this method is unsupervised in the sense that does not need labels for angles and is capable of estimating relative rotation among different views of object.
These supervised methods still require the use of labelled data in good quantity and quality. To circumvent these needs, we are interested in the semi-supervised and unsupervised approaches such as Esteves et al. (2018b a): Rhodin et al.] (2018); Kulkarni et al. (2015). Similar to Kulkarni et al. 2015), our focus to use probabilistic models to capture the topological structure of rotations.
In particular, within the Variational Autoencoders (VAE) Kingma & Welling (2014) context there is a recent growing interest in the study of latent variables with different geometries: hyperspherical Davidson et al. 2018). S3(3) (Falorsi et al. 2018). hyperbolic (Mathieu et al. 2019) and arbitrary closed manifolds (Pérez Rey et al. 2019 et al. 2019a).
VAEs provide an unsupervised method for latent variable estimation and in principle could be used for capturing the underlying geometrical features for a dataset. The Conditional Variational Au- toencoder (CVAE) (Kingma et al. 2014) allows to introduce supervision into the latent variable representations/ Prokudin et al. (2018) used the CVAE
supervised method for pose prediction and adding an uncertainty estimation for the input data. This method could not be directly extended semi-supervised pose estimation without extra modifications.
We use the previous work as a starting point for developing a semi-supervised method for pose estimation based CVAEs with underlying geometrical latent variables for pose estimation.
3 METHODS
ROTATION REPRESENTATION
is important to choose an appropriate latent representation for the rotations that captures their
topological properties. This has been discussed in Liao et al. (2019) and Mahendran et al. (20 Sa) where several representations are described. In the case of 2D planar rotations along one fixed
axis each rotation be described in terms of the I-dimensional circle 5' Liao et al. 2019 Our
method will has focus on the use of a probabilistic models (Variational Autoencoders Kingma & Welling (2014) capable of encoding the input images in circular latent space.
2

---

Under review as a conference paper at ICLR 2019
To the best of our knowledge, there three approaches to implement variational autoencoder with a hyperspherical latent space. Firstly in Invaiidoo et al. (2018). they have focused on a method that is tailored specifically for hypersphere. On the other hand, Pêrez Rey al. (2019) and [i et al. (2019a) propose methods for choosing arbitrary closed spaces. We have chosen to focus on using the work in Pérez Rey et al. (2019) represent the rotations in S since it can provide arbitrary manifolds as latent space which can be used in further experiments on different applications.
CONDITIONAL VARIATIONAL AUTOENCODERS
The Conditional Variational Autoencoders (CVAE) introduced by Kingma et al. (2014) consist of a method for approximating to the true generative process that produced a dataset in a semi- cemi-supervised way. The CVAE assumes a generative process in which the data space X and two latent spaces Rt Z involved. The data space X consists in this case of the set of all possible images with a certain width height and color channels. The latent space R is associated with the variable of interest that have available in the labelled dataset (in this case the rotation), and the second latent space Z corresponds to an unobserved variable.
In the Variational Autoencoder (VAE) context (Kingma & Welling, 2014), family of parametric distributions used to approximate the posterior distributions over the latent spaces. The distribu- tion over the rotations R is denoted by Quzz and the distribution over the latent space Z is denoted by Q2 Such distributions are denominated encoding distributions and their parameters are cal- culated with the aid of neural networks. On the other hand parametric distribution over the data space X is denominated decoding distribution and denoted by |x
The generative model is obtained via variational inference by optimizing the parameters of the encoding and decoding distribution in order to maximize the evidence lower bound of the log- likelihood of the available observations (images and rotations). For the labelled dataset the log- likelihood of the image-rotation pairs is maximized. The evidence lower bound for the labelled dataset C(x,r) r) S log Pxx r) for a pair of image and rotation (z,r) E X R is given by
[(x,r) = EENGuIerr [log (x) + log (PR(r) + log (P2(=)) - log (#)) (I)
Where PR and Pz are the chosen prior probability density functions of the prior distributions over the corresponding latent spaces. In the case of the unlabelled dataset the evidence lower bound u(x) S log Px (z) takes the form
=(x)= = B [log (x) + log (Px(r)) + log (P2(z)) log
(2)
For our experiments, the latent space Z is chosen as d-dimensional Euclidean space Rd with normal posterior approximate with diagonal covariance (#z)0zz) and a standard normal distribution as prior Pz where jzz is the location parameter and az is a vector representing the diagonal of the covariance matrix.
The chosen latent space R that captures the 2D rotations of the object is a circle embedded in R² i.c. R S C 2 The encoding distribution Quunntn) over R is chosen as in Pérez Rey et al. (2019) where it corresponds to the transition kernel of the diffusion equation over the circle such that un and lR correspond to the location and scale parameters respectively. A uniform prior IP over R is selected over this latent space. For the decoding distribution (Px) we propose a Bernoulli X|aar
distribution over each of the pixels with parameter px-
According to Kingma et al. (2044) extra regression loss term is added to the CVAE objective. This loss term can be considered as the regression loss associated to the posterior over R that estimates the rotations. For an input image .i and its corresponding true rotation rı S1, the regression loss corresponds to the negative dot product between the estimated location of the posterior approximate pR (2(Z)) and the true rotation i.e.
S(r,,)) -ri pn(m))
(3)
3

---

Under review as a conference paper at ICLR 2019
n
azixri
C
(izag) 2I
Px
Ma
S(n. (g)
R
X₁
CVAE connections for training using pair of image and rotation. For labelled data, the regression loss takes part in training.
(Mz. Q
P
Px
LaIK
(ps
Xu
(b) CVAE connections for training using unlabelled images.
Figure 1: Training procedure for CVAE. Green trapezoids represent the encoding and decoding neural networks that calculate the parameters the corresponding distributions. Blue rectangles represent the sampling of latent variables according to the corresponding reparameterization tricks.
The maximization of the evidence lower bounds for the unlabelled and labelled dataset can be trans- lated into minimization problem by introducing minus sign. The minimization loss for the labelled and unlabelled datasets is then described by the following formula
U
I ta = Σ (C(a))))) S(n,,z)) Eu(xw)
(4)
a=1
TRAINING THE CVAE
In order to train a CVAE the labelled and unlabelled data need to be fed to the encoding and decoding distributions respectively. Figure shows how the parameters of the corresponding encoding and decoding distributions are estimated for the labelled and the unlabelled datasets. The sampling
from @x is done using the reparametrization trick from Kingma & Welling (2014) while
(uR.RR) done by performing a random walk over the circle as described in sampling from Q RI...
Pérez Rey et al. 2019).
In order train jointly the whole dataset of N L+U unlabelled and labelled images introduce for each available image I; an additional supervision value s; E {0, 1}. the datapoint is labelled then 8; and 8; 0 otherwise. For unlabelled images æ; an abritrary fixed label yi is provided. During training the loss function to be optimized corresponds to
N
N
I (12, = (C(a,,))) - S(ri,z;)) (1 si) Σ u(x.).
(5)
This loss function reduces to Equation but provides practical approach for implementing the training of labelled and unlabelled data.
4

---

Under review as a conference paper at ICLR 2019
4 DATASET GENERATION AND ARCHITECTURE
In order test the capabilities of our method we have restricted the use of our method a controlled setting in which the data annotations and the images provided can be generated arbitrarily. With this purpose we have focused on the use of dataset of 3D models from which we can generate arbitrary amount of rendered images of objects with known rotations.
ModelNer40 by Wu al. (014) dataset of 3D CAD models with object instances from forty categories. For the task of pose estimation, an aligned version was provided by |Sedaghat & Brox (2015) in which, for each category, all object instances were aligned to a certain frame reference. From the 3D models we have focused on the airplane class which has the most training instances (626 instances) and used a regular rendering procedures to generate the images for training and testing the CVAE:In the case of the planar 2D rotations of the object, we have generated the views by rendering images from 64 equally distanced angles within the interval [0, 2n/-
We train with VGG-like architecture for both the encoder and decoder neural networks similar to the work presented in Prokudin et al. (2018). The encoder networks, for the parameters of QRiz and @z consist of six convolutional layers with 24, 48, 48, 64 and 64 feature channels followed by a fully connected layer of 512 neurons. Each convolution and dense layer has rectifying linear unit activations followed by batch normalization. A final dense layer with linear activations is attached to each network to predict the pertaining parameters to be estimated. To enforce the restriction of positive scale parameters the log-scale parameters are estimated for each encoding density instead.
For the latent space Z = Rd the dimension is chosen as d = 50 with location and log-scale param- eters of the same dimension. For the space of rotations S C IR² the location parameter obtained from 2 neuron dense layer followed by Euclidean normalization. The log-scale parameter is the output of a neuron dense layer.
The decoding neural network consists of the reverse VGG network. First dense layer with 512 neurons is followed by a combination of up-sampling and convolutional layers with filters 64, 64, 48, 48, 24. The final convolution has sigmoid activation and returns arrays with same shape as input images.
EXPERIMENTS
In experiments have studied the performance of approach with respect two main factors. The amount of labelled data and the coverage of the rotations that are fed to the network.
Number of renders is was described in the previous section, for each object instance 64 images were generated at different rotations. During training, different subsets of these renders were used to test the effects of the angle coverage. The number of renders used per object instance were (64, 32, 16, 8}.
Label percentage For testing the effects of changing the amount of available labels we select uni- formly randomly certain percentage of object instances for which remove all the labels in their corresponding images. We further divide randomly the available labelled images into train(70%) and validation (30%). In each experiment we train for 200 epochs with early stopping and a tol- erance of 15 epochs for the loss. The model with the lowest validation loss is chosen as the final model for the given experiment.
EVALUATION
Evaluation of the rotation regression is done with respect to the rendered images from object in- stances within the test set of ModelNet40 aligned. For each 3D model instance set of 64 images at angles equally separated from the interval [0, 2n]] were generated. All of the trained models are evaluated with respect to the complete test set of 64 images per object instance. For each experi- ment three repetitions are trained and the relevant metrics are averaged and their standard deviation reported.
5

---

Under review as a conference paper at ICLR 2019
Medion renders
Medion 16 rende's
100
120
degrees
Medinn azsoltt erot in degrees
80
60
10
20
50
60
70
80
90
103
Persontage
Percantage
Median omro for G4 rendars
120
Mediaa abroauut drgrems
atseutt Segress
130
80
60
-00
10
30
40
90
60 70
800
90
100
20
30
40
50
60
80
100
Porcontage of Lab235 94
Parcancage
Figure 2: Results for the experiments trained on the regular angle dataset. Median error with respect to the percentage of labelled data, each experiment averaged over three runs and the corresponding standard deviation bars are shown. Each of the four plots corresponds to the results obtained by training models based on different amounts of rendered images per object instance.
Median Error We consider that the location parameter obtained for an image in the test dataset (*R(2)) E R S corresponds the rotation prediction for the labelled image 2; in the test dataset. The angular error between the label rotation E S and the prediction (R(())) is calculated as their geodesic distance in S
dgi (ri, (x() = arccos(ri pn(()))-
We calculate the median error across the images the test dataset and report it in degrees.
Accuracy For accuracy, the prediction is considered to be correct if its geodesic distance is smaller than a certain threshold i.e. dsi (r): (z()))) < 6. We consider three threshold values { 12 24
}o for our evaluation and report the percentages of correct classifications.
RESULTS
The results from Figure show the performance of the models with respect to the amount of su- pervision. A general trend can be distinguished in all plots i.e. increasing the amount of labelled images decreases the median enror and thus improves the models' performance. This is expected result since the models receive more signals that push the training into the correct regression values. It is important to notice that when relatively small amount of labels is missing (around 80%-90%) the method achieves similar performance. This means that the method can be used effectively in cases with moderately restricted labelling of data.
Focusing on the results with respect to the different number of renders per object instance and evaluated with respect the test images from, it is expected observe higher variance in the methods trained on less images. For such models, the training has been biased towards a restricted set of views which increases the variability in the predictions for the test

---

Under review as a conference paper at ICLR 2019
It is quite interesting find that even though the models trained 8 and 16 renders have a restricted amount of views from the object instances, their performance high when trained on highly labelled data. This means that under high supervision, the amount of example views from an object at different angles can be decreased significantly while maintining competitive performances.
Table shows a comparison of the performance of the method in the fully supervised setting. In such cases the CVAE has a good performance with the median error decreasing with respect to the number of renders provided during training expected. As was previously described, the results show that even with few representative samples of an object's rotation (with 8 renders), the method still achieves low median error and high accuracy (up to 87 = 16.3%). This performance still comes at the cost of high variance in the resulting models' performance.
Renders
Median Error (")
Acc@ (%%
Acc@ T2
(%)
Acc@ 2 (%)
8.5
87.0
77.6:127.3
65.4
16
2.6+-0.5
98.7+-0.3
96.8++0.8
87.6+3..3
32
1.8+1.0
99.120.2
98.3+0.6
93.9#4.2
64
1.2::0.3
99.410.0
99.0=E0.0
97.710.1
Table 1: Statistics for fully supervised method with 100% of labelled data evaluated on the test dataset. Median error (in degrees) and accuracy percentage at different threshold values is shown.
7 CONCLUSION
We have presented a novel approach to the task of semi-supervised pose estimation with limited trained data. The method is capable of receiving dataset of labelled and unlabelled images. As expected, model's performance increases with the amount of supervision provided to the network. Moreover, when just small percentage of the training labels are missing the model can achieve comparable performance to that of the fully supervised method. Some future steps to this work include the use of different sampling schemes for generated the rendered views of an object. New sampling schemes would also include arbitrary 3D rotations. Moreover, testing with more challeng- ing datasets such Pascal3D+ Xiang al. (2014) would provide point of comparison with other pose estimation methods..
REFERENCES
Markus Braun, Qing Rao, Yikang Wang. and Fabian Flohr. Pose-RCNN: Joint object detection and pose estimation using 3d object proposals. IEEE Conference Intelligent Transportation Systems, Proceedings, ITSC, pp. 1546-1551, 2016. doi: 10. 10.1109/TSC.2216..7797777
Taco S Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical CNNs. (3):1- 15, 2018. URL http:/t//aaavv..oo//////////.1./ abs/1801 10130
Tim R. Davidson, Luca Falorsi, Nicola De Cao, Thomas Kipf, and Jakub M. Tomezak. Hyperspher- ical Variational Into-Encoders. 2018. URL h/tp::////// httpg//aas/a000010000 10t./1898800091911
Carlos Esteves, Christine Allen-Blanchette. Ameesh Makadia, and Kostas Danilidis. Learning SO(3) Equivariant Representations with Spherical CNNs. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11217 LNCS(3)554-00, 2018a. ISSN 16113349, doi:
J4.
Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross-Domain 3D Equivariant Image Embeddings. 2018b. UBLiatt:://ariiv..rrr//bb
02716
Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in Homeomorphic Variational Auto Eiooo--nnoing.. 7 2018. httht::
// //aaxiiv..rr//ab///88..... wt
7

---

Under review as a conference paper at ICLR 2019
Matthew Johnson-Roberson Charles Barto, Rounak Mehta, Sharath Nittur Sridhar, Karl Rosaen, and Ram Vasudevan. Driving in the Matrix: Can virtual worlds replace human- -aman-generated an- notations for real world tasks? Proceedings IEEE International Conference Robotics and Automation, pp. 746-753, 2017. ISSN 10504729. doi: 10.1 10/////AA..1....99990.
Diederik P Kingma and Max Welling. Auto- Eitto-nn Variational Bayes. In International Confer- ence on Leurning Representations (ICLR), 2014. ISBN 3312.6114v10 doi: 10.1051/0004-633 201527329.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. Semi- Semi-Supervised Learning with Deep Generative Models. pp. 1-9, 2014. ISSN 10495258.
Tejas D. Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B. Tenenbaum. Deep Convolutional Inverse Graphies Network. pp. 1-10, 2015. ISSN 10897550. doi: 10.1063/1.4914077, URL
http://arxiv.orr http:/iwwwrrgabb////o/ba// abs/ 03167
Henry Li, Ofir Lindenbaum, Xiuyuan Cheng, and Alexander Cloninger. Variational Random Walk Autoencoders. 2019a. URL http::/tp:///tt httpp:://axx...og///bb////////// wttgabs///.......... 12724
Li-Jia Li, Kai Li, Fei Fei Li, Jia Deng, Wei Dong, Richard Socher, and Li Fei-Fei. ImageNet: Large-Scale Hierarchical Image Database Human trajectory forecasting View project Shrimp Project View project ImageNet: A Large-Scale Hierarchical Image Database. 2009 IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109//VR. 2009.5206843 URL/ttpp:: htmp:://wwwet /wwww wwweeeeeeaaaeeeeeeerrrr neteette.eeteeeeeee wwtp/tt..uu... www.iiiiinn//a /osssiinn22213111111
Zhigang Li, Yuwang Wang, and Xiangyang Ji. Monocular Viewpoints Estimation for Generic Ob- jects in the Wild. IEEE Access, 7:94321-94331, 2019b. doi: (00///ccess.0119.22224.
Shuai Liao, Efstratios Gavves, and Cees G. M. Snoek. Spherical Regression: Learning Viewpoints, Surface Normals and 3D Rotations on n-Spheres. 4 2019. URL
1904. 05404
Siddharth Mahendran, Haider Ali, and Rene Vidal. 3D Pose Regression Using Convolutional Neural Networks. Proceedings 2017 IEEE International Conference on Computer Vision Workshops, ICCVW 2017, 2018-Januu:217422182. 2018a. doi: 10.1 10.1109/CCV.22217..555
Siddharth Mahendran, Haider Ali, and Rene Vidal. A Mixed ClassfiiationnRReersssonn Framework for 3D Pose Estimation from 2D Images. pp. 1-16, 2018b. URL htLL:t//p///a/iiooott abs/
Eric Marchand, Hideaki Uchiyama, and Fabien Spindler. Pose Estimation for Augmented Reality: A Hands-On Survey. IEEE Transactions on Visualization and Computer Graphics, 22(12):2633- 2651, 2016. ISSN 10772626. doi: 10. 10..110V.....1..555111
Emile Mathieu, Charline Le Lan, Chris J. Maddison, Ryota Tomioka, and Yee Whye Tch. Hi- crarchical Representations with Poincaré Variational Auto-Encoders. 2019. URL http:
901 06033
Luis A. Pérez Rey, Vlado Menkovski, and Jacobus W. Portegies. Diffusion Variational Autoen- coders. 2019. URL httttttt://aaa orgpgas////........ 08991
Sergey Prokudin, Peter Gehler, and Sebastian Nowozin. Deep Directional Statistics: Pose Es- timation with Uncertainty Quantification. pp. 1-18, 2018. doi: arXiv:180.00330vv.. URL
attp::/arxiv. org. aatpg///b////// 1805 03430
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3D human pose estimation. Lecture Notes in Computer Science (including subseries Lec- ture Notes in Artificial Intelligence and Lecture Notes Bioinformatics). 11214 LNCS:765-782,
2018. ISSN 161 13349. doi: 10. 1111007998---00000124999
J46.
Nima Sedaghat and Thomas Brox. Unsupervised Generation of a View Point Annotated Car Dataset from Videos. Proceedings of the IEEE International Conference on Computer Vision, 2015 Inter 11):334411322, 2015. ISSN 15505499. doi: 10.1 20.110//.C2.221..5555.
8

---

Under review as a conference paper at ICLR 2019
Hao Su, Charles R. Qi, Yangyan Li, and Leonidas J. Guibas. Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views. Proceedings of the IEEE Inter- national Conference Computer Vision, 2015 Inter:2686-2694, 2015. ISSN 15505499. doi: 10.1 101100/2220005.3000
Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. Proceedings of the IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition, 07-12-Jume:1510-1555 2015. ISSN 10636919, doi: 10.109/CVVR.2005572877..
Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. Learning from Synthetic Humans. In 2017 IEEE Conference on Computer sion and Pattern Recognition (CVPR). volume 2017-Janua, pp. 4627 4627-4635, IEEE, 7 2017. ISBN 978-1-5386-0457-1 doi: 10. 10.1109922222221..9222 /rg/document. 80999757
URL http::/eeeepporr..eeee ieee.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3D ShapeNets: A Deep Representation for Volumetric Shapes. 2014. ISSN 10636919. doi: 10.1 101.:///CCVP.225.77288828. / www. wttp:///////...//6//22
URL http: //axivv.//aivvvaa//aa/av
Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond PASCAL: A benchmark for 3D object detection in the wild. 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014, pp. 75 82, 2014. doi: 00.1099/ACY2001466366000
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes. 2018. doi: 10. 1560/rsss 2018 811.v.199.


Reviews:
[{"review_id": "HkgKc3DJqS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the task of rotation estimation in a setting where both labelled and unlabelled examples are available for training. It proposes to learn a generative model of images (a VAE), where the ‘code’ is factored into a latent vector z and the object rotation r.  As in training a VAE, an image encoder that predicts the distribution over (z, r) and the generator are jointly trained, but with additional supervision on the distribution over r for the labelled examples.\n\nI think the overall idea of learning a disentangled generative model in a semi-supervised setting is simple and elegant, and could in principle help leverage unlabelled data. However,  I do have some concerns regarding the specific contributions of this work, and several reservations about the experiments reported, and would overall argue for rejection.\n\nConcerns:\n1) The central empirical result stated is that using this approach allows one to reduce amount of labelled data by 10-20 %. First, even if valid, this is not a very convincing reduction in the amount of supervision. However, I feel this claim is not well-established by the experiments:\n\n1a)  The paper should report a baseline with only using the loss in eqn 3 and only training the encoder (using various fractions of training data) to predict the rotation i.e. purely discriminative training without training a generative model. The current plots of performance vs fraction of labelled data don't mean much until compared to a similar plot for this baseline. The current results don't really highlight the importance of training the generative model or using the unlabelled data.\n\n1b) I think there are some inconsistencies in performances reported in Fig 2. I assume the test set is same despite different training data, because the paper states \"All of the trained models are evaluated with respect to the complete test set\". In this regard, I am puzzled why using 100% labelled data with 16 renders is significantly better than using 50% labelled data with 32 renders -- these should imply similar number of labelled examples, and more unlabelled ones in the former.\n\n2) While the discussion points to this, the paper would really benefit from having results in a real setting, in particular as pose estimation is a field with a lot of prior methods that have been shown to work in these settings. The current results are all in a setup with synthetic, unoccluded data, without background variation, equidistant camera uniformly sampled along a circle. The central idea of using a generative model would be much more difficult to operationalize in a realistic setting where these simplifying assumptions are not made, and I'd only be convinced about the applicability of the approach by results in that setting. As a possible setup, one case use many imagenet images in conjunction with labelled examples in PASCAL3D+ to try this approach.\n\n3) The overall approach maybe novel in context of pose estimation, but this idea of learning a disentangled generative model is not, and there are several papers which do so with varying amount of supervision e.g. see [1] below for similar ideas, and pointers. While some details here may vary, in context of these prior works, I'd view this paper as mostly applying well-established ideas to a new task.\n\n--\nIn addition to the above, I have a question regarding the training/testing data:\nQ: The dataset description only states data was randomly divided - was this random division at an image level, or model level i.e. could different renderings of the same model be in train and test set?\n--\n\n[1] Learning Disentangled Representations with Semi-Supervised Deep Generative Models, NIPS 2017. Siddharth et. al.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byxpp1v0cS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a semi-supervised approach to learn the rotation of objects in an image. The primary motivation is that for rotation estimation datasets may not always be fully labeled, so learning partially from labeled and partially for unlabeled is important. The approach is to use a CVAE with a supervised loss and an unsupervised loss and to jointly train the network. Limited experiments that show performance are presented.\n\nFirst, the paper solves a very interesting problem with potentially wide applications. The paper is reasonably well-written.\n\nUnfortunately, I don't believe that the contributions of the paper meet the standards of ICLR. I justify my opinion below. The experiments are also very weak.\n\n- While the high level goal of \"pose estimation\" is clear. Even after reading the paper multiple times, I did not understand the setting well. It appears like the paper looks at the problem of 2D orientation estimation of objects in images. However, this setting is restrictive and not very practical in reality. We mostly care about 3D pose estimation. It would have been good to see results on 3D rotations at the very least.\n\n- Contribution: It is unclear to me what the primary contribution(s) of the paper is. The entire section on CVAE's and losses are quite standard in literature. The interesting part is in combining the supervised and unsupervised parts of the method for the task for pose estimation. But in the end this is a simple weighted loss function (equation 5). So I wonder what is the novelty? What are the new capabilities enabled by this approach?\n\n- Related Work:\n\nImplicit 3D Orientation Learning for 6D Object Detection from RGB Images, ECCV 18\n\n\n- I would have loved to see a description of the differences in the loss functions (1) and (2). Perhaps this can help elevate the contribution more?\n\n- I also missed justification of why the particular design choice is suitable for this problem? Would direct regression using a simple CNN work better?\n\n- In equation (4), how are the two losses balanced?\n\n- The dataset generation part is just confusing. ModelNet40 is rendered but only 2D rotation is predicted? What does 2D rotation mean for a 3D object?\n\n- Could this method be tested on a dataset like dSprites (https://github.com/deepmind/dsprites-dataset) which has 3D rotations?\n\n- Regarding experiments: I was disappointed to see no comparisons with other approaches or even a simple baseline. A CNN that directly regresses orientation could help put the tables and plots in perspective.\n\nOverall, the problem is important (if lifted to 3D) with important applications. However, the paper does not say anything new about how to solve the problem and the experiments are weak. In its current state, I am unable to recommend acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1gUsOVx5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to employ conditional variational autoencoder (CVAE) to estimate the geometry of 2D rotations of objects given images partially labeled. Here, the label represents the geometry of the 2D rotation. The proposed method introduces two latent representation. z is the ordinal latent variable and r a latent representation for the rotations where the latent variable is defined in the 1-dimensional circle in R^2 so that it can naturally represent a hyperspherical latent space. \nThe construction of the proposed CVAE is straightforward. For labeled images, the (evidence lower bound of the) loglikelihood of the image-rotation pairs is maximized. For labeled images. For labeled images, the (evidence lower bound of the) loglikelihood of the images is maximized. \n\nThe decision of the reviewer of this paper is weak reject. The major reason is the lack of technical originality. The construction itself would be novel while each component (e.g., CVAE,  latent representation for the rotations, and semi-supervised construction of CVAE) have been already known.\n\nExperimental results are not surprising but show that the presented method is useful to some extent. In a sense, it is a bit disappointing that we need 50+% images needed to be labeled to achieve < 20-degree error. One interesting observation of this paper is that more labeled images give better results than giving greater number of renders. Expansion to 3D rotations would be a good challenge. \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of rotation estimation in 2D images. The method attempted to reduce the labeling need by learning in a semi-supervised fashion. The approach learns a VAE where the latent code is be factored into the latent vector and the object rotation. 

All reviewers agreed that this paper is not ready for acceptance. The reviewers did express promise in the direction of this work. However, there were a few main concerns. First, the focus on 2D instead of 3D orientation. The general consensus was that 3D would be more pertinent use case and that extension of the proposed approach from 2D to 3D is likely non-trivial. The second issue is that minimal technical novelty. The reviewers argue that the proposed solution is a combination of existing techniques to a new problem area. 

Since the work does not have sufficient technical novelty to compare against other disentanglement works and is being applied to a less relevant experimental setting, the AC does not recommend acceptance. 

Decision: Reject