Semi-supervised Pose Estimation with Geometric Latent Representations

1 INTRODUCTION . Pose estimation is the computer vision task in which the rotation of an object is estimated with respect to a certain frame of reference . The estimation of the position and the pose of an object within an image is an important challenge of computer vision and can be applied to a wide range of areas such as car navigation ( Braun et al. , 2016 ) , augmented reality ( Marchand et al. , 2016 ) and robot interaction ( Xiang et al. , 2018 ) . Several methods based on the use of convolutional neural networks have been proposed high performance . These approaches are divided into two : classification-based , where the possible rotations that can be predicted are discretized into predefined sets with different coarseness levels and regression-based where a continuous prediction is provided . Most of them are supervised methods which require large datasets with accurately labelled images and with a sufficiently good coverage of all possible rotations of each object instance . One of such datasets obtained from natural images and manually annotating objects in images is the Pascal3D+ dataset ( Xiang et al. , 2014 ) with about 22 thousand images from twelve categories . Compared to other datasets for other computer vision tasks such as classification ( with millions of images ) Li et al . ( 2009 ) the example data is scarce and the labelling process might be cumbersome and prone to errors . Thus , the need for a method to train with a limited amount of labelled data . To solve this problem there have been dataset generation approaches proposed to create new training labels and images ( Su et al. , 2015 ; Varol et al. , 2017 ; Johnson-Roberson et al. , 2017 ) from 3D models and inserting them in natural scenery . Other approaches are focused on reducing the supervision to auxiliary tasks indirectly related to the pose estimation such as in Esteves et al . ( 2018b ) .Prokudin et al . ( 2018 ) addresses the problem of label quality by including an uncertainty estimation in the rotation predictions . The work presented in this paper proposes a semi-supervised probabilistic method for pose estimation that incorporates the geometry of 2D rotations into the latent variable representations . The method is capable of using both labelled and unlabelled images from an object to improve the rotation regressions by performing the auxiliary task of reconstruction from the autoencoders . We have also investigated the effects of having a variable amount of labelled data in a controlled setting and variable coverage of the rotations . In the general setting of object pose estimation an object is located in the scene of interest and an image has been taken of it . The object has a certain rotation with respect to the a fixed frame of reference . The goal is to estimate the rotation that the object has undergone from the available image . In particular we are interested in the case were we might have a dataset in which the objects might be partially labelled . More formally , consider that we have a training dataset consisting of L labelled images with their corresponding rotation representations { ( xl , rl ) } Ll=1 in which xl ∈ X and rl ∈ R correspond to the l-th image and rotation , respectively , and U unlabelled images { xu } Uu=1 . How can we obtain a model for estimating the pose of an object based on an image , taking advantage of both datasets ? How does such a model perform with respect to the amount of supervision provided and the angle coverage ? 2 RELATED WORK . Current approaches for rotation estimation using neural networks can be divided into classification and regression . Classification methods discretize the possible rotations orientations of an object ( Su et al. , 2015 ; Li et al. , 2019b ; Tulsiani & Malik , 2015 ) while regression methods provide a more approach to the continuous nature of rotations ( Mahendran et al. , 2018a ; b ; Liao et al. , 2019 ; Prokudin et al. , 2018 ) . Since rotations have a particular topology , their structure needs to be taken into account for both classification and regression tasks . Mahendran et al . ( 2018a ) studies the performance of regression methods by using different rotation representations in a supervised setting . Liao et al . ( 2019 ) focus on the task of regression by representing rotations in n-spheres for different pose-related tasks . Prokudin et al . ( 2018 ) . In Esteves et al . ( 2018b ; a ) equivariant representations are used which capture the group structure of rotations . Their method is trained with the aid of spherical convolutional neural networks ( Cohen et al. , 2018 ) based on an auxiliary classification task , this method is unsupervised in the sense that it does not need labels for angles and is capable of estimating relative rotation among different views of an object . These supervised methods still require the use of labelled data in good quantity and quality . To circumvent these needs , we are interested in the semi-supervised and unsupervised approaches such as Esteves et al . ( 2018b ; a ) ; Rhodin et al . ( 2018 ) ; Kulkarni et al . ( 2015 ) . Similar to Kulkarni et al . ( 2015 ) , our focus is to use probabilistic models to capture the topological structure of rotations . In particular , within the Variational Autoencoders ( VAE ) Kingma & Welling ( 2014 ) context there is a recent growing interest in the study of latent variables with different geometries : hyperspherical ( Davidson et al. , 2018 ) , SO ( 3 ) ( Falorsi et al. , 2018 ) , hyperbolic ( Mathieu et al. , 2019 ) and arbitrary closed manifolds ( Pérez Rey et al. , 2019 ; Li et al. , 2019a ) . VAEs provide an unsupervised method for latent variable estimation and in principle could be used for capturing the underlying geometrical features for a dataset . The Conditional Variational Autoencoder ( CVAE ) ( Kingma et al. , 2014 ) allows to introduce supervision into the latent variable representations.Prokudin et al . ( 2018 ) used the CVAE as a supervised method for pose prediction and adding an uncertainty estimation for the input data . This method could not be directly extended to semi-supervised pose estimation without extra modifications . We use the previous work as a starting point for developing a semi-supervised method for pose estimation based on CVAEs with underlying geometrical latent variables for pose estimation . 3 METHODS . ROTATION REPRESENTATION It is important to choose an appropriate latent representation for the rotations that captures their topological properties . This has been discussed in Liao et al . ( 2019 ) and Mahendran et al . ( 2018a ) where several representations are described . In the case of 2D planar rotations along one fixed axis each rotation can be described in terms of the 1-dimensional circle S1 Liao et al . ( 2019 ) . Our method will has focus on the use of a probabilistic models ( Variational Autoencoders Kingma & Welling ( 2014 ) capable of encoding the input images in a circular latent space . To the best of our knowledge , there are three approaches to implement a variational autoencoder with a hyperspherical latent space . Firstly in Davidson et al . ( 2018 ) , they have focused on a method that is tailored specifically for hypersphere . On the other hand , Pérez Rey et al . ( 2019 ) and Li et al . ( 2019a ) propose methods for choosing arbitrary closed spaces . We have chosen to focus on using the work in Pérez Rey et al . ( 2019 ) to represent the rotations in S1 since it can provide arbitrary manifolds as latent space which can be used in further experiments on different applications . CONDITIONAL VARIATIONAL AUTOENCODERS The Conditional Variational Autoencoders ( CVAE ) introduced by Kingma et al . ( 2014 ) consist of a method for approximating to the true generative process that produced a dataset in a semi-supervised way . The CVAE assumes a generative process in which the data spaceX and two latent spacesR , Z are involved . The data space X consists in this case of the set of all possible images with a certain width height and color channels . The latent space R is associated with the variable of interest that we have available in the labelled dataset ( in this case the rotation ) , and the second latent space Z corresponds to an unobserved variable . In the Variational Autoencoder ( VAE ) context ( Kingma & Welling , 2014 ) , a family of parametric distributions is used to approximate the posterior distributions over the latent spaces . The distribution over the rotations R is denoted by QR|x and the distribution over the latent space Z is denoted by QZ|x , r . Such distributions are denominated encoding distributions and their parameters are calculated with the aid of neural networks . On the other hand a parametric distribution over the data space X is denominated decoding distribution and denoted by PX|z , r . The generative model is obtained via variational inference by optimizing the parameters of the encoding and decoding distribution in order to maximize the evidence lower bound of the loglikelihood of the available observations ( images and rotations ) . For the labelled dataset the loglikelihood of the image-rotation pairs is maximized . The evidence lower bound for the labelled dataset L ( x , r ) ≤ logPX×R ( x , r ) for a pair of image and rotation ( x , r ) ∈ X ×R is given by L ( x , r ) = Ez∼QZ|x , r [ logPX|z , r ( x ) + log ( PR ( r ) ) + log ( PZ ( z ) ) − log ( QZ|x , r ( z ) ) ] . ( 1 ) Where PR and PZ are the chosen prior probability density functions of the prior distributions over the corresponding latent spaces . In the case of the unlabelled dataset the evidence lower bound U ( x ) ≤ logPX ( x ) takes the form U ( x ) = E ( z , r ) ∼QZ×R|x [ logPX|z , r ( x ) + log ( PR ( r ) ) + log ( PZ ( z ) ) − log ( QZ×R|x ( z , r ) ) ] . ( 2 ) For our experiments , the latent space Z is chosen as a d-dimensional Euclidean space Z = Rd with a normal posterior approximate with diagonal covariance Q ( µZ , σZ ) Z|x , r and a standard normal distribution as prior PZ where µZ is the location parameter and σZ is a vector representing the diagonal of the covariance matrix . The chosen latent space R that captures the 2D rotations of the object is a circle embedded in R2 i.e . R = S1 ⊆ R2 . The encoding distribution Q ( µR , tR ) R|x over R is chosen as in Pérez Rey et al . ( 2019 ) where it corresponds to the transition kernel of the diffusion equation over the circle such that µR and tR correspond to the location and scale parameters respectively . A uniform prior PR overR is selected over this latent space . For the decoding distribution P ( pX ) X|z , r we propose a Bernoulli distribution over each of the pixels with parameter pX . According to Kingma et al . ( 2014 ) an extra regression loss term is added to the CVAE objective . This loss term can be considered as the regression loss associated to the posterior over R that estimates the rotations . For an input image xl and its corresponding true rotation rl ∈ S1 , the regression loss corresponds to the negative dot product between the estimated location of the posterior approximate µR ( xl ) and the true rotation i.e . S ( rl , xl ) = −rl · µR ( xl ) . ( 3 ) The maximization of the evidence lower bounds for the unlabelled and labelled dataset can be translated into a minimization problem by introducing a minus sign . The minimization loss for the labelled and unlabelled datasets is then described by the following formula J ( { xl , rl } Ll=1 , { xu } Uu=1 ) = − L∑ l=1 ( L ( xl , rl ) − S ( rl , xl ) ) − U∑ u=1 U ( xu ) . ( 4 ) TRAINING THE CVAE In order to train a CVAE the labelled and unlabelled data need to be fed to the encoding and decoding distributions respectively . Figure 1 shows how the parameters of the corresponding encoding and decoding distributions are estimated for the labelled and the unlabelled datasets . The sampling from Q ( µZ , σZ ) Z|xl , rl is done using the reparametrization trick from Kingma & Welling ( 2014 ) while the sampling from Q ( µR , tR ) R|xu is done by performing a random walk over the circle as described in Pérez Rey et al . ( 2019 ) . In order to train jointly the whole dataset ofN = L+U unlabelled and labelled images we introduce for each available image xi an additional supervision value si ∈ { 0 , 1 } . If the datapoint is labelled then si = 1 and si = 0 otherwise . For unlabelled images xi an abritrary fixed label yi is provided . During training the loss function to be optimized corresponds to J ( { xi , ri , si } Ni=1 ) = −si N∑ l=1 ( L ( xi , ri ) − S ( ri , xi ) ) − ( 1− si ) N∑ i=1 U ( xi ) . ( 5 ) This loss function reduces to Equation 4 but provides a practical approach for implementing the training of labelled and unlabelled data .

Summary: This paper presents a semi-supervised approach to learn the rotation of objects in an image. The primary motivation is that for rotation estimation datasets may not always be fully labeled, so learning partially from labeled and partially for unlabeled is important. The approach is to use a CVAE with a supervised loss and an unsupervised loss and to jointly train the network. Limited experiments that show performance are presented.