Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

We study the problem of learning to estimate the 3D object pose from a few labelled examples and a collection of unlabelled data . Our main contribution is a learning framework , neural view synthesis and matching , that can transfer the 3D pose annotation from the labelled to unlabelled images reliably , despite unseen 3D views and nuisance variations such as the object shape , texture , illumination or scene context . In our approach , objects are represented as 3D cuboid meshes composed of feature vectors at each mesh vertex . The model is initialized from a few labelled images and is subsequently used to synthesize feature representations of unseen 3D views . The synthesized views are matched with the feature representations of unlabelled images to generate pseudo-labels of the 3D pose . The pseudo-labelled data is , in turn , used to train the feature extractor such that the features at each mesh vertex are more invariant across varying 3D views of the object . Our model is trained in an EM-type manner alternating between increasing the 3D pose invariance of the feature extractor and annotating unlabelled data through neural view synthesis and matching . We demonstrate the effectiveness of the proposed semi-supervised learning framework for 3D pose estimation on the PASCAL3D+ and KITTI datasets . We find that our approach outperforms all baselines by a wide margin , particularly in an extreme few-shot setting where only 7 annotated images are given . Remarkably , we observe that our model also achieves an exceptional robustness in out-of-distribution scenarios that involve partial occlusion . The code is available at https : //github.com/Angtian/NeuralVS . 1 Introduction . Object pose estimation is a fundamentally important task in computer vision with a multitude of real-world applications , e.g . in self-driving cars or augmented reality applications . Current deep learning approaches to 3D pose estimation achieve a high performance , but they require large amounts of annotated data to be trained successfully . However , the human annotation of an object ’ s 3D pose is difficult and time consuming , therefore it is desirable to develop methods for learning 3D pose estimation from as few labelled examples as possible . A powerful approach for training models without requiring a large amount of labels is semi-supervised learning ( SSL ) . SSL mitigates the requirement for labeled data by providing a means of leveraging unlabeled data . Since unlabeled data can often be obtained with low human labor , any performance boost conferred by SSL often comes with low cost . This has led to a multitude of SSL methods , for example for image classification [ 18 , 44 ] , object detection [ 40 ] and keypoint localization [ 31 ] . However , only limited attention was devoted to SSL for 3D pose estimation . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this work , we introduce a semi-supervised learning framework for category-level 3D pose estimation from very few annotated examples and a collection on unlabelled images . Intuitively , our framework follows a spatial matching approach , in which we transfer the 3D pose annotation from the labelled examples to the unlabelled data . In general , matching-based approaches aim at transferring annotations between training examples by estimating correspondences between the respective images [ 29 , 2 , 26 , 15 , 8 , 45 , 19 , 27 ] . However , those prior works mainly focused on transferring 2D annotations , such as segmentation , part annotations , or keypoints , and mostly assume that the objects in the spatially matched images have a similar 3D pose . In this work , we explore the spatial matching of images with objects that have a largely varying 3D pose , in addition to nuisance variations in their shape , color , texture , context and illumination . 3D pose transfer through Neural View Synthesis and Matching ( NVSM ) . The intuition behind our approach is illustrated in Figure 1 using a single annotated training image ( but note that in practice we use several images ) . Our method proceeds in two steps : ( I ) Neural view synthesis and ( II ) Matching . During neural view synthesis , we start with ( 1 . ) extracting the feature map of a training image with a convolutional backbone ( CNN ) that was pre-trained for image classification . ( 2 . ) A mesh cuboid ( purple mesh ) is projected onto the extracted feature map using the ground-truth 3D pose annotation θ to sample the corresponding feature vectors at each visible mesh vertex ( indicated by red arrows ) . ( 3 . ) The mesh cuboid is subsequently rotated into a novel pose θ+ ∆θ . By rasterising the sampled feature vectors at the mesh vertices we synthesize a feature map of the object in a novel pose . Subsequently , ( 4 . ) we compute the feature maps of the unlabelled training images with the CNN . ( 5 . ) We spatially match the synthesized feature map those feature maps of the unlabelled data , resulting in a set of matching scores . ( 6 . ) Finally , we assign the 3D pose θ + ∆θ as pseudo-label to those images with highest matching scores ( marked in red ) . As prior works have shown [ 41 , 58 , 5 ] , the features in pre-trained convolutional networks are surprisingly reliable in spatial matching tasks as they are invariant to small variations in nuisance variables such as shape deformations , or changes in the object texture and illumination . This invariance property enables the spatial matching of annotated training images with unlabelled data across varying 3D poses . However , as we demonstrate in our experiments ( Section 4.3 ) , the features of the classification pre-trained CNN are not invariant to large 3D pose variations . Therefore , this pseudo-labelling process is initially only accurate for those objects in the unlabelled data that have a similar pose as the object in the annotated training image . This raises the need for improving the 3D pose invariance in the CNN features , to be able to annotate images with larger pose variability . Increasing 3D pose invariance in the feature extractor . We aim to increase the 3D pose invariance in the CNN using the pseudo-labelled data obtained from the NVSM process . The pseudo-labels enable us to extract the features vectors at corresponding mesh vertices in the data . To increase the 3D pose invariance in the CNN , we use a contrastive loss that encourages the feature vectors at a particular mesh vertex to become more similar to each other , while at the same time making them different from the features of other mesh vertices . This contrastive training improves the feature representation by making it more invariant to changes in the 3D pose , as well as to category-specific nuisance variables such as the object ’ s color , shape , illumination , while also reducing the ambiguity between nearby feature vectors in the feature map , which benefits the spatial matching quality . The improved CNN features enable us to increase the 3D pose scope ∆θ in the NVSM process , and hence to increase the 3D pose variability in the pseudo-labelled data . The pose diversity in the labelled data , in turn , enables us to improve the 3D pose invariance in the CNN feature extractor . We proceed to iterate between pseudo-labelling training data and training the CNN feature extractor , while continuously enlarging the 3D pose variation ∆θ of the synthesized views in NVSM . After each update of the CNN , we also update the feature representations at the mesh vertices by computing the moving average of the corresponding feature vectors in the pseudo-labelled images . In this way , the feature representation on the mesh cuboid is continuously adapting to the trained feature extractor . After the training , the trained CNN and mesh cuboid are used for 3D pose estimation . Given a test image , we first compute the feature map using the CNN and subsequently optimize the 3D pose of the mesh cuboid such that the distance between the features of the projected mesh vertices and the feature map are minimized . We evaluate our model at 3D pose estimation on the Pascal3D+ [ 53 ] and KITTI [ 13 ] datasets . Our approach proves highly effective in leveraging unlabeled data outperforming all baselines by a wide margin , particularly in an extreme few-shot setting where only 7 or 20 annotated images are given . Remarkably , we observe that our model also achieves an exceptional robustness in out-of-distribution scenarios that involve partial occlusion . 2 Related Work . Object pose estimation . Object pose estimation is an important and well studied computer vision task . [ 47 ] and [ 32 ] formulate the pose estimation problem as single step classification problem . In contrast , [ 25 , 35 , 59 ] solve the object pose estimation problem via a two-step approach , which involves keypoint detection and solving a Perspective-n-Point ( PnP ) process . Recently , [ 51 , 56 ] demonstrate the success of analysis-by-synthesis approaches for object pose estimation , which use a differentiable renderer to generate a synthesised image and estimate the object pose by minimizing a reconstruction loss . [ 49 ] extend the render-and-compare approach for pose estimation to the neural feature level , which significantly decreases the difficulty of the optimization process during pose estimation , while also improving the robustness . However , all of these approaches require a large amount of data with annotated 3D object pose during training , which is time consuming and expensive . In this work , we follow a matching-based approach , in which we transfer the 3D annotation form a few labelled images to a collection of unlabelled data , thus leading to a largely enhanced data efficiency . Spatial Matching . Spatial image matching aims at estimating the correspondence between objects in two different images . Traditional matching algorithms use features from corners [ 30 , 16 ] , blobs [ 28 ] , edges or lines [ 9 , 43 ] . Recent works [ 42 , 24 , 4 ] demonstrate the advantage of utilizing deep neural network features for image matching . [ 52 , 6 ] leverage the 3D structure of objects as additional constraint in the spatial matching process , for objects in similar poses . [ 2 ] demonstrate that neural feature matching can help reducing the amount of training data required for semantic part detection . In this work , we spatially match objects with largely varying 3D pose and leverage this ability for the semi-supervised few-shot learning of a model for 3D pose estimation . Semi-supervised Learning . Semi-supervised learning [ 10 , 60 , 48 ] aims at reducing the amount of annotations by utilizing unlabelled data . Self-training [ 55 , 46 , 33 ] is one of the most widely used semi-supervised learning approaches to create pseudo-labels , which are in turn used as supervision during the learning process . Current works show the success of self-training in classification [ 18 , 44 ] , detection [ 40 ] , and keypoint localization [ 31 ] . However , self-training for object pose estimation has not been well explored yet . In this work , we propose a semi-supervised pose estimation approach , that can be trained using very few annotations , while also being highly robust to partial occlusion . Robust Vision through Approximate Analysis-by-Synthesis . In a broader context , our work builds on and extends a recent line of work that follows an approximate analysis-by-synthesis approach to computer vision [ 49 ] , which formulates vision as an inverse rendering process on the level of neural network features . Several recent works demonstrate that approximate analysis-by-synthesis induces a largely enhanced generalization in out-of-distribution situations such as when objects are partially occluded in image classification [ 21–23 , 57 ] and object detection [ 50 ] , when images are modified through adversarial patches [ 20 ] , or when objects are viewed from unseen 3D poses [ 49 ] . Our work enables the learning of models for approximate analysis-by-synthesis with minimal supervision .

Summary: This paper studies the problem of estimating the pose of an object, given an image of it. The proposed method aims to use as few labeled samples as possible, and generate pseudo-labels for a large collection of unlabelled data during training. The method works in an EM-fashion in which two stages (neural view synthesis and matching) are optimized by alternating.   In the first stage, the method extracts a feature map from the input annotated image and shapes it as a mesh cuboid where each vertex is assigned a feature vector depending on its position. The mesh is then rotated at regular angle intervals and rasterized into the image plane. In the matching part, features from unlabeled images are extracted and matched with rotated view rasterizations using a cosine similarity metric.   The experimental evaluations show that the proposed method is able to learn the pose prediction task from as few as 7 annotated images during training. The method mostly outperforms the baselines in comparisons against them.