Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose
Authors: Angtian Wang
Shenxiao Mei
Alan Yuille
Adam Kortylewski
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Pose Estimation
Semi-supervised Learning
Few-shot Learning
Neural Rendering
Analysis-by-Synthesis.

Abstract: We study the problem of learning to estimate the 3D object pose from a few labelled examples and a collection of unlabelled data. Our main contribution is a learning framework, neural view synthesis and matching, that can transfer the 3D pose annotation from the labelled to unlabelled images reliably, despite unseen 3D views and nuisance variations such as the object shape, texture, illumination or scene context. In our approach, objects are represented as 3D cuboid meshes composed of feature vectors at each mesh vertex. The model is initialized from a few labelled images and is subsequently used to synthesize feature representations of unseen 3D views. The synthesized views are matched with the feature representations of unlabelled images to generate pseudo-labels of the 3D pose. The pseudo-labelled data is, in turn, used to train the feature extractor such that the features at each mesh vertex are more invariant across varying 3D views of the object. Our model is trained in an EM-type manner alternating between increasing the 3D pose invariance of the feature extractor and annotating unlabelled data through neural view synthesis and matching. We demonstrate the effectiveness of the proposed semi-supervised learning framework for 3D pose estimation on the PASCAL3D+ and KITTI datasets. We find that our approach outperforms all baselines by a wide margin, particularly in an extreme few-shot setting where only 7 annotated images are given. Remarkably, we observe that our model also achieves an exceptional robustness in out-of-distribution scenarios that involve partial occlusion.

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose
Angtian Wang
Shenxiao Mei
Alan Yuille
Adam Kortylewski
Johns Hopkins University Maryland, MD 21218, USA
(angtianvang, smei5, ayuillel, akortyl1)@jhu.eed edu
Abstract
We study the problem of learning estimate the 3D object pose from a few labelled examples and a collection of unlabelled data. Our main contribution is a learning framework, neural view synthesis and matching, that can transfer the 3D pose annotation from the labelled to unlabelled images reliably, despite unseen 3D views and nuisance variations such as the object shape, texture. illumination or scene context. In our approach, objects are represented as 3D cuboid meshes composed of feature vectors at each mesh vertex. The model is initialized from a few labelled images and is subsequently used to synthesize feature representations of unseen 3D views. The synthesized views are matched with the feature representations of unlabelled images generate pscudo-labels of the 3D pose. The pscudo-labelled data is, in turn, used to train the feature extractor such that the features at each mesh vertex are more invariant across varying 3D views of the object. Our model is trained in an EM-type manner alternating between increasing the 3D pose invariance of the feature extractor and annotating unlabelled data through neural view synthesis and matching. We demonstrate the effectiveness of the proposed semi-supervised learning framework for 3D pose estimation on the PASCAL3D+ and KITTI datasets. We find that our approach outperforms all baselines by a wide margin, particularly in extreme few-shot setting where only 7 annotated images are given. Remarkably, we observe that our model also achieves an exceptional robustness out-of-distribution scenarios that involve partial occlusion. The code is available at https://github.com com AnggiannNeurallV
1 Introduction
Object pose estimation is fundamentally important task in computer vision with multitude of real-world applications, e.g. in self-driving cars or augmented reality applications. Current deep learning approaches to 3D pose estimation achieve a high performance, but they require large amounts of annotated data to be trained successfully. However, the human annotation of object's 's 3D pose difficult and time consuming, therefore it is desirable to develop methods for learning 3D pose estimation from as few labelled examples as possible.
powerful approach for training models without requiring large amount of labels is remi-supervised learning (SSL). SSL mitigates the requirement for labeled data by providing means of leveraging unlabeled data. Since unlabeled data can often be obtained with low human labor, any performance boost conferred by SSL often comes with low cost. This has led to multitude of SSL methods, for example for image classification [18, 44], object detection [40] and keypoint localization [31]. However, only limited attention was devoted to SSL for 3D pose estimation.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(1) Neural View Synthesis
(1.) Extract feature
(2.] Semple feeture for
(3) 3D rotate mesh cubold and rasterize feature map
Labelled training
CNN
Mesh cuboid novel pose 0 + 60
Syrthesized feature map
cuboid pose
(I) Matching
(4) maps
symthenized map with unlabelled maps
(6.) Imagis with paeudo- label ae
Unlabelled training images
Feature Maps
Computed matching scores
51 0.9
S2 0.2
-3-0.1 54
Figure 1: Illustration of how we transfer the 3D pose annotation from one training image to a set of unlabelled images. A detailed description of this process is given in the introduction section.
In this work, we introduce a semi semi-supervised learning framework for category-level 3D pose esti- mation from very few annotated examples and collection on unlabelled images. Intuitively, our framework follows spatial matching approach, in which we transfer the 3D pose annotation from the labelled examples to the unlabelled data. In general, matching- based approaches aim at transfer- ring annotations between training examples by estimating correspondences between the respective images [29, 2, 26, 15, 8, 45, 19, 27]. However, those prior works mainly focused on transferring 2D annotations, such segmentation, part annotations, or keypoints, and mostly assume that the objects in the spatially matched images have similar 3D pose. In this work, we explore the spatial matching of images with objects that have a largely varying 3D pose, addition to nuisance variations their shape, color, texture, context and illumination.
3D pose transfer through Neural View Synthesis and Matching (NVSM). The intuition behind our approach is illustrated in Figure I using single annotated training image (but note that in practice we use several images). Our method proceeds in two steps: (I) Neural view synthesis and (II) Matching. During neural view synthesis, we start with (1.) extracting the feature map of training image with a convolutional backbone (CNN) that was pre-trained for image classification. (2.) mesh cuboid (purple mesh) is projected onto the extracted feature map using the ground-truth 3D pose annotation 0 to sample the corresponding feature vectors each visible mesh vertex (indicated by red arrows). (3.) The mesh cuboid is subsequently rotated into novel pose 0 + A0. By rasterising the sampled feature vectors at the mesh vertices we synthesize feature map the object in a novel pose. Subsequently, (4.) compute the feature maps of the unlabelled training images with the CNN. (5.) We spatially match the synthesized feature map those feature maps of the unlabelled data, resulting in set of matching scores. (6.) Finally, assign the 3D pose 0 + A0 pscudo-label to those images with highest matching scores (marked in red).
As prior works have shown [41, 58, 5], the features in pre-trained convolutional networks are surprisingly reliable in spatial matching tasks as they are invariant small variations in nuisance variables such as shape deformations, invariance property enables the spatial matching annotated training images with unlabelled data across varying 3D poses. However, we demonstrate in our experiments (Section 4.3), the features of the classification pre-trained CNN are not invariant to large 3D pose variations. Therefore, this geeudo-labelling process is initially only accurate for those objects in the unlabelled data that have similar pose as the object in the annotated training image. This raises the need for improving the 3D pose invariance in the CNN features, to be able to annotate images with larger pose variability.
changes in the object texture and illumination. This
Increasing 3D pose invariance in the feature extractor. We aim to increase the 3D pose invariance in the CNN using the pseudo-labelles data obtained from the NVSM process. The pseudo-labels enable us to extract the features vectors at corresponding mesh vertices in the data. increase the 3D pose invariance in the CNN, we use contrastive loss that encourages the feature vectors at particular mesh vertex to become more similar to each other, while at the same time making them different from the features of other mesh vertices. This contrastive training improves the feature

---

representation by making it more invariant to changes in the 3D pose, as well as to category -ppeciffc nuisance variables such as the object's 's color, shape, illumination, while also reducing the ambiguity between nearby feature vectors in the feature map, which benefits the spatial matching quality.
The improved CNN features enable to increase the 3D pose scope A in the NVSM process, and hence to increase the 3D pose variability in the pseudo-labelled data. The pose diversity in the labelled data, in turn, enables us improve the 3D pose invariance in the CNN feature extractor. We proceed to iterate between pseudo-labelling training data and training the CNN feature extractor, while continuously enlarging the 3D pose variation At of the synthesized views in NVSM After each update of the CNN, we also update the feature representations at the mesh vertices by computing the moving average of the corresponding feature vectors in the pseudo-labelled images. In this way, the feature representation on the mesh cuboid is continuously adapting to the trained feature extractor.
After the training, the trained CNN and mesh cuboid are used for 3D pose estimation. Given test image, we first compute the feature map using the CNN and subsequently optimize the 3D pose of the mesh cuboid such that the distance between the features of the projected mesh vertices and the feature map are minimized. We evaluate our model at 3D pose estimation on the Pascal3D+ [53] and KITTI [13] datasets. Our approach proves highly effective in leveraging unlabeled data outperforming all baselines by a wide margin, particularly in an extreme few-shot setting where only 7 or 20 annotated images are given. Remarkably. we observe that our model also achieves an exceptional robustness in out-of of-distribttion scenarios that involve partial occlusion.
2 Related Work
Object pose estimation. Object pose estimation an important and well studied computer vision task. [47] and [32] formulate the pose estimation problem as single step classification problem. In contrast, [25, 35, 59] solve the object pose estimation problem via a two-step approach, which involves keypoint detection and solving a Perspective-n-Poin (PnP) process. Recently, [51, 56] demonstrate the success of maalysis-by-synthees
approaches for object pose estimation, which use a differentiable renderer to generate synthesised image and estimate the object pose by minimizing reconstruction loss. [49] extend the sonder-and-oomperr approach for pose estimation to the neural feature level, which significantly decreases the difficulty of the optimization process during pose estimation, while also improving the robustness. However, all of these approaches require a large amount of data with annotated 3D object pose during training, which is time consuming and expensive. In this work, we follow a matching- matching-based approach, in which we transfer the 3D annotation form few labelled images to collection of unlabelled data, thus leading to largely enhanced data efficiency.
Spatial Matching. Spatial image matching aims at estimating the correspondence between objects in two different images. Traditional matching algorithms use features from corners [30, 16), blobs [28], edges lines 19, 43]. Recent works [42, 24, 4] demonstrate the advantage of utilizing deep neural network features for image matching. [52, 6] leverage the 3D structure of objects as additional constraint in the spatial matching process, for objects in similar poses. [2] demonstrate that neural feature matching can help reducing the amount of training data required for semantic part detection. In this work, we spatially match objects with largely varying 3D pose and leverage this ability for the semi- semi-supervised few-shot learning of a model for 3D pose estimation.
Semi-supervised Learning. Semi-supervised learning [10, 60, 48] aims at reducing the amount of annotations by utilizing unlabelled data. Self-training [55, 46, 33] is one of the most widely used semi-supervised learning approaches to create pseudo-labels, which are in turn used as supervision during the learning process. Current works show the success of self-training in classification// 18, 44], detection[400 [00]. and keypoint localization 1]. However, self-training for object pose estimation has not been well explored yet. In this work, we propose a semi-supervised pose estimation approach, that can be trained using very few annotations, while also being highly robust to partial occlusion.
Robust Vision through Approximate
bySnthis-b.-SSttsssss... In broader context, our work builds sendlsis-b--sythheee approach to
on and extends recent line of work that follows an approximate computer vision [49], which formulates vision as an inverse rendering process on the level of neural network features. Several recent works demonstrate that approximate analysis-by by-stthssss induces a largely enhanced generalization in out-of-distribution situations such as when objects are partially occluded in image classification [21-23, 57] and object detection [50], when images are modified
3

---

through adversarial patches [20], or when objects are viewed from unseen 3D poses [49]. Our work enables the learning of models for approximate analysis-by-sye by-synthesis with minimal supervision.
3 Method
In this section, we describe our approach for the semi-supervised learning of 3D pose from a few labelled examples and a collection of unlabelled data. The central part of our framework is a novel method for generating pseudo annotations of the 3D pose in unlabelled images through synthesizing novel views of an object on the level of neural network features (Section 3.1) and matching those synthesized views to the features of unlabelled images (Section 3.2). We discuss how this method can be integrated into a semi-supervised learning pipeline for 3D pose estimation in Section 3.3.
3.1 Neural View Synthesis
A key requirement for an efficient cemi-supervised learning is the ability to generate reliable pseudo labels for the unlabelled data using only few labelled images. The challenge when generating pseudo- labels of an objects 3D pose is that it requires generalizing to situations where an object is depicted in a previously unseen 3D pose, which has proven to be a challenging problem for discriminative models [1, 37]. In this work, we generate meeudo-lbbels by synthesizing novel views of an object on the level of neural network feature activations. In the following, we explain the details of this process based on single labelled training image, and we discuss later how this is extended to multiple images.
Feature extraction. Our approach starts with an image / and the corresponding 3D pose annotation 0 E RR We use a convolutional neural network V to extract a feature map F (r( E # with C being the number of channels. To start with, we use an ImageNet [12] pre-trained neural network as feature extractor and we discuss how the feature extractor is trained in Section 3.3.
Mesh cuboid and feature sampling. We aim to synthesize the object in the feature map F' from a novel view. To achieve this, we represent the object as a 3D cuboid mesh composed of a set of vertices r {a, | 1, ...... R} and feature vectors at each mesh vertex Σ = fo, E R r 1...., R). The mesh cuboid serves as 3D representation of the object in the feature map and allows us to change the 3D pose of the object as described in the following. Given an anchor image / and the corresponding feature map F, we sample the mesh features Σ from the feature map using the ground truth pose annotation O. For this, we project the mesh vertices I into the feature map by multiplying the vertices with a weak perspective projection matrix Pg:
or F(Po x.y).
where o, is the sampled feature vector at the position of the r-th projected mesh vertex. Note that the mesh cuboid does not represent the object shape in detail and therefore also feature vectors at the background will be sampled. But intuitively the model will learn that some features are coherent across views and therefore will be able focus more the foreground region.
Synthesizing feature maps in novel views with rasterisation. Given the mesh cuboid and the sampled feature vectors, we can synthesize novel views of the object, by rotating the mesh cuboid into a novel 3D pose 0' A0. Subsequently, we generate feature map of the object this novel view through rasterisation:
FØ = RT, E HXWXC
(2)
where RR denotes the rasterisation process of the cuboid mesh I' with the vertex features Σ in the 3D pose 8'. Note that rasterisation a standard technique in computer graphics to draw 3D models into 2D images [38]. However, instead of drawing images with RGB pixels, in our usecase feature maps. The rasterisation resolves two problems. First, when projecting a 3D mesh into an image, some of the vertices will become occluded due to self-occlusion, and rasterisation accounts for this effect. Second, some pixels in an image might not be covered by a projected vertex and to fill these holes in the image, rasterisation interpolates between the features of the mesh vertices.
draw
3.2 Spatially Matching Synthesized Views to Unlabelled Images
We generate pseudo-labels of an object's 3D pose by spatially matching synthesized object views For to the set of unlabelled images D = {Li...., }.r. During matching, we compute the similarity

---

Anchor Image
Synthesis Pose B+46
With ImageNet Pretrained Meani 48 error 41.5
After Semi-Superviied Training
Mean[| error 8.7
-40
4B errori 140.8
142.9
erter
Groundtruth Pose θ
-20
JAB error
2.2
10.3
140 craaii: 0.1
1.4
1A8 error : 4.3 8.4
Pose Distribution
+20
|60 error :: 29.7
51.1
168 error 9.4
+40
[68 error]: 14.7
14.2
|A0 error|: 4.6
11.5
Figure 2: Spatial matching results of synthesized views. We synthesize novel views the anchor image using the ground-truth pose 9 and varying azimuth angles AØ (here sample A along azimuth). We spatially match the synthesized views with all images in the data and retrieve those with highest matching scores. We also report the pose difference of the retrieved images. The CAD model is only used for visualization purposes and is not used in our method. Note how the retrieval results much accurate after proposed semi-supervised training, compared to when using pre-trained features.
between For and the feature map of an unlabelled image Fm (Im) as the sum of the cosine distances -(-.-) between their corresponding feature vectors:
S(Fo,FFl 1 E d(Fo w), Fm Fm (h, w))).
(3)
HW
We retrieve those images with highest spatial matching similarities that greater than a pre-defined threshold and pseudo-label them with the 3D pose of the mesh cuboid 0'.
The role of pose invariance on the spatial matching quality. The main challenge when spatially matching synthesized view Fe+ to a set of unlabelled images is to achieve reliable spatial matching result when &O is large, and hence when the novel view very different from the original 3D pose of the object in the labelled training image. Figure 2 illustrates this problem. In our work, we start with an ImageNet pre- trained CNN as feature extractor W. We observe that these features have some invariance to 3D pose and other nuisances, therefore the retrieved images consistent with the pscudo-labelled pose when AØ is relatively small. However, with an increased value of Ae the objects in the retrieved images inconsistent with the 3D pose of the mesh cuboid (refer to section 4.3 for more details). Therefore, we keep AA small the initial neural view synthesis and matching process. This allows collect additional data with reliable 3D pose annotation, although with limited pose variability. To increase the pseudo annotation to larger unseen pose angles, we subsequently train the feature extractor to become more pose invariant, using the labele-l data as described in the following section.
3.3 Semi-Supervised Learning of 3D pose
Neural view synthesis (Section 3.1) and matching (Section 3.2) process enables us annotate the 3D pose in unlabelled data, but this requires features that are invariant to 3D pose. However, the training of feature extractor with higher 3D pose invariance requires amount of annotated data. To resolve this chicken-and and-egg problem, we iterate between annotating data and training the feature extractor in Expectation-Mann Sat.aattioo-aaaiiiiitito-o--y--- manner.
Training the feature extractor. After first pass of neural view synthesis and matching with an ImageNet pre-trained feature extractor V, we obtain a set of annotated data PP {(T,, 0,) 1,..., N) that is, turn, used to improve the invariance to viewpoint changes in the feature extractor with contrastive training strategy [3, 11]. Specifically, given data IP and the mesh cuboid I' use
5

---

a contrastive loss to maximize the feature similarity between the feature vectors that correspond to the same vertex different images:
R Σ 1 x,), x,))]
(F, -
(4)
Here F. = (T.)) and P3 (T,) are feature maps of two images with pose annotations 0; and 0z. At the same time, we minimize the similarity between feature vectors of different vertices:
R E d(P.(oo 2,), xr-))
L.... - Σ
(5)
During training, the overall loss is computed over all combinations of images in I.. After training the feature extractor, we can resort to an improved pseudo-labelling through neural view synthesis and matching, which will in turn enable to improve the feature extractor.
Throughout the EM-type learning process, we gradually increase the pose difference AO of the synthesized views from the annotated training data. Figure 1 illustrates the improved retrieval performance after several EM-type iterations of NVSM and training the feature extractor using only the single depicted training image.
Semi-supervised learning with multiple annotated images. While our approach very data efficient, we observe that single training example is not sufficient to bootstrap the full learning process successfully, because objects in images not only vary due their 3D pose, but also in terms of nuisance factors such as their texture, shape, context or illumination conditions. To generalize our approach multiple images, we initialize the mesh features Σ by averaging over the whole set of annotated training images A {(Ta 0a) 1,.... A}:
o, = Σ Fa 2T).
(6)
The training of the feature extractor naturally extends to a training with multiple annotated images, the only thing that changes the size of the pseudo-annotated dataset IP. However, note that after updating the feature extractor, the mesh features Σ also need to be updated. After the first training iteration, we leverage the pseudo-labelles labelled dataset compute the mesh features Σ, since is multiple times larger compared to the annotated dataset A. To compute the update at a reasonable computational cost, we use a moving average update [3]:
c+1 (1 a) of + Er.(Pa, :rr)
where of the current mesh feature at time step t and + the updated feature. The moving average update enables us update the mesh features while training the feature extractor, and hence without requiring to process the whole set P again, after training the feature extractor. In practice, we observe that this improves the runtime of the training, while neither hurting benefiting the overall performance significantly.
3D pose estimation with the mesh cuboid. After training the model, we can leverage the feature extractor and the mesh cuboid directly to estimate the 3D pose of an object. Following related works on amalysi-by-ssynthn by-synhhesis for face analysis [7] or pose estimation with neural network features [49], we implement 3D pose estimation in a render-and-compare manner. Specifically, given test image IT we first compute its feature representation using the trained backbone Fr (I(T). Subsequently, we optimize the 3D pose 0 of the mesh cuboid with learned mesh features Σ, such that the synthesized feature map Fe most closely resembles Fr, iec. such that S(Fr, F6) is minimized.
4 Experiments
We describe the experimental setup in Section 4.1. Subsequently, we study the performance of our approach at semi-supervised few-shot learning of 3D pose estimation in Section 4.2. We diagnose the how the spatial matching quality evolves during training in Section 4.3.

---

4.1 Experiment Setup
Evaluation. The task 3D object pose estimation involves the prediction of three rotation parameters, ic. azimuth, elevation, in-plane rotation, of an object relative to the camera. We follow the evaluation protocol used in previous works [59, 49] and assume the object scale and center are given, and measure the pose estimation error between the predicted rotation matrix and ground-turth rotation
matrix: A (Rpred; Rgt) |log m ( (Rires
We report accuracy w.r.t two thresholds T and is
v2
as well as, the median error of the prediction.
Dataset. We report 3D pose estimation performance for all methods on the PASCAL3D+ dataset [54] and the KITTI dataset [14]. For Pascal3D+, we evaluate 6 vehicle categories (acroplane, bicycle, boat, bus, car, motorbike), which have a relatively evenly distributed pose regarding the azimuth angle. We evaluate few-shot semi- suneriiiee learning in 3 few-shot settings which train 7. 20, and 50 annotated images for each category respectively. The annotated images are selected such that they are spread around the pose space (see supplementary for details) and the remaining images sampled randomly. For the KITTI dataset, we utilize the 3D detection annotation of the car category to crop the objects such that they are located in the image center and compute the 3D object pose. Following the official KITTI protocol, we split the dataset such that it contains 2047 training images and 681 testing images, including 411 fully visible 201 partially occluded and 69 largely occluded images according to the occlusion level annotation.
Training Setup. We use an ImageNet [12] pre-trained ResNet50 [17] feature extractor. The dimensions of the cuboid mesh I are defined such that for each category most of the object area is covered. We sample A at fixed distances of 10 degree during neural view synthesis. During training we continually increase the pose range by 10 degrees after each iteration of training the feature extractor until the whole pose space is covered. For each sampling step, we pseudo label between 20 to 50 images depending on the category and the similarity threshold T = 0.9. In total, we train the whole model by alternating 100 times between neural view synthesis and matching (NVSM) and training the feature extractor, which takes around 5 hours per category on a machine with 4 RTX Titan GPUs. We implement our approach in PyTorch [34] and apply the rasterisation implemented in PyTorch3D [39].
Baselines. We evaluate StarMap [59] our baseline as it is one of the state-of- of-tth--rttttt supervised approaches to 3D pose estimation. Moreover, we evaluate NeMo [49] which is a recently proposed approach for 3D pose estimation that is robust and data efficient. For NeMo, we use the same single mesh cuboid as our method is using. We also implement a bascline that formulates the object pose estimation problem
classification task described in [59]. For the classification approach, we evaluate both a category specific (Res50-Spec) and a non-specific (Res50-Gene) ResNet50 classifier. The former formulates the pose estimation task for all categories as one single classification task, whereas the latter learns one classifier per category.
Note that all baselines are evaluated using semi-supervised few-shot protocol. This means we use 7, 20, and 50 annotated images for training. In order to utilize the unlabelled images, we use common labellil strategy for all baselines. Specifically, annotated images, and use the trained model to predict a pseudo-label for all unlabelled images in the training set. We keep those pseudo-labels with confidence threshold 7 = 0.9, and utilize the pseudo-labeled data as well as the annotated data to train the final model.
first train model on the
4.2 Semi-Supervised Few-Shot 3D Pose Estimation
Pascal3D+. Table shows the performance ouf our approach and all baselines at semi-supervised few-shot 3D pose estimation on the PASCAL33+ dataset. All models are evaluated using 7, 20, and 50 (per class) training images with annotated 3D pose and collection of unlabelled training data (as described in Section 4.1). The Resnet50 classification baselines achieves reasonable performance. Notably, the category classifier trained for all categories, suggesting that the latter benefits from the additional data of all categories. particular, for the extreme low data setting using only 7 annotated examples, the model can share information across categories which greatly improves the accuracy by 5.5%. Remarkably, StarMap performs worse compared to the simple classification baseline. This highlights the relevance the semi-supervised few-shot setting as it shows that current state-of-the-art models for 3D pose
classifier performs significantly lower compared the
7

---

Table 1: Few-shot pose estimation results on PASCAL3D+. We indicate the number of annotations
during training for each category and evaluate all approaches using Accuracy (in percent, higher better) and Median Error (in degree, lower better). We also include the fully supervised baseline [49] (Full Sup.) which is trained from hundreds of images per category.
Metric
ACO= t
ACC r
MedErr +
Num Annos
7
20
50
Mean
20
50
Mean
20
50
Mean
Res5O-Gene
36.1
45.2
54.6
45.3
14.7
25.5
34.2
24.8
39.1
26.3
20.2
28.5
Res50-Spec
29.6
42.8
50.4
40.9
13.3
23.0
29.3
21.9
46.5
29.4
23.0
32.9
StarMap
30.7
35.6
53,8
40.0
4.3
7.2
19.0
10.1
49.6
46.4
27.9
41.3
NeMo
38.4
51.7
69.3
53.1
17.8
31.9
45.7
31.8
60.0
33.3
22.1
38.5
Ours
53.8
61.7
65.6
60.4
27.0
34.0
39.8
33.6
37.5
28.7
30.1
Full Sup. [49]
89.3
66.7
7.7
Table 2: Few-shot pose estimation results on the KITTI dataset at different levels of partial occlusion. We report results using prediction accuracy and median error.
Occ level
Fully visible
Partially occluded
Largely Occluded
Eval Metric
Mean
Num Annos
20
50
7
20
50
7
20
50
NeMo
34.3
83.9
89.8
14.9
58.2
74.6
4.3
27.5
30.4
58.5
ACC; f
Ours
84.2
94.6
97.6
68.2
88.6
92.5
52,2
60.9
63.8
86.1
NeMo
17.3
74.9
81.5
4.5
37.8
61.7
0.0
7.2
11.6
45.8
ACC t
Ours
23.2
81.1
88.3
18.7
81.6
82.1
16.4
39.1
42.0
60.0
NeMo
64.1
5.9
5.6
13.2
99.9
59.8
50.0
32.6
MedErr
Ours
20.3
8.1
4.1
24.0
12.2
5.3
27.0
13.3
12.2
12.4
estimation are designed for data rich training and cannot simply be extended to semi-supervised training setups. This deficit of generalizing from few annotated data is particularly prominent in the high accuracy evaluation 18' In contrast StarMap, the NeMo baseline shows higher data efficiency. This can be attributed to the generative nature of the model which enables it to become more data efficient compared to the discriminatively trained baselines. However, NeMo still requires at least 50 annotated images per category to perform well, while having significantly reduced performance in the training settings with fewer data.
Our proposed approach gives significantly higher accuracy compared to all baselines in the low data regime using 7 and 20 annotated images, while being competitive in the higher data regime using 50 annotated examples. Specifically, our model achieves exceptionally high performance using 7 annotated images improving the accuracy by 15.4% n and 9.2% 18'
6
KITTI and occlusion robustness. Table 2 shows the results on the KITTI dataset. Using the occlusion annotation in the test data we also study the robustness of the models under occlusion. We compare our approach to the NeMo baseline only, as it is the most competitive model (see original paper [49] for comparisons). Moreover, StarMap cannot be trained the KITTI dataset, because it requires keypoint annotations, which are not provided the data. Notably, our approach outperforms NeMo in all experiments by a wide margin. The most prominent performance gain is observed in the extreme few-shot setting of using 7 images only. A notable performance increase can be observed when increasing the annotated data 20, while more data, i.c. 50 data does not result comparable increase. Interestingly, our model is also highly robust to partial occlusion, outperforming NeMo under low and large partial occlusion scenarios. Note that this an -uitt--diitribution scenario, since the training data does not contain partially occluded objects. The overall improved performance compared Table can be attributed to the fact that KITTI only contains 3D annotations of cars, which have a cuboid like overall shape. In contrast, PASCAL3D+ contains other objects, such as aeroplanes, for which the shape is not approximated well by cuboid. This suggests, that a more accurate shape representation could further improve the performance the PASCAL3D4 data. Qualitative prediction results of our method are illustrated Figure 3 for a number of different categories in the PASCAL3D+ and KITTI datasets.
4.3 Quality of Neural View Synthesis and Matching over Time
Figure 4 illustrates the quality of the neural view synthesis and matching process with different feature extractors. We start from set of 20 randomly selected anchor images from the category of

---

(a) PASCAL3D+, motorbike
(b) PASCAL3D+, bicycle
(c) PASCAL3D+, boat
(d) PASCAL3D+, bus
(c) KITTI, car
(f) KITTI, car
Figure 3: Qualitative results on the PASCAL3D+ and the KITTI dataset. We illustrate the predicted 3D pose using CAD model. Note that the CAD model is not used for our approach.
Imagelent Festrnne arme
eporn (mageelett Pranrassdd
epoch
wosch
0.5
spoch
Boleton Door
04-
06-
6.1
02-
29 azmuth
(a) Match quality under different training setup.
(b) Match quality under variant training epochs.
Figure 4: Visualization of the spatial matching quality comparing an ImageNet pre-trained feature extractor and feature extractors trained with NVSM with different amount of annotated data. We plot the 3D pose rotation error spatially matched images (y-axis) as a function of the target azimuth angle used for synthesizing novel views (x-axis). (a) Influence of the number of annotated images. (b) Match quality at different training epochs for model trained with 50 annotated images.
the PASCAL3D+ dataset. For each anchor image, novel views as described in Section 3.1 by varying the azimuth angle of the 3D pose (x-axis). We spatially match the synthesized views the remaining test data described in Section 3.2 to retrieve 3 images that best fit the synthesized views. The y-axis of each plot shows the rotation error between the cuboid pose
the ground-truth 3D pose 0 and synthesize
AO used synthesize the novel view and the retrieved images. Each plot is averaged over all anchor images and plots the error as a function of the azimuth pose in the range from - 3/2 5. Figure 4(a) compares the spatial matching quality of ImageNet pre-trained feature extractor (red) with feature extractors that are trained with our proposed framework and different numbers of annotated images. It can be observed, that the ImageNet pre-trained features reasonably effective when the synthesized pose is close to the ground-truth pose, but they are not reliable when the pose difference is large. Remarkably, when using annotated images and NVSM. our model is able to train the feature extractor that is much more reliable even for very large pose differences. We also observe that additional annotated data further improves the spatial matching quality. Figure 4(b) shows how the matching quality evolves as function of the number of trained epochs using 50 annotated training images. The matching quality improves significantly over the first 20 epochs and further improves, but more slowly, over the remaining training process.

---

(a) Extracted Features
(b) Averaged Features
Figure 5: Visualization for features obtained from semi-supervised trained feature extractor. We use PCA to reduce the features into 3-channels and visualize them in RGB. (a) Features extracted from one testing image via semi-supervised trained feature extractor. (b) The learnt average features of the training set dr.
4.4 Visualizing the Learned Features
In order to qualitatively evaluate the features and the feature extractor learned with our semi- supervised approach, we visualize the features using Principal Component Analysis (PCA) [36]. In Figure 5 (a), we apply PCA on the features extracted from test image, and reduce their dimensional- ity to three using PCA to visualize them as an RGB image. In Figure (b), we conduct the same dimensionality reduction on the averaged feature vectors described Equation 6) at each mesh vertex Σ, and render them using the same 3D pose as in Figure 5 (a).
Figure (a) shows clear boundary between the object and background, which indicates that the ImageNet pre-trained backbone backbone is object-aware and encodes features on the object very differently compared to the background context. However, features within the object are similar to each other. In contrast, Figure 5 (b) demonstrates that the features of our trained features extractor are more similar each other when they are spatially close. Moreover, most features on the mesh have different color embedding, i.e. they are different from each other. This is exactly what the contrastive loss Equations and encourages. However, we note that the first three PCA components only account for about 14.5% of the variation in the data (and hence some features that are very different in the original space might share a similar color in this visualization).
Conclusion
In this work, we introduced semi-supervised learning framework for category-level 3D pose estimation from very few annotated examples and a collection on unlabelled images. We followed a matching-basse approach, in which we transfer the 3D pose annotation from the labelled examples to the unlabelled data. We achieved this by representing objects as 3D cuboid meshes composed of feature vectors each mesh vertex. The mesh features were sampled from the annotated data and subsequently used to generate feature maps of unseen 3D views by rotating the mesh cuboid. The synthesized views were then spatially matched to unlabelled data to generate aseudo-labels. We demonstrated that the proposed neural view synthesis and matching approach enables a highly efficient learning of 3D pose, particularly in extreme few shot scenarios, while the same time achieving an exceptional robustness to out-of-distribution scenarios that involve partial occlusion.
Limitations and Societal Impact. Our work shares common limitation with other works on 3D pose estimation, in that it cannot be trivially generalized to articulated objects. This is even more emphasized in the semi-supervised few-shot setting that we are studying in this paper. For us, the most promising future research direction would therefore be to extend our efficient learning framework to articulated objects such humans or animals. Like other approaches for 3D pose estimation, framework belongs to the type of technical tools that do not introduce any additional foresecable societal problems, but will in the long term make computer vision models generally better.
Acknowledgments
We gratefully acknowledge funding support from Institute for Assured Autonomy at JHU with Grant IAA 80052272, NIH RO1 EY029700, and ONR 2-001--0-11-2000 We also thank Qing Liu, Zihao Xiao, Peng Wang and Fang Yan for suggestions paper.
10

---

References
[1] Michael Alcom, Qi Li. Zhitao Gong, Chengfei Wang. Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4845-48544 2019.
[2] Yutong Bai, Qing Liu, Lingxi Xie, Weichao Qiu, Zheng, and Alan Yuille. Semantic part detection via matching: Learning to generalize to novel viewpoints from limited training data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7535-7545, 2019.
[3] Yutong Bai, Angtian Wang, Adam Kortylewski, and Alan Yuille. Coke: Localized contrastive learning for robust keypoint detection. arXiv preprint arXiv: 14115, 2020.
[4] Vassilcios Baintas, Edgar Riba, Daniel Ponsa, and Krystian Mikolajezyk. Learning local feature deseriptors with triplets and shallow convolutional neural networks. In Brve, volume 1, page 3, 2016.
[5] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantify- ing interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[6] Silvia Biasotti, Andrea Cerri, Alex Bronstein, and Michael Bronstein. Recent trends, applications, and perspectives in 3d shape similarity assessment. Computer Graphics Forum, volume 35, pages 87-119. Wiley Online Library, 2016.
Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187-194, 1999.
[8] Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4080 4088, 2018
[9] John Canny, A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence, (6):679-699) 698, 1986.
[10] Olivier Chapelle, Mingmin Chi, and Alexander Zien. A continuation method for semi-supervised svms. In Proceedings of the 23rd international conference on Machine learning, pages 185-192, 2006.
[11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04220 044.9777... 2020.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[13] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 121112)123111277, 2013.
[14] Andreas Geiger, Philip Leuz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
[15] Kai Han, Rafael S Rezende, Bumsub Ham, Kwan-Yee K Wong, Minsu Cho, Cordelia Schmid, and Jean Ponce. Scnet: Learning semantic correspondence. In Proceedings of the IEEE international conference on computer vision, pages 1831-1840, 2017.
[16] Christopher G Harris, Mike Stephens, et al. A combined corner and edge detector. Alvey vision conference, volume 15, pages 10-5244. Citeseer, 1988.
[17] Kaiming He, Xiangyu 7hang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Thomas N Kipf and Max Welling. Semi superviee classification with graph convolutional networks. arXiv preprint arXiv: 1609:v:660902220 1609.02907 2016.
[19] Bailey Kong, James Supancic, Deva Ramanan, and Charless C Fowlkes. Cross-domain image matching with deep feature maps. International Journal of Computer Vision, (11(11))17381177 2019.
[20] Adam Kortylewski, Ju He, Qing Liu, Christian Cosgrove, Chenglin Yang. and Alan L Yuille. Compositional generative networks and robustness to perceptible image changes. In 2021 55th Annual Conference on Information Sciences and Systems (CISS), pages 1-8. IEEE, 2021.
11

---

[21] Adam Kortylewski, Ju He, Qing Liu, and Alan L Yuille. Compositional convolutional neural networks: A deep architecture with innate robustness partial occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vixion and Puttern Recognition. pages 8940-8949, 2020.
[22] Adam Kortylewski, Qing Liu, Angtian Wang, Yihong Sun, and Alan Yuille. Compositional convolutional neural networks: A robust and interpretable model for object recognition under acclusion. International Journal of Computer Vision, pages 1-25, 2020.
[23] Adam Kortylewski, Qing Liu, Huiyu Wang, Zhishuai Zhang, and Alan Yuille. Combining compositional models and deep networks for robust object classification under occlusion. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 1333-1341, 2020.
[24] Vijay Kumar BG, Gustavo Cameiro, and Ian Reid. Learning local image descriptors with deep siamese and triplet convolutional networks by minimising glohal loss functions. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 5385-5394, 2016.
[25] Vincent Lepetit. Francesc Moreno-Nogues and Pascal Fua. Epup: An accurate (n) solution to the pup problem. International journal of computer vision, (1)))155, 2009.
[26] Liu, Jenny Yuen, and Antonio Torralba. Nonparametric scene parsing via label transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, 33(12)2366 2382, 2011.
[27] Qing Liu, Adam Kortylewski, Zhishuai Zhang, Zizhang Li, Mengqi Guo, Qihao Liu, Xiaoding Yuan, Jiteng Mu, Weichao Qiu, and Alan Yuille. Cgpart: A part segmentation dataset based on 3d computer graphics models. arXiv preprint arXiv:2103. 140998 2021.
[28] David G Lowe. Distinctive image features from ccale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
[29] Jiayi Ma, Weichao Qiu, Ji Zhao, Yong Ma, Alan L Yuille, and Zhuowen Tu. Robust /_ [_[2]e estimation of transformation for non-rigid registration. IEEE Trunsactions on Signal Processing, 63(5):1115-1122
2015.
[30] Hans Moravec. Techniques towards automatic visual obstacle avoidance. 1977.
[31] Olga Moskvyak, Frederic Maire, Feras Dayoub, and Mahsa Baktashmotlagh. Semi-supervised keypoint localization. arXiv preprint 2XXXv:2000.0798. 2021.
[32] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7074-7082, 2017.
[33] Jiteng Mu, Weichao Qiu, Gregory D Hager, and Alan L Yuille. Learning from synthetic animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12386- 12395, 2020.
[34] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv: 1912.01703, 2019.
[35] Georgios Pavlakos, Xiaowei Zhou, Aaron Chan, Konstantinos G Derpanis, and Kostas Daniilidis. 6-dof object pose from semantic keypoints. In 2017 IEEE international conference on robotics and automation (ICRA). pages 2011-2018. IEEE, 2017.
[36] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh. and Dublin philosophical magazine and journal of science, 2(1 2(1)):59-557, 1901.
[37] Weichao Qiu and Alan Yuille. Unrealev: Connecting computer vision to unreal engine. In European Conference on Computer Vision, pages 909-99999999991999 916. Springer, 2016.
[38] Rasterisation. Rasterisation - Wikipedia, the free encyclopedia, 2021. [Online: accessed 26-May-20211
[39] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint arXiv:2007.0850/ 2020.
[40] Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. Semi-supervised self trainiraining of object detection models. 2005.
12

---

[41] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Can features off-the-shelf: astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition workshops, pages 806-813, 2014.
[42] Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, lasonas Kokkinos. Pascal Fua, and Francesc Moreno- Noguer. Discriminative learning deep convolutional feature point descriptors. Proceedings of the IEEE International Conference on Computer Vision. pages 118 1166,, 2015.
[43] Stephen M Smith and Michael Brady, Susan new approach to low level image processing. International journal of computer vision, 23(1):45-77 181455777777 1997.
[44] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001 100600077,.000 2020.
[45] Yicong Tian, Chen Chen, and Mubarak Shah. Cross-view image matching for geo-localization in urban environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608-3616, 2017.
[46] Isaac Triguero, Salvador García, and Francisco Herrera, Self-labeled techniques for semi-superve semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):245 -2,424,,, 2015.
[47] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1510-1519, 2015.
[48] Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learning. 109(2)3373 440, 2020.
[49] Angtian Wang, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. International Conference on Learning Representations, 2021.
[50] Angtian Wang. Yihong Sun, Adam Kortylewski, and Alan L. Yuille. Robust object detection under occlusion with context-aware compositionalnets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[51] He Wang. Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-Ieve 6d object pose and size estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2642-2651, 2019.
[52] Paul Wohlhart and Vincent Lepetit. Learning descriptors for object recognition and 3d pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3109-310 1119-,18,,,,11111 2015.
[53] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision, pages 75-82. IEEE, 2014.
[54] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2014.
[55] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196, 1995.
[56] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung Yi Lin. inerf: Inverting neural radiance fields for pose estimation. arXiv preprint arXiv:2012.05877 2020.
[57] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[58] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprini arXiv: 1412.6866. 2014.
[59] Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang, Starmap for category agnossic keypoint and viewpoint estimation. In Proceedings of the European Conference Computer Vision (ECCV), pages 318-334, 2018
[60] Xiaojin Jerry Zhu. Semi-supervised leamning literature survey. 2005.
13


Reviews:
[{"review_id": "yXiUZaFYqGb", "reviewer": "Reviewer_h5BS", "summary": "This paper studies the problem of estimating the pose of an object, given an image of it. The proposed method aims to use as few labeled samples as possible, and generate pseudo-labels for a large collection of unlabelled data during training. The method works in an EM-fashion in which two stages (neural view synthesis and matching) are optimized by alternating.\n \nIn the first stage, the method extracts a feature map from the input annotated image and shapes it as a mesh cuboid where each vertex is assigned a feature vector depending on its position. The mesh is then rotated at regular angle intervals and rasterized into the image plane. In the matching part, features from unlabeled images are extracted and matched with rotated view rasterizations using a cosine similarity metric.\n \nThe experimental evaluations show that the proposed method is able to learn the pose prediction task from as few as 7 annotated images during training. The method mostly outperforms the baselines in comparisons against them. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is novel in the way that it generates 3D pose pseudo-labels for large amounts of unlabeled data which is difficult to annotate in practice. I believe that the method can be useful in future research in this area.\n\nOverall, I think the paper is organized well. There are a few typos, some of which are included at the end of this review. My questions and comments are listed below.\n\n- The term neural view synthesis sounds somewhat misleading in my opinion, since the proposed method does not synthesize and render RGB images in the way the literature on view synthesis does. I would suggest clarifying what is meant by view synthesis and adding how it is different from the literature. \n\n- How does the method ensure that the extracted features from the image have meaningful 3D information? What does it mean to rotate the features extracted by a network pre-trained on ImageNet?\n\n- How does the method avoid trivial solutions of assigning the nearest neighbor’s label as the pseudo-label? What is the intuition behind preventing such mode collapses? For instance, if there are images with labels of 30 and 45 degrees azimuth in the annotated dataset, how does the method infer an angle of 35 degrees? \n\n- Similarly, how is the model able to differentiate between the pose angles from the front and back views, or from left and right side views? Are there any failure modes involving such examples?\n\n- Do the results in the tables average over all three output angles (azimuth, elevation and in-plane rotations)? How does the method perform in these individual terms?\n\n- Is ∆θ only along the azimuth axis or does it also include the elevation and in-plane rotations?\n\n- Can you detail the optimization at test time more? Does the method use additional labeled data at this stage? What is the computation time of the pose optimization at inference time? \n\n- What is the procedure to select the few images with labels? Does it need to cover the viewing hemisphere? Any discussion would be helpful to include in my opinion.\n\n- Does the approach of averaging features from multiple annotated images have any drawbacks? Can having a large number of labeled data (> 100) degrade the performance due to features averaging out each other? \n\n- What is the curriculum strategy for starting with small ∆θ and increasing it while training?\n\n- Why are the angle errors increased for the case of ∆θ = 0, after the semi-supervised training in Figure 2?\n \n\nMinor comments:\n\n- Why is the feature cuboid named a mesh rather than a voxel grid? Since the mesh is a grid, and the edges and faces are not used, I believe calling it a voxel grid is more accurate.\n- What do the features projected to the image plane look like? I would suggest adding those generated feature images to the paper.\n- Please include the standard deviation of the results in the tables as well, for a more complete evaluation. \n- Typos: line 53: the the; line 63: for for; line 65: annotated -> annotate, line 78: between between, line 161: wholes -> holes\n\nUpdate: after my concerns were clarified in the response, I raised my score to accept.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of this work, along with potential societal impacts are discussed in the conclusion. The inability of trivial generalization to articulated objects such as humans is listed as a limitation. I believe that there are others as well though, such as the existence of a large error margin (40%) for pose estimation even when the model is evaluated at 30 degree thresholds. I encourage the authors to detail the limitations section of the paper. \n", "review_text": "", "extra_scores": {"ethical_concerns": "The paper does not present any ethical concerns to the best of my knowledge.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Zzu0_bWu84U", "reviewer": "Reviewer_kdyM", "summary": "This paper presents a method for 3D pose estimation of PACAL3D objects (6 vehicle categories). The method bootstraps from 7-50 annotations per category. The idea is: given the initial annotations, finetune a Resnet50 and a featurized cuboid mesh with contrastive losses that bring the 2D and 3D features close to each other, and make them discriminative for position/viewpoint. Then, compute 2D features for the entire dataset, and rotate the 3D featuremap a few degrees in each direction, and compute match scores for all pairs. The confident matches are then used for subsequent re-optimization of the 2D and 3D features, and this process repeats ~100 times, gradually increasing the number of rotation bins covered by the 3D featuremap. Results are quite good against non-bootstrapping baselines, trained just with the 7-50 annotations. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper. Pose estimation is a hard problem, and this boot-strapping/EM solution combines insights from pre-trained ImageNet feature-learning and structured 3D representations in a clever way, to gradually cover the full set of viewpoints/angles of the objects. The approach is a little constrained, in terms of assuming perfect detections and known scale/distance and centroid, but maybe this is OK. \n\nI was surprised that the approach does not also involve some optimization of the 3D mesh of the category, and instead keeps it as a cuboid. Have the authors tried optimizing this, or perhaps optimizing occupancy inside the grid, and using occupancy as part of the rendering?\n\nWhat about the drift? EM/bootstrapping methods like this inevitably drift (and also sometimes recover). I think it is important to show pose estimation errors over EM rounds, and maybe to visualize some \"hard\" examples for which the the wrong pose has a better match score than the right pose.\n\nHow does the pose analysis-by-synthesis work exactly? Is it SGD on the three angles? Does this require multiple initializations, or does it work even if it starts 180deg off?\n\nI understand the idea to compare against baselines under the same semi-supervised setting, but I think it is important to also report those models' fully-supervised results, to put things into perspective. \n\nIt would be interesting (but maybe not essential) to compare against \"Image GANs meet differentiable rendering for inverse graphics and interpretable 3d neural rendering\" (ICLR 2020), which I think can also be interpreted as an EM approach: an image generator is used to optimize a 3D inverse graphics net, and the inverse graphics net is used to optimize the latent space of the image generator. I do not think they evaluated on pose estimation, but they could have. \n\nIt would be nice to visualize the 2D and 3D features, with PCA.\n\nSome typos: \n- Groundturth -> Groundtruth\n- wholes -> holes", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "No concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Kn59ITdlHvd", "reviewer": "Reviewer_B1pt", "summary": "The paper presents a learning based method for 3D pose estimation from single image that works with very few annotated data. Given some exemplar images with 3D pose annotation, the 3D pose of a test image could be estimated by two steps: 1) neural view synthesis and 2) feature matching. In the neural view synthesis step, a 3d feature is generated by unprojecting a pretrained 2d feature map of the image. The 3d feature is then probably \"projected\" to a set of novel views for feature matching and 3d pose estimation.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weakness/questions:\n\nThe paper proposed a few shot learning method for 3D pose estimation from 2d images that utilizes the underlying 3D structure. A 2d feature map is unprojected to a spatial grid and this kind of 3D representation is desired to be pose-invariant. However, it is not very clear to me how occlusion reasoning is performed here and how a holistic shape representation is inferred from a single image here. For example, how are the features on the grid are sampled from a 2D view? Is any self-occlusion taken into account when the 2d feature are unprojected to the 3d grid? How about the features at the occluded vertex? Do they take the feature from the object that is in front of them? Also in the neural view synthesis step, how is the rasterization performed? Are all vertex considered to be non-transparent? it is a bit unclear to me for this part and I hope the author could provide more details about this part.\n\nThis paper seems to be similar to another relevent work NeMo which is also cited as [44] in this paper. I am wondering if the author could elaborate more on the difference between those two works and the novelty of this work among NeMo?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2h"}, "extra_text": {}}, {"review_id": "Chn3E16Xpq", "reviewer": "Reviewer_9Mv2", "summary": "This paper investigates a new problem of semi-supervied 3D viewpoint estimation using few-shot labeled examples. It further proposes a new method for the task based on learning a 3D cuboid feature for a category with features at its vertices which are viewpoint invariant. The authors propose a method of rotating the cuboid to generate feature representations for novel views. Another key insight of the paper is how to expand the ranges of novel views for which the cuboid can be rotated by introducing a contrastive loss to update the cuboid feature representation to be more rotation invariant by combining with the pseudo--labeled images. The authors compare their method to two baseline SoTA methods and outperform the existing methods by a large margin in the few-shot setting.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper introduces both a new challenging task of few-shot semi-supervised learning as well as a novel and effective method for it with a strong baseline performance. The work is quite original and opens up a new research area.\n\nQuality: The proposed method is technically sound. The experiments are adequate and well formulated. I note a few shortcoming to improve the paper.\n\n1. The authors describe that they manually chose the 7 few-shot labeled examples per category to train with such that they equally span the entire range of pose angles. It seems to me that this choice is critical to the success of their proposed method. In the worst case, if all the 7 few-shot poses were clustered at a single point, I doubt that the authors would be able to expand the pseudo labels to beyond a small range of angles around the labeled ones. The authors should discuss this limitation of their approach more clearly and provide experiments to clearly show the affect of the choice of few-shot samples on the accuracy of their algorithm. Does their algorithm support a random choice of angles?\n\n2. From Table 1, the proposed method does not seem to perform as well as NeMo for 50 training samples. This behavior for larger number of trained samples is not observed for the KITTI datasets, however. So, perhaps it is an artifact of the Pascal3D+ dataset. I would like to hear a further discussion of this from the authors and whether they have analyzed  the cause of this observation. I am also curious as to what happens with greater than 50 labeled samples and for 100% labeled training data? Does the proposed approach perform as well as NeMo on both datasets in fully labeled scenarios as well or is it limited in applicability only to less labeled scenarios? \n\n3. Few-shot Viewpoint estimation has been explored previously in the work: Few-Shot Viewpoint Estimation, Tseng et al, BMVC 2019. However it was in a different context of exploring few-shot generalization to previously unseen categories of objects, which is different from this current work. Nevertheless it would be good to cite this closely related prior work.\n\nClarity: The paper is mostly clear. Code is provided. There are several small typos throughout the paper which would be helped with a thorough editorial review. \n\nSignificance: Labeling viewpoint or 3D pose is a hard problem. The proposed work opens up a new line of research; proposes a novel method with several interesting new insights, including the use of the 3D feature projection and of using the contrastive loss in an EM-style framework to expand to more viewpoint; performs significantly better than the SoTA approaches and generalized to many different object categories. This is a significant and broadly impactful piece of research.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It would be good to discuss further the points 1 and 2 stated in the previous answer as potential limitations of the method in the paper, if they are indeed limitations of the method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses semi-supervised 3D viewpoint estimation by lifting 2D CNN features into object-centric feature cuboids, which are rotated and rendered to generate features maps of alternative views, while also updates in a contrastive manner exploiting labelled and pseudo-labelled examples in an iterative manner. All reviewers agree on the novelty of the approach pursued in the paper. The rebuttal submitted by the authors clarified many concerns of the reviewers regarding handling of  occlusions and self-occlusions, scalability of the method with respect to the number of annotated examples, evaluation metric and thresholds used. 

Decision: Accept (Poster)