Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization
Authors: Stefan Stojanov
Ngoc Anh Thai
Zixuan Huang
James Matthew Rehg
Venue: NeurIPS 2022 Accept (2022)
Keywords: low-shot
self-supervised
dense descriptor

Abstract: A hallmark of the deep learning era for computer vision is the successful use of large-scale labeled datasets to train feature representations. This has been done for tasks ranging from object recognition and semantic segmentation to optical flow estimation and novel view synthesis of 3D scenes. In this work, we aim to learn dense discriminative object representations for low-shot category recognition without requiring any category labels. To this end, we propose Deep Object Patch Encodings (DOPE), which can be trained from multiple views of object instances without any category or semantic object part labels. To train DOPE, we assume access to sparse depths, foreground masks and known cameras, to obtain pixel-level correspondences between views of an object, and use this to formulate a self-supervised learning task to learn discriminative object patches. We find that DOPE can directly be used for low-shot classification of novel categories using local-part matching, and is competitive with and outperforms supervised and self-supervised learning baselines.

Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalization
Stefan Stojanov, Anh Thai, Zixuan Huang, James M. Rehg Georgia Institute of Technology (sstojanov, athai6, zixuanh, rehg))gatech..en
Abstract
A hallmark of the deep learning era for computer vision the successful use large-scale labeled datasets to train feature representations. This has been done for tasks ranging from object recognition and semantic segmentation to optical flow estimation and novel view synthesis of 3D scenes. In this work, aim learn dense discriminative object representations for low-shot category recognition without requiring any category labels. To this end, we propose Deep Object Patch Encodings (DOPE), which can be trained from multiple views of object instances without any category semantic object part labels. To train DOPE, access to sparse depths, foreground masks and known cameras, to obtain pixel- level correspondences between views of an object, and use this to formulate a self-supervised learning task to learn discriminative object patches. We find that DOPE can directly be used for low-shot classification of novel categories using local-part matching, and is competitive with and outperforms supervised and self- supervised learning baselines. Code and data available at /tttps://gttaiiii com/ rehg-lab. 1es--aabdoopeeeee ht//d/ppp_effssu
assume
Introduction
Achieving high accuracy at object category recognition generally requires deep models with millions of parameters 266 3 and million-scale datasets [I] 37 Alleviating this data requirement is fundamental challenge for computer vision and has driven the development of low-shot [57] 65
17 76 5
and self- self-supervised 43 25 22 methods for learning object representations. Despite the increasing interest in low-shot and supervieed learning, only few works tackle the case where multiple unlabelled views of object instances are available for sulf-super learning of discriminative object representations [27 32]. In addition, their primary focus is not low-shot generalization. Using multi-view data to learn representations for low-shot categorization in self- supervised manner is of practical interest, such data can potentially be obtained from robots moving around and handling objects [99 or by images and videos collected by humans using mobile devices 77 53).
The goal of this paper is address this gap by using multiple-views of many object instances for training, combined with some additional geometric information about the views, to learn representa- tions that can be directly used for low-shot object category recognition. We assume access to sparse depth, known camera pose/calibration, and foreground segmentation during training, with no access to category labels. We use this information to formulate a self-supervised representation learning task. In practice, the additional data necessary for self-supervision can be obtained in controlled indoor settings using robotic arms equipped with RGB-D sensors or by postprocessing RGB-D sensor data and using forward kinematics to obtain the camera pose [19 23). Otherwise, in less constrained settings where the data vidcos of objects captured using mobile devices, the data needed for self-supervision can be obtained from Structure- wenecture-fro---eee from-Motion (SIM) and Multi-View
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Desu Trasn Daspu de
Uiit papeoppax
Docas Raaau Oupaa 250 26.
Inbae: Vize
Dowo Forre Oeqer Min St
24% din
W
Torget
Figure 1: Left: Our contrastive learning formulation: We form positives between features at pixel locations of the same 3D point the object surface, and negatives pixels from different points the object surface. Right: For each feature corresponding to a point on the predicted segmentation mask in the source view assign color. For each point the predicted segmentation for the target views, we assign the color of the highest cosine similarity feature from the source view. We see that model learns dense object encodings which are consistent across many views of object.
Stereo (MVS) pipeline such COLMAP 55 56 Note that we do not assume access to any sort of category-level object or part annotations or any object attribute information for representation learning. At test time, method only uses single-view RGB images.
To solve this self-supervised learning task, we propose Deep Object Patch Encodings (DOPE). DOPE uses pairs of object views to learn a dense, CNN-based local feature descriptor that will map a patch on the surface the object that is visible
both views to the same feature vector, and different surface patches to different feature vectors (see Figure [T]. We define object patch as local neighborhood on the 3D surface of a rigid object. Surprisingly, we find that such a representation, trained only pairs of views of the same object instances, generalizes across novel object categories (illustrated Figures @gguresBandd and This is demonstrated by the ability to encode the same object parts, e.g. motorcycle tire, instances of a category. This generalization ability allows us to directly use the DOPE representation (trained only local patch matches with category labels) for low-shot object recognition.
airplane wing, with the same learned feature vector across multiple
DOPE is trained using a local contrastive learning loss which requires correspondences between views to form positive and negative pairs. To find such pairs, we use sparse depth map and camera information. Similar contrastive formulations have been used before to train instance-specific or class-specifer dense object descriptors [19 23 from RGB images for robotics applications, but did not address low-shot recognition. We hypothesize that learning dense surface patch similarity at the local feature level forees the model to learn encode all visible object parts in two views, and find that such representation is useful for low-shot object categorization.
We find that DOPE is competitive and in some cases outperforms both self-supervised methods 27 and fully supervised low-shot methods 66 62 76 on the synthetic ModelNet 771, ShapeNet and the real-world CO3D [53) datasets. We further find that DOPE can learn representation using objects from the large, uncurated 3D object dataset ABC (35), which directly transfers to low-shot category recognition. In summary, make the following contributions:
Deep Object Patch Encodings (DOPE), self-supervised learning approach based on learning object shape correspondences, generalizes low-shot object categorization.
The first study of how a large, unstructured object instance dataset like ABC [35 can be used effectively to learn representations for low-shot category recognition from images.
Multiple datasets with high photorealism rendered from existing 3D assets that will enable future research in learning representations using multi-view information.

---

2 Related Work
2.1 Low-shot Learning for Object Recognition
The goal of low-shot learning is to build models that can learn how to classify images based few labeled samples. These are typically trained on a large base class dataset, and their low-shot generalization ability is evaluated and testing queries from novel, unseen classes. Such models can be categorized based on how the base class data is used, and whether the model'ss parameters are updated during the low-shot phase. Based on base class data, the meta learning- rearning-based methods [37 72 I7] sample low-shot episodes from the base classes to train under the same setting as during evaluation. In contrast, the whole-clas sttole-classffication--ass
many low-shot episodes with a few labeled training images
62 train the backbone feature representation as a classifier further fine-tune the representation with meta-learning procedure. Based on whether the model parameters are updated, optimization-base techniques [7 [8] aim learn a representation that can be fine-tuned using a few samples while metric learning-baa techniques [68] 72 8 62] learn a discriminative metric space that will directly transfer to low-shot episodes of novel classes. Our self-supervised representations do not require any labels during training, and like metric learning techniques, can be directly applied to novel classes.
techniques 668
on the base classes, whereas 72
8 20
2.2 Self-Supervised Learning
Self-supervised learning aims to formulate tasks that exploit the underlying structure of the data rather than category labels to train deep network-based representations suitable for downstream tasks.
Self-Supervised Learning for Object Classification Early works formulated proxy tasks such as colorization 778). predicting relative positions of patches [12] 43) or predicting rotations (21). More recently, methods based on instance-leve comparisons have led to significant performance improvements. These are trained to map multiple views (usually multiple augmented versions of an image) as the same feature vector under some similarity measure, and different images as different vectors. SimCLR and MoCLR 61 consider augmentations of the same underlying image as positives and other images in the minibatch as negatives, whereas the McCo-basee 6 25 approaches use a continuously updating queue of negative features. The methods presented in 61 use second nomennuu-uudaate views of an instance. Our proposed approach makes use of a momentum encoder as well. Further, some works like [22 2 achieve superior downstream performance for classification without requiring any explicit negatives, while others [24] obtain high performance by revisiting image reconstruction as proxy task using vision transformers [3].
2 all
encoder rather than using the same backbone to encode multiple
Self-Supervised Learning for Dense Vision Tasks The main ideas from self-supervised contrastive learning for classification have been extended learn local, dense, pixel-level representations for tasks like semantic segmentation, keypoint, and object part learning. For segmentation and detection, 775 67 73 45) apply contrastive losses at the pixel level, treating features corresponding to individual pixels or pooled regions as positives or negatives, or alternatively use clustering based [79] losses to fine-tune visual transformers. For keypoint learning, Cheng et al. [00] use a dense contrastive strategy, while Novotny et al. [44) propose probabilistic matching loss. In addition, SCOPS (31) is self-supervised technique for object part segmentation of fine- tne-grained categories. All of these prior works are trained using pairs of images that have undergone 2D geometric transforms, while we use images corresponding to different camera positions and focus on low-shot categorization.
Sele-Supervised Learning using 3D Geometry and Multi- View Images Self-supervised learning
has been studied in settings where multi view RGB-I D information (images and in-camera dense First,
3D Point clouds) is available 29 74
48
36
19
229
[5
38
36 48
use pixel-level
correspondences between views of scene, where a positive pair of pixels represents the same 31) point in the world projected in each image. Such pixel-level correspondences have been used align IT] image and depth- scene representations [29 [:]] 38) for point cloud registration [13) and scene
segmentation 38 28 or learn 3D CNN-based scene representations from multiple views [36] 48)
Herre by align we mean make corresponding features between the 2D and 3D representations have high similarity.
3

---

Synthetic Data Domain
Real Data Domain
STOP
Trnining: ModelNet basc
Training ABC 125K
Tasting ModeINet test
Tranning CO3D basc
Testing CO3D Test
Figure 2: Left: Synthetic data rendered from ModelNet and ABC (first and second columns) for self-supervised learning of representations and low-shot testing data from ModelNet (third column).
Right: Real data from CO3D for superviee representation training and low-shot testing.
for self supervised object tracking and few-shot style/shape learning. Florence al. [19 use RGB images to learn dense object descriptors for robotics applications. comparison, our goal is to use correspondences between multi-view RGB images of objects to learn representations for low-shot object classification. Other works 74 28 directly train on 3D point clouds. Last, assuming only access multiple RGB views, 322 uses encoder/decoder architecture predict images of different viewpoints from a single image, whereas VISPE [27] formulates an instance discrimination task. While share with these works the use of self-supervised representation learning from multiple RGB views for object recognition, we focus on low-shot object classification. We show that our approach outperforms VISPE -ike instance-level global representations multiple datasets.
2.3 Object Classification Synthetic Datasets
Object recognition has been studied using 3D datasets such as ModelNet40
converted into point
clouds [50 52 69], voxels 71), rendered into images [60] 51 16
70 32) We use
photorealistic rendering to generate data, and conduct the first study to investigate the use the large unstructured ABC (35] dataset for representation learning in low-shot object recognition.
3 Learning Deep Object Patch Encodings DOPE
We aim to learn a representation based solely on multi-view images of object instances that can directly generalize low-shot object category recognition tasks. For training, we assume access relative pose between views of object instances, foreground/background segmentation, and sparse depth. However, for low-shot generalization at time, we only assume access single RGB images. We do not make any assumptions about the semantic structure of the data for representation learning, or about the availability of any semantic labels such explicit parts or attribute information.
In this section, we describe our approach for contrastive learning of dense object descriptors from multiple views that generalize object categories. and present simple local nearest neighbor- based method to utilize this learned representation for low-shot classification.
3.1 Local Object Representation Learning
Assume we have dataset D of object instances oN, where each object has N calibrated views with masks and known camera intrinsics and extrinsics. Denote the two views one object as vi and v If these two viewpoints are sufficiently close, there will be points on the surface of the object, e.g. one leg a chair, that will be visible in both views. This means that the same point on the object surface, will get projected to pixel coordinates (u1, v1) in v and (ug, in V2 Let f: TR Hx - x x xD be a learnable feature map that maps RGB images to 2D feature grid with D channels and h and Wy height and width. Let ūg be the location the lower spatial dimensional feature output corresponding for the corresponding pixel locations z = f(v(') 01 and z = f(v?" ) [ūg, ū2 are so that sim(z x²2) 1, where sim is some similarity metric, in our case normalized dot product. Conversely, we want sim(z;) z O, where z a feature from a pixel projected from the object surface in v that is not in correspondence. Further, for a feature 2 extracted from some different object 02, we want sim i 2 0. Note that z E DR_ In other words, following the chair leg
uk. Our goal is to learn f so that the feature vectors

---

256 x
Masked and Normalized
125 x 56 x
Festure Ourut
Nox dooie
Convolutional Map 128 x x
1xl Cony
Projection
Dense Object Patch Encoding
Input Image
Panoptic FPN
Spatially Dense Feature Representation
Mask Predictor
Binury Mask
Figure 3: Our architecture consists of ResNet18 [26] followed by a Panoptic Feature Pyramid network [33 to output 2D grid of convolutional features. We an additional branch from these features to predict a binary mask. After applying a series of Ixl convolution layers, we perform channel-wise unit vector normalization and masking with the binary mask obtain the final feature output we use for contrastive training and at test time
example, we want f to produce the same feature in the 2D feature grid for a point on the chair leg in any viewpoint where that chair leg is visible (Please refer to Fig. for illustration).
To learn such representation, an image pair vi and v22 For each view pair, we sample a set C of n pixels on the object surface in vi using farthest point sampling and find the corresponding pixels in v2 2 For each pixel in C we therefore obtain the feature vector 21th and the corresponding z whose similarity (zi 22th want to maximize. Concurrently, we want to minimize: (1) sim(z)) oN z² for any feature z from
sample batch of objects B, and for each object o, E B we sample
a non-corresponding pixel on the object and (2) sim (z 2z,) for any feature 2 from some other object where 0, E B, where j j k. To achieve this we minimize a normalized and temperature-scaled cross entropy loss [58 6
for each pair of positive correspondences
exp (sim(z?' z.z 3/T)
= log
(1)
Σggx exp (sim / Ej.i.R Ezp exp (zi /ZT))/T
where 7 a temperature parameter controlling how peaky the softmax output is. This formulation is commonly used in contrastive learning. We apply this loss for all sampled points in each image pair and refer to it as Leor Ece.
To avoid learning to encode the background, learn additional module h : XwyyD Rhy to predict binary masks mn which we train with binary cross entropy Lmusk = BCE(m, m). We then apply this mask to the final output of (refer Fig. 3 to mask out the background. We optimize the sum of both losses with equal weight. Fig, I contaiin qualitative examples of how our proposed approach matches object shape across viewpoints.
3.2 Low-Shot Object Recognition using DOPE
Given that we have the means to encode the same object part from different images (see Figures 5 a)d present simple nearest neighbor-based method to make use of this representation for low-shot category recognition. This method is outlined in Figure 4, and we give formal description next. Let Z RDxhy xwy be feature map of query image that wish to classify given labeled support set of N images. Our goal is to find the image from the support set which is the most similar the query and assign its label to the query image.
We first sample pixel locations from the predicted foreground of the query image using farthest point sampling and extract their corresponding feature vectors 2 ......, 2 We then extract complete feature grids for each of the shot images and flatten them into matrices Z!,... ..., Z (h; x Note that the z vectors and the columns of the Z matrices unit normalized. The product s Z,zi, where st E TRyy-w wy is the similarity of the query feature vector 2g the i-th sampled pixel location and all features Z from the nith support image. We compute st for all points in the query image, and use E max(s as the score for that query/support image pairs. We take the label from the
We obtain these correspondences using the segmentation masks, known camera intrinsics and extrinsics and depths. We also the depth information to avoid sampling points that occluded.
5

---

Doop Diject
Porn Etwcoling
Ponme Victos
MAX
0.9
Query
3.0
Excodia
256 5136 Featur Matns
max
0.5
Similuitt Score Between Shot
max
Compute similartiioe and toice max
Shor Motorrycle
Figure 4: To determine a similarity score between a labelled shot and query image, for each query image, we first use farthest point sampling to randomly choose & pixels the object surface, based on a predicted segmentation mask (in this example = 4). We then extract features corresponding to those points from the query, and full feature grid for the shot. After computing the cosine similarity between all query and each shot features, we find the maximum similarity score between each shot and all query features. We sum these scores for the final similarity score.
support image with the highest score and assign it to the query. When there are multiple shot images per category available, we perform 1-nearest neighbor classification.
4 Implementation Details
The backbone of DOPE is based on ResNet18 [26] with a Panoptic Feature Pyramid network [33].
taking as input a 3 x 224 x 224 image and outputting a 256 x 56 x 56 2D feature block. This backbone formulation follows the approach in VADeR [46]. The FPN output is then sent through 256. During training, we use n 32
set of 1x1 convolution layers with dimension
pixel locations on the object surface for each image pair, and use a batch size of 256 view pairs. DOPE is trained using AdamW W (39] and initial learning rate of 10 and weight decay of 10 3000 epochs, using cosine learning rate annealing schedule. For each view pair, one image is sent to a momentun-encoder backbone as in [25]. At test time, we use k 20 pixel locations for our local nearest neighbor-base classification approach. During training, we use the known segmentation mask to randomly remove the background and perform flipping, color jittering and additional photometric augmentation, building on [19]. We empirically found that this random background removal significantly improves the model's generalization ability, forcing to learn to encode cues from the foreground object. The self-supervised baselines and low-shot baselines the for DOPE
Experiments (Section 5) trained using the same background masking strategy to ensure a fair comparison. Further implementation details are provided in the Appendix.
Experiments
In this section, we demonstrate DOPE's low-shot generalization ability of in comparison to both supervised algorithms and strong self-supervised baselines on synthetic and real datasets. A surprising finding is that our model outperforms few-shot baselines trained on category labels for ModelNer40, and delivers comparable performance for CO3D. For thorough comparison, models are evaluated over five different randomly chosen validation and testing splits, where each validation split is used to find the best checkpoint for each test set. In our experiments, parentheses show confidence intervals based 2.5K low-shot episodes. We include further results and ablation studies in the Appendix.
5.1 Datasets
We use Blender 49 with the Cycles ray-tracing engine to create synthetic data, rendering views object with varying camera distance, azimuth, and elevation. Objects are on top of plane with PBR material and illuminated using image-based lighting from an HDRI environment map. These assets were collected from PolyHaven [47]

---

NGl0
3G00
Concgreszonn
Sonnor Incunce
Teigd
Souce
Tergo
Tergei
Scarcc
Tagat
Inaozeee
leazace
Inamnce
Figure Category-level dense correspondences for ShapeNet emerge from training on ABC [3 instance dataset. For each feature corresponding to point on the predicted segmentation mask in the source view we assign color. For each point the predicted segmentation for the target views, we assign the color of the highest cosine similarity feature from the source image. We see that our model learns dense object descriptors which transfer across instances category. We zoom into the object for the dense visualization images improve visibility.
59 and convention in low-shot learning we partition ModelNet (71,, ShapeNet and CO3D 3 into disjoint sets of base classes-used for representation training, validation classes- used for early stopping and hyperparameter tuning, and test classes used to test low-shot category recognition. Further, we use the 3D object instance dataset ABC [35 to investigate whether large and diverse set of object instances without any underlying category structure can be used learn representations useful for low-shot category generalization. For illustration see Figure Following are additional dataset details:
Following 33
ModeINet400LLS(71 is a dataset of 3D object categories split into disjoint sets of 20 training, 10 validation and 10 test categories. We use the base classes with labels for training the supervised models, and without labels for the celf-supervised models.
ShapeNett0-LLS(] is dataset 3D object categories split into 25 training, validation and 20 test categories. We the 20 base classes with labels for training the supervised models, and without labels for the self-supervised models.
ABC dataset of 3D object instances from 3D printing online repositories without any category structure. We randomly select set of 115K instances from this dataset for training and directly evaluate transfer to the ModelNet and ShapeNet low-shot test sets.
CO3D-LS(33) (Common Objects in 3D) is dataset which consists of 20K crowd-sourced videos of objects from 51 categories, where the videos are collected by person moving around an object using a mobile device. We split CO3D into 31 base, 10 validation and 10 classes. The dataset is post- posseproeeeeed using COLMAP [56] 55 to estimate the cameras and reconstruct the scene and object geometry. It also includes object segmentation masks estimated using PointRend [34].
5.2 Baseline Methods
We provide a brief overview of the supervised and self-supervised baseline algorithms we evaluate against. All baselines implemented using the same ResNet18 26 backbone and are tuned to setting (details are included in the Appendix).
SimpleShot 68 is simple and competitive low-shot baseline. A CNN-based backbone is trained with cross-entropy on the base classes and tested with nearest class mean classification.
RFS[62] is strong but simple baseline, trained with cross-entropy on the base classes, and logistic regression-baser classifier trained on the support data each low-shot episode.
FEAT(76] finetunes transformer- based [64] set-to-set function on top of a cross- cross-entropy eetropy-traiied
backbone. Methods that outperform FEAT on minilmageNet like COSOC 40 so by eliminating learning classification by focusing the background. As there is no correlation between classes and backgrounds in our synthetic setting, FEAT represents the state of the in low-shot learning in our synthetic data comparisons.
7

---

Similarity Heatmap Views
Similarity Heatrap Across Instances
Figure 6: Similarity heatmap for the query point (yellow) Left: across different views of the same object and Right: across different object instances from CO3D. The blue point on the heatmap- overlaid image indicates the predicted correspondence of the query point. Note that these objects from validation or test categories, which are different from the training categories.
Table Results on image-only low-shot recognition on the ModelNet40-LS,, DOPE trained on ABC outperforms supervised baselines and the global self supervised learning approaches. Dashed line separates supervised and self- superviseedd methods.
5-way
10-way
Episode Setup
I-shot
5-shot
1-shot
5-shot
SimpleShot 68
56.55 #0.42
69.87
#0.32
41.27
+0.24
54.84
#0.18
RFS 662
57.31 #0.40
73.77
#0.33
42.22
+0.34
59.97
#0.18
FEAT 776
57.46 10.39
71.73
10.32
41.72
10.24
57.84
10.18
SupMoCo
55.32 10.40
71.82
10.33
39.87
4:0.3
57.15
10.17
VISPE 27
56.27 20.44
67.76
20.35
40.41
51.97
10.18
VISPE++- SimSiam 7
53.83 10.29
20.25 68.75
39.84
1:017
$4,34
10.13
VISPE++ MolCoV 25
$7.05 #0.42
+0.35 71.81
43.23
#0.25
58.68
#0.18
DOPE (ours)
57.51 #0.44
70.44
+0.36
42.73 +0026
55.52 #0.19
VISPE++ SimSiam
?
ABC
60.24 10:28
76,55
4:0.22
47.02 1.0.18
64,51
+:0.33
VISPE++ MoCoV2
25 ABC
61.07 40.41
75.96
.0.32
47.67
4:0.25
63.27
10.18
DOPE (ours) ABC
62.76 1:0.43
76,86
-0.33
49.39
10.26
64.77
10.18
DOPE (ours) ABC
66
pred. mask
57.63 40.40
71.62 #0.31
43.43 +0.24
57.72 10.18
Su uses labeled base-class dataset to perform contrastive training, and trains
cosine-classifier (20) for each low-shot episode. We finetune a logistic regression- classifier in RFS due to the smaller episode sizes in our setting, upon consulting the authors. We found that it results in improved performance. SupMoCo the state of the art for low-shot learning on
MetaDataset [63]
22 is a self- self-supe baseline that uses a temperature-scaled cross-entropy to learn global features so that two views of the same object are positives and two views of different objects are negatives. We use a nearest class mean classifier in the learned global feature space for low-shot classification as we found that it gives the highest performance.
VISPE++ We improve the original VISPE implementation by applying more recent contrastive learning techniques based on either MoCoV2 [6) or SimSiam (7], resulting in stronger baselines. We use 1 --nearest neighbor classifier the learned global feature space for low-shot classification as we found that it gives the highest performance.
5.3 DOPE trained on ABC outperforms supervised and self-supervised baselines on ModelNet and ShapeNet
We evaluate DOPE's ability to directly perform low-shot classification without any category su- pervision during representation learning on synthetic data, and demonstrate improved low-shot generalization ability over both supervised and self supfrsupe baselines when trained on the ABC dataset. The results are presented in Table | and Table 2] observe that when trained on ABC without any category labels, DOPE outperforms the self-supervised baselines. Note that both the SimSiam (7] and MoCoV2 (25) versions VISPE+ and our proposed DOPE outperform supervised baselines by large margin when trained ABC.

---

Table 2: Results image-only low-shot recognition the ShapeNet-LSS. DOPE trained ABC outperforms supervised baselines and outperforms the global self supervised learning approaches. Dashed line separates supervised and self sulf--upe methods.
5-way
10-way
Episode Setup +
1-shot
5-shot
1-shot
5-shot
SimpleShot 68
58.07
#0.45
70.06
43.00
#0.29
55.96
#0.25
RFS 162
57.93 #0.46
73.23
#0.37
43.08
#0.28
59.71
+-0.25
FEAT 766
57.83 =0.45
72.41
20.37
42.92
20.29
58.95
10.25
VISPE
57.69 =0.47
68.65
#0.39
42.43
20.29
54.00
#0.21
VISPE++ SimSiam 7
53.22 #0.30
66.75
#0.31
39.12
10.19
52.25
#0.17
V 25
55.83 #0.42
69.11
20.36
41.14
+0.28
54.86
+0.25
DOPE (ours)
58.64 20.48
70.43
20.38
44.26 +0.30
56.07
10.26
VISPE++ SimSiam 7
ABC
58,48 10.32
72.19
20.34
44.59
10.20
59.17 1:018
VISPE++ MoCov2
25
ABC
58.99 10.46
72.26
20.39
44.96
10.30
59.02
2:0.6
DOPE (ours) ABC
62.00 20.48
73.55
20.38
47.93
10.31
60.72 10.27
Table 3: Results on image-only low-shot recognition on CO3D-LS. DOPE outperforms the global self supervised learning-onf, approaches. Dashed line separates supervised and self-supervised methods.
5-way
10-way
Episode Sctup
1-shot
5-shot
1-shot
5-shot
SimpleShot 68
62.44 10
79.18
+0.29
49.43
10.26
68.32
10.17
RFS
62.59 10.41
81.50
10.26
50.16
+0.26
71.89 40.16
FEAT 76
64.48 10.43
79.04
#0.29
51.58
::226
69.66 10.16
SupMoCo
62.34 +0 43
78.72
#0.44
48.79
+0.27
67.84 10.16
VISPE
54,68 10.45 43
70.12
20.35
40.85
+0.25
56,46 10.20
VISPE++ (SimSiam)
56.02 20.27
74.29
#0.21
43.45
10.17
62.88
10.12
VISPE++ (MoCoV2)
60.25 +0.41
76.30
#0030
47.09
40.26
64.98 10.18
DOPE (ours)
61.77 20.44
75.16
#0.32
48.07 40.27
62.97 10.17
We also investigate removing the assumption of known foreground masks for training DOPE, by extracting foreground masks for the ABC dataset using the off-the-shelf unsupervised instance segmentation method FreeSOLO [66]. The result is presented in the bottom row of Table I We observe that even when the assumption of known foreground masks is removed, DOPE's performance drops but remains competitive with supervised baselines when trained on ABC.
5.4 DOPE trained CO3D outperforms other self-supervised baselines
We also evaluate DOPE's low-shot generalization ability on real data and observe that it has improved low-shot generalization ability over other self-supervised baselines. The results are presented in Table 3. We observe that DOPE can successfully be trained on real data with estimated scene geometry and that it outperforms self- eeffsuupervised learning baselines that do not perform any local feature learning in the 1-shot scenario by up to 1.5%-points.
5.5 DOPE Qualitative Results
We present qualitative results for dense object correspondences on ShapeNet in Figure and for CO3D (33 in Figure DOPE learns to map the same object parts across different views of the same instance and generalizes further to different object instances in different viewpoints (the front and back legs of the chairs are correctly matched in Figure 5 and the rear light the cars in Figure (6). Note that for CO3D the local representations were trained different set of object categories, and were trained on the ABC dataset for ShapeNet. Further, we observe an intriguing property that the model is uncertain for object parts that are similar, shown in the similarity heatmap in Figure (e.g. the wheels of the skateboard). This highlights the ability of DOPE to successfully encode local features of objects and generalize unseen data.
5.6 Negative Sampling Strategy
We present an ablation study in Table to understand the effect of how negatives are sampled on low-shot classification performance. Recall from the left side of Figure and Section 3.1 that while positives only be obtained by comparing corresponding pixels in two views of one object,

---

Table 4: Ablation study over different strategies for obtaining negative samples for contrastive training. We observe that is essential to sample from the second object view and from other object instances in the batch to train the best representation. All models are trained on ABC and evaluated on ModeINer-O-L.S.
5-way
10-way
Episode Setup
1-shot
5-shol
1-shot
5-shol
DOPE ABC 2nd view + other objects
62.76 #0.45
76.86
#0.33
49.39
=0.25
64.77
#0.18
DOPE ABC other objects only
61.51
#0.42
75.
10.32
47.95
#0.25
62.65
+0.18
DOPE ABC 2nd view only
43.50 #0.30
56.58 #6.28
30.58
=0.25
39.06 #0.17
Table 5: We investigate the tradeoff between using two views of an object corresponding to two different camera positions in the 3D world against two "views" generated by augmenting a single image twice. We find that using two physically different viewpoints during training has a high impact on generalization ability.
5-way
10-way
Episode Setup 7
I-shot
5-shot
1-shol
5-shol
DOPE - ABC multi-view augmented image pairs
62.76 =0.45
76.86
=0.33
49.39
#0.26
64.77
#0.18 37.19 10.18
DOPE ABC single-viw augmented image pairs
38.36
00.43
51.68
10.33
25.78
10.26
negatives come either from non-corresponding points in the second view of an object or from other objects. We observe that is essential to both sample negatives from the second view and from the other objects in the batch obtain the best representation.
5.7 Training with Multiple 3D Views of Objects is Essential
While prior dense contrastive learning works 46 67 73
generate training pairs by augmenting a single 2D image of an object, our method is trained using two images corresponding to two different camera placements in 3D space. To understand the importance of different 3D viewpoints of an object, train DOPE using the same loss but generate image pairs by applying two augmentations of the same image instead (see Table (5). We find that augmenting the same image to generate view pairs is insufficient for low-shot generalization.
6 Limitations and Discussion
The main limitation of our proposed approach is its requirement of estimated scene geometry in order derive pixel-Ievel correspondences between multiple views of object instances. While this information allows us to train better self-supervised representations that do not require category labels, it comes at a computational cost when applied to real datasets. Future work may include developing means to apply contrastive learning at the local level but without requiring explicit pixel-level correspondences, potentially making use of the temporal contiguity of videos.
Negative Societal Impact Contrastive training generally requires long training runs with multiple GPUs, which has potential for negative environmental impact. This may be addressed in the future through improvements in chip design and optimization of deep models.
7 Conclusion
Progress in few-shot and self-supervised learning is essential for overcoming the current requirements for large labeled datasets. This paper presents self-supervised method that learns from multiple views of object instances and can recognize novel categories based on few labels. Our findings are validated on both synthetic and real datasets. Qualitatively, can match semantic elements on the object surface (e.g. tires, chair legs) in a way that generalizes to novel object categories. Developing means perform local contrastive learning without explicit multi-view pixel-level correspondences is an exciting direction for future work.
observe that our proposed approach
Acknowledgement
This work was supported in part by NIH R01HD104624-01A1. NIH ROIMH11499) NSF OIA2033413, and gift from Facebook. We thank Miao Liu and Wenqi Jia for their helpful feedback and discussion.
10

---

References
[1] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems, 33:9912-9924, 2020.
[2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
[3] Angel Chang, Thomas Funkhouser. Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich model repository. arXiv preprint arXiv: 1512.03012, 2015.
[4] Ting Chen, Simon Kornblihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] Wei-Yr Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. In International Conference on Learning Representations, 2018.
[6] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming Hc. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.00 04297, 2020.
[7] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[8] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Exploring simple meta-learning for fow-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9062-9071 October 2021.
[9] Yunlu Chen, Vincent Tao Hu, Efstratios Gavves, Thomas Mensink, Pascal Mettes, Pengwan Yang. and Cees GM Snoek. Pointmixup: Augmentation for point clouds. In European Conference on Computer Vision, pages 3300-45. Springer, 2020.
[10] Zezhou Cheng, Jong-Clin -hyy Su, and Subhransu Maji. On equivariant and invariant learning of object landmark representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9897-9906, 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255, 2009.
[12] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings the IEEE international conference computer vision, pages 1422-1430, 2015.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[14] Mohamed El Banani and Justin Johnson. Bootstrap your own correspondences. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6433-6442. 2021.
[15] Mohamed El Banani and Justin Johnson. Bootstrap your own correspondences. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6433 6442, October 2021.
[16] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 264-26 262, 2018
[17] Chelsea Finn, Pieter Abbecl, and Sergey Levine. Modcl-agnostic meta-lcarning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1126-1133 135, 2017.
[18] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model- meta-learning. In Advances in Neural Information Processing Systems, pages 9516 077669557,24 2018

---

[19] Peter R Florence, Lucas Manuelli, and Russ Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In Conference on Robot Learning, pages 373-385. PMLR, 2018.
[20] Spyros Gidaris and Nikos Komodakis, Dynamic few-shot visual learning without forgetting. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2018.
[21] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07723 2018.
[22] Jean-Bastie Man-Baatienn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-xupervised learning. arXiv preprint arXiv:2006.07733, 2020.
[23] Denis Hadjivelichkov and Dimitrios Kanoulas. Fully self-supervised class awareness in dense object descriptors. In Conference on Robot Learning, pages 1522-1531. PMLR, 2022.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation lcarning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[27] Chih-Hui Ho. Bo Liu, Tz- Ying Wu, and Nuno Vasconcelos. Exploit clues from views: Self-supervised and regularized learning for multiview object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9090-9100 2020.
[28] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene under- standing with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15587-15597, 2021. 2021.
[29] Ji Hou, Saining Xie, Benjamin Graham. Angela Dai, and Matthias NieBner. Pri3d: Can 3d priors help 2d representation lcarning? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5693-5702 2021.
[30] Han Hu, Jiayuan Gu, Zheng Zhang. Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3588 3597, 2018.
[31] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov. Ming-Hsuan Yang, and Jan Kautz. Scops: Self- Sepe-super co-part segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 869-87 878, 2019.
[32] Dinesh Jayaraman, Ruohan Gao, and Kristen Grauman. Shapecodes: supf-supe feature learning by lifting views to viewgrids. In Proceedings of the European Conference on Computer Vision (ECCV), pages 120 136, 2018.
[33] Alexander Kirillov. Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6399-6408, 2019.
[34] Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image segmentation as rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9799-999 9808,,98,,2220 2020.
[35] Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey Artemov, Evgeny Burnaev, Mare Alexa, Denis Zorin, and Daniele Panozzo. Abe: A big cad model dataset for geometric deep learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9601-9661601 1601-9611, 2019.
[36] Shamit Lal, Mihir Prabhudesai, Ishita Mediratta, Adam W Harley, and Katerina Fragkiadaki. Coconets: Continuous contrastive 3d scene representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11487-12496, 2021.
12

---

[37] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), pages 740-755, 2014.
[38] Yunze Liu, Li Yi, Shanghang Zhang, Qingnan Fan, Thomas Funkhouser, and Hao Dong. P4contrast: Contrastive learning with pairs of point-pixel pairs for rgh-d scene understanding. arXiv preprint arXiv:2012.13089, 2020.
[39] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
[40] Luo, Longhui Wei, Liangjian Wen, Jinrong Yang. Lingxi Xie, Zenglin Xu, and Qi Tian. Rectifying the shortcut learning background for few-shot learning. Advances in Neural Information Processing Systems, 34, 2021.
[41] Orchid Majumdor, Avinash Ravichandran, Subhramsu Maji, Alessandro Achille, Marzia Polito, and Stefano Soatto. Supervised momentum contrastive learning for few-shot classification. arXiv preprint arXiv:2101.1 1058, 2021.
[42] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cuns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 51 5115-5124, 2017.
[43] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pages 69 84. Springer, 2016.
[44] David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. Self-supervised learning of geo- metrically stable features through probabilistic introspection. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 3637-3645, 2018.
[45] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron Courville. Un- supervised learning of dense visual representations. Advances in Neural Information Processing Systems. 33:4489-4500, 2020.
[46] Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Un- supervised learning of dense visual representations. Advances in Neural Information Processing Systems, 33:4489 4500, 2020.
[47] Polyhaven. Poly haven, https:/poolhavve.comm/
[48] Mihir Prabhudesai, Shamit Lal, Darshan Patil, Hsiao Yu Tung, Adam W Harley, and Katerina Fragkiadaki. Disentangling 3d prototypical networks for few-shot concept learning, arXiv preprint arX3v.2011.03367.
2020.
[49] Blender Project. Blender, https://blender.org..
[50] Charles Qi. Hao Su, Kaichun Mo, Leonidas Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 652-660, 660, 2017.
[51] Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cnns for object classification 3d data. In Proceedings the IEEE conference computer vision and pattern recognition, pages 5648-5653 5656-55566750 2016.
[52] Charles Ruizhongtai Qi, Yi, Hao Su, Leonidas Guibas. Pointnet4+:: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pages 5099-5108, 2017.
[53] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901-10911, 2021.
[54] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprini arXiv:1807.050000 2018.
[55] Johannes Lutz Schönberger and Jan-Mich Michael Frahm. dtructurefrrom motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
13

---

[56] Johannes Lutz Schönberger. Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV). 2016.
[57] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 408774087, 2017.
[58] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
[59] Stefan Stojanov, Anh Thai, and James M. Rehg. Using shape to categorize: Low-shot learning with explicit shape bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1798-1808, June 2021.
[60] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Milles Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference computer vision, pages 945-953, 2015.
[61] Yonglong Tian, Olivier J Henaff, and Aäron van den Oord. Divide and contrast: Self- selfrvuuee learning from uncurated data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10074, 2021.
[62] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua Tenenbaum, and Phillip Isola. Rethinking fow-shot image classification: good embedding is all you need? In European Conference on Computer Vision (ECCV) 2020, August 2020.
[63] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu. Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations. 2019.
[64] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[65] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
[66] Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. FreeSOLO: Learning to segment objects without annotations. arXiv preprint 1111:2200..21881 2022.
[67] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self- supervupe visual pre-training. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 0024-3033, 2021,
[68] Yan Wang. Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting neiieot classification for shot learning. arXiv preprint 1XXiv:991100462. 2019.
[69] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph ean for learning on point clouds. Acm Transactions On Gruphics (log), 38(5):1-12, 2019.
[70] Xin Wei, Yifei Gong, Fudong Wang, Xing Sun, and Jian Sun. Learning canonical view representation for 3d shape recognition with arbitrary views. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 407-416, 2021.
[71] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-192 1920, 2015.
[72] Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero- shot lcarning, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5542-5551, 2018
[73] Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10539-10548, 2021.
[74] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In European conference on computer vision, pages 574-591. Springer, 2020.
14

---

[75] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. In Proceedings the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 16684-16693, 2021.
[76] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2020.
[77] Lin Yen-Chen, Pete Florence, Jonathan Barron, Tsung- Yi Lin, Alberto Rodriguez, Phillip Isola. Nerf- supervision: Learning dense object descriptors from neural radiance fields. arXiv preprint arXiv:2203.01913, 2022.
[78] Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In European conference on computer vision, pages 149-666. Springer, 2016.
[79] Adrian Ziegler and Yuki M Asano. Supe-super learning of object parts for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
15


Reviews:
[{"review_id": "oxyeXLzr7Ki", "reviewer": "Reviewer_7WGn", "summary": "This paper aims to learn representative features from multi-view images of objects without requiring category labels. The authors propose a Deep Object Part Encodings (DOPE) framework that leverages sparse depths, foreground masks and camera poses to obtain pixel-level corresponds between different views and performs self-supervised learning from these views. DOPE works well on low-shot classification of novel categories.", "questions": "Please refer to weakness. My major concern is the foreground mask is not always available. It is necessary to show DOPE can work well with unsupervised foreground masking methods.\n\nUpdate:\nA minor update of review comments:\nI think it is fine that the author can also leverage class-agnostic segmentation models or segmentation models pre-trained on other classes instead of unsupervised methods. In other words, any segmentation models that are NOT trained on the target categories will be fine for me. Please feel free to reply this question.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The overall idea is interesting and novel. Compared to previous work [1], this paper explicitly leverages the information of 3D geometry in multi-view images for self-supervised learning.\n2) The performance is good. DOPE outperforms other counterparts and performs competitively with the supervised methods.\n3) The paper is well written.\n\nWeaknesses:\n\nThe key weakness is that DOPE relies on ground-truth foreground masks, which will limit its application. It is necessary to explore using unsupervised foreground masking methods (e.g., [2]) to extract masks and show if DOPE can still work well.\n\n\n\n[1] Exploit clues from views: Self-supervised and regularized learning for multiview object recognition. \n\n[2] Unsupervised Foreground Extraction via Deep Region Competition\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aGOAChi-fjd", "reviewer": "Reviewer_tG2S", "summary": "-- Post rebuttal update --\nThe authors have addressed all the questions and weaknesses in my original review using new experiments. In particular the main thesis of the paper has become clear to me now, that it is a comparison between geometric and semantic supervision for low shot learning. The authors have agreed to change their title and description to reflect that the method is patch level and not part level. New experiments show that the methods is much better than Vader and the message remains the same even after supervised baselines are given the same data augmentation. Multi object experiments show that the method is not severely limited to single object datasets as I had originally anticipated. I have updated my rating to 7.\n-- End of post rebuttal update ---\n\nThis paper presents a method for learning grid structured embeddings from images, called DOPE.\n\nDOPE leverages pairs of images which are two views of a single object with known foreground segmentation, depth maps, camera parameters and camera poses. This information is used to derive pixel level correspondence between the two images which create a positive pair for contrastive learning. Other pairs are treated as negatives. Contrastive learning is used to learn the required embeddings which exhibit the following properties.\n\n1. Embeddings can be used to match across instances of the same category.\n2. Such properties extend to novel classes that were not used in training.\n\nThese properties combined are used to do nearest neighbor classification to demonstrate SoTA low-shot classification.", "questions": "### More questions\n- Why use the predicted foreground mask, \\hat{M}, instead of the ground truth mask, M, in line 182. Same in line 192?\n\n- Is PointRend in line 236 unsupervised / self-supervised?\n\n### Minor comments\n- Please discuss the observation that local embeddings work better than global ones again in the experiments. Which two rows should the reader compare to know the performance gap? Is this an apples-to-apples comparison?\n\n- In section 2.1, please state which bucket DOPE falls into. \"Metric-learning\".\n\n- Do supervised baselines also get the random background masking data augmentation?\n\n### Typos\n- Line 2: datasets for training feature representations\n- Line 151: For illustration 'p'lease refer to Figure 2.\n- Line 245: trained on the support data", "limitations": "Limitations around data simplicity need to be discussed in section 7. Can DOPE be pre-trained using say the epic kitchens dataset?\n\nSocietal impact has been adequately discussed focusing on the environmental effects of training these models.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The main thesis of this paper is that local features matching provides better downstream generalization for low-shot recognition. This has been supported by comparing against baselines.\n\n- Results include both real and synthetic data and compare against both supervised and self-supervised prior work.\n\n- Qualitative visualizations show that the embedding is indeed local. For instance the front of the motorcycle in figure 5 has a different embedding than the middle and back of the motorcycle. Such embeddings might be useful for more fine-grained tasks in future work.\n\n- The tables show that DOPE performs well, beating even supervised baselines on the ModelNet40-LS dataset.\n\n### Weaknesses and questions.\n- While the method does not use category labels for self-supervised pre-training, it requires foreground segmentation, depth maps, camera parameters and camera pose. This is very rich information which they derive using COLMAP for the CO3D-LS dataset, but that might not be work well for in-the-wild large scale video corpora. This limitation has been discussed in section 7.\n\n- The paper refers to embeddings as capturing object parts. However, nothing in the formulation is part specific, rather it goes down to the pixel or patch level. The visualization show a continuous variation from one point on the object to another as opposed to a part level segmentation emerging from the algorithm. The presentation would be much clearer if the paper referred to these as patch embeddings from the beginning. Both the paper title and the method name convey that it is \"part\" based which is confusing.\n\n- All experiments and the formulation assume that there is only one object in the scene. This wasn't discussed explicitly even in section 7. Specifically, what can we expect from DOPE when trained in cluttered natural environments?\n\n- Missing comparison against VADeR (SimCLR style contrastive learning but at the pixel level using two views generated by data augmentation) [1], FLow-E (Similar to VADeR but using BYOL style loss instead of SimCLR) [2], DenseCL [3] or PixPro [4], some recent local embedding learning methods. In fact the architecture borrows from VADeR. FLow-E code is not available. But PixPro has public code [here](https://github.com/zdaxie/PixPro), DenseCL [here](https://github.com/WXinlong/DenseCL). Basically what does a SoTA self-supervised local-embedding learning method do on these datasets? On a related note, how does DOPE perform when the two views come from data augmentation?\n\n[1] Pedro O O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. NeurIPS 2020.\n\n[2] Xiong, Yuwen, et al. \"Self-supervised representation learning from flow equivariance.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3] Wang, Xinlong, et al. \"Dense contrastive learning for self-supervised visual pre-training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[4] Xie, Zhenda, et al. \"Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "On18SlBTImv", "reviewer": "Reviewer_BZQB", "summary": "This paper proposes a novel method for low-shot category generalization via the object part correspondence learning. The object part correspondence is learnt by contrastive learning on multiple views of object instances. The anchors are constructed by positive features at pixel locations of the same 3D point on the object surface and negatives features from different points on the object surface. To obtain the dense object feature, the autors propose a Deep Object Part Encoding (DOPE), which is a simple framework with the Panoptic FPN. During the inference, several keypoints of the shot image are first sampled and extract the pixel-level feature by DOPE. Then, the keypoint features are compared with all pixel feature in gallery image to obtain the similarity score. Finally, the nearest neighbor method is used to determine the category.", "questions": "As mentioned before, the proposed method requires more information, i.e., depth and camera pose. Although these information are free for synthetic data, it still not very easy to capture for real images. I wonder if you have tried to transfer the model trained on synthetic data to the real images directly or with some tricks. I think these results could be interesting and if the generalization ability is not so bad, the contribution of this work would be improved.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method is well-defined and esay to understand and it outperforms the state-of-the-arts approachs on few shot classification.\n\nThe authors conduct comprehensive experiments on several benchmarks including both synthetic data and real-world data. And the consistent improvement can be observed on different few shot learning setting.\n\nWeakness:\n\nIn Figure4, the k pixels on the object surface are randomly sampled from each shot image and compare them with all pixel-wise feature of query image, however, from Line #192 to Line #200, the k pixels are sampled from the query image while compare them with all pixel-wise feature of  the shot image. These two description are conflict, and whick one is correct?\n\nAlthough this paper claims it doesn't require any category-level annotations, but it still need lots of additional supervision signal, i.e., foreground mask, depth map and camera pose during training. I think it is unfair to compare the proposed method with the existing work. Especially, the depth map actually involves lots of geometry information than the pure RGB image.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MwSv3frBshb", "reviewer": "Reviewer_3MZa", "summary": "The paper proposes a self-supervised learning task to perform similarity metric learning across multiple view for the local representation of an object with the assistance of sparse depths, foreground masks, and known camera parameters (from COLMAP.) It achieves superior performance even better than some supervised baseline methods for low-shot classification of novel categories.", "questions": "It is great that the authors show that to enforcing the self-supervised loss for the local invariant representation of an object across different view can benefit low-shot classification. However, I have few questions:\n\n1. How are the views sampled for the training? If the object views are too extreme, the estimation of COLMAP may fail.\n2. How do the preprocessing errors from the camera parameters, sparse depth, and segmentation map affect the performance? Although w in appendix, the authors do provide the ablation studies of w/o random background remove and w/o mask prediction.\n3. Is the 3D-based self-supervision necessary since the correspondence may have errors? Will other some other 2D-based self-supervision-based method for correspondence learning for an image with different transformations could also be used to assist the proposed method? Can the author have a short discussions about them with the proposed method. For some examples, there are 2D-based self-supervised learning work for co-part segmentation and part correspondence.\n\nCo-part segmentation:\n1. Hung, Wei-Chih, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz. \"Scops: Self-supervised co-part segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 869-878. 2019.\n2. Ziegler, Adrian, and Yuki M. Asano. \"Self-Supervised Learning of Object Parts for Semantic Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14502-14511. 2022.\n\nSemantic correspondence:\n1. Novotny, David, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. \"Self-supervised learning of geometrically stable features through probabilistic introspection.\" In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3637-3645. 2018.", "limitations": "Yes, the authors do address the limitations of the proposed method and potential negative societal impact of their work at the end of the main paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The work shows that by performing metric learning for the local representation of an object across multiple view could significantly improve the low-shot classification of novel categories.\n2. They conduct the experiments on both synthetic and real datasets and achieve consistent improvement for different scenarios.\n\nWeakness:\n\n1. The evaluation part is insufficient. The compared methods are all before 2021, and there is no 2022 method for comparison, and only two self-supervised baseline method for comparison, VISPE and VISPE++, which are proposed in 2020.\n2. The training heavily relies on the accuracy of the camera parameter estimator, COLMAP and foreground segmentation method, PointRend. There is not much discussion when the preprocessing results are noisy.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept