Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations
Authors: Xiang Gao
Wei Hu
Guo-Jun Qi
Venue:  (2021)
Keywords: Self-supervised Learning
Multi-View Learning

Abstract: 3D object representation learning is a fundamental challenge in computer vision to draw inferences about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based methods have performed best so far. However, feature learning of multiple views in existing methods is mostly trained in a supervised fashion, which often requires a large amount of data labels with high cost. Hence, it is critical to learn multi-view feature representations in a self-supervised fashion. To this end, we propose a novel self-supervised learning paradigm of Multi-View Transformation Equivariant Representations (MV-TER), exploiting the equivariant transformations of a 3D object and its projected multiple views. Specifically, we perform a 3D transformation on a 3D object, and obtain multiple views before and after transformation via projection. Then, we self-train a representation learning module to capture the intrinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after transformation. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks.

Under review as a conference paper at ICLR 2021
SELF-SUPERVISED MULTI-VIEW LEARNING AUTO-ENCODING 3D TRANSFORMATIONS
VIA
Anonymous authors Paper under double-blind review
ABSTRACT
3D object representation learning is fundamental challenge in computer vision to draw inferences about the 3D world. Recent advances in deep learning have shown their efficiency in 3D object recognition, among which view-based meth- ods have performed best so far. However, feature learning of multiple views existing methods is mostly trained in a supervised fashion, which often requires large amount of data labels with high cost. Hence, it is critical to learn multi-view feature representations in self-supervised fashion. To this end, we propose a novel self-supervised learning paradigm of Multi- View Transformation Equivari- ant Representations (MV-TER), exploiting the equivariant transformations of 3D object and its projected multiple views. Specifically, we perform 3D transforma- tion on 3D object, and obtain multiple views before and after transformation via projection. Then, we self-train representation learning module to capture the in- trinsic 3D object representation by decoding 3D transformation parameters from the fused feature representations of multiple views before and after transforma- tion. Experimental results demonstrate that the proposed MV-TER significantly outperforms the sfte-of--hhe-r view-based approaches in 3D object classification and retrieval tasks.
INTRODUCTION
3D object representation has become increasingly prominent for a wide range of applications, such
as 3D object recognition and retrieval (Maturana & Scherer 2015 Qi et al. 2016 Brock et al. 2016
Qi et al. 20 Klokov & Lempitsky 2017 Su et al.) 2015 Feng et al.) 2018 Yu et al. 2018
Yang & Wang 2019). Recent advances in Convolutional Neural Network (CNN) based methods
have shown their success in 3D object recognition and retrieval (Su et al. 2015 Feng et 2018 Yu ct al. 2018 Yang & Wang 2019). One important family of methods are view-based methods, which project a 3D object into multiple views and learn compact 3D representation by fusing the feature maps of these views for downstream tasks. Feature learning of multiple views in existing approaches are mostly trained in supervised fashion, hinging on a large amount of data labels that prevents the wide applicability. Hence, self-supervised learning is in demand to alleviate the dependencies on labels by exploring unlabeled data for the training of multi-view feature representations in an unsupervised (seri-)supervised fashion.
Many attempts have been made to explore self-supervisory signals at various levels of visual struc- tures for representation learning. The self-supervised learning framework requires only unlabeled data in order to formulate pretext learning task (Kolesnikov et al. 2019), where a target objective can be computed without any supervision. These pretext tasks can summarized into four cate- gories (Jing & Tian 2019): generation-based (Zhang et al.] [016)] Pathak et al. 2016]] Srivastaval al. 2015), context-baseen free semantic label-based (Faktor Irani 2014||Streteu Leordeanu,
2015 Ren & Jac Lee, 22018) and cross modal-basce (Sayed et | [018 Korbar et al.] 2018). Among
them, context-based pretext tasks include representation learning from image transformations, which
well connected with transformation equivariant representations as they transform equivalently as the transformed images.
Transformation Equivariant Representation learning assumes that representations equivarying to transformations are able to encode the intrinsic structures of data such that the transformations can be reconstructed from the representations before and after transformations (Qi 2019). Learning

---

Under review as a conference paper at ICLR 2021
transformation equivariant representations has been advocated in Hinton's seminal work learning transformation capsules (Hinton et al.] 2011). Following this, a variety of approaches have been
proposed to learn transformation equivariant representations (Kivinen & Williams [011]] Sohn &
Lee 2012 Schmidt & Roth 2012 SSkibee, 2013 Lenc & Vedaldi 2015 Gens & Domingos 2014]
Dieleman et al. 2013 2016 Zhang et al.] Qi al.1 Gao al. 2020 Wang al.. 2020).
Nevertheless, these works focus on transformation equivariant representation learning of a single
modality, such as 2D images 3D point clouds.
In this paper, we propose to learn Multi-Viers Transformation Equivariant Representations (MV- TER) by decoding the 3D transformations from multiple 2D views. This is inspired by the equivari- ant transformations of a 3D object and its projected multiple 2D views. That is, when we perform 3D transformations on a 3D object, the 2D views projected from the 3D object via fixed view- points will transform equivariantly. In contrast to previous works where 2D/3D transformations are decoded from the original single image/point cloud and transformed counterparts, we exploit the cquivariant transformations of 3D object and the projected 2D views. We propose to decode 3D transformations from multiple views of a 3D object before and after transformation, which taken as self-supervine supervisorr regularization enforce the learning of intrinsic 3D representation. By estimat- ing 3D transformations from the fused feature representations of multiple original views and those of the equivariantly transformed counterparts from the same viewpoints, we enable the accurate learning of 3D object representation even with limited amount of labels.
3D object (e.g., point clouds, meshes), and render the original and transformed 3D objects into multiple 2D views with fixed camera setup. Then, we feed these views into representation learning module to infer representations of the mul- tiple views before and after transformation respectively. A decoder is set up to predict the applied 3D transformation from the fused representations of multiple views before and after transformation. We formulate multi-view transformation equivariant representation learning as regularizer along with the loss of a specific task (e.g., classification) to train the entire network end-to-end. Exper- imental results demonstrate that the proposed method significantly outperforms the state-of-the-art visw-based models in 3D object classification and retrieval tasks.
Specifically, first perform 3D transformation
Our main contributions are summarized as follows.
We propose Multi View Transformation Equivariant Representations (MV-TER) to learn 3D object representations from multiple 2D views that transform equivariantly with the 3D transformation in a self-supervised fashion.
We formalize the MV-TER as a self-supervisory regularizer to learn the 3D object repre- sentations by decoding 3D transformation from fused features projected multiple views before and after the 3D transformation of the object.
Experiments demonstrate the proposed method outperforms the state-of-the-arr view-based methods in 3D object classification and retrieval tasks in a self-suue fashion.
2
RELATED WORKS
In this section, we review previous works on transformation equivariant representations and multi- view based neural networks.
2.1
TRANSFORMATION EQUIVARIANT REPRESENTATIONS
Many approaches have been proposed to learn equivariant representations, including transform- ing auto-encoders (Hinton al. 2011). equivariant Boltzmann machines (Kivinen & Williams, 2011 Sohn & Lee, (012)) equivariant descriptors (Schmidt & Roth 2012). and equivariant filtering (kkbbeel 2013). Lenc Vedaldi (015) prove that the AlexNet (Krizhessky et al.] 2012) trained
on ImageNet learns representations that are equivariant to flip, scaling and rotation transformations.
Gens & Domingos (2014) propose an approximately equivariant convolutional architecture, which utilizes sparse and digh-dimensional feature maps to deal with groups of transformations. Dieleman ce al. (2015) show that rotation symmetry can be exploited in convolutional networks for effectively learning equivariant representation. Dieleman et al. (2016) extend this work to evaluate other computer vision tasks that have cyclic symmetry. Cohhnn & Welling (2016) propose group equivari- ant convolutions that have been developed equivary more types of transformations. The idea of
2

---

Under review as a conference paper at ICLR 2021
group equivariance has also been introduced to the capsule nets (Lenssen et al. 2018) by ensuring the equivariance of output pose vectors to a group of transformations.
To generalize generic transformations, Zhang al.) (2019) propose to learn unsupervised fea- ture representations via Auto-Encoding Transformations (AET) by estimating transformations from the learned feature representations of both the original and transformed images. [Q al. (2019)
extend AET by introducing a variational transformation decoder, where the AET model is trained from an information-theorede
perspective by maximizing the lower bound of mutual information. Gao et al. 2020) extend transformation equívariant representations to graph data that are irregularly structured, and formalize graph transformation equivariant representation learning by auto- eetoding wode transformations in an unsupervised manner. [Wang et al. (2020) extend the AET to Gener- ative Adversarial Networks (GANs) for unsupervised image synthesis and representation learning.
2.2 MULTI- VIEW LEARNING
Recently, many view-based approaches have been proposed for 3D object learning. These methods project 3D objects (e.g., point clouds, meshes) into multiple views and extract view-wise features receptively via CNNs, and then fuse these features as the descriptor of 3D objects. [Su al. (0113) first propose a multi-view convolutional neural network (MVCNN) to learn a compact descriptor of an object from multiple views, which fuses view-wise features via max pooling layer. Q et al. 2016) introduce a new multi-resolution component into MVCNNs, and improve the classification performance. However, max pooling only retains the maximum elements from views, which leads to information loss. In order to address this problem, many subsequent works have been proposed to fuse multiple view-wise features into an informative descriptor for 3D objects. |Feng et al. (2018) propose group-view convolutional neural network (GVCNN) framework, which produces com- pact descriptor from multiple views using a grouping strategy. Yu et al. (2018) propose a multi-view harmonized bilinear network (MHBN), which learns 3D object representation by aggregating local convolutional features through the proposed bilinear pooling. To take advantage of the spatial re- lationship among views, Han et al. (2088) and Han al. (2019) propose aggregate the global features of sequential views via attention bassd RNN and CNN, respectively. Kanezaki et al. (2018) propose to learn global features by treating pose labels as latent variables which are optimized to self-align in an unsupervised manner. Yang & Wang (2019) propose a relation network to connect corresponding regions from different viewpoints, and reinforce the information of individual view. Jiang et al.) (211)) propose Multi-Loop- Looli-Loop-View View Convolutional Neural Network (MLVCNN) for 3D object retrieval by introducing a novel loop normalization to generate loop-level features. |Wii et al.) (020) design view-based GCN framework to aggregate multi-view features by investigating rela- tions of views.
3 MV-TER: THE PROPOSED METHOD
In this section, we first define multi-view equivariant transformation in Section 3.1 Then we for- mulate the MV-TER model and introduce the MV-TER framework in Section 3.2 and Section 3.4 respectively.
3.1 FULTI-VIEN EQUIVARIANT TRANSFORMATION
2D views are projections of 3D object from various viewpoints, which transform in an equivariant manner as the 3D object transforms. Formally, given 3D object M e [gax3 consisting of n points and a 3D transformation distribution T, we sample a transformation T and apply it to M:
M = t(M).
We project M onto 2D views from m viewpoints, denoted as V (V1,.., Vn), ie..
V = (M).
(2)
where Ps : R + R² is projection function for the ith view. Subsequent to the transformation M. the views transform equivariantly, leading to D> (V1,... Vn). We have
V. pi [M] P4 (t(M)) = fi. (V.) 1,... m,
(3)
3

---

Under review as a conference paper at ICLR 2021
where f 's are 2D transformations that are equivariant under the same 3D transformation t. Though V; and V; are projected along the same viewpoint i (i.e,, the same camera setup), they are projec- tions of the original 3D object and its transformed counterpart, thus demonstrating different per- spectives the same 3D object. Our goal is to learn the representations of 3D objects from their multiple 2D views by estimating the 3D transformation t as pretext task from sampled multiple views before and after the transformation, i.e., V and V.
3.2 THE FORMULATION
Considering the ith views {V., Vi} before and after transformation t. function E(-) is transfor- mation equivariant if satisfies
E(V.) = (V.)) = p(t))((V,),
(4)
where p(t) is homomorphism of transformation t in the representation space.
We aim to train a shared representation module E(-) that learns equivariant representations of mul- tiple views. In the setting of self-superviser learning, we formulate MV-TER as a regularizer along with the (semi- sspeevisuee loss of specific task to train the entire network. Given a neural network with learnable parameters θ, the network is trained end-to-end by minimizing the weighted sum of two loss functions: 1) the loss of a specific task lask (e.g., a cross-entropy loss in 3D object clas- sification); and 2) the MV-TER loss that is the expectation of estimation error (M t) over each sample M given distribution of 3D objects M and each transformation t 'T:
min lask A E E t~T M-M
(M, tt.
(5)
(t, t) is the mean squared error (MSE) between the estimated transformation é and the ground truth t. weighting parameter to strike a balance between the loss of a specific task and the MV-TER loss. Here, the loss lask can be taken over all the data labels (fully-supervised) or partial labels (semi-supervised). In (5). is decoded as function of V and V in multiple views as defined in and and we will present two schemes to decode in the next subsection.
3.3 TWo TRANSFORMATION DECODING SCHEMES
Fusion Scheme. We propose two schemes to decode the transformation in (5) from the feature representations of multiple views E(V,) and E(V,), = 1.... m. The first scheme to decode from fused representations of multiple views (before and after transformations). Suppose the neural network extracts features of V, and V, from representation learning module EE-), and estimates the 3D transformation from both features via transformation decoding module D(-), then we have
6 6= D (F(EEEV))) E(V
B(Vm)) FF BO (E(V)),... E(V.))
(6)
where F(-) is function of feature fusion.
Average Scheme. the second decoding scheme, we estimate the transformation È from each view before and after transformation and then take the average of the estimates. The idea is each view captures the projected 3D structures under a transformation. This essentially models 3D object from different perspectives from which the underlying 3D transformation By averaging the estimated transformations across multiple views, a reasonable estimation of 3D transformation can be made.
be revealed.
This actually pushes the model to learn a good 3D representation from individual 2D views and leads to an estimation &, from the ith view:
? D (E(V),,, , L....,
The final decoded 3D transformation is taken as the expectation of f,
f
(8)
m
4

---

Under review as a conference paper at ICLR 2021
V1
R(V:
V2
(V::)
V/3
E
Fusion
Concatenation
D
(V
3D transformatton
shared weights
shared weights
Vi
E(V))
V4
((V-)
V
E
R(V.))
Fusion
T
Score
E(Vm)
Vm
The architecture of the proposed MV-TER in the fusion decoding scheme. E and D represent the feature representation module and transformation decoding module respectively, and specific task, e.g., 3D object classification or retrieval.
Figure 1:
Hence, we update the parameters θ in the representation learning module E(.) and the transforma- tion decoding module (--) iteratively by backward propagation of the regularized loss in (
Interestingly, the second decoding scheme can reach even better performance than the first scheme experiments. This should not be surprising since our ultimate goal is enable multi- view learning by fusing the representations of individual 2D views to reveal the target 3D objects. The second scheme follows this motivation by pushing each view to encode as much information as possible about the 3D transformation, as implied by the multi-view learning.
3.4 THE ALGORITHM
Given a 3D object M, we randomly draw transformation t T and apply it to M to obtain transformed M. Then have views V {VI. ...., 3 by projecting M 2D views. Accordingly, the views after the 3D transformation are V ( V..,. Vm).
To learn the applied 3D transformation t, we design end-to-end architecture as illustrated in Figure [ for the fusion decoding scheme, while the architecture of the average decoding scheme will be presented in Appendix A We choose existing CNN models as the representation learning module E(-) (e.g., AlexNet Krizhevsky et al. 2012), GoogLeNet (Szegedy et al. 2015)), ). which extract the representation of each view separately. The learned feature representations will be fed into a fusion module and transformation decoding module D(.) respectively. The fusion module is fuse the features of multiple views as the overall 3D object representation, e-g., by view-wise max-pooling layer (Su et al. 2013) or group pooling layer (Feng et al.] 2018). The fused feature will serve the general descriptor of the 3D object for the subsequent downstream learning tasks (e.g., classification and retrieval). The transformation decoding module D(.) is to estimate the 3D transformation parameters from the feature representations of multiple views. Next, we will discuss the representation learning module and transformation decoding module in detail.
3.4.1 REPRESENTATION LEARNING MODULE
The representation learning module E(-) takes the original 2D views V and their transformed coun- terparts V as the input. E-.) learns feature representations of V and 1 through a Siamese encoder network with shared weights. Specifically, we employ the feature learning layers of pre-trained CNN model as the backbone. Then, we obtain the features of each view before and after transfor- mation.
3.4.2 TRANSFORMATION DECODING MODULE
To estimate the 3D transformation t, concatenate extracted features of multiple views before and after transformation at feature channel, which are then fed into the transformation decoder. The decoder consists of several linear layers to aggregate the representations of multiple views for
5

---

the prediction of the 3D transformation. As discussed in Section 3.2 have two strategies for decoding the transformation parameters. We can decode from the fused representations multiple views before and after transformation as in (6), or from each pair of original and equivariantly transformed views (V,, V:) to take average for final estimation as in (8). Based on the loss (5). Under review as a conference paper at ICLR 2021
Table 1: 3D object classification and retrieval results ModelNet40 dataset.
Training Confguration
Classification (Accuracy)
Retrieval
Methods
Modality
Pre-train
Fine tune
(mAP)
VoxNer (Maturana & Scherer 2015
ModelNer40
voxels
83.0
Sub Qu
2016
ModelNorHO
voxels
89.2
Voxception ResNet [Brock et
2016
ModelNer-U
voxels
91.3
PointNet (Qi
70]
ModelNer40
points
89.2
PointNet+t (U)
70
ModelNeran
points
91.9
KD-Networks ( (Klokov & Lempitsky
2017
ModelNer40
points
91.8
(Su
I
ModelNera0
views
MVCNN, merre Sir ml
2015
ImageNet1K
ModelNer40
12 views
89.5
80.2
MVCNN, multi resolntton (U el
2016)
ImageNeilK
ModelNer40
20 views
93.8
RotationNet (Knezaki et
2018)
ImageNer1
ModelNer4n
12 views
90.7
MHBN Yu etal, (011))
Iet
ModelNer40
12 views
93.4
VegVeesZZeqLarree (Hap al.
IetgeNettK
ModelNeriU
views
89.1
3D2Sey Views [I Tan et al (2119
ImaggNNelIK
ModelNer40
12 views
91.4
90.8
Relation Network Yang Wang 2019)
IetgeNet1K
ModelNer4n
views
86.7
MLVCNN, Conter Loks Drakng
2019
I
ModelNer40
36 views
94.2
92.8
View-GCN (Wei et (bz2))
ModelNer40
12 views
96.2
MVCNN (Go)gLaNet)) [reng el 2018)
InngeNetLKK
ModeiNeciu
92.2
74.1
MVCNN (GoogLLaNet), metric (1 al. 2018
ImageNei
ModelNeran
12 views
92.2
83.0
MV-TER (MVCNN), average
Iet
ModelNer4n
views
83.0
MV-TER (MVCNN), fusion
ImageNet1K K
ModelNer40
12 views
93.1
84,9
MV-TER (MYCNN), average. Center Loss
ImageNetIK
ModelNeriu
12 views
95.1
86.6
MV-TER (MVCNN), Fusion. Center Lows
ImageNeilK K
ModelNer40
12 views
93.2
89.0
GVCNN (GoogLeNet) (Feng al. 2018
ImageNetIK
ModelNerAU
12 views
92.6
81.3
(GoogLeNeet) meine (reng et 2018
ImugeNetlK
ModelNer40
views
85.7
MV-TER (GVCNN), average
ImageNetIK
ModelNeran
12 views
97.0
88.8
MV-TER (GVCNN), fasion
I
ModelNer40
12 views
96.4
88.3
MV-TER (GVCNN), average, Center Loss
ImageNNtIK
ModelNer40
12 views
95.7
91.5
MV-TER (GVCNN), fusion, Center Loss
ImageNetIK
ModelNet40
12 views
96.3
91.1
t. is decoded by minimizing the mean squared error (MSE) between the ground truth and estimated
transformation parameters. We will show the estimated 3D transformations of the two decoding schemes in Appendix
4 EXPERIMENTS
In this section, evaluate the proposed MV-TER model two representative downstream tasks: 3D object classification and retrieval.
4.1 DATASET
2015). This dataset contains 12,311 CAD models from 40 categories. We follow the standard training and testing split settings, i.e., 843 models are used for training and 2, 468 models are for testing. To acquire projected 2D views, we follow the experimental settings of MVCNN (Su et al. 2015) to render multiple views of each 3D object. Here, 12 virtual cameras are employed to capture views with an interval angle of 30 degree. Next, we employ rotation as our 3D transformations on objects and perform a random rotation with three parameters all in range [-180", 180%] on the entire 3D object. We also render the views of the transformed 3D object using the same settings as the original 3D object. The rendered multiple views before and after 3D transformations are taken as the input to our method.
We conduct experiments the ModelNet40 dataset (Wu al.
4.2 3D OBJECT CLASSIFICATION
In this task, employ the MV-TER self-supervisory regularizer to two competitive multi-view based 3D object classification methods: MVCNN (Su et al. 2015) and GVCNN (Feng et al. 2018), which are referred to MV-TER (MVCNN) and MV-TER (GVCNN). Further, implement with two transformation decoding schemes as discussed in Section 3.3 including the fusion scheme and average scheme. Then model has four variants presented in Table [
Implementation Details: We deploy GoogLeNet (Szegedy et al. 2015) as our backbone CNN as in GVCNN (Feng et al. 2018). The backbone GoogLeNNe pre-trained ImageNet1K dataset.

---

Under review as a conference paper at ICLR 2021
We remove the last linear layer as the Siamese representation learning module to extract features for each view. Subsequent to the representation learning module, we employ one linear layer as the transformation decoding module. The output feature representations of the Siamese network first go through a channel-wise concatenation, which are then fed into the transformation decoding module to estimate the transformation parameters. The entire network is trained via the SGD optimizer with a batch size of 24. The momentum and weight decay rate are set to 0.9 and 10-4, respectively. The initial learning rate is 0.001, which then decays by factor of 0.5 for every 10 epochs. The weighting parameter À in (5) is set to 1.0. Also note that, MVCNN in Table [ has two variants with different backbones: MVCNN (Su et al. 2015) and MVCNN (GoogLeNet) (Feng et al. 2018).
MVCNN (Su et al.) 2015) uses the VGG-M (Chatfield et al. 2014) as the backbone, while MVCNN
(GoogL implemented (Feng et al. 2018) employs the GoogLeNet (Szegedy et al. 2015). In addition, RotationNet (Kanezaki et al. 2018) and View-GCN (Wei et al. 2020) are set up with 12
views taken by the default camera system for fair comparison.
Experimental Results: As listed in Table D the MV-TER (MVCNN), average and MV-TER (GVCNN), average achieve classification accuracy of 95.5% and 97.0% respectively, which out- perform the state-of-the-an art View-GCN (Wei et al., 2020). Also, the MV-TER (MVCNN), average outperforms its baseline MVCNN (GoogLeNet) by 3.3%, while MV-TER (GVCNN), average out- performs the baseline GVCNN (GoogLeNet) by 4.4%, which demonstrates the effectiveness of our proposed MV-TER as a self-supervisory regularizer.
4.3 3D OBJECT RETRIEVAL
In this task, we directly employ the fused feature representations of MV-TER (MVCNN) and MV. TER (GVCNN) as the 3D object descriptor for retrieval. We denote Fx and Fy as the 3D object descriptor of two 3D objects X and Y respectively, and use the Euclidean distance between them for retrieval. The distance metric is defined as
dist(X, Y) = Fx Fyl12.
We take the mean average precision (mAP) on retrieval as the evaluation metric, and present the com- parison results in the last column of Table I For MVCNN and GVCNN, low-rank Mahalanobis metric learning (Su et al. 2015) is applied to boost the retrieval performance. In comparison, we train our MV-TER model without the low-rank Mahalanobis metric learning, but still achieve better retrieval performance, which validates the superiority of our feature representation learning for 3D objects. Further, apply Triplet Center Loss (He et al. (0188) to MV-TER. With Center Loss, our model further achieves an average gain of 3.3% in mAP. As presented in the last column of Table 1 the MV-TER (GVCNN), average and MV-TER (GVCNN), fusion achieve mAP of 91.5% and 91.1% respectively, which is comparable to MLVCNN with Center Loss (Tiang et al. (019)) while we only take 12 views as input instead of 36 views. We will demonstrate some visual results of 3D object retrieval in Appendix B
4.4 ABLATION STUDIES
4.4.1 ON THE NUMBER OF VIEWS
We quantitatively evaluate the influence of number of views the classification task. Specifically, we randomly choose {8. 12} views from all the views as the input to train MV-TER (GVCNN), average respectively. leading to two learned networks. Then, we randomly select {2, 4, 12) views from all the testing views to evaluate the classification accuracy of the two networks respectively, as reported in Table We see that we constantly outperform GVCNN with different number of training views and testing views. In particular, when the number of testing views reaches the extreme of two views for multi- learning, our MV-TER model is still able achieve the classification accuracy of 99.9% and 91.2%, which outperforms GVCNN by a large margin.
4.4.2 ON DIFFERENT LABELING RATES
We adopt six different label rates in the set {0.01, 0.02, 0.03, 0.04, 0.05, 0.10) to train four models for comparison: MVCNN (AlexNet), GVCNN (AlexNet), MV-TER (MVCNN), average and MV- TER (GVCNN), average. When training MVCNN (AlexNet) and GVCNN (AlexNet), we only use small amount of labeled data minimize the cross entropy loss for training, and then employ all the test data for evaluation. When training MV-TER (MVCNN) average and MV-TER (GVCNN), average, we adopt all the data (labeled and unlabeled) to predict the 3D transformations without the
7

---

Under review as a conference paper at ICLR 2021
Table Comparison between GVCNN and our MV-TER (GVCNN), average with different num-
ber of input views for 3D object classification on ModelNet40 dataset. MV-TER is used for brevity.
Training
Testing Views
Accuracy (%)
Training
Testing Views
Accurncy (%)
Views
GVCNN
MV-TER
Views
GVCNN
MVTTE
91.9
2
76.8
91.2
4
91.1
94.6
4
90.3
94.3
95.4
96.1
12
91.5
96.0
12
92.6
97.0
use of labels, and then adopt only labeled data acquire classification scores. That is, we minimize (5) with a small amount of labels taken for the classification loss l In all the four models, pre-trained AlexNet on ImageNet1K dataset is employed as the backbone CNN.
preseetss the classification accuracy un- der the six label rates on ModelNet40 dataset. When the label rate is 0.10, we see that the four models achieve comparable results, which benefits from the pre- pre-training of the backbone
MVCNN #V--TER (MVUNN) @VENNN #MV-TER (GVCNN)
90
ACCURACY
60
AlexNet. When the label rate keeps decreasing, the performance of both MVCNN and GVCNN drop quickly, while the MV-TER models are much more robust. Even the extremely low label rate 0.01, MV-TER (MVCNN), average and MV-TER (GVCNN), average achieve the
30
0.10
0.05
0.04
0.03
0.02
0.01
classification accuracy of 58.6% and 55.2% re- spectively, thus demonstrating the robustness of the proposed MV-TER model.
LABEL RATE
Figure 2: Classification accuracy with different la- bel rates.
4.4.3 TRANSFER LEARNING
We further show the generalization perfor-
mance of the proposed MV-TER under average and fusion schemes. We take the same network architecture and parameter settings as in Sec. [.2] except that we set À = 0.5 in (5). In particular, we train the feature extractor the ShapeNeiCore55 dataset, and test on the ModelNet40 dataset by linear SVM classifier using the feature representations of dimension 2048 obtained from the second last fully- fullecconnected layer of MV-TER. Table the classification comparison of MV-TER and two baseline methods on ModelNet40 dataset under the ShapeNetCore55 pre-training strategies. As can see, the proposed MV-TER with decoding schemes improves the average classification accuracy by 2.85% and 2.95% respectively under the ShapeNetCore55 pre-training strategy com- pared with the two baseline methods MVCNN and GVCNN, thus validating the generalizability.
Table 3: Classification comparison of MV-TER and two baseline methods ModelNet40 dataset under the ShapeNetCore55 pre-training strategies.
Method
Accuracy (%)
Method
Accuracy (%)
MVCNN
85.9
GVCNN
87.9
MV-TER, average
88.7
MV-TER, average
MV:TER, fusion
89.0
MV. fusion
90.3
CONCLUSION
In this paper, we propose novel self-supervised learning paradigm of Multi- Transforma- tion Equivariant Representations (MV-TER) via auto-encoding 3D transformations, exploiting the cquivariant transformations of 3D object and its projected multiple views. We perform a 3D trans- formation on a 3D object, which leads to equivariant transformations in projected multiple views. By decoding the 3D transformation from the fused feature representations of multiple views be- fore and after transformation, the MV-TER enforces the representation learning module to learn intrinsic 3D object representations. Experimental results demonstrate that the proposed MV-TER significantly outperforms the state-of-the-art view-based approaches in 3D object classification and retrieval tasks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv: 1608. 04236, 2016.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Return of the devil in the details: Delving deep into convolutional nets, In British Machine Vision Conference (BMVC), 2014.
Taco Cohen and Max Welling, Group equivariant convolutional networks. International Confer- ence on Machine Learning (ICML), pp. 2990-2999, 2016.
Sander Dieleman, Kyle W Willett, and Joni Dambre. Rotation-invarian convolutional neural net- works for galaxy morphology prediction. Monthly Notices of the Royal Astronomical Society, 450 (2):1441-1459, 2015.
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convo- lutional neural networks. In International Conference on Machine Learning (ICML), pp. 1889- 1898, 2016.
Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In British Ma- chine Vision Conference (BMVC), volume 2. pp. 8, 2014.
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. GVCNN: Group-view convo- lutional neural networks for 3D shape recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 264-272, 2018.
Xiang Gao, Wei Hu, and Guo-Jun Qi. GraphTER: Unsupervised learning of graph transforma- tion equivariant representations via auto-encoding node-wise transformations. In Proceedings of IEFF/CVF Conferences on Computer Vision and Puttern Recognition (CVPR), 2020.
Robert Gens and Pedro M Domingos. Deep symmetry networks. In Advances in Neural Information Processing Systems (NIPS), pp. 2537-2545, 2014.
Zhizhong Han, Mingyang Shang, Zhenbao Liu, Chi-Man Vong. Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen. Seqviews2seqlabels: Learning 3d global features via aggregat- ing sequential views by ınn with attention. IEEE Transactions on Image Processing (TIP), 28(2): 658-672, 2018
Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen. 3D2Seq Views: Aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation. IEEE Transactions on Image Processing (TIP), 28()):3986-3999, 2019.
Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi- view 3d object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2018
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto auto-encoders. In Interna- tional Conference on Artificial Neural Networks (ICANN), pp. 44 51. Springer, 2011.
Jianwen Jiang. Di Bao, Ziqiang Chen, Xibin Zhao, and Yue Gao. Mlvenn: Multi-loop-view convolu- tional neural network for 3d shape retrieval. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 33, pp. 8513-8520, 2019.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. arXiv preprint 1xX1v:1902.06162, 2019.
Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object catego- rization and pose estimation using multiviews from unsupervised viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5010-5019, 2018.

---

Under review as a conference paper at ICLR 2021
Jyri J Kivinen and Christopher KI Williams. Transformation equivariant boltzmann machines. In International Conference on Artificial Neural Networks (ICANN), pp. 1-9. Springer, 2011.
Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of
3D point cloud models. In Proceedings of the IEEE International Conference on Computer Vision (CVPR), pp. 863-872, 2017.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting supervidd visual repre- sentation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1929, 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Cooperative learning of audio and video models from self-supervised synchronization. In Advances in Neural Information Processing Systems (NIPS), pp. 7763-7774, 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pp- 1097-1105, 2012.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivari- ance and equivalence. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 991-999, 2015.
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In Advances in Neural Information Processing Systems (NIPS), pp. 8844-8853, 2018.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3D convolutional neural network for real-time object recognition. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 922-922-22222 928. IEEE, 2015.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2536-2544 2016.
Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J Guibas. Volumetric and multi-view CNNs for object classification on 3D data. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 5648-5656, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning point sets for 3D classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. PointNet++: Deep hicrarchical feature learning point sets in a metric space. In Advances in Neural Information Processing Systems (NIPS), pp. 5099-5108, 2017b.
Guo-Jun Qi. Learning generalized transformation equivariant representations via autoencoding transformations. arXiv preprint arXiv: 09622990000888....88 2019.
Guo-Jun Qi, Liheng Zhang, Chang Wen Chen, and Qi Tian. AVT: Unsupervised learning of transfor- mation equivariant representations by autoencoding variational transformations. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning us- ing synthetic imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 762-771, 2018
Nawid Sayed, Biagio Brattoli, and Bjöm Ommer. Cross and learn: In German Conference on Pattern Recognition, pp. 228-243. Springer, 2018.
Cross-modal self-supervision.
Uwe Schmidt and Stefan Roth. Learning rotation-aware features: From invariant priors to equivari- ant descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 2050-2057. IEEF, 2012.
10

---

Under review as a conference paper at ICLR 2021
Henrik Skibbe. Spherical Tensor Algebra for Biomedical Image Analysis. PhD thesis, Verlag nicht ermittelbar, 2013.
Kihyuk Sohn and Honglak Lee. Learning invariant representations with local transformations. In International Conference Machine Learning (ICML), pp. 1339-1346, 2012.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International Conference on Machine Learning (ICML), pp. 843- 852, 2015.
Otilia Streteu and Marius Leordeanu. Multiple frames matching for object discovery in video. In British Machine Vision Conference (BMVC), volume 1, pp. 2015.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convo- lutional neural networks for 3D shape recognition. In Proceedings of the IEEE international conference on computer vision (CVPR), pp. 945-953, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 1-9,20 2015.
Jiayu Wang, Wengang Zhou, Guo-Jun Qi, Zhongqian Fu, Qi Tian, and Houqiang Li. Transformation GAN for unsupervised image synthesis and representation learning. In Proceedings of TEFF/CVF Conferences on Computer Vision and Pattern Recognition (CVPR), 2020.
Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), June 2020.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2015.
Ze Yang and Liwei Wang. Learning relationships for multi-view 3D object recognition. In Proceed- ings of the IEEE International Conference Computer Vision (ICCV), pp. 7505-7514, 2019.
Tan Yu, Jingjing Meng, and Junsong Yuan. Multi-view harmonized bilinear network for 3D object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 186-194, 2018
Liheng Zhang, Guo-Jun Qi, Liqiang Wang, and Jicbo Luo. AET vs. AED: Unsupervised represen- tation learning by auto-encoding transformations rather than data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2547-2555 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Con- ference on Computer Vision (ECCV), pp. 649-666. Springer, 2016.

---

Under review as a conference paper at ICLR 2021
A
ARCHITECTURE OF AVERAGE DECODING SCHEME
Vi
E(V.
V2
(
V
E
F(V.)
Concatenation
D
EV...
Pm
transformation
shared weights
((V:)
R(V,)
As!
V/3
B(V))
Fusion
7
Score
E(V..)
Figure 3: The architecture of the proposed MV-TER in the average decoding scheme. and D represent the feature representation module and transformation decoding module respectively, and T is a specific task, e.g. 3D object classification and retrieval.
We demonstrate the architecture of the proposed MV-TER with the average decoding scheme in Figure
B VISUALIZATION OF 3D OBJECT RETRIEVAL Query
Top 10 retrieved 3D objects
Figure 4: 3D object retrieval examples on ModelNet40 dataset. Top 10 matches are shown for each query, with mistakes highlighted in red.
We demonstrate visual results of 3D object retrieval in Figure 4
12

---

Under review as a conference paper at ICLR 2021
C EVALUATION OF THE 3D TRANSFORMATION ESTIMATION
Further, to intuitively interpret the estimated 3D transformations from the proposed fusson and av- erage decoding schemes, we visualize the multiple views projected from 3D objects Car and Bowl with the estimated 3D transformations applied. In Figure 5 (aa and Figure the first, second and fourth rows demonstrate the projected views from the 3D object with the same 3D transformation: the ground truth, the estimation from the fusion scheme and the estimation from the average scheme. In the third row, each view is the result of each individually estimated 3D transformation t, in (7), ie., view-wise transformations. Note that each column is rendered under the same viewpoint. We that MV-TER model estimates more accurate 3D transformations via the average scheme, which is consistent with the objective results.
Trans formed
Views
Estimated
(Fusion)
(a)
Estimated (View-wise)
Estimated (Average)
Trunsforned Views
Estimated
(Fusion)
(b)
Estimated
(View-wise)
Estimated
(Average)
Figure 5: Illustration of multiple views projected from 3D objects in the same posture: (a) Car and (b) Bowl. The four rows of (a) and (b) demonstrate multiple views projected from the 3D object with the following 3D transformations applied respectively: 1) the ground-truth 3D transformation: 2) the estimated 3D transformation of the fusion decoding scheme; 3) the individually estimated 3D transformations i, 's from each view during the average decoding scheme (with i, applied to the ith view); and 4) the finally averaged 3ID transformation of the average decoding scheme.
Moreover, Figure shows the transformation estimation error the ModelNet40 dataset under the average scheme. The horizontal axis the index of the training epoch, while the vertical axis refers to the mean squared error. We observe that the MV-TER loss decreases rapidly in the first 40
13

---

Under review as a conference paper at ICLR 2021
3.5
3-
2,5
MSE
2-
1,5
0.5
0
20
40
60
80
100
120
Epoch
Figure 6: Transformation estimation error from the average scheme.
epochs. Until the 60th epoch, the mean squared error basically converges to very small number, thus validating the effectiveness of our model in the transformation estimation.
D VISUALIZATION OF FEATURE MAPS
We visualize the feature maps of multiple views projected from 3D objects before and after transfor- mation Figure for the same category and for different categories. We see that the feature maps of projected multiple views transform equivariantly with the input views. In Figure 7 the feature maps from the same category are similar. contrast, Figure 3 although the 3D objects from two different categories similar, their feature maps are discriminative. This shows the robustness and effectiveness of the learned descriptor.
(8)
(b)
(c)
(d)
Figure 7: Illustration of feature maps of multiple views projected from 3D objects before and after transformation in the same category Airplane. (a) and (b) demonstrate multiple views projected from the 3D object before and after transformations, respectively; (c) and (d) show the feature maps of the corresponding views above.
14

---

Under review as a conference paper at ICLR 2021
(b)
(d)
Figure 8: Illustration of feature maps of multiple views projected from 3D objects before and after transformation in different categories Sofa (left) and TV Stand (right). (a) and (b) demon- strate multiple views projected from the 3D object before and after transformations, respectively; (c) and (d) show the feature maps of the corresponding views above.
15


Reviews:
[{"review_id": "yT-PdqBuCIA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes a self-supervised learning framework for 3D object classification and retrieval based on multi-view representation, where a sub-task of transformation estimation is adopted as a regularizer. By adding the proposed MV-TER loss, the STOA approaches can gain notable improvement in performance.\n\nStrengths:\n-The paper is well written, and the idea of transformation equivariant representation of 3D objects is easy to understand. \n-The proposed method further improved the stoa methods on 3D object classification, which is already quite high.\n\nAreas for improvement:\n-Firstly, the representation of the transformation that the author used in the paper is not well specified.  I assumed that the Euler angle representation is used in this paper, but I think it is not proper to use the MSE loss in (5) or take the average in (8) for Euler angles. \n-Secondly, since it has been suggested by many previous work that the joint-training of multi-task is helpful for the network, it will be appreciated that the authors provide more analysis and discussion on how the MV-TER loss helps the network learn transformation equivariant representation than simply using rotation as data-augmentation or using pose-estimation as sub-task.\n-Thirdly, any quantitative results of the transformation estimation? From the images in Appendix B, the results of (average) are incredibly accurate.", "extra_scores": {}, "extra_text": {}}, {"review_id": "i_zEsxMKR8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a self-supervised learning technique for multi-view learning based on a simple intuition that the transforms of the 2D views of a 3D object will be in an equivariant manner as the 3D object transforms. (Section 3.1). They show its effectiveness by clearly improvements under two frameworks MVCNN and GVCNN.\n\n+ clear presentation\n+ simple idea\n+ convincing experiments\n+ detailed ablation study\n\nOverall, most parts of this paper are satisfactory. They have successfully backed up their claims by applying the idea in two different frameworks and show clear improvements. While my main complaint is that the idea of this work is very simple and even can be called as common sense, and I have encountered this idea in many other papers, though in different fields, such as human pose estimation and 6d object pose estimation. I recommend the authors adding a full discussion of this simple idea based on transformation invariance.\n\nThe related works:\n3D Human Pose Machines with Self-Supervised Learning, PAMI 2019\nGeometry-Driven Self-Supervised Method for 3D Human Pose Estimation, AAAI 2020\nSelf-Supervised Learning of 3D Human Pose using Multi-view Geometry, CVPR 2019\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WDneMKc6ViR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a self-supervised learning method of 3D shape descriptors for 3D recognition through multi-view 2D image representation learning. To represent the 3D shape, the authors first project the object to a group of 2D project images, which helps apply deep learning due to the image's matrix data format.  The Unsupervised Learning of Transformation Equivariant 2D Representations by Autoencoding Variational Transformations is used for 3D shape descriptor learning, which the authors claimed as \"self-supervised\" learning. The key idea of transformation equivariant representations is directly borrowed from existing works [1][2]. The method designed is almost the same as [1] except for the encoding network.\n\n[1] Zhang et al., AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data, in CVPR 2019. (AET)\n[2] Qi et al., AVT: Unsupervised Learning of Transformation Equivariant Representations by Autoencoding Variational Transformations, in ICCV 2019. (AVT)\n\nBesides, there's no proof to verify transformation equivariant representations learning. The authors need to prove the properties for Transformation Equivariant directly for 3D objects. The current presentation of the paper is based on the 2D project image representation for 3D objects. If the authors wish to use Transformation Equivariant  in [1][2] above, the authors might want to consider adding additional 2D to the 3D reconstruction process, and  then consider the order of doing transformation and doing reconstruction etc.)\n\nMay some visualization results can better convince the audience. For example, the authors would like to add some visualization results for a 3D shape descriptor for objects from the same category and different categories and show how robust the proposed shape descriptor is.\n\nThe proposed approach had experimentally verified its effectiveness in 3D recognition.  However, for a paper being accepted to ICLR, I would like to see more technical novelties/merits beyond directly extending the existing image representation learning approach to 2D projection images of the 3D image. For instance, the authors could propose a method that can be developed based on the \"3D Transformation Equivariant\" to 3D objects directly instead of its 2D projections.\n\n---Additional comments after rebuttal--\nI have carefully reviewed the authors' feedback regarding their comments on how their proposed method differentiates the existing (AVT and AET). Unfortunately, it did not address my concerns about the novel technical contributions in the proposed paper. Obviously, the authors applied the \"Transformation Equivariant Representations by Autoencoding Variational Transformations\" directly to 2D projections of a 3D object and then fused the deep representation by a shared weight NN (shown in Figure 1). I am not sure why in the rebuttal, the authors claimed, \"Their proposed method distinguishes from AET significantly in two aspects\". I would urge the authors to check both papers below, and it clearly defines the Transformation Equivariant Representations learning by Autoencoding Variational Transformations, which could be applied for various types of data. [1] Zhang et al., AET vs. AED: Unsupervised Representation Learning by Auto-Encoding Transformations rather than Data, in CVPR 2019. (AET) [2] Qi et al., AVT: Unsupervised Learning of Transformation Equivariant Representations by Autoencoding Variational Transformations, in ICCV 2019. (AVT)\n\nAnother concern was critical but not yet addressed neither: The authors could propose a method that can be developed based on the \"3D Transformation Equivariant\" to 3D objects directly instead of its 2D projections. For this question, I think the authors should prove the Transformation Equivariant Representations directly on a 3D object (point cloud, voxel, 3D mesh) instead of multi-view 2D images. I am not sure why the authors answered that \"3D objects are unavailable at the testing stage.\" The proof of Autoencoding Variational Transformations for 3D data directly should not depend on the availability of 3D data.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mv9splaJEXw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the Submission:\n\nThis submission proposes a self-supervised learning scheme for 3D object recognition. The basic idea is to predict a 3D transformation from 2D views. The features that facilitate 3D transformation prediction then generalize to other 3D object recognition tasks such as object classification and retrieval. The evaluations on modelnet40 and shapenetcore55 show significant improvement when adding the self-supervised training.\n\nStrengths: Self-supervised formulations are highly beneficial for tasks where no labels are available or where it is expensive to collect labels. The formulation is intuitive, the manuscript is well written and easy to follow.\n\nUpdate after Rebuttal:\n\nAn additional experiment on real data was added which I find a valuable addition to the submission. AR2 points out a similarity with AET which I did not notice in my initial review. I feel this does limit the contribution somewhat. As the rebuttal points out there are differences between the proposed method and AET but the core idea is very similar. In my opinion there is enough difference to still recommend acceptance.\n\nWeakness:\n\nThe evaluations are all on synthetic data. It would be great to see how the learned features generalize to real data such as e.g. pascal.\n\nThe paper title and the writing is in general talking about 3D transformations but it seems experiments have only been conducted with 3D rotations. Maybe it would be more adequate to change it to 3D rotations everywhere or demonstrate at least one additional type of transformation.\n\nThe means squared error as a loss on the 3D rotation is not well motivated. How is the rotation parametrized? Why not using a loss function that penalizes the amount of rotation?\n\nThe views are sampled in a regular spacing and not randomly. There is a worry that this induces a bias and recognition with multiple nearby views at test time might be affected as it will not have been present in the training data. But this mode of operation might be a common one in practice if e.g. a video of an object constitutes the multiple views.\n\nThis paper seems also related and might be an addition to the related work: Rhodin et al. Unsupervised geometry-aware representation for 3d human pose estimation, ECCV 2018\n\nExplanation of Rating:\n\nThis paper shows a self-supervised way to learn features which lead to improved results in classification and recognition on the modelnet40 dataset. The main weakness of the submission is that it was only evaluated on synthetic data. It would be interesting to see if the self-supervised learning helps with generalization to real images.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a method for self-supervised learning of representations of 2D projections of 3D objects. By performing known 3D transformations of an object of interest, a encoder/decoder network is trained to estimate the applied transformation from a series of 2D projections. The proposed method is used as a regularizer and experiments are performed on supervised 3D object classification and retrieval.   

After seeing each others’ reviews, one of the main concerns from the reviewers was the relationship between the proposed method and Zhang et al., CVPR 2019 (i.e. AET). The two methods are conceptually very similar, and the consensus from the reviewers is that the authors did not acknowledge the overlap sufficiently and also did not provide a convincing argument as to why they think the approaches are different. 

In their rebuttal the authors provided some additional results on real data which is a valuable and welcome addition. However, there were still other concerns that the reviewers had e.g. R2 wanted to know why the model could not be applied directly to 3D shapes instead of 2D projections. 

Given the above concerns (specifically the relationship to AET), there is currently not enough support for accepting the paper in its current form. The authors have received detailed feedback and are encouraged to take it onboard when revising the paper in future.  

Decision: Reject