Contrastive Learning of 3D Shape Descriptor with Dynamic Adversarial Views
Authors: Shuaihang Yuan
Yi Fang
Venue: ICLR 2022 Submitted (2022)
Keywords: Dynamic multi-views

Abstract: View-based deep learning models have shown the capability to learn 3D shape descriptors with superior performance on 3D shape recognition, classification, and retrieval. Most popular techniques often leverage the class label to train deep neural networks under supervision to learn to extract 3D deep representation by aggregating information from a static and pre-selected set of different views used for all shapes. Those approaches, however, often face challenges posed by the requirement of a large amount of annotated training data and the lack of a mechanism for the adaptive selection of shape-instance-dependent views towards the learning of more informative 3D shape representation. This paper addresses those two challenging issues by introducing the concept of adversarial views and developing a new mechanism to generate views for adversarial training of a self-supervised contrastive model for 3D shape descriptor, denoted as CoLAV. In particular, compared to the recent advances in multi-view approaches, our proposed CoLAV gains advantages by leveraging the contrastive learning techniques for self-supervised learning of 3D shape representations without the need for labeled data. In addition, CoLAV introduces a novel mechanism for the dynamic generation of shape-instance-dependent adversarial views as positive pairs to adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation. Comprehensive experimental results on 3D shape classification demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other state-of-the-art techniques, even though CoLAV is completely self-trained with unlabeled 3D datasets (e.g., ModelNet40).

Under review as a conference paper at ICLR 2022
CONTRASTIVE LEARNING OF 3D SHAPE DESCRIPTOR WITH DYNAMIC ADVERSARIAL VIEWS
Anonymous authors Paper under double-blind review
ABSTRACT
View-based deep learning models have shown the capability to learn 3D shape de- scriptors with superior performance on 3D shape recognition, classification, and retrieval. Most popular techniques often leverage class labels to train deep neural networks under supervision to learn to extract 3D deep representation by aggre- gating information from static and pre-sclected set of different views used for all shapes. Those approaches, however, often face challenges posed by the require- ment of a large amount of annotated training data and the lack of mechanism for the dynamic mining of shape-instance--epenndde of more informative 3D shape representation. This paper addresses those two challenging issues by introducing the concept of adversarial views and develop- ing a new mechanism generate 2D views for the adversarial training of self- supervised contrastive model for 3D shape descriptor, denoted as CoLAV. In par- ticular, compared to the recent advances multi-view approaches, our proposed CoLAV gains advantages by leveraging the contrastive learning techniques for self-supervised learning of 3D shape representations without the need for labeled data. In addition, CoLAV introduces novel mechanism for the dynamic gener- ation of shape- cesstance-deeendene adversarial views as positive pairs to adversar- ially train robust contrastive learning models towards the learning of more infor- mative 3D shape representation. Comprehensive experimental results on 3D shape classification demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other existing view-based de- scriptor learning techniques, even though CoLAV is completely self-trained with unlabeled 3D datasets (e.g., ModelNet40).
views towards the learning
INTRODUCTION
The impressive performance of Deep Neural Networks (DNNs) for 2D visual feature learning natu- rally motivates 3D computer vision researchers to develop deep learning models for 3D representa- tion for downstream 3D tasks (e.g.. shape classification (Qi al. 2017b), shape correspondence (Li cct al., 2019), shape retrieval (Han et al.) 2018), shape registration (Miao et al.) (016). To this end, researchers developed the first family of 3D deep learning models that operate directly on 3D raw data represented by point clouds (Qi et al. 2017b Shi et al.) 2020), triangulated meshes (Hanockal al. 2019), 3D regular voxels (Choy al.) 2019). The recent advancement of approaches in this category has significantly contributed to the shape classification, correspondence, segmenta- tion, registration, and detection. The second important family of 3D deep learning models starts with rendering multiple 2D projections of a 3D object from static and pre-selected set of different views, followed by learning the compact 3D representation via fusing the deep feature maps of these 2D views. As they can straightforwardly leverage the latest developments of convolutional neural networks, the riew-based methods have shown very attractive capability to learn 3D) shape descrip- tors with superior performance on 3D shape recognition, classification, and retrieval (Li et al., 22220) Yuan & Fang 2020 Shi et al. 2020). Particularly, recent advances in view-based approaches lead to impressive state-of-the-art performances in various 3D vision tasks such as 3D shape classifica- tion, retrieval, correspondence, and segmentation (Wei et al. 2020). The most popular techniques often leverage the ground truth labels to train deep neural networks and learn to extract deep repre- sentations by aggregating information from a static and pre-selected set different views. Those approaches, however, often face challenges posed by the requirement of large amount of annotated

---

Under review as a conference paper at ICLR 2022
training data and the lack of mechanism for the adaptive selection of shape-instance--eeendeee views towards the learning of more informative 3D shape representation.
Augmentation
Augmented, 3D instance! 1(x)
Forward Path
(Descriptor learning module) Shared weights
Diffrrettable Renderer Tt
Generation & Attack
Contrastive Loss
input instance
Adversarial Views Mining
(Descriptor learning module) I Shared weights
Latent space
u.P.y
Contrastive
Learning
(back-propagation)
c,
Adversarial 3D Instance add
(Descriptor learning module)
Rotation Matrix
Figure 1: Illustration of our pipeline.
This paper addresses those two challenging issues by introducing the concept of adversarial views (Xiang et al. 2019) and developing a new mechanism to generate views for the adversarial training of self-supervised model for 3D representation learning, denoted CoLAV. Specifically, different from the recent advances in multi-view approaches (Kanezaki et al. 2018), CoLAV gains advantages by learning the 3D shape representation without the need for labeled data. In addition, in contrast using a static set of views as input during the training of a multiview- bhasee 3D shape representation learning framework, we take further step investigate the effect of using dynamically rendered views as input during the 3D shape representation learning. To this end, we introduce an adversarial view generator that enables the dynamic generation of shape-instance-deeeedee adversarial views, which can fool the deep learning model to misidentify its instance wise identity, positive pairs adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation. Our pipeline can be found in Figure Specifically, our model takes 3D shape instance x as input, augmentation operations t sampled from a pool of predefined augmentations. Then, domly initialized a rotation matrix (characterized by a, 3, and 7) to generate rotated 3D shape r For each 3D instance ær, f(x),, and 2:go render & views using the same differentiable renderer r+()). The rendered views are fed to a descriptor learning module to extract shape descriptors. After obtained the deseriptors, we use contrastive loss measure the mutual information of descrip- tors. Two key components are employed in our 3D shape unsupervised descriptors learning process. The first to a, B, and and update the rotation matrix using the updating rule which introduce in the method section. This rotation matrix updating is represented by the purple path. The second component is the contrastive learning module with phape-instance-dependdee represented the red path, which is adopted towards the learning of more informative and robust 3D shape representations. In this component, new 3D instance rıdu denoted as adversarial 3D instance, can be obtained using the updated rotation matrix. We adopt the same differentiable ren- derer r. (+) to render adversarial views for the adversarial training of the Descriptor learning module in order to produce robust and informative 3D shape descriptor.
we first generate an augmented
shape ((x)) by adopting set of
Adversarial view. We use the contrastive loss calculate the gradient with respect
adversarial view attack,
contribution follows: 1) We introduce novel concept of adversarial views of a 3D shape and develop a new mechanism to adaptively and dynamically render shape-instance dependent views towards the learning of more informative 3D shape representation via CoLAV. 2)
We summarized
2

---

Under review as a conference paper at ICLR 2022
The proposed CoLAV is capable of learning 3D shape representations from multiple 2D views with- out requiring ground truth annotations. 3) Our comprehensive experimental results on 3D shape classification and retrieval demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other techniques, even though CoLAV is com- pletely self-trained with unlabeled 3D datasets (e.g., ModelNet40).
RELATED WORKS
2.1 MULTI- VIEW LEARNING
In recent research, multi-view view-bbsed methods have become a popular topic in 3D computer vision. Basically, these methods start with rendering projections of 3D objects from different view perspec- tives, generate representation features for each projection, and merge them into a global descriptor of the 3D objects. The compact descriptor can be used to improve performance and robustness in 3D object learning tasks such as classification and retrieval. One of the earliest methods in multi-view- based learning is called multi-view convolutional neural network (MVCNN) (Su et al. 2015), which fuses feature representations extracted from multiple views into one compact descriptor by a max- pooling layer. Ruizhongtai Qi et al. (2016) advances the progress by incorporating multi-resoluuiion component into MVCNN to boost the classification accuracy. However, using a max- max--oooiigg layer to fuse representation features from multiple views can only retain the maximum value each di- mension leading to possible losses of information. The group-view view convolutional neural network (GVCNN) (Feng al. 2018) is proposed to solve this problem, which generates descriptor via grouping strategy that keeps more information from different views. There is also research attach- ing attention-based models the framework, which leverages spatial relationship among different views' feature representation (Han et al. 2018 2019). Yang & Wang (2019) also leverages the relationship between views and constructs model that connects corresponding regions on each projection and strengths the representation learned from each individual viewpoint. The aforemen- tioned view-based methods have achieved promising performance on various tasks. However, their training process requires large amount of annotated data. To alleviate the dependency of annotated data, (H et al. (2020) proposed a self-supervised method that randomly selects a view feature out of several pre-selected static views of 3D object as a prototype to learn invariant stochastic object embedding which is further used to generalize the method (Ho et al. 2020) on test data. Different from all previous view-based methods, we investigate the effect of using dynamic views. Instead of using pre-selected static views, in our work, dynamically rendered views are used to learn an infor- mative siltt--vie--baseedd 3D object representation, meanwhile, annotated class labels are given. This encourages the proposed model to better generalize on various unseen views.
2.2 ADVERSARIAL LEARNING
In the concept of adversarial learning, adversarial examples are treated as hard positive samples in the model training process improve the robustness of neural networks (Szegedy et al. [2013).
Goodfellow et al. (2014) introduced the fast gradient sign method (FGSM) which increases the robustness of the model by perturbing input samples along with gradient directions. Following this
direction, numerous methods (Kurakin et al.) 2016 Papernot et al., 2016 Shafahi et al. 2019 Xie
& Yuille 2019: Xie et al. 2020) were proposed and have obtained impressive robustness against adversarial attack in image space. Recently, Zeng et al. (2019) claimed that instead of generating adversarial samples in 2D space, finding adversarial 3D objects behind the projected 2D images is crucial to improve the robustness. In their work (Zeng et al.) (011)). FGSM is used to perturb the input sample in 3D space governed by a supervised loss, and ground truth class labels are required to generate adversarial 3D objects. Unlike Zeng et al. (2019), in this work, we introduce new approach to generate adversarial views based on 3D objects without using ground truth labels.
2.3 CONTRASTIVE LEARNING
The concept of contrastive learning has been extensively used in recent studies, especially in self- supervised (SSL) learning scenarios. In SSL 3D object learning, contrastive learning can help trieve an invariant descriptor for each 3D object by applying random augmentations to each object and minimizing the contrastive loss by measuring the similarity between the feature vectors repre-
3

---

Under review as a conference paper at ICLR 2022
senting original and augmented shapes. There have been certain amount of augmentation strategies that came up by previous studies in the 2D computer vision field (Wu et al. 2018 Ye et al., 2019 Tian et al. 2019), implementing data augmentation by rotation, cropping, and other transformation methods. Such augmentations can force the network to learn sophisticated descriptors to represent input instances while training, thus boosting robustness. By comparing different methods of aug- Chhin
mentation, Chen et al. (2020) indicates that the effectiveness of contrastive learning can be enhanced by employing a stronger data augmentation. On the other hand, although have various augmen- tation methods to boost performance, it is arduous for contrastive learning models to overcome the hard negative mining problem mentioned in [Sohn] (2016); Wuu et al. (2017): [Schroff al. (2015). One of the possible solutions to that problem is to select negative samples from a distribution of noise that treats all negative instances equally, which is proposed by the contrastive noise estimation (NCE) (utmmann & Hyvärinen 2010), by which contrastive learning is directly inspired Bose et al.
(2018: :
Bose et al.) Ho & Vasconcelos (2020).
METHODS
The pipeline of our proposed module is illustrated in Fig. The CoLAV is a generic module that generates instance-wise adversarial views to adversarially train contrastive learning model towards the goal of learning a more informative and robust shape descriptor for a given task.
3.1 ADVERSARIAL VIEWS
We introduce our novel approach obtain adversarial views while alleviating the dependency of annotated data. As we illustrated in Section 2 although many multi-view-ber viwwd methods have been proposed, their performances rely on contents of input views which are controlled camera posi- tions and annotated data. As we observed from existing works (Su et al.] 2015 Feng et al.) 2018)), camera positions are static across the entire 3D shape dataset, which leads the trained model vulner- able to unseen views. To address this issue, we introduce the concept of adversarial views as hard positive training samples during the model training. which aims to fool the predictor to make wrong predictions by maximizing the objective function.
Adversarial Views Generation. Concretely, the generation dynamic adversarial views depends on the generation of an adversarial 3D sample zade, To generate adversarial 3D samples, existing researches focus on perturbing 3D point positions within a certain range in 3D space. This pertur- bation is almost indistinguishable by human eyes, i.e., a perturbed 3D sample is visually identical to the original 3D object. Different from the existing adversarial approaches, we propose a method to generate visually different adversarial views of each 3D instance with the aim fake the model to treat the generated views as new instances. Given a 3D shape from the dataset D, diffcrentiable renderer r, (-) x > rotation matrix M controlled by three randomly initialized parameters a, 3, descriptor learning model fo(-) and a task-specific predictor g (-), the generation of add is defined as:
zadv -{
Mx. Maddy.
i= 0
(1)
i>0 0,
where i denotes the generation iterations and ad denotes the updated rotation matrix controlled by à, B, y which are formulated as follow:
à 3 p(esign( Ca.,,, [a, 7)),
(2)
where C is the loss for training the neural network fa(.) and ga (-) with parameter & and 0, the sign(-) gives the sign of gradients, and e represents the step size of one gradient attack. We em-
ploy a function p(-) as projection function to clip three angles a, B, R with in 11180, The adversarial views aade is rendered by:
adv r(zado)
(3)
To train neural network using adversarial samples generated from Equation 12 we can directly minimize the L over Jndo This adversarial training process can be formulated as follow:
arg min ((r)
max Lo
(4)
8,0
padu
4

---

Under review as a conference paper at ICLR 2022
The computation of loss term L..e, such cross-entropy loss requires the ground truth labels, which prevents us from applying this approach to unannotated datasets. To address this, we alleviate the dependency of ground truth annotations by adopting self-supervised contrastive loss.
Self-Supervised Adversarial Views Generation. Specifically, given a pool of augmentation op- erations T, such as random rotation, random jittering, random scaling, random noise, and random dropout, we randomly select a set of augmentation operations ~ T. The specific data augmenta- tion operation that we adopt our method can be found in Section 4.2 Our goal is to maximize the similarity between fr(r(z)) and fa(((r(x))) since r(æ) and ((r(z)) have the same instance-level proprieties, and minimize the similarity between ((((r)))) and (t((((2z))) where x' a dif- ferent 3D shape instance. For a clear illustration, we denote r(x) as I, [.((r))) as Ipos, and t(r(z') as Ieeg- Then, the constrastive learning process can be formulated by minimizing the following loss function:
L (I, Iposs Ineg)
log
(5)
(p) D)TT + (
where we follow the SimCLR (Chen et al., 2020) to adopt I' as temperature parameter and the sim(:,) function, which is defined as follow:
sim(a, = a" b/lla//laallbb
By adopting this formulation, we can reformulate Equation 2 and make it self-supervised. which relaxes the constrain of requiring annotated datasets during adversarial view generation. The self- supervised version of adversarial view generation can be formulated as follow:
à; = Pelesign(Vo.,,,fcaa (r(x)) (rttttt))))))rr(((((())))))) (/tes
+ [a, B, 7)),
(7)
3.2 CONTRASTIVE LEARNING WITH SHAPE-INSTANCE DAABB---NSTANEEEEEEEEEENT DEPENDENT ADVERSARIAL VIEW ATTACK
Once have the adversarial views our positive samples, generated from Equation 12 we can learn robust and informative 3D shape descriptor through contrastive learning. The learning pro- cess can now be formulated as follow:
arg min Ba-d(max L (x, (t(x), (t(x),t(xadd)) d(a {t(x'))) gade
(8)
0
The final loss function L3Deo can be described as:
LaDeol (xpos (Inegt)
Ej ) ( E(ppen ((iimm//
log
(Fnco)
where {Ipos }o is the positive 3D sample set which is composed by {t(x), t(zadv)) and {zneg} denotes the set of negative 3D samples (1xx'). The training of CoLAV can be found in Algorithm 1.
4 EXPERIMENTS
In this section, we conduct experiments to evaluate the performance of our proposed method on two benchmark datasets. More specifically, we evaluate our module when adopting various multi- view-based backbones, including GVCNN (Feng et al. (018) and MVCNN (Su et al.) (0015). as our descriptor learning module. Section 4.1 provides implementation details of our model and a description of datasets used in experiments. In the following section, carry out experiments to classify and retrieve 3D objects. In Section 4.3 we demonstrate that the descriptor extracted by model is robust to rotation. In Section 4.4 we further investigate model's 's effectiveness by using different settings, such as different representations of 3D inputs, different numbers of views, etc.
5

---

Under review as a conference paper at ICLR 2022
Algorithm The training process of our proposed method
Input: Original 3D shapes: x, augmented 3D shapes: f(x), augmented negative 3D shapes: t(z",, descriptor learning model: f. parameter of model: 0, adversarial view generation iterations: for all iteration E training stage do for all minibatch do for attack iteration E [0, 2) do
update rotation matrix M.
end for generate ad adversarial views. end for calculate objective function L3Deo using Equation update model parameter end for
4.1 DATASET, IMPLEMENTATION AND TRAINING DETAILS
Dataset: We conduct experiments on the ModelNet40 (Wu et al. 2015 dataset. ModelNet40 is one
of the first benchmark datasets that were acquired from CAD objects. This dataset contains 12,311 high-resolution, watertight 3D shapes in the mesh format, and all 3D meshes are labeled into 40 categories. In our experiment, we apply the standard splitting of the dataset. We use 9,843 shapes for training and 2,468 shapes for testing. In addition to ModelNet40 benchmark dataset, we also evaluate our method on ScanObjectNN (Uy et al. 2019) dataset. Due to the page limit, we include this part in our supplementary materials.
Implementation and Training Details:
The entire model is implemented in Pytorch. An Adam optimizer with batch size of 16 is used to train the entire network. The momentum was 0.9, and the weight decay rate is set 10-5, The learning rate starts at 0.0005 and then decreases by factor of 0.5 every 20 epochs.
Method
Modality
Supervision
Classification
Retrieval
PointNet Qi et al. (017a)
point cloud
89.2
70.5
PointNet++ Q1 cl al. (2017b)
point cloud
91.9
85.3
PointCNN Li al. 2018)
point cloud
91.8
83.8
Shi. et. al. Shi et al. (2020)
point cloud
84.7
71.8
VoxNet Maturana Scherer
2015)
voxel grid
83.0
SubvolumeSup Q1 et al. Voxception Brock al.
voxel grid
89.2
2016)
voxel grid
91.3
3D-DescripNNt Xie et al.
(20T8)
voxel grid
X
83.8
3D-GAN Wu et al. (2016)
voxel grid
83.3
T-L Network Shi
(2020)
image voxel
74.4
MVCNN Su et al. (2015
12 views
89.9
70.1
MVCNN, metrie Su al.
(2015)
12 views
89.5
80.2
RotationNet Kanezaki et al. (0118
12 views
90.7
MHBN Yu al. (2018)
12 views
93.4
3D2Seq Views Han et al.
2019
12 views
93.4
90.8
MVCNN Su al. (2013
12 views
92.2
74.1
MVCNN (G.) (G.)Sn Su al. (2015)
12 views
92.2
83.0
GVCNN (G.) Feng et al. 2018
12 views dynamic views
92.6
81.3
Ours (MVCNN)
X
93.6
85.3
Ours (GVCNN)
dynamic views
X
94.3
85.9
Table I: 3D Shape classification and shape retrieval results on ModelNet40 dataset. We com- pare the classification and retrieval performance of our model with existing methods on the Model- Net40 dataset. Our model is trained without any supervision of ground truth but still outperforms most of the supervised methods on classification accuracy and retrieval mAP. "(Go.)" represents the methods use GoogLeNet backbone network.

---

Under review as a conference paper at ICLR 2022
4.2 RESULTS ON MODELNET40
3D Object Classification:
To the best of our knowledge, our work is the first to extract 3D shape descriptors from dynamic views without using ground truth labels. To validate the generalization ability of our proposed model, we conduct experiments for the task of 3D object classification on the ModelNet40 bench- mark and compare our method against various multi-view baed 3D shape descriptor learning meth- ods. In this comparison, we compare not only with multi-view-based methods but also with other types of 3D representation, such as point clouds and voxel grids. For the 3D object classification task,
as part of the inputs, we employ random rotation, random scaling, and random jittering as t(-) to generate t(z). Specifically, for mesh object we randomly rotate the object around three axes (x, y, 2) and randomly scale the mesh object in the range of [0.75, 1.15]. After random rotation and random scaling, we apply random jittering on each vertex of the 3D mesh. For this augmentation operation, we add random displacement to each point and clip this displacement in the range of --.04...044) After random jittering, the connection of all faces remains the same as origin 3D instance, which preserves the watertight property of 3D mesh. We use the same data augmentation but apply on another 3D shape instance to produce t(x²) negative sample. In this experiment, we adopt Pytorch3D as the renderer (-) to render 2D views for each 3D instance on-the-fly. We render k- 12 images of each 3D shape instance and use 5 to generate adversarial views. For each attack iterations i, we use step size c = 0.015 to update rotation parameters a, 33, y. We use GVCNN and MVCNN two different backbone networks to generate 3D shape representation We evaluate the unsupervised learned 3D shape representation z by using a linear SVM with default parameters.
Experiment Results: Table T shows the classification accuracy of our proposed model as well as other methods on the ModelNet40 dataset. In spite of the fact that our method is completely self- supervised, our MVCNN based model achieves 93.6% accuracy on 3D object classification on the ModelNet40 dataset, which surpasses the classification accuracy of previously proposed supervised methods listed on the table. Our GVCNN based model achieves 94.3% accuracy, which further im- proves the classification accuracy and surpasses the supervised methods. Compared with MVCNN, (UVS(M)CNN) has an improvement of 1.5% on accuracy. Compared with GVCNN, OurS(GVCNN) outperforms GVCNN by 1.7% of accuracy. Another notable difference is that in previous multi- view-based methods new sets of views can be generated for each training epoch. Throughout the entire training process, different views can be dynamically rendered for the training of descriptor learning network obtain more robust and informative 3D shape descriptors compared with existing methods. To this end, an advantage of dynamically view generation raises, compared with static view-based multi-view methods.
the table, views are all generated statically before training. In model,
Furthermore, we follow the same training setups that we presented above to conduct the experiment on the task of 3D object retrieval. For each 3D object, its respective shape representation used to perform the object retrieval. We calculate the distance between every two latent representations to conduct retrieval. Specifically, given a learned 3D shape representation of a query shape x, we aim find similar shapes by comparing the distance between 2 and every other shape descriptors. To evaluate the shape retrieval performance, we follow the stander 3D shape retrieval evaluation metric (mAP).
3D Object Retrieval:
Experiment Results: As shown in the last column of Table D our MVCNN based self-supervised method obtains 85.3 mAP when performing 3D object retrieval tasks on the ModelNet40 dataset, and GVCNN based method achieves the mAP of 85.9, which exceeds the performance of most of the supervised methods proposed by previous studies. Furthermore, compared to our baseline method MVCNN and GVCNN, our methods improve retrieval mAPs by 2.3 and 4.6 respectively.
4.3 ROBUSTNESS TO ROTATION
In this section, conduct further experiments to explore the resistance to rotation. Since model dynamically generates 2D views and performs adversarial attacks to force the network to take the worst views into consideration, our model is expected to be highly robust to the rotation. This test
7

---

Under review as a conference paper at ICLR 2022
i=3
j=5
Figure 2: Adversarial Views: We observe the adversarial views generated during the adversarial attack procedure. Our model gradually rotates the 3D objects to can fool the network. These views are utilized to provide difficulty to the training process and urge network to draw more powerful feature representations.
capture less informative views that
is designed measure our performance dealing with 3D objects which are rotated into different angles.
In this experiment, test the rotation robustness of our model on the Mod- clNet40 benchmark dataset. We rotated the testing data around random axis by either 90 or 180 degrees and reported the performance.
Experiment Setup:
Experiment Results: As shown Table 2 our self-supervised network with both views and 12 views configuration achieves attractive robustness to the rotation. Dealing with 90 degrees of rotation along the gravity axis, our views model performs 90.8 accuracy of classification, which is the best among all the previous methods, including all the supervised methods listed on the table. For 180 degrees of rotation along the Y-axis, our 12 views model outperforms all the previous methods, obtaining 90.9 classification accuracy.
Method
Modality
Supervision
90"
180
PointNet QQ ct al. 2017a)
point cloud point cloud
42.5
38.6
PointNet+
al. 2011
47.9
39.7
RSCNN Hu et al.
2021
point cloud
V
90.3
90.3
MVCNN Su al. 2015
12 views 12 views views
V
84.4
84.2
ViewGCN Wei et al. 22000
V
86.0
86.0
GVCNN (Feng et
2018
V
88.9
89.7
Ours(MVCNN)
6 views
X
89.3
89.4
OurS(MVCNN)
views
90.1
89.7
Ours(GVCNN)
views
X
90.8
90.3
Ours(GVCNN)
12 views
X
90.4
90.9
Table 2: Rotation Robustness 3D Methods. We test our model's robustness against rotations around Y-axis (vertical to the ground) ModelNet40. Without using the ground truth supervision, our model with and 12 views configuration performs the highest level robustness to 90º and 180" rotation among all methods.
4.4 ABLATION STUDY
In this section, We study the effect the components of our model. First, we investigate the effect of the number of views during training and testing, then analysis of the adversarial views generated by our network is presented. Due to the page limitation, the in-depth experiment conducted to ex- plore the effect of different augmentation operations adopted during the model training process,
8

---

Under review as a conference paper at ICLR 2022
the effectiveness of our proposed unsupervised loss and the effectiveness of adversarial view included the Appendix.
Training V.
Testing V.
GVCNN
MVCNN
Ours(GVCNN)
Ours(MVCNN)
8
71.2
68.3
91.2
88.7
8
91.1
84.5
92.9
90.1
8
8
93.1
87.4
93.6
91.9
8
12
91.5
86.8
94.1
93.1
12
2
76.8
71.1
91.7
90.2
12
90.3
87.6
92.1
91.6
12
8
92.1
90.5
93.9
92.2
12
12
92.6
92.2
94.3
93.6
Table 3: Effect of Number of Views: We examine the effect of different numbers training and testing views. "Training V." indicates the number of views used during the network training, and "Testing V." indicates the number of views used during the inference period. Our network achieves an impressive accuracy even if only two testing views applied, where "Our (GVCNN)" indicates the GCVNN based network and "Our (MVCNN)" indicates the MVCNN based network.
4.4. EFFECT OF NUMBER OF VIEWS:
In this study, we investigate the impact of different numbers of views on our model performing 3D object classification. We apply {8. 12] views to train model and retrain the MVCNN and GVCNN. We adopt {2,4, 4, 8, 12) views during the testing phase. As it listed in Table our model outperforms our baseline GVCNN and MVCNN in all the combinations of training and testing views selected. Even when we employit two views for testing, which is arduous for other methods, our model still achieve a high level of accuracy. We attribute this success the adversarial attack procedure that generates challenging positive samples for the network to learn from and causes the network to draw informative descriptors from the two views selected via contrastive learning.
4.4.2 ADVERSARIAL VIEW ANALYSIS:
We analyze the effectiveness of our adversarial attack procedure by observing the adversarial views generated by our model. As shown in Figure with i defined in Equation ( represennin the num- ber of iterations of adversarial attack, we can see that our model will gradually rotate the adversarial views to challenging angles which depict less informative details of the 3D object. When i = 5, the adversarial views of the sofa are highly ambiguous that even human eyes cannot recognize the object easily. Moreover, with the help of the contrastive learning procedure in the following steps, the network expected to draw a more sophisticated and invariant descriptor for each 3D object to enhance classification accuracy on these challenging views.
CONCLUSION
In this paper, we introduce novel concept of adversarial views generated via a self-supervised contrastive learning mechanism that augments the positive training samples by performing shape- instance-dependent adversarial attacks and introduce multi-view pipeline that directly takes 3D instance input. More importantly, different from the previous multi-view-baae based methods us- ing pre-rendered static views for the training of deep neural networks, our proposed method can dynamically generate views for a given 3D instance on the fly without requiring additional net- work parameters. Extensive experiments conducted on different benchmark datasets demonstrate the effectiveness of the newly introduced adversarial views. Our self-supervised method boosts the 3D classification accuracy and 3D retrieval mAP to outperform the supervised methods previously proposed. In addition, ablation studies of model prove the effectiveness and efficiency of the proposed components. Our method highly improves robustness to rotation thanks to the adversarial views dynamically generated by our adversarial attack procedure.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Compositional hard negatives for visual semantic embeddings via an adversary.
Avishek Joey Bose, Huan Ling, and Yanshuai Cao. Adversarial contrastive estimation. arXiv preprint arXiv: arXiv:1805.00 (attv:880003444 2018.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv: 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3075-3084, 2019.
Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao. Gvenn: Group-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Zhizhong Han, Mingyang Shang, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL. Philip Chen. Seqviews2seqlahel:: Learning 3d global features via aggre- gating sequential views by rnn with attention. IEFE Transactions on Image Processing, 28(2): 658-672, 2018.
Zhizhong Han, Honglei Lu, Zhenbao Liu. Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL. Philip Chen. 3d2seqviews: Aggregating sequential views for 3d global feature learning by cnn with hierarchical attention aggregation. IEEE Transactions on Image Processing,
388-)99986-3999 2019.
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. Meshenn: a network with an edge. ACM Transactions on Graphics (TOG), 18)4111112, 2019.
Chih-Hui Ho and Nuno Vasconcelos. Contrastive learning with adversarial examples. arXiv preprint arXiv:2010.1 12050, 2020.
Chih-Hui Ho, Bo Liu, Tz-Ying Wu, and Nuno Vasconcelos. Exploit clues from views: Self- supervised and regularized learning for multiview object recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9090-9100, 2020.
Linshu Hu, Mengjiao Qin, Feng Zhang, Zhenhong Du, and Renyi Liu. Rsenn: A cnn-based method to enhance low-light remote-sensing images. Remote Sensing, 13(1):62, 2021.
Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida. Rotationnet: Joint object catego- rization and pose estimation using multiviews from unsupervised viewpoints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5010-5019, 2018
Roman Klokov and Victor Lempitsky. Escape from cells: Deep kd-networks for the recognition of 3d point cloud models. In Proceedings of the IEEE International Conference on Computer Vision, pp. 863-872, 2017.
Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al. Adversarial examples in the physical world, 2016.
10

---

Under review as a conference paper at ICLR 2022
Chun-Liang Li, Tomas Simon, Jason Saragih, Barnabás Póczos, and Yaser Sheikh. Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 11967-11976, 2019.
Xiang Li, Congcong Wen, Lingjing Wang, and Yi Fang. Topology-constrained shape correspon- dence. IEEE transactions on visualization and computer graphics, 2020.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. In Advances in neural information processing systems, pp. 820-830, 2018.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Con- ference on, pp. 922-928. IEFE, 2015.
Shun Miao, Z Jane Wang, and Rui Liao. A cnn regression approach for real-time 2d/3d registration.
IEEE transactions on medical imaging, 35(5):1352-138) 2016.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European sympo- sium on security and privacy (EuroS&P), pp. 372-387. IEEE, 2016.
Charles R Qi, Hao Su, Matthias Nießner Angela Dai Mengyuan Yan, and Leonidas Guibas. Volu- metric and multi-view cnns for object classification on 3d data supplementary material.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. Pruc. Computer Vision and Pattern Recognition (CVPR), IEEE, 1(2)4, 2017a.
Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint 17Xiv:1706.024133 2017b.
Charles Ruizhongtai Qi, Hao Su, Matthias Niessner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view cnns for object classification on 3d data. 04 2016.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering, In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823. 2015.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843, 2019.
Yi Shi, Mengehen Xu, Shuaihang Yuan, and Yi Fang. Unsupervised deep shape descriptor with point distribution learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9353-9363 2020.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 1857-1865, 2016.
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: /00600599000 2019.
Mikaela Angelina Uy Quang-Hies Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revis- iting point cloud classification: A new benchmark dataset and classification model real-world data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1588- 1597, 2019.
11

---

Under review as a conference paper at ICLR 2022
Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1850-1855 2020. 2020.
Chao-Yuan Wu, Manmatha, Alexander Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2840-2845 2017.
Jiajun Wu, Chengkai Zhang. Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning prob- abilistic latent space of object shapes via 3d generative-adversarial modeling. In Advances in neural information processing systems, pp. 82-90, 2016.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1912-1920, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Chong Xiang, Charles R Qi, and Bo Li. Generating 3d adversarial point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9136-9144, 2019.
Cihang Xie and Alan Yuille. Intriguing properties of adversarial training at scale. arXiv preprint MXXvv:996.00370 2019.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille. and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 819-828, 2020.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Learning descriptor networks for 3d shape synthesis and analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8629-8638, 2018.
Ze Yang and Liwei Wang. Learning relationships for multi-view 3d object recognition. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 7505-7514, 2019.
Mang Ye, Xu Zhang, Pong Yuen, and Shih-Fu Chang. Unsupervised embedding learning via in- variant and spreading instance feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6210-6219, 2019.
Tan Yu, Jingjing Meng. and Junsong Yuan. Multi-view harmonized bilinear network for 3d object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 186-194,77 2018.
Shuaihang Yuan and Yi Fang. Ross: Robust learning of one-shot 3d shape segmentation. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1961-1969, 2020.
Xiaohui Zeng, Chenxi Liu, Yu-Siang Wang, Weichao Qiu, Lingxi Xie, Yu-Wing Tai, Chi-Keung Tang, and Alan L Yuille. Adversarial attacks beyond the image space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2019.
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.I DETAILED IMPLEMENTATION
We use the Adam optimizer with batch size of 16 is used to train the entire network. The momen- tum was 0.9, and the weight decay rate is set to 10-5, The learning rate starts at 0.0005 and then decreases by factor of 0.5 every 20 epochs. We train the entire network in 180 epochs and we use a 1024-d latent vector to represent the learned descriptor. To evaluate the shape retrieval scores, we extract 3D descriptors using the trained network. We sort the most relevant shapes for cach query by cosine distance and compute the mAP score.
A.2 3D SHAPE RETRIEVAL TASK ON MODELNET40
We show examples of 3D object retrieval in Figure For each group of 3D shapes, the shape to the left of the vertical line is the query object from the test set, and on the right side of the line, we show the retrieved objects from the training
Query
Figure 3: Qualitative results of 3D shape retrieval task on ModelNet40 dataset:
A.3 SCANOBJECTNN.
addition to ModelNet40 benchmark dataset, we also evaluate our method on ScanObjectNN Uy al. (2019) dataset. ScanObjectNN is the newest and one of the most challenging datasets for 3D object recognition. It is acquired from the real-world objects in 3D point cloud representation and three variants are proposed for different levels of difficulty. 1) Vanilla ScanObjectNN dataset tains clean 3D object point clouds. 2) Background ScanObjectNN dataset which includes real-world background information of each 3D object. 3) Perturbed ScanObjectNN dataset performs random
13

---

Under review as a conference paper at ICLR 2022
Method
Modality
Supervision
Classification
Retrieval
PointNet
point cloud
79.2
64.8
PointNet++
point cloud
84.3
76.6
KD-Networks
point cloud point cloud
84.1
76.1
Shi. et. al.
X
78.7
65.1
MVCNN
12 views
79.8
63.9
MVCNN, metric
12 views
80.1
64.1
RotationNet
12 views
81.4
MHBN
12 views
83.6
3D2Seq Views
12 views
83.8
75.5
MVCNN (GoogLeNet)
12 views
82.0
74.1
MVCNN (GoogLeNet), metric
12 views
82.1
74.3
GVCNN (GoogLeNet)
12 views
83.1
75.9
Ours (MVCNN)
dynamic views
X
84.1
76.2
Ours (GVCNN)
dynamic views
X
85.4
76.8
Table 4: 3D Shape classification and retrieval results on ScanObjectNN dataset. We compare the classification and retrieval performance of our model with previous methods on the ScanOb- jectNN dataset. Our model is trained without any supervision of ground truth but still outperforms most of the supervised methods on classification accuracy.
perturbation the Background ScanObjectNN dataset to increase the difficulty of 3D object recog- nition tasks. In our experiment, we adopt the Vanilla ScanObjectNN dataset to evaluate our method. The reason why we use the "vanilla" version is that other versions of ScanObjectNN contain back- grounds which could lead the propped adversarial view only to reflect the background and occlude the 3D objects. The first variant composed of 2902 3D point clouds and each 3D point cloud has its unique label from 1 to 15. ScanObjectNN (Uy et al., 2019) randomly split into a training set (80%) and test set (20%) and we follow this protocol to conduct our experiment.
3D Object Classification; In this section, we conduct experiments for the task of 3D object clas- sification on the ScanObjectNN benchmark and compare our method with many other methods retrained on the ScanObjectNN dataset. In this comparison, we consider multiple representation learning methods taking different 3D modalities as input, such as point clouds, into consideration. Similar to Section 4.2 we GVCNN and MVCNN as two different backbone modules fe(-). Dif- ferent from Net41, ScanObjectNN is a point-cloud- based 3D representation learning method, we re-render 3D objects from multiple viewpoints. Here, we adopt 12 viewpoints in this experiment. An elevation angle of T/9 9 is applied for all viewpoints with azimuths sampled form [0, T) uniformly.
dataset. To apply the multi-view-
In our proposed model, we adopt Pytorch3D as a differentiable renderer and use its point cloud rendering utilities. We also employ random rotation,randon scaling, and random jittering operations as t/(-) to generate t(x) we specified in Section 4.2 Furthermore, we add the random dropout in t(-) operation as the last augmentation. In this experiment, we use attack with step size e = 0.005 to update rotation parameters a, 3, We adopt the same default linear SVM from the seikit-learn library to perform the downstream task of 3D object classification.
to perform the adversarial
In Table we compare the classification accuracy of our model with other re- trained methods, including PointNet (Qi al. (2017a). PointNet++ (Qi et al. (0076)) KD-Networks KKlokov & Lempitsky 2017). VoxNet (Maturana & Scherer, 2015). 3D-GAN (Wu et al., (016)) MVCNN (uu al. 2015) and GVCNN (Feng et al. (0088), on the ScanObjectNN dataset. As listed in Table 4 Our two backbones achieve 84.1% and 85.4% of accuracy 3D object classification, which outperforms existing supervised methods. This demonstrates our proposed method can ef- fectively find challenging views and hence increase the generalize ability of the backbone model. Specifically, compared with MVCNN, our (VCN()) improves the performance by 2% on accuracy and our Ours(GCVNN) network outperforms GVCNN by 2.3% of accuracy.
Experiment Results:
3D Object Retrieval In this subsection, we follow the protocol that was presented Section 4.2 to
evaluate our proposed method on 3D object retrieval.
14

---

Under review as a conference paper at ICLR 2022
Experiment Results: As shown in the last column of Table our methods achieve the best results compared with other methods listed in the table. Specifically Ours(MVCNN) improves the mAP by 1.9 compared with MVCNN (GoogLeNet), metric (Su et al. 2015). Ours(GVCNN) achieves the retrieval mAP of 76.8 which outperform the baseline result, GVCNN (GoogleNeei (Feng et al.) 2018) 75.9, by 0.9. This improvement validates the effectiveness of our proposed method in the 31> object recognition task.
A.4 EFFECT OF DIFFERENT AUGMENTATION OPERATION:
In section 3.3, we introduce our unsupervised method which leverages the power of contrastive learning which relies on different data augmentation operations. In this section, we conduct various experiments on ModelNet4O to investigate the effect of different data augmentations. As presented T we adopt two different sets of data augmentation operations for mesh objects. The first set (Set 1) of operations contains random rotation, random scaling, and random jittering. The second set (Set 2) of operations contains random scaling and random jittering. The third set (Set 3) of operations only contains random jittering. In addition to mesh objects, we also conduct experiments on point clouds, and three sets of augmentations are used. Moreover, a random dropout of point cloud has the point cloud, first randomly sample
also been added t(-). To validate the performance 2048 points from the ModelNet40 data set, and we use the point cloud rendering function from the that our proposed method
Pytorch3I library to render 3D point clouds. From Table [S] achieves the best result when adopting complex data augmentations. Observing from the second and fourth rows and eighth and tenth rows of Table 5 less during the learning process. This can be attributed to our proposed adversarial attack which can generate rotated adversarial samples.
we find that the rotation operation contributes
Method
Modality
Augmentations
Accuracy
Our((MVCNN)
Mesh
Set I
93.6
Ours(GVNNN)
Mesh
Set
94.3
Our((MVCNN)
Mesh
Set 2
93.5
Ours(GVCNN)
Mesh
Set 2
94.3
Ours(MVCNN)
Mesh
Set 3
93.1
Ours(GVCNN)
Mesh
Set 3
94.1
Ours(MVCNN)
Point Cloud
Set ++dropout
93.1
Ours(GVCNN)
Point Cloud
Set 1+dropout
93.8
Ours(MVCNN)
Point Cloud
Set 2+dropout
93.1
Ours(GVCNN)
Point Cloud
Set 2+dropout
93.78
Ours(MVCNN
Point Cloud
Set 3+dropout
92.6
Ours(GVCNN)
Point Cloud
Set ++dropout
93.2
Table 5: Effect of Different Augmentation We investigate the effect of different augmentations that are adopted during our experiment. Our model is insensitive to rotation augmentation.
Method
Modality
Accuracy
OurS(MVCNN)
Mesh
93.6
Self-Supervised MV.
Mesh
92.3
Ours(GVCNN)
Mesh
94.3
Self-Supervised GV.
Mesh
93.2
OurS(MVCNN)
Point Cloud
93.1
Self-Supervised MV.
Point Cloud
91.8
Ours(GVCNN) Self-Supervised GV.
Point Cloud
93.8
Point Cloud
92.7
Table 6: Effect of Unsupervised Loss We investigate the effect of our proposed adversarial attack by comparing it with a supervised version.
15

---

Under review as a conference paper at ICLR 2022
A.4.1 EFFECT OF UNSUPERVISED LOSS:
In this section, we further investigate the effectiveness of our proposed adversarial attack from two aspects. We train Ours(GVCNN) on the ModelNet40 dataset using 9,843 shapes and compare our proposed method with our method that depreciates the adversarial attack which self-supervised multi-view-based 3D representation learning method. Specifically, we use random rotation, random scaling, and random jittering ((-), and we use
Lad (x, f(z)), t(x'))
(10)
- -log geimtt/oreee),ffooeeeel//l + feiinffine).fffrrteeemnn
the objective function unsupervisedly train the network. The experimental results can be found in Table 6 Our adversarial attack achieved over 1.0% of improvement on classification accuracy comparing with the self-supervised approach. Moreover, our supervised methods also outperform GVCNN and MVCNN by 0.4% and 0. % respectively.
A.5 EFFECT OF ADVERSARIAL VIEWS:
we further conduct an experiment by applying random rotation (as opposed to the gradient-descen guided rotation) to input 3D instance to generate adversarial views to verify the effectiveness of our proposed adversarial views. Specifically, we modify Equation 7 as:
à 3. (p(Aa, A3, (,,3,,)),
(11)
and we follow
{ Mx. i=
add
{
Maddy
i> O,
(12)
generate views by using random Aa, A,3, Ar for each step of adversarial attack, where M is a random initialized rotation matrix, and M is a rotation matrix controlled by â, 3. We use GVCNN the backbone network and follow the same experiment setting in main paper to train the new module. We report the classification accuracy on ModelNet40, denoted as "Random rotation (multiple steps)". Moreover, views. Specifically, we replace the adversarial view generation from our method with a random rotation operation, denoted as "No adversarial views generation", validate its effectiveness. As
compare model with the without using adversarial
Method
Classification acc.
No adversarial views generation Random rotation (multiple steps)
91.4
92.3
Ours
94.3
Table 7: Effect of Adversarial Views. Comparison between our method and random rotation in adversarial view generation.
presented in this table, our method outperforms other two methods by at least 2%, which proves the effectiveness of our proposed adversarial view generation method.
In addition to the aforementioned experiment, we conduct an experiment on the number of back- propagation steps for the rotation matrix to answer this question. We use i = 1, 5, while the other experimental settings remain the same as we presented in Section of the main submission to validate the effectiveness of our method. As listed in Table our method reaches the best perfor- mance when we set i=5 during the training process.
A.6 COMPUTATIONAL OVERHEAD:
We conducted an additional experiment as follows. We first generated a collection of 2D view images offline through rending the 3D objects sampled from ModelNet40. Then we compared the computational overhead for the training with the offline rendered view images and for the training with online view generation (our method). We report the time of forward pass for one input
16

---

Under review as a conference paper at ICLR 2022
Method Classification Acc.
Ours-1
92.4
Ours-3
93.8
Ours-5
94.3
Ours-7
94.2
Table 8: Effect of Adversarial Attack Steps. Ours-i represents the number of adversarial attack adopted during training.
Method
Time
MVCNN 39.89ms
GVCNN
40.12ms
(UVS(M)NNN)
43.39ms
Ours(GVCNN)
43.62ms
Table 9: Computational Overhead Comparison. Please note the time reported in this table for MVCNN and GVCNN exclude the time cost by the offline rendering process. Ours(MVCNN) and Ours(GVCNN) denotes the different backbone networks.
sample. Due to the differentiable rendering process implemented under the Pyotroch framework, the rendering process can be accelerated by GPU which significantly increase the rendering speed. As we can see from the experimental results, our online adversarial views generation during the training does not introduce significant computational overhead.
17


Reviews:
[{"review_id": "T2rcaJbO2jY", "reviewer": "Reviewer_s4iu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- I find the main method of the paper interesting and practical. I particularly like the idea of identifying and rendering challenging viewpoints on the fly, compared to the more static setting of using a pre-rendered and pre-specified set of views adopted by many other works. The combination with contrastive learning is useful to avoid the extra requirement of class labels.\n\n- Regarding rendering new views on the fly, I was wondering if there is a significant computational overhead when it comes to rendering these views, since this happens online during training? Although I like the idea, I can see how this can make training slow.\n\n- In fact, how does this approach compare with just performing extreme data augmentations without mining hard negatives? I think this could be an interesting setting to check for the ablation study. In general, I think, that there are more choices that could be ablated in the main ablation study, instead of just the number of views (e.g., potentially a different hard negative mining method?).\n\n- I like the presentation of the paper overall (e.g., the paper is clearly written, Figure 2 gives intuition about the type of hard examples that the method discovers), but there are quite a few typos (I list some of these below), so I would encourage the authors to do a more thorough proofread to eliminate those.\npg 5 : some wrong pointers leading to ??\npg 5 : \"we can reformulate Equation\" -> equation number missing\npg 6 : \"a PyTorch3D\" -> delete a\npg 6 : one-the-fly -> on-the-fly\npg 7 : we random rotate -> we randomly rotate\npg 7 : mash -> mesh\npg 7 : feces -> faces\npg 7 : Our MVCNN -> Our shouldn't be capitalized\npg 8 : \"In this experiment, test the rotation\" -> In this experiment, we test the rotation\n\n- Evaluation happens on relatively saturated benchmarks, but quantitative performance is solid.\n\n- The citations for the methods in Table 1 are missing. It would be much easier for the readers if the citations were also included in the Table, instead of only the names of the methods.", "summary_of_the_paper": "This paper focuses on the problem of learning a descriptor for 3D shape. The two main contributions of the paper are related to using a mechanism for detecting hard examples for training the descriptor, while also doing it without requiring labels for this procedure. This is done with a combination from techniques inspired by adversarial learning and contrastive learning. The authors compare with a number of methods on the ModelNet40 benchmark.", "summary_of_the_review": "All in all, I think this is an interesting paper, and the method could be applied in other settings as well. In fact, I would like to hear from the authors if they have other ideas for future extensions, because ModelNet is more of a simpler toy setting. I'm currently giving a Weak Accept rating, but I'm very interested in reading the response of the authors and the other reviews.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EQC8dSVoWPC", "reviewer": "Reviewer_6v1k", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The area in which the present paper contributes is important and interesting. The paper proposes an unsupervised descriptor which seems to be efficient. \n\nHowever, I believe in the current form the paper is not ready for a publication. \n\n**1. The presentation is lacking clarity and details.** For example, in section 3.1. Equation 2 uses $t(\\cdot)$ which is not defined until the self-supervised adversarial view generation paragraph. Then $f_{\\theta}$ is introduced prior to eq 2 but not used before eq 3, differential renderer is introduced twice before eq 1. There are missing pointers to sections and equations, notation is not strict $Iadv$ vs $I^{adv}$. All these issues undermine the quality of the paper. Then, in the first sentence of section 4, they say they evaluate on 3 dataset, while in fact they do so on two. Table 1 misses references to baselines, what is VoxNet?. The reader can find VoxNet in references and in supplement, but the name VoxNet is not used in the paper apart from the table 1. Furthermore, important implementation details are missing. It's not clear how they integrate into GVCNN and MVCNN, I guess there can be many ways. I believe, this single concern is sufficient to recommend rejection and encourage resubmission. \n\n**2. Unsupervisedness of the approach.** I cannot really name the current approach unsupervised for object classification at least. MVCNN and GVCNN use labels and it's not clear how it's the descriptors are integrated, whether they're finetuned or anything.\n\n**3. Ablation of adversarial views.** It's not totally clear from the paper if the method is better than just using random augmentations and random rotations, or somehow uniformly distributed.", "summary_of_the_paper": "The paper discusses a method to learning a 3D shape descriptor. They start with meshes of objects, which they render using a differentiable renderer. They pass the rendered images to a feature extraction network $f_{\\theta}$ and obtain a feature embedding. This embedding is used in the contrastive learning procedure. Together with the \"real\" images, they use adversarially perturbed images. The perturbation parameters include rotation angles and random non-linear transformations. At test time they use many images of the object to get object embedding. \n\nThe paper is evaluated on one dataset in the main paper and on another one in the supplement. Two problems are considered: object classification and object retrieval. According to the tables the description can bump the accuracy by a couple of percent. ", "summary_of_the_review": "Overall, I believe the paper is below the bar", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2f1Y4kU0uxW", "reviewer": "Reviewer_GNMf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\nThe method is based on two powerful and novel ideas: \n- using contrastive learning for learning efficient descriptors in a self-supervised way for this task. The self-supervision alone performs impressively well. \n- further improving the method by generating novel adversarial views.  \n\nRemarkably, the method outperforms relevant baselines that are trained in a supervised way.\n\nThe idea of using adversarial views is particularly interesting: in addition to being efficient, adversarial views are conveniently interpretable contrary to imperceptible noise values in more common adversarial methods. They could be useful for other applications in 3D vision due to their interpretability.\n\nThe paper provides comprehensive experiments on multiple datasets while comparing to relevant baselines. Moreover, ablation studies are run extensively on the different parameters and losses. \n\nUp to the results section, the paper is well written and clear. The illustrations are clear and nicely describe the methods and results.\n\n### Cons\n\nThe rule for bold numbers in the tables should be consistent. In table 1, the results in bold are the ones from the presented method (“ours”) (3D2SeqViews/Retrieval is not in bold in table 1). In table 2 the best results are in bold.\n\nStarting from section 4.1 the writing feels rushed and is unclear. It would be good to perform a careful pass over this section. Some sentences are unclear and there are many typos.\n\nA list of a few problems from the text:\n\nSection 3.1:\n- “Section ?? and Section ??” the section numbers are missing\n- “we can reformulate Equation and make” the equation number is missing\n\nSection 4.1:\n- water eight 3D shapes in the mesh format -> watertight (?) 3D shapes in the mesh format \n- we random rotate -> we randomly rotate\n- random scaling the mash object -> randomly scale the mesh\n- “Connotations of all feces remain the same as origin”  this sentence is unclear and there is a typo\n- water-tight propriety -> watertight property\n", "summary_of_the_paper": "The paper proposes a novel method that combines contrastive learning and an adversarial method to improve multi-view  approaches for shape classification and retrieval in a self-supervised way. The authors introduce adversarial views that enable to explicitly train the network on challenging views of a shape. The method is trained in a self-supervised way and does not require any label.", "summary_of_the_review": "The authors propose powerful and novel ideas. They are clearly supported by the performance shown in the extensive experiments and ablation studies. I strongly suggest the authors to work on the writing, especially in the results section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-0hYFvVCOGM", "reviewer": "Reviewer_aAcK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Exploiting a differentiable renderer to optimize for the viewpoints that fool the network the most seems to be effective, as displayed on Fig 2.\n- The method is applied to 2 different backbone networks (MVCNN and GVCNN) and is thus generic.\n- Metrics are indeed in favour of the proposed method compared to a supervised training, with no augmentation or adversarial samples.\n\n\nWeaknesses:\n- The idea of using gradient descent for optimising for the pose of objects that fool a CNN is not novel, and references should be added. For example, see *Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects* (Alcorn et al., CVPR 2019).\n- The benefit of 3D augmentation and adversarial view sampling should clearly be investigated separately. The very last table of the supplementary (Tab. 6) is an attempt at ablating the adversarial view generation but should be made clearer (why is the caption mentioning a “supervised version”?) and moved to the main paper. The impact on final performance on the number of back propagation steps for the rotation matrix would also contribute to prove the method is effective. Similarly, Table 5 in the supplementary is an attempt at quantifying the effect of augmentations, but seems to be carried on already trained networks. What would be interesting is to train networks with different sets of augmentation and describe which augmentations are the most effective.\n- Figure 1 is unclear, and might benefit from being split into 2. Currently it is trying to describe both contrastive learning and adversarial view generation, but both are mixed and therefore not clearly displayed. For example:\n    - Why is a differentiable renderer needed for the augmented instance and the input 3D instance? Having a differentiable renderer is only useful for the adversarial instance.\n    - The caption for “adversarial view generation” introduces the augmented 3D shape $t(x)$, while the adversarial view is not related to this augmentation.\n    - Similarly, a “rotation matrix” is mentioned, but not introduced previously\n- By contrast to the above points, some experiments are displayed in the main paper that are of less relevance, such as “4.4.1 effect of the number of views”: this does not help to quantify the benefits of the proposed method.\n- In Fig. 2, it would be interesting to show how L is increasing during hard sample mining.\n\n- The conclusion mentions a “novel multi-view pipeline”, but the pipeline/network is not novel. What is novel is a way of augmenting data during training, and using a contrastive loss on it.\n\nSmall missing technical details:\n- What is the number of training epochs?\n- What is the size of a descriptor?\n- How is the task of object retrieval carried on?\n\nMany typos, and quite verbose language that does not help having a smooth read:\n- The abstract+intro should state more clearly what the contributions are. The “selection of shape-instance-dependent views towards the learning of more informative 3D shape representation” is not really informative.\n- Fig. 1 “prospective”?\n- 2.2 “required to generatING adversarial…”\n- 3.1 “To this end, the direct adaptation of the adversarial sample generation method prevents us dynamically generating adversarial views due to field viewpoints” is unclear.\n- In the very next sentence the differentiable renderer is introduced twice, and a task specify predictor is introduced but was never mentioned before.\n- Equation (2) refers to t and x’ which were not introduced previously.\n- “Self-supervised Adversarial Views Generation” refers to Sections ?? and ??, and mentions “proprieties” instead of “property”. “We can reformulate Equation ???”\n- 3.2 : “generated from Equation 1, 3, 7” : pick one? Similarly in Algorithm 1: pick one equation and make it self contained?\n- Algorithm 1 refers to L_final, which was not introduced previously.\n- 4.1 Dataset “datasets that WERE acquired” / “WATERTIGHT 3D shapes” \n- 4.1 Implementation and training details: “We adopt a pytorch 3d” -> “We use pytorch3d” . “One-the-fly” -> on-the-fly\n- 4.2 “Random mash” -> mesh. “Connoations of all feces” -> sorry what? “Propriety” -> property.\n- End of Experiment results “ To this end, a significant advantage of our proposed method raises compared” : unclear.\n- …\n- The supplementary has broken references to the main paper (“Section ??”)", "summary_of_the_paper": "The authors train a 3D object descriptor using existing multi view CNN architectures, but improve the training procedure with:\n- augmentations in the form of 3D rotations,\n- adversarial view sampling for forging hard examples during training,\n- an unsupervised (contrastive) loss.\n\nWhen trained with this procedure, CNNs are more performant on downstream tasks such as object retrieval or classification. ", "summary_of_the_review": "The method seems to be working well, but is not very novel and lacks ablation studies to disentangle which components contribute to the performance improvements and how.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received a diverging set of the final ratings: 6, 3, 6, 5. On the positive side, reviewers appreciated practicality of the approach and supporting empirical results. At the same time, all of them expressed concerns with the presentation (typos, unfinished sentences, inconsistent notations). Additional requests for clarifications and ablation studies have been mainly addressed in the rebuttal. The most skeptical reviewer did not participate in the post-rebuttal discussion, thus the final decision took that into account..

The AC has read the paper and verified that the minor technical issues pointed out by the reviewers have been fixed in the updated version (there are still a couple of typos remaining). This submission was further discussed between AC and SAC, as well as in the PC calibration meeting. Both AC and SAC agreed with the comment of Reviewer aAcK who pointed out that generating adversarial samples for mining hard examples has been explored in more general but related contexts before, which limits the novelty of this work to an application of a known idea to a particular domain (3D). At the same time, performance gains on the ModelNet40 dataset are marginal compared to the point cloud based baselines, while the proposed method still uses point clouds for generating adversarial views. In combination with other minor issues pointed out by the reviewers, and given that none of the reviewers was championing the paper, AC and SAC believe that the weaknesses of this paper at the end outweigh its strengths and do not recommend acceptance at this stage.
Decision: Reject