Contrastive Learning of 3D Shape Descriptor with Dynamic Adversarial Views

1 INTRODUCTION . The impressive performance of Deep Neural Networks ( DNNs ) for 2D visual feature learning naturally motivates 3D computer vision researchers to develop deep learning models for 3D representation for downstream 3D tasks ( e.g. , shape classification ( Qi et al. , 2017b ) , shape correspondence ( Li et al. , 2019 ) , shape retrieval ( Han et al. , 2018 ) , shape registration ( Miao et al. , 2016 ) . To this end , researchers developed the first family of 3D deep learning models that operate directly on 3D raw data represented by point clouds ( Qi et al. , 2017b ; Shi et al. , 2020 ) , triangulated meshes ( Hanocka et al. , 2019 ) , or 3D regular voxels ( Choy et al. , 2019 ) . The recent advancement of approaches in this category has significantly contributed to the shape classification , correspondence , segmentation , registration , and detection . The second important family of 3D deep learning models starts with rendering multiple 2D projections of a 3D object from a static and pre-selected set of different views , followed by learning the compact 3D representation via fusing the deep feature maps of these 2D views . As they can straightforwardly leverage the latest developments of convolutional neural networks , the view-based methods have shown a very attractive capability to learn 3D shape descriptors with superior performance on 3D shape recognition , classification , and retrieval ( Li et al. , 2020 ; Yuan & Fang , 2020 ; Shi et al. , 2020 ) . Particularly , recent advances in view-based approaches lead to impressive state-of-the-art performances in various 3D vision tasks such as 3D shape classification , retrieval , correspondence , and segmentation ( Wei et al. , 2020 ) . The most popular techniques often leverage the ground truth labels to train deep neural networks and learn to extract deep representations by aggregating information from a static and pre-selected set of different views . Those approaches , however , often face challenges posed by the requirement of a large amount of annotated training data and the lack of a mechanism for the adaptive selection of shape-instance-dependent views towards the learning of more informative 3D shape representation . This paper addresses those two challenging issues by introducing the concept of adversarial views ( Xiang et al. , 2019 ) and developing a new mechanism to generate views for the adversarial training of a self-supervised model for 3D representation learning , denoted as CoLAV . Specifically , different from the recent advances in multi-view approaches ( Kanezaki et al. , 2018 ) , CoLAV gains advantages by learning the 3D shape representation without the need for labeled data . In addition , in contrast to using a static set of views as input during the training of a multiview-based 3D shape representation learning framework , we take a further step to investigate the effect of using dynamically rendered views as input during the 3D shape representation learning . To this end , we introduce an adversarial view generator that enables the dynamic generation of shape-instance-dependent adversarial views , which can fool the deep learning model to misidentify its instance-wise identity , as positive pairs to adversarially train robust contrastive learning models towards the learning of more informative 3D shape representation . Our pipeline can be found in Figure 1 . Specifically , our model takes a 3D shape instance x as input , and we first generate an augmented 3D shape t ( x ) by adopting a set of augmentation operations t sampled from a pool of predefined augmentations . Then , we use a randomly initialized a rotation matrix ( characterized by α , β , and γ ) to generate rotated 3D shape xadv0 . For each 3D instance x , t ( x ) , and xadv0 we render k views using the same differentiable renderer rτ ( · ) . The rendered views are fed to a descriptor learning module to extract shape descriptors . After we obtained the descriptors , we use a contrastive loss to measure the mutual information of descriptors . Two key components are employed in our 3D shape unsupervised descriptors learning process . The first one is Adversarial view . We use the contrastive loss to calculate the gradient with respect to α , β , and γ and update the rotation matrix using the updating rule which introduce in the method section . This rotation matrix updating is represented by the purple path . The second component is the contrastive learning module with a shape-instance-dependent adversarial view attack , represented in the red path , which is adopted towards the learning of more informative and robust 3D shape representations . In this component , a new 3D instance xadvi , denoted as adversarial 3D instance , can be obtained using the updated rotation matrix . We adopt the same differentiable renderer rτ ( · ) to render adversarial views for the adversarial training of the Descriptor learning module in order to produce robust and informative 3D shape descriptor . We summarized our contribution as follows : 1 ) We introduce a novel concept of adversarial views of a 3D shape and develop a new mechanism to adaptively and dynamically render shape-instancedependent views towards the learning of more informative 3D shape representation via CoLAV . 2 ) The proposed CoLAV is capable of learning 3D shape representations from multiple 2D views without requiring ground truth annotations . 3 ) Our comprehensive experimental results on 3D shape classification and retrieval demonstrate that the 3D shape descriptor learned by CoLAV exhibits superior performance for 3D shape recognition over other techniques , even though CoLAV is completely self-trained with unlabeled 3D datasets ( e.g. , ModelNet40 ) . 2 RELATED WORKS . 2.1 MULTI-VIEW LEARNING . In recent research , multi-view-based methods have become a popular topic in 3D computer vision . Basically , these methods start with rendering projections of 3D objects from different view perspectives , generate representation features for each projection , and merge them into a global descriptor of the 3D objects . The compact descriptor can be used to improve performance and robustness in 3D object learning tasks such as classification and retrieval . One of the earliest methods in multi-viewbased learning is called multi-view convolutional neural network ( MVCNN ) ( Su et al. , 2015 ) , which fuses feature representations extracted from multiple views into one compact descriptor by a maxpooling layer . Ruizhongtai Qi et al . ( 2016 ) advances the progress by incorporating a multi-resolution component into MVCNN to boost the classification accuracy . However , using a max-pooling layer to fuse representation features from multiple views can only retain the maximum value on each dimension leading to possible losses of information . The group-view convolutional neural network ( GVCNN ) ( Feng et al. , 2018 ) is proposed to solve this problem , which generates a descriptor via a grouping strategy that keeps more information from different views . There is also research attaching attention-based models to the framework , which leverages spatial relationship among different views ’ feature representation ( Han et al. , 2018 ; 2019 ) . Yang & Wang ( 2019 ) also leverages the relationship between views and constructs a model that connects corresponding regions on each projection and strengths the representation learned from each individual viewpoint . The aforementioned view-based methods have achieved promising performance on various tasks . However , their training process requires a large amount of annotated data . To alleviate the dependency of annotated data , Ho et al . ( 2020 ) proposed a self-supervised method that randomly selects a view feature out of several pre-selected static views of a 3D object as a prototype to learn an invariant stochastic object embedding which is further used to generalize the method ( Ho et al. , 2020 ) on test data . Different from all previous view-based methods , we investigate the effect of using dynamic views . Instead of using pre-selected static views , in our work , dynamically rendered views are used to learn an informative multi-view-based 3D object representation , meanwhile , no annotated class labels are given . This encourages the proposed model to better generalize on various unseen views . 2.2 ADVERSARIAL LEARNING . In the concept of adversarial learning , adversarial examples are treated as hard positive samples in the model training process to improve the robustness of neural networks ( Szegedy et al. , 2013 ) . Goodfellow et al . ( 2014 ) introduced the fast gradient sign method ( FGSM ) which increases the robustness of the model by perturbing input samples along with gradient directions . Following this direction , numerous methods ( Kurakin et al. , 2016 ; Papernot et al. , 2016 ; Shafahi et al. , 2019 ; Xie & Yuille , 2019 ; Xie et al. , 2020 ) were proposed and have obtained impressive robustness against adversarial attack in image space . Recently , Zeng et al . ( 2019 ) claimed that instead of generating adversarial samples in 2D space , finding adversarial 3D objects behind the projected 2D images is crucial to improve the robustness . In their work ( Zeng et al. , 2019 ) , FGSM is used to perturb the input sample in 3D space governed by a supervised loss , and ground truth class labels are required to generate adversarial 3D objects . Unlike Zeng et al . ( 2019 ) , in this work , we introduce a new approach to generate adversarial views based on 3D objects without using ground truth labels . 2.3 CONTRASTIVE LEARNING . The concept of contrastive learning has been extensively used in recent studies , especially in selfsupervised ( SSL ) learning scenarios . In SSL 3D object learning , contrastive learning can help retrieve an invariant descriptor for each 3D object by applying random augmentations to each object and minimizing the contrastive loss by measuring the similarity between the feature vectors repre- senting original and augmented shapes . There have been a certain amount of augmentation strategies that came up by previous studies in the 2D computer vision field ( Wu et al. , 2018 ; Ye et al. , 2019 ; Tian et al. , 2019 ) , implementing data augmentation by rotation , cropping , and other transformation methods . Such augmentations can force the network to learn sophisticated descriptors to represent input instances while training , thus boosting robustness . By comparing different methods of augmentation , Chen et al . ( 2020 ) indicates that the effectiveness of contrastive learning can be enhanced by employing a stronger data augmentation . On the other hand , although we have various augmentation methods to boost performance , it is arduous for contrastive learning models to overcome the hard negative mining problem mentioned in Sohn ( 2016 ) ; Wu et al . ( 2017 ) ; Schroff et al . ( 2015 ) . One of the possible solutions to that problem is to select negative samples from a distribution of noise that treats all negative instances equally , which is proposed by the contrastive noise estimation ( NCE ) ( Gutmann & Hyvärinen , 2010 ) , by which contrastive learning is directly inspired Bose et al . ( 2018 ) ; Bose et al . ; Ho & Vasconcelos ( 2020 ) . 3 METHODS . The pipeline of our proposed module is illustrated in Fig . 1 . The CoLAV is a generic module that generates instance-wise adversarial views to adversarially train a contrastive learning model towards the goal of learning a more informative and robust shape descriptor for a given task . 3.1 ADVERSARIAL VIEWS . We introduce our novel approach to obtain adversarial views while alleviating the dependency of annotated data . As we illustrated in Section 2 , although many multi-view-based methods have been proposed , their performances rely on contents of input views which are controlled to camera positions and annotated data . As we observed from existing works ( Su et al. , 2015 ; Feng et al. , 2018 ) , camera positions are static across the entire 3D shape dataset , which leads the trained model vulnerable to unseen views . To address this issue , we introduce the concept of adversarial views as hard positive training samples during the model training , which aims to fool the predictor to make wrong predictions by maximizing the objective function . Adversarial Views Generation . Concretely , the generation of dynamic adversarial views depends on the generation of an adversarial 3D sample xadv . To generate adversarial 3D samples , existing researches focus on perturbing 3D point positions within a certain range in 3D space . This perturbation is almost indistinguishable by human eyes , i.e. , a perturbed 3D sample is visually identical to the original 3D object . Different from the existing adversarial approaches , we propose a method to generate visually different adversarial views of each 3D instance with the aim to fake the model to treat the generated views as new instances . Given a 3D shape x from the dataset D , a differentiable renderer rτ ( · ) : x → I , a rotation matrix M controlled by three randomly initialized parameters α , β , γ , a descriptor learning model fθ ( · ) and a task-specific predictor gφ ( · ) , the generation of xadv is defined as : xadv = { Mx , i = 0 M̂xadvi−1 , i > 0 , ( 1 ) where i denotes the generation iterations and M̂ denotes the updated rotation matrix controlled by α̂ , β̂ , γ̂ which are formulated as follow : α̂ , β̂ , γ̂ = p ( sign ( ∇α , β , γLφ , θ ( xadvi−1 ) + [ α , β , γ ] ) , ( 2 ) where L is the loss for training the neural network fθ ( · ) and gφ ( · ) with parameter φ and θ , the sign ( · ) gives the sign of gradients , and represents the step size of one gradient attack . We employ a function p ( · ) as projection function to clip three angles α̂ , β̂ , γ̂ with in [ −180◦ , 180◦ ] . The adversarial views Iadv is rendered by : Iadv = r ( xadv ) ( 3 ) To train a neural network using adversarial samples generated from Equation 12 , we can directly minimize the L over Iadv . This adversarial training process can be formulated as follow : arg min θ , φ E ( I ) ∼r ( D ) [ max Iadv Lφ , θ ] ( 4 ) The computation of loss termLφ , θ , such as cross-entropy loss requires the ground truth labels , which prevents us from applying this approach to unannotated datasets . To address this , we alleviate the dependency of ground truth annotations by adopting a self-supervised contrastive loss . Self-Supervised Adversarial Views Generation . Specifically , given a pool of augmentation operations T , such as random rotation , random jittering , random scaling , random noise , and random dropout , we randomly select a set of augmentation operations t ∼ T . The specific data augmentation operation that we adopt in our method can be found in Section 4.2 . Our goal is to maximize the similarity between fθ ( r ( x ) ) and fθ ( t ( r ( x ) ) ) , since r ( x ) and t ( r ( x ) ) have the same instance-level proprieties , and to minimize the similarity between fθ ( t ( r ( x ) ) ) and fθ ( t ( r ( x′ ) ) ) , where x′ is a different 3D shape instance . For a clear illustration , we denote r ( x ) as I , t ( r ( x ) ) as Ipos , and t ( r ( x′ ) ) as Ineg . Then , the constrastive learning process can be formulated by minimizing the following loss function : Lco , θ ( I , Ipos , Ineg ) = − log e ( sim ( f ( I ) , f ( Ipos ) ) /Γ ) esim ( f ( I ) , f ( Ipos ) ) /Γ + esim ( f ( I ) , f ( Ineg ) ) /Γ , ( 5 ) where we follow the SimCLR ( Chen et al. , 2020 ) to adopt Γ as a temperature parameter and the sim ( · , · ) function , which is defined as follow : sim ( a , b ) = aT b/‖a‖‖b‖ . ( 6 ) By adopting this formulation , we can reformulate Equation 2 and make it self-supervised , which relaxes the constrain of requiring annotated datasets during adversarial view generation . The selfsupervised version of adversarial view generation can be formulated as follow : α̂ , β̂ , γ̂ = p ( sign ( ∇α , β , γLco , θ ( r ( x ) , { r ( t ( x ) ) , r ( t ( xadvi−1 ) ) } , r ( t ( x′ ) ) ) + [ α , β , γ ] ) , ( 7 )

Summary: The paper discusses a method to learning a 3D shape descriptor. They start with meshes of objects, which they render using a differentiable renderer. They pass the rendered images to a feature extraction network $f_{\theta}$ and obtain a feature embedding. This embedding is used in the contrastive learning procedure. Together with the "real" images, they use adversarially perturbed images. The perturbation parameters include rotation angles and random non-linear transformations. At test time they use many images of the object to get object embedding.   The paper is evaluated on one dataset in the main paper and on another one in the supplement. Two problems are considered: object classification and object retrieval. According to the tables the description can bump the accuracy by a couple of percent.