Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations

1 INTRODUCTION . 3D object representation has become increasingly prominent for a wide range of applications , such as 3D object recognition and retrieval ( Maturana & Scherer , 2015 ; Qi et al. , 2016 ; Brock et al. , 2016 ; Qi et al. , 2017a ; b ; Klokov & Lempitsky , 2017 ; Su et al. , 2015 ; Feng et al. , 2018 ; Yu et al. , 2018 ; Yang & Wang , 2019 ) . Recent advances in Convolutional Neural Network ( CNN ) based methods have shown their success in 3D object recognition and retrieval ( Su et al. , 2015 ; Feng et al. , 2018 ; Yu et al. , 2018 ; Yang & Wang , 2019 ) . One important family of methods are view-based methods , which project a 3D object into multiple views and learn compact 3D representation by fusing the feature maps of these views for downstream tasks . Feature learning of multiple views in existing approaches are mostly trained in a supervised fashion , hinging on a large amount of data labels that prevents the wide applicability . Hence , self-supervised learning is in demand to alleviate the dependencies on labels by exploring unlabeled data for the training of multi-view feature representations in an unsupervised or ( semi- ) supervised fashion . Many attempts have been made to explore self-supervisory signals at various levels of visual structures for representation learning . The self-supervised learning framework requires only unlabeled data in order to formulate a pretext learning task ( Kolesnikov et al. , 2019 ) , where a target objective can be computed without any supervision . These pretext tasks can be summarized into four categories ( Jing & Tian , 2019 ) : generation-based ( Zhang et al. , 2016 ; Pathak et al. , 2016 ; Srivastava et al. , 2015 ) , context-based , free semantic label-based ( Faktor & Irani , 2014 ; Stretcu & Leordeanu , 2015 ; Ren & Jae Lee , 2018 ) , and cross modal-based ( Sayed et al. , 2018 ; Korbar et al. , 2018 ) . Among them , context-based pretext tasks include representation learning from image transformations , which is well connected with transformation equivariant representations as they transform equivalently as the transformed images . Transformation Equivariant Representation learning assumes that representations equivarying to transformations are able to encode the intrinsic structures of data such that the transformations can be reconstructed from the representations before and after transformations ( Qi , 2019 ) . Learning transformation equivariant representations has been advocated in Hinton ’ s seminal work on learning transformation capsules ( Hinton et al. , 2011 ) . Following this , a variety of approaches have been proposed to learn transformation equivariant representations ( Kivinen & Williams , 2011 ; Sohn & Lee , 2012 ; Schmidt & Roth , 2012 ; Skibbe , 2013 ; Lenc & Vedaldi , 2015 ; Gens & Domingos , 2014 ; Dieleman et al. , 2015 ; 2016 ; Zhang et al. , 2019 ; Qi et al. , 2019 ; Gao et al. , 2020 ; Wang et al. , 2020 ) . Nevertheless , these works focus on transformation equivariant representation learning of a single modality , such as 2D images or 3D point clouds . In this paper , we propose to learn Multi-View Transformation Equivariant Representations ( MVTER ) by decoding the 3D transformations from multiple 2D views . This is inspired by the equivariant transformations of a 3D object and its projected multiple 2D views . That is , when we perform 3D transformations on a 3D object , the 2D views projected from the 3D object via fixed viewpoints will transform equivariantly . In contrast to previous works where 2D/3D transformations are decoded from the original single image/point cloud and transformed counterparts , we exploit the equivariant transformations of a 3D object and the projected 2D views . We propose to decode 3D transformations from multiple views of a 3D object before and after transformation , which is taken as self-supervisory regularization to enforce the learning of intrinsic 3D representation . By estimating 3D transformations from the fused feature representations of multiple original views and those of the equivariantly transformed counterparts from the same viewpoints , we enable the accurate learning of 3D object representation even with limited amount of labels . Specifically , we first perform 3D transformation on a 3D object ( e.g. , point clouds , meshes ) , and render the original and transformed 3D objects into multiple 2D views with fixed camera setup . Then , we feed these views into a representation learning module to infer representations of the multiple views before and after transformation respectively . A decoder is set up to predict the applied 3D transformation from the fused representations of multiple views before and after transformation . We formulate multi-view transformation equivariant representation learning as a regularizer along with the loss of a specific task ( e.g. , classification ) to train the entire network end-to-end . Experimental results demonstrate that the proposed method significantly outperforms the state-of-the-art view-based models in 3D object classification and retrieval tasks . Our main contributions are summarized as follows . • We propose Multi-View Transformation Equivariant Representations ( MV-TER ) to learn 3D object representations from multiple 2D views that transform equivariantly with the 3D transformation in a self-supervised fashion . • We formalize the MV-TER as a self-supervisory regularizer to learn the 3D object representations by decoding 3D transformation from fused features of projected multiple views before and after the 3D transformation of the object . • Experiments demonstrate the proposed method outperforms the state-of-the-art view-based methods in 3D object classification and retrieval tasks in a self-supervised fashion . 2 RELATED WORKS . In this section , we review previous works on transformation equivariant representations and multiview based neural networks . 2.1 TRANSFORMATION EQUIVARIANT REPRESENTATIONS . Many approaches have been proposed to learn equivariant representations , including transforming auto-encoders ( Hinton et al. , 2011 ) , equivariant Boltzmann machines ( Kivinen & Williams , 2011 ; Sohn & Lee , 2012 ) , equivariant descriptors ( Schmidt & Roth , 2012 ) , and equivariant filtering ( Skibbe , 2013 ) . Lenc & Vedaldi ( 2015 ) prove that the AlexNet ( Krizhevsky et al. , 2012 ) trained on ImageNet learns representations that are equivariant to flip , scaling and rotation transformations . Gens & Domingos ( 2014 ) propose an approximately equivariant convolutional architecture , which utilizes sparse and high-dimensional feature maps to deal with groups of transformations . Dieleman et al . ( 2015 ) show that rotation symmetry can be exploited in convolutional networks for effectively learning an equivariant representation . Dieleman et al . ( 2016 ) extend this work to evaluate on other computer vision tasks that have cyclic symmetry . Cohen & Welling ( 2016 ) propose group equivariant convolutions that have been developed to equivary to more types of transformations . The idea of group equivariance has also been introduced to the capsule nets ( Lenssen et al. , 2018 ) by ensuring the equivariance of output pose vectors to a group of transformations . To generalize to generic transformations , Zhang et al . ( 2019 ) propose to learn unsupervised feature representations via Auto-Encoding Transformations ( AET ) by estimating transformations from the learned feature representations of both the original and transformed images . Qi et al . ( 2019 ) extend AET by introducing a variational transformation decoder , where the AET model is trained from an information-theoretic perspective by maximizing the lower bound of mutual information . Gao et al . ( 2020 ) extend transformation equivariant representations to graph data that are irregularly structured , and formalize graph transformation equivariant representation learning by auto-encoding node-wise transformations in an unsupervised manner . Wang et al . ( 2020 ) extend the AET to Generative Adversarial Networks ( GANs ) for unsupervised image synthesis and representation learning . 2.2 MULTI-VIEW LEARNING . Recently , many view-based approaches have been proposed for 3D object learning . These methods project 3D objects ( e.g. , point clouds , meshes ) into multiple views and extract view-wise features receptively via CNNs , and then fuse these features as the descriptor of 3D objects . Su et al . ( 2015 ) first propose a multi-view convolutional neural network ( MVCNN ) to learn a compact descriptor of an object from multiple views , which fuses view-wise features via a max pooling layer . Qi et al . ( 2016 ) introduce a new multi-resolution component into MVCNNs , and improve the classification performance . However , max pooling only retains the maximum elements from views , which leads to information loss . In order to address this problem , many subsequent works have been proposed to fuse multiple view-wise features into an informative descriptor for 3D objects . Feng et al . ( 2018 ) propose a group-view convolutional neural network ( GVCNN ) framework , which produces a compact descriptor from multiple views using a grouping strategy . Yu et al . ( 2018 ) propose a multi-view harmonized bilinear network ( MHBN ) , which learns 3D object representation by aggregating local convolutional features through the proposed bilinear pooling . To take advantage of the spatial relationship among views , Han et al . ( 2018 ) and Han et al . ( 2019 ) propose to aggregate the global features of sequential views via attention-based RNN and CNN , respectively . Kanezaki et al . ( 2018 ) propose to learn global features by treating pose labels as latent variables which are optimized to self-align in an unsupervised manner . Yang & Wang ( 2019 ) propose a relation network to connect corresponding regions from different viewpoints , and reinforce the information of individual view . Jiang et al . ( 2019 ) propose a Multi-Loop-View Convolutional Neural Network ( MLVCNN ) for 3D object retrieval by introducing a novel loop normalization to generate loop-level features . Wei et al . ( 2020 ) design a view-based GCN framework to aggregate multi-view features by investigating relations of views . 3 MV-TER : THE PROPOSED METHOD . In this section , we first define multi-view equivariant transformation in Section 3.1 . Then we formulate the MV-TER model and introduce the MV-TER framework in Section 3.2 and Section 3.4 , respectively . 3.1 MULTI-VIEW EQUIVARIANT TRANSFORMATION . 2D views are projections of a 3D object from various viewpoints , which transform in an equivariant manner as the 3D object transforms . Formally , given a 3D object M ∈ Rn×3 consisting of n points and a 3D transformation distribution T , we sample a transformation t ∼ T and apply it to M : M̃ = t ( M ) . ( 1 ) We project M onto 2D views from m viewpoints , denoted as V = { V1 , ... , Vm } , i.e. , Vi = pi ( M ) , ( 2 ) where pi : R3 7→ R2 is a projection function for the ith view . Subsequent to the transformation on M , the m views transform equivariantly , leading to Ṽ = { Ṽ1 , ... , Ṽm } . We have Ṽi = pi ( M̃ ) = pi ( t ( M ) ) = fi , t ( Vi ) , i = 1 , ... , m , ( 3 ) where fi , t ’ s are 2D transformations that are equivariant under the same 3D transformation t. Though Vi and Ṽi are projected along the same viewpoint i ( i.e. , the same camera setup ) , they are projections of the original 3D object and its transformed counterpart , thus demonstrating different perspectives of the same 3D object . Our goal is to learn the representations of 3D objects from their multiple 2D views by estimating the 3D transformation t as a pretext task from sampled multiple views before and after the transformation , i.e. , V and Ṽ .

Summary: This paper proposed a self-supervised learning method of 3D shape descriptors for 3D recognition through multi-view 2D image representation learning. To represent the 3D shape, the authors first project the object to a group of 2D project images, which helps apply deep learning due to the image's matrix data format.  The Unsupervised Learning of Transformation Equivariant 2D Representations by Autoencoding Variational Transformations is used for 3D shape descriptor learning, which the authors claimed as "self-supervised" learning. The key idea of transformation equivariant representations is directly borrowed from existing works [1][2]. The method designed is almost the same as [1] except for the encoding network.