Interpreting Representation Quality of DNNs for 3D Point Cloud Processing

1 Introduction . Deep neural networks ( DNNs ) have exhibited superior performance in various tasks , but the blackbox nature of DNNs hampers the analysis of knowledge representations . Previous studies on explainable AI mainly focused on the following two directions . The first is to explain the knowledge encoded in a DNN , e.g . visualizing patterns encoded by a DNN [ 24 , 36 , 50 ] and estimating the saliency/importance/attribution of input variables w.r.t . the network output [ 20 , 30 , 57 ] . The second is to evaluate the representation power of a DNN , e.g . the generalization and robustness of feature representations . This paper focuses on the intersection of the above two directions , i.e . analyzing the quality of knowledge representations of DNNs for 3D point cloud processing . Specifically , we aim to design metrics to illustrate properties of feature representations for different point cloud regions , including various types of regional sensitivities , the spatial smoothness of encoding 3D structures , and the representation complexity of a DNN . These metrics provide new perspectives to diagnose DNNs for 3D point clouds . For example , unlike the image processing relying on color information , the processing of 3D point clouds usually exclusively uses 3D structural information for classification . Therefore , a well-trained DNN for 3D point cloud processing is supposed to just use scale information and 3D structures for inference , and be robust to the rotation and translation . Regional sensitivities . We first propose six metrics to disentangle the overall model vulnerability into the regional rotation sensitivity , the regional translation sensitivity , the regional scale sensitivity , and three types of regional structure sensitivity ( sensitivity to edges , surfaces , and masses ) , so that we can use such sensitivity metrics to evaluate the representation quality of a DNN ( as Fig . 1 ( a-b ) shows ) . Each sensitivity metric ( let us take the rotation sensitivity for example ) is defined as the vulnerability of the regional attribution when we rotate the 3D point cloud with different angles . The regional attribution is computed as the Shapley value , which is widely used as a unique unbiased estimation of an input variable ’ s attribution [ 11 , 20 , 31 , 37 ] from the perspective of game theory . ∗This work was done when Wen Shen was an intern at Shanghai Jiao Tong University . †Quanshi Zhang is the corresponding author . This study was done under the supervision of Dr. Quanshi Zhang . He is with the John Hopcroft Center and the MoE Key Lab of Artificial Intelligence , AI Institute , at the Shanghai Jiao Tong University , China . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . high low sensitivity sensitivity high rotation sensitivity low translation sensitivity scale sensitivity sensitivity to edges sensitivity to surfaces sensitivity to masses ( a ) high low sensitivity sensitivity Based on the sensitivity metrics , we have discovered the following new insights , which have exposed problems with classic DNNs . • Insight 1 . Rotation robustness is the Achilles ’ heel of most DNNs for point cloud processing . The rotation sensitivities of PointNet [ 26 ] , PointNet++ [ 28 ] , DGCNN [ 43 ] , the non-dynamic version of DGCNN ( referred to as GCNN in this paper ) [ 43 ] , and PointConv [ 45 ] are much higher than other sensitivities ( see Fig . 1 ( a ) ) . • Insight 2 . It is usually difficult for DNNs to extract rotation-robust features from 3D points at edges and corners . These points are usually vulnerable to rotation-based attacks . • Insight 3 . What ’ s worse , DNNs usually can not ignore features of such points at edges and corners . Instead , such rotation-sensitive points usually have large regional attributions . • Insight 4 . PointNet fails to model local 3D structures , because convolutional operations in PointNet encode each point independently . Thus , PointNet has low sensitivity to local 3D structures . Spatial smoothness . Beyond the sensitivity metrics , we also evaluate the spatial smoothness of knowledge representations of DNNs . We define the spatial smoothness of knowledge representations as the similarity among the neighboring regions ’ attributions to network output . Most widely-used benchmark datasets for point cloud classification [ 46 , 49 ] only contain objects with simple 3D structures ( e.g . a bed mainly composed of a few surfaces in Fig . 1 ( c ) ) , in which adjacent regions usually have similar 3D structures . Therefore , most adjacent regions are supposed to have similar attributions to the network output . We also find that the adversarial training increases the spatial smoothness of knowledge representations . Representation complexity . Besides regional sensitivities , we further extend the metric of multiorder interaction [ 53 ] to evaluate the representation complexity of a DNN , i.e . the maximum complexity of 3D structures that can be encoded in a DNN . The interaction of the m-th order measures the additional benefits brought by collaboration between two point cloud regions i , j under the contexts of other m regions , where m reflects the contextual complexity of the collaboration between regions i and j. High-order interactions usually represent global and complex 3D structures , corresponding to the collaboration between point cloud regions i , j and massive other regions . Loworder interactions usually describe local and simple 3D structures , without being influenced by many contextual regions ( see Fig . 1 ( d ) ) . Based on such interactions , we have discovered the following new insights . • Insight 5 . Most DNNs fail to encode high-order interactions . I.e . most DNNs mainly focus on local structures and do not extract many global and complex structures from normal samples . • Insight 6 . Rotation-sensitive regions usually activate abnormal high-order interactions ( global and complex structures ) . The very limited global structures modeled by the DNN are usually over-fitted to training samples . Thus , when the sample is rotated , the activated global and complex structures usually appear as abnormal patterns . • Insight 7 . Besides , in terms of model robustness , adversarial training based on adversarial samples w.r.t . using rotations/translations for attack is a typical way to improve the robustness to rotation and translation . Thus , we use the proposed sensitivity metrics to explain the inner mechanism of how and why the adversarially trained DNN is robust to rotation and translation . We find that ( 1 ) the adversarial training shifts the attention from the global orientations and positions to the structural information , thereby boosting the sensitivity to local 3D structures . ( 2 ) The adversarial training usually increases both the quantity and the complexity of 3D structures ( high-order interactions from normal samples ) modeled in a DNN , which boosts the robustness to the rotation and translation . 2 Related work . Deep learning on 3D point clouds : Recently , a series of studies directly used DNNs to process 3D point clouds and have achieved promising performance in various 3D tasks [ 26 , 51 , 42 , 13 , 27 , 47 , 41 , 18 , 10 , 32 , 33 ] . PointNet [ 26 ] was the pioneer to use point-wise multi-layer perceptron to process point clouds and aggregate all individual point features into a global feature . To further extract information from local 3D structures , PointNet++ [ 28 ] recursively applied PointNet to capture hierarchical 3D structures . Likewise , Relation-Shape CNN [ 19 ] and PointCNN [ 18 ] also focused on hierarchical structures and improved the ability to extract contextual information . Due to the irregularity of 3D point clouds , some approaches considered the point set as a graph and further defined graph convolutions . DGCNN [ 43 ] dynamically constructed local graphs and conducted feature extraction via the EdgeConv operation . SPG [ 17 ] built a superpoint graph to process largescale point clouds . GACNet [ 40 ] introduced an attention mechanism in graph convolutional networks . KCNet [ 34 ] proposed kernel correlation and graph pooling to aggregate contextual information . In addition , many approaches have been proposed to apply convolution operators to the point clouds . PointConv [ 45 ] and KPConv [ 39 ] utilized nonlinear functions to construct convolution weights from the input 3D coordinates . InterpCNN [ 23 ] interpolated features of 3D points to the discrete kernel weights coordinates , so as to implement the discrete convolution operator to point clouds . Instead of designing new architectures , our study focuses on the analysis of representation quality of existing classic DNNs for point cloud processing . Visualization or diagnosis of representations : It is intuitive to interpret DNNs by visualizing the feature representations encoded in intermediate layers of DNNs [ 52 , 35 , 50 , 22 , 7 ] , and estimating the pixel-wise saliency/importance/attribution of input variables w.r.t the network output [ 29 , 15 , 8 , 57 , 30 , 5 , 56 ] . Similarly , in the 3D domain , PointNet [ 26 ] visualized the subset of 3D points ( namely the critical subset ) that directly affected the network output . Additionally , Zheng et al . [ 55 ] specified the critical subset via building a gradient-based saliency map . In comparison , our study proposes to evaluate the representation quality of different point cloud regions . Quantitative evaluation of representations : The quantitative evaluation of the representations of DNNs provides a new perspective for explanations . The Shapley value [ 31 ] estimates the attribution distribution over all players in a game and has been applied to quantitative evaluation of the representations of DNNs [ 1 , 9 , 20 ] . Based on Shapley values , some studies have investigated the interaction between input variables of a DNN [ 11 , 21 , 38 , 53 ] . In comparison , our study aims to illustrate distinctive properties of 3D point cloud processing . 3 Analyzing feature representations of DNNs for 3D point cloud processing . We propose six metrics to measure regional sensitivities to rotation , translation , scale , and three types of local 3D structures , respectively , so as to exhibit the representation property of each specific point cloud region . Beyond these metrics , we can further analyze the representation quality from the following two perspectives , i.e . the spatial smoothness of knowledge representations , and the contextual complexity of 3D structures encoded by a DNN . Preliminaries : quantifying the regional attribution using Shapley values . The Shapley value was originally introduced in game theory [ 31 ] . Considering a game with multiple players , each player aims to pursue a high award for victory . The Shapley value is widely considered as a unique unbiased approach that allocates the total reward to each player fairly , which satisfies axioms of linearity , nullity , symmetry , and efficiency [ 44 ] as the theoretical foundation . Please see our supplementary materials for details . Given a point cloud with n regions1 , N = { 1 , 2 , · · · , n } , the Shapley value can be used to measure the attribution of each input region . We can consider the prediction process of a DNN as a game v , and each region in a point cloud as a player i . Let 2N def= { S | S ⊆ N } denote all the possible subsets of N . Let xS denote the point cloud only containing regions in S , in which regions in N\S are removed . For a DNN learned for multi-category classification , we use v ( S ) to denote the network output given the input xS . v ( S ) is calculated as log p1−p , where p = p ( y = y truth | xS ) denotes the probability of the ground-truth category given xS . Note that the number of input points for each DNN is a fixed value . Therefore , in order to make the DNN successfully handle the point cloud xS without regions in N\S , we reset coordinates of points in regions of N\S to the center of the entire point cloud to remove the information of these points , instead of simply deleting these points [ 55 ] . Note that the reason for not simply deleting these points is that the number of input points must be greater than the number of points required for specific layerwise operations of DNNs used in this paper . Specifically , the sampling operation at the first layer of PointNet++ and PointConv requires the minimum number of input points to be greater than 512 points . The k-NN-based grouping [ 43 ] operation of DGCNN and GCNN requires the number of input points to be greater than 20 points . However , the computation of Shapley values requires to make inference on a point cloud fragment with a single region i ( i.e. , x { i } ) , which may have less than 20 points . Therefore , for such DNNs , we can not simply delete 3D points from the point cloud , in order to allow DNNs to work normally . To this end , for a fair comparison , we choose the alternative way , i.e. , placing all points , which are supposed to be masked , to the center of the point cloud . In this way , the numerical attribution of the i-th region to the overall prediction score is estimated by the Shapley value φ ( i ) , as follows . φ ( i ) = ∑ S⊆N\ { i } |S| ! ( n− |S| − 1 ) ! n ! ( v ( S ∪ { i } ) − v ( S ) ) , ( 1 ) The computation of Equation ( 1 ) is NP-hard . Therefore , we use the sampling-based method in [ 4 ] to approximate φ ( i ) .

Summary: This paper studies how to understand the quality of knowledge representations encoded in point cloud DNNs. Based upon the Shapley value, the paper presents six metrics to evaluate the model vulnerability toward different types of input variations. In addition, it presents an analysis regarding the smoothness and multi-order interactions of these attributions, pointing out the restrictions of current point DNNs. The paper makes interesting observations and draws useful conclusions, supported by a range of experimental evaluations.