Understanding Graph Learning with Local Intrinsic Dimensionality

1 INTRODUCTION . Graphs are widely used to model real-life problems owing to their flexible structure and ability to carry different types of information . Graph learning has thus become essential for a wide range of applications in biomedicine ( Zitnik et al. , 2018 ) , physics ( Battaglia et al. , 2016 ) and traffic network ( Yu et al. , 2018 ) . Whilst the rise of Graph Neural Networks ( GNNs ) has enabled important breakthroughs in graph learning ( Senior et al. , 2020 ; Ying et al. , 2018 ) , there is still a lack of understandings of the intrinsic properties of graphs and their impact on learning . In this paper , we narrow this gap by characterizing and analyzing the intrinsic dimensionality ( ID ) of graphs and graph representations based on an expansion-based intrinsic dimensionality measure : Local Intrinsic Dimensionality ( LID ) . Such an analysis is beneficial for the community to better understand the intrinsic difficulty of a graph learning task and motivate advanced GNNs and learning methods . The intrinsic dimensionality of a dataset measures the dimension of its underlying manifold or the minimum number of parameters needed to represent the intrinsic structure of the data ( Bennett , 1969 ; Nakada & Imaizumi , 2020 ) . According to the manifold hypothesis ( Fefferman et al. , 2016 ) in machine learning , the intrinsic dimensionality is often much lower than the representation dimensionality ( the number of features ) for real-world high-dimensional data ( Tenenbaum et al. , 2000 ; Fodor , 2002 ; Cayton , 2005 ; Lin et al. , 2006 ) . LID is an expansion-based ID measure associated with the local neighborhood of data points . In other words , the LID of a point measures the intrinsic dimensionality of the local submanifold surrounding the point and the average LID over all points in a set depicts the dimensionality of the entire manifold . The LID metric has been applied to study the intrinsic complexity of many forms of data , such as images , texts and tabular data ( Pope et al. , 2020 ; Aghajanyan et al. , 2020 ; Ansuini et al. , 2019 ) , as well as the learning and generalization behaviors of deep neural networks . For instance , it has been shown that the LID characteristic of image datasets is closely related to the learning difficulty and generalization performance ( Pope et al. , 2020 ) . For graph learning , we are interested in the intrinsic dimensionality of node features , graph structure , representations learned by GNNs and its indication of the final performance . To this end , we apply LID on a diverse set of graph datasets and estimate the Feature LID ( FLID ) , Structure LID ( SLID ) and Representation LID ( RLID ) for each node . The three graph LID measures are then averaged over all nodes in the graph to reflect the overall intrinsic dimensionality . FLID has the same interpretation as the LID for non-graph data . The SLID of a graph can be interpreted as the expansion rate of the graph as the local neighborhood size of its nodes grows . Both FLID and SLID characterize the properties of the raw graph . RLID , on the other hand , characterizes the properties of the integrated representation of both the node feature and the graph structure . With the three LID measures , we provide the following key insights : • FLID and SLID are good indicators of graph complexity relative to node features and graph structures , respectively . This is verified on synthetic graphs generated using singular value decomposition ( SVD ) and random geometric graph ( RGG ) . • With FLID , we study 5 categories of 12 popular graph datasets including co-author graphs , co-purchase graphs , webpage graphs , citation graphs and Wikipedia graphs , and show that graphs of low FLIDs are generally easier to learn and different GNNs are likely to achieve higher accuracies in downstream node classification tasks . • With RLID and 4 representative GNN models , we show that graph learning is a process that maps the node features and graph structure together onto a simpler manifold that is of a much lower RLID . We also showcase that RLID can be leveraged as a regularizer to improve existing GNN models . • With SLID , we reveal that the underlying graph converges to a complete graph of SLID = 0.5 as the layers of message-passing based GNNs go deep , causing the over-smoothing problem . 2 RELATED WORK . Intrinsic dimensionality analysis plays an important role in dimensionality reduction ( DeMers & Cottrell , 1993 ) , manifold learning ( Law & Jain , 2006 ) , classification ( Gong et al. , 2019 ) , outlier detection ( Houle et al. , 2018 ) , generative modeling ( Li et al. , 2019 ) , adversarial example detection ( Ma et al. , 2018b ) , and deep learning understanding ( Ma et al. , 2018b ; Ansuini et al. , 2019 ; Pope et al. , 2020 ) . The intrinsic dimensionality of a data representation can be estimated either globally on the entire dataset via Principal Component Analysis ( PCA ) ( Wold et al. , 1987 ) , graph based methods ( Costa & Hero , 2003 ) , and fractal models ( Camastra & Staiano , 2016 ) or locally around the individual data points via Local Intrinsic Dimensionality ( LID ) and its variants ( Amsaleg et al. , 2015 ; Houle , 2017 ; Amsaleg et al. , 2019 ) . Different from the global ID measures , LID provides a local view of the intrinsic geometry of the data ( see formal definitions in Section 3 ) . LID has been related to the robustness properties of DNNs to adversarial attacks ( Amsaleg et al. , 2017 ; Ma et al. , 2018a ) and noisy labels ( Ma et al. , 2018b ) . It has been shown that the subspaces around adversarial examples are of much higher LID than of the normal examples in the deep representation space of DNNs ( Ma et al. , 2018a ) . And when there are noisy labels in the training data , DNN learning exhibits two distinctive phases from dimensionality compression to dimensionality expansion and the expansion phase is when the model starts to overfit the noisy labels ( Ma et al. , 2018b ) . The LID of the representations learned by DNNs has also been found to be a good indicator of the generalization performance ( Ansuini et al. , 2019 ) . Both LID and global ID have been applied to characterize the intrinsic dimensionality of image datasets and representations ( Gong et al. , 2019 ; Pope et al. , 2020 ) . The intrinsic dimensionality of the objective space ( defined by the loss function and model parameters ) has also been studied in both natural language processing ( Aghajanyan et al. , 2020 ) and computer vision ( Li et al. , 2018a ) to help understand the parameterization redundancy in DNNs . These understandings have motivated either model compression techniques ( Li et al. , 2018a ) or new theories ( with the intrinsic parameters ) for DNNs ( Aghajanyan et al. , 2020 ) . The current understandings of graphs are mostly focused on the expressive power of GNNs . For example , GNNs have been shown to have equivalent discriminative power to the Weisfeiler-Lehman graph isomorphism test ( Weisfeiler & Leman , 1968 ) . Xu et al . ( 2019 ) showed that GNNs are at most as powerful as the 1-WL test in distinguishing graph structures . Geerts et al . ( 2021 ) further proved that degree-aware Message Passing Neural Networks ( MPNNs ) may be one step ahead of the WL algorithm because of the degree information . Balcilar et al . ( 2021a ) proposed a MPNN model which is experimentally as powerful as a 3-WL test . The learning of GNNs has also been investigated from a spectral perspective . Hoang & Maehara ( 2019 ) argued that GNNs only work as a low-pass filter , which was then verified in Balcilar et al . ( 2021b ) by reformulating most of existing GNNs into one common framework . Oono & Suzuki ( 2019 ) investigated the asymptotic behaviors of GNNs as the layer size tended to infinity and related the expressive power of GNNs to the topological information in the spectral domain . In this work , we apply LID to explore the intrinsic complexity of graphs and graph representations , and provide a set of new and complementary insights into graph learning . 3 LOCAL INTRINSIC DIMENSIONALITY FOR GRAPHS . 3.1 LOCAL INTRINSIC DIMENSIONALITY . Given a data set X ⊂ Rn , X is said to have an intrinsic dimension of m if its elements lie entirely , without information loss , within a m-dimensional manifold of Rn , where m < n ( Fukunaga , 1982 ) . Before introducing LID , we first explain the intuition behind LID based on the expansion-based modeling of dimensionality . Among the family of dimensionality models , the expansion dimension ( ED ) ( Karger & Ruhl , 2002 ) quantify the ID in the vicinity of a point of interest in the data domain . More precisely , it assesses the rate of growth in the number of data points encountered as the distance from the reference point increases . As an example , in the Euclidean space Rm , one can measure the volume Vi of a m-ball of radius ri with i ∈ { 1 , 2 } , taking the logarithm of the ratio would reveal the dimension m : V2V1 = ( r2 r1 ) m ⇒ m = ln ( V2/V1 ) ln ( r2/r1 ) . Transferring the concept of expansion dimension to the statistical setting with neighborhood distance distributions gives us the formal definition of LID ( Houle , 2017 ) . Definition 1 ( Local Intrinsic Dimensionality ) Given a data sample x ∈ X , let R > 0 be a random variable denoting the distance from x to other data samples . If the cumulative distribution function F ( r ) of R is positive and continuously differentiable at distance r > 0 , the LID of x at distance r is given by : LIDF ( r ) ≜ lim ϵ→0 F ( ( 1 + ϵ ) r ) − F ( r ) ) ϵ · F ( r ) = r · F ′ ( r ) F ( r ) , ( 1 ) The local intrinsic dimension at x is then defined as the limit , as the radius r tends to zero , i.e . LIDF ≜ lim r→0 LIDF ( r ) . Here , the CDF F ( r ) is analogous to the volume in the Euclidean example . Since F ( r ) is unknown , estimators are needed for LID . There already exist a number of LID estimators in the literature ( Levina & Bickel , 2005 ; Amsaleg et al. , 2015 ; Liao et al. , 2014 ) . In the following , we will introduce one commonly used LID estimator and how it can be applied on graphs . 3.2 FEATURE AND REPRESENTATION LID . For graphs , we are interested in the LIDs of the nodes features and structure of the graph itself and node representations learned by GNNs . Node features and graph structure are two fundamental information of the graph , while the learned representation for a node is an integration of its feature and the structural information . The existing LID estimators developed for non-graph data can be directly applied to node features and node representation . Here , we first introduce the LID estimation for Feature LID ( FLID ) and Representation LID ( RLID ) . Amongst the existing LID estimators , the Maximum Likelihood Estimator ( MLE ) ( Levina & Bickel , 2005 ; Amsaleg et al. , 2015 ) is one of the most cited estimators . It treats the neighbors of each point x ∈ X as events in a Poisson process and the distance r ( j ) ( x ) between x and its j-th nearest neighbor as the event ’ s arrival time . Since this process depends on the dimensionality d , MLE estimates the intrinsic dimension by maximizing the log-likelihood of the observed process . The node features or representations are represented as vectors in the Euclidean space . Thus , FLID and RLID can be directly estimated by MLE . Let x denote the feature/representation vector of a particular node , the FLID/RLID of x can be estimated as following : FLID/RLID ( x , k ) = ( 1 k k∑ j=1 log r ( k+1 ) ( x ) r ( j ) ( x ) ) −1 , ( 2 ) where k is the neighborhood size ( i.e. , k-nearest ) and r ( i ) ( x ) is the Euclidean distance between x and its i-th nearest neighbor . Averaging FLID ( x , k ) across all nodes x in { xi } Ni=1 leads to the FLID of the entire graph , i.e. , FLIDG ( k ) = 1N ∑N i=1 FLID ( xi , k ) . Similarly , we can obtain RLIDG ( k ) = 1 N ∑N i=1 RLID ( xi , k ) . The k-nearest neighbors are identified based on the pairwise distance between all nodes in the graph .

Summary: In this work, the authors investigate the Local Intrinsic Dimensionality (LID), especially the feature (FLID), structure LID (SLID), and Representation LID (RLID) of a graph. Through experimental analysis, the authors demonstrate that the FLID and SLID are well correlated with the graph complexity, and real-world graphs have a much lower intrinsic dimensionality. In addition, authors also interpret the over-smoothing problem associated with GNN models from the perspective of the SLID's convergence.