Understanding Graph Learning with Local Intrinsic Dimensionality

1 INTRODUCTION . Graphs are widely used to model real-life problems owing to their flexible structure and ability to carry different types of information . Graph learning has thus become essential for a wide range of applications in biomedicine ( Zitnik et al. , 2018 ) , physics ( Battaglia et al. , 2016 ) and traffic network ( Yu et al. , 2018 ) . Whilst the rise of Graph Neural Networks ( GNNs ) has enabled important breakthroughs in graph learning ( Senior et al. , 2020 ; Ying et al. , 2018 ) , there is still a lack of understandings of the intrinsic properties of graphs and their impact on learning . In this paper , we narrow this gap by characterizing and analyzing the intrinsic dimensionality ( ID ) of graphs and graph representations based on an expansion-based intrinsic dimensionality measure : Local Intrinsic Dimensionality ( LID ) . Such an analysis is beneficial for the community to better understand the intrinsic difficulty of a graph learning task and motivate advanced GNNs and learning methods . The intrinsic dimensionality of a dataset measures the dimension of its underlying manifold or the minimum number of parameters needed to represent the intrinsic structure of the data ( Bennett , 1969 ; Nakada & Imaizumi , 2020 ) . According to the manifold hypothesis ( Fefferman et al. , 2016 ) in machine learning , the intrinsic dimensionality is often much lower than the representation dimensionality ( the number of features ) for real-world high-dimensional data ( Tenenbaum et al. , 2000 ; Fodor , 2002 ; Cayton , 2005 ; Lin et al. , 2006 ) . LID is an expansion-based ID measure associated with the local neighborhood of data points . In other words , the LID of a point measures the intrinsic dimensionality of the local submanifold surrounding the point and the average LID over all points in a set depicts the dimensionality of the entire manifold . The LID metric has been applied to study the intrinsic complexity of many forms of data , such as images , texts and tabular data ( Pope et al. , 2020 ; Aghajanyan et al. , 2020 ; Ansuini et al. , 2019 ) , as well as the learning and generalization behaviors of deep neural networks . For instance , it has been shown that the LID characteristic of image datasets is closely related to the learning difficulty and generalization performance ( Pope et al. , 2020 ) . For graph learning , we are interested in the intrinsic dimensionality of node features , graph structure , representations learned by GNNs and its indication of the final performance . To this end , we apply LID on a diverse set of graph datasets and estimate the Feature LID ( FLID ) , Structure LID ( SLID ) and Representation LID ( RLID ) for each node . The three graph LID measures are then averaged over all nodes in the graph to reflect the overall intrinsic dimensionality . FLID has the same interpretation as the LID for non-graph data . The SLID of a graph can be interpreted as the expansion rate of the graph as the local neighborhood size of its nodes grows . Both FLID and SLID characterize the properties of the raw graph . RLID , on the other hand , characterizes the properties of the integrated representation of both the node feature and the graph structure . With the three LID measures , we provide the following key insights : • FLID and SLID are good indicators of graph complexity relative to node features and graph structures , respectively . This is verified on synthetic graphs generated using singular value decomposition ( SVD ) and random geometric graph ( RGG ) . • With FLID , we study 5 categories of 12 popular graph datasets including co-author graphs , co-purchase graphs , webpage graphs , citation graphs and Wikipedia graphs , and show that graphs of low FLIDs are generally easier to learn and different GNNs are likely to achieve higher accuracies in downstream node classification tasks . • With RLID and 4 representative GNN models , we show that graph learning is a process that maps the node features and graph structure together onto a simpler manifold that is of a much lower RLID . We also showcase that RLID can be leveraged as a regularizer to improve existing GNN models . • With SLID , we reveal that the underlying graph converges to a complete graph of SLID = 0.5 as the layers of message-passing based GNNs go deep , causing the over-smoothing problem . 2 RELATED WORK . Intrinsic dimensionality analysis plays an important role in dimensionality reduction ( DeMers & Cottrell , 1993 ) , manifold learning ( Law & Jain , 2006 ) , classification ( Gong et al. , 2019 ) , outlier detection ( Houle et al. , 2018 ) , generative modeling ( Li et al. , 2019 ) , adversarial example detection ( Ma et al. , 2018b ) , and deep learning understanding ( Ma et al. , 2018b ; Ansuini et al. , 2019 ; Pope et al. , 2020 ) . The intrinsic dimensionality of a data representation can be estimated either globally on the entire dataset via Principal Component Analysis ( PCA ) ( Wold et al. , 1987 ) , graph based methods ( Costa & Hero , 2003 ) , and fractal models ( Camastra & Staiano , 2016 ) or locally around the individual data points via Local Intrinsic Dimensionality ( LID ) and its variants ( Amsaleg et al. , 2015 ; Houle , 2017 ; Amsaleg et al. , 2019 ) . Different from the global ID measures , LID provides a local view of the intrinsic geometry of the data ( see formal definitions in Section 3 ) . LID has been related to the robustness properties of DNNs to adversarial attacks ( Amsaleg et al. , 2017 ; Ma et al. , 2018a ) and noisy labels ( Ma et al. , 2018b ) . It has been shown that the subspaces around adversarial examples are of much higher LID than of the normal examples in the deep representation space of DNNs ( Ma et al. , 2018a ) . And when there are noisy labels in the training data , DNN learning exhibits two distinctive phases from dimensionality compression to dimensionality expansion and the expansion phase is when the model starts to overfit the noisy labels ( Ma et al. , 2018b ) . The LID of the representations learned by DNNs has also been found to be a good indicator of the generalization performance ( Ansuini et al. , 2019 ) . Both LID and global ID have been applied to characterize the intrinsic dimensionality of image datasets and representations ( Gong et al. , 2019 ; Pope et al. , 2020 ) . The intrinsic dimensionality of the objective space ( defined by the loss function and model parameters ) has also been studied in both natural language processing ( Aghajanyan et al. , 2020 ) and computer vision ( Li et al. , 2018a ) to help understand the parameterization redundancy in DNNs . These understandings have motivated either model compression techniques ( Li et al. , 2018a ) or new theories ( with the intrinsic parameters ) for DNNs ( Aghajanyan et al. , 2020 ) . The current understandings of graphs are mostly focused on the expressive power of GNNs . For example , GNNs have been shown to have equivalent discriminative power to the Weisfeiler-Lehman graph isomorphism test ( Weisfeiler & Leman , 1968 ) . Xu et al . ( 2019 ) showed that GNNs are at most as powerful as the 1-WL test in distinguishing graph structures . Geerts et al . ( 2021 ) further proved that degree-aware Message Passing Neural Networks ( MPNNs ) may be one step ahead of the WL algorithm because of the degree information . Balcilar et al . ( 2021a ) proposed a MPNN model which is experimentally as powerful as a 3-WL test . The learning of GNNs has also been investigated from a spectral perspective . Hoang & Maehara ( 2019 ) argued that GNNs only work as a low-pass filter , which was then verified in Balcilar et al . ( 2021b ) by reformulating most of existing GNNs into one common framework . Oono & Suzuki ( 2019 ) investigated the asymptotic behaviors of GNNs as the layer size tended to infinity and related the expressive power of GNNs to the topological information in the spectral domain . In this work , we apply LID to explore the intrinsic complexity of graphs and graph representations , and provide a set of new and complementary insights into graph learning . 3 LOCAL INTRINSIC DIMENSIONALITY FOR GRAPHS . 3.1 LOCAL INTRINSIC DIMENSIONALITY . Given a data set X ⊂ Rn , X is said to have an intrinsic dimension of m if its elements lie entirely , without information loss , within a m-dimensional manifold of Rn , where m < n ( Fukunaga , 1982 ) . Before introducing LID , we first explain the intuition behind LID based on the expansion-based modeling of dimensionality . Among the family of dimensionality models , the expansion dimension ( ED ) ( Karger & Ruhl , 2002 ) quantify the ID in the vicinity of a point of interest in the data domain . More precisely , it assesses the rate of growth in the number of data points encountered as the distance from the reference point increases . As an example , in the Euclidean space Rm , one can measure the volume Vi of a m-ball of radius ri with i ∈ { 1 , 2 } , taking the logarithm of the ratio would reveal the dimension m : V2V1 = ( r2 r1 ) m ⇒ m = ln ( V2/V1 ) ln ( r2/r1 ) . Transferring the concept of expansion dimension to the statistical setting with neighborhood distance distributions gives us the formal definition of LID ( Houle , 2017 ) . Definition 1 ( Local Intrinsic Dimensionality ) Given a data sample x ∈ X , let R > 0 be a random variable denoting the distance from x to other data samples . If the cumulative distribution function F ( r ) of R is positive and continuously differentiable at distance r > 0 , the LID of x at distance r is given by : LIDF ( r ) ≜ lim ϵ→0 F ( ( 1 + ϵ ) r ) − F ( r ) ) ϵ · F ( r ) = r · F ′ ( r ) F ( r ) , ( 1 ) The local intrinsic dimension at x is then defined as the limit , as the radius r tends to zero , i.e . LIDF ≜ lim r→0 LIDF ( r ) . Here , the CDF F ( r ) is analogous to the volume in the Euclidean example . Since F ( r ) is unknown , estimators are needed for LID . There already exist a number of LID estimators in the literature ( Levina & Bickel , 2005 ; Amsaleg et al. , 2015 ; Liao et al. , 2014 ) . In the following , we will introduce one commonly used LID estimator and how it can be applied on graphs . 3.2 FEATURE AND REPRESENTATION LID . For graphs , we are interested in the LIDs of the nodes features and structure of the graph itself and node representations learned by GNNs . Node features and graph structure are two fundamental information of the graph , while the learned representation for a node is an integration of its feature and the structural information . The existing LID estimators developed for non-graph data can be directly applied to node features and node representation . Here , we first introduce the LID estimation for Feature LID ( FLID ) and Representation LID ( RLID ) . Amongst the existing LID estimators , the Maximum Likelihood Estimator ( MLE ) ( Levina & Bickel , 2005 ; Amsaleg et al. , 2015 ) is one of the most cited estimators . It treats the neighbors of each point x ∈ X as events in a Poisson process and the distance r ( j ) ( x ) between x and its j-th nearest neighbor as the event ’ s arrival time . Since this process depends on the dimensionality d , MLE estimates the intrinsic dimension by maximizing the log-likelihood of the observed process . The node features or representations are represented as vectors in the Euclidean space . Thus , FLID and RLID can be directly estimated by MLE . Let x denote the feature/representation vector of a particular node , the FLID/RLID of x can be estimated as following : FLID/RLID ( x , k ) = ( 1 k k∑ j=1 log r ( k+1 ) ( x ) r ( j ) ( x ) ) −1 , ( 2 ) where k is the neighborhood size ( i.e. , k-nearest ) and r ( i ) ( x ) is the Euclidean distance between x and its i-th nearest neighbor . Averaging FLID ( x , k ) across all nodes x in { xi } Ni=1 leads to the FLID of the entire graph , i.e. , FLIDG ( k ) = 1N ∑N i=1 FLID ( xi , k ) . Similarly , we can obtain RLIDG ( k ) = 1 N ∑N i=1 RLID ( xi , k ) . The k-nearest neighbors are identified based on the pairwise distance between all nodes in the graph .

Summary: The paper characterizes the intrinsic dimensionality of node features, graph structures, and representations learned by GNNs via the so-called Local Intrinsic Dimensionality (LID) measure, intending that it can benefit the community in understanding the difficulty of an underlying graph learning task. In addition, estimators for Feature LID (FLID), Structure LID (SLID), and Representation LID (RLID) were introduced. This work showed that real-world graphs have much lower intrinsic dimensionality when compared to their extrinsic dimensionality.