On the Relationship between Heterophily and Robustness of Graph Neural Networks

1 INTRODUCTION . Graph neural networks ( GNNs ) aim to translate the enormous empirical success of deep learning to data defined on non-Euclidean domains , such as manifolds or graphs ( Bronstein et al. , 2017 ) , and have become important tools to solve a variety of learning problems for graph structured and geometrically embedded data . However , recent works show that GNNs—much like their “ standard ” deep learning counterparts—have a high sensitivity to adversarial attacks : intentionally introduced minor changes in the graph structure can lead to significant changes in performance . This finding , first articulated by Zügner et al . ( 2018 ) and Dai et al . ( 2018 ) , has triggered studies that investigated different attack scenarios ( Xu et al. , 2019 ; Wu et al. , 2019 ; Li et al. , 2020a ; Ma et al. , 2020 ) . A different aspect of GNNs that has been scrutinized recently is that most GNNs do not perform well with many heterophilous datasets . GNNs generally perform well under homophily ( or assortativity ) , i.e. , the tendency of nodes with similar features or class labels to connect ( Pei et al. , 2020 ; Zhu et al. , 2020 ) . Such datasets are thus called homophilous ( or assortative ) . While homophilous datasets dominate the study of networks , homophily is not a universal principle ; certain networks , such as romantic relationship networks or predator-prey networks in ecology , are mostly heterophilous ( or disassortative ) . Employing a GNN which does not account for heterophily can lead to significant performance loss in heterophilous settings ( Abu-El-Haija et al. , 2019 ; Zhu et al. , 2020 ; Bo et al. , 2021 ) . Previous works have thus proposed architectures for heterophilous data . While previous work has focused on naturally-occurring heterophily , heterophilous interactions may also be introduced as adversarial noise : as many GNNs exploit homophilous correlation , they can be sensitive to changes that render the data more heterophilous . A natural follow-up question is if and how this observation manifests itself in previously proposed attacking strategies on GNNs . In this work , we thus investigate the relation between heterophily and robustness of GNNs against adversarial perturbations of graph structure , focusing on semi-supervised node classification tasks . More specifically , our main contributions are : • Formalization : We formalize the relation between adversarial structural attacks and the change of homophily level in the underlying graphs with theoretical ( §3.1 ) and empirical ( §5.1 ) analysis . Specifically , we show that on homophilous graphs , effective structural attacks lead to increased heterophily , while , on heterophilous graphs , they alter the homophily level contingent on node degrees . To our knowledge , this is the first formal analysis of such kind . • Heterophily-inspired Design : We show how the relation between attacks and heterophily can inspire more robust GNNs by demonstrating that a key architectural feature in handling heterophily , separate aggregators for ego- and neighbor-embeddings , also improves the robustness of GNNs against attacks ( §3.2 ) . • Extensive Empirical Analysis : We show the effectiveness of the heterophilous design in improving empirical ( §5.2 ) and certifiable ( §5.3 ) robustness of GNNs with extensive experiments on real-world homophilous and heterophilous datasets . Specifically , we compare GNNs with this design , which we refer to as heterophily-adjusted GNNs , to non-adjusted models , including stateof-the-art models designed with robustness in mind . We find that heterophily-adjusted GNNs are up to 5 times more certifiably robust and have stronger performance under attacks by up to 32.92 % compared to non-adjusted , standard models . Moreover , this design can be combined with existing vaccination mechanisms , yielding up to 18.33 % higher accuracy under attacks than the best non-adjusted vaccinated model . 2 NOTATION AND PRELIMINARIES . Let G = ( V , E , X ) be a simple graph with node set V , edge set E and node attributes X . The one-hop neighborhood N ( v ) = { u : ( u , v ) ∈ E } of a node v ∈ V is the set of all nodes directly adjacent to v ; the k-hop neighborhood of v ∈ V is the set of nodes reachable by a shortest path of length k. We represent the graph G algebraically by an adjacency matrix A ∈ { 0 , 1 } |V|×|V| and node feature matrix X ∈ R|V|×F . We use As = A+ I to denote the adjacency matrix with self-loops added , and denote the corresponding row-stochastic matrices as Ā = D−1A and Ās = D−1s As , respectively , where D is a diagonal matrix with Dii = ∑ jAij ( Ds is defined analogously ) . We further assume that there exists a vector y , which contains a unique class label yv for each node v. Given a training set TV = { ( v1 , y1 ) , ( v2 , y2 ) , ... } of labeled nodes , the goal of semi-supervised node classification is to learn a mapping ` : V → Y from the nodes to the set Y of class labels . Graph neural networks ( GNNs ) . Most current GNNs operate according to a message passing paradigm where a representation vector rv is assigned to each node v ∈ V and continually updated by K layers of learnable transformations . These layers first aggregate representations over neighboring nodes N ( v ) and then update the current representation via an encoder ENC . For prevailing GNN models like GCN ( Kipf & Welling , 2017 ) and GAT ( Veličković et al. , 2018 ) , each layer can be formalized as r ( k ) v = ENC ( AGGR ( { r ( k−1 ) u : u ∈ N ( v ) ∪ { v } } ) ) , where AGGR is the mean func- tion weighted by node degrees ( GCN ) or an attention mechanism ( GAT ) , and ENC is a learnable ( nonlinear ) mapping . Adversarial attacks on graphs . Given a graph G = ( V , E , X ) and a GNN f that processes G , an adversarial attacker tries to create a perturbed graph G′ = ( V , E ′ , X ) with a modified edge-set E ′ such that the performance of the GNN f is maximally degraded . The information available to the attacker can vary under different scenarios ( Jin et al. , 2020a ; Sun et al. , 2020 ) . Here , we follow the gray-box formalization by Zügner et al . ( 2018 ) , where the attacker knows the training set TV , but not the trained GNN f . The attacker thus considers a surrogate GNN and picks perturbations that maximize an attack loss Latk ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019a ) , assuming that attacks to the surrogate model are transferable to the attacked GNN . For node classification , the attack loss Latk quantifies how the predictions zv ∈ [ 0 , 1 ] |Y| made by the GNN f differ from the true labels y . For a targeted attack of node v with class label yv ∈ Y , we adopt the negative classification margin ( CM-type ) ( Zügner et al. , 2018 ; Xu et al. , 2019 ) : Latk = −∆c = − ( zv , yv − maxy 6=yv zv , y ) . The attacker usually has additional constraints , such as a limit on the size of the perturbations allowed ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019a ) . Taxonomy of attacks . We follow the taxonomy of attacks introduced in ( Jin et al. , 2020a ; Sun et al. , 2020 ) . For node classification , the attacker may aim to change the classification of a specific node v ∈ V ( targeted attack ) , or to decrease the overall classification accuracy ( untargeted attack ) . Attacks can also happen at different stages of the training process : we refer to attacks introduced before training as ( pre-training ) poison attacks , and attacks introduced after the training process ( and before potential retraining on perturbed data ) as ( post-training ) evasion attacks . While our theoretical analysis ( §3 ) mainly considers targeted evasion attacks , we consider other attacks in our empirical evaluation ( §5 ) . Characterizing homophily and heterophily in graphs . Using class labels , we characterize the types of connections in a graph contributing to its overall level of homophily/heterophily as follows : Definition 1 ( Homo/Heterophilous path and edge ) A k-hop homophilous path from node w to u is a length-k path between endpoint nodes with the same class label yw = yu . Otherwise , the path is called heterophilous . A homophilous or heterophilous edge is a special case with k = 1 . Following ( Zhu et al. , 2020 ; Lim et al. , 2021 ) , we define the homophily ratio h as : Definition 2 ( Homophily ratio ) The homophily ratio is the fraction of homophilous edges among all the edges in a graph : h = | { ( u , v ) ∈ E|yu = yv } |/|E| . When the edges in a graph are wired randomly , independent to the node labels , the expectation for h is hr = 1/|Y| for balanced classes ( Lim et al. , 2021 ) . For simplicity , we informally refer to graphs with homophily ratio h 1/|Y| as homophilous graphs ( which have been the focus in most prior works ) , graphs with homophily ratio h 1/|Y| as heterophilous graphs , and graphs with homophily ratio h ≈ 1/|Y| as weakly heterophilous graphs . 3 RELATION BETWEEN GRAPH HETEROPHILY & MODEL ROBUSTNESS . In this section , we first show theoretical results on the relation between adversarial structural attacks and the change in the homophily level of the underlying graphs . Though empirical analyses from previous works have suggested this relation on homophilous graphs ( Wu et al. , 2019 ; Jin et al. , 2020a ) , to our knowledge , we are the first to formalize it with theoretical analysis and address the case of heterophilous graphs . As an implication of the relation , we then discuss how a key design that improves predictive performance of GNNs under heterophily can also help boost their robustness . 3.1 HOW DO STRUCTURAL ATTACKS CHANGE HOMOPHILY IN GRAPHS ? . Homophilous Graphs : Structural Attacks are Mostly Heterophilous Attacks . Our first result shows that , for homophilous data , effective structural attacks on GNNs ( as measured by loss Latk ) always result in a reduced level of homophily where either new heterophilous connections are added or existing homophilous connections are removed . It also states that direct perturbations on 1-hop neighbors of the target nodes are more effective than indirect perturbations ( influencer attacks ( Zügner et al. , 2018 ) ) on multi-hop neighbors . For simplicity , akin to previous works ( Zügner et al. , 2018 ; Zügner & Günnemann , 2019a ) we establish our results for targeted evasion ( post-training ) attacks in a stylized learning setup with a linear GNN . However , our findings generalize to more general setups on real-world datasets as we show in our experiments ( §5.1 ) . In the theorems below , we use the notion of gambit node : node u is called a gambit if a perturbation that targets node v ∈ V adjusts the connectivity of node u ∈ V . Theorem 1 Let G = ( V , E , X ) be a self-loop-free graph with adjacency matrix A and node features xv = p · onehot ( yv ) + 1−p|Y| · 1 for each node v , where 1 is an all-1 vector , and p is a parameter that regulates the signal to noise ratio . Assume that a fraction h of each node ’ s neighbors belong to the same class , while a fraction 1−h|Y|−1 belongs uniformly to any other class . Consider a 2-layer linear GNN f ( 2 ) s ( A , X ) = Ā2sXW trained on a training set TV ⊆ DV , with at least one node from each class y ∈ Y , and degree d for all nodes with a distance less than 2 to any v ∈ DV . For a unit structural perturbation that involves a target node v ∈ DV , and a correctly classified gambit node with degree da , the following statements hold if h ≥ 1|Y| : 1. the attack loss Latk ( §2 ) of the target v increases only for actions increasing heterophily , i.e. , when removing a homophilous edge or path , or adding a heterophilous edge or path to node v ; 2. direct perturbations on edges ( or 1-hop paths ) incident to the target node v lead to greater increase in Latk than indirect perturbations on multi-hop paths to target node v. We give the proof in App . C.1 . Intuitively , the relative inability of existing GNNs to make full use of heterophilous data ( Pei et al. , 2020 ; Zhu et al. , 2020 ) can be exploited by inserting heterophilous connections in graphs where homophilous ones are expected . Though the theorem shows that effective attacks on homophilous graphs necessarily reduce the homophily level , the converse is not true : not all perturbations which reduce the homophily level are effective attacks ( Ma et al. , 2021 ) . Heterophilous Graphs : Structural Attacks Can Be Homophilous or Heterophilous , depending on Node Degrees . When a graph displays heterophily , our analysis shows a more complicated picture on how the level of homophily in the graph is changed by effective structural attacks : in heterophilous case , the direction of change is dependent on the degrees of both the target node v and the gambit node u of the attack . Specifically , if the degree of either node is low , attacks increasing the heterophily are still effective ; however , if the degrees d and da of both nodes are high , attacks decreasing the heterophily will be effective . Similar to the homophilous case , we formalize our results below for targeted evasion attacks in a stylized learning setup . Theorem 2 Under the setup of Thm . 1 , for a unit perturbation that involves a target node v with degree d , and a correctly classified gambit node with degree da , the following statements hold : 1 . ( Low-degree target node ) if 0 < d ≤ |Y| − 2 , for any da ≥ 0 and h ∈ [ 0 , 1 ] , the attack loss Latk ( §2 ) of v increases only under actions increasing heterophily in the graph ; 2 . ( High-degree target node ) if d > |Y| − 2 , conditioning on the degree da of the gambit node : ( a ) ( Low-degree gambit node ) if da < ( d+2 ) ( |Y|−1 ) d−|Y|+2 , for any h ∈ [ 0 , 1 ] , the attack loss Latk ( §2 ) of v increases only under actions increasing heterophily in the graph ; ( b ) ( High-degree gambit node ) if da ≥ ( d+2 ) ( |Y|−1 ) d−|Y|+2 , for 0 ≤ h < da ( d−|Y|+2 ) − ( d+2 ) ( |Y|−1 ) ( d+1 ) |Y|da < 1 |Y| , Latk ( §2 ) of v increases only under actions reducing heterophily in the graph . In the statements above , the actions increasing heterophily in the graph include removing a homophilous edge or adding a heterophilous edge to node v , and the actions reducing heterophily in the graph include when adding a homophilous edge or removing a heterophilous edge to node v. The above theorems cover the situation when the gambit nodes are initially classified correctly ( where attacks introducing heterophily can be unambiguously defined using the ground-truth class labels of the nodes involved ) . However , in §5.1 , we show on real-world datasets that a relaxed interpretation of the theorems , where heterophily is instead defined by the predicted class labels of GNNs , can explain the behavior of the attacks regardless of the initial correctness of the gambits .

Summary: The authors present a study on the robustness on GNNs on homophilic and heterophilic datasets. The authors first show in a stylized theoretical setting that GNNs with a *heterophilic design* (separate aggregation of ego- and neighborhood representations) should show increased robustness against structure perturbations. Then, they follow up with an experimental study where they compare homophily-based GNNs with GNNs following a heterophilic design on both homophilic and heterophilic datasets. In addition, the authors augment three GNN models with the SVD preprocessing by [Entezari et al. 2020] in order to improve robustness. The authors study both empirical as well as certifiable robustness (via randomized smoothing). In general, GNNs with heterophilic design show higher empirical and certifiable robustness across datasets.