MONET: Debiasing Graph Embeddings via the Metadata-Orthogonal Training Unit

1 INTRODUCTION . Graph embeddings – continuous , low-dimensional vector representations of nodes – have been eminently useful in network visualization , node classification , link prediction , and many other graph learning tasks ( 10 ) . While graph embeddings can be estimated directly by unsupervised algorithms using the graph ’ s structure ( e.g . 24 ; 28 ; 15 ; 25 ) , there is often additional ( non-relational ) information available for each node in the graph . This information , frequently referred to as node attributes or node metadata , can contain information that is useful for prediction tasks including demographic , geo-spatial , and/or textual features . The interplay between a node ’ s metadata and edges is a rich and active area of research . Interestingly , in a number of cases , this metadata can be measurably related to a graph ’ s structure ( 21 ) , and in some instances there may be a causal relationship ( the node ’ s attributes influence the formation of edges ) . As such , metadata can enhance graph learning models ( 31 ; 20 ) , and conversely , graphs can be used as regularizers in supervised and semi-supervised models of node features ( 32 ; 11 ) . Furthermore , metadata are commonly used as evaluation data for graph embeddings ( 8 ) . For example , node embeddings trained on a Flickr user graph were shown to predict user-specified Flickr “ interests '' ( 24 ) . This is presumably because users ( as nodes ) in the Flickr graph tend to follow users with similar interests , which illustrates a potential causal connection between node topology and node metadata . However , despite the usefulness and prevalence of metadata in graph learning , there are instances where it desirable to design a system to avoid the effects of a particular kind of sensitive data . For instance , the designers of a recommendation system may want to make recommendations independent of a user ’ s demographic information or location . At first glance , this may seem like an artificial dilemma – surely one could just avoid the problem by not adding such sensitive attributes to the model . However , such an approach ( ignoring a sensitive attribute ) does not control for any existing correlations that may exist between the sensitive metadata and the edges of a node . In other words , if the edges of the graph are correlated with sensitive metadata , then any algorithm which does not explicitly model and remove this correlation will be biased as a result of it . Surprisingly , almost all of the existing work in the area ( 31 ; 35 ) has ignored this important realization.1 In this work , we seek to refocus the discussion about graph learning with node metadata . To this end , we propose a novel , general technique for extending graph representations with metadata embedding dimensions while debiasing the remaining ( topology ) dimensions . Specifically , our contributions are the following : 1 . The Metadata-Orthogonal Node Embedding Training ( MONET ) unit , a novel GNN algorithm which jointly embeds graph topology and graph metadata while enforcing linear decorrelation between the two embedding spaces . 2 . Analysis which proves that a naive approach ( adding metadata embeddings without MONET ) leaks metadata information into topology embeddings , and that the MONET unit does not . 3 . Experimental results on real world graphs which show that MONET can successfully “ debias '' topology embeddings while relegating metadata information to separate metadata embeddings . 2 PRELIMINARIES . Early graph embedding methods involved dimensionality reduction techniques like multidimensional scaling and singular value decomposition ( 8 ) . In this paper we use graph neural networks trained on random walks , similarly to DeepWalk ( 24 ) . DeepWalk and many subsequent methods first generate a sequence of random walks from the graph , to create a “ corpus '' of node “ sentences '' which are then modeled via word embedding techniques ( e.g . word2vec ( 19 ) or GloVe ( 23 ) ) to learn low dimensional representations that preserve the observed co-occurrence similarity . Let W be a d-dimensional graph embedding matrix , W ∈ Rn×d , which aims to preserve the lowdimensional structure of a graph ( d < < n ) . Rows of W correspond to nodes , and node pairs i , j with large dot-products WTi Wj should be structurally or topologically close in the graph . As a concrete example , in this paper we consider the debiasing of a recently proposed graph embedding using the GloVe model ( 6 ) . Its training objective is : GloVe ( U , V , a , b|C ) = ∑ i , j≤n fα ( Cij ) ( ai + bj + U T i Vj − log ( Cij ) ) 2 , ( 1 ) where U , V ∈ Rn×d are the “ center '' and “ context '' embeddings , a , b ∈ Rn×1 are the biases , C is the walk-distance-weighted context co-occurrences , and fα is the loss smoothing function ( 23 ) . We use the GloVe model in the next section to illustrate topology/metadata embeddings and metadataorthogonal training . However , the MONET unit we propose is broadly generalizable . To illustrate this , we also describe a MONET unit for DeepWalk ( 24 ) , a popular graph embedding algorithm . Notation . In this paper , given a matrix A ∈ Rn×d and an index i ∈ 1 , . . . , n , Ai denotes the d× 1 i-th row vector of A . Column indices will not be used . 0n×d denotes the n × d zero matrix , and || · ||F denotes the Frobenius norm . 3 METADATA EMBEDDINGS AND ORTHOGONAL TRAINING . In this section we present MONET , our proposed method for separating and controlling the effects of metadata on topology embeddings . First , we begin by outlining the straightforward extension of metadata to traditional embedding models in Section 3.1 . Next , in Section 3.2 , we prove that such a simple model will leak information from the metadata to the topology ( structural ) embeddings . Then , in Section 3.3 we present MONET , our proposed approach for training embeddings of a graph ’ s 1While preparing this manuscript , we have become aware of a recent independent result ( 5 ) in this area for recommender graphs . In contrast to that work , we use a substantially different methodology which offers guarantees about the debiasing process . structure which are not correlated with metadata . Finally , we conclude with some analysis of MONET in Section 3.4 3.1 JOINTLY MODELING METADATA & TOPOLOGY . A natural first approach to modeling the effects of metadata on the graph is to explicitly include the node metadata as part of a node embedding model . For instance , to extend Eq . ( 1 ) , in addition to U and V ( the “ topology embeddings '' ) , we can consider the node metadata M directly ( M ∈ Rn×m , row vector Mi is the metadata for node ui ) . We then can define metadata embeddings X = MT1 , Y = MT2 , where T1 , T2 are trainable transformations , and propose the concatenations [ U , X ] and [ V , Y ] as full-graph representations . The GloVe loss with metadata embeddings is : GloVemeta ( U , V , T1 , T2 , a , b|C , M ) = 1 2 ∑ i , j≤n fα ( Cij ) ( ai+bj+U T i Vj+X T i Yj− log ( Cij ) ) 2 . ( 2 ) While in this paper we demonstrate metadata embeddings within the GloVe model , they can be incorporated in any dot-product-based graph neural network . For instance , the well-known DeepWalk ( 24 ) loss , which is based on word2vec ( 19 ) , would incorporate metadata embeddings as follows : DeepWalkmeta ( U , V , T1 , T2|W , M ) = − ∑ i , j∈W log ( UTi Vj +X T i Yj ) − ∑ k∈Ki log ( −UTi Vk−XTi Yk ) . ( 3 ) Above , W is the set of context pairs from random walks , andKi is a set of negative samples associated with node i . For GloVe , DeepWalk , and many other GNNs , this approach augments the overall graph representation by concatenating metadata-learned dimensions . However , this naïve approach does not guarantee that the topology embeddings converge to be decorrelated from the metadata embeddings . Suppose that the metadata ( like demographic information ) are indeed associated with the formation of links in the graph . In this case , any algorithm which does not explicitly model and remove the association will be biased as a result of it . In the next section we formalize this concept , which we call metadata leakage . 3.2 METADATA LEAKAGE IN GRAPH NEURAL NETWORKS . Here , we formally define metadata leakage for general topology and metadata embeddings , and show how it can occur even in embedding models with separate metadata embeddings . All proofs appear in the Appendix . Definition 1 . The metadata leakage of metadata embeddings Z ∈ Rn×dZ into topology embeddings W ∈ Rn×d is definedML ( Z , W ) : = ||ZTW ||2F . We say that there is no metadata leakage if and only ifML ( Z , W ) = 0 . Without a more nuanced approach , metadata leakage can occur even in embedding models that explicitly include the metadata , like Eqs . ( 2 ) and ( 3 ) . To demonstrate this , we consider for simplicity a reduced metadata-aware GloVe loss with W : = U = V ∈ Rn×d as the sole topology embedding and T : = T1 = T2 ∈ Rm×dZ as the sole metadata transformation parameter . With Z : = MT , the reduced loss is : GloVe∗meta ( W , T , a|C , M ) = 1 2 ∑ i , j≤n ( ai + aj +W T i Wj + Z T i Zj − log ( Cij ) ) 2 ( 4 ) We now show that under a random update of the GloVe∗meta model in Eq . ( 4 ) , the expected metadata leakage is non-zero . Specifically , let ( i , j ) be a node pair from C , and define δW ( i , j ) as the incurred Stochastic Gradient Descent update W ′ ← W + δW ( i , j ) . Suppose there is a “ ground-truth '' metadata transformation B ∈ Rm×dB , and define ground-truth metadata embeddings Z̃ : = MB , which represent the “ true '' dimensions of the metadata effect on the co-occurrences C. Define ΣB : = BB T and ΣT : = TTT . With expectations taken with respect to the sampling of a pair ( i , j ) for Stochastic Gradient Descent , define µW : = E [ Wi ] and ΣW : = E [ WiWTi ] . Define µM , ΣM similarly . Then our main Theorem is as follows : Algorithm 1 MONET Unit Training Step Given : topology embedding W , metadata embedding Z 1 : procedure FORWARD PASS DEBIASING ( W , Z ) 2 : Compute Z left-singular vectors QZ and projection PZ ← In×n −QZQTZ 3 : Compute orthogonal topology embedding W⊥ ← PZW 4 : return debiased graph representation [ W⊥ , Z ] 5 : procedure BACKWARD PASS DEBIASING ( δW ) 6 : Compute orthogonal topology embedding update δ⊥W ← PZδW 7 : Apply update W⊥ ←W⊥ + δ⊥W 8 : return debiased topology embedding W⊥ Theorem 1 . Assume ΣW = σW Id for σW > 0 , µW = 0d×1 , and µM = 0m×1 . Suppose for some fixed θ ∈ R we have log ( Cij ) = θ + Z̃Ti Z̃j . Let ( i , j ) be a randomly sampled co-occurrence pair and W ′ the incurred update . Then if E [ MiWTi ] = β ∈ Rm×d , we have E [ ML ( Z , W ′ ) ] ≥ 2||TT [ ΣM ( ΣB − ΣT ) + ( n− σW ) Im ] β||2F . ( 5 ) Importantly , ΣT and σW are neural network hyperparameters , so we give a useful Corollary : Corollary 1 . Under the assumptions of Theorem 1 , E [ ML ( Z , W ) ] = Ω ( n||TTβ||2F ) as n→∞ . Note that under reasonable GNN initialization schemes , T and β are random perturbations . Thus , Corollary 1 implies the surprising result that incorporating feed-forward metadata embeddings is not sufficient to prevent metadata leakage in practical settings .

Summary: The paper introduces a GNN model (MONET) for debiasing graph embeddings, by enforcing orthogonality between the embedding spaces of the graph topology & the graph metadata. They show that unsupervised learning induces bias from important graph metadata, when the metadata is correlated with the node edges. They show experimental results on real world graphs (political blogs network & graph-based recommendation systems), where MONET can debias  graph embeddings and prevent metadata leakage.