GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation

1 INTRODUCTION . Learning from graph-structured data has seen explosive growth over the last few years , as graphs are a convenient formalism to model the broad class of data that has objects ( treated as vertices ) with some known relationships ( treated as edges ) . Example usages include reasoning about physical and biological systems , knowledge bases , computer programs , and relational reasoning in computer vision tasks . This graph construction is a highly complex form of feature engineering , mapping the knowledge of a domain expert into a graph structure which can be consumed and exploited by high-capacity neural network models . Many neural graph learning methods can be summarised as neural message passing ( Gilmer et al. , 2017 ) : nodes are initialised with some representation and then exchange information by transforming their current state ( in practice with a single linear layer ) and sending it as a message to all neighbours in the graph . At each node , messages are aggregated in some way and then used to update the associated node representation . In this setting , the message is entirely determined by the source node ( and potentially the edge type ) and the target node is not taken into consideration . A ( partial ) exception to this is the family of Graph Attention Networks ( Veličković et al. , 2018 ) , where the agreement between source and target representation of an edge is used to determine the weight of the message in an attention architecture . However , this weight is applied to all dimensions of the message at the same time . A simple consequence of this observation may be to simply compute messages from the pair of source and target node state . However , the linear layer commonly used to compute messages would only allow additive interactions between the representations of source and target nodes . More complex transformation functions are often impractical , as computation in GNN implementations is dominated by the message transformation function . However , this need for non-trivial interaction between different information sources is a common problem in neural network design . A recent trend has been the use of hypernetworks ( Ha et al. , 2017 ) , neural networks that compute the weights of other networks . In this setting , interaction between two signal sources is achieved by using one of them as the input to a hypernetwork and the other as input to the computed network . While an intellectually pleasing approach , it is often impractical because the prediction of weights of non-trivial neural networks is computationally expensive . Approaches to mitigate this exist ( e.g. , Wu et al . ( 2019 ) handle this in natural language processing ) , but are often domain-specific . A more general mitigation method is to restrict the structure of the computed network . Recently , “ feature-wise linear modulations ” ( FiLM ) were introduced in the visual question answering domain ( Perez et al. , 2017 ) . Here , the hypernetwork is fed with an encoding of a question and produces an element-wise affine function that is applied to the features extracted from a picture . This can be adapted to the graph message passing domain by using the representation of the target node to compute the affine function . This compromise between expressiveness and computational feasibility has been very effective in some domains and the results presented in this article indicate that it is also a good fit for the graph domain . This article explores the use of hypernetworks in learning on graphs . Sect . 2 first reviews existing GNN models from the related work to identify commonalities and differences . This involves generalising a number of existing formalisms to new formulations that are able to handle graphs with different types of edges , which are often used to model different relationship between vertices . Then , two new formalisms are introduced : Relational Graph Dynamic Convolutional Networks ( RGDCN ) , which dynamically compute the neural message passing function as a linear layer , and Graph Neural Networks with Feature-wise Linear Modulation ( GNN-FiLM ) , which combine learned message passing functions with dynamically computed element-wise affine transformations . In Sect . 3 , a range of baselines are compared in extensive experiments on three tasks from the literature , spanning classification , regression and ranking tasks on small and large graphs . Experiments were performed on re-implementations of existing model architectures in the same framework and hyperparameter setting searches were performed with the same computational budgets across all architectures . The results show that differences between baselines are smaller than the literature suggests and that the new FiLM model performs well on a number of interesting tasks . 2 MODEL . Notation . Let L be a finite ( usually small ) set of edge types . Then , a directed graph G = ( V , E ) has nodes V and typed edges E ⊆ V × L × V , where ( u , ` , v ) ∈ E denotes an edge from node u to node v of type ` , usually written as u →̀ v. Graph Neural Networks . As discussed above , Graph Neural Networks operate by propagating information along the edges of a given graph . Concretely , each node v is associated with an initial representation h ( 0 ) v ( for example obtained from the label of that node , or by some other model component ) . Then , a GNN layer updates the node representations using the node representations of its neighbours in the graph , yielding representations h ( 1 ) v . This process can be unrolled through time by repeatedly applying the same update function , yielding representations h ( 2 ) v . . .h ( T ) v . Alternatively , several GNN layers can be stacked , which is intuitively similar to unrolling through time , but increases the GNN capacity by using different parameters for each timestep . In Gated Graph Neural Networks ( GGNN ) ( Li et al. , 2016 ) , the update rule uses one linear layer W ` per edge type ` to compute messages and combines the aggregated messages with the current representation of a node using a recurrent unit r ( e.g. , GRU or LSTM cells ) , yielding the following definition . h ( t+1 ) v = r ( h ( t ) v , ∑ u→̀v∈E W ` h ( t ) u ; θr ) ( 1 ) The learnable parameters of the model are the edge-type-dependent weights W ` and the recurrent cell parameters θr . In Relational Graph Convolutional Networks ( R-GCN ) ( Schlichtkrull et al. , 2018 ) , the gated unit is replaced by a simple non-linearity σ ( e.g. , the hyperbolic tangent ) . h ( t+1 ) v = σ ( ∑ u→̀v∈E 1 cv , ` ·W ` h ( t ) u ) ( 2 ) Here , cv , ` is a normalisation factor usually set to the number of edges of type ` ending in v. The learnable parameters of the model are the edge-type-dependent weights W ` . It is important to note that in this setting , the edge type set L is assumed to contain a special edge type 0 for self-loops v 0→ v , allowing state associated with a node to be kept . In Graph Attention Networks ( GAT ) ( Veličković et al. , 2018 ) , new node representations are computed from a weighted sum of neighbouring node representations . The model can be generalised from the original definitional to support different edge types as follows ( we will call this R-GAT below ) .1 eu , ` , v = LeakyReLU ( α ` · ( W ` h ( t ) u ‖W ` h ( t ) v ) ) av = softmax ( eu , ` , v | u →̀ v ∈ E ) h ( t+1 ) v = σ ( ∑ u→̀v∈E ( av ) u→̀v ·W ` h ( t ) u ) ( 3 ) Here , α ` is a learnable row vector used to weigh different feature dimensions in the computation of an attention ( “ relevance ” ) score of the node representations , x‖y is the concatenation of vectors x and y , and ( av ) u→̀v refers to the weight computed by the softmax for that edge . The learnable parameters of the model are the edge-type-dependent weights W ` and the attention parameters α ` . In practice , GATs usually employ several attention heads that independently implement the mechanism above in parallel , using separate learnable parameters . The results of the different attention heads are then concatenated after each propagation round to yield the value of h ( t+1 ) v . More recently , Xu et al . ( 2019 ) analysed the expressiveness of different GNN types , comparing their ability to distinguish similar graphs with the Weisfeiler-Lehman ( WL ) graph isomorphism test . Their results show that GCNs and the GraphSAGE model Hamilton et al . ( 2017 ) are strictly weaker than the WL test and hence they developed Graph Isomorphism Networks ( GIN ) ( Xu et al. , 2019 ) , which are indeed as powerful as the WL test . While the GIN definition is limited to a single edge type , Corollary 6 of Xu et al . ( 2019 ) shows that using the definition h ( t+1 ) v = ϕ ( ( 1 + ) · f ( h ( t ) v ) + ∑ u→v∈E f ( h ( t ) u ) ) , there are choices for , ϕ and f such that the node representation update is sufficient for the overall network to be as powerful as the WL test . In the setting of different edge types , the function f in the sum over neighbouring nodes needs to reflect different edge types to distinguish graphs such as v 1→ u 2← w and v 2→ u 1← w from each other . Using different functions f ` for different edge types makes it possible to unify the use of the current node representation h ( t ) v with the use of neighbouring node representations by again using a fresh edge type 0 for self-loops v 0→ v. In that setting , the factor ( 1 + ) can be integrated into f0 . Finally , following an argument similar to Xu et al . ( 2019 ) , ϕ and f at subsequent layers can be “ merged ” into a single function which can be approximated by a multilayer perceptron ( MLP ) , yielding the final R-GIN definition h ( t+1 ) v = σ ( ∑ u→̀v∈E MLP ( h ( t ) u ; θ ` ) ) . ( 4 ) The learnable parameters here are the edge-specific weights θ ` . Note that Eq . ( 4 ) is very similar to the definition of R-GCNs ( Eq . ( 2 ) ) , only dropping the normalisation factor 1cv , ` and replacing linear layers by an MLP . While many more GNN variants exist , the four formalisms above are broadly representative of general trends . It is notable that in all of these models , the information passed from one node to another is based on the learned weights and the representation of the source of an edge . In contrast , the representation of the target of an edge is only updated ( in the GGNN case Eq . ( 1 ) ) , treated as another incoming message ( in the R-GCN case Eq . ( 2 ) and the R-GIN case Eq . ( 4 ) ) , or used to weight the relevance of an edge ( in the R-GAT case Eq . ( 3 ) ) . Sometimes unnamed GNN variants of the above are used ( e.g. , by Selsam et al . ( 2019 ) ; Paliwal et al . ( 2019 ) ) , replacing the linear layers to compute the messages for each edge by MLPs applied to the concatenation of the representations 1Note that this is similar to the ARGAT model presented by Busbridge et al . ( 2019 ) , but unlike the models studied there ( and like the original GATs ) uses a single linear layer to compute attention scores eu , ` , v , instead of simpler additive or multiplicative variants . of source and target nodes . In the experiments , this will be called GNN-MLP , formally defined as follows.2 h ( t+1 ) v = σ ( ∑ u→̀v∈E 1 cv , ` ·MLP ( h ( t ) u ‖h ( t ) v ; θ ` ) ) ( 5 ) Below , we will instantiate the MLP with a single linear layer to obtain what we call GNN-MLP0 , which only differs from R-GCNs ( Eq . ( 2 ) ) in that the message passing function is applied to the concatenation of source and target state .

Summary: The paper proposes a new Graph Neural Network (GNN) architecture that uses Feature-wise Linear Modulation (FiLM) to condition the source-to-target node message-passing based on the target node representation. In this way, GNN-FiLM aims to allow a GNN's message propagation to "focus on feature that are especially relevant for the update of the target node." The authors clearly describe prior GNN architectures, showing that the do not incorporate such forms of message propagation. The authors then describe several intuitive ways of adding such a form of message propagation, before describing why those approaches do not work in practice. Finally, the authors introduce GNN-FiLM, which is computationally reasonable and works well in practice, as evaluated according to several GNN benchmarks. The GNN-FiLM model is also quite simple and elegant, which makes me think it is likely to work on more tasks than the authors experiment on.