Bridging Graph Network to Lifelong Learning with Feature Interaction

1 Introduction . Graph neural networks ( GNN ) have received increasing attention and proved to be useful for many tasks with graph-structured data , such as citation , social , and protein networks [ 1 ] . However , graph data is usually formed in a streaming fashion and real-world data are continuously evolving over time , thus learning a streaming graph is expected in many cases [ 2 ] . For example , in a social network , the number of users often grows over time and we expect that the model can learn continuously with new users . In this paper , we extend graph neural networks to lifelong learning , which is also known as continual or incremental learning [ 3 ] . However , lifelong learning easily suffers from “ catastrophic forgetting ” if the models are simply updated for new samples via back-propagation [ 4 ] . Although some strategies have been developed to alleviate the forgetting problem for convolutional neural networks ( CNN ) , lifelong learning is still difficult for graph networks . This is because existing graph models such as graph convolutional networks ( GCN ) [ 5 ] require the entire graph for training . This is infeasible for lifelong learning , since the graph size can increase over time and we have to drop off old data or samples to learn new knowledge . To reduce the memory consumption , some graph sampling strategies were proposed , but they are also difficult to directly apply to lifelong learning [ 6–8 ] , since they require a pre-processing of the entire graph [ 8 ] . Preprint . Under review . Recall that regular CNNs are trained in a mini-batch manner where the model can take samples as independent inputs [ 9 ] . Our question is : can we convert a graph task into a traditional CNN-like classification problem , so that ( i ) nodes can be predicted independently and ( ii ) the lifelong learning techniques developed for CNN can be easily adopted for GNN ? However , this is not straightforward as node connections can not be modeled by a regular CNN-like classification model . To solve this problem , we propose to construct a new graph topology , the feature graph in Fig . 1 , to bridge GNN to lifelong learning . It takes the features as nodes and turns the nodes into graphs . This converts the problem of node classification to graph classification and the increasing nodes become independent training samples , thus the feature graphs can be trained in a regular mini-batch manner . The contribution of this paper includes : ( 1 ) We introduce a new graph topology , i.e. , feature graph , to convert a problem of growing graph to increasing training samples , which makes lifelong learning techniques applicable to graph neural networks ; ( 2 ) We take the cross-correlation of neighbor features as the feature adjacency matrix , which is able to explicitly model feature “ interaction ” , that is crucial for many graph-structured tasks ; ( 3 ) Feature graph is of constant computational complexity with the increased learning tasks . We demonstrate its efficiency and effectiveness by applying it to classical graph datasets ; ( 4 ) We further demonstrate its superiority by applying it to a practical application , i.e. , distributed human action recognition based on subgraph classification using wearable devices . 2 Related Work . 2.1 Lifelong Learning . Non-rehearsal Methods Lifelong learning methods in this category do not preserve any old data . To alleviate the forgetting problem , progressive neural networks [ 10 ] leveraged prior knowledge via lateral connections to previously learned features . Learning without forgetting ( LwF ) [ 11 ] introduced a knowledge distillation loss [ 12 ] to neural networks , which encouraged the network output for new classes to be close to the original outputs . Distillation loss was also applied to learning object detectors incrementally [ 13 ] . Learning without memorizing ( LwM ) [ 14 ] extended LwF by adding an attention distillation term based on attention maps and demonstrated that it is helpful for retaining information of the old classes . EWC [ 15 ] remembered old tasks by slowing down learning on important weights . RWalk [ 16 ] generalized EWC and improved weight consolidation by adding a KL-divergence-based regularization . Memory aware synapses ( MAS ) [ 17 ] computed an importance value for each parameter in an unsupervised manner based on the sensitivity of output function to parameter changes . [ 18 ] presented an embedding framework for dynamic attributed network based on parameter regularization . A sparse writing protocol is introduced to a memory module [ 19 ] , ensuring that only a few memory spaces is affected during training . Rehearsal Methods Rehearsal lifelong learning methods can be roughly divided into rehearsal with synthetic data or rehearsal with exemplars from old data [ 20 ] . To ensure that the loss of exemplars does not increase , gradient episodic memory ( GEM ) [ 3 ] introduced orientation constraints during gradient updates . Inspired by GEM , [ 21 ] selected exemplars with a maximal cosine similarity of the gradient orientation . iCaRL [ 22 ] preserved a subset of images with a herding algorithm [ 23 ] and included the subset when updating the network for new classes . EEIL [ 24 ] extended iCaRL by learning the classifier in an end-to-end manner . [ 25 ] further extended iCaRL and discovered that updating the model with class-balanced exemplars further improved the performance . Similarly , [ 26 , 27 ] further added constraints to the loss function to mitigate the effect of imbalance . To reduce the memory consumption of exemplars , [ 28 ] applied the distillation loss to feature space without having to access to the corresponding images . Rehearsal approaches with synthetic data based on generative adversary networks ( GAN ) were used to reduce the dependence on old data [ 29–32 ] . 2.2 Graph Neural Networks . Graph neural networks have been widely used to solve problems about graph-structured data [ 33 ] . The spectral network was the first to extend convolution to the graph domain [ 34 ] . Later the graph convolutional network ( GCN ) [ 5 ] followed this idea and alleviated the problem of overfitting on local neighborhoods via first order Chebyshev expansion . This obtained a good generalization ability and inspired more recent work . To identify the importance of neighborhood features , graph attention network ( GAT ) [ 35 ] added an attention mechanism into GCN , which further improved the performance on citation networks and the protein-protein interaction dataset . GCN and its variants require the entire graph during training , thus they can not scale to large graphs . To solve this problem and train GNN with mini-batches , a sampling method , SAGE [ 6 ] is introduced to learn a function to generate node embedding by sampling and aggregating neighborhood features . JKNet [ 36 ] followed the same sampling strategy and demonstrated a significant accuracy improvement on GCN with jumping connections . DiffPool [ 37 ] learned a differentiable soft cluster assignment to map nodes to a set of clusters , which then formed a coarsened input for the next layer . [ 38 ] designed a training strategy that relied on harder-and-harder training examples to improve the robustness and convergence speed of the model . S-GCN [ 39 ] developed control variate-based algorithms to sample small neighbor sizes and used historical activation in the previous layer to avoid re-evaluation . FastGCN [ 7 ] applied importance sampling to reduce variance and performed node sampling for each layer independently , which resulted in a constant sample size in all layers . [ 40 ] sampled the lower layer conditioned on the top one ensuring higher accuracy and fixed-size sampling . Subgraph sampling techniques were also developed to reduce memory consumption . ClusterGCN [ 41 ] sampled a block of nodes in a dense subgraph identified by a clustering algorithm and restricted the neighborhood search within the subgraph . GraphSiant [ 8 ] constructed mini-batches by sampling the training graph . Nevertheless , most of the sampling techniques still require a pre-processing of the entire graph to determine the sampling process or require a complete graph structure , which makes those algorithms not directly applicable to lifelong learning . In this paper , we hypothesize that a different graph structure is required for lifelong learning , and the new structure is not necessary to maintain its original meaning . We noticed that there is some recent work focusing on continuously learning a graph problem , but they have different formulation . For example , several exemplar selection methods are tested in ER-GNN [ 42 ] . A weight preserving method is introduced to growing graph [ 43 ] . A combined strategy of regularization and data rehearsal is introduced to streaming graphs in [ 2 ] . To overcome the incomplete structure , learning a temporal graph in sliding window is introduced in [ 44 ] . 3 Problem Formulation . We start from defining regular graph learning before lifelong graph learning for coherence . An attribute graph is defined as G = ( V , E ) , where V is the set of nodes and E ⊆ { { va , vb } | ( va , vb ) ∈ V2 } is the set of edges . Each node v ∈ V is associated with a target zv ∈ Z and a multi-channel feature vector xv ∈ X ⊂ RF×C . Each edge e ∈ E is associated with a vector we ∈ W ⊂ RW . In regular graph learning , we aim to learn a predictor f to associate a node xv , v ∈ V ′ with a target zv , given graph G , node features X , edge vectorsW , and part of the targets zv , v ∈ V \ V ′ . In lifelong graph learning , we have the same objective with the regular graph learning , but we are only able to obtain the graph-structured data as a data continuum GL = { ( xi , ti , zi , Nk=1 : K ( xi ) , Wk=1 : K ( xi ) ) i=1 : N } , where each item is formed by a node feature xi ∈ X , a task descriptor ti ∈ T , a target vector zi ∈ Zti , a k-hop neighbor set Nk=1 : K ( xi ) , and an edge vector set Wk=1 : K ( xi ) associated with the k-hop neighbors . For simplicity , we will use the symbol N ( xi ) to denote the available neighbor set and their associated edges . We assume that every item ( xi , N ( xi ) , ti , zi ) satisfies ( xi , N ( xi ) , zi ) ∼ Pti ( X , N ( X ) , Z ) , where Pti is a probability distribution of a single learning task . In lifelong graph learning , we will observe , item by item , the continuum of the graph-structured data as ( x1 , N ( x1 ) , t1 , z1 ) , . . . , ( xN , N ( xN ) , tN , zN ) ( 1 ) While observing ( 1 ) , our goal is to learn a predictor fL to associate a test sample ( x , N ( x ) , t ) with a target z such that ( x , N ( x ) , z ) ∼ Pt . Such test sample can belong to a task that we have observed in the past , the current task , or a task that we will observe ( or not ) in the future . The task descriptors ti is defined for compatibility with lifelong learning that requires it [ 3 ] , but it is not used in this paper . Note that samples are not drawn locally identically and independently distributed ( iid ) from a fixed probability distribution , since we don ’ t know the task boundary . In the continuum , we only know the label of xi , but have no information about the labels of its neighbors N ( xi ) . Note that items in ( 1 ) are unavailable once they are observed and dropped off . This is in contrast to the settings in [ 2 ] , where all historical data are available during training . As shown in the experiments , lifelong graph learning in practice often requires that the number of GNN layers L is larger than the availability of K-hop neighbors , i.e. , L > K , which also leads many existing graph models inapplicable .

Summary: This paper aims to bridge GNNs with life-long learning so that the catastrophic forgetting problem in graph-structured tasks is alleviated. Specifically, the major contribution seems to be transforming the original graph into a feature graph so that the node classification problem is transferred into a graph classification problem with isolated samples. Meanwhile, feature interactions are modeled in constructing edges of the feature graph. Experiments on three citation graphs demonstrate the effectiveness of the proposed method.