Lifelong Graph Learning
Year: 2021

Abstract:
Graph neural networks (GNNs) are powerful models for many graph-structured tasks. Existing models often assume that a complete structure of a graph is available during training, however, in practice, graph-structured data is usually formed in a streaming fashion, so that learning a graph continuously is often necessary. In this paper, we aim to bridge GNN to lifelong learning by converting a graph problem to a regular learning problem, so that GNN is able to inherit the lifelong learning techniques developed for convolutional neural networks (CNNs). To this end, we propose a new graph topology based on feature cross-correlation, called the feature graph. It takes features as new nodes and turns nodes into independent graphs. This successfully converts the original problem of node classification to graph classification, in which the increasing nodes are turned into independent training samples. In the experiments, we demonstrate the efficiency and effectiveness of feature graph networks (FGN) by continuously learning a sequence of classical graph datasets. We also show that FGN achieves superior performance in human action recognition with distributed streaming signals for wearable devices.

heading: 1 Introduction
text: Graph neural networks (GNN) have received increasing attention and proved to be useful for many tasks with graph-structured data, such as citation, social, and protein networks [1]. However, graph data is usually formed in a streaming fashion and real-world data are continuously evolving over time, thus learning a streaming graph is expected in many cases [2]. For example, in a social network, the number of users often grows over time and we expect that the model can learn continuously with new users. In this paper, we extend graph neural networks to lifelong learning, which is also known as continual or incremental learning [3]. However, lifelong learning easily suffers from “catastrophic forgetting” if the models are simply updated for new samples via back-propagation [4].
Although some strategies have been developed to alleviate the forgetting problem for convolutional neural networks (CNN), lifelong learning is still difficult for graph networks. This is because existing graph models such as graph convolutional networks (GCN) [5] require the entire graph for training. This is infeasible for lifelong learning, since the graph size can increase over time and we have to drop off old data or samples to learn new knowledge. To reduce the memory consumption, some graph sampling strategies were proposed, but they are also difficult to directly apply to lifelong learning [6–8], since they require a pre-processing of the entire graph [8].
Preprint. Under review.
Recall that regular CNNs are trained in a mini-batch manner where the model can take samples as independent inputs [9]. Our question is: can we convert a graph task into a traditional CNN-like classification problem, so that (i) nodes can be predicted independently and (ii) the lifelong learning techniques developed for CNN can be easily adopted for GNN? However, this is not straightforward as node connections cannot be modeled by a regular CNN-like classification model. To solve this problem, we propose to construct a new graph topology, the feature graph in Fig. 1, to bridge GNN to lifelong learning. It takes the features as nodes and turns the nodes into graphs. This converts the problem of node classification to graph classification and the increasing nodes become independent training samples, thus the feature graphs can be trained in a regular mini-batch manner.
The contribution of this paper includes: (1) We introduce a new graph topology, i.e., feature graph, to convert a problem of growing graph to increasing training samples, which makes lifelong learning techniques applicable to graph neural networks; (2) We take the cross-correlation of neighbor features as the feature adjacency matrix, which is able to explicitly model feature “interaction”, that is crucial for many graph-structured tasks; (3) Feature graph is of constant computational complexity with the increased learning tasks. We demonstrate its efficiency and effectiveness by applying it to classical graph datasets; (4) We further demonstrate its superiority by applying it to a practical application, i.e., distributed human action recognition based on subgraph classification using wearable devices.

heading: 2 Related Work

heading: 2.1 Lifelong Learning
text: Non-rehearsal Methods Lifelong learning methods in this category do not preserve any old data. To alleviate the forgetting problem, progressive neural networks [10] leveraged prior knowledge via lateral connections to previously learned features. Learning without forgetting (LwF) [11] introduced a knowledge distillation loss [12] to neural networks, which encouraged the network output for new classes to be close to the original outputs. Distillation loss was also applied to learning object detectors incrementally [13]. Learning without memorizing (LwM) [14] extended LwF by adding an attention distillation term based on attention maps and demonstrated that it is helpful for retaining information of the old classes. EWC [15] remembered old tasks by slowing down learning on important weights. RWalk [16] generalized EWC and improved weight consolidation by adding a KL-divergence-based regularization. Memory aware synapses (MAS) [17] computed an importance value for each parameter in an unsupervised manner based on the sensitivity of output function to parameter changes. [18] presented an embedding framework for dynamic attributed network based on parameter regularization. A sparse writing protocol is introduced to a memory module [19], ensuring that only a few memory spaces is affected during training.
Rehearsal Methods Rehearsal lifelong learning methods can be roughly divided into rehearsal with synthetic data or rehearsal with exemplars from old data [20]. To ensure that the loss of exemplars does not increase, gradient episodic memory (GEM) [3] introduced orientation constraints during gradient updates. Inspired by GEM, [21] selected exemplars with a maximal cosine similarity of the gradient orientation. iCaRL [22] preserved a subset of images with a herding algorithm [23] and included the subset when updating the network for new classes. EEIL [24] extended iCaRL by learning the classifier in an end-to-end manner. [25] further extended iCaRL and discovered that updating the model with class-balanced exemplars further improved the performance. Similarly, [26, 27] further added constraints to the loss function to mitigate the effect of imbalance. To reduce the memory consumption of exemplars, [28] applied the distillation loss to feature space without having to access to the corresponding images. Rehearsal approaches with synthetic data based on generative adversary networks (GAN) were used to reduce the dependence on old data [29–32].

heading: 2.2 Graph Neural Networks
text: Graph neural networks have been widely used to solve problems about graph-structured data [33]. The spectral network was the first to extend convolution to the graph domain [34]. Later the graph convolutional network (GCN) [5] followed this idea and alleviated the problem of overfitting on local neighborhoods via first order Chebyshev expansion. This obtained a good generalization ability and inspired more recent work. To identify the importance of neighborhood features, graph attention network (GAT) [35] added an attention mechanism into GCN, which further improved the performance on citation networks and the protein-protein interaction dataset.
GCN and its variants require the entire graph during training, thus they cannot scale to large graphs. To solve this problem and train GNN with mini-batches, a sampling method, SAGE [6] is introduced to learn a function to generate node embedding by sampling and aggregating neighborhood features. JKNet [36] followed the same sampling strategy and demonstrated a significant accuracy improvement on GCN with jumping connections. DiffPool [37] learned a differentiable soft cluster assignment to map nodes to a set of clusters, which then formed a coarsened input for the next layer. [38] designed a training strategy that relied on harder-and-harder training examples to improve the robustness and convergence speed of the model. S-GCN [39] developed control variate-based algorithms to sample small neighbor sizes and used historical activation in the previous layer to avoid re-evaluation.
FastGCN [7] applied importance sampling to reduce variance and performed node sampling for each layer independently, which resulted in a constant sample size in all layers. [40] sampled the lower layer conditioned on the top one ensuring higher accuracy and fixed-size sampling. Subgraph sampling techniques were also developed to reduce memory consumption. ClusterGCN [41] sampled a block of nodes in a dense subgraph identified by a clustering algorithm and restricted the neighborhood search within the subgraph. GraphSiant [8] constructed mini-batches by sampling the training graph.
Nevertheless, most of the sampling techniques still require a pre-processing of the entire graph to determine the sampling process or require a complete graph structure, which makes those algorithms not directly applicable to lifelong learning. In this paper, we hypothesize that a different graph structure is required for lifelong learning, and the new structure is not necessary to maintain its original meaning. We noticed that there is some recent work focusing on continuously learning a graph problem, but they have different formulation. For example, several exemplar selection methods are tested in ER-GNN [42]. A weight preserving method is introduced to growing graph [43]. A combined strategy of regularization and data rehearsal is introduced to streaming graphs in [2]. To overcome the incomplete structure, learning a temporal graph in sliding window is introduced in [44].

heading: 3 Problem Formulation
text: We start from defining regular graph learning before lifelong graph learning for coherence. An attribute graph is defined as G = (V, E), where V is the set of nodes and E ⊆ { {va, vb} |(va, vb) ∈ V2
} is the set of edges. Each node v ∈ V is associated with a target zv ∈ Z and a multi-channel feature vector xv ∈ X ⊂ RF×C . Each edge e ∈ E is associated with a vector we ∈ W ⊂ RW . In regular graph learning, we aim to learn a predictor f to associate a node xv, v ∈ V ′ with a target zv, given graph G, node features X , edge vectorsW , and part of the targets zv, v ∈ V \ V ′. In lifelong graph learning, we have the same objective with the regular graph learning, but we are only able to obtain the graph-structured data as a data continuum GL = {(xi, ti, zi, Nk=1:K(xi), Wk=1:K(xi))i=1:N}, where each item is formed by a node feature xi ∈ X , a task descriptor ti ∈ T ,
a target vector zi ∈ Zti , a k-hop neighbor set Nk=1:K(xi), and an edge vector set Wk=1:K(xi) associated with the k-hop neighbors. For simplicity, we will use the symbol N (xi) to denote the available neighbor set and their associated edges. We assume that every item (xi,N (xi), ti, zi) satisfies (xi,N (xi), zi) ∼ Pti(X ,N (X ),Z), where Pti is a probability distribution of a single learning task. In lifelong graph learning, we will observe, item by item, the continuum of the graph-structured data as
(x1,N (x1), t1, z1), . . . , (xN ,N (xN ), tN , zN ) (1)
While observing (1), our goal is to learn a predictor fL to associate a test sample (x,N (x), t) with a target z such that (x,N (x), z) ∼ Pt. Such test sample can belong to a task that we have observed in the past, the current task, or a task that we will observe (or not) in the future. The task descriptors ti is defined for compatibility with lifelong learning that requires it [3], but it is not used in this paper.
Note that samples are not drawn locally identically and independently distributed (iid) from a fixed probability distribution, since we don’t know the task boundary. In the continuum, we only know the label of xi, but have no information about the labels of its neighbors N (xi). Note that items in (1) are unavailable once they are observed and dropped off. This is in contrast to the settings in [2], where all historical data are available during training. As shown in the experiments, lifelong graph learning in practice often requires that the number of GNN layers L is larger than the availability of K-hop neighbors, i.e., L > K, which also leads many existing graph models inapplicable.

heading: 4 Feature Graph Network
text: To better show the relationship with a regular graph, we first review GCN [5]. Given an graph G = (V, E) described in Section 3, the stacked node features can be written as X = [x1,x2, · · · ,xN ]T ∈ RN×FC , where xi ∈ RFC is a vectorized node feature. The GCN takes feature channel as C = 1, so that xi ∈ RF ,X ∈ RN×F . The l-th graph convolutional layer is defined as
X(l+1) = σ ( Â ·X(l) ·W ) , (2)
where σ( · ) is a non-linear activation function, W ∈ RF(l)×F(l+1) is a learnable parameter, and Â ∈ RN×N is a self-connected symmetrically normalized adjacency matrix of A [5]. For unweighted graph, A[i,j] = 1 if (i, j) ∈ E and A[i,j] = 0 otherwise. A graph convolutional layer doesn’t change the number of nodes (rows of X), but change the node feature dimension from F(l) to F(l+1).
GCN has been applied to many graph-structured tasks due to its simplicity and good generalization ability. However, the problems of GCN are also obvious. The node features in the next layer is a linear combination of the node features in the current layer, thus its graph propagation cannot directly model the feature “interaction”. Moreover, to propagate node features across layers, a full adjacency matrix A is required, which means that the entire graph topology has to be presented to the model. This limits its usage to large graphs and makes it inapplicable to lifelong learning either. As aforementioned, to scale GCN to large graphs, many sampling techniques are proposed but requires a pre-processing of the entire datasets [8], thus most of them are still difficult to directly apply to lifelong learning. We argue that such difficulty is caused by the inappropriate graph topology and introduce the feature graphs starting from the definition of feature nodes.

heading: 4.1 Feature Nodes
text: Recall that each node v in a regular graph G = (V, E) is associated with a multi-channel feature vector x = [ xT[1,:], · · · ,x T [F,:] ] ∈ RF×C , where x[i,:] is the i-th feature (row) of x. An attribute feature graph takes the features of a regular graph as nodes. It can be defined as GF = (VF , EF ), where each node vF ∈ VF is associated with a feature xT[i,:] and will be denoted as x F i . Intuitively,
the number of nodes in the feature graph is the feature dimension in the regular graph, i.e., ∣∣VF ∣∣ = F , and the feature dimension in feature graph is the feature channel in a regular graph, i.e., xFi ∈ RC . Therefore, we define the feature nodes for feature graph as
VF = { xF1 ,x F 2 , · · · ,xFi , · · · ,xFF } . (3)
In this way, for each node v ∈ V , we have an associated feature graph GF . We next establish their relationship by defining the feature adjacency matrices via feature cross-correlation.

heading: 4.2 Feature Adjacency Matrix
text: For each item in continuum (1), the edges between x and its neighbors N (x) imply the relationship of their features. Therefore, feature “interaction” can be modeled as the cross-correlation matrix of the feature vectors, so that the k-hop c-th channel feature adjacency matrix can be defined as:
AFk,c(x) , sgnroot ( E [ wx,yx[:,c]y T [:,c] ]) , subject to ∀y ∈ Nk(x), c = 1, 2, · · · , C, (4)
where wx,y is the associated edge weight and sgnroot(x) = sign(x) √ ( |x|) is introduced to retain the magnitude of node features and the sign of their inner products. Note that x[:,c] in (4) is the c-th column (channel) of x, while the feature node x[i,:] is the i-th row of x. Intuitively, we have AFk,c(x) ∈ RF×F , where F N due to the settings of lifelong learning. We can calculate multiple feature adjacency matrices if the edge weight is a high dimensional vector. In practice, the expectation in (4) is approximated by averaging over the observed neighbors in the continuum.
E [ wx,yx[:,c]y T [:,c] ] ≈ ∑ y∈Nk(x) wx,yx[:,c]y T [:,c]
|Nk(x)| , (5)
where we need to change x[:,c]yT[:,c] to (x[:,c]y T [:,c] + y[:,c]x T [:,c]) for undirected graphs. Therefore, the feature adjacency matrix can be constructed online via (5), together with the availability of new samples in the continuum. In this way, the continuum (1) is converted to
(GF1 , t1, z1), . . . , (GFi , ti, zi), . . . , (GFN , tN , zN ) (6)
where GF = ( VF ,AF (x) ) . This means that our objective of learning a node predictor becomes learning a graph predictor fF : GF × T 7→ Z to associate a test sample (GF , t) to target vector z such that (GF , z) ∼ PFt . Note that the feature graphs in the new continuum (6) are still non-iid, and each feature graph is associated with a unique feature adjacency matrix.
In this way, a growing adjacency matrix is converted to multiple small adjacency matrices of feature graphs. Hence, a lifelong graph learning problem becomes a regular lifelong learning problem similar to [3] and the problem of increasing nodes in graph learning can be solved by applying lifelong learning to the continuum. To be more clear, we list their detailed relationship in Table 1, where the arrow 7→ refers to the conversion from a regular graph to feature graphs.

heading: 4.3 Feature Graph Layers
text: Since feature graph is a new topology given by feature adjacency matrix, we are able to define many different types of layers. In this section we present three types of layers.

heading: 4.3.1 Feature Broadcast Layer
text: Inspired by GCN, the l-th broadcast layer is defined as xF(l+1) = σ ( ÂFk ·x F (l) ·W F ) , (7)
where σ( · ) is a non-linear activation function, ÂFk is the associated normalized feature adjacency matrix, and W ∈ RC(l)×C(l+1) is a learnable parameter. For simplicity, the channel c is left out in (7) and the layer channel is broadcasted independently. It is worth noting that, although the definition of (7) appears similar to (2), they have different dimension and represent different meanings.

heading: 4.3.2 Feature Transform Layer
text: Similar to (2), the feature broadcast layer doesn’t change the number of feature nodes. However, this is not always necessary for feature graphs, as our objective has been turned into graph classification. In this section, we define a feature transform layer which can change the number of feature nodes and will be helpful for further reducing the number of learnable parameters. Different from the feature broadcast layer, we need to re-calculate the feature adjacency matrices from transformed neighbors. Therefore, given the feature graph GF , the l-th feature transform layer can be defined as
AF(l)(x) , A F k ( x(l),y(l) ) , ∀y ∈ Nk(x), (8a)
xF(l+1) = σ ( WF · ÂF(l)(x) ·x F (l) ) , yF(l+1) = σ ( WF · ÂF(l)(x) ·y F (l) ) , (8b)
where WF ∈ RF(l+1)×F(l) is a learnable parameter, σ( · ) is a non-linear activation function, and ÂF(l)(x) ∈ R
F(l)×F(l) is the self-connected normalized feature adjacency matrix of AF(l)(x), which is calculated from transformed feature and neighbors following (4). In some cases, the node features can be smoothed due to graph propagation, hence we can replace ÂF(l)(x) ·y F (l) in (8b) to [ÂF(l)(x) ·y F (l),y F (l)] to concatenate input features to prevent over-smoothing.

heading: 4.3.3 Feature Attention Layer
text: In the cases that a graph is fully connected or the edges have no weights, wx,y in (4) will be not well defined. Prior methods often rely on an attention mechanism, e.g., GAT [35], to focus on the most important neighbors. Inspired by this, we can define the edge weights as an attention in (9).
wx,y = exp(ex,y)∑
z∈N (z) exp(ex,z) , ex,y = LeakyReLU(a
T xx+ a T y y + b), (9)
where ax,ay ∈ RF , b ∈ R are learnable attention parameters. Note that we can construct other types of layers based on the topology GF , e.g., extending convolution to kervolution [45] to improve the model expressivity or combining the pagerank algorithm [46] to further reduce feature oversmoothness. In this paper, we will mainly demonstrate the effectiveness of the above three layers.

heading: 4.4 Analysis and Computational Complexity
text: We next provide an intuitive explanation for feature graphs. A feature graph doesn’t retain the physical meaning of its regular graph. Take a social network as an example, in a regular graph, each user is a node and user connections are edges. In feature graphs, each user is a graph, while the user features are nodes, including the users’ age, gender, etc. Therefore, user behavior prediction becomes a problem of graph classification from node classification and the increasing users simply become multiple training samples. Since the number of user features is more stable than the number of users, the size of a feature graph is more stable than its regular graph. This simplifies the learning process dramatically since the lifelong graph learning can be solved by regular lifelong learning techniques.
On the other hand, a regular graph usually assumes that some useful information of a node is often encoded into its neighbors, thus graph propagation is able to improve the model performance. A feature graph has the same assumption. However, feature graph doesn’t directly propagate the neighbor features, but encode the neighbors into the feature adjacency matrices (4). Regardless of the nonlinear function, existing methods [5, 8, 35, 46] propagate neighbor features as ÂX, where X = [x1, · · · ,xn], which means that they can only propagate neighbors in an element-wise manner, as nodes in the next layer are weighted average of the current nodes. In contrast, each feature graph propagate features via ÂFxF , which explicitly model “interaction” between features. In this sense, feature graph doesn’t lose information of edges but encode them into the feature adjacency. This explains the phenomenon that it also outperforms the graph models in some cases of regular graph learning presented in Section 5. Take a citation graph as an example, a keyword in an article may have an impact on another keyword in the article citing the former one. However, element-wise graph propagation like (2) [5, 8, 35, 46], cannot explicitly model this relationship. Therefore, feature graph will also work for regular graph learning, although we mainly focus on lifelong graph learning.
Feature graphs have a low computational complexity. Concretely, the complexity for calculating the feature adjacency is O(nF 2(l)), where n is the expected number of neighbors of each node in the continuum. Therefore, regardless of the number of layers which is a constant for specific models,
the complexity of graph propagation for a feature broadcast layer and a feature transform layer is of O(F 2(l)C(l)C(l+1)) and O(F(l)F(l+1)C 2 (l)), respectively. If we take E(l) = F(l)C(l) as the number of elements of the features, the complexity for calculating each new item in the continuum is roughly of O(E2 + nF 2), where we leave out the layer index for simplicity. Note that they are irrelevant to the task number, thus feature graphs have constant complexity with the increased learning tasks.

heading: 5 Experiments
text: Implementation Details We take the one-hot vector as the target vector zi, adopt the crossentropy loss, and use the softsign [47] function σ(x) = x/(1+|x|) as the non-linear activation. Our implementation is based on PyTorch [48]. We perform comprehensive tests on popular graph datasets including citation graph Cora, Citeseer, Pubmed [49], and ogbn-arXiv [50]. We list more details for these datasets in Appendix A. We use two feature broadcast layers (7) for Cora, Citeseer, and Pubmed. Feature channel is set as C(1) = 1 and C(2) = 2 for hidden layers. For obgn-arXiv, we found that node features can be easily smoothed due to multiple feature propagation, hence we use the feature transform layers to concatenate its input features. We adopt the SGD [51] optimizer for Cora, Citeseer, and Pubmed and the Adam optimizer [52] and obgn-arXiv. We choose the most popular graph models including GCN [5], GAT [35], SAGE [6], and APPNP [46] as our baseline. The other methods, such as SIANT [44] require pre-processing entire dataset, thus it cannot be applied to lifelong learning. The overall test accuracy after learning all tasks is adopted as the evaluation metric [21]. For each dataset, we construct two different continuum, i.e., data-incremental and class-incremental. For data-incremental tasks, all samples are streamed randomly, while in class-incremental tasks, i.e., all samples from one class are streamed before switching to the next class. In both tasks, each node can only be present to the model once, i.e., a sample cannot update model parameters again once dropped.
Sequence Invariant Sampling We adopt the lifelong learning method described in [21] with a minor improvement for all the baseline models. As reported in the Section 4.2 of [21], it tends to select less earlier items in the continuum and this has a negative effect on remembering earlier knowledge. We find that this is because that a uniform sampling is adopted for all items. To compensate for such effect, we set a customized selection probability for different items and present it in the Appendix B.
Performance Lifelong learning has a performance upper bound given by the regular learning settings, thus the performance in regular learning is also an important indicator for the effectiveness of feature graph. We present the overall performance of regular learning in Table 2 and denote this task as “R”, where the same dataset settings are adopted for different models. To demonstrate the
necessity of graph models, we also give the performance of MLP, which neglects of the graph edges. It can be seen that feature graph achieves a comparable overall accuracy in all the datasets. This means that feature graph may also be useful for regular graph learning, although we focus on lifelong graph learning in this paper. We next show that feature graph in lifelong learning is approaching this upper bound and dramatically alleviate the issue of “catastrophic forgetting”.
The overall averaged performance on the data-incremental tasks is reported as task of “D” in Table 2, where all performance is an average of three runs. We use a memory size of 500 for the datasets Cora, Citeseer and Pubmed, and a memory size of 512 for ogbn-arXiv. It is also worth noting that we have a low standard deviation, which means that its performance is stable and
not sensitive to the model initialization. Samples in this class-incremental tasks are non-iid and we have no prior knowledge on the task boundary, thus it is generally more difficult than data-incremental tasks. The overall test accuracy on class-incremental tasks are listed as “C” in Table 2. Similar to the data-incremental tasks, we use a memory size of 500 for Cora, Pubmed, and Citeseer, and a memory size of 4096 for ogbn-arXix, which is grouped into 5 tasks, each of which contains 8 classes. Despite its difficult, FGN is still able to obtain meaningful performance on ogbn-arXix in task “C”, compared to other state-of-the-art methods, which verifies its superiority in lifelong graph learning. We further report the forgetting rate in the task “C” for the best models in Table 3, which is the averaged class precision dropping compared to the regular learning. It can be seen that FGN achieves the least forgetting rate, which further demonstrates its effectiveness in lifelong graph learning.

heading: 6 Distributed Human Action Recognition with Wearable Devices
text: Implementation Details To demonstrated the flexibility of FGN, we apply it to an application, i.e., distributed human action recognition using wearable motion sensor networks in Fig. 2a (adapted from [53]). Five sensors, each of which consists of a triaxial accelerometer and a biaxial gyroscope, are located at the left and right forearms, waist, left and right ankles, respectively. Each sensor produces 5 data streams and totally 5 × 5 data streams is available. The stream is recorded at 30Hz and is comprised of human subjects with ages from 19 to 75 and 13 daily action categories, including rest at standing (ReSt), rest at sitting (ReSi), rest at lying (ReLi), walk forward (WaFo), walk forward left-circle (WaLe), walk forward right-circle (WaRi), turn left (TuLe), turn right (TuRi), go upstairs (Up), go downstairs (Down), jog (Jog), jump (Jump), and push wheelchair (Push).
We take each 25 sequential data points from a single sensor as a node and perform recognition for every 50 sequential points (1.67s) shown in Fig. 2b, which is also adopted by [54]. This results in a temporal growing (streaming) graph, where each node is associated with a multi-channel (C = 5) 1-D signal xit ∈ R25×5, where i = 1, 2 . . . 5 denotes the i-th sensor and t = 1, 2 . . . T is the time index. In this way, the problem of distributed human action recognition becomes a problem of sub-graph (10 nodes) classification and lifelong human action recognition becomes a lifelong graph learning problem. In practice, the temporal growing graph can only be learned sequentially, thus we take the first 80% of each sequence for training and the remaining 20% for testing. Specifically, we define the radius of neighborhood as the temporal distance. Therefore, all the nodes at the same instant are 1-hop neighbors of each other. For each feature graph we have K = 2 in the continuum (1). We construct FGN using two feature transform layers (8) with attention weights (9) to construct the sub-graph embedding. For fairness, we use C(1) = 5, F(1) = 25, C(2) = 32, and F(2) = 12 for all models. During training, we find that GCN, APPNP are able to obtain best overall performance using the SGD optimizer, while MLP, GAT, and FGN performed the best using the Adam optimizer.
Performance We list the overall performance of regular and lifelong learning in Table 4, which is obtained from an average of three runs. It can be seen that FGN achieves the best overall performance in regular learning and a much higher performance in lifelong learning compared to the state-of-the-art methods. This means that FGN has a lower forgetting rate, demonstrating its effectiveness.
We present the overall test accuracy during the lifelong learning process in Fig. 2c. It can be seen that FGN has a much higher and stable performance than all the other methods. We also shown their final per-class precision in Fig. 2d, which indicates that FGN achieves a much higher performance in
nearly all the categories. Note that all models have a relatively low performance in the three activities including ReSt, ReSi, and ReLi. This is because their signals are relatively flat and similar due to their static posture. This phenomenon is also observed by a template matching method KCC [54]. Note that we don’t directly compare against to KCC, since KCC mainly performs matching for single subject, while we aim for prediction across different subjects.
We argue that the superior performance of FGN is because of that FGN is able to explicitly model the relationship of different features. Take the walking action as an example, it is well known that the movement of a left arm is always related to the movement of the right leg. Such information can be explicitly modeled by the cross-correlation by the feature adjacency matrix. Moreover, it is also able to model the feature relationship at different time step, e.g., the movement of a left arm at time t is also related to the movement of the right arm at time t + 1. As aforementioned, FGN can explicitly learn such relationship with k = 2 in (4), while the other methods can’t directly do this. Besides, FGN also achieves the least forgetting rate in Table 5, which is calculated from the difference between the final and best performance during the entire learning process.

heading: 7 Conclusion
text: In this paper, we focus on the problem of lifelong graph learning and propose the feature graph as a new graph topology, which solves the challenge of increasing nodes in a streaming graph. It takes features of a regular graph as nodes and takes the nodes as independent graphs. To construct the feature adjacency matrices, we accumulate the cross-correlation matrices of the connected feature vectors to model feature interactions. This successfully converts the original node classification to graph classification and turns the problem from lifelong graph learning into regular lifelong learning. The comprehensive experiments show that the feature graph achieves superior performance in both data-incremental and class-incremental tasks. The practical applications on distributed action recognition also demonstrates its superiority in lifelong graph learning.

heading: Acknowledgment
text: This work was sponsored by ONR grant #N0014-19-1-2266.

heading: A Dataset Details
text: We perform comprehensive experiments on popular graph datasets including citation graph Cora, Citeseer, Pubmed, and ogbn-arXiv [49, 50, 55]. Their statistics are listed in Table 6.

heading: B Proof of Sequence Invariant Sampling
text: Let P be the probability that the observed items are selected at time t, thus the probability that one item is still kept in the memory after k selection is P k. This explains that earlier items have lower probability to be kept in the memory and this phenomenon was reported in the Section 4.2 of [21]. To compensate for such effect and Proposition B.1. To ensure that all items in the continuum have the same probability to be kept in the memory at any time t, we can set the probability that the n-th item is selected at time t as
Pn(t) =
{ 1 t 6M
M/n t > M, t = n (n−1)/n t > M, t > n , (10)
where M denotes the memory size.
Proof. It is obvious for t 6M as we only need to keep all items in the continuum. For t > M , the probability that the n-th item is still kept in the memory at time t is
Pn,t = Pn(n) ·Pn(n+ 1) · · ·Pn(t− 1) ·Pn(t),
= M
n ·
n n+ 1 · · · · t− 2 t− 1 · t− 1 t ,
= M
t .
(11)
This means the probability Pn,t is irrelevant to n and all items in the continuum share the same probability. In practice, we always keep M items and sample balanced items accross classes.

heading: C Efficiency Analysis
text: Due to its simplicity, feature graph requires less learnable parameters. Take the ogbn-arXiv dataset as an example, the number of model parameters including MLP, GCN, SAGE, GAT, APPNP, and FGN used in Section 5 are listed in Table 7. As reported in [56], GCN is the fastest graph learning method, however, FGN requires only 58% parameters compared to GCN, which demonstrates its efficiency.

Review summaries:
This paper aims to bridge GNNs with life-long learning so that the catastrophic forgetting problem in graph-structured tasks is alleviated. Specifically, the major contribution seems to be transforming the original graph into a feature graph so that the node classification problem is transferred into a graph classification problem with isolated samples. Meanwhile, feature interactions are modeled in constructing edges of the feature graph. Experiments on three citation graphs demonstrate the effectiveness of the proposed method. 

This paper aims to solve the problem of lifelong graph learning. Thus far, the topic about graph learning and lifelong learning is still underexplored. This paper proposes a new graph topology based on feature interaction, which takes the features as nodes and turns the nodes into graphs, and thus formulates a regular lifelong learning problem by defining the feature graph continuum. The authors conduct experiments on three popular citation graph datasets including Cora, Citeseer, and Pubmed.