Bridging Graph Network to Lifelong Learning with Feature Interaction
Authors: Chen Wang
Yuheng Qiu
Sebastian Scherer
Venue:  (2021)
Keywords: Graph Neural Network
Continual Learning

Abstract: Graph neural networks (GNN) are powerful models for many graph-structured tasks. In this paper, we aim to bridge GNN to lifelong learning, which is to overcome the effect of ``catastrophic forgetting" for continuously learning a sequence of graph-structured tasks. Although many lifelong learning techniques for convolutional neural networks (CNN) have been developed, lifelong learning for GNN is still underexplored and suffers from incomplete graph structure during learning. This is because in lifelong learning the nodes increase dynamically and can only be present to the model once, which makes many graph models and sampling strategies inapplicable. To solve this problem, we propose a new graph topology based on feature interaction, called the feature graph. It takes features as new nodes and turns nodes into independent graphs. This successfully converts the original problem of node classification to graph classification, in which the increasing nodes are turned into training samples. Therefore, the lifelong learning techniques developed for CNN become applicable to GNN for the first time. In the experiments, we demonstrate both the efficiency and effectiveness of feature graphs for lifelong learning tasks using a rehearsal method. We expect that it will have broad potential applications for graph-structured tasks in lifelong learning.

Lifelong Graph Learning
Chen Wang
Yuheng Qiu
Dasong Gao
Sebastian Scherer bast Dasti0cmml @0mmu edu
cherwangedz.com
yuhengg@andrew. cmn edu
dasonggdandrew. cmu. eduu
The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213
Abstract
Gruph neural networks (GNNs) are powerful models for many graph-structured tasks. Existing models often assume that a complete structure of graph is available during training. In practice, however, graph-structured data is usually formed in a streaming fashion so that learning a graph continuously is often necessary. In this paper, we aim to bridge GNN to lifelong learning by converting a graph problem to a regular learning problem, so that GNN can inherit the lifelong learning techniques developed for convo- lutional neural networks (CNNs). To this end, we propose new graph topology based on feature cross-corvelation, namely, the feature graph. It takes features as new nodes and turns nodes into independent graphs. This success- fully converts the original problem of node classification to graph classification, in which the increasing nodes are turned into independent training samples. In the experi- ments, we demonstrate the efficiency and effectiveness of feature graph networks (FGN) by continuously learning sequence of classical graph datasets. We also show that FGN achieves superior performance in two applications. i.e. lifelong human action recognition with wearable devices and feature matching. To the best of our knowledge, FGN is the first work to bridge graph learning lifelong learn- ing via novel graph topology. Source code is available at https: //g1 thub. com/ www/wang-ooceecoooooeeoewo.ooog www./whennnaaaaaaeeeeeaea LGL.
Z.
Z.
as
(a) Regular graph G.
(b) Feature graph G.F.
Figure 1. We introduce feature graph network (FGN) for lifelong graph learning. A feature graph takes the features as nodes and turns nodes into graphs, resulting in graph predictor instead of the node predictor. This makes the lifelong learning techniques for CNN applicable GNN, the new nodes in regular graph become individual training samples. Take the node a with label za in the regular graph & as an example, its features x. = [1,0, 0, are nodes { {a1, a2, 0g au} in feature graph G. The feature adja- cency is established via feature cross-coreelation between and its neighbors M(a) {a, b, e, d, e} to model feature "interaction."
we extend graph neural networks to lifelong learning, which is also known as continual or incremental learning [26].
Lifelong learning often suffers from "catastrophic forget- ting" if the models are simply updated with new samples [35]. Although some strategies have been developed alleviate the forgetting problem for convolutional neural networks (CNN), they are still difficult for graph networks. This is because in the lifelong learning setting, the graph size can in- crease over time and we have to drop off old data or samples to learn new knowledge. However, the existing graph model cannot directly overcome this difficulty. For example, graph convolutional networks (GCN) require the entire graph for training [20]. SAINT [58] requires pre- pre-processing for the entire dataset. Sampling strategies [7, 13, 58] easily forget old knowledge when learning new knowledge.
1. Introduction
Graph neural networks (GNN) have received increas- ing attention and proved usefuu for many tasks with graph- structured data, such as citation, social, and protein networks [52]. However, graph data is sometimes formed in a stream- ing fashion and real-world datasets are continuously evolving over time, thus learning a streaming graph is expected in many cases [46]. For example, in a social network, the num- ber of users often grows over time and we expect that the model can learn continuously with new users. In this paper,
Recall that regular CNNs are trained in a mini-batch man- ner where the model can take samples as independent inputs [23]. Our question is: can we convert graph task into a traditional CNN-like classification problem, so that (I) nodes can be predicted independently and (II) the lifelong learning

---

techniques developed for CNN can be easily adopted for GNN? This is not straightforward as node connections can- not be modeled by a regular CNN-like classification model. To solve this problem, we propose to construct a new graph topology, the feature graph Figure 1, to bridge GNN to lifelong learning. It takes features nodes and turns nodes into graphs. This converts node classification to graph clas- sification where the node increments become independent training samples, enabling natural mini-batch training.
the loss of exemplars does not increase, gradient episodic memory (GEM) [26] introduced orientation constraints dur- ing gradient updates. Inspired by GEM, [2] selected exem- plars with a maximal cosine similarity of the gradient orien- tation. iCaRL [32] preserved subset of images with a herd- ing algorithm [49] and included the subset when updating the network for new classes. EEIL [5] extended iCaRL by learning the classifier in an end-to-end manner. [51] further extended iCaRL by updating the model with class-balanced exemplars. Similarly, [3, 16] further added constraints to the loss function to mitigate the effect of imbalance. To reduce the memory consumption of exemplars, [18] applied the dis- tillation loss to feature space without having access to the corresponding images. Rehearsal approaches with synthetic data based on generative adversary networks (GAN) were used to reduce the dependence on old data [14, 40, 50, 533.
The contribution of this paper includes: (1) We introduce a novel graph topology, i.e. feature graph, to convert a prob lem of growing graph to an increasing number of training samples, which makes existing lifelong learning techniques developed for CNNs applicable GNNs. (2) We take the cross-correlation of neighbor features as the feature adja- cency matrix, which explicitly models feature "interaction", that is crucial for many graph-structured tasks. (3) Feature graph is of constant computational complexity with the in- creased learning tasks. We demonstrate its efficiency and effectiveness by applying it to classical graph datasets. (4) We also demonstrate its superiority in two applications, i.e. distributed human action recognition based on subgraph clas- sification and feature matching based on edge classification.
2.2. Graph Neural Networks
Graph neural networks have been widely used to solve problems with graph-structured data [60]. The spectral net- work extended convolution to graph problems [4]. Graph convolutional network (GCN) [20] alleviated over-fitting on local neighborhoods via the Chebyshev expansion. To identify the importance of neighborhood features. graph at- tention network (GAT) [42] added an attention mechanism into GCN, further improving the performance on citation networks and the protein-protein interaction dataset.
2. Related Work
2.1. Lifelong Learning
Lifelong learning methods in this category do not preserve any old data. To alleviate the for- getting problem, progressive neural networks [36] leveraged prior knowledge via lateral connections to previously Iearned features. Learning without forgetting (LwF) [24] introduced knowledge distillation loss [15] to neural networks, which encouraged the network output for new classes to be close to the original outputs. Distillation loss was also applied to learning object detectors incrementally [41]. Learning without memorizing (LwM) [10] extended LwF by adding an attention distillation term based on attention maps for retaining information of the old classes.
Non-rehearsal Methods
GCN and its variants require the entire graph during train- ing, thus they cannot scale to large graphs. To solve this prob- lem and train GNN with mini-batches, a sampling method, SAGE [13] is introduced to learn a function to generate node embedding by sampling and aggregating neighborhood features. JK-Net [54] followed the same sampling strategy and demonstrated significant accuracy improvement on GCN with jumping connections. DiffPool [57] learned a differentiable soft cluster assignment to map nodes to set of clusters, which then formed a coarsened input for the next layer. Ying et al. [56] designed a training strategy that relied on harder-and-harder training examples to improve the robustness and convergence speed of the model.
EWC [21] remembered old tasks by slowing down leam- ing on important weights. RWalk [6] generalized EWC and improved weight consolidation by adding a KL-divergence- based regularization. Memory aware synapses (MAS) (1] computed an importance value for each parameter in unsu- pervised manner based on the sensitivity of output function to parameter changes. [48] presented an embedding frame- work for dynamic attributed network based on parameter regularization. A sparse writing protocol is introduced to memory module [43], ensuring that only a few memory spaces is affected during training.
FasIGCN [7] applied importance sampling to reduce vari- ance and perform node sampling for each layer indepen- dently, resulting in a constant sample size in all layers. [17] sampled lower layer conditioned on the top one ensuring higher accuracy and fixed-size sampling. Subgraph sam- pling techniques were also developed to reduce memory consumption. [8] sampled a block of nodes in a dense sub- graph identified by a clustering algorithm and restricted the neighborhood search within the subgraph. SAINT [58] con- structed mini-batches by sampling the training graph.
Rehearsal Methods Rehearsal lifelong learning methods can be roughly divided into rehearsal with synthetic data or rehearsal with exemplars from old data [33]. To ensure that
Nevertheless, most of the sampling techniques still re- quire pre-processing of the entire graph determine the sampling process require complete graph structure,
2

---

which makes those algorithms not directly applicable to lifelong learning. In this paper, we hypothesize that a dif- ferent graph structure is required for lifelong learning, and the new structure is not necessary to maintain its original meaning. We noticed that there some recent work focusing on continuously learning graph problem, but they have dif- ferent formulation. For example, several exemplar selection methods are tested in ER-GNN [59]. A weight preserving method is introduced to growing graph [25]. A combined strategy of regularization and data rehearsal is introduced to streaming graphs in [46]. To overcome the incomplete structure. [12] learns a temporal graph in sliding window.
dropped. This is in contrast to the settings in [46], where all historical data are available during training. As shown in the experiments, lifelong graph learning in practice often requires that the number of GNN layers is larger than the availability of K K-hop neighbors, i.e. L K, which also leads many existing graph models inapplicable.
4. Feature Graph Network
To better show the relationship with a regular graph, we first review GCN [20]. Given an graph G (V. described in Section 3, the stacked node features can be written as X X2, xx]] E NxFC where x; RF is a vectorized node feature. The GCN takes feature channel as C = 1, so that x; E RR.X. E RNKF, The 1-th graph convolutional layer is defined as
3. Problem Formulation
We start by defining regular graph learning before life- long graph learning for completeness. An attribute graph is defined as G = (V,E), where V is the set of nodes and 3 S. {fra 06 (( 1y) E }²) is the set of edges. Each node v E V is associated with target Zy 2 and a multi- channel feature vector x, E X C IF and each edge E E is associated with a vector W. E W C R In regu- lar graph learning, we learn a predictor f to associate node Xv, U.G. E V with a target 2,, given graph G, node features X, edge vectors W, and part of the targets 2,, V E V V.
X = (A (A.X() W)
(2)
where o(-) is an activation function, W E R ( is a learnable parameter, and Á E NRNNN is a normalized adja- cency for A (refer to [20] for details). A graph convolutional layer doesn't I change the number of nodes (rows of X) but change it feature dimension from F()) to F((+1)-
GCN has been applied to many graph-structured tasks due to its simplicity and good generalization ability. How- ever, the problems of GCN are also obvious. Besides the forgetting problem in lifelong learning, its node features in the next layer is a linear combination of the current layer, thus GCN and its variants cannot directly model the feature "interaction". To this end, we introduce the feature graphs by defining feature nodes and feature adjacency matrix.
In lifelong graph learning, we have the same objective, but can only obtain the graph-structured data from a data contin- uum Gr. = {(x; 4, Z; Wi 1:K (x;;) (x;), Wh= 1:K (x.) i=))))) }. where each item is formed by node feature x, X, task descriptor 4 E 'T a target vector 2, E Zr,- a h--hop neighbor set NN= (x:), and an edge vector set W (x,) associ- ated with the k-hop neighbors. For simplicity, we will use the symbol (xx)) to denote the available neighbor set and their edges. We assume that every item satisfies (xxx) z.) ~ Pt. (X,, 2), where Pt, is probability distribution of single learning task. In lifelong graph learning, we will observe, item by item, the continuum of the graph-structured data as
4.1. Feature Nodes
4,, 2;)
Recall that each node v in a regular graph G = (V.E) is associated with multi multi-channel feature vector x = TT... , x [F..] E FxC where X|I.]] is the i-th feature (row) of x. An attribute feature graph takes the features of a regular graph as nodes. It can be defined as G (VT &T), where each node of E y is associated with feature x and will be denoted as x²" Intuitively, the number of nodes in the feature graph is the feature dimension in the regular graph, i.e. | = F, and the feature dimension in feature graph is the feature channel in regular graph, i.e. xf" E RC. Therefore, we define the feature nodes for feature graph as
(x,,N(x1), (1, 21),.... ... NN
2N)
(1)
While observing (1), our goal is to lear a predictor fL to associate a test sample (x,,),),, () with a target z such that (x,N(xx,,) z)- ~ P.. Such test sample can belong to a task observed in the past, the current task, or a task observed (or not) in the future. The task descriptors t, is defined for compatibility with lifelong learning that requires them [26] but is not used in the experiments.
D (xf
(3)
In this way, for each node v E V, we have feature graph g². We next establish their relationship by defining the feature adjacency matrices via feature cross-correlatio.
Note that samples are not drawn locally identically and independently distributed (i.i.d.) from a fixed probability distribution, since we don't know the task boundary. In the continuum, we only know the label of x,, but have no information about the labels of its neighbors N(xx)). The items in (1) are unavailable once they are observed and
4.2. Feature Adjacency Matrix
For each item in continuum (1), the edges between and its neighbors.WV (xx) imply the existence of correlations
3

---

Table 1.
The relationship of a graph and feature graph.
where a(-) is non-linear activation function, A is the associated normalized feature adjacency matrix, and W e IR Ct XC is a learnable parameter. For simplicity. the channel e is left out in (7) and the layer channel broad- casted independently. is worth noting that, although the definition of (7) appears similar to (2), they have different dimension and represent different meanings.
Graph
Feuture Graph
Relationship
Node
H+
Graph
Node classification I+
Graph classification
Feature
H
Nade
Fixed Feature + Fixed Nodes
Edgo
Edges
Growing Adjacency
Multiple Adjaceecy
Graph
Samples
Dynamic graph -
Multiple samples
between their features. We model the feature adjacency the correlation over the k-hop neighborhood (x) and for each of the channels independently, where e = I,...,C. C:
4.3.2 Feature Transform Layer
(x) A sgnroot (Ex-Ni<x) (oo (..0 (4)
Similar to graph convolutional layer (2), the feature broad- cast layer doesn't It change the number of feature nodes. How- ever, this is not always necessary, as the objective has been turned into graph classification. Therefore, we define a fea- ture transform layer which can change the number of feature nodes and will be helpful for further reducing the number of learnable parameters. Different from the feature broadcast layer, we need to re-calculate the feature adjacency matrices from transformed neighbors. Therefore, given the feature graph gF, the 1-th feature transform layer can be defined as
where Wx,y E IR is the associated edge weight and (g)root(z) sign(x) //xx| retains the magnitude of node features and the sign of their inner products. Note that k.c preserves the connectivity information by only encoding in- formation from connected nodes. For each sample x, this produces C' matrices of size F x F, where F < N due to the lifelong learning settings. For undirected graphs, we [.o) to (Xj.et [.c] + Y| M1s.e for symmetry. In practice, the expectation in (4) is approximated by averaging over the observed neighborhood:
change Xi..e)V(..)
A (x) & A (x0)) Vy E NN((x), x = a (w (x) xx))
(8a)
ExeM((x) Wn W Ge
(8b)
(x.y [.d]
(5)
[xx(x)
v6+1 = a (W A.. (x) vō)
(8c)
In this way. the feature adjacency matrix is constructed dy- namically and independently from neighborhood samples via (5), so that the continuum (1) is converted to graphs:
where W E Ra a learnable parameter, a(.) is (-)
a non-linear activation function, and A (x) E P is the normalized feature adjacency A (1) (xx). The node features sometimes can be smoothed due to graph propagation, hence we can replace (m) (x) yTT in (8c) to [A (x) (x)) vo by concatenating input features to prevent over-smoothing.
(of 11, (1),,,..., (oT li, 2i),...,,( (G (N, ZN)
(6)
where g. (V.,AF(x:)) Ax())) This means that our objec- tive of learning a node predictor becomes learning a graph predictor f= : GF x T + Z that predicts target z for test sample (gF, t) so that (G,,t.z) feature graphs in the new continuum (6) are still non-i.i.d.
P.T. Note that the
4.3.3 Feature Attention Layer
In this way, growing adjacency matrix is converted to multiple small adjacency matrices. Hence, lifelong graph learning problem becomes a regular lifelong learning prob- lem similar to [26] and the problem of increasing nodes can be solved by applying lifelong learning to the graph contin- uum (6). To be more clear, we list their detailed relationship in Table 1, where the arrow r> refers to the conversion from regular graph to multiple feature graphs.
In the cases that a graph is fully connected or the edges have no weights, Wayyy in (4) will be not well defined. Prior methods often rely on an attention mechanism, e.g. GAT [42], to focus on important neighbors. Inspired by this, we define the edge weights as attention in (9).
exp(ex.r)
Wx.y E ExEN(E) exp(ex,.)
(9a)
4.3. Feature Graph Layers
cz,y LeakyReLUUaffa an. b),
(9b)
Since feature graph is a new topology given by feature adjacency matrix, we are able to define many different types of layers. In this section we present three types of layers.
where ag, ag E RR E IR are learnable attention parame- ters. We can also construct other types of layers based on the topology g.F, egg. extending convolution to kervolution [44] to improve the model expressivity or combining the pagerank algorithm [22] to further reduce feature overssmoothness, In this paper, we will mainly demonstrate the effectiveness of the three types of layers introduced above.
4.3.1 Feature Broadcast Layer
Inspired by GCN, the 1-th broadcast layer is defined as
x x((1)> = a (A x w=).
4

---

4.4. Analysis and Computational Complexity
layer (8) as a feature extractor to project the raw features onto a lower dimension. For example, we reduced the feature dimension on Cora from F(i) = 1433 to F(2) = 10, which is also adopted by GCN, GAT, APPNP, SAGE, etc.
We next provide an intuitive explanation for feature graphs. A feature graph doesn't retain the physical meaning of its regular graph. Take social network an example, in regular graph, each user is node and user connections are edges. In feature graphs, each user is a graph, while the user features, including the users' age, gender, etc., are nodes. Therefore, user behavior prediction becomes graph classi- fication based on node information and new users simply become multiple training samples. Since the number of user features is more stable than the number of users, the size of feature graph is more stable than its regular graph. This simplifies the learning on growing graphs dramatically by reducing the problem to the sample-incremental learning.
5. Experiments
Implementation Details We perform comprehensive tests on popular graph datasets including citation graph Cora, Cite- seer, Pubmed [38], and ogbn-arXiv [29]. For each dataset, we construct two different continuum: iata-inccrmmetal and class- plass-incremental. In data-incremental tasks, all samples are streamed randomly, while in class-increman incremental tasks, all samples from one class are streamed before switching to the next class. In both tasks, each node can only be present to the model once, i.e., a sample cannot be used to update the model parameters again once dropped. For this experiment, we implement two-layer feature graph network in Py ToTch [31] ] and adopt the SGD [19] optimizer. See further details in Appendix A. We choose the most popular graph models including GCN [20]. GAT [42]. SAGE [13], and APPNP [22] as our baseline. Comparison with other methods such as SAINT [12] are omitted as they require a pre- proocssessi of the entire dataset, which is incompatible with the lifelong learning setting. The overall test accuracy after learning all tasks is adopted as the evaluation metric [2].
On the other hand, regular graph usually assumes that some useful information of node is often encoded into its neighbors, thus graph propagation is able to improve the model performance. A feature graph has the same assump- tion. However, feature graph doesn't directly propagate the neighbor features, but encode the neighbors into the feature adjacency matrices (4). Regardless of the nonlinear func- tion, existing methods propagate neighbor features as Åx, where X [xi..... x..], meaning they can only propagate information clement-wische as features in the next layer are weighted average of the current features [20, 22, 42, 58).
Sequence Invariant Sampling We adopt the lifelong learn- ing method described in [2] with a minor improvement for all the baseline models. As reported in the Section 4.2 of (2], it tends to select less earlier items in the continuum, which discourages the memorization of earlier knowledge. We find that this is because uniform sampling is adopted for all items. To compensate for such effect, we set a customized selection probability for different items (See Appendix B). Performance Lifelong learning has a performance upper bound given by the regular learning settings, thus the per- formance in regular learning also an important indicator for the effectiveness of feature graph. We present the overall performance of regular learning in Table 2 and denote this task as "R", where the same dataset settings are adopted for different models. To demonstrate the necessity of graph models, we also report the performance of MLP, which ne- glects the graph edges. It can be seen that feature graph achieves a comparable overall accuracy in all the datasets. This means that feature graph may also be useful for regular graph learning. We next show that feature graph in lifelong learning is approaching this upper bound and dramatically alleviate the issue of "catastrophic forgetting".
In contrast, each feature graph propagate features via A Fxx, which explicitly model "interaction" between fea- tures. In this sense, feature graph doesn't 't lose information of edges but encode them into the feature adjacency. This explains its superiority over regular graph models in some cases of conventional graph learning as shown in Section 5. Take a citation graph as example, keyword in an article may influence another keyword in the article citing the former. However, element-wise graph propagation like (2) cannot explicitly model this relationship [20, 22, 42, 58]. Although feature graph is also suitable for regular graph learning, we mainly focus our discussion on lifelong graph learning.
Feature graphs have a low computational complexity. Concretely, the complexity for calculating the feature adja- cency is (nF ), where n is the expected number of node neighbors in the continuum. Therefore, regardless of the number of layers which is a constant for specific models, the complexity of graph propagation for a feature broadcast layer and a feature transform layer is of (F)) C(ay ((()))) and (F C&t) respectively. If we take Eg) = Fig as the number of elements of the features, the complexity for calculating each sample is roughly of O(E + F where we leave out the layer index for simplicity. Note that they do not depend on the task number, thus feature graphs have constant complexity with the increased learning tasks.
The overall averaged performance on the data- incremental tasks is reported as task "D> in Table 2, where all performance is an average of three runs. We use a mem- ory size of 500 for the datasets Cora, Citeseer and Pubmed, and memory size of 512 for agbn-arXiv. It is also worth noting that we have low standard deviation, which means
FGN is applicable even when the number of features F is very large. In this case, we often use feature transformation
5

---

Table Overall performance comparison on all the datasets in Section 5.
Method
Task
Cora
Citeseer
Pubmed
ogbn-arXiv
MLP
0.673
0.702
0.832
0.462
GCN
0,850
0.768
0.868
0.557
SAGE
0,850
0.768
0.858
0.615
GAT
0.877
0.776
0.872
0.619
APPNP
0.883
0.777
0.870
0.615
FGN
0.887
0.785
0.884
0.631
MLP
0.652 + 0.006
0.707 + 0.008
0,812 + 0.006
0.349 + 0.006
GCN
0.827 + 0.003
0.760 + 0.004
0.845 + 0.002
0.397 + 0.012
SAGE
0.778 + 0.016
0.712 + 0.015
0,816 = 0.015
0,531 - 0,006
GAT
0.857 = 0.007
0.735 = 0.009
0,847 = 0.002
0.512 = 0.003
APPNP
0.861 + 0.001
0.725 1 0.016
0,851 T 0.003
0.511 + 0.002
FGN
0.870 + 0.011
0.762 1 0.003
0,872 TT 0.009
0.555 + 0.010
MLP
0.558 0.010
0.597 + 0.012
0,816 1 0.013
0,237 + 0.011
GCN
0.796 + 0.022
0.695 + 0.003
0,851 + 0.012
0.328 + 0,011
SAGE
0.826 + 0.014
0.716 + 0.001
0,841 + 0.011
0.455 + 0.003
GAT
0.833 + 0.032
0.698 + 0.009
0.833 + 0.006
0.409 + 0.006
APPNP
0.818 + 0.017
0.689 + 0.003
0.844 + 0.013
0.423 + 0.007
FGN
0.853 = 0.014
0.733 + 0.009
0.857 + 0.003
0.494 = 0.003
We denote the best performance in bold and second best with underline for each task. Due to the settings of lifelong learning, dataset split is different from the original one, as only test the final optimized model and don't 't need a validation set (Appendix A). TTas of "R", "D", "C" denote regular, data-incremental, and class-incremental learning, respectively.
Table 3. Average class forgetting rate in task "C" (%)
Table 4. The number of model parameters used in Section 5.
Model
Cora
Citeseer
Pubmed
ogbn-arXiv
Model
MLP
GCN
SAGE
GAT
APPNP
FGN
APPNP
6.38
8.03
3.61
16.3
# Parameters
37,888 37,888 37,932 38,700
38,184 21,872
GAT
5.50
7.05
3.50
17.7
FGN
1.35
0.41
3.49
14.8
6. Distributed Human Action Recognition
that its performance is stable and not sensitive to the model initialization. Samples in the class-incremental tasks are non-i..dd and we have no prior knowledge on the task bound- ary, thus it is generally more difficuli than data-incremental tasks. The overall test accuracy on class-incremental tasks are listed as "C" in Table 2. Similar to the cata-incremental tasks, we use a memory size of 500 for Cora, Pubmed, and Citeseer, and a memory size of 4096 for ogbn-arXix, which is grouped into 8 tasks, each of which contains 5 classes. Despite the difficulty. FGN still obtains higher performance ogbn-arXix in task "C", compared to other state-of-the- art methods, which verifies its superiority in lifelong graph learning. We further report the forgetting rate [6] in the task "C" for the best models in Table 3, which is the aver- age precision drop compared to the regular learning. It can be seen that FGN achieves the least forgetting rate, which demonstrates its effectiveness in lifelong graph learning- Memory Efficiency We report the number of learnable parameters in Table 4. Due its simplicity, FGN only requires 42% less parameters compared GCN and SAGE, which demonstrates its memory efficiency.
Implementation Details To demonstrate the flexibility of FGN, apply it an application, i.e. distributed human action recognition using wearable motion sensor networks in Figure 2a (Data from [55]). Five sensors, each of which consists of triaxial accelerometer and a biaxial gyroscope, are located at the left and right forearms, waist, left and right ankles, respectively. Each sensor produces data streams and totally 5 x 5 data streams available. The stream is recorded at 30Hz and is comprised of human subjects with ages from 19 to 75 and 13 daily action categories, includ- ing rest at standing (ReSt), rest at sitting (ReSi), rest at lying (ReLi), walk forward (WaFo), walk forward left-circle (WaLe), walk forward right-circle (WaRi), turn left (TuLe), turn right (TuRi), go upstairs (Up), go downstairs (Down), jog (Jog). jump (Jump), and push wheelchair (Push).
We take every 25 sequential data points from a single sensor as a node and perform recognition for every 50 se- quential points (1.67s) shown in Figure 2b, which is also adopted by [45]. This results in a temporally growing graph: Each node is associated with multi-channel (C = 5) 1-D signal x E 255x5 where
1,2. 5 is the sensor index
6

---

0.95
Push
GCN
Jump
GCN
S Dimension
Right Right Left
Precision [%)
0.9
APPNP
Down
1gg
APPNP
Up
FGN
TuRi
FGN
0.85
TuLe
WaRi
WaLe
0.8
WaFo
ReLi
ReSi
5
0.75
ReSt
Sub-Graph (b) The temporal growing graph.
25 Points
Node
Sequence
0.5
Sensors.
(c) Lifelong learning.
(d) Per-class precision.
Figure 2. The wearable sensor networks, the streaming graph, precesion during learning, and the final performance comparison.
Table 5. Backward maximum forgetting rate of all models all actions (%).
Model
ReSt
ReSi
ReLi
WaFo
WaLc
WaRi
TuLe
TuRi
Up
Down
Jog
Jump
Push
Overall
GCN
5.34
12.49
0.99
1.32
2.88
0.59
0.89
1,58
1.30
2.81
1.24
0.26
5.31
2.85
APPNP
0.81
2.57
7.17
1.37
2.70
0.80
0.94
0.44
0.11
3.97
0.99
0.00
3.66
1.96
GAT
1.34
6.12
4.23
0,00
2.79
0.76
0.82
0.83
0.81
2.27
0.62
0.87
4.79
2.02
FGN
0.19
0.56
3.32
0.95
2.67
0.53
1.15
1.04
0.00
3.52
0.46
1.02
2.28
1.36
Table 6. Precision comparison the action recognition.
of different features. Take the walking action as an example, it is well known that the movement of a left arm is always related to the movement of the right leg. Such information can be explicitly modeled by the cross-corrllation the fea- ture adjacency matrix. Moreover, it is also able to model the feature relationship at different time step, e.g. the movement of left arm at time / is also related to the movement of the right arm at time t + 1. As aforementioned, FGN can explic- itly learn such relationship with A: = 2 in (4), while the other methods can't 't directly do this. Besides, FGN also achieves the least forgetting rate in Table 5 over all classes, which the difference between the final and best performance during the entire learning process.
MLP
GCN
APPNP
GAT
FGN
Regular L0.046 0,881 10.016 0.002
0.91110.004
0.95410.004
Lifelong -1
0.86420.013 0. 0.88 0.826#000144 0.914110.099
Thee notation "." that cannot obtain meaningful performance this task.
and / 1, is the time index. We assume all nodes at adjacent time index connected, i.e. for all t, nodes (x?...., .x,, x x(+1) form fully connected sub- graph. Therefore, the problem of human action recogmition becomes a problem of sub-graph (10 nodes) classification. Performance We list the overall performance of regular and lifelong learning in Table 6, which is obtained from an average of three runs. It can be seen that FGN achieves the best overall performance in regular learning and much higher performance in lifelong learning compared the state-of-the-ar methods. This means that FGN has lower forgetting rate, demonstrating its effectiveness.
7. Image Feature Matching
Implementation Details We next extend FGN more challenging application, i.e. image feature matching. It is crucial for many 3-D computer vision tasks including si- multaneous localization and mapping (SLAM). As shown in Figure 4, the interest point and their descriptors form an infi- nite temporal growing graph, in which the feature points are nodes and their descriptors are the node features as defined in Section 3. this way, the problem of feature matching becomes edge prediction for temporal growing graph. We next show that the performance of SuperGlue [37], which used a regular graph attention model for feature matching, can be simply improved by changing the graph-attention matcher to our proposed FGN. For simplicity, we adopt a framework similar to that of SuperGlue but removed the cross-atee coos-atttntionn layers in the matching network. In image fea- ture matching, we normally have layered temporal graph, where edge weights undefined. Therefore, we construct
We present the overall test accuracy during the lifelong learning process in Figure 2c. It can be seen that FGN has a much higher and stable performance than all the other methods. We also shown their final per-class precision in Figure 2d, which indicates that FGN achieves a much higher performance in nearly all the categories. Note that all mod- els have relatively low performance on ReSt, ReSi, and ReL.i This is because their signals are relatively flat and similar due to their static posture. This phenomenon is also observed by a template matching method KCC [45]. Note that we don't directly compare against to KCC, since KCC mainly performs matching for single subject, while we aim for prediction across different subjects.
We argue that the superior performance of FGN is be- cause of that FGN is able explicitly model the relationship
7

---

Indoors
Outdoors
Natural
Artificial
100
100
100
100
Ours
Precision [%]
Differnnce
0.5
Difference (%)
Precision [%)
Diference
0.5
Difference (%)
Precision [%)
Diffrrncc
0.5
Difference [%)
Precision [%)
Diffrrncee
0.5
Difference [%6
50
0
50
0
50
50
-0.5
-0.5
-0.5
-0.5
0
1
10c
10
10c
10
10
10
10
Threshold [px]
Tord [px]
Toreshold [px]
Threshold [px]
Mixed
Dark
Hard
Overall
100
100
100
100
Ours
Ours
Ours
Ours
Precision [%)
Difference
0.5
Difference (%)
Precision [%)
Diffrrncee
0.5
Difference [%)
Precisian [%)
Difference
0.5
Difference [%)
Precisoo [%)
Differncce
05
Difference
50
50
0
50
.0.5
.0.5
-0.5
-0.5
10
10
10
10 Threshald [px]
10a
10
10
10
Threshold [px)
Threshold [px)
Threshold [px]
Figure 3. The matching precision of our method (blue) and performance gap between our method and GAT (orange) at different level of tolerances. Our method outperforms GAT at sub-pixel precision in mildly difficult categories.
Table 7. Mean Error (pixels) in image feature matching."
Category
0
A
M
Overall
# Sequence
17
63
19
53
11
6
27
80
GAT
1.70
2.73
4.17
2.31
2.19
6.00
4.68
2.64
FGN (Ours)
1.52 2.05
2.88
1.85
1.56 4.75
3.25 2.00
Reduction (%) -10.6 224.9 -30.9 -19.9 -28.8 -20.8 -30.6 -24.2
!Scene categories include: (Indoor and (O)utdoor, (N)aturalistic (woods and sea floor), (A )ttificial (strects and buildings), (M)ixed (containing both natural and artificial objects), (D)ark (poor light- ing condition), (H)ard (violent motion and/or complex textures).
Figure 4. Matching examples on TartanAir dataset. Feature match- ing is problem edge prediction for temporal growing graph.
ronments, which is beneficial for downstream tasks such as simultaneous localization and mapping (SLAM).
FGN by concatenating two feature broadcast layers (7) with the attention edge weights Wz,y defined in (9).
8. Conclusion
Performance We categorize the 80 test sequences from Tar- tanAir dataset [47] into groups based on their characteristics. Figure 4 shows several examples of consecutive matching in Indoors, Outdoors&Artificii
In this paper, we focus on the problem of lifelong graph learning and propose the feature graph new graph topol- ogy, which solves the challenge of increasing nodes in a streaming graph. It takes features of a regular graph as nodes and takes the nodes as independent graphs. To con- struet the feature adjacency matrices, we accumulate the cross-correlation matrices of the connected feature vectors to model feature interaction. This successfully converts the original node classification to graph classification and turns the problem from lifelong graph learning into regular life- long learning. The comprehensive experiments show that feature graph achieves superior performance in both data- incremental and class-incremental tasks. The applications on action recognition and feature matching demonstrates its superiority in lifelong graph learning. To the best of our knowledge, feature graph is the first work to bridge graph learning to lifelong learning via novel graph topology.
and Ouxdd en- vironments. We report the mean matching error in Table 7, compared with the GAT-based matcher. It can be observed that our method outperforms GAT by a noticeable margin on all categories, especially on the difficult ones (Outdoors, Natural, Hard). Specifically, FGN achieves an overall error reduction of 24.2% compared with GAT.
We further report the difference in error landscape of FGN and GAT in Figure 3. Although there is no significant difference in matching precision at the single-pixel level, our method experiences a greater boost in precision in the sub-pixel region when error tolerance increases. This effect is especially noticeable for the Artificial and Outdoors cate- gories. This suggests that our method is more advantageous at identifying high quality matches in mildly difficult envi-
8

---

Acknowledgment
informatioon processing systems, pages 1024-10344 2017. 1,
2,5
This work was sponsored by ONR grant #N0014-19-1-2 266 and ARL DCIST CRA award 1911NF-11-2-0181.
[14] Chen He, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exemplar-supported generative reproduction for class incre- mental learning. In BMVC, page 98, 2018. 2
References
[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint
[1] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelarrs. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages
orXiv:1503.02333 02531. 2015. 2
[16] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE Conference on Com- puter Vision and Puttern Recognition (CVPR), pages 831-839,
139- 339-154.22 2018. 2
[2] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Ben-
2019. 2
gio. Gradient based sample selection for online continual
[17] Wenbing Huang, Tong Zhang, Yu Rong. and Junzhou Huang. Adaptive sampling towards fast graph representation learning. In Advances in neural information processing systems, pages
learning. Advances in Neural Information Processing Sys-
tems, pages 11816-11822 1825, 2019. 2, 5, 11
[3] Eden Belouadah and Adrian Popescu. 112m: Class incre- mental learning with dual memory. In Proceedings of the
4558-4567, 2018. 2
[18] Ahmet Iscen, Jeffrey Zhang, Svetlana Lazebnik, and Cordelia Schmid. Memory-efficient incremental learning through fea- ture adaptation. In European Conference on Computer Vision
IEEE International Conference on Computer Vision, pages
583-592, 2019. 2
[4] Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann Le- Cun. Spectral networks and locally connected networks on graphs. In International Conference on Learning Representa- tions (ICLR), 2014.
(ECCV). 2020. 2
[19] Jack Kiefer, Jacob Wolfowitz, et al. Stochastic estimation of the maximum of a regression function. The Annals of Mathematical Statistics, 23(3):462-466, 1952. 5
[5] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremen-
[20] Thomas N. Kipf and Max Welling. Semi-supervised classifi- cation with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017. 1, 2,
tal learning. In Proceedings of the European Conference Computer Vision (ECCV). pages 233-248, 2018. 2 [6] Arslan Chaudhry. Puneet K Dokania, Thalaiyasingam Ajan- than, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Pro- ceedings of the European Conference on Computer Vision
3,5
[21] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran John Quan, Tiago Ramalho, Agnieszka Grabska- Barniskk,, et al. Overcoming catastrophic forgetting in neural net- works. Proceedings of the national academy of sciences.
Milan,
(ECCV), pages 532-547, 2018. 2, 6
[7] Jie Chen, Tengfei Ma, and Cao Xiao. Fastgen: fast learning with graph convolutional networks via importance sampling.
114(13):351133526 2017. 2
[22] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Güi-- nemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations (ICLR), 2019. 4,5
arXiv preprini arXiv:/801. 10247, 2018. 1.2
[8] Wei-Lin Chiang. Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. Cluster-gen: An efficient algorithm for training deep and large graph convolutional networks. Pru- ceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 257-266, 2019. 2
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Im- agenet classification with doep convolutional neural networks. In Advances in neural information processing systems, pages
1097-1105, 2012.
[9] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabi- novich. Superpoint: Self-supervised interest point detection and description. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 224-236. 2018 12
[24] Zhizhong Li and Derck Hoiem. Learning without forget- ting. IEEE Iransactions on paltern analysis and machine
intelligence, 40(12):2935-2947, 201
2017. 2
[25] Huihui Liu, Yiding Yang. and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In AAAI
[10] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without mem- orizing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5138-5146, 2019. 2 [11] David L Elliott. A better activation function for artificial neural networks. Technical report, University of Maryland, 1993. 11
Conference on Artificial Intelligence (AAAI), 2021. 3
[26] David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual leariig. In Advances Neural Information Processing Systems, pages 6467-6476,
2017. 2,
[27] David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 1150-1157. leee, 1999. 11
[12] Lukas Galke, Iacopo Vagliano, and Ansgar Scherp. Incre- mental training of graph neural networks on temporal graphs under distribution shift. arXiv preprint
[28] Julian McAuley and Jure Leskovec. Image labeling on a net- work: using social-network metadata for image classification. In European conference on computer vision, pages 928-841.
2020. 3,5
[13] Will Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in neural

---

Springer. 2012. 11
estingness via unsupervised online learning. In European Conference on Computer Vision (ECCV), 2020. [44] Chen Wang. Jianfei Yang. Lihua Xie, and Junsong Yuan. Kervolutional neural networks. In The IEFF Conference on Computer Vision and Pattern Recognition (CVPR), pages
[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 31 1 1-3119, 2013. 5,
31-40. 2019. 4
[30] David Novotny, Samuel Albanie, Diane Larlus, and Andrea Vedaldi. supe--ipe learning of geometrically stable features through probabilistic introspection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3637-3645, 2018. 12
[45] Chen Wang. Le Zhang. Lihua Xie, and Junsong Yuan, Kernel cross- corsecoor....or. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 6,
[46] Junshan Wang. Guojie Song. Yi Wu, and Liang Wang. Stream- ing graph neural networks via continual learning. In Proceed- ings of the 29th ACM International Conference on Informa- tion & Knowledge Management, pages 1515-1524, 2020. 1,
[31] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differ- entiation in pytorch. In Conference on Neural Information Processing Systems (NIPS), 2017. 5
[47] Wenshan Wang. Delong Zhu, Xiangwei Wang. Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafci Hu, Ashish Kapoor, and Schastian Scherer. Tartanair. A dataset to push the limits of visual slam. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020. 8, 122
[32] Sylvestre-Ale Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017. 2
[48] Hao Wei, Guyu Hu, Wei Bai, Shiming Xia, and Zhisong
Pan. Lifelong representation learning in dynamic attributed networks. Neurocomputing, 358:1-9, 2019. 2
[33] Anthony Robins. Catastrophic forgetting, rehearsal and pseu- dorehearsal. Connection Science, 7(2):123-14,, 1995. 2 [34] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary Bradski. Orb: An efficient alternative to sift or surf. In 20/7 International conference on computer vision, pages 2564-
[49] Max Welling. Herding dynamical weights to learn. In Pro- ceedings of the 26th Annual International Conference on Muchine Learning, pages 1 1121-1128, 2009. 2
[50] Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. In Advances Neural Information Processing Systems, pages 5962-5972,
2571. Icce, 2011. 11
[35] David E Rumelhart, Geoffrey E Hinton, and Ronald Williams. Learning representations by back-propagating er- rors, nature, 3238))8
1986.
2018. 2
[36] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Raz- van Pascanu, and Raia Hadsell. Progressive neural networks.
[51] Yuc Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale in- cremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 374-382,
arXiv preprint /o..vv/006.0000 0806004671.00 2016. 2
[37] Paul-Ea Edouard Sarlin, Daniel DeTone. Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature match- ing with graph neural networks. In CVPR, 2020. 7, 12
2019. 2
[52] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2020. 1
[38] Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classifica- tion in network data. Al magazine, 29(3):93-99) 93, 2008, 5,
[53]
Ye Xiang, Ying Fu, Pan Ji. and Hua Huang. Incremental learn- ing using conditional adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 6619-6628, 2019. 2
[39] Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and Service Roboticx, 2017.
[54] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representa- tion learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, 2018. 2 [55] Allen Y Yang. Roozheh Jafari, S Shankar Sastry, and Ruzena Bajesy. Distributed recognition of human actions using wear- able motion sensor networks. Journal of Ambient Intelligence and Smart Environments. 1(2):003 115, 2009. 6
12
[40] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pages 2990-2999,
2017. 2
[41] Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. Incremental learning of object detectors without catastrophic forgetting. Proceedings of the IEEE International Confer-
[56] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Pro- ceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983,
ence on Computer Vision, pages 14000-3099, 2017. 2 [42] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adri- ana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representa- tions (ICLR), 2018. 2, 5,
2018. 2
[43] Chen Wang. Wenshan Wang, Yuheng Qiu, Yafei Hu, and Sebastian Scherer. Visual memorability for robotic inter-
[57] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph repre-
10

---

sentation learning with differentiable pooling. In Advances in neural information processing systems, pages 4800-4810, 2018. 2
at time as
1
t E M
[58] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna. GraphSAINT: Graph sampling based inductive learning method. International Conference on Learning Representations (ICLR), 2020. 1,
(n(()
M/n
1> M, t=
(10)
(n-11/n
t
M, t>
where M denotes the memory size.
[59] Fan Zhou and Chengtai Cao. Overcoming catastrophic for- getting in graph neural networks with experience replay. In AAAI Conference Artificial Intelligence (AAAI), 2021. 3 [60] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint 18/ 2018. 2
Proof. It obvious for t & M we only need keep all items in the continuum. For t > M, the probability that the n-th item is still kept in the memory at time / is
Pn. = (n(n)
Pn + 1) P. (t))..
M
n
1-2
1-1
(11)
n
n n+1
1-1 t
A. Dataset Details
M
t
We perform comprehensive experiments on popular graph datasets including citation graph Cora, Citeseer, Pubmed, and ogbn-arXiv [28, 29, 38]. Their statistics are listed in Table 8. Note that the dataset split a little different from their original settings,
This means the probability P.. is irrelevant to n and all items in the continuum share the same probability. In prac- tice, we always keep M items and sample balanced items accross classes.
can only test the final optimized model due to the requirement of lifelong learning, thus we don't tt need a validation set. For Cora, Citeseer, and Pubmed, the model consists of two feature broadcast layers (7) with C()) = 1 and C(2) = 2 channels each. For obgn-arXiv, we found that node features can be easily smoothed due to multiple feature propagation, hence we use the feature transform layers to concatenate its input features. We take the one-hot vector as the target vector z,, adopt the cross- entropy loss, and use the softsign [11] function (xx) */(++(=)) as the non-linear activation.
C. Distributed Human Action Recognition
Implementation In practice, the temporal growing graph only be learned sequentially, thus take the first 80% of each sequence for training and the remaining 20% for testing. Specifically, we define the radius of neighborhood as the temporal distance. Therefore, all the nodes at the same instant are 1-hop neighbors of each other. For each feature graph we have K in the continuum (1). We construct FGN using two feature transform layers (8) with attention weights (9) and one fully connected layer to predict for the sub-graph classification. For fairness, we use C(1) 5. F(1) = 25, C(2) = 32, and F(2) = 12 for all models. In the experiments, we find that GCN, APPNP obtain the best overall performance using the SGD optimizer, while MLP, GAT, and FGN performed the best using the Adam optimizer. Running time We also report the average running time for the models in Table 9. Note that the efficiency of FGN is on par with other methods. Considering that FGN has a much better performance, we believe that FGN is more promising.
Table 8. The statisties of the datasets used for lifelong learning.
Dataset
Nodes
Edges
Classes
Features Labels
Cora
2,708
5,429
7
1,433
0.421
Citeseer
3,327
4,732
6
3,703
0.337
Pubmed
19,717
44,338
3
500
0.054
obgn-arXiv
169,343
1,166,243
40
128
0.5
B. Proof of Sequence Invariant Sampling
Table 9. Running time comparison on the action recognition.
Let P be the probability that the observed items are se- lected at time f, thus the probability that one item is still kept in the memory after k selection is p This explains that earlier items have lower probability to be kept in the memory and this phenomenon was reported the Section 4.2 of [2]. To compensate for such effect and
MLP GCN APPNP GAT FGN
Runtime (ms)
3.51
5.38
5.36
5.48
5.76
D. Image Feature Matching
Proposition B.1. To ensure that all items in the continuum have the same probability to be kept in the memory at any time f, we can set the probability that the n-th item is selected
Although many hand-crafted feature descriptors such SIFT [27] and ORB [34] have been proposed decades ago, their performance is still unsatisfied for large view

---

point changes. Due to the well generalization ability, deep learning-based feature detectors have received increasing attentions. For example, SuperPoint [9] introduced a self- supervised framework for extracting interest point detectors and descriptors. SuperGlue [37] introduced graph attention model into SuperPoint for feature matching.
verse motion patterns. In addition, the dataset also includes other traditionally challenging factors in SI.AM tasks such as moving objects, changing lighting conditions, and extreme weather. We randomly select 80% the sequences for train- ing and take the remaining for testing. We recommend the readers refer [47] for more details of the dataset.
Implementation In the experiments, we adopt C = 1, K 1, = 256 in both FGN and FGN for fairess.. The training loss function is adapted from [30] which maximizes the likelihood predicting similar node embeddings cor- responding to their spatial location. We recommend the readers refer SuperPoint [9], SuperGlue [37], and [30] for more details of the loss functions.
E. Limitation
Although we have shown that FGN outperformed the SOTA methods in node classification, sub-graph classifi- cation, and edge prediction, it also has several limitations. First, our current implementation is not vectorized for tak- ing a varying number of neighbors, which is less computa- tionally efficient. Second, in the experiments, we assumed scalar edge weights, while in the general case, the graph edge weights represented by vector, defined in the fea- ture graphs. Third, since our main contribution is the novel graph topology, i.e., feature graph, we mainly compared it with the SOTA graph models such as GAT by applying an off-the-shelf lifelong learning algorithm. However, might also be applicable to other lifelong learning algorithms. In the future, plan fully optimize the codes, extend it applications with vector edge weights, and apply more lifelong learning algorithms.
Dataset We perform training and evaluation the Tar- tanAir dataset [47]. TartanAir is large (about 3TB) and very challenging visual SLAM dataset consisting of binoc- ular RGB-D video sequences together with additional per- frame information such as camera poses, optical flow, and semantic annotations. The sequences are rendered in AirSim [39], a photo-realistic simulator, which features modeled environments with various themes including urban, rural, nature, domestic, public, sci-fi, etc. Figure contains sev- eral example video frames from TartanAir. The dataset is collected such that it covers challenging viewpoints and di-
12


Reviews:
[{"review_id": "iu7XpLObDGp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The continuous learning on the graph neural network is restricted by the mechanism of the graph neural network itself. In order to enable the continuous learning approaches to be directly applied to GNN, the authors propose Feature Graph, which converts the node classification problem into graph classification by converting nodes into graphs. Therefore, we can apply current lifelong learning techniques to GNNs. Experimental results show that the proposed method overperforms the baselines. The main contribution of the paper is the idea of the converting technique, which makes it possible to employ existing methods to deal with the problem of continuous learning on the graph neural network. Overall, the paper is well written and is easy to follow.\n\nConcerns:\n1. Using the feature's cross-correlation matrix as the adjacency matrix of the neural network on the node sounds reasonable, but the validity of this idea lacks experimental explanation. In addition, how much do features contain the structure information of a graph? It is not discussed in this paper.\n\n2. The dynamic increase of nodes is the main problem that limits the application of continuous learning approaches in graph neural networks. This problem still exists in Feature Graph. The author should discuss it.\n\n3. The usage of task descriptors should be clarified, since their different usages may have a great impact on difficulty of problem\n\n4. The experiments are not solid enough. Although Table 2 & 3 show positive result supporting proposed method, the experimental results are too brief to be convincing. Other results such as accuracy curves, or other baselines such as joint training should be provided. In addition, Cora, Citeseer and Pubmed are small graph datasets. Experiments on big graph datasets should be conducted.\n\nMinor comments:\n1. In Formula (9), the parentheses are redundant.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "g62uJnOJlmI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to solve the problem of lifelong graph learning. Thus far, the topic about graph learning and lifelong learning is still underexplored. This paper proposes a new graph topology based on feature interaction, which takes the features as nodes and turns the nodes into graphs, and thus formulates a regular lifelong learning problem by defining the feature graph continuum. The authors conduct experiments on three popular citation graph datasets including Cora, Citeseer, and Pubmed.\n\nPros:\n1. This paper presents a novel strategy to transform the regular graph to a feature graph. This converts the original problem of node classification to graph classification, where the increasing nodes are turned into training samples. It makes the graph learning applicable to the continual learning.\n\nCons:\n1. In Section 1, the authors mention that--“It takes the features as nodes and turns the nodes into graphs. This converts the problem of node classification to graph classification. In this way, the increasing nodes become training samples”. In the lifelong learning, this strategy will not only increase the node samples but also the edges between new and old nodes. Although the feature graph continuum and random sample rehearsal strategy are proposed, scalability might still be a concern.\n\n2. It is unclear whether the proposed feature graph and feature adjacency matrix could effectively capture useful information from neighbors. More justifications and theoretical analysis shall be provided. In addition, evaluating the performance of feature graph in some traditional graph learning tasks would be helpful. \n\n3. In the experiments, the authors only compare their method with the modified GraphSAGE method. Although this topic about the combination of graph learning and continual learning is relatively new, there exists several relevant papers such as Continual Graph Learning (March, 2020). It would be more convincing if the authors could conduct comparisons with other graph continual learning or lifelong learning methods. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RWKU9tysxIp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:  \nThis paper aims to bridge GNNs with life-long learning so that the catastrophic forgetting problem in graph-structured tasks is alleviated. Specifically, the major contribution seems to be transforming the original graph into a feature graph so that the node classification problem is transferred into a graph classification problem with isolated samples. Meanwhile, feature interactions are modeled in constructing edges of the feature graph. Experiments on three citation graphs demonstrate the effectiveness of the proposed method. \n\nPros:   \n(+) GNNs + lifelong learning seems to be a novel problem that has been seldomly studied.  \n(+) The proposed framework can be applied to different GNNs.   \n(+) The paper is overall well written and easy to follow.  \n\nNegative points are as follows:   \n(1) Converting the original graph into a feature graph is not adequately justified and may have severe drawbacks. The main motivation for such a conversion is to transform connected nodes in the original graph into isolated samples so that the existing lifelong learning methods can be applied. However, the whole point of using GNNs is to pass and exchange messages between different nodes. If each node is regarded as an isolated sample, the relationships between nodes are completed ignored (except the feature co-occurrence statistics). In other words, the resulted model is essentially feature-centric and basically does not preserve any structural information in the original graph. For example, it is highly likely that the proposed model cannot preserve motifs or structural roles of nodes, nor to handle structural-driven tasks such as link prediction. The authors need to further clarify this major model design.  \n(2) Following how to convert nodes into isolated samples in GNNs, a well-known method is to regard each node as an ego-graph, since the representation of a node in GNNs only depends on its k-hop neighbors (see GraphSAGE and [1-2]). Such a conversion will also transform the node classification problem into a graph classification problem, similar to the paper’s arguments, but without losing structural information. Thus I am wondering whether or why such a method cannot be directly applied in the lifelong setting (from Section 3, the k-hop neighbors’ information should be available).  \n(3) In experiments, the authors adopt three citations graphs. Though I acknowledge they were commonly used as benchmarks in GNNs, recent studies suggest these small datasets may be not adequate in comparing different methods [3-5]. Thus, more experiments on larger datasets may be needed. (The experimental results on the Flickr dataset in the Appendix are puzzling since the results show that not using memory outperforms using memory, indicating that graph lifelong learning may even not be a proper setting.)   \n(4) There are also a few missing related works [6-7].  \n\nMinor:  \n(1)\tIn related works, JK-Net and DiffPool are GNN architectures (proposing jumping connections and differentiable pooling) rather than new sampling techniques.  \n(2)\tIt should be noted that building feature graphs and considering feature interactions in GNNs have also been studied recently, see [8-10]. But since they are informal publications or very recent w.r.t. the submission, I only suggest the authors compare them in an updated version and do not consider this as a negative point.  \n\n[1] Link Prediction Based on Graph Neural Networks, NeurIPS 2018  \n[2] Graph Meta Learning via Local Subgraphs, arXiv:2006.07889  \n[3] Pitfalls of Graph Neural Network Evaluation, arXiv:1811.05868  \n[4] Open Graph Benchmark: Datasets for Machine Learning on Graphs, arXiv: 2005.00687.  \n[5] Benchmarking Graph Neural Networks, arXiv: 2003.00982  \n[6] Lifelong representation learning in dynamic attributed networks, Neurocomputing 2019.   \n[7] Streaming Graph Neural Networks via Continual Learning, CIKM 2019.  \n[8] Cross-GCN: Enhancing Graph Convolutional Network with k-Order Feature Interactions, arXiv:2003.02587.  \n[9] CatGCN: Graph Convolutional Networks with Categorical Node Features, arXiv:2009.05303.   \n[10] AM-GCN: Adaptive Multi-channel Graph Convolutional Networks, KDD 2020.  \n\nBased on the above comments, I am currently leaning towards rejection. I am happy to improve my scores if the authors can further justify their proposed method.   \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mm7CjYzVxsG", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to extend GCN to the lifelong learning setting. The idea is to transform the nodes of a graph into feature graphs where each feature is a node and the edges represent feature correlations estimated from the K-hop neighborhood of a node in the original graph.\n\n[Pros] The idea of converting nodes of a graph into feature graphs is novel and interesting (as far as I know). In this way, the model depends on feature graphs rather than the original graph which continually grows in the online learning setting. This breaks down the original graph into individual nodes and their contexts which are then fed to the model to generate results. Very interesting.\n\n[Cons] However, some major issues still hinder the paper from publication: \n(1) There is a lack of necessary details. First, the constructed adjacency matrix $A^F_{k,c}$ (Eq.(5)) has two subscripts $k$ and $c$. But the subscript $c$ just disappears in Eq.(9) without any explanation. Therefore, it is not known how the final adjacency matrix is built in Eq.(9) from Eq.(5). Second, how do you set the edge wights of the original graph? Moreover, at first the problem formulation says each edge is associated with a weight vector, but it seems finally it is just a scalar. This is a little confusing. The above issues affect the reproducibility of this work. \n(2) The notations are hard to follow. For example, in the definition of the graph lifelong learning, the weight vector set $\\mathcal{W}$ is said to contain weight vectors associated with the $K$-hop (Strictly speaking, I think here it should be $K$ rather than $k$. More about this issue below) neighbors. Does this mean these weights are associated with nodes rather than edges? According to the notation of $\\mathcal{W}$ ($k=1:K$), I think $\\mathcal{W}$ contains neighbors up to $K$-hop rather than only the $K$-th hop. The use of $k$ and $K$ is very confusing, which makes related content hard to understand. Another question is, how do you set the parameter $K$? The last sentence said “a maximum $k=1$ is required...”. I am not sure whether this is how $K$ is set. Furthermore, what do you mean by maximum? Is it possible that $K=0$? The last question is, how do you set the target vector $z$ for the training data? I cannot find the related content.\n(3) What is the motivation of the design of the feature transform layer? Why do we need to change the number of feature nodes? The motivation is not clear. Furthermore, there is no comparison to using broadcast layers in experiments. In the appendix F, why do you use the Flickr dataset for evaluating transform layers? Why not use this dataset in the main experiments?\n\nSome minor issues:\n(1) The writing needs some improvement. Some language errors include “an importance”, two sentences in a sentence.\n(2) What is the motivation of using sgnroot in Eq.(5)?\n(3) Traditional methods also exploit the interactions between features in a neighborhood, by concatenating feature vector of the target node and the aggregated neighborhood feature vector and do affine transformation. Hence, I think “existing methods cannot model feature interactions well” is not precise. Besides, “useful information might be encoded in one’s neighbor features” is vague. Some concrete examples are needed to motivate the modeling of feature interactions in node neighborhoods.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers initially assessed this paper as slightly below the acceptance threshold. The reviewers seem to agree on the novelty and potential impact of this project, but they also highlighted the lack of clarity of the manuscript including lack of clarity in the method used to encode the graph data. 

As the authors noted, graph-related questions were the focus of most of the comments and questions from the reviewers. This is not because the reviewers did not understand and assess the method from the continual-learning side (I am also meta-reviewing several continual-learning papers and I believe that I can assess the novelty of this work). As I wrote above, reviewers were convinced of the paper's motivation. 

The authors provided good responses and discussed with at least one reviewer thoroughly. These interactions seem to have clarified important aspects of your proposed methodology and notably the properties of your graph-construction method. I found that your new results on larger datasets also provide an improvement. However, to be properly assessed, this number of clarifications regarding the core method requires a new round of reviews. The discussions have also highlighted some of the limits of your approach which do not seem to be acknowledged in your paper. This includes the discussion with reviewer2 regarding constraints on L & K, node classification (also I find that one to less important), and comparison to GraphSage on the non-lifelong learning scenario.

Overall, and while I agree that continual learning from graph data is an important and unexplored problem, I also find that the current manuscript lacks clarity and, even though the ICLR discussion allowed reviewers to discuss these with the authors, there are still significant ways to improve the clarity of the current manuscript. As a result, I do not recommend acceptance of the current manuscript. 

I strongly suggest the authors keep on working on their manuscript as their idea seems to have potential and I would imagine that it may become one of the first works in a new interesting line of research.
Decision: Reject