Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims to solve the problem of lifelong graph learning. Thus far, the topic about graph learning and lifelong learning is still underexplored. This paper proposes a new graph topology based on feature interaction, which takes the features as nodes and turns the nodes into graphs, and thus formulates a regular lifelong learning problem by defining the feature graph continuum. The authors conduct experiments on three popular citation graph datasets including Cora, Citeseer, and Pubmed.

Pros:
1. This paper presents a novel strategy to transform the regular graph to a feature graph. This converts the original problem of node classification to graph classification, where the increasing nodes are turned into training samples. It makes the graph learning applicable to the continual learning.

Cons:
1. In Section 1, the authors mention that--“It takes the features as nodes and turns the nodes into graphs. This converts the problem of node classification to graph classification. In this way, the increasing nodes become training samples”. In the lifelong learning, this strategy will not only increase the node samples but also the edges between new and old nodes. Although the feature graph continuum and random sample rehearsal strategy are proposed, scalability might still be a concern.

2. It is unclear whether the proposed feature graph and feature adjacency matrix could effectively capture useful information from neighbors. More justifications and theoretical analysis shall be provided. In addition, evaluating the performance of feature graph in some traditional graph learning tasks would be helpful. 

3. In the experiments, the authors only compare their method with the modified GraphSAGE method. Although this topic about the combination of graph learning and continual learning is relatively new, there exists several relevant papers such as Continual Graph Learning (March, 2020). It would be more convincing if the authors could conduct comparisons with other graph continual learning or lifelong learning methods. 

review_title: Interesting idea but theoretical justification and experiments are not convincing 

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:  
This paper aims to bridge GNNs with life-long learning so that the catastrophic forgetting problem in graph-structured tasks is alleviated. Specifically, the major contribution seems to be transforming the original graph into a feature graph so that the node classification problem is transferred into a graph classification problem with isolated samples. Meanwhile, feature interactions are modeled in constructing edges of the feature graph. Experiments on three citation graphs demonstrate the effectiveness of the proposed method. 

Pros:   
(+) GNNs + lifelong learning seems to be a novel problem that has been seldomly studied.  
(+) The proposed framework can be applied to different GNNs.   
(+) The paper is overall well written and easy to follow.  

Negative points are as follows:   
(1) Converting the original graph into a feature graph is not adequately justified and may have severe drawbacks. The main motivation for such a conversion is to transform connected nodes in the original graph into isolated samples so that the existing lifelong learning methods can be applied. However, the whole point of using GNNs is to pass and exchange messages between different nodes. If each node is regarded as an isolated sample, the relationships between nodes are completed ignored (except the feature co-occurrence statistics). In other words, the resulted model is essentially feature-centric and basically does not preserve any structural information in the original graph. For example, it is highly likely that the proposed model cannot preserve motifs or structural roles of nodes, nor to handle structural-driven tasks such as link prediction. The authors need to further clarify this major model design.  
(2) Following how to convert nodes into isolated samples in GNNs, a well-known method is to regard each node as an ego-graph, since the representation of a node in GNNs only depends on its k-hop neighbors (see GraphSAGE and [1-2]). Such a conversion will also transform the node classification problem into a graph classification problem, similar to the paper’s arguments, but without losing structural information. Thus I am wondering whether or why such a method cannot be directly applied in the lifelong setting (from Section 3, the k-hop neighbors’ information should be available).  
(3) In experiments, the authors adopt three citations graphs. Though I acknowledge they were commonly used as benchmarks in GNNs, recent studies suggest these small datasets may be not adequate in comparing different methods [3-5]. Thus, more experiments on larger datasets may be needed. (The experimental results on the Flickr dataset in the Appendix are puzzling since the results show that not using memory outperforms using memory, indicating that graph lifelong learning may even not be a proper setting.)   
(4) There are also a few missing related works [6-7].  

Minor:  
(1)	In related works, JK-Net and DiffPool are GNN architectures (proposing jumping connections and differentiable pooling) rather than new sampling techniques.  
(2)	It should be noted that building feature graphs and considering feature interactions in GNNs have also been studied recently, see [8-10]. But since they are informal publications or very recent w.r.t. the submission, I only suggest the authors compare them in an updated version and do not consider this as a negative point.  

[1] Link Prediction Based on Graph Neural Networks, NeurIPS 2018  
[2] Graph Meta Learning via Local Subgraphs, arXiv:2006.07889  
[3] Pitfalls of Graph Neural Network Evaluation, arXiv:1811.05868  
[4] Open Graph Benchmark: Datasets for Machine Learning on Graphs, arXiv: 2005.00687.  
[5] Benchmarking Graph Neural Networks, arXiv: 2003.00982  
[6] Lifelong representation learning in dynamic attributed networks, Neurocomputing 2019.   
[7] Streaming Graph Neural Networks via Continual Learning, CIKM 2019.  
[8] Cross-GCN: Enhancing Graph Convolutional Network with k-Order Feature Interactions, arXiv:2003.02587.  
[9] CatGCN: Graph Convolutional Networks with Categorical Node Features, arXiv:2009.05303.   
[10] AM-GCN: Adaptive Multi-channel Graph Convolutional Networks, KDD 2020.  

Based on the above comments, I am currently leaning towards rejection. I am happy to improve my scores if the authors can further justify their proposed method.   

review_title: Interesting problem, but some model designs need further clarifications

final_score: rating: 6: Marginally above acceptance threshold
review_content: The continuous learning on the graph neural network is restricted by the mechanism of the graph neural network itself. In order to enable the continuous learning approaches to be directly applied to GNN, the authors propose Feature Graph, which converts the node classification problem into graph classification by converting nodes into graphs. Therefore, we can apply current lifelong learning techniques to GNNs. Experimental results show that the proposed method overperforms the baselines. The main contribution of the paper is the idea of the converting technique, which makes it possible to employ existing methods to deal with the problem of continuous learning on the graph neural network. Overall, the paper is well written and is easy to follow.

Concerns:
1. Using the feature's cross-correlation matrix as the adjacency matrix of the neural network on the node sounds reasonable, but the validity of this idea lacks experimental explanation. In addition, how much do features contain the structure information of a graph? It is not discussed in this paper.

2. The dynamic increase of nodes is the main problem that limits the application of continuous learning approaches in graph neural networks. This problem still exists in Feature Graph. The author should discuss it.

3. The usage of task descriptors should be clarified, since their different usages may have a great impact on difficulty of problem

4. The experiments are not solid enough. Although Table 2 & 3 show positive result supporting proposed method, the experimental results are too brief to be convincing. Other results such as accuracy curves, or other baselines such as joint training should be provided. In addition, Cora, Citeseer and Pubmed are small graph datasets. Experiments on big graph datasets should be conducted.

Minor comments:
1. In Formula (9), the parentheses are redundant.

review_title: the paper presents a method by which the lifelong learning techniques developed for CNN can be applicable to GNN

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper aims to extend GCN to the lifelong learning setting. The idea is to transform the nodes of a graph into feature graphs where each feature is a node and the edges represent feature correlations estimated from the K-hop neighborhood of a node in the original graph.

[Pros] The idea of converting nodes of a graph into feature graphs is novel and interesting (as far as I know). In this way, the model depends on feature graphs rather than the original graph which continually grows in the online learning setting. This breaks down the original graph into individual nodes and their contexts which are then fed to the model to generate results. Very interesting.

[Cons] However, some major issues still hinder the paper from publication: 
(1) There is a lack of necessary details. First, the constructed adjacency matrix $A^F_{k,c}$ (Eq.(5)) has two subscripts $k$ and $c$. But the subscript $c$ just disappears in Eq.(9) without any explanation. Therefore, it is not known how the final adjacency matrix is built in Eq.(9) from Eq.(5). Second, how do you set the edge wights of the original graph? Moreover, at first the problem formulation says each edge is associated with a weight vector, but it seems finally it is just a scalar. This is a little confusing. The above issues affect the reproducibility of this work. 
(2) The notations are hard to follow. For example, in the definition of the graph lifelong learning, the weight vector set $\mathcal{W}$ is said to contain weight vectors associated with the $K$-hop (Strictly speaking, I think here it should be $K$ rather than $k$. More about this issue below) neighbors. Does this mean these weights are associated with nodes rather than edges? According to the notation of $\mathcal{W}$ ($k=1:K$), I think $\mathcal{W}$ contains neighbors up to $K$-hop rather than only the $K$-th hop. The use of $k$ and $K$ is very confusing, which makes related content hard to understand. Another question is, how do you set the parameter $K$? The last sentence said “a maximum $k=1$ is required...”. I am not sure whether this is how $K$ is set. Furthermore, what do you mean by maximum? Is it possible that $K=0$? The last question is, how do you set the target vector $z$ for the training data? I cannot find the related content.
(3) What is the motivation of the design of the feature transform layer? Why do we need to change the number of feature nodes? The motivation is not clear. Furthermore, there is no comparison to using broadcast layers in experiments. In the appendix F, why do you use the Flickr dataset for evaluating transform layers? Why not use this dataset in the main experiments?

Some minor issues:
(1) The writing needs some improvement. Some language errors include “an importance”, two sentences in a sentence.
(2) What is the motivation of using sgnroot in Eq.(5)?
(3) Traditional methods also exploit the interactions between features in a neighborhood, by concatenating feature vector of the target node and the aggregated neighborhood feature vector and do affine transformation. Hence, I think “existing methods cannot model feature interactions well” is not precise. Besides, “useful information might be encoded in one’s neighbor features” is vague. Some concrete examples are needed to motivate the modeling of feature interactions in node neighborhoods.

review_title: The idea of this work is novel, but the paper is not clear enough.

Meta-review: The reviewers initially assessed this paper as slightly below the acceptance threshold. The reviewers seem to agree on the novelty and potential impact of this project, but they also highlighted the lack of clarity of the manuscript including lack of clarity in the method used to encode the graph data. 

As the authors noted, graph-related questions were the focus of most of the comments and questions from the reviewers. This is not because the reviewers did not understand and assess the method from the continual-learning side (I am also meta-reviewing several continual-learning papers and I believe that I can assess the novelty of this work). As I wrote above, reviewers were convinced of the paper's motivation. 

The authors provided good responses and discussed with at least one reviewer thoroughly. These interactions seem to have clarified important aspects of your proposed methodology and notably the properties of your graph-construction method. I found that your new results on larger datasets also provide an improvement. However, to be properly assessed, this number of clarifications regarding the core method requires a new round of reviews. The discussions have also highlighted some of the limits of your approach which do not seem to be acknowledged in your paper. This includes the discussion with reviewer2 regarding constraints on L & K, node classification (also I find that one to less important), and comparison to GraphSage on the non-lifelong learning scenario.

Overall, and while I agree that continual learning from graph data is an important and unexplored problem, I also find that the current manuscript lacks clarity and, even though the ICLR discussion allowed reviewers to discuss these with the authors, there are still significant ways to improve the clarity of the current manuscript. As a result, I do not recommend acceptance of the current manuscript. 

I strongly suggest the authors keep on working on their manuscript as their idea seems to have potential and I would imagine that it may become one of the first works in a new interesting line of research.
Decision: Reject