Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: For representation learning on textual graphs, Cascaded Transformers-GNN architectures have been used, as language models and GNN are independently and consecutively deployed. The authors' claims, however, this approach has a limitation of information fusion of text encoding and graph structures. The authors proposed GNN-nested Transformers named as GraphFormers, where GNN components are nested alongside the transformer layers of language models. Also, the author proposed a progressive learning strategy and unidirectional graph attention to effectively train the graph network modules. The efficacy of the GraphFormers is demonstrated on 3 textual graph datasets (DBLP, Wiki, and Product Ads datasets), achieving SOTA performance.
main_review: 1. Originality
    1. The problem this paper tackle is interesting in which utilizing the neighborhood information during text encoding, and the proposing approach that inserting GNN layers into the transformer layers of PLM seems novel.
    2. However, the nested GNN module's functionality is utilizing the related information of neighborhood nodes to learn the center node's representation using multi-headed attention layers. My main concern is that the proposed model should be theoretically and experimentally explained the differences to feeding all concatenated texts (i.e., $[g_1; g_2; ... ; g_n] , g_i \in G_x$)  into transformer-based PLMs (especially large scale PLMs such as GPT-x, T5, etc) as an input. The concatenated text input approach may have advantages in the sense of more flexible and rich attention representation.
2. Quality
    1. This paper's claims are technically sound, and experimental results support the efficacy of GraphTransformers.
    2. The proposed uni-directional graph attention for model simplification and two-stage progressive learning methods are well-studied through the ablation studies.
3. Clarity
    1. Overall, the paper is well-organized and well-written.
    2. In figure 2, the asymmetric MHA is not correctly represented. From the figure, readers can misunderstand as symmetric MHA of $\hat{H_g^l}$). Furthermore, in Algorithm 1 (in Appendix), Transformers[l]( Hg_hat[l], asymmetric = True) is not understandable. According to eq.4, the MHA^asy requires two different inputs.
4. Significance
    1. Lack of baseline of recent previous work (TwinBERT, AdsGNN, TextGNN, MetaBERT)
    2. Lack of comparison with other PLMs. 
        - What is 'BERT-like PLM' exactly?
        - Lack of leveraging recent large-scale PLMs such as T5 and GPT-x.
    3. Lack of **comparison with feeding concatenated text inputs into large scale PLMs**
    4. The Product Ads dataset is not published so that it is hard to reproduce the results. Since the dataset is new, the details such as how the dataset is collected, categories of the items, etc. should be explained. 
    5. Lack of qualitative analysis.
    6. From table 4, there was little performance decrease when the learnable position bias is disabled, which concluding the bias is unnecessary.
limitations_and_societal_impact: The authors addressed the limitations of this work in terms of the problem scopes. From the author's checklist, I could find the Product Graph dataset is a private industrial dataset. I would like to recommend denote this and that the dataset will not be published due to the privacy issue in the main manuscript.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes GraphFormer, an architecture built on top of GAT and Transformers for textual graphs. The motivation behind the architecture is that the existing techniques on textual graphs involve a cascade process where the nodes are first encoded using a language model after which a GNN is applied. With GraphFormers, the graph aggregation and encoding can happen iteratively, allowing for better contextualization of nodes with text like “notes on transformers” since transformers can be interpreted as ML models or electrical devices depending on the surrounding context.

The authors also lay out a 2-step training procedure whereby they first train on a version of the graph with injected noise via link prediction before training on the actual graph. The motivation behind this is given as the fact that training on the graph with injected noise makes the GNN component of the GraphFormer model more robust without which the transformer part will dominate.
main_review: Overall, this paper is a good one.  I like the motivation presented by the authors for their work. Indeed the cascade process for encoding textual graphs is not optimal, and the community as a whole will benefit from having a more concerted approach.

Strengths of the paper:
* The presented architecture is intuitive to understand where GNN and transformer are iteratively applied in the neighborhood aggregation process to allow for text at a node to be encoded taking into account the graph context.
* Several experiments are performed to show the superiority of GraphFormer over cascaded approaches in link prediction task on three benchmark datasets.
* Authors utilize caching in a succinct manner to decrease the recomputation needed for encoding different nodes in the graph; the caching works by preventing nodes that are neighbors of a central node to aggregate from the neighbors.

Weaknesses of the paper:
* The organization of the paper and the way in which the equations for GraphFormer are presented make the paper a bit hard to follow.
* There are other works on making GNNs and aggregations within them scalable via caching (e.g., https://arxiv.org/pdf/2001.07524.pdf) but the authors have not made use of that, instead reinventing the wheel a little.
limitations_and_societal_impact: The authors do clearly call out the limitation of the work in that GraphFormers are only applicable to textual graphs and does not take edge information like edge type or edge features into account. Therefore, they can’t be applied to heterogenous graphs.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a means to encode text feature for nodes in textual graphs, where a layerwise aggregation module (implemented as a multi-head self-attention) is appended after each Transformer encoder block that aggregates the hidden representations of (sampled) neighbor nodes. The produced contextual representations are dispatched to each node for the computation of next Transformer encoder block. To mitigate the possible model degradation issue that the aggregation module might be insufficiently trained, the authors also propose a two-stage training schedule where the model is first trained on nodes with corrupted inputs until convergence and then trained on normal data.
main_review: This paper proposes a novel text feature extraction method for textual graphs where intermediate representations of each node are contextualized using representations from neighbor nodes. It also proposes a two-stage training schedule to balance the capacity of the pre-trained text encoder and the aggregation modules.

The motivation that node representations should be contextualized in order to distinguish between nodes with similar texts but very different semantics is interesting. And the main method in this paper is clearly represented overall. However, there are still some concerns that I'd prefer the authors to address:

- Position bias: In Eq. (1),  a learnable position bias is added to the unnormalized attention score where three types of bias (c2n,c2c,n2n) are utilized. What is the purpose of the bias term? It doesn't seem important to distinguish between the center node and the neighbor nodes. The ablation result seems to be insignificant and is not elaborated. 
- On data settings: In DBLP citation graph, the text feature is chosen to be the title of each paper, which I suspect might be not informative enough to encode the semantics of the paper. This issue is especially crucial when the titles are short. I was wondering if the authors have considered using abstracts as well.
- Performance on Wiki: In line 274, the authors provide justification for the inferior performance of PLM+GAT on Wiki (see Table 2) from the heterogeneity of Wiki graph and that the incorporation of neighbor nodes might not contribute to the link prediction task. This justification seems at odds with the motivation and the performance of the proposed model (i.e., that the neighbor nodes can serve as additional information in generating the node representations).

Overall I think the paper is built upon good motivations, but some aspects of the empirical results are not explained or not convincing enough. 

( Several minor notation issues and typos:

- In Eq. (1), a multi-head self-attention is used to aggregate the information between nodes, in which a learnable position bias $\mathbf{b}$ is added to the attention score. I believe the notation should be $\mathbf{B}$ by convention.
- Other typos: line 188: leverage (what?) of; line 257: varnishing -> vanishing; line 267: be attribute(d) to

)


limitations_and_societal_impact: In the last section, the authors list some of the limitations of the proposed method: 1. it doesn't take the edge information into account; 2. It is specifically designed for textual graphs.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes GraphFormers for text graph tasks where GNNs are nested in each layer of the transformer-based pretrained language model. In this way, each node can appropriate graph structure to build semantic representations more effectively. Two-stage progressive training and unidirectional graph aggregation are further proposed to improve representation quality and avoid unnecessary computation. They use three real-world million-scale textual graph datasets (DBMLP, Wikidata, Product Graph) and demonstrate consistent improvements compared with traditional cascaded transformers-GNN models.
main_review: Originality: The author first identify the weakness of the "cascaded transformers-GNN" architecture: the text encoding and graph aggregation are in two consecutive with no information exchanged in text embeddings. Based on this insight, the proposed approach differs from the traditional "cascaded transformers-GNN" by interleaving transformers and GNNs. They have adequately discussed a comprehensive set of related work in pretrained LM and GNNs.

Quality: While this modification is simple, it is reasonable to nest GNNs into transformers to better encode information incorporating graph structures. The example in line 37-40 is helpful to illustrate the usefulness. The early usage and repetitive usage of graph structure is the main advantage of GraphFormers versus traditional manners. The added nested GNN operation is also small compared to transformers as discussed algorithmically and empirically.

In experiments, through comparison with other SOTA models and different variants of PLM+GNNs, the advantages of GraphFormers are supported and verified. 

However, the authors should clarify two things:
1. Clearly define the center nodes against the neighbour nodes. Which node is defined as center nodes in each tasks? and how this will affect the results?
2. In each layer of GNN, is the parameter shared? In other words, do you make sure the number of parameters is the same between GNN-nested transformers and Cascaded Transformers-GNN?

Clarify: The paper is well written and easy to follow. The technical components are documented in detail with equations and visualizations. However, the center node should be clearly defined.

Significance: In general, I believe their results of this new technique are solid. While the technique is simple and straightforward, comprehensive results have been presented to valid its effectiveness and efficiency. The proposed method can potentially change how other researchers use PLM and GNNs in a wide range of NLP tasks.

limitations_and_societal_impact: The authors have acknowledged the limitations of their current work including (1) not including edge information, (2) only used for textual graphs. This opens up further directions.

I don't see any potential negative societal impact.
review_title: null

Meta-review: This paper proposes a new model for learning representations on a textual graph. The key idea is to combine Transformer and graph neural network. Experimental results show that the proposed approach works better than the traditional cascade approach.

Strength
* The paper is generally clearly written, although there is still room for improvements on the presentation.
* The proposed model appears to be reasonable and technically sound.
* Experimental results have demonstrated the effectiveness of the proposed method.
* The reviewers pointed out some issues with the paper. The authors have successfully addressed most of them, particularly they provide new experimental results.

Weakness
* There are details that are not explained very clearly. The authors are encouraged to further improve the presentation.
* There are still problems with the English of the paper. For example, “graph aggregation and text encoding are iterative performed” --> it should be "iteratively". The authors are encouraged to do further proofreading. Please significantly revise the paper based on your replies in the rebuttal if the paper is accepted.

Minor comments: It is not clear for non-experts what the downstream tasks are from the explanation in the abstract and introduction. Please add that.


Decision: Accept (Poster)