Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a methodology to perform classification on text-attributed graphs. This is an important problem that arises in many practical applications where the nodes of the graph are associated with some textual information. The paper studies a methodology that combines pre-trained language models (LM) and GNNs. The proposed framework, called GLEM, is trained using a variational EM algorithm. Specifically, fixing the GNN model which characterizes the global label distribution, the parameters of the LM are updated in the E-step. Then, in the M-step, the parameters of the GNN are optimized. The paper puts emphasis on the choice of the likelihood function to deal with scalability constraints. The proposed methodology is evaluated on several node classification tasks.
strength_and_weaknesses: **Strengths:**
* Interesting formulation of GLEM. Jointly optimizing the LM and the GNN within a variational EM algorithm constitutes an elegant formulation of the problem.

* The paper has performed experiments on large-scale datasets.

**Weaknesses:**
* The paper has mostly examined datasets that contain textual features. However, similar settings arise while dealing with the task of text classification using graph-based models. This problem, although it is highly relevant to the formulation studied here, it is not discussed at all in the paper. In the past, several approaches have been introduced for graph-based text classification —  none of them is mentioned in the paper. For instance, one of the very first papers for this task “Graph Convolutional Networks for Text Classification” at AAAI ’19, follows a similar formulation where a graph composed of document nodes is constructed. Is there any specific reason why such methodologies have not been used?

* Despite targeting scalability, the paper does not discuss the convergence of the proposed model. The training step is quite complex, therefore studying the convergence of the model is important.

* The settings of structure-free inductive learning are not very clearly presented. Since GLEM assumes that there is a graph structure that captures the relationships among text nodes, it is not straightforward to extend it to an inductive setting. How is this problem tackled here?

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written, and most of the different concepts are clearly presented. At the same time, the paper keeps a good balance between theoretical contributions and empirical evaluation.  

Below, I list a few other points that could improve the clarity and presentation of the paper:
* In the presentation of GNNs for node classification in Sec. 3, it is not clear what are the initial node feature vectors. This becomes more clear afterward but would be helpful to clarify it here as well.
* In the experiments, two variants of GLEM, namely GLEM-GNN and GLEM-LM are used. These models have not been formally defined. I assume each one of them corresponds to the predictions made by the GNN and LM part respectively of the overall pipeline. 

summary_of_the_review: Overall, I believe it’s an interesting formulation of the node classification task on text-attributed nodes. Nevertheless, I still have some concerns about the absence of baseline models. This point needs further clarification from the authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose the node representation method GLEM for Text-Attributed Graphs (TAG) based on a pseudo-likelihood variational framework.

Specifically, they alternatively update GNN (M-step) and LM (E-step) with pseudo-labels from each other.

In experiments, they validate their method achieves SOTA on large-scale TAG while maintaining a high level of efficiency.
strength_and_weaknesses: ### Strengths

- **Simple but powerful idea;** the authors prove that their method outperforms previous baselines on three benchmark datasets. This work has enough impact on the field of representation learning on TAG.

### Weaknesses

- **Lack of ablation studies;** In E-step and M-step, there are hyperparameters that balance the weight of two terms. However, I fail to find any design choice or ablation study on this. In addition, more details (optimizer, learning rate, batch size, …) should be described in the paper (at least appendix) to fully reproduce the methods and experiments in this work.

### Suggestions

- Questions
    
    - What is the difference between GLEM-LM and GLEM-GNN? Do they differ based on the starting point of optimization (E-step or M-step first)? 
    
- Typos & suggestions
    - Table 3: Dataset name (ogbg-papers) is missing in Table 3.
    - Table 4: In Arxiv-MLP experiments, the boldface on diff is wrong. (on -1.67 instead of -1.58
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The overall method and experimental setup are clearly written. More clarifications on implementation details are still needed.

### Quality
The quality of the paper is good.

### Novelty
The method is novel enough.

### Reproducibility
I cannot find the reproducibility statement in the main manuscript. The authors do not provide any codes or software to reproduce their results. 
More clarifications on implementation details are needed for reproducibility.
summary_of_the_review: To my knowledge, this paper has sufficient merits to be accepted.

For reproducibility, there should be more information provided about implementation and experimental settings.

Please be aware that since I am unfamiliar with the works on representation learning of TAGs, I may have missed significant prior works.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a variational expectation maximization framework to jointly train the language model and the graph neural network for representation learning for text-attributed graphs. The approach is clearly motivated, mathematically principled, and empirically effective and efficient. I believe this paper makes a clear contribution to graph representation learning and opens new possibilities for scaling graph representation learning to even larger graphs and language models. 
strength_and_weaknesses: **Strength**

**Clear motivation**: this paper is clearly motivated to target on the problem of joint learning LM and GNN efficiently. 

**Mathematically principled**: the variational EM framework is elegant and very smart. Although variational EM is very well studied and already used for learning GNNs, I think the application of variational EM in this paper is still novel. 

**Solid experiment**: the experiment is conducted well, with solid comparison with GNNs and LMs and joint methods.The improvements are also clearly marked in Table 2. 

**Efficiency and scalability**: The efficiency (quick enough training time) and scalability (setting larger batch size) is also clear in table 5. 

**Weakness**

**Multiple approximation happens in training:** the authors have clearly discussed all the places where approximation happens, which on itself is a good thing. Yet all these approximations (mean-field, weak-sleep, pseudo label, balancing parameters .etc) intuitively makes the final objective deviate very much from the original variational bound (or not sure if the final objective is a bound anymore). Although being orthodoxical to the original variational objective may not be practically good, it would still be mathematically more principled if the authors could establish more rigid analysis of the final objective (e.g., its relationship to the likelihood).
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and quite novel. I encourage the authors to open-source the code to improve its reproducibility and impact
summary_of_the_review: This paper proposes a variational expectation maximization framework to jointly train the language model and the graph neural network for representation learning for text-attributed graphs. The method novel and smart. Experiments demonstrate its effectiveness, efficiency and scalability. I believe this paper would make a valid contribution to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new method for node classification where nodes are associated with text attributes. The proposed framework consists of a language model (LM) and a graph neural network (GNN) and uses the variational EM algorithm that learns LM and GNN in the E and M steps respectively. The proposed method is evaluated on several large-scale graphs with text attributes in comparison with different LMs and GNNs used separately. 
strength_and_weaknesses: Strength:

1. The general idea of this paper is performing joint training for LM and GNN and letting them help each other during training. This might be done with heuristic approaches. But in this paper, it is interesting to wrap up the alternative training of LM and GNN into an EM framework, which adds credit to the technical depth of the paper.

2. The datasets and the baselines are well selected and the experiments are comprehensive in general including different settings and comprehensive ablation studies.

Weaknesses:


1. Some of the performance improvement of the proposed method is a bit marginal. For example, in Table 2, GLEM-GNN (the proposed method) is marginally better than the second best (X_{GIANT}) on arxiv and products (e.g., 75.90 ± 0.19 VS 76.74 ± 0.08 with RevGAT), not mentioning that GLEM-GNN did not outperform X_{GIANT} on papers.

2. Several clarity issues (please see comments in the next section).

clarity,_quality,_novelty_and_reproducibility: Clarity issues/questions:

1. More details of the implementation are needed, including the detailed architecture, learning rates, optimizers, ...

2. What's the proportion of the node with/without labels?

3. What's the model for the implementation of the right-hand-side term of Eq (7)?

4. What's the difference between GAMLP and GAMLP+?

Minor clarity issues:

1. SAGN+ in Table 2 is not defined.


2. A typo in Eq (6)

3. In Table 1, the train/val/test proportions are 8/2/90?

Without the details of the implementation or the release of the code, I have concerns to the reproducibility of the paper.

summary_of_the_review: The variational EM formulation is interesting but the quality of the paper needs to be improved.

The author provided detailed information to my concerns of clarity. I've increased my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of learning representations for large-scale text-attributed graphs. The authors propose a novel variational EM algorithm incorporating textual and structural information in such graphs.

All reviewers agree this is a nice paper, with sound theoretical and empirical components, integrating the popular pre-trained language models and the graph neural networks. The execution of the work was also excellent, with comprehensive experiments and analyses. The paper’s quality is high.

There are some concerns about the experiments (baselines, settings, ablations, etc.), clarity, and reproducibility. The authors have addressed them well in the rebuttals.
justification_for_why_not_higher_score: N/A.
justification_for_why_not_lower_score: Among all the papers under me, this is the best paper. The paper also addresses an important issue with a nice solution (involving both PLM and GNN). This paper deserves to be a highlight at the conference.
Decision: Accept: notable-top-5%