Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose the method Edgeformers, which represents the text-attributed edges in a graph. 

They utilize the pre-trained language model (BERT) and the concept of virtual node to represent each text in the edge with regards to its adjacent neighboring edges.

Specifically, in Edgeformer-E, they represent each edge attribute with its text and node embedding from their adjacent nodes using a transformer layer.
In Edgeformer-N, the node embeddings are represented as the aggregation of its adjacent edges’ embeddings from Edgeformer-E.

In experiments, they demonstrate the proposed method outperforms previous baselines in both edge classification and link prediction tasks.
strength_and_weaknesses: ### Strengths

- **Important problem and valid approach;** This paper addresses an important problem in edge representation learning where the text (document) is provided as an edge attribute. By reflecting graph structure into edge representations using pre-trained language models (PLM), the proposed method successfully handles the given problem.

### Weaknesses

- **Recent baselines are missing;** Relformer [1] is one of the advanced version of Graphformer, which is specialized for link prediction. Otherwise, EHGNN [2] has a similar approach which represents edges based on adjacent nodes and edges with dual hypergraph transformation. In my opinion, such powerful recent algorithms on edge representation learning should be compared as baselines for the proposed method.
- **Hard to understand;** The suggested approach is challenging to comprehend. In order to comprehend the suggested method completely, I read the method section several times. Particularly, Figure 1 is hard to comprehend because there isn't a subscription for each component. Does the transformer layer of PLM have node information appended to it? If not, is Edgeformer another module that integrates the node information into the text representation after the PLM layers? It would be preferable, in my opinion, to incorporate some notations into Figure 1 or include additional, in-depth figures for each method.

[1] Bi et al., Relational Graph Transformer for Knowledge Graph Representations, WWW 2022

[2] Jo et al., Edge Representation Learning with Hypergraphs, NeurIPS 2021
clarity,_quality,_novelty_and_reproducibility: ### Clarity
The method section is hard to understand. More clarification and improvements in Figure 1 are needed.

### Quality
Overall, the paper is well-structured.

### Novelty
Edge representation learning with its neighboring node is not a novel concept. This work, however, focuses on applying such an idea to graphs with textual edge attributes first.

### Reproducibility
 The authors provide the code for their method and experiments and explain the details in the Appendix.
summary_of_the_review: This paper has merit on its own in terms of how it affects learning edge representation on graphs with textual edge attributes.

However, to fully demonstrate the validity of their method, more comparisons against recent edge-aware representation learning methods are required.
Additionally, more details on their approach in section 3 could make the writing clearer.

As a result, I give this paper a weak rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper the authors address the problem of graph representation learning where edges are associated with text. In doing so the authors propose to combine pretrained networks with GNN with local network aggregation to inject text representation of nodes to its neighbors. The authors conduct experiments on various datasets and tasks, along with visualization, to show the performance of the proposed method.

strength_and_weaknesses: Overall this paper is clearly writing, well motivated. The method proposed is substantial and the results are convincing. 

That said, there is still some issues that are better addressed:

1. A lot of similar works have been researched on the graph with text attributes, like PLM-GNN and GraphFormers mentioned in the introduction. Although for them the text attributes are on the node, a straightforward adaptation of these methods to the current setting would be creating an extra node with text attribute for each edge. It's unclear how do they perform and compare with the proposed method

2. In ablation study, are the differences statistically significant?

3. in equation (4), since $z$s are of $R^d$ while $H$ is of $R^{d \times n}$, are such concatenations "balancing"? (e.g. would $H$ dominates the final representation?)

clarity,_quality,_novelty_and_reproducibility: Overall the quality, clarity and novelty of this paper is okay. 
summary_of_the_review: 
Since the strengths are dominant, I would recommend 6: marginally above the acceptance threshold
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: EdgeFormers is a new deep learning architecture to deal with networks with textual edges (e.g., email network or product reviews). Actually, it integrates two sequential modules: the first module (Edgeformer-E) aims at learning edge text-aware representations, the second module (Edgeformer-N) aims at learning node representation based on the edge representations of its neighbors. This work doesn't deal with attributed nodes, even though it should be easily adapted to do so. The main point is to go beyond considering the information associated to edges as a set of features, but treating it as a full text by the mean of Transformer-like mechanisms (multihead attention). EdgeFormers is trained on both edge classification (for Edgeformer-E module) and link prediction (for Edgeformer-N module). It's favorably compared to several recent architectures on 5 datasets.
strength_and_weaknesses: Strength:

- dealing with the textual information in a proper, modern way
- experiments on various and quite big datasets with positive, consistant results

Weaknesses:

- The architecture is quite complex with multiple mechanisms that are combined and it's not always easy to figure out what's really going on in there. For instance, Edgeformer-E integrates node vectors noted z_vi (for vertices vi) in order to contextualized the edge vector h_eij. However, Edgeformer-N uses another node vector noted h_vi which is the aggregation of the surrounding h_eij. In this end, I wonder what's the relation between both types of vectors. I guess that z_vi is a *first version* that has been a priori initialized with another node embedding technique, and h_eij a *second (and better) version* that is maybe more contextualized.
- Some recent works have pointed out the bias of GNNs that tend to "oversmooth" the information because of the propagation mechanism (see for instance the discussion here: https://towardsdatascience.com/over-smoothing-issue-in-graph-neural-network-bddc8fbc2472). It seems to me that EdgeFormer has the same king of limitation. If I'm correct, I think the authors should discuss this point and test on other datasets to clearly show when their model is applicable.
- Even though the authors make some effort for reproducibility, I don't see any information related to the usual train/test procedure we expect in a machine learning paper.
- It seems to me that the framework is transductive. What would be its ability for more inductive tasks? For instance, what about a new node (e.g., a new user) that posts a review on a product which is in the network? Is EdgeFormer able to deal with such a case?

Typo:
- I think "z_i" and "z_j" should be replaced by "z_vi" and "z_vj" in Eq.12 
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear, even though I had to take a look at the appendices to have a clearer understanding of the final procedure (Alg.1&2). The authors give access to their code and the paper contains many details about the experimental setting. 

I think the novelty is quite limited because the authors combine already well-known techniques. There is no new idea nor insight that can foster further research.

We can find many details of the implementation so that the experiments can be reproduced. As I said earlier, it's surprising that the usual information about how data have been split into training/test sets is not given.
summary_of_the_review: The proposed architecture is not fundamentally novel but it seems more fitted to deal with networks with textual edges. The model is quite well designed, even though some parts should be made clearer (relation between h_vi and z_vi, see above). EdgeFormer leads to slightly better results on 5 datasets, which finally explains my recommandation.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors want to design a more expressive edge model for graph neural networks based on transformer models. They propose Edgeformers to improve edge and node representation learning by modeling texts on edges in a contextualized way. The idea is to inject rich text descriptions / context in the edge representation with a transformer architecture instead of just using a simple relation name. The AC thinks that this approach is simple but reasonable, and the authors have agreed that this paper has created a meaningful design to model edges and nodes. The authors did a great job responding to the reviewers, and one reviewer has increased their score after seeing the authors' response. Overall, we recommend an acceptance of the paper based on reviews.
justification_for_why_not_higher_score: BERT was from 2018, so the experiments could be improved by incorporating more recent backbones.
justification_for_why_not_lower_score: This is a clear acceptance since all reviewers voted acceptance.
Decision: Accept: poster