Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose CMI-GCL to generate contrastive pairs in real-world imbalanced graph datasets automatically. At first, it designs the inter-modality GCL module by co-training encoders in different modalities to automatically generate contrastive pairs. To deal with the data imbalance problem, CMI-GCL employs network pruning to prune graph encoders and image encoders to boost the weights of monitor samples during GCL optimization. In addition, it designs an intra-modality GCL module to guarantee that similar nodes in the graph have similar embeddings. Experimental results on two real-world datasets show that CMI-GCL outperforms many state-of-the-art baseline models and shows the strong applicability of the CMI-GCL framework.
strengths_and_weaknesses: $\textbf{Strengths:}$

i) The model idea in this paper is novel. Most GCL models focus on benchmark datasets, but they cannot be applied to real-world data. This paper points out the gaps of existing GCL models to real-world data and proposes to co-train encoders across different modalities to generate contrastive pairs. I know there are some cross-modality learning works in image-text, or video-text fields (e.g., CLIP), but co-train the cross-modality encoders in graph-image or graph-text fields is a good try. It looks like that cross-modality GCL has gained excellent performance in two real-world datasets based on the results of the ablation study.  Besides, the idea that applying the network pruning to detect minority nodes in self-supervised learning is also a good try, especially pruning the encoders in different modalities.  

ii) This paper is well organized with clear motivations and sufficient comparison experiments in two real-world datasets, which makes the paper easy to understand and follow. 

iii) The detailed discussion about experiments is impressive. To evaluate the performance of CMI-GCL, this paper defines two fine-tuning modes for imbalanced datasets and balanced datasets, respectively. By comparing CMI-GCL with state-of-the-art baselines models on two real-world datasets (GitHub and Instagram), CMI-GCL shows its superiority. I think Figure 2 is impressive to show that network pruning is useful to detect minority nodes in GCL pre-training. 

iv) As CMI-GCL is developed as a plug-and-play tool, I think CMI-GCL can be considered as a general framework for real-world graph datasets and it can be applied to most real-world graphs with multi-modal data.  Figure 3 describes the performance of the encoder’s combination, showing that CMI-GCL is applicable to mainstream models in different modalities, like BERT and DistilBERT for text content, GCN, GAT, and SAGE for graph, ResNet and Swin Transformer for image content. But I wonder about the number of models that are developed for this framework. With more models considered in CMI-GCL framework, CMI-GCL will be more applicable to real-world datasets.

$\textbf{Weaknesses:}$

i) Some definitions should be cited if it is not defined by this work. For instance, PIEs (pruning identified exemplars) in Section 4.2.2 should be cited as it is from the previous work [1].

ii) More explanations about the experimental results should be provided so that it will facilitate the understanding of model improvement. For instance, in Table 3, why does co-modality have the biggest contribution to CMI-GCL?  Could you explain more about Figure 2?

ii) There are some typos in this paper. 
E.g., in section 5.5, CMI-GCL is applicable to --> CMI-GCL is applicable to
Line 240, inspired by GCC we introduce --> inspired by GCC, we introduce
DistillBert --> DistilBert

[1] What do compressed deep neural networks forget? arXiv 2019.

questions: i) In table 3, why does the co-modality module have the biggest contribution to CMI-GCL?

ii) Could you explain more about Figure 2?  why does subset D0.1 can detect the most minority samples in the beginning? Why does subset D init have the lowest percentage in the beginning? Why does D 0.01 have the highest percentage in the end?

iii) In Section 5.5, it claims that the framework CMI-GCL is designed as a play-and-play tool and has very strong applicability to mainstream models. I wonder how many models are already considered in this framework. Could you list some examples except these models listed in Figure 3?
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper explored tackling data imbalance problem on graph with contrastive learning. This work proposes a principled framework called Co-Modality Imbalanced Graph Contratsive Learning with Network Pruning (CMI-GCL) to generate contrastive pairs and further learn balanced representation over unlabeled data. Inter- and intra-modality GCL are designed to learn general node representation. Network pruning is used to detect minority nodes. Experiments are conducted on two experiments to verify the effectiveness.
strengths_and_weaknesses: For the strengths, the paper tackles the important data imbalance problem on graph data and designs effective co-modality contrastive learning strategy to learn multimodal representation. Extensive ablation study demonstrates the effectiveness of the proposed method. However, I'm confused about why network pruning can resolve the problem of data imbalance. The paper mentioned minority samples can be “forgotten” by pruning deep neural networks. But why the paper said pruning and releasing weights in each round can effectively boosts minority samples’ weights in the section "Network Pruning for Imbalance Data"? For the experiments, despite the datasets you used, OGB node property benchmark provides numerous large-scale graph datasets with raw text. Current experiments are only conducted on two datasets, which seems not very convincing. Moreover, it'll be better if you could provide the standard deviation of the experiments.  

There are also some minor issues. In the abstract, the paper said "Hence, we co-train two pruned encoders ...". What is the causal relationship here? In both Figure 1 (above "Content Encoder") and 3, there are some weird gray lines, please consider removing them in the future revision.
questions: Please refer to the weak points I mentioned above.
limitations: The limitations are discussed in the conclusion. I believe the authors can find more related datasets at OGB benchmark. I’m not aware of any potential negative societal impact.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a model to learn representations over graph nodes such that: (1) the representations can address imbalanced nature of classes, and (2) they use multimodality to learn representations on nodes. The learned representations show good performance compared to baselines in various settings on two benchmarks. As the authors mention, the limit of their approach is that it assumes corresponding textual or visual data is available.
strengths_and_weaknesses: Strengths:
- The paper tackles the problem of imbalanced data in node-level representation learning while assuming that corresponding modalities are available. This setting can be important in some real-world graph data such as social networks.
- The ideas in the paper are easy-to-follow.
Weaknesses
- The paper needs some revisions to address typos, e.g., "Existing graph representation learning methods [9, 41, 29] generally implement the pre-trained language model (e.g., BERT [6])".
- Experimental results are carried over only two datasets (Github and Instagram) which is not enough. There are other multimodal graph benchmarks publicly available that authors could have used.
- Important citations are missing from GCL literature are missing [1, 2, 3, 4]  

[1] https://arxiv.org/abs/2006.05582
[2] https://arxiv.org/abs/2201.09830
[3] https://arxiv.org/abs/2106.07594
[4] https://arxiv.org/abs/2102.06514
questions: - How would your model perform in case that some corresponding modality is missing from some nodes?
- How would the model perform on other publicly available benchmarks?
- How you would use your model for single modality graph data?
limitations: NA
review_title: null

Meta-review: metareview: All the reviewers finally give prone-to-accept score 5/5/8 to this paper after the authors' informative communication and adding more experiments on two additional datasets. I agree that the paper in general is novel and the approach is reasonably designed. I suggest to accept this paper while it can be also improved in many aspects especially for clarity as pointed out by the reviewers.
confidence: Less certain
Decision: Accept