Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work aims to alleviate the effect of popularity bias and noisy user-item interactions in graph convolutional networks (GCNs) for recommendation. To this end the authors propose a GCN-based recommender method called Contrastive Graph Structure Learning via Information Bottleneck (CGI). In addition to the standard GCN layer producing node representations from the observed user-item graph, CGI includes a node and an edge dropping components whose role is to generate augmentations of the observed graph by dropping some nodes and edges respectively. To improve the quality of the learned augmentations, the authors introduce (in addition to the recommendation objective) an Information Bottleneck (IB)-based contrastive objective, which encourages the representations learned from the augmented graphs to be independent from those learned from the observed graph. The proposed method is evaluated and compared to some existing recommender models on three real-world datasets.  
strengths_and_weaknesses: Strengths.
1. The paper is well written and technically sound. 
2. This work investigates interesting ideas such as the IB-based contrastive objective, as well as learning the graph augmentations instead of relying on random or predefined procedures to generate them. 
3. Empirical results show that CGI outperforms the chosen baselines on the recommendation task and seems more robust to noisy user-item interactions.

Weaknesses.
1. The claim regarding the ability of the proposed method to alleviate the popularity bias is not well supported in the paper, nor by theoretical analyses, neither by convincing targeted experiments. For examples, I would recommend reporting statistics about the popularity distribution of the recommended items by the different baselines. Also, some quantitative and qualitative experiments on how popular/rare items are ranked by the different models for a given set of users would be meaningful (see for instance Figures 2 and 3 in [1]).
2. Experiments are weak. Three datasets are considered for evaluation: Yelp, MovieLens and Douban. However, most of the results are on Yelp and/or MovieLens, except in Table 1.  I would recommend reporting the results of every experiment across all the three datasets.

Additional comments/questions.

- In section 4.3, the part related to proposition 1 is a bit hard to follow and connect to the objective of eq. 13. This is due to using different notations for the mutual information terms in the proposition and in eq. 13. Please consider improving the notations.  
- I would recommend keeping the legend consistent across all the experiments. For instance, CGI is represented by a green bar in figures 3 and 4, while in figure 5 it is represented by a blue bar.  
- What type of significance test is used in the experiments, and how many trials are performed for every algorithm in Table 1. 

References.

[1] Liang, Dawen, et al. "Factorization meets the item embedding: Regularizing matrix factorization with item co-occurrence." Proceedings of the 10th ACM conference on recommender systems. 2016.


questions: Please refer to the strengths and weaknesses section. 
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a graph neural network model based on LightGCN using learned parameters to adaptively dropout nodes and edges to obtain multi-view representation and contrastive learning with information bottleneck. Experiments on public datasets show improvements compared to existing models. Ablation studies show effectiveness of different components in the model.
strengths_and_weaknesses: Strength
- The paper explains the motivation behind different model components, e.g., adaptive dropping of edges and nodes to reduce popularity bias and contrastive learning via information bottleneck to reduce noise.
- The paper conducts thorough experiments using 3 public datasets and compared with multiple models. There are also ablation studies to verify different hypothesis in the model. The paper also provides detailed hyperparameters and time complexity analysis. Code was also provided for reproducing the results.

Weakness
- While the paper provides empirical results showing that the model can reduce popularity bias and provide robustness to noise, there are no clear theoretical link provided.
questions: The main GNN component is based on LightGCN, have you tried with other GNN variants? Do they work well?

From the details of the dataset, it seems that the paper only considers a binary case for Movielens and Douban (1 vs 0) and this seems to simplify the problem. For real recommendation systems, there's a different between recommending an item that user would rate 5 vs 4 (assuming 5 is the highest rating). It would be worth trying to fully utilize the ratings information in these datasets.

If the proposed model aims to address the popularity bias issue, why are items with fewer interactions removed (Yelp with <10, Movielens with <3, Douban with <5)? These are real data and should be considered in the model training and evaluation unless they are corrupt or spammy data points.
limitations: The model assumes that the data is relatively clean (without much spam or misleading content/item). Real recommendation systems would need to consider filtering out spammy content, misleading and low quality content.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes CGI, a contrastive model for item-user bipartite graphs in recommendation systems. The CGI consists of two parts: learnable graph structure augmentation and information bottleneck losses. Each of these modules reduces the influence of popularity biases and noisy interactions. The experiments on three real-world recommendation tasks demonstrate that CGI outperforms baselines, especially for unpopular items and noisy settings.

strengths_and_weaknesses: ## Strengths

This paper deals with an important problem in recommendation systems. The model design aligns with the motivation, and extensive studies under various scenarios are well-presented. The authors design experiments for sub-problems of this particular domain (popularity biases and interaction noises) and present them in an appropriate format. I like the Effectiveness of Information Bottleneck part (line 340) since this is an essential analysis to support the model design choice of MI minimization rather than maximization.

## Weaknesses

Although I have decided to rate this paper as borderline accept, there are two flaws in this paper. One is a minor problem that can be solved by simple revision, but the other can be critical. Even if a rejection decision is made, I will not object to it.

First, some related works of information bottleneck models on graphs are missing.

- Yu, Junchi, et al. "Graph Information Bottleneck for Subgraph Recognition." International Conference on Learning Representations. 2020.
- You, Yuning, et al. "Bringing your own view: Graph contrastive learning without prefabricated data augmentations." Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. 2022.
- Sun, Q., et al. "Graph Structure Learning with Variational Information Bottleneck." Proceedings of the AAAI Conference on Artificial Intelligence. 2022.
- Yu, Junchi, Jie Cao, and Ran He. "Improving subgraph recognition with variational graph information bottleneck." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. (Yes, I know this paper is very recently published, but it was on arXiv more than three months ago. Moreover, it is deeply related to the authors' work. See the 'Second' section below)

Second, the CGI model might be a straightforward combination of existing approaches but for a specific domain. The technical novelty is somewhat limited. Learnable masking of nodes or edges with reparameterization is a well-known approach for learning graph structures, and using it as augmentations have been proposed recently (as the authors say). The authors use information bottleneck losses in the same form as the original GIB paper (and other variants for graph tasks) with similar motivations. I believe this is an appropriate choice for this problem, but it is difficult to call it novel. In addition, the theoretical result (proposition 1) is a simple application of Proposition 3.1 in [1] and is very similar to Lemma 4.1 in [2].

[1] Achille, Alessandro, and Stefano Soatto. "Emergence of invariance and disentanglement in deep representations." The Journal of Machine Learning Research 19.1 (2018): 1947-1980.

[2] Yu, Junchi, Jie Cao, and Ran He. "Improving subgraph recognition with variational graph information bottleneck." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. 

questions: - It is nice to see the statistical significance tests in the result tables. Could you write the standard deviation and the number of actual runs in the experiments?
- How about changing the name from CGI to CGIB? Surveying existing works, models that end with ‘I’ are conventionally InfoMax models (DGI, HDGI, GMI, GCI, …) and models that end with ‘IB’ are usually Information bottleneck models (GIB, HGIB, VIB-GSL, VGIB, …).

limitations: The authors fairly discussed limitations.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new deep learning architecture for graphs, named CGI. CGI aims at build a vector space for users and items that can be used for recommander systems. The main novelty lies in the way two views are built (one for the nodes, one for the edges), by using data augmentation with perturbation (for dealing with high-degree nodes) and Information Bottleneck framework to make the representation space more robust to noise. CGI is favorably compared to several recent models on three different datasets with a proper experimental framework (good metrics, ablation studies, etc.). I don't see any import flaw in the methodology. All in all, it seems to be as a good, rational contribution to the field.
strengths_and_weaknesses: Strengths:
- new interesting model with good motivations
- good results when compared to recent sota
 
Weaknesses:
I don't see any important weakness.
questions: Is it possible to design a more accurate baseline by mitigating the weight of node centrality (like a normalization) before using solutions from the literature?

Section 4.3 is not perfectly clear to me. I don't understand to what extent Proposition 1 is a real new contribution (that can be used for other research, and in what context), or if it's a rather straightforward derivation.
limitations: I don't see any important limitation. I've not given the maximal score because it's really interesting but not fundamentally novel (like, "groundbreaking").

However, I'm bothered by the fact the authors don't point out any possible limitation... and I cannot believe that. From my opinion, a paper like this should tell when the model isn't fit to the problem and what may be the possible extensions. It's maybe a problem of the (possible) computation cost of optimization, or when the two views (node vs. edge) disagree?
review_title: null

Meta-review: metareview: All reviews were on the side of acceptance, but opinion varied from borderline to strong acceptance.

Several aspects of the paper were appreciated:
- The problem was considered important, and the model and its technical ideas were considered interesting. - - The model and its components were considered clearly motivated.
- The performance and robustness advantage over baselines in empirical results was appreciated, as were the ablation studies, and the analysis of the information bottleneck part.
- The paper was considered well written and technically sound, and code was provided.

The reviewers dis have several concerns:
- A claim about alleviating popularity bias was considered not well supported. One reviewer considered it was not well supported either by theory or experiments, whereas another reviewer felt that empirical results did show reduction of popularity bias and noise robustness but no clear theoretical link was provided.
- Opinion on experiments varied: three reviewers considered the experiments extensive / thorough / having good results, but one considered the experiments weak and not giving results on all three data sets in each experiment; additional information was provided by authors in a response.
- Some related work on information bottleneck was missing. 
- The novelty was considered limited as a combination of existing approaches. 

The discussion with the authors resolved at least in part the latter two concerns.

Overall, it seems the paper can be publishable in NeurIPS provided the improvements provided during the discussion phase are sufficiently taken into account in the final manuscript.
confidence: Certain
Decision: Accept