Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To tackle the noisy negative sampling from the unlabeled data, the paper proposes an inverse dual loss (IDL) to assign larger weights to the more confident loss (either negative loss or positive loss) on the unlabeled data to achieve automatic labeling. To further improve learning robustness, it proposes a meta-learning procedure to determine whether a direct gradient, an inverse gradient (IG), or a pass gradient of the inverse dual loss will be applied based on the meta-test performance. Experiments on two datasets demonstrate the method's superiority over naive negative sampling and a denoising method that only focuses on denoising the labeled data. The effectiveness of IDL to classify the unlabeled data and the effectiveness of IG to assist convergence are also analyzed in the experiments.
strength_and_weaknesses: Strengths
- The paper focuses on solving an important and challenging issue of noisy negative sampling in recommender systems.
- The idea of employing the inverse dual loss to annotate the unlabeled data and applying a meta-learning framework to determine the update gradient for more robust learning is novel.
- A processed Micro-Video dataset is released to the public, where labeling has been done based on the playing time of videos. This I believe will become a good contribution to the community.
- The experiments to analyze the effectiveness of IDL and IG are helpful.

Weaknesses (see the concerns below)
- Some claims in the paper are not well-supported.
- Several parts of the paper lack clarity, which affects the understanding of some important ideas.
- Experiments can be further improved to include the implicit feedback setting.

Major Concerns
1. In section 2.3.1, the authors claim that "when the sampled data is hard, exploiting the inverse gradient to update the model will gain a smaller test loss on the training-test data, and thus we exploit inverse gradient here". However, I find that the link between the hard instances and the inverse gradient is not clear, and hence the claim that the inverse gradient helps improve learning with the hard instances is not supported.\
Based on the descriptions in the previous sections (and illustrations in Fig. 1), hard instances are points that are close to the decision boundary (and easy instances are those that are far from the decision boundary). This means that for these hard unlabeled instances, the negative loss and positive loss values are close to each other (as the classifier has a hard time determining labels for them), which results in similar weights for the negative and positive loss in the inverse dual loss in eq. 4. Hence, for hard instances, the resulting effect on learning is just that the model will hardly change as both the positive and negative loss are optimized equally. I don't see why in this case, employing an inverse gradient of the inverse dual loss will help achieve better learning and result in "a smaller test loss on the training-test data" (similarly for the case of easy instances, why employing direct gradient for them is better). The authors may want to provide more explanations on this to support the claim.
2.  The experiments in this paper analyze two scenarios: passive feed recommendations (Micro-Video) and explicit user rating (ML1M). For both cases, the authors pre-processed the datasets to have binary labeling where the labeled user-item pairs can be either positive or negative. However, for the implicit feedback setting (i.e., users provide interactions like clicks and purchases as feedback), all the labeled/interacted user-item pairs are considered positive. For this case, the test loss in the meta-learning procedure will only have positive data points. How will this affect the annotation quality of the inverse dual loss for the unlabeled/uninteracted data? Since the implicit setting is quite a prevalent scenario in the industry, I suggest the authors incorporate experiments on the implicit setting (e.g., using Yelp, Amazon datasets) to make the evaluation of the proposed method more comprehensive.

Minor Concerns
1. In Definition 1, it is not clear why the normalization parameter z_w is formulated in this way. Can the authors provide some intuitions on this?
2. In the summary of the main contributions, the authors claim that they are "the first to perform denoising from both positive and negative perspectives in recommendation". According to my understanding, the paper only proposes a method to denoise the negative sampling (avoid false-negative) from the unlabeled data. Can the authors explain why they claim that their method also serves to perform denoising from a positive perspective? 
3. There is this recent work [1] that aims to tackle noises in both positive and negative implicit feedback. The authors may consider including a comparison with this work (adding it as a baseline in experiments or at least discussing it in the related work).

Parts that need further clarification/adjustment
1. Based on the descriptions in "Baselines and Backbones" and section 3.1, it is not clear whether IG is adding on top of IDL or the two methods are applied separately.
2. In the second paragraph of section 3.1, what does it mean by "IDL only outperforms the NeuMF" and "it is even outperformed by GMF", as both NeuMF and GMF are just the backbones but not the methods to compare? The authors may want to rephrase this part.
3. In section 3.3, the authors state something about the effects of pre-training on the training-train data. However, experiments in this section all involve pre-training, and a set of experiments without pre-training is required to make comparisons and support such claims.
4. In Fig. 5, for the two plots on NeuMF, the legend boxes are too large and block out some important parts of the learning curves.

[1] Denoising Neural Network for News Recommendation with Positive and Negative Implicit Feedback, NAACL 2022

clarity,_quality,_novelty_and_reproducibility: - Clarity is lacking in certain parts of the paper (see concerns).
- Quality of the work is not up to the standard of acceptance as several parts cause serious confusion and the experiments are not comprehensive (see concerns).
- The method proposed in this work is considered quite novel in my opinion, especially the inverse dual loss and the application of the meta-learning framework.
- The work is reproducible as both the source code and the datasets used are released.

summary_of_the_review: Since some important claims made in the paper are ambiguous/not well-supported and the experiments lack evaluation in an important aspect (i.e., implicit setting), I feel that the paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the de-noising recommendation problem. The authors propose a meta learning method to annotate the unlabelled data from both the loss and gradient perspective. They propose the inverse dual loss to boost the true label learning and prevent false label learning. Moreover, they also propose the inverse gradient to explore the correct updating gradient. To demonstrate the effectiveness of the proposed method, the authors perform extensive experiments on two real datasets. 
strength_and_weaknesses: Strengths:
1.	The authors propose to perform denoising from both positive and negative perspectives in recommendation. 
2.	The authors propose two new methods, i.e., inverse dual loss and inverse gradient, to improve the model performance.
3.	The authors perform extensive experiments on two real datasets to demonstrate the effectiveness of the proposed method.

Weaknesses:
1.	The proposed method is only studied with the logloss. It is not clear whether the proposed method can help other ranking-based recommendation loss, e.g., Bayesian personalized ranking loss. 
2.	The backbone models, i.e., GMF and NeuMF, are a little weak. The authors are recommended to consider other advanced recommendation models as backbone models, e.g., LightGCN and other sequential recommendation models.
3.	Some claims in this paper are not supported by experimental analysis. 

clarity,_quality,_novelty_and_reproducibility: 1.	The motivation to develop the inverse gradient method is to achieve more robust sampling on hard instances. However, there is no experimental analysis studying the “robustness” of sampling on hard instances.

2.	In Section 2.2.1, the authors claim that the objective of this work is to propose a noiseless solution for unlabeled data. However, the definition of “noiseless” is not clear. Moreover, there is no experimental analysis studying the “noiseless” property of the proposed method. 

3.	The item recommendation based on implicit feedback is more likely a ranking problem. It should be better to use some ranking loss, e.g., Bayesian personalized ranking (BPR) loss, instead of logloss, to study the research problem. To demonstrate the effectiveness of the proposed method, the authors are recommended to also apply the proposed method with other recommendation loss functions, e.g., BPR loss.

4.	In this work, the authors use GMF and NeuMF as the backbone to demonstrate the effectiveness of the proposed method. However, these two methods are a little weak. The authors can consider LightGCN or other GNN-based recommendation methods as the backbone models to study the effectiveness of the proposed method.

5.	Negative sampling is a general problem in many recommendation problems, e.g., general top-N recommendation and sequential recommendation. The authors may also consider one more recommendation task, e.g., sequential recommendation, to study the effectiveness of the proposed method.

6.	Why choose the Micro-video dataset for experiments instead of other publicly available datasets, e.g., Amazon Review dataset?

7.	In the experiments, the definition of positive and negative feedback is based on some pre-defined rules, e.g., a finishing rate greater than 80% as positive feedback and less than 20% as negative feedback. Are these kinds of definitions reasonable?

8.	In the experiments, the authors claimed that “we split 60%, 20%, and 20% of the data as training, validation, and test data, for these two datasets”. However, it is not clear whether this data splitting is based on each individual user’s behavior or all users’ behavior data. The authors are recommended to include a more clear description. 

summary_of_the_review: Overall, this paper provides some novel ideas to solve the drawbacks of existing negative sampling strategies widely used in existing recommendation research. Some claims in this paper are not supported by experiments. Moreover, the experimental analysis is not sufficient. The authors only study the effectiveness of the proposed method on the Logloss. The quality of this work can be improved by studying the performance of the proposed method with other recommendation functions, e.g., BPR loss. In addition, the backbone models are a little weak. The authors need to consider other advanced recommendation models, e.g., LightGCN or other sequential recommendation models, to demonstrate the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a heuristic method to better exploit unlabeled data in recommender system in order to boost performance and also avoid too strong introduced noise. The method has components such as a re-weighted loss based on current model prediction and an explorative gradient selection logic. The performance gain demonstrates the necessarily of the gradient selection logic.  
strength_and_weaknesses: Strength 
The paper is clear to follow. The motivation and intuition is clear and valid.

Weakness
There is no rigorous theoretical guarantee. There is no data point in a complicated experimental set up such as industrial level production A/B testing results.

clarity,_quality,_novelty_and_reproducibility: Clarity is good and easy to follow.

Quality is fair.

Novelty is fair that the idea is not surprising.

Reproducibility is good since code is/will be public according to the main text.
summary_of_the_review: The author should also benchmark with semi-supervised learning in recommender system since both trying to exploit (assign a label) to large volume unlabeled data.

When model is not well trained (such as in early stage training), there will be a negative feedback loop to use such inverse loss and inverse gradient. The method is too heuristic to cope with such edge cases.

Flaws:
Usually the predicted label is between 0 and 1 and the loss in Eq one is always positive. In Eq 4, the absolute symbol doesn't make sense.

The inverse gradient is too ad hoc. For example, in the proof, we see a very much simplified case, this is not sufficient proof for a theorem. It's very hard to define easy example, hard example and guarantee the gradient triage to work.

Typo:
in abstract line 3, it should be "often introduces false-negative noises", instead of "often introduces false-positive noises"?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors proposed a meta-learning framework based on a new inverse dual loss to generate gradients to improve the negative sampling quality for recommender systems. Though the high-level idea is interesting, the experimental results fail to support the claims made by the authors. Moreover, some experimental setups are not convincing.

In summary, this paper is not ready for publication in ICLR.
justification_for_why_not_higher_score: Many claims made by the authors are not supported by experiments.
Decision: Reject