Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
The authors propose a novel targeted doubly robust (TDR) algorithm for collaborative learning in recommender systems that alternately optimizes the predictor and propensity estimator encountered in a typical DR estimator along with the error imputation estimate and the step size estimate encountered in a targeted approach. Ablation studies on real data clearly demonstrate the efficacy of the proposed novel components. The authors also convincingly demonstrate improved metrics (AUC, NDCG, etc.) with respect to competing methods.

strength_and_weaknesses: Strengths:

1. The paper appears to be theoretically strong. Proposition 1, Theorem 1 and Theorem 2 appear to be novel and interesting.

2. Algorithm 1 is itself well-motivated and the emprical results on semi-synthetic data in Table 2 and real data in Tables 3, 4 are quite convincing.

Weaknesses:
1. The authors correctly credit van der Laan and Rose for developing TDR estimators, but they do not relate the multi-step algorithm in Sec. 4.1 on TDR with previous work by van der Laan and Rose. Theorem 1 and Theorem 2 in Sec 4.1 appear to be new, but the reader is not explicitly made aware of this fact.

2. The authors ignore temporal variability in recommender systems and assume that features $x_{ui} $ for a user-item pair are static rather than temporally varying.
Recurrent Recommender Networks, WSDM 2017, shows that this assumption often does not hold for real world recommender systems.

3. The presentation of the material could be improved. For instance, the columns "Without extrapolation" and "Boundedness" in Table 1 are not explained in the paper and one needs to refer to appendices, whereas the symbol \eta in eq (4) changes to \epsilon when we reach Algorithm 1 and its description. The meaning of the columns in Table 2 is not clear until one reads Appendix E.

clarity,_quality,_novelty_and_reproducibility: Weaknesses 1, 2 and 3 all include concerns about clarity. In addition, typos exist, e.g., "RQ1. Do the ... estimator ... have ..." should be "RQ1. Does the ... estimator ... have ...".

The quality of the theoretical and experimental work appears to be solid as mentioned above.

The approach does not appear to be particularly novel, but it is an interesting combination of existing ideas.

The code is provided in the supplementary file for reproducibility and sufficient experimental details seem to be provided in the appendices.
summary_of_the_review: Given the concerns regarding clarity discussed under weaknesses as well as the second weakness regarding general practical applicability, I weakly recommend acceptance of this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a new estimator for debiasing Recommender Systems called Targeted Doubly Robust (TDR) which addresses some limitations of the Doubly Robust (DR) and Error Imputation Based (EIB) estimators. 

They show first that DR has a large variance and is sensitive to small propensities. The variance of DR, while being lower than variance of Inverse Propensity Score (IPS), is higher than variance of Error Imputation Based (EIB) method. The objective is to come up with an estimator with the best of the two worlds: DR and EIB.

By rewriting DR estimator as a sum of the EIB estimator and a correction term, they model the error imputation by any arbitrary function (without any assumptions) so that the correction loss is equal to zero to ensure the lowest variance of EIB but smaller bias than EIB (Theorem 1). 

TDR benefits from all other advantages of EIB: boundedness and robustness to small properties. The authors show TDR does not increase the bias compared to DR. Theorem 2 proves its double robustness. 

The authors finally show how TDR is used in a collaborative-filtering-based recommender system. 

strength_and_weaknesses: Strengths:

The paper is quite clear and enjoyable to read. Motivation of the paper is clearly stated and this is easy to follow for the reader how the authors approached the problem and solved it. 

Each of the desirable properties for debiasing estimators (doubly robust, low variance, robust to small properties, without extrapolation and boundedness) is investigated theoretically. Definitions of the concepts are recalled in the appendix.

The theoretical contributions of the paper are significant and are supported by extensive experiments on synthetic and real datasets (best performances are obtained by TDR estimator and its variants). 

Weaknesses:

The section on “semi-parametric collaborative learning” might be a little bit dense.

It might be worth also to recall briefly the differences between the competing methods used for comparison in the Section “Real-World Experiments”.

It would be finally helpful for the reader to give a little bit more details on targeted learning and the link with the proposed estimator.

Which stopping criteria and number of steps do you suggest for Algorithm 1 in practice?

Minor (typos):
-p.4: “Specifically, It”
clarity,_quality,_novelty_and_reproducibility: The paper is rather clear and well written. 

They recall all the desirable properties of existing estimators and compare them in a summary table. This easily highlights what TDR is bringing to the field and its novelty. 

In terms of reproducibility, use of public datasets and release of the code is very helpful.

summary_of_the_review: The paper proposes a new estimator TDR for debiasing Recommender Systems which is thoroughly investigated theoretically and well-supported experimentally. TDR obtains best performance on semi-synthetic and real datasets and gathered all the desirable theoretical properties.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper argues that existing doubly robust (DR) estimators usually have a large variance, a large bias, and poor generalization ability given inaccurate error imputation. The paper proposes a novel estimator, called targeted doubly robust (TDR), to simultaneously reduce the bias and variance of existing DR estimators. The paper demonstrates that the imputed errors by the TDR estimator satisfy two important properties called validity and preservation in Theorem 1 and shows that the TDR estimator preserves the desired property of double robustness in Theorem 2. The paper further proposes a semi-parametric collaborative learning approach (TDR-CL) whose key idea is to decompose imputed errors into parametric term and a non-parametric correction term. The paper conducts semi-synthetic experiments to demonstrate that the proposed TDR estimator achieves the lowest absolute relative error among four baseline estimators (Naive, EIB, IPS, and DR) under six rating missing scenarios (ONE, THREE, FIVE, Rotate, SKEW, and CRS). The paper also conducts experiments on two real-world datasets (COAT and YAHOO) and the experimental results show that the proposed approach significantly outperforms baseline approaches under MSE, AUC, NDCG@5, and NDCG@10.


strength_and_weaknesses: S1: The idea of applying targeted learning techniques to effectively reduce the bias and variance of existing DR estimators is novel. The paper provides theoretical results showing that the imputed errors obtained with the proposed TDR estimator satisfy two desired properties (validity and preservation) and the proposed TDR estimator has the desired property of being doubly robust.

S2: The paper conducts pretty extensive semi-synthetic experiments and real-world experiments to validate the effectiveness of the proposed TDR estimator and TDR-CL approach, respectively. In the semi-synthetic experiments, the paper simulates a total of six different rating missing scenarios and compares the proposed TDR estimator against four baseline estimators under these diverse scenarios. In the real-world experiments, the paper uses two real-world datasets, compares against a wide range of baseline approaches (IPS, SNIPS, CVIB, DR, DR-JL, etc.), applies the proposed TDR-CL approach to two representative DR approach (DR-JL and MRDR), and computes various evaluation metrics (MSE, AUC, NDCG@5, NDCG@10). The extensive experimental results make the argument that the proposed estimator and approach are more effective than existing approaches much convincing.

W1: In Section PRELIMINARIES, it is not clear why a prediction error is defined as an expectation of the real prediction error given features. It is important to add more explanations and clarifications here.

W2: It would be better if the paper can move the descriptions of the six rating missing scenarios, ONE, THREE, FIVE, Rotate, SKEW, and CRS, from the Appendix to the main text.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and is of a good quality.
summary_of_the_review: This paper is marginally above the acceptance threshold based on its novelty and significance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work argues that existing doubly robust (DR) estimators tend to have large variance and bias, with poor generalization ability with inaccurate error imputation. The work then proposes a novel estimator, called targeted doubly robust (TDR), to simultaneously reduce the bias and variance of existing DR estimators. TDR is designed for collaborative learning in recommender systems that alternately optimizes the predictor and propensity estimators. By rewriting DR estimator as a sum of the Error Imputation Based (EIB)  estimator and a correction term, they model the error imputation through an arbitrary function.

The work tackles an important task with an innovative solution. Reviewers agree that the idea of applying targeted learning techniques to effectively reduce the bias and variance of existing DR estimators is novel.
justification_for_why_not_higher_score: The work applies existing methods (e.g., targeted learning) to a new context (doubly robust (DR) estimators) for recommender systems. The method is interesting but unclear there is significant novelty for a spotlight or oral.
justification_for_why_not_lower_score: The paper makes a clear contribution in doubly robust (DR) estimators for recommender systems.
Decision: Accept: poster