Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a debiasing method in the learning to rank (L2R) regime. They advance the hypothesis that contemporary debiasing methods are lacking since they assume a click model where the effect of the position of an item in a ranked list on the probability of observation by the user is constant across different contexts.

The authors instead propose learning parametric models for observation probability and relevance probability. Since only clicks are observed in practice and they assume P(click|X) = P(observation|X, position) * P(relevance|X), the two quantities of observation/relevance are not identifiable. As a result the authors instead propose two conditions to fulfill a "soft decoupling" condition that maintains ranks in their observational model when compared to the true probabilities of observation.
1. lipschitz decoupling: where P(observation|X, position) is alpha-lipschitz in X and its corresponding estimation model is also beta-lipschitz in X, for a value of beta that falls within a bound dependent on alpha.
2. bernoulli decoupling: the observation probability is once again alpha-lipschitz, but here the corresponding estimation model is instead forced to output 1 with probability t, forcing the ranker to learn directly from click data. If t falls within a bound dependent on alpha, then soft decoupling also applies here.

The authors then validate their model across a linear and neural network ranker and compare to a number of unbiased baselines in semi-synthetic L2R settings.
strengths_and_weaknesses: This is a good contribution to the area of L2R. The model they study is elegant and natural and I was surprised there has been no prior work making similar assumptions.

Overall the clarity of exposition was very good and the paper was easy to follow. However, I felt that the authors could have provided more intuition in the bernoulli decoupling section (see questions).

While the ideas presented are quite valuable I felt the results in the experiments were a bit underwhelming. The authors assume that their hypothesis about disparate effects of rankings is correct in their simulations and only show marginal improvements in nDCG when compared to DLA unless their coupling parameter is significantly increased (the nDCG difference between DLA and LBD are smaller than the differences between LBD and itself for two different values of the hyperparameter t that are apart by 0.1). However, I did appreciate the completeness of their experimental results: the authors included results across a wide range of simulation parameters and compared their method with a broad range of baselines.

Overall, I'm leaning towards an accept although I would be willing to change my score up from a weak accept to an accept if the authors are able to make a compelling case for the strength of their experimental results and address my questions below.
questions: 1. Why is the bernoulli model the right model here? While I read the paper the choice felt a bit arbitrary, and rather felt like a hack to enforce soft coupling, rather than reflecting any real data generating process.
2. Does soft decoupling give us any guarantees about ranking accuracy? For example, if I have a softly decoupled observational model with probabilities that are way off from the true probabilities, can I still have some guarantee about my click probability?
3. There seems to be a gap between the theoretical and empirical sections of the paper since you tune the beta and t hyperparameters using a validation set. Given that you have access to the true underlying probabilities in the simulation environment, what happens if you explicitly compute the lipschitz bound
4. I'm confused how you evaluated nDCG in your validation set and test set. I had assumed you only observed click data across all splits of the data, hence shouldn't all your items in each query have a score of 0 except for the clicked item? In that case, wouldn't it be more interpretable to use a metric such as the average position of the clicked item, or the mean reciprocal rank?
limitations: Given that the intent of L2R models is for them to be deployed on large internet platforms, I would have liked to have seen some discussion about the societal impact of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an Unbiased Learning-to-rank (ULTR) method which relies on only clicks and performs decoupling to separate relevance and observation parts from the click model. 

Given that we observe only clicks, it is very difficult to retrieve the true relevance and observation parameters, because multiple combinations of relevance and observation can result in the same click. 

The authors propose a disentangling method to decouple the two parameters, with theoretical guarantees. 

Empirical results on the standard semi-synthetic benchmark demonstrate the efficacy of the method over established baselines. 

-- I have read and considered the author's response. 
strengths_and_weaknesses: Some strong points of the paper:

- The paper is well-written, with a proper introduction to the area of ULTR and biases in click data. Given the limited space in the NeuRIPS format, the authors have done a commendable job of introducing the related works and the problem itself. 
- The assumptions are laid out in an intuitive manner and the method is supported by theory. 
- Empirical results indicate strong performance as compared to baselines. 

Some weak points/comments on the draft:
- During training, when clicks are estimated via a product of predicted relevance and predicted observations, even with the regularization term on the norm of gradient, a trivial solution would be to make the observation model's prediction constant, and relevance equal to clicks. How will the model recover the true relevance in such a case? (even with soft decoupling, it needs to estimate relevance up to a constant). 

Overall, I think the paper presents an interesting theoretically motivated algorithm for ULTR. 
questions: I couldn't find the proof for soft decoupling and hard decoupling in the draft. It points to an Appendix, but I couldn't find one. 

limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors studied and proposed a Lipschitz and Bernoulli Decoupling (LBD) model to decouple relevance and observation at individual document level in unbiased learning-to-rank. The authors first explained about the coupling of relevance and observation, as there are common factors affecting both. They then translated the coupling effect into the main assumption of unbiased click prediction and defined both hard decoupling (model decomposes as the ground truth) and soft decoupling (model decomposition preserves the order of relevance function) based on that. Assuming the Lipschitz continuity of the observation function, the authors proved that the soft decoupling could be achieved by enforcing a Lipschitz continuity on the observation model with a finite Lipschitz constant and gave the upper and lower bound. Based on the same continuity assumption, the authors also showed Bernoulli observation model (BOM) can achieve the soft decoupling within the bounds of the observation dropout rate. Finally, the authors described the objective that combines both Lipschitz and Bernoulli Decoupling methods. In the experiment, the authors showed on the synthetic dataset, LBD could achieve the best unbiased results and also did the ablation studies on Lipschitz and Bernoulli alone. With a brief discussion of limitations, the authors concluded the paper. In the Appendix, the authors extended related works, proved the main theorems, and included more experiment details. 
strengths_and_weaknesses: Strengths:
1. The work studies a general and important but previously ignored issue in unbiased learning to rank, which has been the core of industrial application in recent years.
2. Well written and solid theory.
3. Complete study with sufficient ablation study and discussion on limitations.

Weaknesses:
1. The authors only demonstrated their method on the synthetic dataset, despite the method is claimed to be applicable to more nontrivial bias. It could be more convincing if authors could test the method on real-world click data.
2. The authors should better double-check their significance analysis. There are some results appearing absurd: e.g. in Table 1, 0.659 of LBD_Lips Linear model on Istella-S is significant worse while 0.592 of Unlimited Linear model is not significant? Also, not sure which exactly the baseline was when applied the significance analysis, why not 0.686 and 0.687 are not significant if 0.707 is significant, assuming the baseline is the best one with 0.701.
3. In Figure 2(c), when authors claimed about better performance of LBD at low eta. In fact, they only have a single point LBD performs better, which is eta=0, exactly no coupling. Isn't that somewhat deviated from the main claim of the paper. Can authors add some points, for example, 0<eta<0.1, to show LBD does work better at low eta end?
4. Minor points on notations: a. variable t appears to be doubly used in Theorem 2 and in Bernoulli model parameter; b. Appendix chapters should better always be clarified, somewhat confused when not knowing there is an appendix.
questions: Good that authors discussed the limitations on Assumption 1. On the other hand, based on my understanding, both Lipschitz and Bernoulli methods are based on the Assumption 2, i.e., the Lipschitz continuity assumption of observation. In general, I tend to accept the assumption. However, on the other hand, it implies the function is smooth up to certain scale. If we need to learn the smooth function and extend to below the scale, we need in our dataset at least dense enough to cover to the scale, especially, when our observation is now in a much higher dimension space. Can authors discuss this limitation on Assumption 2 as well? And explain the requirements on the dataset, e.g. data size, and relation to the smoothness alpha?

Please also check my other questions in weakness.
limitations: Please see my comments in questions and weakness above.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper studies the unbiased learning to rank problem. Different from existing work, the paper first argues that the bias factor should use any existing features, thus can go to individual-level. Then the authors argue that there will be a coupling effect between the relevance tower and bias tower as they share a lot of input signals. Then two simple methods are proposed. The first one enforces a smoothness constraint on the bias model (the so-called “Lipschitz Observation Model”). The second one uses dropout on the the bias model output (the so-called “Bernoulli Decoupling Model”). The paper provides some theoretical analysis on when decoupling is achievable. Experiments are done on semi-synthetic datasets with synthetic generated clicks on public datasets. 

Overall the paper can be strengthened with more compelling theoretical analysis and experimental evaluation. 

The reviewer acknowledges the author's reply. The dropout argument is decent. As some of the major concerns are still there (strong assumptions, only on semi-synthetic dataset, smoothness method only works on one dataset (more concerning as the datasets are manually controlled)). Will keep the score as the reviewer was initially between 3 and 4.
strengths_and_weaknesses: Strength
The problem studied is important and the motivation is solid.
The theoretical analysis is interesting, though the assumption is strong and the usefulness is unclear.

Weakness
The methods are simple (which is good), but not novel. In fact, the dropout method was proposed 3 years ago (“recommending what video to watch next: a multitask ranking system”) but it was not cited. The smoothness method has several concerns (see below) and do not seem to perform well in experiments.
The theoretical analysis is not very compelling to the reviewer. The assumption is strong (the authors acknowledged it which is good). For example, the smoothness constraint is not very convincing - the authors already mentioned the position-only case. Also, it is intuitive that different query types can have quite different browsing behaviors, easily violating the smoothness constraint. It only shows feasibility but no guarantees. In practice it’s about hyper-parameter tuning.
The experiments are not satisfactory. First, using semi-synthetic dataset is common in the literature, so it does not add extra points (while many people use real-world datasets). Also, the reviewer feels it is more important to test this paper on more realistic datasets. Indeed, as the authors' motivation states, the bias can have complex correlation with input features. Existing work mainly generate bias only depends on position so it could be quite simple, but this work has a different motivation and the way it generates bias can be too simplified. Third, the authors are suggested to compare with some more recent methods (the most recent baseline was in 2019).

questions: The paper describes HTE as baseline in the text but it’s not shown in the results?
On Istella, “Click Data” can outperform several unbiased methods? 

limitations: NA
review_title: null

Meta-review: metareview: The paper studies the unbiased learning to rank problem, and introduces new assumptions and techniques to learn good ranking policies from biased data. The key insight is to use smoothness assumptions to decouple the effect of observation from relevance. The reviewers appreciated this novel attack on a significant problem, and the authors clarified how existing position-based models and other debiasing strategies emerge as special cases of their smoothness assumptions. 

Some reviewers pointed out deficiencies in the empirical study -- lack of results on real-world click data, questions around statistical significance and finer-grained sweeps of hyper-parameter ranges (which the authors subsequently clarified during the feedback phase). Adding a real-world experiment, even on the limited TianGong-ST dataset that the authors identified but rejected, would substantially strengthen the claims in the paper. That said, the Lipschitz and Bernoulli decoupling are likely to be of interest to the learning to rank community, and spur follow-up work on better user modeling.
confidence: Less certain
Decision: Accept