LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank
Authors: Mouxiang Chen
Chenghao Liu
Zemin Liu
Jianling Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: learning to rank
unbiased learning to rank
decouple

Abstract: Using Unbiased Learning to Rank (ULTR) to train the ranking model with biased click logs has attracted increased research interest. The key idea is to explicitly model the user's observation behavior when building the ranker with a large number of click logs. Considering the simplicity, recent efforts are mainly based on the position bias hypothesis, in which the observation only depends on the position. However, this hypothesis does not hold in many scenarios due to the neglect of the distinct characteristics of individuals in the same position. On the other hand, directly modeling observation bias for each individual is quite challenging, since the effects of each individual's features on relevance and observation are entangled. It is difficult to ravel out this coupled effect and thus obtain a correct relevance model from click data. To address this issue, we first present the concept of coupling effect for individual-level ULTR. Then, we develop the novel Lipschitz and Bernoulli Decoupling (LBD) model to decouple the effects on relevance and observation at the individual level. We prove theoretically that our proposed method could recover the correct relevance order for the ranking objective. Empirical results on two LTR benchmark datasets show that the proposed model outperforms the state-of-the-art baselines and verify its effectiveness in debiasing data. 

LBD: Decouple Relevance and Observation for Individual-Level Unbiased Learning to Rank
Mouxiang Chen Chenghao Liu Zemin Liu Jianling Sun
'Zhejiang University, SSaleforce Research Asia, National University of Singapore, "Aiijaba-Zhejiang University Joint Institute of Frontier Technologies (chenmx,sunjil @z edu. en, Beanghao.Libbsalessoorc.. com, zeminl wuwuuuu.uuu... edu.sg
Abstract
Using Unbiased Learning Rank (ULTR) train the ranking model with bi- ased click logs has attracted increased research interest. The key idea is to ex- plicitly model the user's observation behavior when building the ranker with a large number of click logs. Considering the simplicity, recent efforts are mainly based on the position bias hypothesis, in which the observation only depends the position. However, this hypothesis does not hold in many scenarios due to the neglect the distinct characteristics of individuals in the same po- sition. On the other hand, directly modeling observation bias for each individ- ual is quite challenging, since the effects of each individual's features on rel- evance and observation effect and thus obtain a correct relevance model from click data. To address this issue, we first present the concept of coupling effect for individual-level ULTR. Then, we develop the novel Lipschitz and Bernoulli Decoupling (LBD) model to decouple the effects relevance and observation the individual level. We prove theoretically that our proposed method could recover the correct rel- evance order for the ranking objective. Empirical results on two LTR bench- mark datasets show that the proposed model outperforms the state-of-the-ar base- lines and verify its effectiveness in debiasing data. Our codes available htg.haa..cco//yttoo////pppphhhca
entangled. It is difficult to ravel out this coupled
Introduction
Learning to Rank (LTR) has been widely used in modern information retrieval systems, which aims learn ranking model that sorts documents with their relevance. Recently, using implicit feedback (e.g. clicks) instead of relevance labels (typically obtained by human annotation) to train the ranking model has become popular since implicit feedback an attractive proxy of relevance, which is cheap and relatively easy to obtain on large scale [38 However, they inherently contain lot of bias from user behavior (28] Using Unbiased Learning to Rank (ULTR) remove these biases has attracted increasing research interest 29 The key is factorize the clicks into relevance probabilities and observation probabilities and model a user's observation probability on an item in a ranking list. By reweighing the click signals based on the reciprocal of observation probabilities, ULTR provides an unbiased estimate the ranking objective.
Most of the existing ULTR methods are based on the position bias hypothesis 29 14], which assumes that the observation only depends on the position. We refer this method as group-level ULTR method, in which various heterogeneous features (typically encoded with the
17 3
Equual contribution. Corresponding authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

information of query, document, and user) share the same observation probability according to some property. However, the observation bias inherently varies across individuals. For example, the exact match in result titles and abstracts (which is usually shown in different presentation style) affects users' attention and judgment, known as attractive bias 45 7 Except for attractive bias, there are lots of factors that can influence a user's observation [T 47 37 7 31 32]. These factors are encoded in (or can be inferred from) the features, which leads to heterogeneous observation.
To be in line with these real phenomena, we argue there is an urgent need to develop an individual- level ULTR, which can capture the effect of heterogeneous features for observation behavior, That is to say, the observation model should be function related to documents.
Unfortunately, this problem is very challenging, As shown in Figure [ there exists coupling effect from features to observation and rele- vance. There are infinite solutions to factorize the click rate into the relevance function and observation function (all of them are functions of features), but few of them imply a correct ranking model. In the ex- treme case, a naive ranking model which outputs identical relevance probability still has a chance to perfectly fit the click data with a properly designed association between features and observation [41]. A straightforward solution to avoid coupling is to divide the features manually into two separate parts: one related to relevance and the other related to obser- vation. But it's intractable in practice because some of these features affect both. For example, the exact match in result titles and abstracts affects observation (attractive bias), but on the other hand, it is naturally an important feature encoding how relevant the document is. Besides, result domain 7 and users' search intent (37 are also common factors affecting both.
presentation, quality, click necessity, domain, search intent,
Features (Encoded with query, document user)
Observation (bias)
TF-IDE, BM25, domain. search intent.... ...
Relevance
Click
Figure I: The graphical model when indvidual--eeed observa- tion bias exists.
To address the coupling effect problem, we find that it is unnecessary to correctly estimate the true relevance probability (named hard decoupling). yet only need to recover the correct relevance order from click data (named soft decoupling) because of the ranking objective. Given that we don't 't know the true relevance, this goal is difficult to achieve directly on the ranking model. We instead modify the observation model, which helps to achieve soft decoupling when the click probability estimation is unbiased. We propose two special observation models, the Lipschitz Observation Model (LOM), which has Lipschitz constraint 33 on the observation function, and the Bernoulli Observation Model (BOM) which randomly cancels debiasing operation with a probability tion output. The two models act different mechanisms, thus they are complementary and can be combined. We provide a theoretical guarantee that when 3 and t are bounded, these two observation models can achieve soft decoupling whenever estimating unbiased click probability with a ranking model. Given this, we propose a principled novel individual-based ULTR framework, called Lip- schitz and Bernoulli Decoupling (LBD) model. We conducted comprehensive experiments on two LTR benchmark datasets, which shows that the proposed LBD outperforms the baseline methods and verifies its effectiveness in debiasing data. We further studied the different application areas of the two decoupling techniques and found that the performance can be better when they are combined.
l on the observation func-
To the best of our knowledge, we are the first to study the coupling effect for ULTR and propose feasible solution to decouple the effects. The main contributions of this work are:
1. We present the concept of coupling effect for individual-level ULTR, as well as two targets to achieve decoupling.
2. We propose a novel Lipschitz and Bernoulli Decoupling model, which could help to decouple the effects on relevance and observation. We provide theoretical proof to guarantee the decoupling ability of our proposed techniques.
3. We conducted comprehensive experiments on two LTR benchmark datasets in the coupling effect scene, which shows that the two decoupling techniques have different application areas, and combining them can outperform the baseline methods.

---

2 Related Work
Debiasing Click Data for LTR. Debiasing click data is an important research direction in the infor- mation retrieval field. We review this line of related work in A
Individual-Lewe Observation. There is much work finding that the observation bias can vary from instance to instance. The user's observation can influenced by result type [32,, result quality (IO),
search intent [37], result domain 7 24 exact match [45, and even the relevance itself [1. These
factors can be inferred from document features. Particularly, [44] found that in mobile search, the user may be directly satisfied with the result without clicking any links certain kinds of
documents. 37 argued that users with different search intent might have different examination behavior. All these discoveries suggest the necessity to estimate the bias the individual level.
Recently, researchers attempted to find solutions ULTR fields to deal with individual-level ob- servation bias. 23 proposed to manually impose stronger regularization the position bias estimated by Unbiased LambdaMART, to alleviate the influence of user's different click behaviors, while it's heuristic and has no theoretical proof. [30] proposed wLambdaMART that estimates the confidence of click data with a few labeled data to correct bias, without the need to design a specific bias type. But this method requires extra labeled data. [39] employed heterogeneous treatment ef- fect estimation techniques to estimate the bias between the position and the position 1. However, this method is based the assumption that the click rates at the first position reveal their accurate relevance. This is only true when the observation probability at the first position is constant.
Preliminaries
In this work, we use bold-faced letters to denote vectors, upper-case letters to denote random vari- ables, and the corresponding lower-case letters to denote values. P--) denotes the distribution of a random variable.
Generally, the core of LTR is learn ranking model f. which assigns a relevance score document with its query document features X E R.. Then, the documents for a query can be sorted in descending order of their scores. the true relevance score R to make the ranking order of the query list close to the optimal order (29). In the click setting, users' click logs are used as substitutes of relevance due to their abundance and cheapness. While click signals are naturally biased from the true relevance (27), much prior work follows the examination hypothesis to address it, which can be formulated as:
traditional LTR, the ranking model is directly learned with
(p(x) (x) r(=) ( (p())
where r(x) x
and c,(x) Pr(C X xz P = p) denote the probabilities of relevance, observation and click, and P denotes the position. By explicitly modeling the bias effect via observation probability, it is able to attain an
x). (z) Pr(O = P
unbiased estimate of the ranking objective.
4
Our method
In this section, we first introduce the coupling effect in $ 4.1 and propose two decoupling methods
in 4.2 and 4.3
4.1 Coupling Effect
Let f(æ; 02) denote the estimation of ranking model, and o(æ; 02) denote the estimation of obser- vation model. We use clicks to train the two models using T. to let their production close to the click. What we about is if the relevance model is correct when the model converges. To simplify the later analysis, we suppose that the model has a universal approximation capacity (2 to predict an unbiased click probability, and make the following assumption.
Assumption (Unbiased Click Prediction). There exists a parameter space e which consists of pairs of ranking model parameter and observation model parameter. such thatfor all (0) 02) O", the predictive click probability is unbiased:
fxx: 01) (x; 02) cxp cx()), Va E R2 p E [n].
3

---

From Assumption ! we can find that the factorization of relevance probability f( 08) and ob- servation probability Öp(2; 05) is unidentifiable given the click probability cp(x), which makes challenging to uncover the correct ranking model. This is because the features X have an effect on both relevance R and observation O. It's difficult to distinguish which parts of the features affect relevance and observation, respectively. In the extreme case, the ranking model will degenerate into a trivial case that always predicts the same scores for all documents [41]. We refer to this problem as the coupling effect.
Suppose the correct factorization is cp(x) = r(2))(020)00)) where r(xz) the true relevance and o, (x(2)) is the true observation. To achieve good decoupling of the features' effect on relevance and ob- servation, we aim to make f(: 01) close to r(z) whenever predicting an accurate click probability. Based on this, we introduce the definition of hard decoupling as follows.
Definition (Hard Decoupling). For any features x with the true relevance r(æz), a parameter space O' C 0 can do hard decoupling, if for all (6i, 82) O', we have r(x) f(x; 01). O* is defined in Assumption 1
Considering the pairwise comparison of the ranking objective, we further relax the hard decoupling constraint and refer to this as soft decoupling as follows.
Definition 2 (Soft Decoupling). For any two features œ1 and 2z. without loss of generality we suppose r(x1) 2 r r(x2). A parameter space O' C O can do soft decoupling, if for all (6), 02) e O', we have ;( 01) 2 f(1:2; 0f). . O* is defined Assumption !
Soft decoupling requires identifying the relevance, which is challenging when relevance is unob- served. In this subsection, we propose two mechanisms (Lipschitz. Decoupling and Bernoulli De- coupling) for soft decoupling with theoretical guarantees. From this point forward, we assume that all of the parameters 0 take the value from O*, and omit @ of f(:00) and (:(33) for ease of notation.
4.2 Lipschitz Decoupling
Usually, the rate of change between the observation and features will not be substantial in real scenarios. Therefore, we propose modifying the observation model's smoothness to make it match the true observation function, which helps to achieve soft decoupling.
We start with defining the Lipschitz continuity for function. Let ||-|| be norm on R'. In this work,
we employ the Euclidean norm (L-2) when not mentioned otherwise: |||||| = ||^||| = v Vi x7
Definition (Lipschitz Continuity). A function f(x) RR + IR is L-iischhizz to the input x if
|f(x)) f(x)] al|æx æx|,
where a is referred to as the Lipschitz constant.
Particularly, if an a-Lipschitz function f is differentiable, we have [|V S a. We suppose r(æ)), 0p (2), f()) and 0, (x) are all differentiable.
Assumption 2 (True Observation Lipschitz). For each p E [n], the true observation (x(2) is a- Lipschitz.
assumes that there is an upper bound on the rate at which the true observation changes with the features. This is a reasonable extension for well-known position-based model [29], which assumes that a O. Similarly, we assume the observation estimation is also Lipschitz. We refer to this kind of observation model as Lipschitz Observation Model (LOM).
Definition 4 (Lipschitz Observation Model), An observation model is 3-Lipschitz observation model (3-LOM), if its estimation (x) is B B-Lipschitz for all p [n].
We claim that soft decoupling can be achieved by selecting a lower 3, as the following theorem.
Theorem (Lipschitz Decoupling). For any two features æi and æ2, without loss of generality we suppose r (21) 2 r(x2). For a B-LOM, if the following condition is satisfied,
r(x1)
0p(x1)
inf J. 30p(x)) + =dp(xx) ||xx|
112
r r(x2)
(TEI(e). (x))
x2)
0z(x)

---

where ((x1,22) denotes the collection of all curves from point σ1 to point 22. Then we haver ((x1) 2 f(x2).
Remark 1. the gap between r(æ1) and r(x2)) is large (i.e., r(a1) is large) and is small enough, r(æz)
there exists a 3 such that the condition can be satisfied. Namely, when the true observation doesn't change too fast with the features, we can select lower 3 to achieve soft decoupling.
At a high level, 3-LOM controls the capacity of the function space @* by restricting the Lipschitz constant B. While low / benefits the decoupling, it also suffers from the risk of underfiting. For example, if we adopt a 0-LOM (known as the Position-based Model) in the setting that the true observation has a large Lipschitz constant, we may not find a f(xr) and a 6p(x)) such that their product is equal to given click rate (x))). In other words, 3 has a lower bound. This can be shown formally in the following theorem.
Theorem (Lower Bound of (3). Let c(x) denote a vector containing click rates at each position:
c(x) (r( rioo)
For any two different features 1 and Ø2 we have
[mm((1)) nc(())))))))
B 2 inf m.221
[ x2112
Remark We consider a special situation where the true observation is O-Lipschitz (a = 0). In this case, is no longer a function of but a constant, thus can find m, n such that me(xx)-nc(()) a zero vector, and the lower bound of 3 can be zero. When the true observation has a large Lipschitz number, mc(x1) ne((g) can no longer be zero vector and 3 should be larger. It means the models should change quickly with the feature to fit clicks.
Theorem I and Theorem 2 give the upper bound and lower bound of 3, respectively. The proofs of these two theorems are provided in Appendix $ B Since the bounds focus on the pairwise compar- ison of documents and it's difficult to find the best parameter for the global dataset, in practice we can tune the value of 3 to achieve the best global effect for soft decoupling, We refer to this method as Lipschitz Decoupling.
4.3 Bernoulli Decoupling
Unfortunately, when is large in given dataset, it's difficult to select a proper 3 such that the upper bound and the lower bound are satisfied at the same time. Under this circumstance, it may perform even worse than that without any debiasing operation (see $ 6.3 for discussion). As complementary, we propose another decoupling technique, named Bernoulli Decoupling. Unlike Lipschitz Decoupling, which makes constraints on the observation model's estimation (x; 02), we explicitly add an additional sampling step to transform its output, as follows:
7f ~ Bernoulii - t). (x; 02, = 1 (0p 02) 1)
That is to say, we randomly cancel the debiasing operation with a certain probability (i.e., force the observation model to predict I as an observation score and let the ranking model learn directly from clicks). Since the features are generally collected for the ranking task, the effects from features to relevance are usually stronger than the ones from features to observation. Thus, we hope the relevance model could Iearn more effects from features to clicks than the observation model. The wrong observation model
Bernoulli sampling prevents the relevance model from over-relying and encourages the relevance model to relearn what was learned by the observation model. The probability t can be seen as global parameter to control the strength of relevance effects. Similar to our method, Zhao et al. [46) also uses a dropout technique to prevent the model over-relying on the position. We define the observation model based on this new sampling process as t- t-Bernoulli Observation Model (t-BOM).
For convenience, we denote the expectation B(ö (z:; 02, ()) as the estimation of the observation model after transformation, and Assumption T hollds based on this expectation. We can prove that by choosing sufficiently large I, BOM can achieve soft decoupling:
5

---

Theorem (Bemoulli Decoupling). For any two different features and 22 without loss of gen- erality we suppose r(z2) 2 r( For a BOM, if the following condition is satisfied.
r(
al|æ1 -221-
r(22)
supop(
then we have: f(x2) 2 f(x1)-
Remark 3. If the true observation 0p(2)) is O-Lipschitz (i.e., a = 0), there must exist a t S 1 making the above error condition satisfied to achieve soft decoupling. For a larger Lipschitz number, we can increase the value of appropriately.
Similar to LOM, BOM is another method to control the capacity of O", and it also suffers from the risk of underfitting. order to make the click rates, has an upper bound.
Theorem (Upper Bound of l). For any feature a and a L- BOM, we have
inf (x())
t S sup (x)
Theorem aad Theorem give the lower bound and the upper bound of respectively. The proofs of the theorems are provided in Appendix { Like LOM, we take as a tuning parameter in practice. We refer this method as Bernoulli Decoupling.
Bernoulli Decoupling is a complement to Lipschitz Decoupling, and they can be used simultane- ously to improve the efficiency of decoupling. This is because the range in which the two hyper- parameters take effect is an "or" relationship: Even 3 is outside the scope of Theorem and Theorem [ long is within the scope of Theorem 3 and Theorem [ the combined model can still achieve soft decoupling. This shows that it is beneficial to combine the two models together.
Finally, we propose our (3, t)-iisschitz and Bernoulli Decoupling (LBD) model, which combines the 3-LOM and I-BOM. We treat LBD as our complete solution for individual-basse ULTRR.
Model Implementation
gradient penalty
Lipschitz Decoupling
Bernoulli Decoupling
Figure 2: Framework of the proposed LBD.
Up this point, have shown that (3, f)-LBD effectively decouple clicks relatively low- Lipschitz observation dataset. In this section, we first introduce the implementation of the proposed LBD, then present the optimization objective of the proposed model. The overall model architecture illustrated in Figure 2
We first use neural network f as the ranking model (take features as input) and neural network g as the observation model (take features and position as inputs), and output the relevance and the observation: f = f(x), 0n (x)

---

Lipschitz Decoupling Based on Theorem 1 and we need to enforce a proper Lipschitz con- straint on the observation model employ Lipschitz Decoupling. Since we use a neural net- work to implement the observation model, it's difficult to directly control the Lipschitz constant without adjusting the network structure too much. As an easy alternative, [19] proposed to add penalty on the gradient norm. Similar to them, we employ a gradient penalty loss function:
LipshtyLipschitz
(x) = > > [Vo(6.)]12 This is based on the fact that a differentiable function is k-Lipschtiz if and only if has gradients with the norm at most k everywhers The gradient penalty loss encour- ages the model to have a low Lipschitz constant, and we can adjust À to control the penalty level and obtain models with different Lipschitz constants. Thus, À can be a substitute for B.
Bernoulli Decoupling Based on Theorems and 4 in order to implement the Bernoulli Decou- pling technique and avoid degradation of the debiasing process, we sample y from Bernoulli distri- bution parameterized by t-. Our final logarithmic observation probability generated by log 8, = y log oz, where y ~ Bernoulli(1 - t). The final predictive click score is: log Cp log f + log 6/.-
Objective Function and Training Process Similar to [4] we adopt a list-wise loss based on softmax cross entropy. Given query q with the document ranking list my { "., Fp J. generate click &p for feature Ep at each position p. Suppose the real click signals for each document in n are C1. C2:" "2 Cn. we use the following supervise loss to train our models:
exp(log (4)
Lo) Σ
exp(1og 2.))
Finally, the objective function based on a query q can be written as L(g) Lsw (1) Lap(a)). We jointly update the ranking model and the observation model with the derivatives of L(g) and repeat the process until the algorithm converges.
6
Experiments
In this section, we describe our experimental setup and show the empirical results.
6.1 Experimental Setup
39 and conducted semi-synthetic experiments two widely used benchmark datasets: Yahoo! LETOR [12] and Istella-s [33] provide further details for these datasets in Appendix $ C.I
Dataset We followed the standard setup in CLTR 5
4
15
(33 We
We followed the given data split of training, validation and testing. To generate initial ranking Tists for click simulation, we followed the standard process [29] 4 [5] to train a Ranking SVM model (26) with 1% of the training data with relevance labels, and sort the documents. Based on these initial ranking lists, we sampled clicks according to the examination hypothesis. Following the steps proposed by [3, we set the relevance probability be:
2
Pr(R= = = =xx==++(1 -
29max 1
where 3/2
[0, ymax] is the relevance level of x by human annotation, and Vmax 4 in both of the two datasets. é is the click noise level and we set = 0.1 as the default setting.
To the best of our knowledge, we are the first to study the coupling effects, thus there is no precedent for the observation simulation. In order to simulate the observation, We first select some crux features as the factors that are related to observation, such as result type, result quality or search intent, etc. Because the exact meaning of the features given by the datasets is unknown, we follow the method in previous work [39] [5] to select some features as substitutes: (1) use normalized features and relevances in the total dataset to train ExtRa Trees [18]; (2) based on this model, select and combine the top- 10 important features crux features. We adopted the methodology in [39 to model the feature's 's influence on observation. The observation probability is set to be:
Pr(O
X x,, P= p)
u " fa(=')+1,))
httpe://eebscope.eeet wwwdppp.peeadboooooeeoaooooooo.. waanddbxx.yaho...o hom...cco./aaoa......
http://qu.ckraak..itt.. cnr wttp::ssi...attttttttteeeddddaaa
7

---

where is the position iesition-based examination probability at position by eye-tracking studies [27]] w 10-dimensional vector uniformly drawn from [- where 7 is a hyperparameter to control the dependency between the observation and crux features. x'' denotes the crux features of æ. Note that ry can be explained as the coupling level between relevance and observation.
Baselines We combined the state-of-the-art ULTR methods and two LTR models for comparison.
Debiasing methods include Vectorization [14] RegressionEM [43]. DLA (4) PairDebias [23],
HTE (39), Labeled Data (uses human-annotated relevance labels to train the ranker directly) and Click Data (uses the click data train the ranker directly). Ranking models include DNN and Linear. We adopted the codes in ULTRA framework 13 to implement RegressionEM. DLA, PairDebias and the ranking models, and kept the same hyperparameters. Note that Regression-EM, DLA and PairDebias are all group-level ULTR methods. HTE is an individual-level ULTR, but it's not a jointly learning algorithm, so we compared it in another scene.
Our method is referred to as LBD. We also use three degenerate versions for ablation study: (1) set & = remove the Bernoulli sampling step (named as Lép)) (2) set À = 0 remove the gradient penalty loss function (named as Ber)er), and (3) remove both (named as Unlimited). Training details can be found in S C.2
6.2 Performance on different rankers and datasets
Table 1: nDCG@11 performance of different methods on two datasets (1 = 0.1). Significant performance (p-value < 0.005) improvement/degraatiin compared to the best baseline (except for Labeled Data) by t-test is denoted as +/- Table 4 in the appendix shows the full results.
Dataset
Yahoo!
Istella-S
Ranker
DNN
Linear
DNN
Linear
Labeled Data
0.764
0.747
0.729
0.689
DLA
0.755
0.741
0.696
0.667
Vectorization
0.751
0.741
0.691
0.672
PairDebias
0.739
0.738
0.691
0,678
RegressionEM
0.746
0.720
0.623
0.594
Click Data
0.741
0.737
0.701
0.680
Unlimited
0.752
0.719
0.687
0.592
LBDraps
0.757
0.744
0.686
0.659
LBD Ber
0.754
0.742
0.707
0.679
L.BD
0.758
0.746
0.709
0.680
We first study the performance of jointly learning- ULTR models. Table I summarizes the NDCG@ 10 results about the performance of different rankers different datasets. We that our proposed LBD significantly outperforms almost all the other baseline methods since we properly handle the coupling effect. The results of other baselines compatible with those reported in ( Particularly, we have the following findings:
1. LBD works better than LBD and LBD which indicates that using Lipschitz Decoupling and Bernoulli Decoupling at the same time better than using them alone. The reason seems to be that the two decoupling methods have different conditions (see Theorem and Theorem When using them together, the scope of decoupling can be expanded.
2. Unlimited works worst than almost all the best baselines, indicating the existence of the coupling effect. If we train the models unlimitedly, the performance would be quite bad. Additionally, the coupling effect is more serious when using the Linear ranker.
Lipschitz Decoupling performs better than Bernoulli Decoupling on Yahoo! dataset, while the opposite is true on the Istella-S dataset. It shows that the application of two decoupling methods is tightly related to the data distribution.
6.3 Performance on different coupling levels
In this experiment, we investigate the performance under different degrees of coupling level on the Yahoo! dataset, where the ranker is DNN. We vary from 0 to 0.6 when generating clicks.

---

0.76
0.76
0.75
0.72
0.73
LBD
LBO,er
0.72
PairDesias
0.72
LBD
0.74
LBD-ITE-ORE Orace
RegressionEM
Uniimted
HTE
Click Data
Clock Data
HTE Oracie
0.73
0.0
0.2
04
0.6
0.0
0.4
0.6
0.0
0.2
0.4
0.5
(a) With Baselines
(b) With Decoupling Techniques
(c) With Intervention Methods
Figure 3: Performance across different coupling levels 17. The larger the value of 1/, the stronger the
influence of features on observation. The variance is displayed with the shadow areas.
Figure ((a) shows the influence of coupling levels on different offline ULTR methods. We can observe that our model achieves the best performance under all degrees of coupling level. When the observation is O-Lipschitzz (i.e., 7) = 0), our model and DLA perform similarly, since both of them employ a similar listwise loss function. The debiasing performance of other baselines diminishes with the increased coupling level. For a large coupling level (1) 2 0.3), all basclines perform worse than the raw click data. In contrast, our proposed method always performs better than click data, which shows the strong decoupling ability of LBD. Besides, our method exhibits a downward trend with the increased coupling level, which demonstrates that it's more difficult to decouple the effects when the true observation has large Lipschitz constant.
Figure 3(b) shows the performance of different decoupling methods we propose this work. One can see that for a low degree of coupling level (1) < 0.2), LBDLiips performs better than LBD which implies that LOM can decouple the effects under low Lipschitz observation dataset. For large coupling level, LOM is worse than BOM in turn. One explanation is that BOM can avoid the deterioration of ULTR when the relevance and observation effects are difficult to decouple since randomly drops out the debiasing operation. Besides, LBD always performs better than LBD and LBD Ber. once again showing that using them together could expand the scope of decoupling.
The previous study focuses most on joint-learning ULTR methods. To compare the Heterogeneous Treatment Effect (HTE), we conduct another experiment based on intervention data. Following [39] we simulate 2,560,000 queries for the intervention experiment before training the ranking model (this number is equal the number of total queries we used to train models). For each query, we randomly chose k E [n]] and swapped the document at the first position with the one the k-tth position. We collected clicks generated on these ranking lists and calculated the HTE estimator (z). Based on fx (z)), we debiased the click data and use it to train ranking model directly. also used an accurate HTE estimator Ti (ar) from the click model we used, which can be regarded as the upper bound of the HTE method. Based on the clicks corrected with Th (r( we train a ranking model directly (named HTE--raclee. We also use these clicks to train our model (named LBD-HTE-oracle
We present the results in Figure 3(c) We can observe that our LBD (trained without any intervention data) performs even better than HTE and HTE-oracle in a low degree of coupling level because HTE based on an assumption that the click rates at the first position can reveal the correct relevance, which is biased when the observation probability is not constant. For a high degree of coupling level, HTE performs better than our method. One possible reason is that intervention can augment the dataset (since each document has opportunity to display at the top position), which is helpful when the effects are difficult to decouple. Besides, when our model is trained on the clicks corrected with the oracle HTE, it can always outperform the vanilla HTE method. This demonstrates once again that HTE is still biased and verifies our decoupling effectiveness.
6.4 Hyper Parameter Analysis
From the previous discussion, we know that the hyperparameters should have an upper bound and lower bound depending on the data. To investigate it, we study the performance with different hyper- parameters. Figure shows the results when changing hyper parameters. We can see that there is focal point each heat map, and the grid points around the focal point have lower performance,

---

0.76
0.751
6.756
8.758
0.758
0.732
0.732
2.738
0.736
0.737
0.718
0.726
0.726
0.75
0.75
0.75
0.753
6.754
0.700
0.7
0,750
0.750
0.746
0.746
0.737
0.737
0.732
0.732
0.73
0.73
0.73
0.72
633
0.747
6.748
0.748
6.749
9.747
0.746
0.746
0.745
0.738
0.737
0.738
-0.71
100
10000
100
10000
100
10000
(a)
0.0
(b) = 0.3
(c)
0.6
Figure 4: Performance with different hyper parameters across different coupling levels. A darker color indicates better nDCG@ 10 performance.
which verifies our discussion. Besides, the focal point gradually shifts from the upper right cornrr to the lower-left corner as an increase. This shows again that Lipschitz Decoupling takes better effect on a low Lipschitz observation dataset, while Bernoulli Decoupling performs better on large Lipschitz observation dataset.
7 Conclusion
In this work, we take the first step to studying the coupling effect in individual-level ULTR. We propose two goals: hard decoupling, where the true relevance probability should be estimated ac- curately, and soft decoupling, where the correct relevance order can be recovered. We modify the observation model to achieve soft decoupling when the click probability estimation is unbiased. We propose two special observation models, Lipschitz Observation Model (LOM) and Bernoulli Observation Model (BOM). We provide theoretical guarantee that when their hyperparameters are bounded, soft decoupling can be achieved by these methods. We conduct comprehensive ex- periments two LTR benchmark datasets, which shows that the proposed LBD outperforms the baseline methods and verifies its effectiveness in debiasing data.
Limitations. (1) For simplicity, theoretical results are based on Assumption which requires the models to fit clicks perfectly. It would be interesting to investigate how the decoupling effect will be affected if the click estimation not accurate. (2) Our bounds in the theorems contain value a from Assumption 2 which related to the real data generating process. In some cases such sparse, high dimensional, or high noise cases, the value of alpha may be too large and invalidate the derived bounds. How to decouple data in these cases is an open question.
Societal Impact
To the best of our knowledge, the approaches in this paper raise no major ethical concerns societal consequences. Researchers and practitioners from the ULTR domain may benefit from our research since debiasing implicit feedback possible outcome when the proposed approach fails is that it reduces to the standard position-bar based observation estimation and stops making the desired impact. Finally, the proposed approach aims at solving the coupling effects of the data, the extent of which depends the properties of the data.
significant challenge in real-world applications. The worst
Acknowledgments
Thanks to Yusu Hong for the discussion on the theorems, and the reviewers for their valuable com- ments and suggestions.
References
[1] Aman Agarwal, Kenta Takatsu, Ivan Zaitsev, and Thorsten Joachims. 2019. A general frame- work for counterfactual learning-to-rank. In Proceedings of the 42nd International ACM SIGIR Conference Research and Development in Information Retrieval, 5-14.
10

---

[2] Aman Agarwal, Xuanhui Wang, Cheng Li, Michael Bendersky. and Mare Najork. 2019. Ad- dressing trust bias for unbiased learning-to-rank In The World Wide Web Conference. 4-14.
[3] Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Marc Najork, and Thorsten Joachims. 2019. Estimating position bias without intrusive interventions. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. 474-482.
[4] Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018, Unbiased learning to rank with unbiased propensity estimation. In The s1 International ACM SIGIR Conference on Research & Development in Information Retrieval. 385-394.
[5] Qingyao Ai, Jiaxin Mao, Yiqun Liu, and W. Bruce Croft. 2018. Unbiased Learning to Rank: Theory and Practice. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy) (CIKM '18). ACM, New York, NY, USA, 2305- 2306.
https: wtg.d/o.....g///// 1 11111//26906..2777221
[6] Qingyao Ai, Tao Yang, Huazheng Wang, and Jiaxin Mao. 2021. Unbiased Learning to Rank: Online or Offline? ACM Transactions on Information Systems (TOIS) 39, 2 (2021), 1-29.
[7] Judit Bar-Ilan, Kevin Keenoy, Mark Levene, and Eti Yaari. 2009. Presentation bias is signifi- cant in determining user preference for search results-A user study. Journal of the American Society for Information Science and Technology 60, (2009), 135-149.
[8] Alexey Borisov, Iya Markov, Maarten De Rijke, and Pavel Serdyukov. 2016. A neural click model for web search. In Proceedings of the 25th International Conference on World Wide Web. 531-541.
[9] Alexey Borisov, Martijn Wardenaar, Ilya Markov, and Maarten de Rijke. 2018. A click se- quence model for web search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 45-54.
[10] Georg Buscher, Susan T Dumais, and Edward Cutrell. 2010. The good, the bad, and the random: an eye-tracking study of ad quality in web search. In Proceedings of the 33rd interna- tional ACM SIGIR conference on Research and development information retrieval. 42-49.
[11] Hui Cai,, Chengyu Wang, and Xiaofeng He. 2020. Debiasing Learning to Rank Models with Generative Adversarial Networks. In Asia-Pacific Web (APWeb) and Web-Age Information Management (WAIM) Joint International Conference on Web and Big Data. Springer, 45-60.
[12] Olivier Chapelle and Yi Chang. 2011. Yahoo! learning to rank challenge overview. In Pro- ceedings of the learning rank challenge. PMLR, 1-24.
[13] Olivier Chapelle, Donald Metizer, Ya Zhang, and Pierre Grinspan. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management. 62 1-660.
[14] Mouxiang Chen, Chenghao Liu, Zemin Liu, and Jianling Sun. 2022. Scalar Not Enough: Vectorization-Basse Unbiased Learning to Rank. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '22). 135-145.
[15] Mouxiang Chen, Chenghao Liu, Jianling Sun, and Steven CH Hoi. 2021. Adapting Interac- tional Observation Embedding for Counterfactual Learning to Rank. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 285-294.
[16] Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations.. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 331-338.
[17] Zhichong Fang, Aman Agarwal, and Thorsten Joachims. 2019. Intervention harvesting for context-dependent examination-bian estimation. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 825-834.
[18] Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Ma- chine learning 63, 1 (2006), 3-42.
[19] Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved training of wasserstein GANs. In Proceedings of the 31 International Con- ference on Neural Information Processing Systems. 5769-5779.
11

---

[20] Fan Guo, Chao Liu, Anitha Kannan, Tom Minka, Michael Taylor, Yi-Min Wang, and Christos Faloutsos. 2009. Click chain model in web search. In Proceedings of the 18th international conference on World wide web. 11-20.
[21] Fan Guo, Chao Liu, and Yi Min Wang. 2009. Efficient multiple-click models in web search. In Proceedings of the second acm international conference on web search and data mining. 124-131.
[22] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward net- works are universal approximators. Neural networks 2, 5 (1989), 359-366.
[23] Ziniu Hu, Yang Wang, Qu Peng, and Hang Li. 2019. Unbiased LambdaMART: An unbiased pairwise learning-to-rank algorithm. In The World Wide Web Conference. 2830-2836.
[24] Samuel leong, Nina Mishra, Eldar Sadikov, and Li Zhang. 2012. Domain bias in web search. In Proceedings of the fifth ACM international conference on Web search and data mining- 413-422.
[25] Jiarui Jin, Yuchen Fang, Weinan Zhang, Kan Ren, Guorui Zhou, Jian Xu, Yong Yu, Jun Wang, Xiaoqiang Zhu, and Kun Gai. 2020. A deep recurrent survival model for unbiased ranking. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval. 29-38.
[26] Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 217-226.
[27] Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accu- rately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval.
154-161.
(28) Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, Filip Radlinski, and Geri Gay. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions Information Systems (TOIS) 25, 2 (2007), 7-es.
[29] Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased learning-to- rank with biased feedback. In Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. 781-789.
[30] Jae-woong Lee, Young-In Song, Deokmin Haam, Sanghoon Lee, Woo-sik Choi, and Jongwuk Lee. 2020. Bridging the Gap between Click and Relevance for Learning-to-Pe to--ank with Mini- mal Supervision. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2109-2112.
[31] Yiqun Liu, Chao Wang, Ke Zhou, Jianyun Nie, Min Zhang, and Shaoping Ma. 2014. From skimming to reading: A two-stage examination model for web search. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Manage- ment. 849 858.
[32] Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of verti- cal result in web search examination. In Proceedings of the 38th International ACM SIGIR Conference Research and Development in Information Retrieval. 193-202
[33] Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando. Raffaele Perego, Fabrizio Sil- vestri, and Salvatore Trani. 2016. Post-learning optimization of tree ensembles for efficient ranking. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval. 949-952 9522.
[34] Harrie Oosterhuis and Maarten de Rijke. 2020. Policy-aware unbiased learning to rank for top-k rankings. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 498.
[35] Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva. 2020. Cor- recting for selection bias in learning-to-rank systems. In Proceedings of The Web Conference 2020. 1863-1873.
[36] Zohreh Ovaisi, Kathryn Vasilaky, and Elena Zheleva. 2021. Propensity-Indeeendent Bias Re- covery in Offline Learning-to -ortinggtt--aak- Rank Systems. (2021).
12

---

[37] Yingcheng Sun, Richard Kolacinski, and Kenneth Loparo. 2020. Eliminating search intent bias in learning rank. In 2020 IEEE 14th International Conference on Semantic Computing (ICSC). IEEE, 108-115.
[38] Joachims Thorsten, Granka Laura, Pan Bing, Hembrooke Helene, and Gay Geri. 2005. Accu- rately Interpreting Clickthrough Data as Implicit. In Proceedings of the 28th annual interna- tional ACM SIGIR conference. 154-161.
[39] Mucun Tian, Chun Guo, Vito Ostuni, and Zhen Zhu. 2020. Counterfactual Learning to Rank using Heterogeneous Treatment Effect Estimation. arXiv preprint WKKiv2007.0097990 (2020).
[40] Ali Vardasbi. Maarten de Rijke, and Ilya Markov. 2020. Cascade Model-based Propensity Estimation for Counterfactual Learning to Rank. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Jul 2020). https:
aroi..org/100 33945/3 3401299
[41] Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 1475-1484.
[42] Xuanhui Wang, Michael Bendersky, Donald Metzler, and Mare Najork. 2016. Learning to rank with selection bias in personal search. In Proceedings of the 39th International ACM SIGIR conference on Research and Development Information Retrieval. 124.
[43] Xuanhui Wang. Nadav Golbandi, Michael Bendersky, Donald Metzler, and Mare Najork. 2018. Position bias estimation for unbiased learning to rank in personal search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 610-618.
[44] Kyle Williams, Julia Kiseleva, Aidan C Crook, Imed Zitouni, Ahmed Hassan Awadallah, and Madian Khabsa. 2016. Detecting good abandomment in mobile search. In Proceedings of the 25th International Conference on World Wide Web. 495-505.
[45] Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. In Proceedings of the 19th international conference World wide web. 1011 1011-1018.
[46] Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. 2019, Recommending what video to watch next: multitask ranking system. In Proceedings of the 13th ACM Conference on Recommender Systems. 43-51.
[47] Yukun Zheng, Jiaxin Mao, Yiqun Liu, Cheng Luo, Min Zhang, and Shaoping Ma. 2019. Con- structing click model for mobile search with viewport time. ACM Transactions on Information Systems (TOIS) 37, 4 (2019), 1-34.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] This work is not resource- intensive.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you re using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tEknj0esTQl", "reviewer": "Reviewer_FSFX", "summary": "This paper studies the unbiased learning to rank problem. Different from existing work, the paper first argues that the bias factor should use any existing features, thus can go to individual-level. Then the authors argue that there will be a coupling effect between the relevance tower and bias tower as they share a lot of input signals. Then two simple methods are proposed. The first one enforces a smoothness constraint on the bias model (the so-called “Lipschitz Observation Model”). The second one uses dropout on the the bias model output (the so-called “Bernoulli Decoupling Model”). The paper provides some theoretical analysis on when decoupling is achievable. Experiments are done on semi-synthetic datasets with synthetic generated clicks on public datasets. \n\nOverall the paper can be strengthened with more compelling theoretical analysis and experimental evaluation. \n\nThe reviewer acknowledges the author's reply. The dropout argument is decent. As some of the major concerns are still there (strong assumptions, only on semi-synthetic dataset, smoothness method only works on one dataset (more concerning as the datasets are manually controlled)). Will keep the score as the reviewer was initially between 3 and 4.", "questions": "The paper describes HTE as baseline in the text but it’s not shown in the results?\nOn Istella, “Click Data” can outperform several unbiased methods? \n", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\nThe problem studied is important and the motivation is solid.\nThe theoretical analysis is interesting, though the assumption is strong and the usefulness is unclear.\n\nWeakness\nThe methods are simple (which is good), but not novel. In fact, the dropout method was proposed 3 years ago (“recommending what video to watch next: a multitask ranking system”) but it was not cited. The smoothness method has several concerns (see below) and do not seem to perform well in experiments.\nThe theoretical analysis is not very compelling to the reviewer. The assumption is strong (the authors acknowledged it which is good). For example, the smoothness constraint is not very convincing - the authors already mentioned the position-only case. Also, it is intuitive that different query types can have quite different browsing behaviors, easily violating the smoothness constraint. It only shows feasibility but no guarantees. In practice it’s about hyper-parameter tuning.\nThe experiments are not satisfactory. First, using semi-synthetic dataset is common in the literature, so it does not add extra points (while many people use real-world datasets). Also, the reviewer feels it is more important to test this paper on more realistic datasets. Indeed, as the authors' motivation states, the bias can have complex correlation with input features. Existing work mainly generate bias only depends on position so it could be quite simple, but this work has a different motivation and the way it generates bias can be too simplified. Third, the authors are suggested to compare with some more recent methods (the most recent baseline was in 2019).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "g-yF75ZFMa", "reviewer": "Reviewer_7htM", "summary": "The authors propose a debiasing method in the learning to rank (L2R) regime. They advance the hypothesis that contemporary debiasing methods are lacking since they assume a click model where the effect of the position of an item in a ranked list on the probability of observation by the user is constant across different contexts.\n\nThe authors instead propose learning parametric models for observation probability and relevance probability. Since only clicks are observed in practice and they assume P(click|X) = P(observation|X, position) * P(relevance|X), the two quantities of observation/relevance are not identifiable. As a result the authors instead propose two conditions to fulfill a \"soft decoupling\" condition that maintains ranks in their observational model when compared to the true probabilities of observation.\n1. lipschitz decoupling: where P(observation|X, position) is alpha-lipschitz in X and its corresponding estimation model is also beta-lipschitz in X, for a value of beta that falls within a bound dependent on alpha.\n2. bernoulli decoupling: the observation probability is once again alpha-lipschitz, but here the corresponding estimation model is instead forced to output 1 with probability t, forcing the ranker to learn directly from click data. If t falls within a bound dependent on alpha, then soft decoupling also applies here.\n\nThe authors then validate their model across a linear and neural network ranker and compare to a number of unbiased baselines in semi-synthetic L2R settings.", "questions": "1. Why is the bernoulli model the right model here? While I read the paper the choice felt a bit arbitrary, and rather felt like a hack to enforce soft coupling, rather than reflecting any real data generating process.\n2. Does soft decoupling give us any guarantees about ranking accuracy? For example, if I have a softly decoupled observational model with probabilities that are way off from the true probabilities, can I still have some guarantee about my click probability?\n3. There seems to be a gap between the theoretical and empirical sections of the paper since you tune the beta and t hyperparameters using a validation set. Given that you have access to the true underlying probabilities in the simulation environment, what happens if you explicitly compute the lipschitz bound\n4. I'm confused how you evaluated nDCG in your validation set and test set. I had assumed you only observed click data across all splits of the data, hence shouldn't all your items in each query have a score of 0 except for the clicked item? In that case, wouldn't it be more interpretable to use a metric such as the average position of the clicked item, or the mean reciprocal rank?", "limitations": "Given that the intent of L2R models is for them to be deployed on large internet platforms, I would have liked to have seen some discussion about the societal impact of the work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a good contribution to the area of L2R. The model they study is elegant and natural and I was surprised there has been no prior work making similar assumptions.\n\nOverall the clarity of exposition was very good and the paper was easy to follow. However, I felt that the authors could have provided more intuition in the bernoulli decoupling section (see questions).\n\nWhile the ideas presented are quite valuable I felt the results in the experiments were a bit underwhelming. The authors assume that their hypothesis about disparate effects of rankings is correct in their simulations and only show marginal improvements in nDCG when compared to DLA unless their coupling parameter is significantly increased (the nDCG difference between DLA and LBD are smaller than the differences between LBD and itself for two different values of the hyperparameter t that are apart by 0.1). However, I did appreciate the completeness of their experimental results: the authors included results across a wide range of simulation parameters and compared their method with a broad range of baselines.\n\nOverall, I'm leaning towards an accept although I would be willing to change my score up from a weak accept to an accept if the authors are able to make a compelling case for the strength of their experimental results and address my questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GOhclaIXjrt", "reviewer": "Reviewer_jZaM", "summary": "The authors propose an Unbiased Learning-to-rank (ULTR) method which relies on only clicks and performs decoupling to separate relevance and observation parts from the click model. \n\nGiven that we observe only clicks, it is very difficult to retrieve the true relevance and observation parameters, because multiple combinations of relevance and observation can result in the same click. \n\nThe authors propose a disentangling method to decouple the two parameters, with theoretical guarantees. \n\nEmpirical results on the standard semi-synthetic benchmark demonstrate the efficacy of the method over established baselines. \n\n-- I have read and considered the author's response. ", "questions": "I couldn't find the proof for soft decoupling and hard decoupling in the draft. It points to an Appendix, but I couldn't find one. \n", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Some strong points of the paper:\n\n- The paper is well-written, with a proper introduction to the area of ULTR and biases in click data. Given the limited space in the NeuRIPS format, the authors have done a commendable job of introducing the related works and the problem itself. \n- The assumptions are laid out in an intuitive manner and the method is supported by theory. \n- Empirical results indicate strong performance as compared to baselines. \n\nSome weak points/comments on the draft:\n- During training, when clicks are estimated via a product of predicted relevance and predicted observations, even with the regularization term on the norm of gradient, a trivial solution would be to make the observation model's prediction constant, and relevance equal to clicks. How will the model recover the true relevance in such a case? (even with soft decoupling, it needs to estimate relevance up to a constant). \n\nOverall, I think the paper presents an interesting theoretically motivated algorithm for ULTR. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0FhQmr-Rt2h", "reviewer": "Reviewer_wGHG", "summary": "In this work, the authors studied and proposed a Lipschitz and Bernoulli Decoupling (LBD) model to decouple relevance and observation at individual document level in unbiased learning-to-rank. The authors first explained about the coupling of relevance and observation, as there are common factors affecting both. They then translated the coupling effect into the main assumption of unbiased click prediction and defined both hard decoupling (model decomposes as the ground truth) and soft decoupling (model decomposition preserves the order of relevance function) based on that. Assuming the Lipschitz continuity of the observation function, the authors proved that the soft decoupling could be achieved by enforcing a Lipschitz continuity on the observation model with a finite Lipschitz constant and gave the upper and lower bound. Based on the same continuity assumption, the authors also showed Bernoulli observation model (BOM) can achieve the soft decoupling within the bounds of the observation dropout rate. Finally, the authors described the objective that combines both Lipschitz and Bernoulli Decoupling methods. In the experiment, the authors showed on the synthetic dataset, LBD could achieve the best unbiased results and also did the ablation studies on Lipschitz and Bernoulli alone. With a brief discussion of limitations, the authors concluded the paper. In the Appendix, the authors extended related works, proved the main theorems, and included more experiment details. ", "questions": "Good that authors discussed the limitations on Assumption 1. On the other hand, based on my understanding, both Lipschitz and Bernoulli methods are based on the Assumption 2, i.e., the Lipschitz continuity assumption of observation. In general, I tend to accept the assumption. However, on the other hand, it implies the function is smooth up to certain scale. If we need to learn the smooth function and extend to below the scale, we need in our dataset at least dense enough to cover to the scale, especially, when our observation is now in a much higher dimension space. Can authors discuss this limitation on Assumption 2 as well? And explain the requirements on the dataset, e.g. data size, and relation to the smoothness alpha?\n\nPlease also check my other questions in weakness.", "limitations": "Please see my comments in questions and weakness above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The work studies a general and important but previously ignored issue in unbiased learning to rank, which has been the core of industrial application in recent years.\n2. Well written and solid theory.\n3. Complete study with sufficient ablation study and discussion on limitations.\n\nWeaknesses:\n1. The authors only demonstrated their method on the synthetic dataset, despite the method is claimed to be applicable to more nontrivial bias. It could be more convincing if authors could test the method on real-world click data.\n2. The authors should better double-check their significance analysis. There are some results appearing absurd: e.g. in Table 1, 0.659 of LBD_Lips Linear model on Istella-S is significant worse while 0.592 of Unlimited Linear model is not significant? Also, not sure which exactly the baseline was when applied the significance analysis, why not 0.686 and 0.687 are not significant if 0.707 is significant, assuming the baseline is the best one with 0.701.\n3. In Figure 2(c), when authors claimed about better performance of LBD at low eta. In fact, they only have a single point LBD performs better, which is eta=0, exactly no coupling. Isn't that somewhat deviated from the main claim of the paper. Can authors add some points, for example, 0<eta<0.1, to show LBD does work better at low eta end?\n4. Minor points on notations: a. variable t appears to be doubly used in Theorem 2 and in Bernoulli model parameter; b. Appendix chapters should better always be clarified, somewhat confused when not knowing there is an appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept