Learning Listwise Domain-Invariant Representations for Ranking
Authors: Ruicheng Xian
Honglei Zhuang
Zhen Qin
Hamed Zamani
Jing Lu
Ji Ma
Kai Hui
Han Zhao
Xuanhui Wang
Michael Bendersky
Venue: Submitted to ICLR 2023 (2023)
Keywords: learning to rank
domain adaptation
text ranking

Abstract: Domain adaptation aims to transfer models trained on data-rich domains to low-resource ones, for which a popular method is invariant representation learning. While they have been studied extensively for classification and regression problems, how they would apply to ranking problems, where the metrics and data follow a list structure, is not well understood. Theoretically, we establish a generalization bound for ranking problems under metrics including MRR and NDCG, leading to a method based on learning listwise invariant feature representations. The main novelty of our results is that they are tailored to the listwise approach of learning to rank: the invariant representations our method learns are for each list of items as a whole, instead of the individual items they contain. Our method is evaluated on the passage reranking task, where we adapt neural text rankers trained on a general domain to various specialized domains.

Under review as a conference paper at ICLR 2023
LEARNING LISTWISE DOMAIN-INVARIANT REPRE- SENTATIONS FOR RANKING
Anonymous authors Paper under double-blind review
ABSTRACT
Domain adaptation aims to transfer the knowledge acquired by models trained (data-rich) source domains to (low-resource) target domains, for which pop- ular method is invariant representation learning. While they have been studied extensively for problems including classification and regression, how they would apply to ranking problems, where the data and metrics follow a list structure, is not well understood. Theoretically, we establish a domain adaptation generalization bound for ranking under listwise metrics such as MRR and NDCG, that naturally suggests an adaptation method via learning listwise invariant feature representa- tions. Empirically, we demonstrate the benefits of listwise invariant representa- tions by experiments for unsupervised domain adaptation on real-world ranking tasks, including passage reranking. The main novelty of our results is that they are tailored to listwise ranking: the invariant representations are learned at the list level rather than at the item level.
INTRODUCTION
Learning to rank applies machine learning to solve ranking problems that are at the core of many ev- eryday products and applications, including and not limited search engines and recommendation systems (Liu 2009). The availability of ever-increasing amounts of training data has enabled larger and larger models to improve the state-of-the art on more ranking tasks. A prominent example is text retrieval and ranking, where neural language models with billions of parameters easily outper- form traditional ranking models, e.g., BM25 (Nogueira et al. 2020). But the need for abundant data means that large neural models may not benefit tasks with little could actually fare worse than baselines such as gradient boosted decision trees (Qin et al., 2021).
annotated data, where they
Techniques for extending the benefits of large models to low-resource domains include zero-shor learning and domain adaptation. In the former, instead of directly optimizing for the task of interest with limited data, referred to the target domain, the model is trained a data-rich source domain that has a similar data distribution. The latter considers the scenario where (abundant unlabeled) data from the target domain is available, which can be leveraged to estimate the domain shift and improve transferability, e.g., by learning invariant feature representations. This setting and its algorithms are studied extensively for problems including classification and regression (Blitzer al. 2008 Ganin iet al.] 2016 Zhao et al. (2018). For ranking problems, however, existing methods are mostly limited fact, due to the inherent list structure of the metrics and data, theoretical explorations of domain adaptation for ranking are only nascent.
to specific tasks and applications.
To this end, we provide the first analysis of domain adaptation for listwise ranking via domain- invariant representations (Section building on the foundational work by Ben- David et al. (2007) for domain adaptation in the binary classification setting. One of the results from our theory is that, when the domain shift is small in terms of the Wasserstein distance, ranking model optimized on the source is transferrable to the target domain, whose performance under metrics such as MRR and NDCG can be bounded.
'In this paper, by listwise ranking, we mean that the data are detined over lists and the ranking metric is listwise. It is not a statement of the ranking model, e.g., whether the predicted rank assignments are obtained in a pointwise, pairwise or listwise manner (Joachims) 2006 Cao al. (007). or the interactions between items
in the same list modeled (Pang et al. 2020).

---

Under review as a conference paper at ICLR 2023
Inspired by our theory, we propose an adversarial training method for learning listwise domain- invariant representations, called ListDA (Section 4. that minimizes the distributional shifts be- tween source and target domains in the feature space for improving generalization domain. Unlike traditional classification and regression settings, in ranking, each input follows list structure, containing the items be ranked. The main technical novelty of ListDA is that the invariant representations it learns are of each list as a whole rather than the individual items they contain. Empirically, we evaluate ListDA for unsupervised domain adaptation on two ranking tasks (Section Sjan Appendix C including passage reranking
the target
fundamental task in information retrieval (Craswell et al.) 2019 where the goal rerank list of candidate documents retrieved by first-stage retrieval model in response to search query. We adapt T5 neural rerankers (Raffel jet al., 2020) fine-tuned on the general domain MS MARCO dataset (Bajaj et al., 2018) to two spe- cialized domains: biomedical and news articles. Our results demonstrate the benefits of invariant representations on the transferability of rankers trained with ListDA.
2 PRELIMINARIES
Learning to Rank. A ranking problem defined by a joint distribution over list: X E X of items and nonnegative relevance scores Y (Y Ye) E TR We assume that all lists are length-L, and the ground-truth scores are function of the lists, y (.) so that ranking problem is equivalently defined by marginal distribution μ of lists along with a scoring function y X R
The goal is to train a ranker f X + St that maps each list x to rank assignments r = f(x) e St, where T; represents the predicted rank of item i, and Se denotes the set of permutations on f1,2, ... f). that recover the descending ordering of the relevance scores yr. i.e., y/ > yj -
Ti < r, for all i & j. The more common setup is to train a scoring function h : X + R whose output is list of ranking scores, s.t. s; h(z); correlates with y; in each list, and its ordering agrees that of the tround-truth scores. Rank assignments could be obtained from the ranking scores by taking their descending ordering via probabilistic models (Section 3)
The quality of the predicted ranks is measured by ranking metrics u : S₁ x RS + R20. which are functions that take inputs the rank assignments of the list along with the ground-truth relevance scores and output utility score. Popular listwise metrics in information retrieval include reciprocal rank and normalized discounted cumulative gain (Voorhees) 1999. Därvelin & Kekäläinen 2002):
Definition 1 (RR). Suppose the ground-truth relevance scores E {0, are binary, then the reciprocal rank of the rank assignments r E Se is
RR(r, v) max(r, :11 f, (0}.
The expectation of RR over the dataset is called mean reciprocal rank (MRR), E[RR(J(X), x(X))- Definition 2 (NDCG). The discounted cumulative gain (DCG) and the normalized DCG (with iden- tity gainfunction functions, w.l.o.g.) of the rank assignments E Sq are
and NDCG y) = DCG(r.
DCG(r, y)
log(r:+
IDCG
1=1
where IDCG(y) max DCG(r', y), the ideal DCG, is the maximum DCG value of a list and is attained by the descending ordering of the ground-truth ya 's.
Domain Adaptation. The present work studies the adaptation of a scorer from a source domain (ng ys) to target domain (u² yr). When domain shift is small, i.c.. f μu and ys yr, scorers trained on the source are expected to be transferrable to the target without the explicit need of labeled target data. Indeed, the target performance is bounded by the source performance in such cases. As an example, for binary classification, we have the following generalization bound for randomized classifiers (Shen al. 2018):
AA natural and common choice for the space of lists X' is the f-times Cartesian product of It R meaning that each list (Ir,,..,,d ..,)) a stack of items represented by k-dimensional feature vectors. But generally and more abstractly, the lists need not follow this structure; e.g., A' can also be Bd (not scaling with provided mechanism represent lists by fixed-length vectors.
2

---

Under review as a conference paper at ICLR 2023
Theorem 1.
Let binary classification problems a source and targer domain be given by joint
distributions Jss, pr over inputs and labels (X, Y) E X' x {0.1}. Let F c [0, 1] be a class of L-Lipschitz predictors, and define the enror rate of E by
E(f) == -(xxy)- + Y)]
E(.y) [f() 1(Y + (1 (1-f(x) 1(Y + 0)].
meaning that the output classifications are probabilistic according to P(Й = x) = (xx)
Define X :==:mnn)) -ming (Es + Er(")), then for all F,
ET()) S Es(J) + 2L W Wi(S .TT) + X"
where Sxx denotes the marginal distribution of the input X.
The domain shift in Theorem is measured by the Wasserstein-1 distance between source and target marginal input distributions, whose Kantorovich-Rubinstei dual form is given by (Edwards) 201
Definition 3 (Wasserstein-1 1). Let p.q be probability measures on a metric space (X, dx), their Wasserstein-I distance is Wi(p, q) supyelıp()) (Jx ((x) dp(xx) - Sx fx) dg(x)).
Where the supremum is taken over 1-L-ipschizz functionals :
Definition 4 (Lipschitz). Let (X, (X', dx)) be metric spaces. A function f X + X' is L- L-Lipschitz, denoted by f Lip(L), if dx (f())) S Ldx(F1, for all I1, E X.
3 DOMAIN ADAPTATION GENERALIZATION BOUND FOR RANKING
We establish the first domain adaptation generalization bound for ranking problems under listwise ranking metrics. Specifically, we consider the setting of learning scoring functions and (trans- ferrable) representations, where the end-to-end scorer f = h o g is composition of a shared feature map X' > E and a scoring function h Z + R on the learned list representations. For instance, if the end-to-end scorer an m-layer MLP, could treat the first (m 1) layers as g and the last as h.
For the bound, we let the rank assignments Se be generated from the output scores ho)(
probabilistically via Plackett-Luce model (Plackett) [975] Luce 1959). with the exponentiated
scores exp(s;) as its parameters (Cao al. [N07/Guiver & Snelson 2009).
Definition (P-L model). A Plackett-Luce model with parameters u IR specifies a distribution over Sj, and the probability mass function, denoted by pe. is defined for all E St by
(r)
II
Us(r)
Ej= vr(r),
where I(r); the index of the item ranked i, so that TT(r); = ii
With the above probabilistic procedure of generating rank assignments, the performance (or utility) of a scorer I w.r.t. ranking metric S2 x 20 > Ix is evaluated by
E()) = Exxxxx max u(r, x(X) E R rES;
(n [u(R, [())
which computes its suboptimality relative to the maximum attainable utility. The randomization of the rank assignments using the P-I. model is analogous to that of the classifications (using the Bernoulli model) in Theorem D and its purpose, together with the exponentiation, is to make the ranking metrics continuous w.r.t. to the raw scores output by the model.
Our analysis, however, differs from that of Theorem due to difficulties arising from the list structure of listwise ranking metrics. For instance, in the realizable setting, the tightness of Theorem Thinges on the uniqueness of the optimal classifier. The optimal ranker on ranking problems, in contrast, is generally nonunique. Consider list with scores y (1, 1, 0) as example: the maximum utility is attained by both rank assignments r = (1,2,3) and (2, 3). To proceed, we require the following I. Lipschitz assumptions.
3

---

Under review as a conference paper at ICLR 2023
The ranking metric u Se B220 -+ +R2o is bounded by B and (Euclidean) Lu- Lipschitz in the second argument- the ground- truth relevance scores y.
Assumption 1.
Assumption 2. The ground-truth scoring function X - RR20 is Lng-Lipschizz (Euclidean on the output space).
We will show that RR and NDCG satisfy Assumption II Assumption 2 says that similar lists (i.e., close in X) should have similar ground truth scores. It is satisfied, for instance, when X' is finite and the scores are bounded (this argument used in Corollary 33; this setup is typical with text data, where the inputs are sequences of one-hot vocabulary encodings.
Assumption 3. The space of input lists X' is metric space, and the class H of scoring functions h: 2 + R is Ln -iiischhit (Euclidean on the output space).
Assumption 4. The feature space Z is metric space, and the class G of feature maps - Z such that, Vig G. the restrictions of g to the supports of p and μ²² Ilsupp(uz) and g glsupp(uk) respectively, are both invertible with Lipschhhh inverses.
Assumption 3 is standard generalization and complexity analyses and could enforced with e.g. Il-rreularizzttin (Anthony & Bartlett 1999 Bartlett et al.) 2017). The last assumption is technical, which says that the original inputs are recoverable from their feature representations via Lipschitz g meaning that the feature map g should retain as much information from the inputs each domain. Note that this assumption does not hinder domain-invarian representation learning; as long as G is sufficiently expressive, Eg satisfying 2 147
We are now ready to state our domain adaptation generalization bound for learning to rank:
Theorem 2. Under Assumptions 7 1 for any E G, define Xy == mins (Es(W & then for all E H.
Er(hog) S g) + 2(2L, + BL Vt) W1 14) + Xz.
where 2 denotes the marginal distribution of the feature 7, n. (x) (0 (z)).
The bound says that if the features computed by are domain-invariant, i.c., μ = μzz and the optimal joint risk X remains low, then scorer h optimized for the source domain will also perform well on the target. Thii suggests that domain adaptation for ranking can be achieved via listwise invariant representation learning, where we optimize the feature map g using (unlabeled) source and target domain data to minimize Wi (ug. /T) by aligning the list feature distributions, and simulta- neously optimize h on the source domain with labeled data. We propose such method in Section 4 Indeed, generalization bounds of the type of Theorems [] and [ (originally derived by al. (2007)) form the basis of family of domain adaptation methods based on invariant represen- tation learning, which has seen empirical success in fields ranging from vision (Zhao et al. 2022) to language (Ramponi & Plank 2020
The key distinction of our bound is that the features Z are defined on each list X as a whole, rather than the individual items contained in the lists. To illustrate, suppose the setup where the feature representation of each list is stack of E k-dimensional vectors, g(x) = (u1,"- Mz) E R and u, E _* corresponds to the i-th item in the list. For invariant representation learning, we aim for the distributional alignment of not just the items but also the lists, 2/ μ Concretely, denote the distribution over item feature vectors by n² (u) == Pz~zz (u E 2), then 3 = - u4 but the converse is not true! We demonstrate this point empirically in Section 5 that listwise invariant representations are more appropriate for domain adaptation for listwise ranking, where the data and metrics all follow list structure, compared to the pointwise method of learning invariant representations of items.
ng
Lastly, to instantiate our bound on MRR and NDCG, we simply verify their Lipschitzness:
Corollary (Bound for MRR). RR is 1-Lipsehitz in y, thereby
[RR o g)] 2 Es RR (h g)l 2(22 + LAVO). W1 W133 H2Z) Xy. where for brevity we wrote E|RR (h g)] Exaux BRoeuuRRRPeuuee R
[RR( R, (X))]
4

---

Under review as a conference paper at ICLR 2023
Corollary 4 (Bound for NDCG). Suppose Unin & IDCG(y) $ Umax for some Umins Umax (0, and all y E ys (supp(() )) U yr (suppp(u then NDCG Lipp)-Lipschitz in y, thereby
En NDCG o g)و 2 (D - g)] - (LV + Ln)) W W1(u3, X.x
where for brevity we wrote BINDCG(h g)] == E xx=xxx B.Ceaa,,B--.aaa (--aaptho (NDCG(R,
4 LEARNING LISTWISE IIVAAIN-IIARIANT REPRESENTATIONS
As discussed in Section g Theorem 2 suggests that domain adaptation for ranking could be achieved with listwise invariant representations via learning a feature map g E 9 that minimizes the distri- butional shifts between source and target on the feature space 2, measured by Wi Wi(ug, 2z). Specifically, we consider the setup where the feature representation Z := g of each input list containing l items, ( .zt), , is the stacking of l feature vectors, i.e., 2 = R and each 2 := (u...... .ue) where u, E R is the learned feature vector of the i-th item in the list. This setup is standard in many learning to rank implementations, e-g., in neural text ranking, each feature vector is an embedding of the input text computed by a language model (Guo et al. (020).
Learning invariant representations is similar to generative modeling, and a well- technique in GAN literature is adversarial training (Goodfellow et al. 2014; Ganin et al. 2016). which solves a minimax problem max, min Lad (g. faa) of two players- the feature map g and an adversary fau : z + R. The objective is defined with an adversarial loss function lat : IR x {0, 1} + I, whose inputs are the adversary output á == (2) and the domain identity a (set to 1 for target):
Lgd fua) == Easu (fa g(x),, 0)] + E [fad 9(x),
The adversarial loss corresponds to probability metries between 2Š n under specific choices of ldd (Goodfellow et al. 2014 Arjovsky al. 2017). With the 0-1 loss of la a) (1 (1-a) a) 1 2 0) +a- i( < 0), Lud becomes the (balanced total) classification error of faa as a domain discrimina- for on predicting the domain identities, (g fui) = T -n-uz (faa o g(x) 2 0) + z~ (fus o og(z)) <
0), and it gives an upper bound on W₁ Wi 1&4) under optimality of saa:
Proposition 5. Denote the metric on Reck by d, and define B == (up( xnz) d(z, If
la is the 0-1 loss, then (1 n$) < B(1 - min Jaa))
In practice, the 0-1 loss replaced by a surrogate loss when training Saa to minimize the classifica- tion error, for which we use the logistic loss in our experiments:
la a) = log(1 + el 22)))).
Finally, by parameterizing the function class Ful 3 fui, e.g., neural networks, the minimax problem can be optimized with gradient descent-ascent (w.r.t. g and f respectively), implemented with a gradient reversal layer top of g (Ganin et al., 2016). To prevent g from converging to trivial solutions like x + and causing information loss in the learned features (thereby increasing the minimum achievable Eg and Xz). g is optimized together with the scorer h under a joint objective:
Cjoine(hs g)
min hEH.gE9
(Crmn g) > min Laa Sma) SuEFF
where Crank is ranking loss of choice (surrogate the ranking metric), and À > 0 hyperpa- rameter controlling the strength of domain-invariant feature learning.
Choosing Fus. The only missing piece is the choice of the discriminator function class. Unlike prior work, however, the distributions n3 nu being modeled here are defined not over items but rather lists (or sets, to be more precise), z (w,, ... ur). hence common choices e.g. MLP may not be appropriate. Indeed, a possible design choice is to flatten each list into single @k-dimensional vector and feed into an MLP discriminator, but does not capture the permutation -omuutation-inaaiancce prop- erty of the lists list and its permutations inator under this design, despite them being identical as far as the ranker is concerned. Without inmuutttioniinvaiiann built-in, the optimization of sas is data inefficient.
perceived as distinct inputs by the MLP discrim-
5

---

Under review as a conference paper at ICLR 2023
Quaryy kind bear best??
Gradient Reversal
Transformer Discriminator
Document There are basically two..
Mean-pool
Document Bears eal beets, Bears...
Predicted
Document Allbaar
great..
domain identity
Target
T5 Encoder
BM25
Dense
List of retrieved documents
List of embeddings
Ranking scores
Figure 1: Block diagram of ListDA instantiated on the cross-attention T5 ranker for text ranking.
Therefore, tailored to list-like inputs, we use transformers (no positional encoding) with mean- pooling for Fad as novelty in our implementation (Vaswani et al. 2017). which permutation- invariant, continuously differentiable, and has good expressive power. Fig. includees block dia- gram of our method instantiated on the RankT5 model Section referred as ListDA.
EXPERIMENTS ON PASSAGE RERANKING
In this section, evaluate ListDA the passage reranking task, where given text query, the goal is to rank candidate passages in a retrieved set based on their relevance to the query. In Appendix C we additionally evaluate ListDA on the ranking task from the Yahoo! Learning to Rank Challenge. Reranking is employed in scenarios where the corpus is too large for all (millions of) documents be exhaustively ranked by more accurate but expensive models such as SOTA cross-attention rankers based on language models; rather, a simpler but efficient first-stage model such sparse or dense retrievers e.g. BM25 and DPR is used to retrieve a candidate set of (hundreds or thousands of) passages (Robertson & Zaragoza 2009 Karpukhin et al. 2020), whose ranks are then refined and improved by a more sophisticated reranking model.
We consider unsupervised domain adaptation. Training neural rerankers requires large amounts of queries and secument-relevaann annotations. While such data can be obtained relatively casily from search engines under weak supervision for general text domains, annotations specialized domains such as scientific literature are costly to acquire. Since unannotated documents are almost always readily available, this makes for suitable candidate apply unsupervised domain adaptation: spe- cialized rerankers are adapted from ones trained on general domains.
Models. We use BM25 as the first-stage retriever for simplicity and focus on the adaptation of the reranker-a a attss-attnntion model based on T5 Base with 250 million parameters (Zhuang et al.) 2022). Given a query q and documents dj, ... de retrieved by BM25, the list formed by concate- nating the query and each document (with the title, if available), = ([g, di], [g, d.). [g, de)). We follow the setup in Section 4 and use encoder as the feature map g. so that the feature representation of the list is the first-token output embeddings that T5 computes
each query- document (q-d) pair pair = (W,,... ue) = (T5(x1),... T5(xz)) E R 1024 The ranking scores then obtained by projecting each q-d embedding by dense layer, (h(u))... .h(ue)). We train the ranking model by minimizing the listwise softmax cross-entropy ranking loss:
exp(si)
Crank (s. v) = Σ y/s log
E =1 exp(s;))
ListDA adversarial training also follows Section The discriminator faa is stack of three transformer blocks with the same architecture those of the T5 encoder. Given list feature z = (u,,-.. ... Uz), we obtain its domain prediction by feeding all vectors through the transformer blocks at once as a sequence, taking the mean-pool of the outputs and projecting a logit with
Thee terms document, text, and passage are used interchangeably here. Under our setup, the feature vectors are computed on each item independently, but generally and ideally would also model the interactions hetween items the same list (Pang al. 2020

---

Under review as a conference paper at ICLR 2023
Table 1: Transfer performance of T5 reranker on top 1000 BM25-retrieved passages.
Target Domain
Method
MAP
MRR @ 10
NDCG@5
NDCG@ 10
NDCG@20
BM25
0.2282
0.6801
0.4396
0.4088
0.3781
Zero-shot
0.2759
0.7977
0,5857'
0,5340*
0.4856'
Robust04
QGen PL
0.2693
0.7644
0,5406
0.5034
0.4694
TeeDDA
0.2822
0.8037
0,5822
0.53961
0I.4922
LisIDA
0.2901
0.8234
0,59791t
0,5573
0,5126
BM25
0.2485
0.8396
0,7163
0,6559
0.6236
7ero-shot
0,3083
0.9217
0.8328
0.8200
0.78226
TREC-COVID
QGen PL
0.3180
0.8907
0,8373
0.8118
0.7861
ItmDA
0.3087
0.9080
0.8276
0.8142
0.7697
ListDA
0.3187
0.9335
0.8693
0.8412
0.7985
BM25
0,4088
0,5612
0,4580
0,4653
0,4857
Zero-shot
0,5008
0,6465
0,5484
0,5542
0,5796
BioASQ
QGen PL
0.5143*4
0.6551
0.5538
0,5643#
0.5915
IteDDA
0.4781
0.6383
0,5315
0,5343
0.5604
ListDA
0.5191
0.6666
0.5639
0,5714
0.5985
Source domain is MS MARCO. Gain function NDCG is the identity map. 'Improves upon zero-shot base- line with statistical significance S 0.05) under the two-tailed Student's t-test. 'Improves upon QGen PL. 'Improves upon IteDAA.
dense layer. We ensemble of five discriminators in (Elazar & Goldberg 2018) reduce the sensitivity to the randomness the initialization and the training process, Σi=1 Cad(g, f(i)..
Datasets. The source domain in our experiments MS MARCO for passage ranking, a large- scale dataset containing 8 million passages from the web that covers a wide range of topics, and 532,761 search query and relevant passage pairs (Bajaj et al. 2018). The target domains are biomedi- cal (TREC-COVID, BioASQ) and news articles (Robust04) Voorhees et al. [02 Tsatsaronis et al.)
2013
Voorhees 2005). The data are collected and preprocessed as in the BEIR benchmark (Thakur 2021); their paper also contains statistics of the datasets.
al.
Recall from above that the inputs to our cross- attssntt model are q-d pairs. However, there are no training queries available two out of three target domains, so following (Ma et al. 2021), we synthesize training queries on all target domains in a zero-shot manner, via a T5 XI. query generator (QGen) trained MS MARCO relevant q-d pairs. QGen synthesizes each query seq-to-seq task given a passage from the target corpus as input, whereby we expect the synthesized queries to be related the input passages. See Table for samples of QGen q-d pairs.
Baseline Methods. We compare ListDA to three baseline methods. In zero-shot learning, the reranker is trained on MS MARCO only and directly evaluated on the targets. In QGen PL, we treat target domain QGen synthesized q-d relevant pairs and train the reranker and QGen q-d pairs (PL as in these q-d pairs are "pseudolabeled" by QGen). This method underlies several recent works on domain adaptation of text retrievers and rerankers (Ma al.) [021] Sun ct al. 2021 Wang et al. 2022). All adaptation methods, including ListDA, are applied on the target domains separately, i.e., we train a model for each source-target pair.
both MS MARCO
Lastly, prior work that applies invariant representation learning to domain adaptation all performs feature alignment at the item level (Cohen al., 2018 Tran et al., 2019 Xin et al.) (022) instead of the list level (our ListDA). Yet, the key message of this paper is that the former not suitable for listwise ranking. To verify and demonstrate our claim, we also compare ListDA to ItemDA, which learns pointwise invariant representations with three-layer MLP discriminator (no improvements from going larger) whose inputs are the q-d embeddings individually (similar to the DANN model by Ganin et al. (2016)).
Due to space constraints, experiment details including hyperparameter settings and the construc- tion of training example lists are relegated to Appendices [B.1] and B.2] We also include case studies and additional results with the pairwise logistic ranking loss and the hybrid method of ListDA QGen PI. in Appendix

---

Under review as a conference paper at ICLR 2023
bust00
Robust04
Robust04
0.600
0.600
0.59
0.575
0.575
0.58
0.57
TREC-COVID
CRVID
TREC.COMID
ND0085
0.875
0.875
0.86
0.850
0.850
0.825
0.84
0.825
BloASO
BloASQ
BloASQ
0.56
0.56
0.56
0.54
0.54
1000 3000
0.001
0.01
10
20
Number of target queries
A
Oud
K/fron
Figure 2: ListDA under different target sizes. Lower grey horizon- tal line is zero-shot, upper red line is ListDA using all QGen queries.
Figure 3: ListDA performance under different hyperparame- ter settings for X and Tad- Grey horizontal line is zero-shot. On the left, 7ad 0.004 is fixed and À varies. On the right, À 0.02 is fixed and nad varies.
5.1 RESULTS
The main results are presented in Table We report metrics are that are commonly used in the literature (e.g.. TREC-COVID uses NDCG@P @ 20), and evaluate rank assignments given by the de- scending ordering of the ranking scores. Since TREC-COVID and Robust04 are annotated with 3-level relevancy. the scores binarized for mean average precision (MAP) and MRR as follows: for TREC-COVID, 0 (not relevant) and I (partially relevant) are negative, 2 (fully relevant) is posi- tive; for Robust04, 0 (not relevant) is negative, 1 (relevant) and 2 (highly relevant) are positive.
Across all three datasets, ListDA has the best performance, and the fact that it uses the same resource as QGen PL demonstrates the benefits of invariant representations. Furthermore, the favorable com- parison of LisDD to ItemDA supports the discussion in Section 3 that for domain adaptation for listwise ranking, the invariant representations the model learns should be of each list as a whole, and not of the items individually.
Quality of QGen. An explanation for why QGen PL underperforms ListDA despite sharing re- sources is that the negative sampling of irrelevant q-d pairs could lead to false negatives in the train- ing data. [Snn et al. (202T) observed that queries synthesized by QGen lack specificity and could be relevant to many documents. While QGen PL trains the QGen pseudolabels and the randomly sampled irrelevant documents by treating them as ground truth, they are not assumed by ListDA, which is thereby less likely be affected by false negatives false positives-wher synthesized queries turn out to be irrelevant to the input passages (see Table 9 for samples). While out of the scope, improving query generation should boost the performance both QGen PL and ListDA.
5.2 ANALYSIS OF LISTDA
Size of Target Data. Unsupervised domain adaptation requires sufficient unlabeled data, but not all domains have the same amount: BioASQ has 14 million documents (also the total number of QGen queries, as we synthesize one per document), but Robust04 only 528, and TREC-COVID 171,332. In Fig. 2 we plot the performance of ListDA under varying numbers of target QGen Robust04 and TREC-COVID,
queries (also the number of target training lists). Surprisingly, using just~100 ~100 target QGen queries (0.03% and 0.06% of all, respectively) is sufficient for ListDA to achieve full performance! Although the number of queries is small, since 1,000 documents are retrieved per query, the total number of distinct target documents is still substantial up to 100,000, or 29.5% and 60.7% of the respective corpora. Performance begins drop when reduced to -10 queries, capping the number of documents at 10,000 (2.7% and 5.8%, respectively). The same data efficiency, however, is not observed on BioASQ, likely due to the hardness of the dataset from e.g. the extensive use of specialized biomedical terms (Tables 7 to 9)
Sensitivity to Hyperparameters. ListDA introduces two main hyperparameters for the discrimi- nator fat the learning rate Nad and the strength of invariant feature learning X. We plot in Fig. 3 the
8

---

Under review as a conference paper at ICLR 2023
sensitivity of their settings by fixing one and varying the other. It observed that balanced choice of À is needed to elicit the best performance from ListDA, but the same choice largely works well across datasets. We set 7ad to be multiples of the reranker learning rate Trank and the results show that each dataset prefers different settings Vad. probably due to their distinct domain characteristics.
RELATED WORK
Learning to Rank and Text Ranking. Traditional learning to rank concerns tabular datasets with numerical features for which a wide array of models are developed (Liu 2009). ranging from SVMs (Joachims) 2006), gradient boosted decision trees (Burges) 2010). to neural rankers (Burges eet al. 2005] Pang et al. 2020) Qin et al., (02T). Another research direction is the design of ranking losses (surrogate to ranking metrics), which are categorized into pointwise, pairwise, and listwise
approaches (Cao et al. 2007: Bruch al. 2020. Zhu & Klabjan 2020 Jagerman et al. 2022a).
Recent advances in large neural language models have spurred interest in applying them on text
ranking tasks (Lin et al. 2022). leading cross-attention models (Han et al. 2020; Nogueira &
Cho Nogueira et al. [2020] Pradeep et al. 2021) and generative models based on query likeli-
hood (dos Santos al. 2020, /Zhuang & Zuccon 2021 Zhuang et 2021 Sachan et al. 2022). A
different line of work is neural text retrieval models, which emphasizes efficiencc, including dual-
encoder (Karpukhin et al. 2020 Zhan et al. 2021), late-interaction (Khattab & Zaharia
2020 Hui
al. 2022). and models based on transformer memory (Tay et al. 2022).
Domain Adaptation. Following (Ben-David et al. 2007 Blitzer et al., 2008), a family of do-
main adaptation methods is based learning (adversarial) domain-invarian feature representa- tions (Long et al. 2015 Ganin et al. 2016, Courty et al. 2017). These methods are applied fields including NLP and tasks ranging from cross-domain sentiment analysis, question-answering (Li eet al. 2017| Vernikos et al. 2020). to unsupervised cross- cross-lingual learning and machine transla- tion (Xian et al. 2022] Lample al. 2018). Our method also belongs to this family, but to the best of our knowledge, no prior work considers learning domain-invariant representations of lists/sets.
Recently, /Zhao al. (019) point that for classification, Theorem 1 admits lower bound un- der perfect feature alignment and source accuracy when there are distribution shifts class priors between source and target domains. Their result does not apply to ranking problems, and we leave investigations on the effects of shifts in marginal distributions of relevance scores to future work.
Domain Adaptation for Information Retrieval. Work on this subject is categorized into super- vised and unsupervised domain adaptation. The former assumes access to labeled source data and (a small amount of; few-shot) labeled target data (Sun al. 2021). The present work focuses on the latter, only assuming access to unannotated target documents. Cohen et al.) (2018) apply invariant representation learning to unsupervised domain adaptation for text ranking, followed by Tran et al.) (209)) for enterprise email search, and Xin et al. (2022) for dense retrieval. However, their ap- proach learns invariant representations of items and not lists. Another family of approaches is based query generation (Ma et al. 2021 Wang al. 2022). initially proposed for dense retrieval.
7 CONCLUSION
We analyze domain adaptation for ranking theoretically and establish a generalization bound un- der listwise ranking metrics. Our bound leads to an adaptation method based on learning listwise domain-invariant representations, called ListDA, which is empirically demonstrated to be effective for unsupervised domain adaptation on various ranking tasks.
The novelty of our results is that they are tailored to listwise ranking, where the data the metrics all follow a list structure. A key message from our theory and experiments is that when applying invariant representation learning learned should be at the list level rather than at the item level. We believe our theoretical and empirical contributions provide foundation for future studies on domain adaptation for ranking.
domain adaptation for ranking problems, the representations

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations. Cam- bridge University Press, 1999.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In Proceedings of the 34th International Conference on Machine Learning, pp. 214-223, 2017.
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COm- prehension Dataset, 2018. arXiv: 1.09668 [cs.CL].
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalizeed margin bounds for neural networks. In Advances in Neural Information Processing Systems, 2017.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of Representations for Domain Adaptation. In Advances Neural Information Processing Systems, 2007.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning Bounds for Domain Adaptation. In Advances in Neural Information Processing Systems, 2008
Sebastian Bruch, Shuguang Han, Michael Bendersky, and Marc Najork. A Stochastic Treatment of Learning to Rank Scoring Functions. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 61-69, 2020.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to Rank using Gradient Descent. In Proceedings of the 22nd International Conference Machine Learning, pp. 89-96, 2005.
Christopher J.C. Burges. From RankNet to LambdaRank to LambdaMART: An Overview. Techni- cal Report MSR-TR-200 TR-2010-82, Microsoft Research, 2010.
Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning, pp. 129-136, 2007.
Olivier Chapelle and Yi Chang. Yahoo! Learning to Rank Challenge Overview. Journal of Machine Learning Research: Workshop and Conference Proceedings, 14:1-24, 2011.
Daniel Cohen, Bhaskar Mitra, Katja Hofmann, and W. Bruce Croft. Cross Domain Regularization for Neural Ranking Models using Adversarial Learning. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1025-1028, 2018.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamoniy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, 2017.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 Deep Learning Track. In Proceedings of the Twenty-Fighth Text Retrieval Conference, 2019.
Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. Be- yond [CLS] through Ranking by Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1722-1727, 2020.
D.A. Edwards. On the Kantonovich-Rubinstin theorem. Expositiones Mathematicae, 29(4))387~ 398, 2011.
Yanai Elazar and Yoav Goldberg. Adversarial Removal of Demographic Attributes from Text Data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 11-21, 2018
10

---

Under review as a conference paper at ICLR 2023
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. lnverinad Training of Neural Net- works. Journal of Machine Learning Research, 17(59):1-35, 2016.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Infor- mation Processing Systems, 2014.
John Guiver and Edward Snelson. Bayesian inference for Plackett-Luce ranking models. In Pro- ceedings of the 26th International Conference on Machine Learning, pp. 377-384, 2009.
Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. A Deep Look into neural ranking models for information retrieval. Information Processing & Management, 57(6), 2020.
Shuguang Han, Xuanhui Wang, Mike Bendersky, and Mare Najork. Learning-to-A -aainngttt--akkk with BERT in TF- Ta-Ranning, 2020. AXXV:2004.084700 [cs.IR].
Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ma, Jai Gupta, Ci- cero Nogueira dos Santos, Yi Tay, and Donald Metzler. ED2LM: Model for Faster Document Re-ranking Inference. In Findings of the Association for Computa- fional Linguistics: ACL 2022, pp. 3747-3758, 2022.
to Language
Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. On Optimizing Top-K Metrics for Neural Ranking Models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2307, 2022a.
Rolf Jagerman, Xuanhui Wang. Honglei Zhuang, Zhen Qin, Michael Bendersky, and Marc Najork. Rax: Composable Leamning-toRaan Using JAX. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 3051-3060, 2022b.
Kalervo Järvelin and Jaana Kekäläinen. Cumulated Based Evaluation of IR Techniques. ACM Transactions Information Systems, 20(4):422 446, 2002.
Thorsten Joachims. Training Linear SVMs Linear Time. Proceedings of the 12th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 217-226, 2006.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6784 6781, 2020.
Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextu- alized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Confer- on Research and Development in Information Retrieval, pp. 39-48, 2020.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised Machine Translation Using Monolingual Corpora Only. In International Conference on Learning Representations, 2018
Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-End Adversarial Memory Network for Cross-domans Sentiment Classification. In Proceedings of the Twenty- Sixth Interna- fional Joint Conference on Artificial Intelligence, IJCAI-17, pp. 2237-2243, 2017.
Chen Liang, Haoming Jiang, Simiao Zuo Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models. In International Conference on Learning Representations, 2022.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates, Pretrained Transformers for Text Ranking: BERT and Beyond. Springer International Publishing, 2022.

---

Under review as a conference paper at ICLR 2023
Tie- Yan Liu. Learning Rank for Information Retrieval. Foundations and Trends@ in Information Retrieval, 3(3):225-33) 2 2009.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Ma- chine Learning, pp. 97-105, 2015.
R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. Wiley, 1959.
Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. Zero-shot Neural Passage Retrieval via Domain- borain-t Synthetic Question Generation. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1075-1088, 2021.
Rodrigo Nogueira and Kyunghyun Cho. Passage Re-ranking with BERT, 2020. arXiv:1901.04633 [cs.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document Ranking with Pretrained Aequence-to-Sqqueeee Model. In Findings of the Association for Computational Lin- guistics: EMNIP 2020, pp. 708-718, 2020.
Liang Pang. Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. SetRank: Learning cnttnnttationnnvvrrian Ranking Model for Information Retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 499-508, 2020.
Robin L. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193-202, 1975.
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The Expando-Mono-Duus Design Pattern for Text Ranking with Pretrained Sequence-to te-quueneee-eeeeee Models, 2021. arXiv:2101.05667 [cs.IR].
Zhen Qin, Le Yan, Honglei Zhuang. Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Ben- dersky. and Mare Najork. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees? In International Conference on Learning Representations, 2021.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An Optimized Training Approach to Dense Passage Re- trieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp.
2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer. Journal of Machine Learning Research, (1(140):166 2020.
Alan Ramponi and Barbara Plank. Neural Unsupervised Domain Adaptation in NLP -A Survey. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6838-6855, 2020.
Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Be- yond. Foundations and Trends@ in Information Retrieval, 3(4):333-389) 2009.
Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving Passage Retrieval with Zero-Shot Question Genera- tion, 2022. arXiv:2204.07450 [cs.CL).
Shai Shalev-Shwartz: and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representa- tion Learning for Domain Adaptation. In Proceedings of the Thirty- -Scond AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp. 4058-44440 4065, 2018.
12

---

Under review as a conference paper at ICLR 2023
Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, and José Esquivel. A Data Collection for Evaluating the Retrieval of Related Tweets to News Articles. In Advances in Information Retrieval, pp. 780-786, 2018.
Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang. Jie Bao, Zhiyuan Liu, and Paul Bennett. Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5030-5043, 2021.
Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer Memory as a Differentiable Search Index, 2022. urXiv:2202.0699 [cx.CL].
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Thirty- Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Brandon Tran, Maryam Karimzadehgan, Rama Kumar Pasumarthi, Michael Bendersky, and Donald Metzler. Domain Adaptation for Enterprise Email Search. In Proceedings of the 42nd Interna- fional ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 25-34, 2019.
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenbom,, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio- Alvers. Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. An overview of the BioASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16(1):138. 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, tukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural In- formation Processing Systems, 2017.
Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, and Ion Androutsopoulos. Do- main Adversarial Fine-Tuming as an Effective Regularizer. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp- 3103-3112, 2020.
Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushmant William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. ACM SIGIR Forum, 54:1-12, 2021.
Ellen M. Voorhees. The TREC-8 Question Answering Track Report. Proceedings of the Eighth
Text Retrieval Conference, pp. 77-82, 1999.
Ellen M. Voorhees. The TREC Robust Retrieval Track. ACM SIGIR Forum, 39:1 1--11-0,,100 2005.
Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 2345-2360) 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrie Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Ar Natural Language Processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 2020.
Ruicheng Xian, Heng Ji, and Han Zhao. Cross- Lingual Transfer with Class- Weighte Language- Invariant Representations. In International Conference on Learning Representations, 2022.
13

---

Under review as a conference paper at ICLR 2023
Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. Zero- Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 4008-4020 2022.
Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the Use of Lucene for Information Retrieval Research, In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1253-1256, 2017.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing Dense Retrieval Model Training with Hard Negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 1503-1512, 2021.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, Joao P. Costeira, and Geoffrey J. Gordon. Adversarial Multiple Source Domain Adaptation. In Advances in Neural Information Processing Systems, 2018.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey Gordon. On Learning Invariant Representations for Domain Adaptation. In Proceedings of the 36th International Conference on Machine Learning, pp. 7523-7532, 2019.
Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen Wu, Ravi Krishna, Joseph E. Gonzalez, Alberto L. Sanatiovanni-Viceenelle Sanjit A. Seshia, and Kurt Keutzer. A Review of Single- Deep Unsupervised Visual Domain Adaptation. IEEE Transactions on Neural Networks and Learning Systems, 33(2):473-493, 2022.
Xiaofeng Zhu and Diego Klabjan. Listwise Learning Rank by Exploring Unique Ratings. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 798-806, 2020.
Honglei Zhuang. Xuanhui Wang. Michael Bendersky, and Marc Najork. Feature Transformation for Neural Ranking Models. In Proceedings of the 43rd International ACM SIGIR Conference Research and Development in Information Retrieval, pp. 1649-16522 2020.
Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. RankTS: Fine-Tuning T5 for Text Ranking with Ranking Losses, 2022. 2rXiv:2210.10633 [cs.
Shengyao Zhuang and Guido Zuccon. TILDE: Term Independent Likelihood moDEI for Passage Re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1483-1492, 2021.
Shengyao Zhuang, Hang Li. and Guido Zuccon. Deep Query Likelihood Model for Information Retrieval. In Advances in Information Retrievl,, pp. 463-470, 2021.
14

---

Under review as a conference paper at ICLR 2023
A OMITTED PROOFS
Before proving the generalization bounds for binary classification (Theorem T and ranking (Theo- rem 2 recall the following properties of Lipschitz functions:
Fact 6 (Properties of Lipschitz functions).
1. f >> is differentiable, then (Euclidean) L-Lipschitz if and only if fV S L.
2. If = - TR. is I-Iiipsciitz and g : X - is M-Lipschitz, then of + by is (la) I+ (b|M)) Lipschitz, and max(f,9) 9) is max L, M --Ippscchit.
3. If X' + y is L-Lipschitz and y + Z is M- M-Lipschitz, then g o f is LM- -Lpishhhhitz
Proof. For the first statement, suppose bounded gradient norms, then by mean value theorem Er [0, s.t. f(y) f(x) fz(= (y 2) with 2 (1 +)z + ty, so by Cauchy-Schwarz
(1f(y) - f())lle S (=))(y) - 1/22 S L zl|2.
Next, suppose L-Lipschitzness, then by differentiabilityy vf(x) T = = f(x + - f(x) + o(11=112)
Set = = (x), we have
{ = f( +VV f()) + ((V](())) s (((lV//)))
and the result follows by dividing both sides by fx)]22 and taking +00.
For the second,
(af(x) bg(x)= (af(y) + bo(v))
S |ellf(()) - f(w) + (b]l(()) - g(v)l = (la|L + [6/M)dx(z v). Next, assume w.l.o.g. max(f(=),g(zz) - max ((v)) 2 0, then
Imax(f(x), - max(f())
[f(z) - max(f()). s f(x)- - -f(y)= s Ldx(xx.) if max((()),g(z)) = f(x)
g(()) max(f(y) S 5g(z) - -g(v) s Mdx(I, else
I mL, M))
For the third, dz(g (xx go (w)) < (d))) (()) < LM,x(s,y).
We first prove Theorem D as it shares the same organization of the arguments with Theorem 2
Proof of Theorem
Define random variable 17 == 1(Y 1). then E(f) Exx In (2n
1x)( Note that
E(f) E(f') = E(x,)))m(17 (2n- - [x) - B(x,Y) - (2n - (22
= E(x,))~ (21) - 1) (f - f(X))
I Ex-px [I) f(x)]
because (2n- - 1) = +1. On the other hand,
Ex-pxx ( mx (277- 1) (f(X) - -7
< Ex (22) 1)f(X) - -nl] + Exxr)~~ -2n 1)f((X) + m
(2n
-(x,y)= (2n - 1)f(x)) + -x -2n 12n-1)n(p [x)(
=E(f)) + E(().
Then by Fact 6 the fact that taking absolute value is 1-Lipschitzz and Definition 3 for all fif' F, (f) = Es(() + (Er(f) (f() - (Es(f) + (f') + +Es(())
<E((s) + (Ex~ux (1(()) - ('()I] E x [[(X) ((x0)) ((S())))))(()+++((((())
< (E(())+ + sup
(Ex (x(X) - Ex=ax (()))) (Es())
vELip(2L)
<Es(f) + 2L Wi xx) + (Es()') + Er)("))
and the result follows by taking the min over f'
15

---

Under review as a conference paper at ICLR 2023
Next, we prove Theorem The main idea behind the proof is that under our setup and assumptions, we could write Eg and ET as expectations of Lipschitz functions of Z ~ pg and م²z respectively, so by Definition their difference can be upper bounded by Wi mG))
While omitted, Theorem 2 can be extended to the cutoff version of the ranking metric u with simple modification of the proof. Also, a finite sample generalization bound could obtained using e.g., Rademacher complexity, assuming Lipschitzness of the end-to-end scorer (Blitzer et al. 2008
Shalev-Shwartz & Ben-Davidl 2014))
Fix g G, which has L,pLisschitz inverse g 1 to supp(uş) by Assump- tion 4 Define en.g Z + Rz0 for any given h 2 Re by
Proof of Theorem 2
(2) == max u(r, rESt
(=)) Bn-Posp(A(ss)
[u us o (x))
exp(h())(()))
max u(r, 9 (=))
E g
II
rESt
I=1 Σj exp((()))m))
rESe
and note that Es(h g) = Ex leh (g( =: Ez (2)] An analogous analysis holds for Er.
We show that et.g as written above is Lipschitz function of if Lipschitz. For the first term, because is Lu-Liscchitz in ys og 11gg1(1 Lipschitz in and so is max u(r, ys og (z)) by Fact Now we bound the second term. We show that it is Lipschitz in both Us (2) =: y and h(2) =: s under the Euclidean distance. By Jensen's inequality,
(=) (=) and ys (z) is Ly -Lipscchii 2, so is Lu Lng-
V PR-Pesc)
[u( R, v))ll2 ||nnpasp(s
V (R, y)lll2 S ER (a)p(e)
S T.a.
Next,
V. Σ u(r v) II exp($1(r)a)
V Er Respis
(u (R,
S BBV
Ej= exp(31(r)))
reSS
k
where the inequality is due to product rule and
exp(s1(M)))
Σ
u(r, v)
@si(r)
j= exp(s1(M)))
PESee
exp(s1(m)))
exp(ss(m)
Σ u(r,y) v) rESt
a (8)()) Ej= (x==xxxp(ss0)))
j== exp(ss(())
1- exp((1())))
exp($1()))
Σ u(r v)
Σ 1(m ii)
ン== exp((1()))
exp(sro
rESe
exp(s1(m)))
B
B; Σ == exp(81(0)))
rESt ==1 1=1 k=1
dz, recall that softmax (x); (x) j) softmax(x)j). Suppose h E Lip(La), then
2 + Enspeopeaed [u( R, ys o 9- (z)) is Lipschitz because
E Eanpesotite (RR,ys g (2)) Enopappee)
/u (R (2\\)]
< E ER-Pesptcco) |u( R ys (x))) E Rnm [u( ys g (
E R ER-pooonnn (R, ys oggggg (z')) ER-POEIME [u(R, ys 9 (2')))
< L, lys o (2) - us . BVE(k(2)) h(22))l2
(Lu + BLA (()xx
16

---

Under review as a conference paper at ICLR 2023
Putting everything together, Ch is (LLuLyLg + BL, Li)-Lisshits in for any h E Lip(Ln).
Then by Fact and Definition for all E 9 and h, h E H.
( o g) (h g) + (Er Er o g)) - (Es(h g) Es(h' og)
+ (Es(h' 9) + Er og)
s o + (Er(h - (n' o g)) - (Es(h 9) Es(h'og))
(Es(hl o g) + & og)
Es(h + Ez~ (en. lz.nglen. (2) en' .0 (2) E3 z~ (2) (2)
(Es(h' o g) + Er o))
{E(hhog)+
sup
(Bz~uzl9(2)) Ez~nz [g(())
(Es(h' og) &r og) Es(h - g) + 2(2L,, + BL, Vt) (M5 14) + (Es(h' og) - Er o g)),
and the result follows by taking the min over
Finally, we verify the Lipschitz conditions for RR and NDCG.
Proof of Corollary 3 fact that RR < 1 and yy v'll2 2 1 for all y, y E 10, , *
It suffices to verify that y + RR(r, y) is 1-Lipschitz, which follows from the
Proof of Corollary
It suffices to verify that
+ NDCG(r, = DCG(r, v) IDCG(y)
yi
y/
log(r; + 1)
log(r: + +
is Lipschitz. Note that IDCG(v) max, DC) a max of continuous functions, is piecewise continuous where each piece defined by an E Sy: {y:r arg max, DCG(r, v))
Let r E St, and y E TR s.t. arg max, DCG((', v) = arg max, DCG((', ,0') =: r", i.e., they are on the same piece for IDCG. Then
a NDCG(r, v) Oyk
IDCG
θ
y/
DCG(r,))
Oyk Σ
Yi log(r; + 1)
Oy:
log + 1)
IDCG(y) log(rs 1) + DCG v) log(ri 1)²|
IDCG 1og(2) + DCGG(r v) log(f + 1)
< U-1 + Uma log(! 1)²
so NDCG is Vi(Um1 min + U max log(E (f + Lipschitz by combining the above and the fact that IDCG
is continuous.
Proof of Proposition First,
/
W1 W1
inf d(z,z')d7(zz)) SB B inf
1( + =')d7(z,2))
Ter(n3 2x2
rEr(ugNNG) 2x2
1 sup
((2 = 2') =2)22(2))))
TET(Ng (z4) 22z
1
min (pš(2), (1),, xz)))) dz
B
B
max (0 (z) (z(()) dz 2
(z)z) m²((2) dz,
17

---

Under review as a conference paper at ICLR 2023
because -z) 0. Note that the last term is the total variation between p3.
On the other hand, define (z) = ((fa(2) 2 0). Then the balanced total error rate of Y on predicting the domain identities is
Em(() | ( (1-)) (+
de =1 / (P(2) - 1 (1š(2) (15(2)-14(=)) dz.
This quantity is minimized with y²(x) - 11 (1)( 2 uz(2)) (z)). whereby
Emr(Y')= - 1 2 |uī(z) - (=) dz < 3 wi
The result follows from an algebraic rearrangement of the terms.
B ADDITIONAL EXPERIMENTS ON PASSAGE RERANKING AND DETAILS
In this section, additional experiment results for unsupervised domain adaptation on the passage reranking task considered in Section [ are provided, along with case studies on ListDA vs. zero- shot and QGen PL (Tables [ and 99. hyperparameeter settings (Appendix B.1 and details on the construction of training example lists (Appendix B.2).
Pairwise Logistic Ranking Loss. We experiment with the pairwise logistic ranking loss in place of the listwise softmax cross-entropy loss (Eq. 4 used in Section []] defined as
exp(8;) exp(s;) exp(s;)
(rank(s, v) = I(y: Uj) log
The results on the Robust04 dataset are provided in Table 2a Since the pairwise logistic loss does not perform better than softmax cross-entropy (cf. Table I also see (Jagerman al. 2022b)), further experiments with this ranking loss are not pursued.
As implementation remark, in this set of experiments, we do not perform pairwise comparisons during inference to obtain the predicted rank assignments due to the high time complexity (this high complexity is avoided during training due to training list truncation, see Appendix [B.2). However, our theory is applicable to any (black-box) model that produces list level representations, and our ListDA method is also more generally applicable any models whose list level representations stackings of feature vectors, i.e., = (u1. U2, ... ur.) and uz. E RR& (here I. & f, and can he arbi- trary). For instance, while not pursued in this work, ListDA could be instantiated on DuoT5 (Pradeep) et al. 2021), ), SOTA pairwise ranking model, by treating the stackings of pairwise q-d;-d, embed- dings the list feature representations (thereby L l(e- 1)/2).
ListDA QGen PL Method and Signal-1M Dataset. We experiment with supplementing ListDA with QGen pseudolabels by (uniformly) combining the training objectives of ListDA and QGen PL methods (ListDA + QGen PL). The results on the three datasets considered in Section are included in Table 2b This method is also applied on the Signal-IM (RT) dataset (Suarez et al. 2018. with results Table 3 It is noted that reranking using neural rerankers transferred from MS MARCO source domain do not give better performance than BM25 on Signal- 1M, which is also observed in prior work (Thakur al., 2021 Liang et al. 2022). This does not mean that neural rerankers are worse than BM25, but that MS MARCO is not good choice for source domain when Signal-IM is the target, because of the arguably large domain shift between tweet retrieval and MS MARCO web search qualitt-iaatttiiellyy it can be seen from Table 7 that the text styles and task seman- tics of the two domains are very different. Hence, the following discussions on Signal- 1M results focused reranking models.
On Signal- QGen PL improves upon the zero-shot baseline but ListDA does not, which is likely due the large domain shift between MS MARCO and Signal-IM that prevented ListDA from finding the correct source-target feature alignment without supervision. With QGen pseudolabels, improved I ListDA performance is observed with + QGen PI., which could have benefited from QGen
18

---

Under review as a conference paper at ICLR 2023
Table Transfer performance of T5 reranker on top 1000 BM25-retrieved passages; in addition to Table 1 results.
(a) With pairwise logistic ranking loss place softmax cross-entropy on Robust04.
Target Domain
Method
MAP
MRR@10
NDCG@s
NDCG@10 10
NDCG@20
Zero-shot
0.2656
0.7894
0.5671
0.5163
0.4729
Robust04
QGen
0,2776
0.7975
0.5576
0.5267
0.4892
ItemDA
0,2766
0.8021
0.5794
0.5355'
0.4917
LisiDA
0.2893"1t
0.8103
0.5935
0.5524
0.5044
Source domain is MS MARCO. Gain function in NDCG is the identity map. "Improves upon zero-shot baseline with statistical significance (p I 0.05) under the two-tailed Student's f-test. "Improves upon QGen PL.. 'Improves upon TremDA.
(b) With ListDA + QGen PL domain adaptation method.
Target Domain
Method
MAP
MRR@ 10
NDCG@5
NDCG@
NDCG@ 20
Rohust04
0
0.8039t 0.8950
0.5761
0.5386
0.4975
TREC-COVID
LisIDA +
QGen PL
0,3168
0.8539
0.8292
0,7820
BioASQ
0.6538
0,5158
0.5547F
0,5671
0,5931
Source domain is MS MARCO. Gain function in NDCG is the identity map. "Improves upon zero-shot baseline with statistical significance (p < 0.05) under the two-tailed Student's 's t-test. 'mprrvves upon QGen PL. *Improves upon ItemDA.
Table 3: Transfer performance of T5 reranker top 1000 BM25-retrieved passages on Signal- -M.
Target Domain
Method
MAP
MRR@ 10
NDCG@s
NDCG@
NDCG@20
BM25
0.1740
0.3639
0.3215
0.2905
Zern-shot
0.1511
0.4804
0.3068
0.2685
0,2410
Signal-IM
QGen
0.1541
0.5043
0.3238
0.2799
0.2497
ListDA
0,1456
0.4629
0.3002
0.2602
0.2328
ListDA QGen PL
0.1549
0.5170
0.3261
0.2817
0,2505
Source domain MARCO. Gain function in NDCG identity map.
q-d pairs acting anchor points for ListDA find the correct correspondence between source and target.
Overall, ListDA + QGen PL is the only method that consistently improves upon the zero-shot base- line on all four datasets including Signal- M, although underperforns ListDA on the other three. Further improvements this method may be possible with better strategies of balancing the stituent training objectives of ListDA and QGen PI..
Case Studies. In Tables 8 aand we include examples where the ranks (top-1) predicted by models trained with ListDA have higher utilities v.s. zero-shot and QGen PL results, respectively, to provide qualitative understanding of the benefits and advantages of ListDA.
In zero-shot learning, the reranker is trained on MS MARCO general domain data only and directly evaluated on the target domains. When the target domain differs from MS MARCO stylistically or consists of passages containing domain-specifi words, as in the TREC-COVID and BioASQ datasets for biomedical retrieval, the zero-shot model, which barely exposed to the specialized language usages, may resort to keyword matching. Examples of such cases are presented in Table where the top passages returned by the zero-shot model contain keywords from the query are irrelevant.
In QGen PL, the reranker is trained on the pseudolabels generated during the query synthesis proce- dure, treating the passages from which the queries are generated relevant. However, as remarked in Section because QGen is deployed in a zero-shot manner, the pseudolabels are not guaranteed to be valid, meaning that they could be false positives. This is observed in presented in Tables and One specific scenario where false positives hurt transfer performance is when the synthesized queries (of false positive documents) coincide with queries from the evaluation set, which is indeed
19

---

Under review as a conference paper at ICLR 2023
Robust04
Ro Rubust04
0.58
0.565
0.560
0.56
0.555
TREC-COVID
TREC-COVID
NDOG
NDCG@5
0.825
0.800
0.81
BioASQ
BioASO
0.55
0.53
0.54
0.53
0.52
0,00005
0.0001
0,0002
0,00005
0.0001
0.0002
Onnwc
(a) With softmax cnoss-entropy ranking loss.
(b) With pairwisc logistic ranking loss.
Figure 4: Zero-shot performance under different hyperparameter settings for Trank
the case on TREC-COVID and BioASQ in Table Since ListDA does not assume the pseudolabels in its training objective, it does not suffer the same pitfalls QGen PL.
B. HYPERPARAMETER SETTINGS
We discuss model hyperparameter settings for our passage reranking experiments in this section.
For BM25, on MS MARCO source domain, and J1 = 0.9 and b = 0.4 on all target domains without tuning. As in (Thakur al. 2021,, titles are indexed as separate field with equal weight the document body, if available.
the implementation of Anserini (Yang al. 2017). set h 0.82 and 0.68
For the T5 reranker, the model is fine-tuned from the T5 v1.1 Base checkpoint on a Dragonfish TPU with 8x8 topology for 100,000 steps with batch size of 32 (each example is a list containing 31 items) per domain. We tune the learning rate Trank E (5e-5, le-4, 2e-4), and select the one that gives the best zero-shot performance to use on all models for each dataset (see Fig. for zero-shot sweep results). We apply learning rate schedule on Tran that decays (exponentially) by a factor of 0.7 every 5,000 steps. Each concatenated query-document text input are truncated to 512 tokens.
For the domain discriminators, there are two hyperparameters: the strength of invariant feature learning X, and the discriminator learning rate 1aad We tune both by sweeping À E (0.01, 0.02), and Tad E: (10, 20, 40) x Traak. multiples of the reranker learning rate (see Fig. [ in Section for ListDA sweep results). The tuned hyperparameter settings for Trank, 7ad and À used in our experiments are included in T
As remark running time, ListDA and ItemDA take roughly the same time train QGen PL because the overhead of the domain discriminators is small. Compared to zero-shot, they have double the training time due data loading: the adaptation methods trained on target domain data in addition to source domain ones.
B.2 TRAINING EXAMPLE LIST CONSTRUCTION
Recall from the description in Section 5 that in listwise ranking, the inputs are defined over lists and the invariant representations for domain adaptation are learned at the list level. In other words, the ranking loss and the adversarial loss need to computed on ranking scores and feature representa- tions that the model outputs on lists of documents (containing both relevant and irrelevant ones) for each query.
Under our reranking setup, each list would be the top-r documents retrieved by BM25 query, and we set r = 1000 in our experiments. However, there are two complications. The first that due to memory constraints. it is not feasible to feed all 1000 documents from each list simultane- ously through the T5 encoder during training. The second is that the source domain MS MARCO dataset only contains annotations of one relevant document per query, meaning that out of the 1000
20

---

Under review as a conference paper at ICLR 2023
Table Hyperparameter settings of TT reranker and domain discriminators.
(a) With softmax cross-entropy ranking loss.
Target Domain
Method
Rusk
Rad
Zero-shot
QGen PL
Robust04
IteDA
lc-4
1c-3
0.01
ListDA
4e-3
0.01
ListDA QGen PL
Ic-3
0.02
Zero-shot
QGen
TREC-COVID
IteDA
2e-4
8c-3
0.01
LisIDA
4e-3
0.01
ListDA + QGen
4e-3
0.02
Zero-shot
QGen PL
BioASQ
ItemDA
20-4
4e-3
0.01
LisIDA
8e-3
0.01
ListDA QGen
4e-3
0.02
Zero-shot
QGen PL
Signal-IM
5e-5
ListDA
2e-3
0.01
ListDA + QGin
1c-3
0.02
(b) With pairwise logistic ranking loss.
Targel Domain
Method
fad
Zero-shot
QGen PL
Robust04
lc-4
ItemDA
2e-3
0.01
ListDA
20-3
0.01
documents retrieved by BM25 for each query, would only know that one of them is relevant; the ground-truth relevance scores for the remaining 999 documents is unknown.
Example List Construction with Negative Sampling. To address both issues, truncate the list to length e = 31 during training, and perform random negative sampling from the BM25 results to gather irrelevant documents.
On the MS MARCO source domain, given a query and top 1000 documents retrieved by BM25, ...... d1000, we construct the example list with the document d labeled as relevant in the dataset along with 30 randomly sampled documents ".. .N treated as irrelevant, and get (lq, d=), [g, dn,,..... [9, dN and y (1,0.-...,
For the target domains, we perform the same procedure. Given a QGen synthesized query and top 1000 documents retrieved by BM25, the example list consists of the pscudolabeled document (i.e., the document with which the query was synthesized) and 30 randomly sampled irrelevant doeu- ments, so that æ = ([g, d), [g, dv,,..... [q, do and y = (1,0, y are used by QGen PL but discarded by ListDA.
, ,0). Note that the pseudolabels
Reducing MS MARCO False Negatives. One potential problem that arises with negative sam- pling is that the constructed lists may contain false negatives (i.e., relevant documents that are incor- rectly marked as irrelevant); fact, false negatives are prevalent in the MS MARCO dataset. While such false negatives are relatively harmless for source domain supervised training because the true positive document to which they are similar is labeled and trained on so that the effects are can- celed out, they have larger impact source-target feature alignment for invariant representation learning.
One potential negative impact is that the duplicates in the lists (which have identical feature vectors) will cause ListDA to collapse distinct documents on the target domain to the same feature vector for achieving alignment, which is artifact that will cause information loss in target domain feature
21

---

Under review as a conference paper at ICLR 2023
Table 5: Transfer performance of 3-layer MLP ranker on Yahoo! LETOR (Set 2).
Target Domain
Method
MAP
MRR@ 10
NDCG@S
NDCG@10
NDCG@2 20
Zero-shot
0,5138
0.6558
0.7302
0.7627
0.8199
Yahoo! LETOR (Sct 2)
Supervised
0,5389
0.6785
0.7523
0.7829
0.8341
ItemDA
0.5315
0
0.7402"
0.7708
0.8255'
ListDA
0,5370
0.6771
0.7442
0.7735
0.8269
Source domain Yahoo! LETOR (Set 1). Gain function in NDCG the identity map. Bold indicates the hest unsupervised result. Results are from ensembles of five models. "Improves upon zero-shot baseline with statistical significance (p < 0.05) under the two-tailed Student's f-test. 'Improves upon ItemDA. Sig- nificance tests are not performed on supervised results.
representations. Another is that the inclusion of duplicates will alter the marginal distribution of scores on the source domain al. 2019).
To lower the chance of selecting false negatives on MS MARCO, we rerank each BM25-retrieved results using a ranker that is pre-trained on MS MARCO, and sample negatives from documents that are ranked at 300 or higher, since the duplicates and relevant documents will be concentrated at the top (Qu al. (02I). We only apply this method when constructing source domain lists for feature alignment (namely, in ListDA and ItemDA), and it only affects the computation of adversarial loss. This sampling method not applicable to target domains because we do not have reliable pre-traine trained rankers for them. It is also not used for source domain supervised training (i.e., the computation of ranking loss), as decreased performance was observed our preliminary experiments with this method, likely due to the exclusion of "hard" negatives.
C EXPERIMENTS ON YAHOO! LETOR DATASET
Our method is also evaluated on the ranking task from the Yahoo! Learning to Rank Chal- lenge v2.0 (Chapelle & Chang 2011), referred to as the Yahoo! LETOR dataset. This is a web search ranking dataset in numerical format, where each item is represented by 700-d vector with values in the range of [0, 1]. It has two subsets, called "Set I" and "Set 2", whose data originate from the US and Asian country respectively. Among the 700 features, 415 are defined on both sets (shared), and the other 285 are defined only on Set I or 2 only (disjoint); we hence write each item x == aahardd Fdisjoint] as concatenation of shared features Tshared E R-15 and disjoint ones Edisjoint E 2225
We consider unsupervised domain adaptation from Set I to Set 2. Our code is implemented with the Hugging Face Transformers library (Wolf et al. 2020
Models. Our models has the same setup that of the passage reranking experiments in Section 5
except that the TS reranker replaced by haree-hidden-layer MLP following (Zhuang et al, 2020). and we treat the list of 256-d outputs on the second layer the feature representations.
g(x); u; ReLU Wa RRLLU(W,ReLU(W) JG shared bb) b2)
+ 6y
ReLU((WRRLLUWW disjoint + b1) + bb)
M(x); = Wa4: + ba:
where W, TR. 1024x1115 Wi 10244x285 W2, W/ 255xx1024 W3 R 2565512 and W4
R k2x2 all randomly initialized.
Each model in the ensemble of five domain discriminators is a stack of three T5 encoder transformer blocks, with 4 attention heads (num_heads)) size-32 key, query and value projections per attention head (d kv). and 1024 intermediate feedforward layers (d ff).
Considering the small dataset size and number of training steps, each method is evaluated by an ensemble of five separately trained models to reduce the variance the results due to the randomness in the initialization and the training process. Since Yahoo! LETOR is annotated with 5-level relevancy, the scores are binarized for MAP and MRR metrics by mapping 0 (bad) and 1 (fair) to negative, and 2 (good), 3 (excellent), 4 (perfect) to
Results.
The results are presented in Table 5
22

---

Under review as a conference paper at ICLR 2023
Table 6: Hyperparameter settings of 3-layer MLP ranker and domain discriminators.
Target Domain
Method
Thakk
Thd
Zero-shot
4e-4
Yahoo! LETOR (Set
Supervised
4e-5
ItemDA
2e-4
4e-4
LisiDA
2a-4
1.6e-3
positive. Because of the availability of labeled data Set also include Supervised results, where the model is trained on labeled data from both Set 1 and 2.
The best unsupervised transfer performance achieved by ListDA, particular, the favorable com- parison of ListDA to ItemDA again corroborates our discussion in Section 3 that listwise invari- ant representation learning more appropriate for listwise ranking, although their gap is smaller compared to passage reranking results (Table [ which we suspect is because the weak contex- defining the structure on data too weak in this numerical dataset. Lastly, thanks to the availability of labeled data, we also provide Supervised results where the model is trained on both Set 1 and 2, serving as an upper bound for ListDA.
tual (query) information
Hyperparameters. The model is trained from scratch NVIDIA A6000 GPU for 5,000 steps with batch size of 32 (each example is a list containing one to no more than 140 items) per domain. We apply learning rate schedule on Thank that decays (exponentially) by a factor of 0.7 every 500 steps.
We exhaustively tune ranker and domain discriminator learning rates and the strength of feature alignment from Trank (le-5, 2c-5, 4e-5, 8c-5, Ic-4, 2c-4, 4c-4, 8c-4, 1c-3), nad e (0.2, 0.4, 0.8, 1, 2, 8, 10) Trank and X E {0.01, 0.02, 0.04, 0.08, 0.1, 0.2). The tuned settings are included in T
23

---

Under review as a conference paper at ICLR 2023
Table 7: Samples of test set relevant q-d pairs and QGen synthesized q-d pairs from domains sidered in Section 5 "I..."".
passage reranking experiments. Truncated or omitted texts are indicated by
Dataset
Ground- Truth Relevant Q-D Pairs
QGen Q-D Pairs
MS MARCO
D:
What cartography? A. the science of map- making B. the science shiphuikding C. the science of charting direction ship D. the science of measuring distances the occan. Carlography is the science of map making A. Q: what is the science of mapmaking called
D: The flu shot also contains the following in- gredients: sodium phosphate & buffered iso- tonic sodium chloride solution, formalde- hyde, octylphenul ethoxylate, and gelatin, ac- cording to the FDA.
Q: what's in the flu shot
TREC-COVID
D: An Evidence Based Perspective mRNA-
D: Impact of arterial load the agreement be- tween pulse pressure analysis and esophageal Doppler. INTRODUCTION. The reliability of pulse pressure analysis to estimate cardiac output is known be affected by arterial load changes. [...]
SARS-CoV-2 Vaccine Develupment. [..] The production of mRNA-baaed vaccines is a promising recent development the pro- duction of vaccines. However, there remain significant challenges the development [...]
Q: what is known about an mRNA vaccine for the SARS-CoV-2 virus?
QGen: what is arterial load for pulse pressure analysis
D: The possible pathophysiology mechanism of cytokine storm in elderly adults with COVID-19 infection: the contribution of "inflame-aging" PURPOSE: Novel Corn- avirus disease 2019 (COVID-19), 19), is an acute respiratory distress syndrome (ARDS). [..]]
D: Opportunity Costs Pacifism. If the resnurces used wage wars could be spent elsewhere and save more lives, does this mean that wars are unjustified? This article considers this question, which has been largely overlooked by Just War Theorists and pacifists. It focuses on whether the opportunity costs of war [...] QGen: opportunity cost pacifism
Q: What is the mechanism of cytokine storm syndrome on the COVID-19?
BioASQ
D: The role of extended-release amantadine for the treatment of dyskincsia in Parkinson's disease patients. [...] Extended-releadd amantadine (amantadine ER) is the first ap- proved medication for the treatment dysk- inesia. When is given bedtime, [...]
D: Subluxation of the femoral head in coxa plana. Twenty-two patients who had severe сoxa plana had closed reduction for Interal subluxation the femoral head, [...] The av- erage age when the patients were seen was eight years and six months, [...] QGen: average age of femoral subluxation
Q: amantadine ER the first approved treat- ment for akinesia?
D: I...] comparison of proxy assessment and putient self-rating using the discasc-specific Huntington's discase health-related quality of life questionnaire (HDQoL). [...] Specific Scales of the HDQoL. On the Specific Hopes and Worries Scale, proxies on average rated HrQul as better than patients' [...]
D: [...] We investigated the health-related qual- ity of life (HRQoL) of long-term prostate cancer patients who received leuprorelin ac- etate in microcupsules (LAM) for androgen- deprivation therapy (ADT). [...]
Q: Can leuprorelin acelate be used as androgen deprivation therapy?
QGen: which scale is used for proxy assessment of hrqol
Signal-IM
D: BJP terms party MP R.K Singh's allegation that money has changed hands for tickets in #RiharPolls as baseless.
D: Black lives matler: thoughts from the deliv- cry ward in St. Louis: #mustread
QGen: where is black lives matter?
Q: Party MP calls BJP 'Baura Jayewala Party"
D: RETWEET if "Brenda's 's Got A Baby" is one of your favorite @ 2Pac songs. #RIP2Pac QGen: brenda got baby pac
D: Kerry: US plans military talks with Russia over Syria
Q: Kerry: US plans military talks with Russia over Syria
24

---

Under review as a conference paper at ICLR 2023
Table 8: Samples of passage reranking results where ListDA achieves higher utilities v.s. zero-shot. Truncated omitted texts are indicated by "[...'".
Dataset
Zero-Shot Shot Top Results
ListDA Top Results (Ground-Trum Relevant)
(Ground- Truth Irrelevant)
Robust04
Q:
Find information on prostate cancer detection and treatment.
D: [...] FIRST PATIENT UNDERGOES GENE INSERTION IN CANCER TREATMENT
D:
[...] Little knowledge goes a long way Can- cer prostate need not a killer/ Ilealth Check. Earlier this year, 13-year-old [...] died from cancer of the bladder and prostate. Iis death is grim reminder that no male should consider himself immune from water- works trouble. The prostate, a gland about size and shape a chestnut, lies deep in the pelvis just below the bladder. Because surrounds the urethra, has the potential to block the flow of urine completely. [...]
This first round of gene transfer exper- iments, in which gene inserted into patient's white blood cells, is not expected to directly benefit an individual putient. Instead, the inserted gene being used to track the movement in the body of the cancer- cancer-fighting white blood cells. [...] Inserting human genes
repair defects may one day help with host of inherited disorders, [...]
TREC-COVID
Q: What the longer-term complications of those who recover from COVID-19?
D: [...] Our previous experience with members of the same corona virus family (SARS and MERS) which have caused two major epi- demics in the past albeit of much lower mag- nitude, has taught us that the harmful effect such outbreaks are not limited to acute complications alone. Long term cardiopul- monary, glucometabolic and neuropsychi- atric complications have been documented following these infections. [..]]
D: Up 20-30% of patients hospitalized with coronavirus disease (COVID-19) have evi- dence of myocardial involvement, Acute cardiac injury in patients hospitalized with COVID-19 is associated with higher morbid- ity and mortality. There are no data how treatment for COVID-19 may affect convalescent phase long-term cardiac covery and function. Myocarditis from other viral pathogens can evolve into overt or sub- clinical myocardial dysfunction, [...]
BioASQ
Q: What is the interaction between WAPL and PDSS proteins?
D: Pds5 and Wpll act anti- anti-establishment fac- tors preventing sister-chromatid cohesion un- til counteracted in S-phase by the cohesin acetyl-transfert Esol. [..]] Here, we show Pds5 is essential cohesin acetylation by Esol and
D: [..]] Here, we show that cohesin suppresses compartments but is required for TADs and loops. that CTCF defines their boundaries. and that the cohesin unloading factor WAPL its PDS5 binding partners control length loops. In the absence of WAPL and PDS5 proteins, cohesin forms extended loops, presumnably by passing CTCF sites, accumulates in axial chromosomal positions (vermicelli). and condenses chromosomes. [...]
the maintenance of co- hesion promoting stable cohesin inter- action with replicated chromosomes. The latter requires Esol only in the presence of Wapl. indicating that cohesin stabilization relies Esol only to neutralize the anti- establishment activity. [...]
25

---

Under review as a conference paper at ICLR 2023
Table 9: Samples of passage reranking results where ListDA achieves higher utilities v.s. QGen PL. Truncated omitted texts are indicated by "[...]..
Dataset
QGen PL. Top Results (Ground- Truth Irrelevant)
ListDA Top Results
(Ground-Truil Sound-Truthhhu Relevant)
RohustO4
Q: Identify outhreaks of Legionnaires' disease.
D: [..]] Care of Patients with Trachcostomy D: [...] LEGIONNAIRE'S DISEASE STRIKES
Suctioning Respiratory Tract Secre- tions III. Modifying Host Risk for Infection A. Precautions for Prevention of Endogenous Pneumonia 1. Prevention of Aspiration 2. Prevention of Gastric Colonization B. Pre- vention of Postoperative Pncumonia C. Other Prophylactic Procedures for Pueumonia 1. Vaccination Patients 2. Systemic Antimi- crobial Prophylaxis 3. Use of Rotating "Ki- netic" Beds Prevention and Control of Le- gionnaires' Disease [...]
16 AT REUNION IN COLORADO; DIF. outbreak of legionnaire's 's discase 50th high school reunion was blamed Thurs- day for the deaths of three elderly celebrants and the pncumonia-like illness of 13 others. State health officials contacted 250 other pco- ple from 21 states who attended the Lamar High School reunion for the classes of 1937 through 1941 but found no cases, Dr. Ellen Mangione, Colorado Department of Health epidemiologist, said. [...]
QGen: what kind of precautions used to pre- veni pneumonia
TREC-COVID
Q: what drugs have been active against SARS-CoV or SARS-CoV-2 animal studics?
D: Different treatments currently used for
D: [...] the antiviral efficacies of lopinavir- ritonavir, hydroxychhomoquine sulfate, and emtricctaainne-eendee for SARS-CoV-2 in- fection were assessed in the ferret infec- tion model. [...] all antiviral drugs tested marginally reduced the overall clinical scores of infected ferrets but did not significanlly af- feet in vivo virus titers. Despite the potential discrepancy of drug eflicacies between ani- mals and humans, these preclinical ferret data should be highly informative to future thera- peutic treatment COVID-19 patients.
clinical management of SARS-CoV-2 infec-
tion, but little is known about their efficacy yet. Here we present ongoing results to com- pare currently available drugs for a variely of diseases find out they counteract SARS- CoV-2-induced cytopathic effect vitro. L...] We will provide results come available, [...] QGen: what the treatment
they be-
BioASQ
Q: What is the function of the Spt6 gene in yeast?
D: As means to study surface protcins involved in the yeust to hypha transition, human mono- clonal antibody fragments (single-chain vari- able fragments, scf'v) have been generated that bind to antigens expressed the surface of Candida albicans yeast and/or hyphae. [..] To ussess C. albicans SPT6 function, expres- sion of the C. albicans gene was induced in a defined cerevisiacsptó mutant. Partial complementation was seen, confirming that the C. albicans and cerevisiae genes functionally related in these species.
D: Sptó is a highly conserved histone chapcrone that interacts directly with both RNA poly- merase Il and histones to regulate gene ex- pression. [...] Our results demonstrate dra- matic changes to transcription and chromatin structure in the mutant, including elevated antisense transcripis at >70% of all genes and general loss of the +1 nucleosome. Further- more, Sptó is required for marks associated with active transcription, including trimethy- lation of histone I13 lysine 4, previously observed in humans but not Saccharomyces cerevisiae, and lysine 36. [...]
QGen: what the function sptó gene can- dida albicans
26


Reviews:
[{"review_id": "uJXPQ9ILOR", "reviewer": "Reviewer_SYBU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on Domain Adaptation (DA) in the context of learning to rank. The authors follow the invariant representation learning approach, which they extend to the listwise learning to rank scenario. The final objective consists of a listwise ranking loss and an adversarial loss encouraging invariance across domains.  The authors also derive a domain adaptation generalization bound for listwise learning to rank scenario. The proposed method is evaluated on the passage/text re-ranking task. ", "summary_of_the_review": "Please refer to the Strength and Weaknesses section above. ", "strength_and_weaknesses": "Strengths.\n- The paper is well written and technically sound\n- The paper proposes a domain adaptation method for learning to rank tailored for the listwise scenario. Compared to existing methods, the proposed approach seeks invariance at the list representation level instead of the item level. \n- Theoretically, the authors derive a domain adaptation generalization bound for the listwise learning to rank case. \n\nWeaknesses.\n\n- The experiments are weak. (I) Evaluations are curried out on one task and one data type (text) only. (II) The list of baselines can be improved, for instance by including other domain adaptation learning to rank approaches, such as the one mentioned in the related work section. (III) Some results are not reported for all datasets and baselines (e.g., the results of figures 2 and 3). \n- The technical novelty of the proposed objective is somewhat limited. \n\nAdditional comments. \nI would recommend presenting section 4 first followed by the current section 3 as supportive theoretical results for the proposed objective.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "t0l8rAsvZRp", "reviewer": "Reviewer_U7gj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on domain-adaptation of ranking problem.\nThey show generalization bound for ranking problem and propose an approach to improve adaptation to minor domain shift.\nThrough a set of experiments on some open-source datasets they show the effectiveness of their approach.", "summary_of_the_review": "Overall I think it is a nice idea to improve the robustness of the model trained for re-ranking. \nWhile, they have bounds on domain adaptation generalization, I did not verify the correctness of the proof.\nThe experiments demonstrate superiority of their approach across three datasets.\nProvided the questions mentioned in the above section is addressed, I would be inclined to accept the paper.", "strength_and_weaknesses": "Strengths:\nPaper is fairly self contained\nPaper shows both theoretical bounds and practical approach.\n\nWeakness:\nThe writing in approach section blends both previous work and their approach making a bit hard to discern their contribution.\nThe experimental set-up might have some issues:\n* Why was the better negative sampling not used for other approaches.\n* While it is true that the paper focuses on ranking and hence retrieval model can be basic, it is unclear how their approach might be affected by a better retriever that could be encoder-based.\n* The model training is a bit confusing, should the adversarial training be done for every new target distribution or can it be trained once and re-used?\n* Q-D generation is a bit questionable, if the source target dont have many documents from the new domain would it still be ok? what is the threshold that determines that? Would be good if authors mention some details.\n\nMinor comments:\n* There are assumptions made in section 3 for the proof, but not all assumptions are justified. Would be good if they can add a couple of lines on why they are reasonable.\n* There are some minor grammatical errors which sometimes make it hard to understand like: \"The final missing piece is the choice of $F_{ad}$ that can model lists $z = (z1 , · · · , zl )$ of feature vectors zi ∈ Rd and is continuously differentiable.\"\n* It would be nice to also add and show what the upper bound of metric is if trained on the training dataset (target domain) when it exists.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VQJpDHgU06", "reviewer": "Reviewer_3EyT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a domain-invariant adaptation approach for listwise learning to rank (l2r). The authors extend a bound from the literature to l2r setting. Then a typical approach for domain adaptation is used with adversarial learning. The authors experiment in the textual domain with one source and three target domains. They compare with two baselines and two relevant approaches showing that the proposed method outperforms them.", "summary_of_the_review": "Generally a good paper, with a lack of connection between the theoretical and the method part. It builds upon an existing framework and extends for the l2r framework. The experimental part is weak in my opinion as it only applied on the textual domain and is hard to asses if this approach would work in other domains. Already the baseline works quite well.", "strength_and_weaknesses": "- This is an interesting problem/setting as in many cases it is very expensive to collect labeled data.\n- The authors extend work on domain adaptation on the l2r setting which is an important task in many applications\n- The results seem to be promising although only applied in the textual domain.\n\n- The proposed approach is not so tightly connected with the theoretical part. Indeed the authors use standard adversarial setting for domain adaptation. I would propose the authors to better describe the connection. These approaches are already presented, see for example Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. ICML 2017.\nGanin et al., Domain-adversarial training of neural networks. Journal of Machine Learning Research, 2016.\n\n- I think the authors should discuss this work \"On Learning Invariant Representations for Domain Adaptation\" regarding the optimal joint risk.\n\n- Generally, while the experimental part show promising results, the fact that is only applied in the textual domain makes it weak. Also, already the zero-shot baseline has a good performance, and to be honest I am not convinced how much this small difference would make in a production system. I am just curious here if we carefully tune the zero-shot baseline what the result would be.\n- Is there any reason to restrict to softmax listwise loss function? Did you try other ones? For example BoltzRank?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4-tjKW08jt", "reviewer": "Reviewer_HnLS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method to learn invariant listwise representations for ranking. A theoretical generalization bound is presented by analyzing domain adaptation for learning to listwise rank. Experimental results demonstrate the effectiveness of the proposed method on unsupervised domain adaptation for passage reRanking on a diversity of domains including biomedical and news articles. ", "summary_of_the_review": "An adversarial training method is introduced in this paper for learning domain-invariant representations to generalize the ranking model from the source to the target domain, in the setting of listwise ranking. Experiments on passage reRanking demonstrate the superiority of the proposed method. \n\nThe paper is clear and illustrative with concrete proofs and diagrams. \n\nHowever, the application and evaluation of this paper are limited to the reRanking problem. The contribution of this paper can be enhanced if the authors could generalize the proposed method to the fundamental ranking problem.\n\nAlso, along with listwise ranking and pointwise ranking, it'd be great if the authors could discuss more about pairwise ranking. It'd be also interesting to compare with some of the SOTA pairwise ranking methods. \n\nPresenting a complexity analysis of the proposed method and comparison to baselines is recommended, in order to show the efficiency of ListDA.\n\n", "strength_and_weaknesses": "Strength: \n\n1. This paper is theoretically sound, providing an important insight that when the domain shift is small according to the Wasserstein distance, a trained ranking model can be domain-adaptable, and its performance under regular ranking metrics can be bounded.\n\n2. The experimental results are reasonable and the proposed ListDA methods outperform baselines significantly by minimizing the source and target distributional shifts for learning domain-invariant representations. \n\nWeakness:\n\n1. Although the comparison between learning domain-invariant representation in the listwise and pointwise settings is given, the connection to the pairwise ranking has been little discussed.\n\n2. The evaluation is on the reRanking problem, that is reranking a list of candidate documents retrieved by a first-stage retrieval model in response to a search query, rather than the fundamental ranking problem. This seems to be one of the main limitations of this paper. The discussion about why the proposed method is not applied to the fundamental ranking problem can be elaborated.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject