Learning List-Level Domain-Invariant Representations for Ranking
Authors: Ruicheng Xian
Honglei Zhuang
Zhen Qin
Hamed Zamani
Jing Lu
Ji Ma
Kai Hui
Han Zhao
Xuanhui Wang
Michael Bendersky
Venue: NeurIPS 2023 spotlight (2023)
Keywords: learning to rank
domain adaptation
text ranking
TL;DR: We revisit invariant representation learning for ranking, propose list-level alignment, and establish a domain adaptation generalization bound for ranking.

Abstract: Domain adaptation aims to transfer the knowledge learned on (data-rich) source domains to (low-resource) target domains, and a popular method is invariant representation learning, which matches and aligns the data distributions on the feature space. Although this method is studied extensively and applied on classification and regression problems, its adoption on ranking problems is sporadic, and the few existing implementations lack theoretical justifications. This paper revisits invariant representation learning for ranking. Upon reviewing prior work, we found that they implement what we call item-level alignment, which aligns the distributions of the items being ranked from all lists in aggregate but ignores their list structure. However, the list structure should be leveraged, because it is intrinsic to ranking problems where the data and the metrics are defined and computed on lists, not the items by themselves. To close this discrepancy, we propose list-level alignment—learning domain-invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method, and it achieves better empirical transfer performance for unsupervised domain adaptation on ranking tasks, including passage reranking.

Learning List-Level Domain-Invariant Representations for Ranking
Ruicheng Xian
Honglei Zhuang
Zhen Qin
Hamed Zamani
Jing Lu²
Ji Ma
Kai Hui
Han Zhao'
Xuanhui Wang
Michael Bendersky?
University Illinois Urbana-Chann Chamaaa-
{rxian2, hanzhao J inoii..eeeie
Google Research
(hlz, (hlz,zhenqin, wwwinniwwnnnniiiiiiiiiii maji, kaihuibj. xuanhui, bemikee
University of Massachusetts Amherst
zamani&ce umass.
Abstract
Domain adaptation aims to transfer the knowledge learned on (data-rich) source domains to (low-resource) target domains, and popular method is invariant repre- sentation learning, which matches and aligns the data distributions on the feature space. Although this method is studied extensively and applied on classification and regression problems, its adoption on ranking problems is sporadic, and the few existing implementations lack theoretical justifications. This paper revisits invariant representation learning for ranking. Upon reviewing prior work, we found that they implement what call item-level alignment, which aligns the distributions of the items being ranked from all lists aggregate but ignores their list structure. However, the list structure should be leveraged, because it intrinsic to ranking problems where the data and the metrics are defined and computed on lists, not the items by themselves. To close this discrepancy, we propose list-level penumentt-eamming
invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method, and it achieves better empirical transfer performance for unsupervised domain adaptation on ranking tasks, including passage reranking.
Introduction
Learning to rank applies machine learning techniques to solve ranking problems that are at the core of many everyday products and applications, including search engines and recommendation systems (34). The availability ever-increaaing amounts of training data and advances in modeling are constantly refreshing the state-of-the off-the-art of many ranking tasks [40]. Yet, the training of practical and effective ranking models on tasks with little to no annotated data remains challenge [57]. A popular transfer learning framework for this problem is domain adaptation [4I]: given source domains with labeled data that are relevant to the target domain of interest, domain adaptation methods optimize the model on the source domain and make the knowledge transferable by utilizing (unlabeled) target data.
For domain adaptation, there are task aase-spccific and general-purposs. (unsupervised) methods. E.g.. for text ranking information retrieval where only have access target documents without any training query or relevance annotation, a method is to use generative language models to synthesize
Work performed while Google Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Lists of items (q-d pairs) to be ranked
Item-level alignment
List-level alignment
Source
9 2
92
4
item
M
Target
c. 9
a. d.
H
item
94
9
Distributions
(list structure ignored)
Distributions of lists
Figure I: Item-level and list-level alignment on ranking problem. Each list contains items to be ranked. Whereas ttemllevel alignment aligns the distributions of items but ignores the list structure, list-level alignment preserves this structure and aligns the distributions of lists.
relevant queries for the unannotated documents, then train the ranking model on the synthesized query- For general-purpose methods, which are applicable all kinds of tasks, perhaps the most well-known is domain- invariin- representation learning [38] 17]. While it is studied extensively on classification and regression problems 77, applications of invariant representation learning on ranking problems are sporadic [12 58 68). and the existing implementations lack
document pairs 37
54 65
theoretical justifications, which casts doubt on their applicability learning to rank.
In this paper, we revisit the method of learning invariant representations for domain adaptation on ranking problems. We found that, from our review of prior work [12 (58 68), existing implementations of the method all perform what we call item-level alignment (Fig. and Section (.2.1). which aligns the distributions of the items being ranked (e.g., query-document pairs) aggregated from all lists and discards their list structure (as in, e.g.. q-d pairs with the same query all belong to the same list). However, both the data and the metrics for ranking are defined and computed lists, not the items by themselves, so intuitively the list structure should be leveraged in the feature learning process as it intrinsic the problem. To close this discrepancy, we propose and analyze list-level alignment. which aligns the distributions of the lists and preserves their list structure (Section 3.2.2).
The conceptual leap from item-level to list-level alignment is significant. First, by analyzing list-level representation invariance, we establish the first domain adaptation generalization bound for ranking (Section 4 in turn providing theoretical support for our proposed method well foundation for future studies. Second, list-level alignment provides empirical benefits to domain adaptation on ranking problems, which demonstrate on two ranking tasks, passage reranking (Section 5 and Yahoo! LETOR web search ranking (Appendix DD, where it outperforms zero-shot learning, item-level alignment, as well as a method specific text ranking based on query synthesis (37).
2 Problem Setup
We define a ranking problem by joint distribution of length-[ lists and their nonnegative scores, denoted by X = -.., Xe) retrieval, the items in each list are query-document pairs of the same query, and the scores indicate the relevance of the documents to the query. Note that the lists are permulation- permulation-iivvriann, because switching the order of the items (and the corresponding scores) does not change the content of the list; concretely, let denotes the set of permutations on [6] 21,2.... ,L}. then permutation invariance Dxz (Bm1. Fxz)) )) for all (a, y) and xX)), so the problem can be equivalently given by the marginal distribution yx of lists and scoring function y X + B
X and Y (Yi, .Y) R [T For example, in information
means that μ(([11, ... .I),, (21, .ee) [((
E Se. We assume that the scores are a function of the list,
Learning to rank aims to learn a ranker that takes lists x as input and outputs rank assignments E Se, where r, E [6] is the predicted rank of item i, such that recovers the descending order of the scores y. i.e., y/ > Yj + r < ry for all j.j. The more common setup, which we will adopt, is
Although here the input lists are written to be decomposable into the & items they contain, generally and abstractly, they need not be so.

---

to train scorer, f & - R', with the rank assignments obtained by first computing the ranking scores s = f(x) E IR,, where s; the predicted score of item then taking the descending order of (alternatively, the rank assignments can be generated probabilistically; see Section +
To quantitatively measure the quality of the predicted ranks, St x IR20 Rzo. which computes utility score via comparing the predicted rank assignments the ground-truth scores of the list. Common metries information retrieval include reciprocal rank and normalized discounted cumulative gain [63 26
(listwise) ranking metrics,
Definition 2.1 (Reciprocal Rank). If the ground truth scores are binary, i.e., y E {0, 13", then the reciprocal rank (RR) of a predicted rank assignments E Se is
RR(r; v) max S S D = 1) (0)).
The average RR of ranking model f' X' + St on dataset, (xx,Y) (R) Y)], referred to as the mean reciprocal rank (MRR).
Definition 2.2 (NDCG). The discounted cumulative gain (DCG) and the normalized DCG (with identity gain function) of predicted rank assignments E St are
DCG(r, IDCG(y)
DCG(r, v)
Σ
log(r; 1) and NDCG(r, v)
where IDCG(y) = maxp'ss, DCG(r', y), called the ideal DCG, is the maximum DCG of a list, which is attained by the descending order of y.
Domain Adaptation. In this learning setting, we have a source domain (PŠ) Us) and target domain (n²²5 yr) with different data distributions and the goal is to learn a good ranking model for the target domain by leveraging all available resources: whereas access source domain labeled training data is always assumed, we may only be given unlabeled data for the target domain (this scenario is called unsupervised domain adaptation). popular method for this adaptation domain- invariant representation learning, which matches and aligns the source and target domain data distributions on the feature space so that their distributions appear similar to the model.
3 Learning Domain-Invariant Representations for Ranking
We begin by reviewing invariant representation learning for domain adaptation and the optimization technique of adversarial training, then describe and compare two instantiations this framework to ranking problems in Section 3.2 -item-level alignment, which is implemented in prior work, and our proposed list-level alignment.
For representation learning, we train composite models of the form f f= h o g. where g - Z shared feature map and h task head on the shared features. As an example. if the model ff is end-to-end implemented as an m-layer multilayer perceptron (MLP), then we could treat the first (m 1) layers as g and the last as h.
And, recall the definitions of Lipsehitz continuity and Wasserstein distance
15
Definition 3.1 (Lipschitz Continuity). A function f - ) from metrie space (XX dxx dy) is L--iipchitz, denoted by f E Lip( (L), if (x))
< L dx(x,x') for all x E X.
Definition 3.2 (Wasserstein Distance). The Wasserstein-1 distance hetween probability measures p. on metric space X' is denoted and given by Wi(p, sup -..Pp.X-R, fxx (x))((()) g(x)) dr.
3.1 Invariant Representation Learning via Adversarial Training
Given a source domain μs and target domain pr, invariant representation learning aims to learn shared feature map g s.t. the distributions are aligned the feature space, i.e., D(u /4) Rs 0, where D is divergence measure or probability metric, e.g., Wasserstein distance, and the source feature distribution 2 (analogously for target μ is defined be the distribution of (X)), X n The idea is that composite model with aligned feature representation is transferable between
For adapting from multiple source domains,
71
3

---

domains this is supported by the following generalization bound on classification problems We will establish a similar result for ranking Section A
52 5
Theorem 3.3. Let binary classification problems on source and target domain be given by joint distributions pas, por of imputs and labels, (X, Y) E X x {0,1}. Define the error rate of a predictor
f: :X + [0, 1] on p by R(f)) = Exxy (X 1 * + f(X)) 1[ + +0]lAwher 1[:] is the indicator function.
Let H c [0, 1 be an L-Lipschitz class of prediction heads, and for any g G, define the minimum joint evror rate by X² = min (Rs (h' / g)) + Rrr('' o g)), then for all E H.
Rr o g) S Rs o 9) 2L Wi (PS pTf Xz.
This result bounds the target domain risk of the composite model h by its source task risk, plus the divergence of the feature distributions and the optimal joint risk. It therefore suggests that good model for the target could be obtained by learning an informative domain-invariant feature map (s.t. X9 and W1 (MS, ) se 0) and training to minimize the source task risk Rss.
Invariant representation learning algorithms capture the aforementioned objectives with the joint training objective minp,g ( + A D(S (f4) [7], where Ls is the source task loss, and À > 0 is hyperparameter that controls the strength of invariant representation learning. This objective learns and minimizing source risk at the same time s.t. the learned features are useful for the task. Since it does not require labeled data but only unlabeled ones for estimating Wi (N frf), it is applicable to unsupervised domain adaptation. If labeled target data are available, they can be incorporated by, e.g.. adding a target loss Lr, which will help keep X low.
features by optimizing g to minimize the distributional discrepancy,
Adversarial Training. When the divergence measure D is simple and admits closed-form expres- sions, e.g., maximum-mean discrepancy [19), the objective above can minimized directly [35]. but they are typically too weak modeling complex distributions that arise in real-world problems. To achieve feature alignment under stronger measures, c.g., JS divergence or Wasserstein distance, a well-known approach is adversarial training [18 17 21, which reformulates the above objective into
Cjoim (h, 9)
min keH.
(s À min Caa(g, fud) fuEFF
and optimizes it using gradient descent-ascent with gradient reversal layer added on top of in the adversarial component.
The adversarial component, - -iin Lad(y, faa), can be shown to upper bound the divergence between n3. 1z- involves an adversary fat : 2 + (parameterized by neural networks) and an adversarial loss function Ead R x {0, 1 R. whose inputs are the output of faa and the domain identity a (we set target domain to a 1). The adversarial loss is computed over both domains:
Lad(g, faa) = Ex [Ex o g(X), 0)] + Ex (fa o g(X). 1)]-
If we choose the 0-1 loss, las(à, a) (1 a) 1/a 0] + L[ā 0j. then the adversarial component upper bounds Wi (23 HG) (see Proposition A.1;; here, fal acts as a domain discriminator for distinguishing the domain identity, and Las is the balanced classification enror of faa- For training, the 0-1 loss is replaced by surrogate loss, e.g., the logistic loss [18] la a) log(1 (1 + (1-2a)0).
3.2 Invariant Representation Learning for Ranking
This section describes and compares two instantiations of the invariant representation learning framework above for ranking: item-level alignment, and our proposed list-level alignment. The key difference between them is the choice of μ whose divergence W, (u pT) is to be minimized.
Model Setup. We consider a composite scoring model f h g where the feature map s.t. given an input list x = (x1, z.)) it outputs a list of k-dimensional feature vectors, g(x) = (M,,".,, ve) E Rexk where v; E R corresponds to item Ranking scores are then
TThis definition assumes that the output class is probabilistic according to P(F = = z) f(x).

---

Each domain contains
Representation
Representation
2 querles and 3 documents
Source
9g d
4g d
9 d
4 d
9,d.
9.
9, dg
3 d
4 d
93 d
4 d
Target
4 ds
44 0
9 d
94 d
c Q 4g 6
a.G
9. 4
Aligned itern-level but lest-leeell
Aligned at item--leee and at list level
Items with same belong same list
Figure 2:
List-level alignment is a stronger requirement than item-level alignment, which, in addition to pairing target domain items to source items, it also pairs target lists to source lists.
obtained by, e.g., projecting each v to I with a (shared) linear layer. This setup, which we will adopt experiments, common with listwise ranking models and in many ranking systems [10]: in neural text ranking, each feature vector can be the embedding of the input text computed by
language model 33 76
3.2.1 Item-Level Alignment
In item-level alignment (AtemDA), the distributions of the k-dimensional feature vectors from all lists in aggregate are aligned 212
58 68
i.e., Ms
Zittem
~ prT
Z.item
where
1 Z. (v) == P 2~
Z) = xx (v E e())), supp(uz2.iem) Z,item S IR
In other words (when both domain have the same number of lists), finds a matching between the items of the source and target domains that pairs each target item with source one (Fig-2) Note that the list structure is not retained in this definition, because one cannot necessarily tell whether two items drawn from the bag of feature vectors, vf p.Z..lemm belong to the same list.
To implement item-level alignment, the discriminator faa (usually an MLP) set up to take individual feature vectors E R as input. In each forward-pan pastd the feature map computes a batch of feature vector lists, {2, NejM = {(111 ... Use) ke[M] then the discriminator takes the items batched from all lists (this step discards the list structure), {vig [Ej and predicts their domain identities.
Item-level aligmment is identical to invariant representation learning implementations for classification and regression, c.g., DANN [17), which also operate "items", since neither the data nor the metrics in those problem settings have any explicit structural assumptions. However, ranking problems are inherently defined with list structures: the inputs to the model are organized by lists, and ranking metrics are evaluated on lists rather than the items by themselves. This discrepancy casts doubt on the applicability of item-level alignment ranking problems, and moreover, it is unclear whether domain adaptation for ranking via item-level alignment is justified from a theoretical perspective.
3.2.2 List-Level Alignment
Closing the discrepancy discussed above, propose list-level alignment (ListDA), which aligns the distributions of the lists of feature vectors on the fx space. i...,
7,
us
23
PT
where
2 Jist = P2 (2) = Px x (a g(X), supp(u 2.lis R
This means finding matching that not only pairs all source and target items, but also pairs items from the same target list to items from the same source list. Therefore, list-level alignment a stronger requirement than level alignment, μlg 1T generally not true (see Fig. for picture, or Example A.2
2 Zist
Z.item Ziitem while the converse is Ms
To implement list-level alignment, the discriminator has to make predictions on lists, = Rexk i.e.
in each forward-pase pass it would be fed batch of feature vector lists, (2) {(a,, ... Use) [ej)) and output predictions. For this, possible design choice to flatten each feature vector list
into long (ek) dimenniinso vector and use an MLP as fad. But recall from Section 2 that the
5

---

input lists are permutation-invariant invariant, and so are the feature vector lists. Since the above setup does not incorporate this property and an exponential amount of samples is required see all possible permutations, this would be inefficient to optimize; yet, the optimality of fat is essential to the success of adversarial training. So instead, we use transformers with mean-pooling less positional encoding as the discriminator [60], which is cormmuattioninvvarian to the input.
Compared item-level alignment, our list-level alignment is supported by a domain adaptation generalization bound (Theorem 4.7] established in the next section), and achieves better transfer performance in our Section and Appendix ArpendixDDevalleee evaluations. These results suggest that the list structure is essential for an effective domain adaptation on ranking problems.
4 Generalization Bound for Ranking with List-Level Alignment
Based on the list-level alignment proposed above, we establish domain adaptation generalization bound for ranking by analyzing list-level representation invariance. The discussions in this section consider learning a composite scoring model, f h o g X - R but need not assume that the list feature representation 2 = g(x) can be decomposed into item-level feature representations as in the model setup of Section 3.2.2] only that it resides in metric space 2.
Our result can be viewed as an instantiation of the framework established originally for classification by Ben-David et al. [5]. But since ranking is very different task with its unique technical challenges, our analysis differs from those in prior work [52]. We will introduce appropriate assumptions for handling these difficulties leading up to the main Theorem 4.7 which the readers may skip to.
The first hurdle the discontinuity of the sorting operation. So rather than taking the descending order of the predicted scores to obtain the rank assignments, we generate them probabilistically using a Plackett model [43 36 with the exponentiated scores as parameters
[10] 20 This makes the computation of the utility (Lipschitz) continuous w.r.t. the raw output scores of the model.
Definition 4.1 (Plackett-Lucc Model). A Plackett-Luce (P-L) model with parameters E RSo0 specifies a distribution over St, whose probability mass function Pw
Par (r)
II
WI(r)
Vr E St
j= WI(r);
where I(r); the index of the item with rank rI(r). = i.
Assumption 4.2. Given predicted scores s f(x) E Re, generate the rank assignments probabilistic from a P-L model by R ~ Pexp(s) E Se, where exp is taken coordinate-wise.
Under this assumption, the utility of scorer f on (4**, v) w.r.t. ranking metric Se x RSo - R20 is computed by (overloading u for future brevity)
[m(x(() == Exxpx ER-Perp((()) [u(R, x())]-
We define the risk (or suboptimality) of ff as the difference between its utility to the maximum- attainable utility on the problem (e.g., the maximum is I when u = NDCG):
R() Ex~ax
max max u(r, ((,, - [[]]]()]] rES,
The next technical challenge is that unlike in classification where the perfect classifier is unique (i.e., achieving zero error), the perfect ranker is generally not: e.g., both rank assignments 2,33
and (2, 1) achieve maximum utility (i.e.. R. 0) on a list with ground-truth scores (1, 1, 0). Prior analyses of domain adaptation leverage this uniqueness [5] [2 instead, ours does not rely on uniqueness (else the bound would be loose) with the following Lipschitz assumptions:
Assumption 4.3. The ranking metric S x RR R2o is upper bounded by B, and is Lu-- I Lipschitz w.r.t. the ground-truth scores y in the second argument (in Fuclidean distance).
A Assumption 4.4. The input lists X reside in metric space X, and the ground-truth scoring function y X R$ is Ly Lipschittz (in Euclidean distance the output space).

---

We will show that Assumption 4.3 is satisfied by both RR and NDCG. Assumption 4.4 says that similar lists (i.e., close in X) should have similar ground-trutt truth scores, and is satisfied, e.g., when X' is finite; such is the case with text data, which are one-hot encoded after tokenization.
Assumption 4.5. The list features Z reside in a metric space 2, and the class H. of scoring functions, h: 2 + R is Ln Lipsschtz (in Euclidean distance on the output space).
Assumption 4.6. The class G of feature maps, 9 X - 2, satisfies that Vg E G, the restrictions of 9 to the supports of pg and μ4 Iluuptuu y Ilappp(()) respectively, are both invertible with T.y-Lipschitz inverses.
Assumption 4.5 is standard in generalization and complexity analyses, and could be enforced, e.g.. with 1.tth.. r.-rraulariization 4 The last assumption is technical, saying that the inputs can be recovered from their feature representations by Lipschitz inverse g g_1 This means that the feature map should retain as much information from the inputs (on each domain), which is a desideratum of representation learning. Note that this assumption does not conflict with the goal of invariant representation learning: there is always a sufficiently expressive G s.t. Eg E 9 satisfying the assumption and μ p4 1is
Z list
We are now ready to state our domain adaptation generalization bound for ranking:
Theorem 4.7. Under Assumptions 4.2 4.6 for any g E G. define the minimum joint risk by X miny (Rs(k' g) + Rh'h's o g)), then for all h E H.
4 + BETN) Wi (1'1s 2.list 2.1laty + N/ Rr(h o 9) & Rs(h ( where y 2 is the marginal distribution of the list features Z g(X), (z) x.x (0 (z)).
,TT
The key feature of this bound is that depends on list-level alignment, Wi ( MT 1in hence it pro-
2,
vides theoretical support for the list-level alignment proposed in Section 3.2.2 It can be instantiated to specific ranking metrics by simply verifying the Lipschitz condition Lu of Assumption 4.3 Corollary 4.8 (Bound for MRR). RR is 1 1-Lipschitz in y. thereby
Epe (R g)] E,s [RR(h g)] - 4,,, + (L,) Wi W 2,ii
.1T (.Zat
g.
Corollary 4.9 (Bound for NDCG). Cf < IDCG - C for some C E (0, co) on (uz Us) and (UY yr) almost surely then NDCG is OCC
in y almost surely. thereby
(NCG o g)l 2 Eps (NDC g)] - -(CC + (Ln) Wi Z.list 2.iss
.PT X..
5 Experiments on Passage Reranking
To demonstrate the empirical benefits of invariant representation learning with list-level align- ment (ListDA) for unsupervised domain adaptation, we evaluate it on the passage reranking task 55 |nd compare zero-shot transfer, item-level alignment (ItemDA), well query synthesis (37]. Note that our method is not specialized to text (re) (re)ranking; in Appendix D we evaluate ListDA on a web ranking task from the Yahoo! Learning to Rank Challenge.
recent method based on
In passage reranking, given a text query, we are to rank candidate passages based on their relevance to the query from retrieved set. Reranking is employed in scenarios where the corpus is too large for using accurate but expensive models such as cross-attention neural rankers to rank every (millions of) documents. Rather, two-stage approach is taken: simple but efficient model such as sparse or dense retrievers (e.g., BM25 49 and DPR (28 is first used to retrieve a candidate set of (hundreds of) passages, then a more sophisticated (re)ranking model is used refine and improve their ranks.
We use BM25 to retrieve 1,000 candidate passages in the first-stage, and focus on the adaptation of a Ra listwise reranker (76). cross-attention model derived from the T5 base language model with 250 million parameters 488. It takes the concatenation the query and the document (q-d pair) as input and outputs an embedding. We treat the q-d embeddings computed on the same list as the list feature- consistent with the setting in Section 3.2 For the task training loss function, we use the
softmax cross-entropy ranking loss: ((s, = i= y/ (og(ee(pss)) (s:)/ exp(s;)).
'IDCG lower bounded, e.g.. every list contains at least one relevant item, and upper bounded when the ground-truth scores are upper bounded. We will use the terms document and passage interchangeably.
7

---

Query: besically Iwo..
Gradient Reversal
Discriminator
Document Bears not Bears.
(transformer)
Mean-pool Dense
Document All beer species are greet..
Predicted
domaiin identity
Source or target
gdd qd2
T5 Encoder
(cross-attention)
BM25
q gd
Dense
List of qud pairs
List of embeddings
Ranking scores
Figure 3: Block diagram of the cross-attention RankT5 text ranking model with ListDA.
Datasets. In our domain adaptation experiments, we use the MS MARCO dataset for passage ranking 3 as the source domain, which contains 8 million passages crawled from the web covering a wide range of topics with 532.761 search query and relevant passage pairs. The target domains
biomedical (TREC-COVID 2 BioASQ [59] and news articles (Robust04 (64). The data
preprocessed consistently with the BEIR benchmark [77 their paper includes dataset statistics.
Since the training split of the target domain datasets does not contain q-d relevance annotations, setting is unsupervised. It also does not contain any queries, but these are required for performing domain adaptation, because the input lists are defined consist of q-d pairs. To acquire training queries on the target domains, we synthesize them in a zero-shot manner following (37), by using a T5 XI. query generator (QGen) trained MS MARCO relevant q-d pairs. QGen synthesizes each query as a seq-to-seq task given passage, and the generated queries are expected to be relevant to the input passages. See Table for sample QGen q-d pairs.
Methods In zero-shot transfer, the reranker trained on MS MARCO and evaluated on the target domain without adaptation. In QGen pscudolabel (QGen PL), treat QGen q-d pairs synthesized on the target domain as relevant pairs, and train the reranker on them in addition to MS MARCO. This method is specifie to text ranking, and is used in several recent works domain adaptation of text retrievers and rerankers 37 54 65).
ItemDA (item-level alignment) is the prior implementation of invariant representation learning for and set up according to Section 3.2.1 with adversarial training described in Section 3.1 The adversarial loss is aggregated from the losses of five three-layer MIPP discriminators
ranking 12
58
(no improvements from using wider or more layers), Σ :=1 Lad f ). for reducing the sensitivity to the randomness in the initialization and training process [16. Our ListDA (list-level alignment) set up according to Section 3.2.2 and the discriminator is an ensemble of five stacks of three T5 (transformer) encoding blocks. To predict the domain identity of a list feature = = (VI, WE.) we feed the vectors into the transformer blocks all at once as sequence, take the mean-pool of the output embeddings, then project it to logit with a linear layer. A block diagram of ListDA is in Fig. 3
Further details including hyperparameters and the construction of training lists (i.e., sampling negative pairs) are relegated to Appendix C where we also include additional results such as using the pairwise logistic ranking loss, a hybrid adaptation method combining ListDA and QGen PL, and using transformers the discriminator for ItemDA.
5.1 Results
The main results are presented in Table [ where we report metrics that are standard in the litera- ture (c.g.. TREC-COVID uses NDCG@20, and evaluate rank assignments by the descending order of the predicted scores. Since TREC-COVID and Robust04 are annotated with 3-level relevancy, the scores are binarized for mean average precision (MAP) and MRR as follows: on TREC- COVID, we map 0 (not relevant) and (partially relevant) to negative, and 2 (fully relevant) to positive; on Robust04, 0 (not relevant) to negative, and (relevant) and (highly relevant) to positive.
All adaptation methods applied each source- strretttrgetteeee domain pair separately.

---

Table 1: Reranking performance of RankT5 on top 1000 BM225-etrrieee passages.
Target domain
Method
MAP
MRR@ 10
NDCG@5
NDCG@10
NDCG@20
BM25
0.2282
0.6801
0.4396
0,4088
0.3781
Zero-shot
0.2759
0.7977
0.5857
0.5340
0.4856"
RohustO4
QGen PL
0.2693
0,7644
0.5406
0.5034
0.4694
ItemDA
0.2822*1
0,8037*
0.58221
0,5396
0,4922'
ListDA
0.2901
0.8234
0.5979ft
0,5573
0,5126
BM25
0.2485
0.8396
0.7163
0.6559
0.6236
Zero-shot
0,3083
0.9217
0.8328
0.8200
0.7826
TREC-COVID
QGen PL
0.3180
0,8907
0.8373
0,8118
0.7861
ItemDA
0.3087
0,9080
0.8276
0.8142
0.7697
ListDA
0.3187
0.9335
0.8693
0.8412
0.7985
BM25
0.4088
0,5612
0.4580
0.4653
0.4857
Zero-shot
0.5008#
0.6465
0.5484
0,5542
0.5796
BioASQ
QGen PL
0,5143*
0.6551
0,5538
0,5643
0.5915
ItemDA
0.4781
0.6383
0.5315
0.5343
0.5604
ListDA
0.5191
0.6666
0.5639
0.5714
0.5985
Souree domain MS MARCO. Gain function in NDCG is the identity map. 'Improves upon zero-shot with statistical significance (p S 0.05) under the two-tailed Student's (-test. "mproves upon QGen PL.. 'Improves upon ItemDA.
Across all three datasets, ListDA achieves the best performance, and the fact that it uses the same training resource as QGen PL demonstrates the added benefits of list-level invariant representations for domain adaptation. The favorable comparison of ListDA to ItemDA corroborates the discussion in Sections and that for domain adaptation on ranking problems, item-level alignment insufficient for transferability, sometimes even resulting in negative transfer (vs. zero-shot). Rather, list-level alignment is the more appropriate approach.
5.2 Analyses of ListDA
Quality of QGen. An explanation for why ListDA outperforms QGen PL despite the same resources is that the sampling procedure (Appendix C.3) of negative q-d pairs could introduce false negatives into the training data. This supported by the observation in [54) that QGen synthesized queries lack specificity and could be relevant to many documents. While these false negative labels are used for training in QGen PL, they are discarded ListDA, which is thereby less likely to be affected by false negatives, even false positives-we wheen synthesized queries are not relevant to the input passages (see Table samples). Although out of the scope, better query generation expected to improve the performance of both QGen PL and ListDA.
Size of Target Data. Unsupervised domain adaptation re- quires sufficient unlabeled data, but not all domains have the same amount: BioASQ has 14 million documents (so is the total number of QGen queries, as we synthesize one per document), but Robust04 only has 528,155, and TREC-COVID 171,332.
Robust04
0.600
0.575
TREC COVID
NO0G85
In Fig. 4 we plot the performance of ListDA under varying numbers of target QGen queries (which is the number of target training lists). Surprisingly, on Robust04 and TREC-COVID, using just ~ 100 target QGen queries (0.03% and 0.06% of all, respectively) is sufficient for ListDA to achieve full performance! Although the number of queries is small, since retrieve 1,000 documents per query, the amount distinct target documents in those 100 lists could be substantial up 100,000, 29.5% and 60.7% of the respective corpora.
0.850
0.825
BloASQ
0.56
10
100
1000
10000
Number of target queries
Figure 4: ListDA performance un- der different target sizes. Lower grey horizontal line is zero-shot, upper red line is ListDA using all QGen queries.
The performance begins to drop when reduced to - 100 queries, which caps the amount of distinct documents at 10,000 (2.7% and 5.8%, respectively). The same data efficiency, however, is

---

not observed on BioASQ, likely due to the size and hardness of the dataset, such as the use of specialized biomedical terms (see examples in Tables to
Related Work
Learning to Rank and Text Ranking. Learning to rank historically focuses on tabular data, for which, wide range of models are developed [34], from SVMs [27). gradient boosted decision
trees (9), to neural rankers 8 42 45 Another traditional research direction the design of ranking
losses (surrogate ranking metrics), which can be categorized into pointwise, pairwise, and listwise
approaches 10
74 24
Recent advances in large neural language models have spurred interest in applying them on text ranking tasks 33 46 leading to cross-attention models [22] 39 40 44 and generative models based on query likelihood [1
77 78 50]. A different line of work is neural text retrieval models, which emphasizes efficiency, and has seen the development of dual-encoder 28 70. late-interaction
and models leveraging transformer memory [56).
Domain Adaptation in Information Retrieval. Work on this subject can be categorized into supervised and unsupervised domain adaptation. The former assumes access labeled source data and (a small amount of) labeled target data (a.k.a. few-shot learning)
[54]. This work focuses on the unsupervised setting, where target domain data do not have annotations. Cohen al. [12] apply invariant representation learning to unsupervised domain adaptation for text ranking, followed by Tran et al. (58) for enterprise email search, and Xin al. [68] for dense retrieval. However, unlike proposed list-level alignment method (ListDA), they learn invariant representations at the item-level. recent family of adaptation methods is based on query generation 377 65 originally proposed for dense retrieval.
Domain-invariant representation learning is popular family of adaptation methods [35] 17 13 to which our proposed ListDA belongs. Besides ranking, these methods also applied in fields including vision and language, and tasks ranging from cross- domain sentiment analysis, question-answering
Invariant Representation Learning.
31 61,, and to cross-lingual learning and machine
translation (67 30).
Zhao et al. 72. and Tachet des Combes et al. (55 point out that on classification problems, attaining perfect feature alignment and high source accuracy insufficient to guarantee good target perfor- mance. This occurs when the marginal distributions of labels differ, or the learned features contain domain-specifee and nontransferable components. Although their findings do not directly apply to ranking, still, they suggest two potential directions for future work: one is whether Theorem 4.7 would admit fundamental lower bound under distributional shifts in the ground-truth scores, and another is to modify ListDA to include a component for isolating out nontransferable features, as in domain separation networks (6].
7 Conclusion
We proposed an implementation of invariant representation learning for ranking via list-level feature alignment, and based which, established domain adaptation generalization bound for ranking. Our theoretical and empirical results illustrate the significance of preserving the list structure for achieving effeective domain adaptation, which is overlooked in prior work.
broader message that when working with (feature) representations, either for domain adaptation or other purposes, they should be analyzed at the same level (or structure) at which the data are defined for the task of interest and the metrics are computed. The results of this paper such an example-by by moving from item-level alignment to the more appropriate list-level alignment, we extracted more potentials from invariant representation learning for domain adaptation on ranking problems.
10

---

Acknowledgments
We thank the anonymous reviewers for their comments and suggestions on improving the presentation. This research was supported in part by the Google Visiting Researcher program and in part by the Center for Intelligent Information Retrieval. Han Zhao's work was partly supported by the Defense Advanced Research Projects Agency (DARPA) under Cooperative Agreement Number: HR001 12320012. Any opinions, findings, and conclusions or recommendations expressed this material are those of the authors and do not necessarily reflect those of the sponsors.
References
[1] Martin Authony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations. Cambridge University Press, 1999.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning, pages 2017.
[3] Payal Bajaj. Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: Human Generated MAchine Reading COmprehension Dataset, 2018. arXiv:161 1.09268 [cs. CLJ.
[4] Peter L.. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally normalized margin bounds for neural networks. In Advances Neural Information Processing Systems, 2017.
[5] Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of Repre- sentations for Domain Adaptation. In Advances in Neural Information Processing Systems, 2007.
[6] Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain Separation Networks. Advances in Neural Information Processing Systems, 2016.
[7] Sebastian Bruch, Shuguang Han, Michael Bendersky, and Mare Najork. A Stochastic Treatment of Learning Rank Scoring Functions. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 61-69, 2020.
[8] Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning Rank using Gradient Descent. In Proceedings of the 22nd International Conference on Machine Learning, pages 89-96, 2005.
[9] Christopher J.C. Burges. From RankNet to LambdaRank to LambdaMART: Overview. Technical Report MSR-TR- 2R-2000-00--, Microsoft Research, 2010.
[10] Zhe Cao, Tao Qin, Tic- Liu, Ming-Feng Tsai, and Hang Li. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning, pages 129-136, 2007.
[11] Olivier Chapelle and Yi Chang. Yahoo! Learning Rank Challenge Overview. Journal of Machine Learning Research: Workshop and Conference Proceedings, 14:1-24, 2011.
[12] Daniel Cohen, Bhaskar Mitra, Katja Hofmann, and W. Bruce Croft. Cross Domain Regulariza- tion for Neural Ranking Models using Adversarial Learning, In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1025-1028, 2018.
[13] Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, 2017.

---

[14] Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. Beyond [CLS] through Ranking by Generation. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1722-1727, 2020.
[15] D.A. Edwards. On the Kantorovich-Rubinsee theorem. Expositiones Mathematicae, 29(4): 387-398, 2011.
[16] Yanai Elazar and Yoav Goldberg. Adversarial Removal Demographic Attributes from Text Data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, 2018.
[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-Adversaril Training of Neural Networks. Journal of Machine Learning Research, 17(59):1-33 2016.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems, 2014.
[19] Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25))723-773, 2012.
[20] John Guiver and Edward Snelson. Bayesian inference for Plackett-Luuc ranking models. In Proceedings of the 26th International Conference on Machine Learning, pages 377-384, 2009.
[21] Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. A Deep Look into Neural Ranking Models for Information Retrieval. Information Processing Management, 57(6), 2020.
[22] Shuguang Han, Xuanhui Wang. Mike Bendersky, and Marc Najork. Learning-to-Rann tank with BERT TF-Ranning, 2020. arXiv:2004.08: 0.846 [Cs.
[23] Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, and Donald Metzler. ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3747-3758, 2022.
[24] Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Mare Najork. On Optimizing Top-K Metrics for Neural Ranking Models. In Proceedings of the 45th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 2303-2307, 2022.
[25] Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Michael Bendersky, and Marc Najork. Rax: Composable Learning-to-Ran Rank Using JAX. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3051-3060, 2022.
[26] Kalervo Järvelin and Jaana Kekäläänen. Cumulated Gain-Based Evaluation of IR Techniques.
ACM Transactions Information Syslems, 20(4):422-446, 2002.
[27] Thorsten Joachims. Training Linear SVMs in Linear Time. Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 217-226. 2006.
[28] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 77691 2020.
[29] Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contexmuliized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 39-48, 2020.
12

---

[30] Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsuper- vised Machine Translation Using Monolingual Corpora Only. International Conference on Learning Representations, 2018
[31] Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-End Adversarial Memory Network for Cross-domain Sentiment Classification. In Proceedings of the Twenty-Sixth International Joint Conference Artificial Intelligence, pages 2237-2243, 2017.
[32] Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models. In International Conference on Learning Representations, 2022.
[33] Jimmy Lin, Rodrigo Nogueira, and Andrew Yates, Pretrained Transformers for Text Ranking: BERT and Beyond. Springer International Publishing, 2022.
[34] Tie- Yan Liu. Learning Rank for Information Retrieval. Foundations and Trends in Information Retrieval, 3(3):225-3311 331, 2009.
[35] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Machine Learning, pages 97-105, 2015.
[36] R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis. Wiley, 1959.
[37] Ma, Ivan Korotkov, Yinfci Yang, Keith Hall, and Ryan McDonald. Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1088, 2021.
[38] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. In Proceedings of the 30th International Conference on Machine Learning, pages 10-18, 2013.
[39] Rodrigo Nogueira and Kyunghyun Cho.
Passage Re-ranking with BERT, 2020.
19X.v:191..04085 [cs.
[40] Rodrigo Nogucira, Zhiying Jiang, Ronak Pradcep, and Jimmy Lin. Document Ranking with a Pretrained Sequence-to-See to-Sequence Model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708-718, 2020.
[41] Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering, 22110):334--135) 2010.
[42] Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. SetRank: Learning Bermutation-Invariin Ranking Model for Information Retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 499-508, 2020.
[43] Robin L. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society Series C (Applied Statistics), 193-222,32 1975.
[44] Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The Expando-Mono-Duu Design Pattern for Text Ranking with Pretrained sannence-to-eqquenn Models, 2021. arXiv:2101.05667 [cs.IR].
[45] Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Mare Najork. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees? In International Conference on Learning Representations, 2021.
[46] Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, 2023. arxiv:2306.174 17563 [cs. [cs.
13

---

[47] Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5835-5847, 2021.
[48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[49] Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends@ in Information Retrieval, 3(4):333-389 2009.
[50] Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving Passage Retrieval with Zero-Shov Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3781-3797 2022.
[51] Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014.
[52] Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representation Learning for Domain Adaptation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pages 4058- 4065, 2018.
[53] Axel Suarez. Dyaa Albakour, David Corney, Miguel Martinez, and José Esquivel. A Data Collection for Evaluating the Retrieval of Related Tweets News Articles. In Advances in Information Retrieval, pages 780-786, 2018
[54] Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Bennett. Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, volume 1, pages 5030-5043, 2021.
[55] Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoff Gordon. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. In Advances in Neural Information Processing Systems, 2020.
[56] Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer Memory Differentiable Search Index. In Advances in Neural Information Processing Systems, 2022.
[57] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Mod- els. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
[58] Brandon Tran, Maryam Karimzadehgan, Rama Kumar Pasumarthi, Michael Bendersky, and Donald Metzler. Domain Adaptation for Enterprise Email Search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25-34, 34, 2019.
[59] George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana Barrio- Alvers, Michael Schroeder, lon Androutsopoulos, and Georgios Paliouras. An overview of the BioASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16(1):138, 2015.
14

---

[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems, 2017.
[61] Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, and lon Androutsopoulos. Domain Adversarial Fine-Tuning an Effective Regularizer. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3 3103-3112, 12, 2020.
[62] Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. ACM SIGIR Forum, 54:1-12, 2021.
[63] Ellen M. Voorhees. The TREC-8 Question Answering Track Report. In Proceedings of the Eighth Text Retrieval Conference, pages 77-82, 1999.
[64]
Ellen M. Voorhees. The TREC Robust Retrieval Track. ACM SIGIR Forum, 39:11-20, 2005.
[65] Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pages 2345-23600 2022.
[66] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-AM Natural Language Processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, 2020.
[67] Ruicheng Xian, Heng Ji, and Han Zhao. Cross-Lingual Transfer with Class- Weighted Language- Invariant Representations. In International Conference on Learning Representations, 2022.
[68] Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 4008 4020, 2022.
[69] Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the Use of Lucene for Information Retrieval Research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development Information Retrieval, pages 1253-1256, 2017.
[70] Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing Dense Retrieval Model Training with Hard Negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1503- 1512, 2021.
[71] Han Zhao, Shanghang Zhang, Guanhang Wu, José M. Moura, Joao P. Costeira, and Geoffrey J. Gordon. Adversarial Multiple Source Domain Adaptation. In Advances in Neural Information Processing Systems, 2018
[72] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On Learning Invariant Representations for Domain Adaptation. In Proceedings of the 36th International Conference Machine Learning, pages 7523-7532, 2019.
[73] Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen Wu, Ravi Krishna, Joseph E. Gonzalez, Alberto L. aanggiooannn-Vincetteee Sanjit A. Seshia, and Kurt Keutzer. A Review of Single-Source Deep Unsupervised Visual Domain Adaptation. IEEE Transactions on Neural Networks and Learning Systems, 33(2):473 493, 2022.
[74] Xiaofeng Zhu and Diego Klabjan. Listwise Learning to Rank by Exploring Unique Ratings. In Proceedings of the 1 13th International Conference on Web Search and Data Mining, pages 798-806, 2020.
15

---

[75] Honglei Zhuang, Xuanhui Wang. Michael Bendersky, and Marc Najork. Feature Transformation for Neural Ranking Models. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1649-1652, 2020.
[76] Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang. and Michael Bendersky. RankT5: Fine- Tuning T5 for Text Ranking with Ranking Losses. In Proceedings of the 46th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 2308-233 2313, 2023.
[77] Shengyao Zhuang and Guido Zuccon. TILDE: Term Independent Likelihood moDEl for Passage Re-ranking. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Reirieval, pages 1483-1492, 2021.
[78] Shengyao Zhuang, Hang Li, and Guido Zuccon. Deep Query Likelihood Model for Information Retrieval. In Advances in Information Retrieval, pages 463-470, 2021.
16

---

A Proofs for Section
Proposition A.l. Let p,p' be distributions supported on a metric space (X, d), define and assume
R = (2)x:,))))supppxxppp d(x, S oc, and let [(f) denote the balanced 0-1 loss of a discriminator
f X {0, 1}, given by
C(f) = Ex
(1- f(X'),,
then Wi (1, S R(1 min, (())).
Proof. Let I' n²) denote the collection of couplings between μl nl, then by the definition of the Wasserstein-1 distance,
Wi(un') inf nEF(mN") XxX
d(xr,x') d'(
/ S R inf
1 & a²) d7)))')x
TEr(mN") xxxx
/
R
1 sup
1|x x x')dr((z
TEL(mN") xxx
R
min(u(x)),x(xx)) dz
R
f max(0, mn,x(0,,11 - pl(x)) dx
R
2 2.xx S. | lu'(x) p(xx) da,
because (x)'(x) p(x)) dr 0. Note that the last line R times the total variation between 14, J..
On the other hand, rewrite
( /ttp:///-oooo-----------//-/--/- - (1:)) dr.
(()) = (I(z) + (1- ((=)) -C- =1 dz)
We know that the minimum 0-1 loss is attained by the Bayes predictor,
(x) 1/x((x) 2 m(,)),
min L() = -(5) = [m(=) (w(x) (x))
1
W (i(
R
by Eq. (T.. The result then follows from an algebraic rearrangement of terms in Eq.
L
Next, we provide concrete example (supplementing the picture Fig 2 for showing that item-level alignment does not necessarily imply list-level alignment. Example A.2. Consider two uniform distributions jllit pulst of two lists containing three real-valued items, where the lists are
supp(u ((1,2,3) (4,5, supp(e (11,3, (2,4,))
The level distributions, pm jutrm which are obtained by aggregating the items from all lists, both the uniform distribution of the same six items:
supp(u supp(u"")) 2,,,,,,,4,,,
Note that item-level representations are automatically aligned since 4 peee but the list-level representations are not, because supp(ulitt) list) suppp(ptt)) = 0.
B Proofs for Section 4
This section provides the proof to Theorem 4.7 the domain adaptation generalization bound for ranking with list-level alignment. First, recall the following properties of Lipschitz functions:
17

---

Fact B.1 (Properties of Lipschitz Functions).
1. If R + R is differentiable, then it is L-Lipschitz (in Euclidean distance) if and only if
|V/lla S L.
2. If f xX-+ R is L-Lipschitz and + IR is M-Lipschitz, then (af + bg) E Lip([a|L [6[M), and max(f.g)
(M
3. If X + is L-Lipschitz and y + 2 is M-Lipschitz, then g s E Lip(LM))
Proof.
1. For the backward direction, suppose bounded gradient norm, |V I L, then by the mean value theorem, 3t E [o, 1] s.t. f() - f(x) = fz) (y - x) with 2 = (1 t)a + ty, so by Cauchy-Schwand inequality,
11(() f())ll2 < [f(2)]l2 byy =112 T.yy x 21|2-
For the forward direction, suppose E Li)(L), then by differentiabilityy vf(x)² f(r - 2) - f(x) + 0(12112). Set = tV (x), we have
f f(=+ f(x)
otV & +fl
and the result follows by dividing both sides by f ll and taking 0.
2. First,
(af())) bo(()) s - f()))+ -(v]
- (|a|L= + [b]M) dx(,,)).
Next, assume w.l.o.g. (fx) max((m)) 0, then
(max(f(z), max
[f(x) max(f(y).g()) - f(x) - f(y) s L dx((z,)
ff) 2 g(x) else
(g(x) max(f()).g(v) S g(r) g(v) s dx(x,))
< M) (x
3. dz(g f(x),g f()) < M dy ((()), S LM dx (2,
We first prove Theorem 3.3 as a warm-up (restated below), the domain adaptation generalization bound for binary classification [22. since its proof shares the same organization that of our Theorem 4.7 and it helps familiarizing readers of prior domain adaptation results and analysis techniques.
Theorem B.2. Let binary classification problems source and a target domain be given by joint distributions ps, pT of inputs and labels, (X,Y) X x {0, 1}. Define the error rate of a predictor f:X X [0, 1] on n by
R(f) = Exx,)) (f(x YY 1] + (1 f(X)) 1|Y *0].
Let F C [0. be an I.-Lipschitz class of predictors. Define the minimum joint error rate by X" minp (Rs(f") + Rr(f"), then for all f E F.
Rr(f) < Rs + 2L Wi(PS 1x +
Proof. Define random variable 7 = 1| 1]. then we may rewrite
R(f) = Exx,y)=ul1 (2n 1x)(
Note that because 27) - 1 E (-1, +1},
R(f) R(f')) = -x ~M [m]
(2n - 1)f(x)] - E(x.r> - (2n (x)]
-xx (21 - (x) - -f(x))
$ Ex [I - f(X)]..
18

---

On the other hand,
Ex-px ( E(x.)~ (2 (f(()) f(x)) - -7+ 7
S Exxy)~lll (22) 1)f(X) - nl + Exx,)~al - (X) + ml
E(x,))=l1 (2n 1)f(x)) + +Bxx-)-ll -(2 10 1x)
R(f)) + R(f)..
Then by Fact I.I the fact that the operation of taking the absolute value is I-Lipschitz, and Defini- tion 3.2 of W1 distance, for all f. f F.
Rr( = Rs(f) +
- -RT(()-Rr(F))) - + (0000000000000000010---------0--
Rs((s) + Ex-ux [1 - ['(x]]
(xx]] + RT()))
Rs((f) + sup (B (Exmui [g(x)] - -Exun [x())]) + (Rs(f))RRr(()
vELip(21.)
S Rs(() + 2T (in 1 + (Rs( + R. (f')),
and the result follows by taking the min over f'.
Next, we prove Theorem 4.7 The main idea is that under the setup and assumptions in Section Rg and Ry can be written as the expectation of some Lipschitz function of Z n and μzz respectively, so by Definition 3.2 their difference is upper bounded by a Lipschitz constant times W WH mz).
Although omitted, remark that Theorem 4.7 be extended to the cutoff version of the ranking metric u (e.g., NDCG@p) p) with a simple modification of the proof. Also, finite sample generalization bound could be obtained using, e.g., Rademacher complexity, and additionally assuming Lipschitzness of the end-to-end scoring model [ 51].
Proof of Theorem 4.7 Fix g E G, and consider fuz for now (analogous results hold for (K)). Which
by Assumption 4.6 when restricted to supp(rg) ), has an Ly -Lpischct inverse g Define function
Eh,9 : Z - R>o for given h : Z + Re as
(2) = max (r, Us og (z))
E ER- [u ys og (2))l
(3)
rESt
exp(h())m())
max
u(, ys og (x)) reSt
Σ rESt
u(r,ys o (z))
II
E ex)))
and note that (s o g) = Ex=uğ (x()] =: Ez (z)]. We show that en.g is Lipschitz provided that h is Lipschitz, from establishing the Lipschitzness of both terms in Eq. (3)
For the first term, because is L,, Lipschitt w.r.t. ys (x) and ys o g (2) is in turn Lyg.g-
Lipschitz w.r.t. by Assumptions 4.3 and 4.4 it follows that u(r, us g (2)) is L. by Lg-
Lipschitz w.r.t. for any fixed r E Se, and therefore so is the function > maxres, w(r, usog (z))
by Fact B.
For the second term, we show that is Lipschitz (in Fuclidean distance) w.r.t. both ys o g (2( =: and h(2) s. By Jensen's 's inequality and Fact B.I
V y E ER-Pececc (u(R, y)]ll2 = Rt
V Vou(R R. vill2 $ ER-pertte Vun(R, v)11a] < L..
19

---

And by the definition that Vaf(2)= = 1am f(z), ...
0 [
a Oza
f(x)] for any f: - R.
V LE Respasepl [u( R. y)ll
v B Reapco
[u(E v)ll1
8 exp(sn(rii)
Σ u(ry)) m r
@s((r)m
Ej=, exp()1(ms)
exp(s1(ri))
exp(si(r)..)
u v)
(s1(r)m Σラ= exp(s1(1)))
exp($1(0)))
m= rESt
exp exp($1(0)m)
exp(si(r)m)
u(r.y)
II
1(m
Žj=n exp(ss()))
Σラ exp(s1(7)))
rES,
exp(sI(()m))
exp(s1(r)m)
B
[mm [
Anml Ejen exp(si(())
Σラ exp(s1(r)))
mml 1 rESt
2m1
exp(s1(()))
exp(s1(r)m)
B
1 1mm 8 + "xp(xxp(10))
exp(s1(M))
rESt
exp(s1(r)a)
Σ m=i exp($1(r)m)
B
exp(s1(m)))
Ej= exp(s1(M))
rES
exp($1(m).)
2BB
A= Žj=n exp(sr()))
rES,
-28,
where the second equality is due to the product rule, the third equality to d softmax(()); (1ft = o - softmax(z);), the last inequality to Assumption 4.3 and the last equality identifying the pmf of the P-L model (Definition 4.1). Because h Lip(Ln. by Assumption 4.5)
az,
the two results above imply that the second term in Eq. 3 is Lipschitz: for all 2, z' E Z,
E Reeeettttt) (u(R, ys o
(x))] - ER- [u (s og g
$ E ER-Post(Ms)) (u R ys o g (x))] E ER-PeSothtdn [ul Rs g (='))]
+ E Peppnn)) (R, us og (x'1)] ER [u(R, us o (2'))]
< Lullus og ( us (x')ll2 2BE((((( M(=')l2
$ (LaLnLy + 2BLL,) dz dz
Combining the two parts, we have that en.g is 2Lo + BfL,) Bipssciis in Finally, by Fact B.ld Definition 3.2 for all g E 9 and h, h¹ E H.
Rr(h o g) Rs(h + (R RT(h' g)) - (Rs (h g) + Rs(h' o g))
(Rs o gg) Rr g))
s Rs(h + (Rr(h g) Rr(h's o g)) (Rs (h g) - Rs(h's og)
(Rs(N' og) Rr g)
= =Rs(hog) + E Ez (en FM (2)]
E ENzuz (2) - E (
(R o og) + R g))
< <Rs(hog) +
sup vEL.p(4(LL.,,, + (,
( (Rz~ny [7) - E Ezunng (g()))
(R o og) Ry (h' og) < Rs(h o + 4(Ln BELa) W Wi mz) + (Rs(h' gg) + R (Ri og) and the result follows by taking the min over h'.
20

---

Finally, we verify the Lipschitzness of RR and NDCG.
Proof of Corollary 4.8
Because RR < uniformly and lyy 1lu-v'lla
for for v. E (0.1),
so y + (R( y) is 1-- Lipschitz.
0
Proof of Corollary 4.9
show that NDCG is Lipsehitz w.r.t. y under the assumptions. Recall that
DC NDCG v) IDCG(y)
y/3 log(r; + +1)
y/1
log +1
where r" is the descending order of y-
Note that IDCG continuous and piecewise differentiable w.r.. y (each piece is associated with some r' E Sq and supported on {y arg max, DCG(r, y) - r')), then is IDCG on the set y" = {v : C < IDCG(y) S C} (which is our assumption) by chain rule. Clearly, DCG is continuous and differentiable w.r.t. y. so NDCG is continuous and piecewise differentiable on ) by product rule. By Fact to show that NDCG is Lipschitz, we just need to show that its gradient norm w.r.t. y bounded when evaluated on (the interior of) each piece.
Let E St be arbitrary, and E Y" be s.t. it lies the interior of the respective piece, namely, by denoting r" the descending order of y. the interior of the set (y' arg max, DCG(r, y²) = r*) {v/ : V 2 V)(r')2 2 ... 2 ) (recall that I(r); denotes the index of the item with rank i).
Then, for all & E [e].
0 NDCG(r, Oyk
IDCG(1) 0
0 Oyk
DCG((, v)
Pyk
1=1 log(r: (T: 1)
log (rz + 1)
< IDCGG))) + 1)
+ w)ogg( + 1)²
IDCGGG)) log 11 DDCG(r; v) log(e 1)
CC+ log(( 1)
hence V NDCG 12 < Va(C C Colog(e 1))) < Ö(CVE), and NDCG is Ö
w.r.t. y. Here, Ô hides polylogarithmic terms.
C Additional Experiments and Details on Passage Reranking
We perform additional experiments for unsupervised domain adaptation the passage reranking task considered in Section 5 in Appendix [C.I] provide case studies on ListDA vs. zero-shot and
QGen PL (Tables and hyperparameter settings in Appendix C.2 and details the construction of training lists in Appendix C.3
C.1 Additional Results
Pairwise Logistic Ranking Loss. Besides the listwise softmax cross-entropy ranking loss used in
Section 5
exp(s;)
((s = Σ
log 1=11
=1 exp(s;)
we experiment with the pairwise logistic ranking loss
8
exp(si)
((.
Σ
Vz log
cxp(s:) + cxp(s;))
i=1j=1
The results with using the pairwise logistic loss for training on the Robust04 dataset are provided in Table [a]] which are not better than using the softmax cross-entropy loss (cf. Table T see also (5)).
hence further experiments with this loss pursued.
21

---

Table 2: Reranking performance of RankT5 top 1000 BM25-retrieved passages; additional results extending Table
(a) With pairwise logistic ranking loss in place of softmax cross-entropy Robust04.
Target domain
Method
MAP
MRR@10 10
NDCG@5
NDCG@ 10
NDCG@20
Zero-shot
0.2656
0.7894
0.5671
0.5163
0,4729
Robust04
QGen PL
0.2776
0.7975
0.5576
0.5267
0.4892*
ItemDA (MLP)
0.2766
0.8021
0.5794
0.5355
0.4917'
ListDA
0.2893
0.8103
0,5935
0.5524
0,5044
Source domain MS MARCO. Gain function in NDCG identity map. 'Improves upon zero-shot with statistical
significance 0.05) under the two-tailed Student's 1-test. 'Improves upon QGen PL. *Improves upon ItemDA.
(b) With ItemDA with a transformer discriminator.
Target domain
Method
MAP
MRR@
NDCG@5
NDCG@
NDCG@20
Robust04
0,2848
0.8018
0,5851
0,5406
0,4982
TREC-COVID BioASQ
ItemDA (transtormer)
0,3088
0.8907
0.8287
0.8069
0.7691
0,4739
0.6420
0.5256
0.5312
0.5567
Source domaii MS MARCO. function in NDCG identity map.
(c) With ListDA + QGen PL method.
Target domain
Method
MAP
MRR@ 10
NDCG@5
NDCG@
NDCG@20
Robust04
0.2851
0.8039
0.5761
0.5386
0.49751
TREC-COVID
ListDA + QGen PL
0.3168
0,8950
0,8539
0,8292
0.7820
BioASQ
0.6538
0,5158
0.5547
0.5671 4
0.5931
Source domain MS MARCO. Gain function in NDCG is the identity map. 'Improves upon zero-shot with statistical signiticance s 0.05) under the two-tailed Student's f-test. 'Improves upon QGen PL. *Improves upon ItemDA.
Table 3: Reranking performance of RankT5 on top 1000 BM25-rctrieved passages on Signal-
Targel domaii
Method
MAP
MRR@10 10
NDCG@5
NDCG@ 10
NDCG@20
BM25
0.1740
0.5765
0.3639
0.3215
0.2905
Zero-shot
0.1511
0.4804
0.3068
0,2685
0,2410
Signal-IM
QGen PL
0.1541
0.5043
0.3238
0.2799
0.2497
ListDA
0.1456
0.4629
0.3002
0,2602
0,2328
LisIDA QGen
0.1549
0.5170
0.3261
0,2817
0.2505
Source domain is MS MARCO. Gain function in NDCG is the identity map.
As an implementation remark, in this set experiments, we do not perform pairwise comparisons obtain the predicted rank assignments during inference (where lists have length-1000) due the high time complexity (the loss is aggregated pairwise during training, since we truncate training lists to length-31; Appendix
Whether not the forward pass of the model involves pairwise computations does not matter to ListDA, which is applicable to any (pointwise, pairwise, or listwise) model long as we can gather list-level representations: ListDA could be instantiated on pairwise models e.g. DuoT5 although not pursued in this work.
ItemDA with Transformer Discriminator. To show that the improvements brought by ListDA over ItemDA is due to learning list-level invariant representations and not because of the use of the more expressive transformer as the discriminator vs. MLP, we experiment with variant of ItemDA that uses the same transformer discriminator used in our ListDA runs. Although, whereas each input to the ListDA discriminator is a list of feature vectors as a length- sequence, each input to the ItemDA transformer discriminator is single feature vector, sequence of length-1. So, the attention mechanism of the transformer becomes pointless, and the ItemDA transformer discriminator collapses
remark that
MLP (with skip connections and LayerNorm).
The results are presented Table 2b (cf. Table T.. Across all datasets and metrics, no consistent improvement of ItemDA (transformer) over ItemDA (MLP) is observed.
22

---

ListDA + QGen PL Method, and Signal-1M Dataset. We experiment with supplementing ListDA with QGen pseudolabels by (uniformly) combining the training objectives of ListDA and QGen PI. (ListDA QGen PL.) The results are presented in Table We also apply this method on the Sig (RT) dataset [53], with results Table 3]
observed that reranking using neural rankers transferred from MS MARCO source domain does not perform better than BM25 (i.e.,
without any reranking) on Signal- IM, which is also noted in prior work [57] 32 This does not mean that neural rerankers are worse than BM25, but could be that MS MARCO is not good choice as the source domain for the target of Signal- MM, because of the arguably large domain shift between tweet retrieval and MS MARCO web search- qualitatively, it can be seen from Table 7 that the text styles and task semantics of the two domains are very different. Hence, the following discussions on Signal- 1M results focus comparing reranking results.
On Signal-IM, QGen PL. improves upon the zero-shot baseline, but ListDA does not, which likely due to the large domain shift between MS MARCO and Signal-1M that prevented ListDA from finding the correct source-target feature alignment without supervision. By incorporating QGen pscudolabels, ListDA performance improves with QGen PL, which could have benefited from QGen q-d pains acting as anchor points for ListDA to find the correct correspondence between source and target.
Overall, ListDA QGen PL the only method that consistently improves upon the zero-shot baseline on all four datasets, including Signal- IM, although it underperforms ListDA without QGen PL on the other three. Further improvements to this hybrid method may be possible with better strategies for balancing the training objectives of ListDA and QGen PI..
Case Studies. In Tables S and 9 we include examples where the ListDA top-1 reranked results are better than those of zero-shot and QGen PL, respectively, for qualitatively understanding the benefits and advantages of ListDA.
In zero-shot, the reranker is trained the general domain MS MARCO dataset and applied on the target domains without adaptation. When the target domain differs from MS MARCO stylistically or consists of passages containing domain-specific words, as in the TREC-COVID and BioASQ datasets for biomedical retrieval, the zero-shot model, which has not seen the specialized language usages, may resort keyword matching. Examples of such cases are presented in Table 8 where the top-1 passages reranked by the zero-shot model contain keywords from the query but are irrelevant.
In QGen PL, the reranker is trained by assuming the QGen synthesized query-document pairs to be relevant. However, as remarked in Section 5 because QGen is trained on MS MARCO and applicd on the target domains in zero-shot manner, the pseudolabels are not guaranteed to be valid, meaning that there could be false positives. This is observed in examples presented in Tables 7 and 9 One specific scenario where false positives hurt transfer performance is when the synthesized queries (of false positive documents) coincide with queries from the evaluation set, observed from the examples presented in Table on TREC-COVID and BioASQ datasets. On the other hand, ListDA does not assume the pseudolabels in its training objective, so it does not suffer from these potential issues with QGen PL.
C.2 Hyperparameter Settings
For BM25, we use the implementation of Anserini 69. set k; 0.82 and b = 0.68 on MS MARCO source domain, and k, = 0.9 and b = 0.4 on all target domains without tuning. As in [57] titles are indexed separate field with equal weight as the document body, when available.
For the RankT5 reranking model, it is fine-tuned from the T5 v1.1 base checkpoint on a Dragonfish TPU with 8x8 topology for 100,000 steps with batch size of 32 per domain (cach training list contains 31 items). We tune the learning rate Thank E [5e-5, Ic-4, 2c-4], and select the one with the best zero-shot performance to use all adaptation methods, each dataset (see Fig. 5 for zero-shot sweep results). We apply a learning rate schedule on Tank that decays (exponentially) by a factor of 0.7 every 5,000 steps. The concatenated docymett text inputs are truncated to 512 tokens.
For the domain discriminators, there are two hyperparameters: the strength of invariant feature learning X, and the discriminator learning rate 7hd- We tune both by sweeping À E [0.01, 0.02), and 1/a E (10, 20, 40) x Thank as multiples of the reranker learning rate. The tuned hyperparameter settings of Trank lad and À used in our experiments are included in Table 4
23

---

Robust04
Robust04
0.58
0.565
0.560
0.56
0.555
TREC-COVID
TREC-COVID
NDCG@5
NDCG@5
0.825
0.800
BloASO
BloASQ
0.55
0.53
0.54
0.53
0.52
0,00005
0,0001
0.0002
0.00005
0,0001
0,0002
Orakk
(a) With softmax cross-entropy ranking loss.
(b) With pairwise logistic ranking loss.
Figure 5: Zero-shot performance under different hyperparameter settings for Trank-
Robust04
Robust04
0.600
0.575
ND0000
0.875
TREC-COVID
NDCG
0.86
TREI-COVID
0.850
0.825
BloASQ
BloASQ
0.56
0.56
0.54
0.54
0.001
0.01
0.1
10
20
40
fid
Xhak
(a) Fix 7ht 0.004 and sweep A.
(b) Fix À = 0.02 and sweep Tae (as multiples of Trank).
Figure 6: ListDA performance under different hyperparameter settings for X and Mad- Grey horizontal line is zero-shot performance with the best 7rank setting.
The sweep results for ListDA are plotted in Fig. 6 It is observed that balanced choice of À is needed to clicit the best performance from ListDA, and the same choice works fairly well across datasets. But for 7nd. each dataset prefers different settings, probably due to their distinct domain characteristics.
In terms of running time, the adaptation methods of ListDA, ItemDA, and QGen PL all have double the training time compared zero-shot learning due to data loading. This is because in addition to source domain data, the adaptation methods require target domain (unlabeled) data. Among ListDA, ItemDA, and QGen PL, the training times roughly the same, because the overhead of the domain discriminators is not significant.
C.3 Training List Construction
Recall from the setup of ListDA ranking problems that, the inputs are defined over lists and the invariant representations are learned at the list level. In other words, the ranking loss and the adversarial loss are to be computed on lists of ranking scores and feature vectors for each query (containing both relevant and irrelevant ones).
In our reranking setup, each list would be the top-r passages retrieved by BM25 on a query, where we set r = 1000 in our evaluations. However, there are two complications. The first that due to memory constraints, it is not feasible to feed all 1000 q-d pairs from each list through the RankT5 model in one pass during training (or time-consuming if via gradient accumulation). The second is that the source domain MS MARCO dataset only labels one relevant passage for each query, meaning that out of the 1000 passages retrieved by BM25 for each query, we may only know that (at most) one of them is relevant; the labels for the remaining 999 passages are unknown.
Negative Sampling. To address both issues, we truncate the list to length-31 for training and perform random sampling to gather irrelevant q-d pairs from the top 1000 BM25 retrieved passages.
24

---

Table 4: Hyperparameter settings of RankT5 model and domain discriminators for passage reranking
experiments.
(a) With softmax cross-entropy ranking loss.
Target domain
Method
Baak
A
Zero-shot
QGen PL.
Robust04
ItemDA (MLP)
le-4
le-3
0.01
ItemDA (transforme)
2c-3
0.01
ListDA
4c-3
0.01
LisIDA QGen PT
le:3
0.02
Zero-shot
QGen PT.
ItemDA (MLP)
8c-3
0.01
TREC-COVID
2e-4
ItemDA (transforme)
2c-3
0.01
ListDA
4e-3
0.01
ListDA + QGen PL
4e:3
0.02
Zero-shot
QGen PT.
BioASQ
ItemDA (MLP)
2e-4
4c-3
0.01
IteDDA (transformer)
2c:3
0.01
ListDA
8e-3
0.01
ListDA + QGen PL
4e:3
0.02
Zero-shot
QGen PT.
Signal-IM
5c-5
ListDA
2e-3
0.01
ListDA QGen PL
1c-3
0.02
(b) With pairwise logistic ranking loss.
Target domain
Method
Thank
That
Zero-shot
QGen PL
RohustO4
le-4
ItemDA (MLP)
2e-3
0.01
ListDA
2c:3
0.01
On the MS MARCO source domain, given query q and the top 1000 passages retrieved by BM25, d1, ... d10w0, we yield a training list by bundling the one passage d' labeled as rel- evant with 30 randomly sampled passages dy..... .d be treated as irrelevant, i.e., (Ig, dr), [g, d, ... 1q, dy and = (1,0,--. 1..00,....000 .0)
On the target domains, we follow the same procedure but with QGen synthesized queries. Given QGen query q and the top 1000 retrieved passages, the training list consists of the pseudolabeled passage d (i.e., the passage on which the query q was synthesized) and 30 randomly sampled irrelevant passages, i.e., x = ([g. d]. [g, dN ..., y the target domain are used by QGen PL for training but discarded by ListDA.
...,,,e [g. d and y = (1,0..... 0). Note that the pseudolabels
Reducing False Negatives on MS MARCO. One potential problem arising from sampling is that the constructed lists may contain false negatives (i.e., relevant documents that incorrectly treated as irrelevant). In fact, false negatives are prevalent in the MS MARCO dataset (e.g., due to duplicate passages). While these false negatives are mostly harmless source domain supervised training because the effects are canceled out when there are positively labeled q-d pairs to which the false negative q-d pairs are similar, false negatives can affect invariant representation learning.
One way that may negatively affect invariant representation learning is that the duplicates in the lists (which have identical feature vectors) will cause ListDA to collapse distinct passages on the target domain to the same feature vector for achieving alignment, which is an artifact that will cause information loss in the target domain feature representations.
To lower the chance sampling false negatives on MS MARCO, we rerank BM25-retrieved passages using ranker pre-trained on MS MARCO, and sample negatives from passages ranked at 300 or lower, as the duplicates and (unlabeled) relevant passages are concentrated the top [47]. We apply this sampling method only when constructing source domain lists for feature alignment (namely, in
25

---

ListDA and ItemDA), and it only affects the computation of adversarial loss. This sampling method not applicable to the target domains because we do not have reliable pre-trained rankers for them. is also not used for constructing the lists for source domain supervised training (i.e., the computation of ranking loss), because it excludes "hard negatives" from the training lists and results in a weaker reranker, as observed in our preliminary experiments.
D Experiments on Yahoo! LETOR
We evaluate ListDA on the Yahoo! Learning to Rank Challenge v2.0 [11], and compare it to zero-shot and ItemDA (QGen PL. is not applicable to tabular data). It is a web search ranking dataset with two subsets, "Set 1" and "Set 2", whose data originate from the US and country in Asia, respectively; the training set sizes are 19,944 and 1,266, and the test set sizes are 6,983 and 3,798 (of which there are 47 and 17 lists of length-1). The dataset is tabular, and each item is represented by 700-d vector with values in the range of [o, 1]. Among the 700 features, 415 are defined on both sets (shared), and the other 285 are defined on either Set or (disjoint); we hence write each item x = [astared, Fdiijoiinl as concatenation of the shared features and the disjoint ones, 3ihared E R 115
Tdissjoi <R285
We consider unsupervised domain adaptation from Set to Set 2 (i.c., we discard the labels in Set 2). Our implementation uses Py PoTch and the Hugging Face Transformers library [66]
Models. Our models have the same setup as that of the passage reranking experiments in Section 5
except that the RankT5 text ranking model is replaced by a three- hee-hhddde-layye layer MLP following [75] and we treat the list of 256-d outputs at the last hidden layer as feature representations:
g(x); c2 ReLU W3 ReLLUWWaRLLUTE
shhared + bi) + 62) ReLLU( TT disijoiint + 1/1 + 1/2)
+ tbs)
h(x); =: W47; + b4,
where W1 E I 10202X4155 W1 E R1021X285 W2, W E 2 W3 E R256x512 and W4 E R! x and are randomly initialized. Note that we use separate input layers (W1, W() and first hidden layers (W2, Wt)) for the shared and disjoint features, then concatenate the hidden representations. This tweak improves performance of the zero-shot bascline.
For ItemDA, we use the same ensemble of five three-layer MLPs as in Section For the ListDA discriminator, an ensemble of five transformers is used, where each transformer is stack of three T5 encoder blocks, with 4 attention heads (num_heads)) size-32 key, query and value projections per attention head (d_vv) and size-1024 intermediate feedforward layers (d_ff).
Results. The results are presented in Table 5 which are ensembles of 10 separately trained models to reduce the variance from the randomness in the initialization and training process, due the small dataset size. Yahoo! LETOR is annotated with 5-level relevancy, and the scores are binarized for MAP and MRR metrics by mapping 0 (bad) and 1 (fair) negative, and 2 (good), 3 (excellent), 4 (perfect) to positive. Thanks to the availability of labeled data on Set 2, we also include supervised results from training on Set I and 2 to serve as an upper bound for unsupervised domain adaptation.
ListDA again outperforms ItemDA for unsupervised domain adaptation, collaborating the discussions in the main sections of the paper. Although, here, their performance gap is smaller compared to passage reranking results, which we suspect is because the contextual (i.e., query) information for defining the list structure of the data is too weak or discarded when the numerical features generated (e.g.. the numerical features could represent the difference between query and document, hence may not retain information about the original query).
Hyperparameters. The model is trained from scratch on an NVIDIA A6000 GPU for 10,000 steps with a batch size of 32 per domain (length of training lists varies from one to 140 items). We apply a learning rate schedule on Thank that decays (exponentially) by a factor of 0.7 every 500 steps.
We tune the hyperparameters by performing grid search over: Thakk E [le-5, 2e-5, 4c-5, 8e-5, lc-4, 2e-4, 4c-4, 8e-4, ),- Rod E f0.2, 0.4, 0.8, 2, 4, 8, 10) x Thank as multiples of the ranker learning rate, and À E [0.08, 0.1, 0.2, 0.4, 0.8, 1). The tuned settings are included in Table 6
"Our code for this set of experiments is available in the supplementary material, and the data can be requested hhtttps://websoost sandbox. yahoo. www.//otcatt.aee catalog. phtttttaty
26

---

Table 5: Ranking performance MI.P ranker on Yahoo! LETOR (Set 2).
Target dommain
Method
@RR@10
NDCG@5
NDCG@ 10
NDCG@20
Zero-shot
0.5464
0.6796
0.7466
0,7778
0.8308
Yahoo! LETOR (Set 2)
Supervised
0.5714
0.7045
0.7760
0,8012
0.8479
ItemDA
0.5604
0.6984"
0.7563
0.7822
0.8350"
LisIDA
0.5633
0.7026
0.7599
0.7845'
0.8355
Results are from an ensemble of five scoring models. Bold indicates the best unsupervised result. Source do- main Yahoo! LETOR (Set 1). Gain function in NDCG identity map. 'Improves upon zero-shot with statistical significance (p I 0.05) under two-tailed Student's f-test. 'Improves upon TtemDA, Significance tests performed supervised results.
Table 6: Hyperparameter settings of MLP ranker and domain discriminators for Yahoo! LETOR experiments.
Target domain
Method
Bonk
A
Zero-shol
8e-4
Yahoo! LETOR (Set 2)
Supervised
4c-5
ItemDA
8c-4
1.6e-3
0.4
LisIDA
8e-4
1.6e-3
0.8
27

---

Table 7: Examples relevant q-d pairs from the test set and pairs synthesized by QGen on domains considered in Section 5 experiments. Text truncation omission are indicated by "[...]'.
Dataset
Relevant q-d pains
QGen q-d pairs
MS MARCO
What cartography? A. science of map- making B. the science of shiphuilding C. the science of charting direction on ship D. the science of measuring distances on the occan. Cartography is the science of map making A. Q: what is the science of mapmaking called
D:
D:
The fl shot also contains the following in- gredients: sodium phosphate & buffered isn- lonic sodium chloride solution, formaldehyde,, octylphenol ethoxylate, and gelatin, accord- ing the FDA.
Q: what's in the flu shot
TREC-COVID
D: An Evidence Based Perspective on mRNA- SARS-CoV-2 Vaccine Development. [...] The production mRNA- -asedd vaccines is promising recent development in the pro- duction of vaccines. However, there remain significant challenges in the development [..]]
D: Impact of arterial load on the agreement be- tween pulse pressure analysis and esophageal Doppler. INTRODUCTION. The reliahility of pulse pressure analysis to estimate cardiac output is known to be affected by arterial load
changes. [...]
Q: what is known about mRNA vaccine for the SARS-CoV-2 virus?
QGen: what is arterial load for pulse pressure
analysis
D: The possible pathophysiology mechanism of cytokine storm in elderly adults with COVID- 19 infection: contribution of "intlame- aging". PURPOSE: Novel Coronavirus dis- case 2019 (COVID-19)) 19), is acute respiratory distress syndrome (ARDS), [...]
D: Opportunity Costs Pacifism. If the resources used to wage wars could be spent elsewhere and save more lives, does this mean that wars unjustified? This article considers this question, which has been largely overlooked by Just War Theorists and pacifists. It focuses on whether the opportunity costs of war [...] QGen: opportunity pacifism
Q: What the mechanism of cytokine syn- drome on the 19?
BioASQ
D: role of extended-release amantadine the treatment dyskinesia Parkinson's dis- case patients. [..]] extended-release aman- ladine (ammntadiie ER) is the first approved medication for the treatment of dyskinesia. When it is given at bedtime, it [...] Q: Is amantadine ER the first approved treatment for akinesia?
D: Subluxation the femoral head coxa plana. Twenty-two patients who had severe coxa plana had closed reduction for lateral subluxa- tion of the femoral head, [...] The average age when the patients were first seen was eight years and six months. L...]
QGen: average age of femoral subluxation
D: [...] comparison of proxy assessment and patient self-rating using the discase-specific Huntington's disease health-related quality of life questionnaire (HDQoL.) [...] Specific Scales of the HDQoL. On the Specitic Hopes and Worries Scale, proxies on average rated HrQul. as hetter than patients' [...]
D: [...] We investigated the health-related qual- ity life (HRQoL.) of long-term prostate cancer patients who received leuprorelin etate in microcapsules (LAM) for androgen- deprivation therapy (ADT). [...]
Q: Can leuprorelin acetate be used androgen deprivation therapy?
QGen: which scale is used for proxy assessment of hrqol
Signal-IM
D: BJP party MP R.K Singh's allegation that money has changed hands for tickets in #BiharPolls as bascless.
D: Black lives matter: thoughts from the delivery ward in St. Louis: #mnustread
QGen: where is black lives matter?
Q: Party MP calls BJP 'Baura Jayewala Party"
D: RETWEET if "Brenda's Got A Baby" is of your favorite @ 2Pac songs. #RIP2Pac QGen: brenda got baby pac
D: Kerry: US plans military talks with Russia over Syria
Q: Kerry: US plans military talks with Russia over Syria
28

---

Table 8: Examples where ListDA top-1 reranked result is better than zero-shot. Text truncation or omission are indicated by "[...]"-
Dataset
Zero-shot top-1 passage (that are irrelevant)
ListDA top-1 passage (that relevant)
Robust04
Q: Find information on prostate cancer detection and treatment.
D: [...] FIRST PATIENT UNDERGOES GENE INSERTION IN CANCER TREATMENT
D:
[...] Little knowledge goes long way Can- prostate need not a killer/ Health Check, Farlier this year, 13-year-old [...] died from cancer of the bladder and prostate. His death is a grim reminder that no malc should consider himself immune from waterworks trouble. The prostate, a gland about the size and shape chestnut, lics deep the polvis just below the bladder. Because it surrounds the urethra, has the potential to block the flow of urine completely. [...]
[..] This first round of gene transfer exper- iments, in which
was inserted into patient's 's white blood cells, is not expected to directly benefit an individual patient. Instead, the inserted gene being used track the movement in the body of the cancer-fighting white blood cells. [..]] Inserting human genes to repair defects may one day help with host of inherited disurders, [...]
TREC-COVID
Q: What are the longer-term complications of those who recover from COVID-19?
D: [...] Our previous experience with members of the same corona virus family (SARS and MERS) which have caused two major epi- demics the past albeit of much lower mag- nitude, has taught us that harnful effect of such outbreaks complications alone. Long term cardiopul- monary, glucometabolic and neuropsychiatric complications have been documented follow- ing these infections. [...]
D: Up 20-30% of patients hospitalized with coronavirus disease (COVID-19) have evi- dence of myocardial involvement. Acute cardiac injury patients hospitalized with COVID-19 19 is associated with higher morbid- ity and mortality. There acute treatment for COVID- 19 may affect valescent phase or long-term cardiac recovery and function. Myocarditis from other viral pathogens can evolve into overt or subclinical myocardial dysfunction, [...]
not limited to acute
data how
BioASQ
Q: What is the interaction between WAPL and PDS5 proteins?
D: Pds5 and Wpll act as anti-ostablishment fac- tors preventing sister-chromatid cohesion un- counteracted S-phase by the cohesin acetyl-transfcrase Esol. [...] Here, show that Pds5 is essential for cohesin acctylation by Esol and ensures the maintenance of co- hesion by promoting a stable cohesin interac- tion with replicated chromosomes. The latter requires Esol only the presence Wapl, indicating that cohcsin stabilization relies Esol only neutralize the anti-establishmen activity. [...]
D: [...] Here, we show that cohesin suppresses compartments but is required for TADs and loops, that CTCF defines their boundaries, and that cohesin unloading factor WAPL and its PDS5 binding partners control the length of loops. In the absence of WAPL and PDS5 proteins, cohesin forms extended loops. presumably by passing CTCF sites, accumu- lates in axial chromosomal positions (vermi- celli), and condenses chromosomes. [...]
29

---

Table 9: Examples where ListDA top-1 reranked result is better than QGen PL. along with the QGen synthesized query for the top-1 passage. Text truncation omission indicated by "[...]'
Dataset
QGen PL top-1 passage (that are irrelevant)
ListDA top-1 passage (that are relevant)
RobustU4
Q: Identify outbreaks of Legionnaires' discuse.
D: [..]] 3. Care Patients with Trachcostomy D: [...] LEGIONNAIRE'S DISEASE STRIKES
Suctioning of Respiratory Tract Secretions III. Modifying Host Risk for Infection A, Pre- cautions for Prevention of Endogenous Pncu- monia I. Prevention of Aspiration 2. Preven- tion of Gastric Colonization B. Prevention of Postuperative Pneumonna C. Other Prophylac- tie Procedures for Pneumonia 1. Vaccination of Patients 2. Systemic Antimicrobial Prophy- laxis 3. Use of Rolating "Kinetic" Beds Pre- vention and Control of Legionnaires' Diseuse [..]]
16 REUNION IN COLORADO; 3 DIE. An outbreak of legionnaire's 's disease S0th high school reunion was blamed Thursday for the deaths of three elderly celebrants and the pneumonia- illness of 13 others. State health officials contacted 250 other people from states who attended the Lamar High School reunion for the classes of 1937 through 1941 but found no new cases, Dr. Ellen Man- gione, Colorado Department of Health epi- demiologist, said. [...]
QGen: what kind of precautions are used to pre-
vent pneumonia
TREC-COVID
Q: what drugs have been active against SARS-CoV or SARS-CoV-2 in animal studies?
D: Different treatments currently used for clinical management of SARS-CoV-2 infec- tion, but little is known about their efficacy yet. Here we present ongoing results to com- pare currently available drugs for a variely of diseases to find if they counteract SARS- CoV-2-induced cytopathic effect in vitro. [..] We will provide results soon they be- come available, [...] QGen: what is the treatment for sars
D: [...] the antiviral efficacies of lopinavir- ritonavir, hydrooychlorrquuie sulfate, and cnntricttahineteeofooo for SARS-CoV-2 in- fection were assessed in the ferret infec- tion model. [...] all antiviral drugs tested marginally reduced the overall clinical scores of infected ferrets but did not significantly af- fect in vivo virus titers. Despite the polential discrepancy of drug efficucies between ani- mals and humans, these preclinical ferret data should be highly informative to future thera- peutic treatment of COVID-19 patients.
BioASQ
Q: What is the function of the Spt6 gene yeast?
D: As means to study surface proteins involved the yeast to hypha transition, human mono- clonal antibody fragments (single-chain vari- able fragments, scFv) have been generated that bind to antigens expressed on the surface of Candida albicans yeast and/or hyphac. [...] To assess C. albicans SPT6 function, expres- sion the C. alhicans gene was induced in defined cerevisiaesptó mutant. Partial com- plementation was seen, confirming that the C. albicans and S. cerevisiae genes are func- tionally related in these species.
D: Sptó is a highly conserved histone chaperone that interacts directly with both RNA poly- merase II and histones to regulate gene pression. [...] Our results demonstrate dra- matic changes to transcription and chromatin structure in the mutant, including clevated an- tisense transcripts at >70% of all genes and general loss of the nucleosome. Further- more, Sptó is required for marks associated with active transcription, including trimethy- lation of histone H3 on lysine 4, previously observed in humans but not Saccharomyces cerevisiae, and lysine 36. [...]
QGen: what is the function of sptó gene in can- dida albicans
30


Reviews:
[{"review_id": "MLrwDMdXbA", "reviewer": "Reviewer_3wjd", "summary": "This paper proposes a novel approach for domain adaptation on list ranking problems. In significant contrast with previous works, they adopt aligning list-level representations directly instead of item-level representations, and argue for its relevance in improving on list-level metrics pertinent to the problem of ranking. They also devise a theoretical bound for the target error in terms of source error, a joint hypothesis error and the list-level domain divergence, under suitable assumptions. Results on passage re-ranking task shows their strength in practical applications.", "questions": "Is there a typo in L98? Since both $\\mathcal{L}_\\text{rank}$ and $D(\\mu_s,\\mu_t)$ need to be minimized according L100-102, but the negative sign in between does the opposite.", "limitations": "The work is self-sufficient, and presents good evaluation to convey the effectiveness of the proposed method. Although, I feel that the scope of this or a follow-up work could be broadened a bit, to also incorporate problems beyond passage ranking into the evaluation.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well written, easy to follow and extremely well presented.\n\n- The idea of extending UDA approaches to list-level re-ranking problems is interesting and highly relevant. The ideas discussed in the approach might be relevant to problems beyond ranking (like object detection in computer vision which also uses unique evaluation metrics).\n\n- The empirical results on the passage re-ranking task is strong.\n\n- The simple of idea of using a transformer discriminator to allow permutation in-variance is impressive.", "weaknesses": "- Few choices, like the use of transformer discriminator instead of MLP, wasn't evaluated or ablated empirically, although it makes sense intuitively. \n\n- The authors might also consider showing results on diverse ranking tasks beyond passage re-ranking, possibly in tasks like fine-grained computer vision as in [1], although I agree that this is much beyond the scope of this work.\n\n[1] Wang, Xinshao, et al. \"Ranked list loss for deep metric learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time, comments, and support! We hope that the following addresses the concerns:\n\n- **“results on diverse ranking tasks… like fine-grained computer vision”**\n\n\tThank you for the suggestion, and it would be indeed interesting to see whether ListDA would be useful in (vision) tasks that reframe fine-grained classification as ranking. Regarding task diversity, in Appendix D, we also included experiments on the Yahoo! Learning to Rank (LETOR) Challenge v2.0 dataset, a web search ranking task on numerical data. The results are consistent with those from the passage reranking experiments and also support our findings in the main sections.\n\n- **“Is there a typo in L98?”**\n\n\tThank you for pointing this out! It will be fixed in the revision."}, {"review_id": "eGvPVukrSV", "reviewer": "Reviewer_sWYy", "summary": "This paper proposes to learn domain-invariant representations for ranking. In contrast to prior works that typically consider item-level alignment, they introduce the concept of list-level alignment to learn higher-level invariant representations on the lists. The domain adaptation generalization bound is explicitly established and extensive experiments on benchmark datasets reveal the effectiveness of the proposed approach. \n\nPost rebuttal: my major concerns have been adequately addressed, and thus I've decided to change my score from 4 to 5.", "questions": "Please refer to the weakness.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Domain adaptation for ranking problems is significant but receives scant attention.\n\n- The paper in general is well-motivated and presents major challenges that need to be overcome to address DA for ranking. A new paradigm termed list-level alignment is proposed. \n\n- They evaluate and ablate the method in multiple benchmark datasets.", "weaknesses": "- The proposed alignment relies on the traditional domain adversarial training, offering no significant advancements. What are the differences between the proposed alignment and prior adversarial alignment, and why the proposed list-level alignment is preferable in the context of ranking problems? The primary distinctions between list-level alignment and other paradigms are not well-articulated.\n\n- According to the reported results, the proposed ListDA shows marginal improvements over baseline methods in terms of MAP.\n\n- The connection between the theory and the proposed method is not sufficiently clear and precise, leaving readers uncertain about their relationship.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time, comments, and support! We hope that the following addresses the concerns:\n\n- **“The proposed alignment relies on the traditional domain adversarial training, offering no significant advancements. What are the differences between the proposed alignment and prior adversarial alignment”**\n\n\tThe key distinction between adversarial training for our list-level invariance and prior work’s item-level invariance is the distribution we try to align. For item-level invariance (e.g., ItemDA), we try to align the distribution of item-level representations, while for list-level invariance, we treat the entire list as a whole, derive a list-level representation, and aim to align the distribution of the list-level representation (please also kindly find the figure in our global response).\n\n\tAnother distinction lies in the choice of the discriminator (lines 130–140). ListDA requires list-level invariance, which means that the input of the discriminator is list-like. For modeling such data, we used a transformer model architecture for our ListDA discriminator; specifically, the transformer does not have positional encoding, so it is compatible with the permutation-invariance property of ranking data. In contrast, there is more freedom in the choice of discriminator for item-level domain alignment, and most prior work uses MLP.\n\n\tIn short, our contribution is to provide an appropriate alignment objective for ranking problems—with rigorous justification (Theorem 4.6), and our focus is not on the optimization side (we note that besides adversarial training, invariant representations can be learned using other methods, incl. optimal transports [1, 2]). We adapted adversarial training for our use case because of its popularity in the literature.\n\n- **“why the proposed list-level alignment is preferable in the context of ranking problems”**\n\n\tOur generalization bounds in Corollaries 4.7–4.8 (derived from Theorem 4.6) state that, if the representations are aligned at the list-level, then target domain performance—in terms of ranking metrics (e.g., NDCG, MRR)—can be lower bounded by source domain performance. But we are not aware of any bound for ranking via item-level invariance; moreover, no such bound has been established for ranking in prior work, because the majority of them studied classification problems (which use accuracy as the metric).\n\n\tThis means that item-level invariance does not have performance guarantees, but list-level invariance does so, and is hence more appropriate on ranking problems. This is verified in our experiments, where ListDA always outperforms ItemDA; in particular, because of the lack of guarantee, the latter can sometimes result in negative transfer. Another intuitive reason is that ItemDA does not take into account the list structure inherent in the data of ranking problems, and ListDA does; the distinction between list-level and item-level invariance is discussed on lines 141–149 (please also kindly find the new figure attached to the global response): list-level invariance is a stricter requirement than item-level invariance, suggesting that ItemDA is not sufficient for ranking problems.\n\n- **“The connection between the theory and the proposed method is not sufficiently clear and precise”**\n\n\tCorollaries 4.7–4.8 (derived from Theorem 4.6) hold when the representations are aligned at the list-level, and on the other hand, our ListDA method is proposed to learn list-level domain invariant representations.\n\n\tSo the bounds provide guarantees for the method, and the method is aimed at maximizing the bounds. The two are connected by the concept of list-level invariance. We will elaborate upon the existing discussion on lines 203–209 and emphasize this connection.\n\n- **“ListDA shows marginal improvements over baseline methods in terms of MAP”**\n\n\tThe ListDA method consistently achieves significantly better mean average precision (MAP) than ItemDA and zero-shot baselines on all three datasets. While the improvement over QGen PL may seem marginal, it is important to note that MAP is just one of several evaluation metrics used in our experiments, and ListDA still outperforms baseline methods on multiple evaluation metrics (including NDCG@k and MRR@10) across different datasets. We believe that the combination of these results supports the effectiveness of ListDA.\n\n\tFurthermore, the contribution of this paper extends beyond the proposed ListDA method. The list-level alignment framework provides a foundation for future research on domain adaptation in ranking tasks (e.g., more sophisticated discriminator structures).\n\n[1] Chen et al. A Gradual, Semi-Discrete Approach to Generative Network Training via Explicit Wasserstein Minimization. ICML 2019.  \n[2] Zhou et al. Iterative Alignment Flows. AISTATS 2022.\n"}, {"review_id": "g9g5fpzAgo", "reviewer": "Reviewer_D69g", "summary": "This paper focuses on domain adaptation in the context of ranking problems. The authors propose a novel approach called list-level alignment to learn domain-invariant representations for ranking tasks. They demonstrate the benefits of this approach through theoretical analysis and empirical experiments. \n\nExisting domain adaptation methods for ranking, particularly invariant representation learning, have been sporadically used and lack theoretical guarantees.\n\nGiven the limitations of previous approaches, the authors aim to address the conceptual discrepancy between item-level alignment and the intrinsic list structure of ranking problems. They propose list-level alignment as a solution to learn higher-level invariant representations on rank lists. This approach not only provides a theoretical foundation for domain adaptation in ranking but also achieves better empirical performance.", "questions": "In my first point at my weakness part, if I misunderstand the theoretical contributions, the author could correct me.", "limitations": "Yes, it's properly stated in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think this problem setting is interesting and practical. I have the following views on this paper's strengths part.\n\nFrom the perspective of problem and method, this paper addresses the challenge of training ranking models in domains with limited annotated data. It introduces domain adaptation as a transfer learning technique and highlights the need for invariant representation learning in the ranking literature. The authors propose a new approach called \\list-level alignment\\ that preserves the list structure of ranking problems and achieves higher-level invariant representations on the lists. The benefits of this approach include the establishment of the first domain adaptation generalization bound for ranking and better empirical performance on unsupervised domain adaptation tasks.\n\nExperimentally, the authors conducted experiments on unsupervised domain adaptation tasks to evaluate the performance of their proposed approach, list-level alignment. The proposed approach achieved better empirical performance on unsupervised domain adaptation tasks compared to existing methods. The approach also established the first domain adaptation generalization bound for ranking, providing theoretical guarantees for its implementation.", "weaknesses": "I have the following views on this paper's weakness part.\n\n- The paper acknowledges that existing domain adaptation methods for ranking lack theoretical guarantees, and it claims to have theoretical foundations for its proposed approach of list-level alignment. It's already a good contribution. However, the contribution of this paper in the theoretical aspect may be considered weak, since the contribution of this paper can be seen as an extension of existing domain adaptation theorems[1] applied to the ranking problem. Correct me if I misunderstand it.\n\n- One curious aspect here is that while the paper discusses domain adaptation (DA) in the ranking problem, the experiments are conducted on reranking. In the context of reranking, which involves reordering a list of candidate documents generated by an initial retrieval model in response to a search query. However, it does not directly address the fundamental ranking problem. This limitation of the paper should be further discussed to provide clarity.\n\n[1] Ben-David, Shai, et al. \"Analysis of representations for domain adaptation.\" Advances in neural information processing systems 19 (2006).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time, comments, and support! We hope that the following addresses the concerns:\n\n- **“the theoretical aspect may be considered weak… can be seen as an extension of existing domain adaptation theorems”**\n\n\tThe result established in [1] is in terms of accuracy for binary classification problems, not ranking, and we point out that their proof technique cannot be extended to get a bound in terms of ranking metrics (e.g., NDCG, MRR) for ranking problems, due to technical difficulties that arise from the problem setup for ranking. For instance, the tightness of the bound in [1] depends on the uniqueness of the optimal Bayes classifier, but the optimal ranker/scorer on ranking problems are generally not unique, as discussed in Section 4.\n\n\tIn addition, our bound can only be obtained after recognizing the need for list-level representation alignment, rather than item-level alignment as done in prior work on (ranking) domain adaptation. This necessary conceptual leap—which enabled both the theoretical result and the empirical improvements—is also our main theoretical contribution.\n\n- **“the experiments are conducted on reranking… does not directly address the fundamental ranking problem”**\n\n\tIn Appendix D, we performed experiments on the Yahoo! Learning to Rank (LETOR) Challenge v2.0 dataset, a web search ranking task on numerical data, where the lists to be ranked are directly defined by the dataset (as opposed to obtained from a retrieval stage), hence the setting is a “fundamental ranking problem”. The results are consistent with those from the passage reranking experiments and also support our findings in the main sections.\n\n\tNevertheless, our theory applies to any data/problem setting that involves lists of items ($x$) and scores ($y$) with the metrics being ranking metrics (e.g., NDCG, MRR), and this includes both passage reranking and Yahoo! LETOR. The only difference between them, in our opinion, is how the raw data is stored (the former as a corpus of queries and documents; the latter directly stores the lists) and how the lists are formed (the former uses a retriever to form the lists, i.e., the retriever, which we do not adapt, is treated as part of the problem setup; the latter as-is)."}, {"review_id": "ljI9dFDmwZ", "reviewer": "Reviewer_nxwQ", "summary": "The authors propose a new domain-invariant ranking method which enforces list-wise feature invariance as opposed to item-wise feature invariance. The authors prove a novel domain adaptation bound specifically for the ranking problem. Authors show strong empirical improvement over item-wise invariance methods.", "questions": "I find the statement in line 141 confusing (even after looking at Appendix A.2). I would think that item-level invariance implies list-level invariance. For example, consider the case where $g$ was trained to output a domain-invariant feature representation of query-document pairs (at the item level). The ranker $h$ operates on these domain invariant features to form a list. How could the resulting list be different between the domains, if the ranker is constant and the output of $g$ is domain-invariant? Perhaps it would be helpful to elaborate on this point further in the paper.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The theorem is new and not a trivial extension of the existing domain adaptation generalization bound from Ben-David (which is for classification). In particular, some new techniques are required to analyze the ranking problem.\n\n(2) The method is novel and results are good. ", "weaknesses": "(1) My main concern is with the empirical improvement between ItemDA and ListDA. I remain unconvinced that this improvement is coming from the list-wise invariance and not from the improvement in discriminator architecture. The authors mention that ItemDA uses a three-layer MLP, while ListDA uses a stack of three transformer blocks. is there anyway to compare the two methods using the same discriminator architecture? \n\n(2) While the novel generalization bound and algorithm are both novel. The theory doesn't explain why ListDA is better than ItemDA. Personally, I do not see this as an issue. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time, comments, and support! We hope that the following addresses the concerns:\n\n- **“the statement in line 141 [is] confusing”**\n\n\tThank you for the question—this was also a conceptual hurdle for us when we worked on this paper (please kindly find the new figure attached to the global response that illustrates the difference between list-level and item-level alignment).\n\n\tIn ranking problems (for information retrieval), the list is defined to be the q-d pairs with the same query q. Although in our experiment setup, $g$ computes feature vector (in $\\mathbb R^k$) for each q-d pair in isolation, it does not mean that the list structure is lost: the list feature representation is the set of features $\\\\{g(q,d_1),\\cdots, z_(q,d_\\ell)\\\\}$ computed on q-d pairs with the same q.\n\n\tThis means that the list-level feature distribution $\\mu_Z$ is defined over the set of features vectors computed on q-d pairs with the same q (the support of this distribution is the space of sets that contain $\\ell$ length-$k$ vectors). This is in contrast to the item-level feature distribution $\\mu_V$, which is defined over the feature vectors, ignoring the list structure (the support is $\\mathbb R^k$, the space of length-$k$ vectors).\n\n\tTherefore, even if item-level distributions are aligned, after imposing the list structure on the items based on whether they have the same q (each list contains not one but $\\ell$ items), the resulting list-level distributions are no longer necessarily aligned. Or, from a different perspective, there are strictly more ways of learning representations that are aligned at the item-level than there are at the list-level, and this is what we intended to demonstrate with the example in Appendix A.2.\n\n- **“The theory doesn't explain why ListDA is better than ItemDA”**\n\n\tWhile we did not prove a separation, we discussed (in the first point above) and showed that item-level alignment does not imply list-level alignment, which is required by the generalization bound for domain transfer measured in terms of ranking metrics. This suggests that ItemDA, which discards the list structure, is not sufficient for domain alignment on ranking problems, and ListDA is the more appropriate choice.\n"}]

Author rebuttal:
We thank all reviewers for their time and the remarks!

- **Figure on item-level alignment vs. list-level alignment.**

	We created a new figure (in the attached PDF) to better illustrate their difference as well as the point that the latter is a stronger requirement than the former. This figure will be included in our revision.

- **(Reviewers 3wjd and nxwQ) “the use of transformer discriminator instead of MLP [for ListDA]” and “unconvinced that this improvement is coming from the list-wise invariance and not from the improvement in discriminator architecture”**

	First, we discuss the justifications behind this choice, then discuss the results on a new set of experiments where we used a transformer discriminator for ItemDA (ItemDA-transformer), as well as a MLP discriminator for ListDA (ListDA-MLP).

	We used transformers for ListDA not mainly for its representation power, but for its efficiency in modeling sequential and list/set-like data. It is indeed possible to use discriminators with a MLP architecture for ListDA, e.g., one design (ListDA-MLP) would be to concatenate all feature vectors (each $\in\mathbb R^k$) of a list (of length $\ell$) and form a long vector $\in\mathbb R^{\ell\times k}$, and feed it into the MLP. However, its optimization process will be inefficient because the MLP in this design does not recognize the permutation invariance property of rank lists (i.e., swapping two items in a list with the scores does not alter the list), so it will be much slower to train (exponentially in $\ell$, for iterating through all orderings of the concatenation). Therefore, we used a transformer (without positional encoding) because it is invariant to permutations in the input sequence, and is efficient to optimize, since MLP is not appropriate for ListDA.

	Now, the transformer is indeed a more capable model than MLP, but its additional modeling power is only useful on sequential data. Note that each input (unbatched) to the list-level discriminator (for ItemDA) is a list of $\ell$ feature vectors in $\mathbb R^k$ as a sequence, whereas each input to the item-level discriminator (for ListDA) is a single $\mathbb R^k$ vector. Therefore, if we were to use transformer for item-level alignment (ItemDA), since each input is a sequence of length-1, the attention mechanism becomes pointless, and the transformer collapses to a MLP (with skip connections and layernorm)—in this regard, the transformer is as expressive as the MLP for ItemDA (we also tried wider and deeper MLPs in our experiments, but they did not provide additional benefits). The transformer discriminator is only meaningful for ListDA, where the inputs are sequential. This is not to rule out the possibility of better discriminator architectures, but is outside the scope of our work.

- **(Continuing from above) Results on ItemDA-transformer and ListDA-MLP.**

	We ran additional experiments (ablations) to show that **using a transformer discriminator for ListDA provides no additional benefits**, where we see that ItemDA-transformer has the same performance as ItemDA-MLP on all datasets:

	| Robust04 | MAP | MRR@10 | NDCG@5 | NDCG@10 | NDCG@20 | 
	| --- | --- | --- | --- | --- | --- |
	| ItemDA-MLP | 0.2822 | 0.8037 | 0.5822 | 0.5396 | 0.4922 | 
	| ItemDA-transformer | 0.2848 | 0.8018 | 0.5851 | 0.5406 | 0.4982 | 
	| ListDA | 0.2901 | 0.8234 | 0.5979 | 0.5573 | 0.5126 | 

	| TREC-COVID | MAP | MRR@10 | NDCG@5 | NDCG@10 | NDCG@20 | 
	| --- | --- | --- | --- | --- | --- |
	| ItemDA-MLP | 0.3086 | 0.9080 | 0.8276 | 0.8142 | 0.7697 | 
	| ItemDA-transformer | 0.3088 | 0.8907 | 0.8287 | 0.8069 | 0.7691 | 
	| ListDA | 0.3184 | 0.9335 | 0.8693 | 0.8412 | 0.7985 | 

	| BioASQ | MAP | MRR@10 | NDCG@5 | NDCG@10 | NDCG@20 | 
	| --- | --- | --- | --- | --- | --- |
	| ItemDA-MLP | 0.4781 | 0.6383 | 0.5315 | 0.5343 | 0.5604 | 
	| ItemDA-transformer | 0.4739 | 0.6420 | 0.5256 | 0.5312 | 0.5567 | 
	| ListDA | 0.5191 | 0.6666 | 0.5639 | 0.5714 | 0.5985 | 

	To show that **MLP discriminator is inappropriate for ListDA**, we used the concatenation strategy described above and applied it on the Yahoo! LETOR dataset. We used two training strategies: in ListDA-MLP-1, both the ranking model and the discriminator are updated in every step (with gradient reversal), and in ListDA-MLP-10, the model is updated every 10 discriminator update steps.

	The results are as follows. Because of the difficulty in optimizing ListDA-MLP mentioned above (the complexity is exponential in list-size, and the list size in Yahoo! LETOR dataset can be as large as 139), it is hard to maintain the discriminator to near-optimality (which is required in adversarial training), and this suboptimality leads to negative transfer. Note that ListDA-MLP-10 > ListDA-MLP-1 because the former discriminator updates more frequently (also takes ten times as long to train), suggesting that the negative transfer is due to inefficiencies in optimization.

	| Yahoo! LETOR | MAP | MRR@10 | NDCG@5 | NDCG@10 | NDCG@20 | 
	| --- | --- | --- | --- | --- | --- |
	| ItemDA | 0.5315 | 0.6717 | 0.7402 | 0.7708 | 0.8255 | 
	| ListDA-transformer | 0.5370 | 0.6771 | 0.7442 | 0.7735 | 0.8269 | 
	| ListDA-MLP-1 | 0.4265 | 0.5190 | 0.6385 | 0.6958 | 0.7740 | 
	| ListDA-MLP-10 | 0.5041 | 0.6480 | 0.7157 | 0.7508 | 0.8117 | 

Meta-review: The paper presents a novel method to learn domain-invariant representations for domain adaptation on ranking problems, which is well received by all reviewers (2x accept, 1x weak accept, 1x borderline accept). In particular, it is well-written and motivated, and addresses an interesting and practically important problem. Moreover, the work establishes a new generalization bound for domain adaptation for ranking, and achieves good empirical results. 

The rebuttal has addressed most of the initial concerns raised by the reviewers by providing additional justification for the transformer-based discriminator and clarification. Therefore, the AC recommends acceptance of the paper and encourages the authors to revise the paper according to the rebuttal discussion.
Decision: Accept (spotlight)