Title: Learning List-Level Domain-Invariant Representations for Ranking
Venue: NeurIPS 2023
Accepted: yes
Keywords: learning to rank

domain adaptation

text ranking
Abstract: Domain adaptation aims to transfer the knowledge learned on (data-rich) source domains to (low-resource) target domains, and a popular method is invariant representation learning, which matches and aligns the data distributions on the feature space. Although this method is studied extensively and applied on classification and regression problems, its adoption on ranking problems is sporadic, and the few existing implementations lack theoretical justifications. This paper revisits invariant representation learning for ranking. Upon reviewing prior work, we found that they implement what we call item-level alignment, which aligns the distributions of the items being ranked from all lists in aggregate but ignores their list structure. However, the list structure should be leveraged, because it is intrinsic to ranking problems where the data and the metrics are defined and computed on lists, not the items by themselves. To close this discrepancy, we propose list-level alignment—learning domain-invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method, and it achieves better empirical transfer performance for unsupervised domain adaptation on ranking tasks, including passage reranking.

Learning List-Level Domain-Invariant Representations for Ranking

Abstract

Domain adaptation aims to transfer the knowledge learned on (data-rich) source domains to (low-resource) target domains, and a popular method is invariant representation learning, which matches and aligns the data distributions on the feature space. Although this method is studied extensively and applied on classification and regression problems, its adoption on ranking problems is sporadic, and the few existing implementations lack theoretical justifications. This paper revisits invariant representation learning for ranking. Upon reviewing prior work, we found that they implement what we call item-level alignment , which aligns the distributions of the items being ranked from all lists in aggregate but ignores their list structure. However, the list structure should be leveraged, because it is intrinsic to ranking problems where the data and the metrics are defined and computed on lists, not the items by themselves. To close this discrepancy, we propose list-level alignment -learning domain-invariant representations at the higher level of lists. The benefits are twofold: it leads to the first domain adaptation generalization bound for ranking, in turn providing theoretical support for the proposed method, and it achieves better empirical transfer performance for unsupervised domain adaptation on ranking tasks, including passage reranking.

1 Introduction

Learning to rank applies machine learning techniques to solve ranking problems that are at the core of many everyday products and applications, including search engines and recommendation systems [34]. The availability of ever-increasing amounts of training data and advances in modeling are constantly refreshing the state-of-the-art of many ranking tasks [40]. Yet, the training of practical and effective ranking models on tasks with little to no annotated data remains a challenge [57]. A popular transfer learning framework for this problem is domain adaptation [41]: given source domains with labeled data that are relevant to the target domain of interest, domain adaptation methods optimize the model on the source domain and make the knowledge transferable by utilizing (unlabeled) target data.

For domain adaptation, there are task-specific and general-purpose (unsupervised) methods. E.g., for text ranking in information retrieval where we only have access to target documents without any training query or relevance annotation, a method is to use generative language models to synthesize

Lists of items (

q

-

d

pairs) to be ranked

Target

q

1

d

1

q

1

d

2

…

q

1

d ℓ

q

2

d

1

q

2

d

2

…

q

2

d ℓ

…

q

3

d

4

q

3

d

5

…

q

3

d ℓ

q

4

d

4

q

4

d

5

…

q

4

d ℓ

…

Source

Item-level alignment

Distributions of items

(list structure is ignored)

List-level alignment

Distributions of lists

μ

S

item

μ

T

item

…

…

…

…

μ

S

list

μ

T

list

…

…

relevant queries for the unannotated documents, then train the ranking model on the synthesized querydocument pairs [37, 54, 65]. For general-purpose methods, which are applicable to all kinds of tasks, perhaps the most well-known is domain-invariant representation learning [38, 17]. While it is studied extensively on classification and regression problems [71, 73], applications of invariant representation learning on ranking problems are sporadic [12, 58, 68], and the existing implementations lack theoretical justifications, which casts doubt on their applicability to learning to rank.

In this paper, we revisit the method of learning invariant representations for domain adaptation on ranking problems. We found that, from our review of prior work [12, 58, 68], existing implementations of the method all perform what we call item-level alignment (Fig. 1 and Section 3.2.1), which aligns the distributions of the items being ranked (e.g., query-document pairs) aggregated from all lists and discards their list structure (as in, e.g., q-d pairs with the same query all belong to the same list). However, both the data and the metrics for ranking are defined and computed on lists, not the items by themselves, so intuitively the list structure should be leveraged in the feature learning process as it is intrinsic to the problem. To close this discrepancy, we propose and analyze list-level alignment , which aligns the distributions of the lists and preserves their list structure (Section 3.2.2).

The conceptual leap from item-level to list-level alignment is significant. First, by analyzing list-level representation invariance, we establish the first domain adaptation generalization bound for ranking (Section 4), in turn providing theoretical support for our proposed method as well as a foundation for future studies. Second, list-level alignment provides empirical benefits to domain adaptation on ranking problems, which we demonstrate on two ranking tasks, passage reranking (Section 5) and Yahoo! LETOR web search ranking (Appendix D), where it outperforms zero-shot learning, item-level alignment, as well as a method specific to text ranking based on query synthesis [37].

2 Problem Setup

We define a ranking problem by a joint distribution µ of lengthℓ lists and their nonnegative scores, denoted by X = ( X 1 , · · · , X ℓ ) ∈ X and Y = ( Y 1 , · · · , Y ℓ ) ∈ R ℓ ≥ 0 . 1 For example, in information retrieval, the items in each list are query-document pairs of the same query, and the scores indicate the relevance of the documents to the query. Note that the lists are permutation-invariant , because switching the order of the items (and the corresponding scores) does not change the content of the list; concretely, let S ℓ denotes the set of permutations on [ ℓ ] := { 1 , 2 , · · · , ℓ } , then permutation invariance means that µ (( x 1 , · · · , x ℓ ) , ( y 1 , · · · , y ℓ )) = µ (( x π 1 , · · · , x π ℓ ) , ( y π 1 , · · · , y π ℓ )) for all ( x, y ) and π ∈ S ℓ . We assume that the scores are a function of the list, Y = y ( X ) , so the problem can be equivalently given by the marginal distribution µ X of lists and a scoring function y : X → R ℓ ≥ 0 .

Learning to rank aims to learn a ranker that takes lists x as input and outputs rank assignments r ∈ S ℓ , where r i ∈ [ ℓ ] is the predicted rank of item i , such that r recovers the descending order of the scores y , i.e., y i > y j ⇐⇒ r i < r j for all i, j . The more common setup, which we will adopt, is

to train a scorer , f : X → R ℓ , with the rank assignments obtained by first computing the ranking scores s = f ( x ) ∈ R ℓ , where s i is the predicted score of item i , then taking the descending order of s (alternatively, the rank assignments can be generated probabilistically; see Section 4).

To quantitatively measure the quality of the predicted ranks, we use (listwise) ranking metrics, u : S ℓ × R ℓ ≥ 0 → R ≥ 0 , which computes a utility score via comparing the predicted rank assignments to the ground-truth scores of the list. Common metrics in information retrieval include reciprocal rank and normalized discounted cumulative gain [63, 26]:

Definition 2.1 (Reciprocal Rank) . If the ground-truth scores are binary, i.e., y ∈ { 0 , 1 } ℓ , then the reciprocal rank (RR) of a predicted rank assignments r ∈ S ℓ is

The average RR of a ranking model f ′ : X → S ℓ on a dataset, E ( X,Y ) ∼ µ [ RR ( f ′ ( X ) , Y )] , is referred to as the mean reciprocal rank (MRR).

Definition 2.2 (NDCG) . The discounted cumulative gain (DCG) and the normalized DCG (with identity gain function) of a predicted rank assignments r ∈ S ℓ are

where IDCG ( y ) = max r ′ ∈ S ℓ DCG ( r ′ , y ) , called the ideal DCG, is the maximum DCG of a list, which is attained by the descending order of y .

Domain Adaptation. In this learning setting, we have a source domain ( µ X S , y S ) and a target domain ( µ X T , y T ) with different data distributions, 2 and the goal is to learn a good ranking model for the target domain by leveraging all available resources: whereas access to source domain labeled training data is always assumed, we may only be given unlabeled data for the target domain (this scenario is called unsupervised domain adaptation). A popular method for this adaptation is domaininvariant representation learning, which matches and aligns the source and target domain data distributions on the feature space so that their distributions appear similar to the model.

3 Learning Domain-Invariant Representations for Ranking

We begin by reviewing invariant representation learning for domain adaptation and the optimization technique of adversarial training, then describe and compare two instantiations of this framework to ranking problems in Section 3.2-item-level alignment, which is implemented in prior work, and our proposed list-level alignment.

For representation learning, we train composite models of the form f = h ◦ g , where g : X → Z is a shared feature map and h is a task head on the shared features. As an example, if the model f is end-to-end implemented as an m -layer multilayer perceptron (MLP), then we could treat the first ( m -1) layers as g and the last as h .

And, recall the definitions of Lipschitz continuity and Wasserstein distance [15]:

Definition 3.1 (Lipschitz Continuity) . A function f : X → Y from metric space ( X , d X ) to ( Y , d Y ) is L -Lipschitz, denoted by f ∈ Lip( L ) , if d Y ( f ( x ) , f ( x ′ )) ≤ L d X ( x, x ′ ) for all x, x ′ ∈ X .

Definition 3.2 (Wasserstein Distance) . The Wasserstein1 distance between probability measures p, q on metric space X is denoted and given by W 1 ( p, q ) = sup ψ : X→ R ,ψ ∈ Lip(1) ∫ X ψ ( x )( p ( x ) -q ( x )) d x .

3.1 Invariant Representation Learning via Adversarial Training

Given a source domain µ S and a target domain µ T , invariant representation learning aims to learn a shared feature map g s.t. the distributions are aligned on the feature space, i.e., D ( µ Z S , µ Z T ) ≈ 0 , where D is a divergence measure or probability metric, e.g., Wasserstein distance, and the source feature distribution µ Z S (analogously for target µ Z T ) is defined to be the distribution of Z = g ( X ) , X ∼ µ X S . The idea is that a composite model with an aligned feature representation is transferable between

domains-this is supported by the following generalization bound on classification problems [52, 5]. We will establish a similar result for ranking in Section 4.

̸

Theorem 3.3. Let binary classification problems on a source and a target domain be given by joint distributions µ S , µ T of inputs and labels, ( X,Y ) ∈ X × { 0 , 1 } . Define the error rate of a predictor f : X → [0 , 1] on µ by R ( f ) = E ( X,Y ) ∼ µ [ f ( X ) 1 [ Y = 1] + (1 -f ( X )) 1 [ Y = 0]] , 3 where 1 [ · ] is the indicator function.

̸

Let H ⊂ [0 , 1] Z be an L -Lipschitz class of prediction heads, and for any g ∈ G , define the minimum joint error rate by λ ∗ g = min h ′ ( R S ( h ′ ◦ g ) + R T ( h ′ ◦ g )) , then for all h ∈ H ,

This result bounds the target domain risk of the composite model h ◦ g by its source task risk, plus the divergence of the feature distributions and the optimal joint risk. It therefore suggests that a good model for the target could be obtained by learning an informative domain-invariant feature map g (s.t. λ ∗ g and W 1 ( µ Z S , µ Z T ) ≈ 0 ) and training h to minimize the source task risk R S .

Invariant representation learning algorithms capture the aforementioned objectives with the joint training objective of min h,g ( L S ( h ◦ g )+ λ D ( µ Z S , µ Z T )) [35, 17, 13], where L S is the source task loss, and λ > 0 is a hyperparameter that controls the strength of invariant representation learning. This objective learns domain-invariant features by optimizing g to minimize the distributional discrepancy, and minimizing source risk at the same time s.t. the learned features are useful for the task. Since it does not require labeled data but only unlabeled ones for estimating W 1 ( µ Z S , µ Z T ) , it is applicable to unsupervised domain adaptation. If labeled target data are available, they can be incorporated by, e.g., adding a target loss L T , which will help keep λ ∗ g low.

Adversarial Training. When the divergence measure D is simple and admits closed-form expressions, e.g., maximum-mean discrepancy [19], the objective above can be minimized directly [35], but they are typically too weak at modeling complex distributions that arise in real-world problems. To achieve feature alignment under stronger measures, e.g., JS divergence or Wasserstein distance, a well-known approach is adversarial training [18, 17, 2], which reformulates the above objective into

and optimizes it using gradient descent-ascent with a gradient reversal layer added on top of g in the adversarial component.

The adversarial component, -min f ad L ad ( g, f ad ) , can be shown to upper bound the divergence between µ Z S , µ Z T . It involves an adversary f ad : Z → R (parameterized by neural networks) and an adversarial loss function ℓ ad : R ×{ 0 , 1 } → R , whose inputs are the output of f ad and the domain identity a (we set target domain to a = 1 ). The adversarial loss is computed over both domains:

If we choose the 0-1 loss, ℓ ad (ˆ a, a ) = (1 -a ) 1 [ˆ a ≥ 0] + a 1 [ˆ a < 0] , then the adversarial component upper bounds W 1 ( µ Z S , µ Z T ) (see Proposition A.1); here, f ad acts as a domain discriminator for distinguishing the domain identity, and L ad is the balanced classification error of f ad. For training, the 0-1 loss is replaced by a surrogate loss, e.g., the logistic loss [18], ℓ ad (ˆ a, a ) = log(1 + e (1 -2 a )ˆ a ) .

3.2 Invariant Representation Learning for Ranking

This section describes and compares two instantiations of the invariant representation learning framework above for ranking: item-level alignment, and our proposed list-level alignment. The key difference between them is the choice of µ Z whose divergence W 1 ( µ Z S , µ Z T ) is to be minimized.

Model Setup. We consider a composite scoring model f = h ◦ g where the feature map g is s.t. given an input list x = ( x 1 , · · · , x ℓ ) , it outputs a list of k -dimensional feature vectors, z = g ( x ) = ( v 1 , · · · , v ℓ ) ∈ R ℓ × k , where v i ∈ R k corresponds to item i . Ranking scores are then

Representation 1

q

1

d

1

q

1

d

2

q

1

d

3

q

2

d

1

q

2

d

2

q

2

d

3

q

3

d

4

q

3

d

5

q

3

d

6

q

4

d

4

q

4

d

5

q

4

d

6

Aligned at item-level

but not at list-level

Representation 2

q

1

d

1

q

1

d

2

q

1

d

3

q

2

d

1

q

2

d

2

q

2

d

3

q

3

d

6

q

4

d

5

q

3

d

5

q

4

d

6

q

3

d

4

q

4

d

4

Aligned at item-level

and at list-level

Each domain contains

2 queries and 3 documents

Source

q

1

d

1

q

1

d

2

q

1

d

3

q

2

d

1

q

2

d

2

q

2

d

3

Target

q

3

d

4

q

3

d

5

q

3

d

6

q

4

d

4

q

4

d

5

q

4

d

6

Items with same q belong to same list

obtained by, e.g., projecting each v i to R with a (shared) linear layer. This setup, which we will adopt in our experiments, is common with listwise ranking models and in many ranking systems [10]: in neural text ranking, each feature vector can be the embedding of the input text computed by a language model [39, 21, 76].

3.2.1 Item-Level Alignment

In item-level alignment (ItemDA), the distributions of the k -dimensional feature vectors from all lists in aggregate are aligned [12, 58, 68], i.e., µ Z, item S ≈ µ Z, item T , where

In other words (when both domain have the same number of lists), it finds a matching between the items of the source and target domains that pairs each target item with a source one (Fig. 2). Note that the list structure is not retained in this definition, because one cannot necessarily tell whether two items drawn from the bag of feature vectors, v, v ′ ∼ µ Z, item , belong to the same list.

To implement item-level alignment, the discriminator f ad (usually an MLP) is set up to take individual feature vectors v ∈ R k as input. In each forward-pass, the feature map g computes a batch of feature vector lists, { z i } i ∈ [ b ] = { ( v i 1 , · · · , v iℓ ) } i ∈ [ b ] , then the discriminator takes the items batched from all lists (this step discards the list structure), { v ij } i ∈ [ b ] ,j ∈ [ ℓ ] , and predicts their domain identities.

Item-level alignment is identical to invariant representation learning implementations for classification and regression, e.g., DANN [17], which also operate on 'items', since neither the data nor the metrics in those problem settings have any explicit structural assumptions. However, ranking problems are inherently defined with list structures: the inputs to the model are organized by lists, and ranking metrics are evaluated on lists rather than the items by themselves. This discrepancy casts doubt on the applicability of item-level alignment to ranking problems, and moreover, it is unclear whether domain adaptation for ranking via item-level alignment is justified from a theoretical perspective.

3.2.2 List-Level Alignment

Closing the discrepancy discussed above, we propose list-level alignment (ListDA), which aligns the distributions of the lists of feature vectors on the R ℓ × k space, i.e., µ Z, list S ≈ µ Z, list T , where

This means finding a matching that not only pairs all source and target items, but also pairs items from the same target list to items from the same source list. Therefore, list-level alignment is a stronger requirement than item-level alignment, µ Z, list S = µ Z, list T = ⇒ µ Z, item S = µ Z, item T , while the converse is generally not true (see Fig. 2 for a picture, or Example A.2)!

To implement list-level alignment, the discriminator has to make predictions on lists, z ∈ R ℓ × k , i.e., in each forward-pass it would be fed a batch of feature vector lists, { z i } i ∈ [ b ] = { ( v i 1 , · · · , v iℓ ) } i ∈ [ b ] , and output b predictions. For this, a possible design choice is to flatten each feature vector list into a long ( ℓk ) -dimensional vector and use an MLP as f ad . But recall from Section 2 that the

input lists are permutation-invariant, and so are the feature vector lists. Since the above setup does not incorporate this property and an exponential amount of samples is required to see all possible permutations, this would be inefficient to optimize; yet, the optimality of f ad is essential to the success of adversarial training. So instead, we use transformers with mean-pooling less positional encoding as the discriminator [60], which is permutation-invariant to the input.

Compared to item-level alignment, our list-level alignment is supported by a domain adaptation generalization bound (Theorem 4.7, established in the next section), and achieves better transfer performance in our Section 5 and Appendix D evaluations. These results suggest that the list structure is essential for an effective domain adaptation on ranking problems.

4 Generalization Bound for Ranking with List-Level Alignment

Based on the list-level alignment proposed above, we establish a domain adaptation generalization bound for ranking by analyzing list-level representation invariance. The discussions in this section consider learning a composite scoring model, f = h ◦ g : X → R ℓ , but need not assume that the list feature representation z = g ( x ) can be decomposed into item-level feature representations as in the model setup of Section 3.2.2, only that it resides in a metric space Z .

Our result can be viewed as an instantiation of the framework established originally for classification by Ben-David et al. [5]. But since ranking is a very different task with its unique technical challenges, our analysis differs from those in prior work [52]. We will introduce appropriate assumptions for handling these difficulties leading up to the main Theorem 4.7, which the readers may skip to.

The first hurdle is the discontinuity of the sorting operation. So rather than taking the descending order of the predicted scores to obtain the rank assignments, we generate them probabilistically using a Plackett-Luce model [43, 36] with the exponentiated scores as its parameters [10, 20]. This makes the computation of the utility (Lipschitz) continuous w.r.t. the raw output scores of the model.

Definition 4.1 (Plackett-Luce Model) . A Plackett-Luce (P-L) model with parameters w ∈ R ℓ > 0 specifies a distribution over S ℓ , whose probability mass function p w is

where I ( r ) i is the index of the item with rank i , r I ( r ) i = i .

Assumption 4.2. Given predicted scores s = f ( x ) ∈ R ℓ , we generate the rank assignments probabilistic from a P-L model by R ∼ p exp( s ) ∈ S ℓ , where exp is taken coordinate-wise.

Under this assumption, the utility of a scorer f on ( µ X , y ) w.r.t. ranking metric u : S ℓ × R ℓ ≥ 0 → R ≥ 0 is computed by (overloading u for future brevity)

We define the risk (or suboptimality) of f as the difference between its utility to the maximumattainable utility on the problem (e.g., the maximum is 1 when u = NDCG):

The next technical challenge is that unlike in classification where the perfect classifier is unique (i.e., achieving zero error), the perfect ranker is generally not: e.g., both rank assignments (1 , 2 , 3) and (2 , 1 , 3) achieve maximum utility (i.e., R = 0 ) on a list with ground-truth scores (1 , 1 , 0) . Prior analyses of domain adaptation leverage this uniqueness [5, 52]; instead, ours does not rely on uniqueness (else the bound would be loose) with the following Lipschitz assumptions:

Assumption 4.3. The ranking metric u : S ℓ × R ℓ ≥ 0 → R ≥ 0 is upper bounded by B , and is L u -Lipschitz w.r.t. the ground-truth scores y in the second argument (in Euclidean distance).

Assumption 4.4. The input lists X reside in a metric space X , and the ground-truth scoring function y : X → R ℓ ≥ 0 is L y -Lipschitz (in Euclidean distance on the output space).

We will show that Assumption 4.3 is satisfied by both RR and NDCG. Assumption 4.4 says that similar lists (i.e., close in X ) should have similar ground-truth scores, and is satisfied, e.g., when X is finite; such is the case with text data, which are one-hot encoded after tokenization.

Assumption 4.5. The list features Z reside in a metric space Z , and the class H of scoring functions, h : Z → R ℓ , is L h -Lipschitz (in Euclidean distance on the output space).

Assumption 4.6. The class G of feature maps, g : X → Z , satisfies that ∀ g ∈ G , the restrictions of g to the supports of µ X S and µ X T , g | supp( µ X S ) , g | supp( µ X T ) respectively, are both invertible with L g -Lipschitz inverses.

Assumption 4.5 is standard in generalization and complexity analyses, and could be enforced, e.g., with L 2 -regularization [1, 4]. The last assumption is technical, saying that the inputs can be recovered from their feature representations by a Lipschitz inverse g -1 . This means that the feature map g should retain as much information from the inputs (on each domain), which is a desideratum of representation learning. Note that this assumption does not conflict with the goal of invariant representation learning: there is always a sufficiently expressive G s.t. ∃ g ∈ G satisfying the assumption and µ Z, list S = µ Z, list T .

We are now ready to state our domain adaptation generalization bound for ranking:

Theorem 4.7. Under Assumptions 4.2 to 4.6, for any g ∈ G , define the minimum joint risk by λ ∗ g = min h ′ ( R S ( h ′ ◦ g ) + R T ( h ′ ◦ g )) , then for all h ∈ H ,

where µ Z, list is the marginal distribution of the list features Z = g ( X ) , µ Z, list ( z ) := µ X ( g -1 ( z )) .

The key feature of this bound is that it depends on list-level alignment, W 1 ( µ Z, list S , µ Z, list T ) , hence it provides theoretical support for the list-level alignment proposed in Section 3.2.2. It can be instantiated to specific ranking metrics by simply verifying the Lipschitz condition L u of Assumption 4.3:

Corollary 4.8 (Bound for MRR) . RR is 1 -Lipschitz in y , thereby

Corollary 4.9 (Bound for NDCG) . If C -1 ≤ IDCG ≤ C for some C ∈ (0 , ∞ ) on ( µ X S , y S ) and ( µ X T , y T ) almost surely, 4 then NDCG is ˜ O ( C √ ℓ ) -Lipschitz in y almost surely, thereby

5 Experiments on Passage Reranking

To demonstrate the empirical benefits of invariant representation learning with list-level alignment (ListDA) for unsupervised domain adaptation, we evaluate it on the passage reranking task 5 and compare to zero-shot transfer, item-level alignment (ItemDA), as well as a recent method based on query synthesis [37]. Note that our method is not specialized to text (re)ranking; in Appendix D, we evaluate ListDA on a web ranking task from the Yahoo! Learning to Rank Challenge.

In passage reranking, given a text query, we are to rank candidate passages based on their relevance to the query from a retrieved set. Reranking is employed in scenarios where the corpus is too large for using accurate but expensive models such as cross-attention neural rankers to rank every (millions of) documents. Rather, a two-stage approach is taken: a simple but efficient model such as sparse or dense retrievers (e.g., BM25 [49] and DPR [28]) is first used to retrieve a candidate set of (hundreds of) passages, then a more sophisticated (re)ranking model is used to refine and improve their ranks.

We use BM25 to retrieve 1,000 candidate passages in the first-stage, and focus on the adaptation of a RankT5 listwise reranker [76], a cross-attention model derived from the T5 base language model with 250 million parameters [48]. It takes the concatenation of the query and the document (q-d pair) as input and outputs an embedding. We treat the q-d embeddings computed on the same list as the list feature-consistent with the setting in Section 3.2. For the task training loss function, we use the softmax cross-entropy ranking loss: ℓ ( s, y ) = -∑ ℓ i =1 y i log(exp( s i ) / ∑ ℓ j =1 exp( s j )) .

…

q

d

1

q

d

2

q

d ℓ

…

…

Gradient

Reversal

Mean-pool

+ Dense

Dense

Ranking scores

List of

q

-

d

pairs

List of embeddings

Query:  What kind of bear is best?

Document 1:  There are basically two…

Document 2:  Bears eat beets. Bears…

Document

ℓ

:  All bear species are great…

Discriminator

(transformer)

T5 Encoder

(cross-aLention)

BM25

Predicted

domain identity

or target

…

Source

Datasets. In our domain adaptation experiments, we use the MS MARCO dataset for passage ranking [3] as the source domain, which contains 8 million passages crawled from the web covering a wide range of topics with 532,761 search query and relevant passage pairs. The target domains are biomedical (TREC-COVID [62], BioASQ [59]) and news articles (Robust04 [64]). The data are preprocessed consistently with the BEIR benchmark [57]; their paper includes dataset statistics.

Since the training split of the target domain datasets does not contain q-d relevance annotations, our setting is unsupervised. It also does not contain any queries, but these are required for performing domain adaptation, because the input lists are defined to consist of q-d pairs. To acquire training queries on the target domains, we synthesize them in a zero-shot manner following [37], by using a T5 XL query generator (QGen) trained on MS MARCO relevant q-d pairs. QGen synthesizes each query as a seq-to-seq task given a passage, and the generated queries are expected to be relevant to the input passages. See Table 7 for sample QGen q-d pairs.

Methods. 6 In zero-shot transfer, the reranker is trained on MS MARCO and evaluated on the target domain without adaptation. In QGen pseudolabel (QGen PL), we treat QGen q-d pairs synthesized on the target domain as relevant pairs, and train the reranker on them in addition to MS MARCO. This method is specific to text ranking, and is used in several recent works on domain adaptation of text retrievers and rerankers [37, 54, 65].

ItemDA (item-level alignment) is the prior implementation of invariant representation learning for ranking [12, 58, 68], and is set up according to Section 3.2.1 with adversarial training described in Section 3.1. The adversarial loss is aggregated from the losses of five three-layer MLP discriminators (no improvements from using wider or more layers), ∑ 5 i =1 L ad ( g, f ( i ) ad ) , for reducing the sensitivity to the randomness in the initialization and training process [16]. Our ListDA (list-level alignment) is set up according to Section 3.2.2, and the discriminator is an ensemble of five stacks of three T5 (transformer) encoding blocks. To predict the domain identity of a list feature z = ( v 1 , · · · , v ℓ ) , we feed the vectors into the transformer blocks all at once as a sequence, take the mean-pool of the ℓ output embeddings, then project it to a logit with a linear layer. A block diagram of ListDA is in Fig. 3.

Further details including hyperparameters and the construction of training lists (i.e., sampling negative pairs) are relegated to Appendix C, where we also include additional results such as using the pairwise logistic ranking loss, a hybrid adaptation method combining ListDA and QGen PL, and using transformers as the discriminator for ItemDA.

5.1 Results

The main results are presented in Table 1, where we report metrics that are standard in the literature (e.g., TREC-COVID uses NDCG@20), and evaluate rank assignments by the descending order of the predicted scores. Since TREC-COVID and Robust04 are annotated with 3-level relevancy, the scores are binarized for mean average precision (MAP) and MRR as follows: on TREC-COVID, we map 0 (not relevant) and 1 (partially relevant) to negative, and 2 (fully relevant) to positive; on Robust04, 0 (not relevant) to negative, and 1 (relevant) and 2 (highly relevant) to positive.

Source domain is MS MARCO. Gain function in NDCG is the identity map. * Improves upon zero-shot with statistical significance ( p ≤ 0 . 05 ) under the two-tailed Student's t -test. † Improves upon QGen PL. ‡ Improves upon ItemDA.

Across all three datasets, ListDA achieves the best performance, and the fact that it uses the same training resource as QGen PL demonstrates the added benefits of list-level invariant representations for domain adaptation. The favorable comparison of ListDA to ItemDA corroborates the discussion in Sections 3.2 and 4 that for domain adaptation on ranking problems, item-level alignment is insufficient for transferability, sometimes even resulting in negative transfer (vs. zero-shot). Rather, list-level alignment is the more appropriate approach.

5.2 Analyses of ListDA

Quality of QGen. An explanation for why ListDA outperforms QGen PL despite the same resources is that the sampling procedure (Appendix C.3) of negative q-d pairs could introduce false negatives into the training data. This is supported by the observation in [54] that QGen synthesized queries lack specificity and could be relevant to many documents. While these false negative labels are used for training in QGen PL, they are discarded in ListDA, which is thereby less likely to be affected by false negatives, or even false positives-when synthesized queries are not relevant to the input passages (see Table 9 for samples). Although out of the scope, better query generation is expected to improve the performance of both QGen PL and ListDA.

Size of Target Data. Unsupervised domain adaptation requires sufficient unlabeled data, but not all domains have the same amount: BioASQ has 14 million documents (so is the total number of QGen queries, as we synthesize one per document), but Robust04 only has 528,155, and TREC-COVID 171,332.

In Fig. 4, we plot the performance of ListDA under varying numbers of target QGen queries (which is the number of target training lists). Surprisingly, on Robust04 and TREC-COVID, using just /tildelow 100 target QGen queries (0.03% and 0.06% of all, respectively) is sufficient for ListDA to achieve full performance! Although the number of queries is small, since we retrieve 1,000 documents per query, the amount of distinct target documents in those 100 lists could be substantial-up to 100,000, or 29.5% and 60.7% of the respective corpora.

The performance begins to drop when reduced to /tildelow 10 queries, which caps the amount of distinct documents at 10,000 (2.7% and 5.8%, respectively). The same data efficiency, however, is

0.575

0.600

Robust04

0.825

0.850

0.875

TREC-COVID

10

100

1000

10000

Number of target queries

0.54

0.56

BioASQ

NDCG@5

not observed on BioASQ, likely due to the size and hardness of the dataset, such as the use of specialized biomedical terms (see examples in Tables 7 to 9).

6 Related Work

Learning to Rank and Text Ranking. Learning to rank historically focuses on tabular data, for which, a wide range of models are developed [34], from SVMs [27], gradient boosted decision trees [9], to neural rankers [8, 42, 45]. Another traditional research direction is the design of ranking losses (surrogate to ranking metrics), which can be categorized into pointwise, pairwise, and listwise approaches [10, 7, 74, 24].

Recent advances in large neural language models have spurred interest in applying them on text ranking tasks [33, 46], leading to cross-attention models [22, 39, 40, 44] and generative models based on query likelihood [14, 77, 78, 50]. A different line of work is neural text retrieval models, which emphasizes efficiency, and has seen the development of dual-encoder [28, 70], late-interaction [29, 23], and models leveraging transformer memory [56].

Domain Adaptation in Information Retrieval. Work on this subject can be categorized into supervised and unsupervised domain adaptation. The former assumes access to labeled source data and (a small amount of) labeled target data (a.k.a. few-shot learning) [54]. This work focuses on the unsupervised setting, where target domain data do not have annotations. Cohen et al. [12] apply invariant representation learning to unsupervised domain adaptation for text ranking, followed by Tran et al. [58] for enterprise email search, and Xin et al. [68] for dense retrieval. However, unlike our proposed list-level alignment method (ListDA), they learn invariant representations at the item-level. A recent family of adaptation methods is based on query generation [37, 65], originally proposed for dense retrieval.

Invariant Representation Learning. Domain-invariant representation learning is a popular family of adaptation methods [35, 17, 13], to which our proposed ListDA belongs. Besides ranking, these methods are also applied in fields including vision and language, and on tasks ranging from crossdomain sentiment analysis, question-answering [31, 61], and to cross-lingual learning and machine translation [67, 30].

Zhao et al. [72] and Tachet des Combes et al. [55] point out that on classification problems, attaining perfect feature alignment and high source accuracy are insufficient to guarantee good target performance. This occurs when the marginal distributions of labels differ, or the learned features contain domain-specific and nontransferable components. Although their findings do not directly apply to ranking, still, they suggest two potential directions for future work: one is whether Theorem 4.7 would admit a fundamental lower bound under distributional shifts in the ground-truth scores, and another is to modify ListDA to include a component for isolating out nontransferable features, as in domain separation networks [6].

7 Conclusion

We proposed an implementation of invariant representation learning for ranking via list-level feature alignment, and based on which, established a domain adaptation generalization bound for ranking. Our theoretical and empirical results illustrate the significance of preserving the list structure for achieving effective domain adaptation, which is overlooked in prior work.

A broader message is that when working with (feature) representations, either for domain adaptation or other purposes, they should be analyzed at the same level (or structure) at which the data are defined for the task of interest and the metrics are computed. The results of this paper is such an example-by moving from item-level alignment to the more appropriate list-level alignment, we extracted more potentials from invariant representation learning for domain adaptation on ranking problems.

Acknowledgments

We thank the anonymous reviewers for their comments and suggestions on improving the presentation. This research was supported in part by the Google Visiting Researcher program and in part by the Center for Intelligent Information Retrieval. Han Zhao's work was partly supported by the Defense Advanced Research Projects Agency (DARPA) under Cooperative Agreement Number: HR00112320012. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsors.

References

Martin Anthony and Peter L. Bartlett. Neural Network Learning: Theoretical Foundations . Cambridge University Press, 1999.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning , pages 214-223, 2017.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset, 2018. arXiv:1611.09268 [cs.CL] .

Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems , 2017.

Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of Representations for Domain Adaptation. In Advances in Neural Information Processing Systems , 2007.

Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain Separation Networks. In Advances in Neural Information Processing Systems , 2016.

Sebastian Bruch, Shuguang Han, Michael Bendersky, and Marc Najork. A Stochastic Treatment of Learning to Rank Scoring Functions. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 61-69, 2020.

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. Learning to Rank using Gradient Descent. In Proceedings of the 22nd International Conference on Machine Learning , pages 89-96, 2005.

Christopher J.C. Burges. From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report MSR-TR-2010-82, Microsoft Research, 2010.

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to Rank: From Pairwise Approach to Listwise Approach. In Proceedings of the 24th International Conference on Machine Learning , pages 129-136, 2007.

Olivier Chapelle and Yi Chang. Yahoo! Learning to Rank Challenge Overview. Journal of Machine Learning Research: Workshop and Conference Proceedings , 14:1-24, 2011.

Daniel Cohen, Bhaskar Mitra, Katja Hofmann, and W. Bruce Croft. Cross Domain Regularization for Neural Ranking Models using Adversarial Learning. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval , pages 1025-1028, 2018.

Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems , 2017.

Cicero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. Beyond [CLS] through Ranking by Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages 1722-1727, 2020.

D.A. Edwards. On the Kantorovich-Rubinstein theorem. Expositiones Mathematicae , 29(4): 387-398, 2011.

Yanai Elazar and Yoav Goldberg. Adversarial Removal of Demographic Attributes from Text Data. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 11-21, 2018.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-Adversarial Training of Neural Networks. Journal of Machine Learning Research , 17(59):1-35, 2016.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Advances in Neural Information Processing Systems , 2014.

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A Kernel Two-Sample Test. Journal of Machine Learning Research , 13(25):723-773, 2012.

John Guiver and Edward Snelson. Bayesian inference for Plackett-Luce ranking models. In Proceedings of the 26th International Conference on Machine Learning , pages 377-384, 2009.

Jiafeng Guo, Yixing Fan, Liang Pang, Liu Yang, Qingyao Ai, Hamed Zamani, Chen Wu, W. Bruce Croft, and Xueqi Cheng. A Deep Look into Neural Ranking Models for Information Retrieval. Information Processing & Management , 57(6), 2020.

Shuguang Han, Xuanhui Wang, Mike Bendersky, and Marc Najork. Learning-to-Rank with BERT in TF-Ranking, 2020. arXiv:2004.08476 [cs.IR] .

Kai Hui, Honglei Zhuang, Tao Chen, Zhen Qin, Jing Lu, Dara Bahri, Ji Ma, Jai Gupta, Cicero Nogueira dos Santos, Yi Tay, and Donald Metzler. ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inference. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 3747-3758, 2022.

Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky, and Marc Najork. On Optimizing Top-K Metrics for Neural Ranking Models. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2303-2307, 2022.

Rolf Jagerman, Xuanhui Wang, Honglei Zhuang, Zhen Qin, Michael Bendersky, and Marc Najork. Rax: Composable Learning-to-Rank Using JAX. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3051-3060, 2022.

Kalervo Järvelin and Jaana Kekäläinen. Cumulated Gain-Based Evaluation of IR Techniques. ACM Transactions on Information Systems , 20(4):422-446, 2002.

Thorsten Joachims. Training Linear SVMs in Linear Time. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 217-226, 2006.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing , pages 6769-6781, 2020.

Omar Khattab and Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 39-48, 2020.

Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. Unsupervised Machine Translation Using Monolingual Corpora Only. In International Conference on Learning Representations , 2018.

Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-End Adversarial Memory Network for Cross-domain Sentiment Classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence , pages 2237-2243, 2017.

Chen Liang, Haoming Jiang, Simiao Zuo, Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models. In International Conference on Learning Representations , 2022.

Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained Transformers for Text Ranking: BERT and Beyond . Springer International Publishing, 2022.

Tie-Yan Liu. Learning to Rank for Information Retrieval. Foundations and Trends® in Information Retrieval , 3(3):225-331, 2009.

Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable Features with Deep Adaptation Networks. In Proceedings of the 32nd International Conference on Machine Learning , pages 97-105, 2015.

R. Duncan Luce. Individual Choice Behavior: A Theoretical Analysis . Wiley, 1959.

Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. Zero-shot Neural Passage Retrieval via Domain-targeted Synthetic Question Generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics , pages 1075-1088, 2021.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain Generalization via Invariant Feature Representation. In Proceedings of the 30th International Conference on Machine Learning , pages 10-18, 2013.

Rodrigo Nogueira and Kyunghyun Cho. Passage Re-ranking with BERT, 2020. arXiv:1901.04085 [cs.IR] .

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document Ranking with a Pretrained Sequence-to-Sequence Model. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 708-718, 2020.

Sinno Jialin Pan and Qiang Yang. A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering , 22(10):1345-1359, 2010.

Liang Pang, Jun Xu, Qingyao Ai, Yanyan Lan, Xueqi Cheng, and Jirong Wen. SetRank: Learning a Permutation-Invariant Ranking Model for Information Retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 499-508, 2020.

Robin L. Plackett. The Analysis of Permutations. Journal of the Royal Statistical Society Series C (Applied Statistics) , 24(2):193-202, 1975.

Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models, 2021. arXiv:2101.05667 [cs.IR] .

Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. Are Neural Rankers still Outperformed by Gradient Boosted Decision Trees? In International Conference on Learning Representations , 2021.

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, 2023. arxiv:2306.17563 [cs.IR] .

Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5835-5847, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research , 21(140):1-67, 2020.

Stephen Robertson and Hugo Zaragoza. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval , 3(4):333-389, 2009.

Devendra Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. Improving Passage Retrieval with Zero-Shot Question Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 3781-3797, 2022.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein Distance Guided Representation Learning for Domain Adaptation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence , pages 40584065, 2018.

Axel Suarez, Dyaa Albakour, David Corney, Miguel Martinez, and José Esquivel. A Data Collection for Evaluating the Retrieval of Related Tweets to News Articles. In Advances in Information Retrieval , pages 780-786, 2018.

Si Sun, Yingzhuo Qian, Zhenghao Liu, Chenyan Xiong, Kaitao Zhang, Jie Bao, Zhiyuan Liu, and Paul Bennett. Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing , volume 1, pages 5030-5043, 2021.

Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoff Gordon. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. In Advances in Neural Information Processing Systems , 2020.

Yi Tay, Vinh Q. Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, Tal Schuster, William W. Cohen, and Donald Metzler. Transformer Memory as a Differentiable Search Index. In Advances in Neural Information Processing Systems , 2022.

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , 2021.

Brandon Tran, Maryam Karimzadehgan, Rama Kumar Pasumarthi, Michael Bendersky, and Donald Metzler. Domain Adaptation for Enterprise Email Search. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 25-34, 2019.

George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R. Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artiéres, Axel-Cyrille Ngonga Ngomo, Norman Heino, Eric Gaussier, Liliana BarrioAlvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. An overview of the BioASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics , 16(1):138, 2015.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems , 2017.

Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, and Ion Androutsopoulos. Domain Adversarial Fine-Tuning as an Effective Regularizer. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 3103-3112, 2020.

Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection. ACM SIGIR Forum , 54:1-12, 2021.

Ellen M. Voorhees. The TREC-8 Question Answering Track Report. In Proceedings of the Eighth Text Retrieval Conference , pages 77-82, 1999.

Ellen M. Voorhees. The TREC Robust Retrieval Track. ACM SIGIR Forum , 39:11-20, 2005.

Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 2345-2360, 2022.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, 2020.

Ruicheng Xian, Heng Ji, and Han Zhao. Cross-Lingual Transfer with Class-Weighted LanguageInvariant Representations. In International Conference on Learning Representations , 2022.

Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. Zero-Shot Dense Retrieval with Momentum Adversarial Domain Invariant Representations. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 4008-4020, 2022.

Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Enabling the Use of Lucene for Information Retrieval Research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1253-1256, 2017.

Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing Dense Retrieval Model Training with Hard Negatives. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 15031512, 2021.

Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, Joao P. Costeira, and Geoffrey J. Gordon. Adversarial Multiple Source Domain Adaptation. In Advances in Neural Information Processing Systems , 2018.

Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On Learning Invariant Representations for Domain Adaptation. In Proceedings of the 36th International Conference on Machine Learning , pages 7523-7532, 2019.

Sicheng Zhao, Xiangyu Yue, Shanghang Zhang, Bo Li, Han Zhao, Bichen Wu, Ravi Krishna, Joseph E. Gonzalez, Alberto L. Sangiovanni-Vincentelli, Sanjit A. Seshia, and Kurt Keutzer. A Review of Single-Source Deep Unsupervised Visual Domain Adaptation. IEEE Transactions on Neural Networks and Learning Systems , 33(2):473-493, 2022.

Xiaofeng Zhu and Diego Klabjan. Listwise Learning to Rank by Exploring Unique Ratings. In Proceedings of the 13th International Conference on Web Search and Data Mining , pages 798-806, 2020.

Honglei Zhuang, Xuanhui Wang, Michael Bendersky, and Marc Najork. Feature Transformation for Neural Ranking Models. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1649-1652, 2020.

Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2308-2313, 2023.

Shengyao Zhuang and Guido Zuccon. TILDE: Term Independent Likelihood moDEl for Passage Re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1483-1492, 2021.

Shengyao Zhuang, Hang Li, and Guido Zuccon. Deep Query Likelihood Model for Information Retrieval. In Advances in Information Retrieval , pages 463-470, 2021.

A Proofs for Section 3

Proposition A.1. Let µ, µ ′ be distributions supported on a metric space ( X , d ) , define and assume R = sup ( x,x ′ ) ∈ supp( µ × µ ′ ) d ( x, x ′ ) ≤ ∞ , and let L ( f ) denote the balanced 0-1 loss of a discriminator f : X → { 0 , 1 } , given by

then W 1 ( µ, µ ′ ) ≤ R (1 -min f L ( f )) .

Proof. Let Γ( µ, µ ′ ) denote the collection of couplings between µ, µ ′ , then by the definition of the Wasserstein1 distance,

̸

because ∫ ( µ ′ ( x ) -µ ( x )) d x = 0 . Note that the last line is R times the total variation between µ, µ ′ . On the other hand, rewrite

We know that the minimum 0-1 loss is attained by the Bayes predictor, f ∗ ( x ) = 1 [ µ ′ ( x ) ≥ µ ( x )] , so

by Eq. (1). The result then follows from an algebraic rearrangement of terms in Eq. (2).

Next, we provide a concrete example (supplementing the picture in Fig. 2) for showing that item-level alignment does not necessarily imply list-level alignment.

Example A.2 . Consider two uniform distributions µ list S , µ list T of two lists containing three real-valued items, where the lists are

The item-level distributions, µ item S , µ item T , which are obtained by aggregating the items from all lists, are both the uniform distribution of the same six items:

Note that item-level representations are automatically aligned since µ item S = µ item T , but the list-level representations are not, because supp( µ list S ) ∩ supp( µ list T ) = ∅ .

B Proofs for Section 4

This section provides the proof to Theorem 4.7, the domain adaptation generalization bound for ranking with list-level alignment. First, recall the following properties of Lipschitz functions:

Fact B.1 (Properties of Lipschitz Functions) .

If f : R d → R is differentiable, then it is L -Lipschitz (in Euclidean distance) if and only if ∥∇ f ∥ 2 ≤ L .

If f : X → R is L -Lipschitz and g : X → R is M -Lipschitz, then ( af + bg ) ∈ Lip( | a | L + | b | M ) , and max( f, g ) ∈ Lip(max( L, M )) .

If f : X → Y is L -Lipschitz and g : Y → Z is M -Lipschitz, then g ◦ f ∈ Lip( LM ) .

Proof.

For the backward direction, suppose bounded gradient norm, ∥∇ f ∥ 2 ≤ L , then by the mean value theorem, ∃ t ∈ [0 , 1] s.t. f ( y ) -f ( x ) = ∇ f ( z ) ⊤ ( y -x ) with z := (1 -t ) x + ty , so by Cauchy-Schwarz inequality,

For the forward direction, suppose f ∈ Lip( L ) , then by differentiability, ∇ f ( x ) ⊤ z = f ( x + z ) -f ( x ) + o ( ∥ z ∥ 2 ) . Set z := t ∇ f ( x ) , we have

and the result follows by dividing both sides by t ∥∇ f ( x ) ∥ 2 and taking t → 0 .

2. First,

Next, assume w.l.o.g. max( f ( x ) , g ( x )) -max( f ( y ) , g ( y )) ≥ 0 , then

We first prove Theorem 3.3 as a warm-up (restated below), the domain adaptation generalization bound for binary classification [52], since its proof shares the same organization as that of our Theorem 4.7, and it helps familiarizing readers of prior domain adaptation results and analysis techniques.

Theorem B.2. Let binary classification problems on a source and a target domain be given by joint distributions µ S , µ T of inputs and labels, ( X,Y ) ∈ X × { 0 , 1 } . Define the error rate of a predictor f : X → [0 , 1] on µ by

̸

̸

Let F ⊂ [0 , 1] X be an L -Lipschitz class of predictors. Define the minimum joint error rate by λ ∗ = min f ′ ( R S ( f ′ ) + R T ( f ′ )) , then for all f ∈ F ,

Proof. Define random variable η = 1 [ Y = 1] , then we may rewrite

Note that because 2 η -1 ∈ {-1 , +1 } ,

On the other hand,

Then by Fact B.1, the fact that the operation of taking the absolute value is 1 -Lipschitz, and Definition 3.2 of W 1 distance, for all f, f ′ ∈ F ,

and the result follows by taking the min over f ′ .

Next, we prove Theorem 4.7. The main idea is that under the setup and assumptions in Section 4, R S and R T can be written as the expectation of some Lipschitz function of Z ∼ µ Z S and µ Z T , respectively, so by Definition 3.2, their difference is upper bounded by a Lipschitz constant times W 1 ( µ Z S , µ Z T ) .

Although omitted, we remark that Theorem 4.7 can be extended to the cutoff version of the ranking metric u (e.g., NDCG@ p ) with a simple modification of the proof. Also, a finite sample generalization bound could be obtained using, e.g., Rademacher complexity, and additionally assuming Lipschitzness of the end-to-end scoring model [5, 51].

Proof of Theorem 4.7. Fix g ∈ G , and consider µ X S for now (analogous results hold for µ X T ). Which by Assumption 4.6, when restricted to supp( µ X S ) , has an L g -Lipschitz inverse g -1 . Define function ϵ h,g : Z → R ≥ 0 for a given h : Z → R ℓ as

and note that R S ( h ◦ g ) = E X ∼ µ X S [ ϵ h,g ( g ( X ))] =: E Z ∼ µ Z S [ ϵ h,g ( z )] . We show that ϵ h,g is Lipschitz provided that h is Lipschitz, from establishing the Lipschitzness of both terms in Eq. (3).

For the first term, because u is L u -Lipschitz w.r.t. y S ◦ g -1 ( z ) and y S ◦ g -1 ( z ) is in turn L y L g -Lipschitz w.r.t. z by Assumptions 4.3 and 4.4, it follows that z ↦→ u ( r, y S ◦ g -1 ( z )) is L u L y L g -Lipschitz w.r.t. z for any fixed r ∈ S ℓ , and therefore so is the function z ↦→ max r ∈ S ℓ u ( r, y S ◦ g -1 ( z )) by Fact B.1.

For the second term, we show that it is Lipschitz (in Euclidean distance) w.r.t. both y S ◦ g -1 ( z ) =: y and h ( z ) =: s . By Jensen's inequality and Fact B.1,

̸

where the second equality is due to the product rule, the third equality to d d x j softmax( x ) i = softmax( x ) i ( 1 [ i = j ] -softmax( x ) j ) , the last inequality to Assumption 4.3, and the last equality to identifying the pmf of the P-L model (Definition 4.1). Because h ∈ Lip( L h ) by Assumption 4.5, the two results above imply that the second term in Eq. (3) is Lipschitz: for all z, z ′ ∈ Z ,

Combining the two parts, we have that ϵ h,g is 2( L u L y L g + BℓL h ) -Lipschitz in z .

and the result follows by taking the min over h

.

Finally, we verify the Lipschitzness of RR and NDCG.

Proof of Corollary 4.8. Because RR ≤ 1 uniformly and ∥ y -y ′ ∥ 2 ≥ 1 for all y, y ′ ∈ { 0 , 1 } ℓ , y = y ′ , so y ↦→ RR ( r, y ) is 1 -Lipschitz.

̸

Proof of Corollary 4.9. We show that NDCG is Lipschitz w.r.t. y under the assumptions. Recall that

where r ∗ is the descending order of y .

Note that IDCG is continuous and piecewise differentiable w.r.t. y (each piece is associated with some r ′ ∈ S ℓ and supported on { y : arg max r DCG ( r, y ) = r ′ } ), then so is IDCG -1 on the set Y ′ := { y : C -1 ≤ IDCG ( y ) ≤ C } (which is our assumption) by chain rule. Clearly, DCG is continuous and differentiable w.r.t. y , so NDCG is continuous and piecewise differentiable on Y ′ by product rule. By Fact B.1, to show that NDCG is Lipschitz, we just need to show that its gradient norm w.r.t. y is bounded when evaluated on (the interior of) each piece.

Let r ∈ S ℓ be arbitrary, and y ∈ Y ′ be s.t. it lies in the interior of the respective piece, namely, by denoting r ∗ the descending order of y , the interior of the set { y ′ : arg max r DCG ( r, y ′ ) = r ∗ } = { y ′ : y ′ I ( r ∗ ) 1 ≥ y ′ I ( r ∗ ) 2 ≥ · · · ≥ y ′ I ( r ∗ ) ℓ } (recall that I ( r ) i denotes the index of the item with rank i ). Then, for all k ∈ [ ℓ ] ,

hence ∥∇ y NDCG ( r, y ) ∥ 2 ≤ √ ℓ ( C + C log( ℓ +1) 2 ) ≤ ˜ O ( C √ ℓ ) , and NDCG is ˜ O ( C √ ℓ ) -Lipschitz w.r.t. y . Here, ˜ O hides polylogarithmic terms.

C Additional Experiments and Details on Passage Reranking

We perform additional experiments for unsupervised domain adaptation on the passage reranking task considered in Section 5 in Appendix C.1, provide case studies on ListDA vs. zero-shot and QGen PL (Tables 8 and 9), hyperparameter settings in Appendix C.2, and details on the construction of training lists in Appendix C.3.

C.1 Additional Results

Pairwise Logistic Ranking Loss. Besides the listwise softmax cross-entropy ranking loss used in Section 5:

we experiment with the pairwise logistic ranking loss [8]:

The results with using the pairwise logistic loss for training on the Robust04 dataset are provided in Table 2a, which are not better than using the softmax cross-entropy loss (cf. Table 1; see also [25]), hence further experiments with this loss are not pursued.

(a) With pairwise logistic ranking loss in place of softmax cross-entropy on Robust04.

Source domain is MS MARCO. Gain function in NDCG is the identity map. * Improves upon zero-shot with statistical significance ( p ≤ 0 . 05 ) under the two-tailed Student's t -test. † Improves upon QGen PL. ‡ Improves upon ItemDA.

(b) With ItemDA with a transformer discriminator.

Source domain is MS MARCO. Gain function in NDCG is the identity map.

(c) With ListDA + QGen PL method.

Source domain is MS MARCO. Gain function in NDCG is the identity map. * Improves upon zero-shot with statistical significance ( p ≤ 0 . 05 ) under the two-tailed Student's t -test. † Improves upon QGen PL. ‡ Improves upon ItemDA.

Source domain is MS MARCO. Gain function in NDCG is the identity map.

As an implementation remark, in this set of experiments, we do not perform pairwise comparisons to obtain the predicted rank assignments during inference (where lists have length-1000) due to the high time complexity (the loss is aggregated pairwise during training, since we truncate training lists to length-31; see Appendix C.3). Whether or not the forward pass of the model involves pairwise computations does not matter to ListDA, which is applicable to any (pointwise, pairwise, or listwise) model as long as we can gather list-level representations; ListDA could be instantiated on pairwise models e.g. DuoT5 [44], although not pursued in this work.

ItemDA with Transformer Discriminator. To show that the improvements brought by ListDA over ItemDA is due to learning list-level invariant representations and not because of the use of the more expressive transformer as the discriminator vs. MLP, we experiment with a variant of ItemDA that uses the same transformer discriminator used in our ListDA runs. Although, we remark that whereas each input to the ListDA discriminator is a list of feature vectors as a lengthℓ sequence, each input to the ItemDA transformer discriminator is a single feature vector, or, a sequence of length-1. So, the attention mechanism of the transformer becomes pointless, and the ItemDA transformer discriminator collapses to an MLP (with skip connections and LayerNorm).

The results are presented in Table 2b (cf. Table 1). Across all datasets and metrics, no consistent improvement of ItemDA (transformer) over ItemDA (MLP) is observed.

ListDA + QGen PL Method, and Signal-1M Dataset. We experiment with supplementing ListDA with QGen pseudolabels by (uniformly) combining the training objectives of ListDA and QGen PL ( ListDA + QGen PL ). The results are presented in Table 2c. We also apply this method on the Signal-1M (RT) dataset [53], with results in Table 3. It is observed that reranking using neural rankers transferred from MS MARCO source domain does not perform better than BM25 (i.e., without any reranking) on Signal-1M, which is also noted in prior work [57, 32]. This does not mean that neural rerankers are worse than BM25, but could be that MS MARCO is not a good choice as the source domain for the target of Signal-1M, because of the arguably large domain shift between tweet retrieval and MS MARCO web search-qualitatively, it can be seen from Table 7 that the text styles and task semantics of the two domains are very different. Hence, the following discussions on Signal-1M results focus on comparing reranking results.

On Signal-1M, QGen PL improves upon the zero-shot baseline, but ListDA does not, which is likely due to the large domain shift between MS MARCO and Signal-1M that prevented ListDA from finding the correct source-target feature alignment without supervision. By incorporating QGen pseudolabels, ListDA performance improves with + QGen PL, which could have benefited from QGen q-d pairs acting as anchor points for ListDA to find the correct correspondence between source and target.

Overall, ListDA + QGen PL is the only method that consistently improves upon the zero-shot baseline on all four datasets, including Signal-1M, although it underperforms ListDA without QGen PL on the other three. Further improvements to this hybrid method may be possible with better strategies for balancing the training objectives of ListDA and QGen PL.

Case Studies. In Tables 8 and 9, we include examples where the ListDA top-1 reranked results are better than those of zero-shot and QGen PL, respectively, for qualitatively understanding the benefits and advantages of ListDA.

In zero-shot, the reranker is trained on the general domain MS MARCO dataset and applied on the target domains without adaptation. When the target domain differs from MS MARCO stylistically or consists of passages containing domain-specific words, as in the TREC-COVID and BioASQ datasets for biomedical retrieval, the zero-shot model, which has not seen the specialized language usages, may resort to keyword matching. Examples of such cases are presented in Table 8, where the top-1 passages reranked by the zero-shot model contain keywords from the query but are irrelevant.

In QGen PL, the reranker is trained by assuming the QGen synthesized query-document pairs to be relevant. However, as remarked in Section 5, because QGen is trained on MS MARCO and applied on the target domains in a zero-shot manner, the pseudolabels are not guaranteed to be valid, meaning that there could be false positives. This is observed in examples presented in Tables 7 and 9. One specific scenario where false positives hurt transfer performance is when the synthesized queries (of false positive documents) coincide with queries from the evaluation set, observed from the examples presented in Table 9 on TREC-COVID and BioASQ datasets. On the other hand, ListDA does not assume the pseudolabels in its training objective, so it does not suffer from these potential issues with QGen PL.

C.2 Hyperparameter Settings

For BM25, we use the implementation of Anserini [69], set k 1 = 0 . 82 and b = 0 . 68 on MS MARCO source domain, and k 1 = 0 . 9 and b = 0 . 4 on all target domains without tuning. As in [57], titles are indexed as a separate field with equal weight as the document body, when available.

For the RankT5 reranking model, it is fine-tuned from the T5 v1.1 base checkpoint on a Dragonfish TPU with 8x8 topology for 100,000 steps with a batch size of 32 per domain (each training list contains 31 items). We tune the learning rate η rank ∈ {5e-5, 1e-4, 2e-4}, and select the one with the best zero-shot performance to use on all adaptation methods, on each dataset (see Fig. 5 for zero-shot sweep results). We apply a learning rate schedule on η rank that decays (exponentially) by a factor of 0.7 every 5,000 steps. The concatenated query-document text inputs are truncated to 512 tokens.

For the domain discriminators, there are two hyperparameters: the strength of invariant feature learning λ , and the discriminator learning rate η ad . We tune both by sweeping λ ∈ {0.01, 0.02}, and η ad ∈ {10, 20, 40} × η rank as multiples of the reranker learning rate. The tuned hyperparameter settings of η rank , η ad and λ used in our experiments are included in Table 4.

0.56

0.58

Robust04

0.800

0.825

TREC-COVID

0.00005

0.0001

0.0002

η

rank

0.53

0.54

0.55

BioASQ

NDCG@5

(a) With softmax cross-entropy ranking loss.

0.555

0.560

0.565

Robust04

0.81

0.82

TREC-COVID

0.00005

0.0001

0.0002

η

rank

0.52

0.53

BioASQ

NDCG@5

(b) With pairwise logistic ranking loss.

0.575

0.600

Robust04

0.825

0.850

0.875

TREC-COVID

0.001

0.01

0.1

λ

0.54

0.56

BioASQ

NDCG@5

(a) Fix η ad = 0 . 004 and sweep λ .

0.57

0.58

0.59

Robust04

0.84

0.86

TREC-COVID

10

20

40

η

ad

0.54

0.56

BioASQ

×

η

rank

NDCG@5

(b) Fix λ = 0 . 02 and sweep η ad (as multiples of η rank ).

The sweep results for ListDA are plotted in Fig. 6. It is observed that a balanced choice of λ is needed to elicit the best performance from ListDA, and the same choice works fairly well across datasets. But for η ad, each dataset prefers different settings, probably due to their distinct domain characteristics.

In terms of running time, the adaptation methods of ListDA, ItemDA, and QGen PL all have double the training time compared to zero-shot learning due to data loading. This is because in addition to source domain data, the adaptation methods require target domain (unlabeled) data. Among ListDA, ItemDA, and QGen PL, the training times are roughly the same, because the overhead of the domain discriminators is not significant.

C.3 Training List Construction

Recall from the setup of ListDA on ranking problems that, the inputs are defined over lists and the invariant representations are learned at the list level. In other words, the ranking loss and the adversarial loss are to be computed on lists of ranking scores and feature vectors for each query (containing both relevant and irrelevant ones).

In our reranking setup, each list would be the topr passages retrieved by BM25 on a query, where we set r = 1000 in our evaluations. However, there are two complications. The first is that due to memory constraints, it is not feasible to feed all 1000 q-d pairs from each list through the RankT5 model in one pass during training (or time-consuming if via gradient accumulation). The second is that the source domain MS MARCO dataset only labels one relevant passage for each query, meaning that out of the 1000 passages retrieved by BM25 for each query, we may only know that (at most) one of them is relevant; the labels for the remaining 999 passages are unknown.

Negative Sampling. To address both issues, we truncate the list to length-31 for training and perform random sampling to gather irrelevant q-d pairs from the top 1000 BM25 retrieved passages.

(b) With pairwise logistic ranking loss.

On the MS MARCO source domain, given a query q and the top 1000 passages retrieved by BM25, d 1 , · · · , d 1000 , we yield a training list by bundling the one passage d ∗ labeled as relevant with 30 randomly sampled passages d N 1 , · · · , d N 30 to be treated as irrelevant, i.e., x = ([ q, d ∗ ] , [ q, d N 1 ] , · · · , [ q, d N 30 ]) and y = (1 , 0 , · · · , 0) .

On the target domains, we follow the same procedure but with QGen synthesized queries. Given a QGen query q and the top 1000 retrieved passages, the training list consists of the pseudolabeled passage ˆ d (i.e., the passage on which the query q was synthesized) and 30 randomly sampled irrelevant passages, i.e., x = ([ q, ˆ d ] , [ q, d N 1 ] , · · · , [ q, d N 30 ]) and y = (1 , 0 , · · · , 0) . Note that the pseudolabels y on the target domain are used by QGen PL for training but discarded by ListDA.

Reducing False Negatives on MS MARCO. One potential problem arising from sampling is that the constructed lists may contain false negatives (i.e., relevant documents that are incorrectly treated as irrelevant). In fact, false negatives are prevalent in the MS MARCO dataset (e.g., due to duplicate passages). While these false negatives are mostly harmless to source domain supervised training because the effects are canceled out when there are positively labeled q-d pairs to which the false negative q-d pairs are similar, false negatives can affect invariant representation learning.

One way that it may negatively affect invariant representation learning is that the duplicates in the lists (which have identical feature vectors) will cause ListDA to collapse distinct passages on the target domain to the same feature vector for achieving alignment, which is an artifact that will cause information loss in the target domain feature representations.

To lower the chance of sampling false negatives on MS MARCO, we rerank BM25-retrieved passages using a ranker pre-trained on MS MARCO, and sample negatives from passages ranked at 300 or lower, as the duplicates and (unlabeled) relevant passages are concentrated at the top [47]. We apply this sampling method only when constructing source domain lists for feature alignment (namely, in

ListDA and ItemDA), and it only affects the computation of adversarial loss. This sampling method is not applicable to the target domains because we do not have reliable pre-trained rankers for them. It is also not used for constructing the lists for source domain supervised training (i.e., the computation of ranking loss), because it excludes 'hard negatives' from the training lists and results in a weaker reranker, as observed in our preliminary experiments.

D Experiments on Yahoo! LETOR

We evaluate ListDA on the Yahoo! Learning to Rank Challenge v2.0 [11], and compare it to zero-shot and ItemDA (QGen PL is not applicable to tabular data). It is a web search ranking dataset with two subsets, 'Set 1' and 'Set 2', whose data originate from the US and a country in Asia, respectively; the training set sizes are 19,944 and 1,266, and the test set sizes are 6,983 and 3,798 (of which there are 47 and 17 lists of length-1). The dataset is tabular, and each item is represented by a 700-d vector with values in the range of [0 , 1] . Among the 700 features, 415 are defined on both sets (shared), and the other 285 are defined on either Set 1 or 2 (disjoint); we hence write each item x := [ x shared , x disjoint ] as a concatenation of the shared features and the disjoint ones, x shared ∈ R 415 , x disjoint ∈ R 285 .

We consider unsupervised domain adaptation from Set 1 to Set 2 (i.e., we discard the labels in Set 2). Our implementation uses PyTorch and the Hugging Face Transformers library [66]. 7

Models. Our models have the same setup as that of the passage reranking experiments in Section 5, except that the RankT5 text ranking model is replaced by a three-hidden-layer MLP following [75], and we treat the list of 256-d outputs at the last hidden layer as feature representations:

where W 1 ∈ R 1024 × 415 , W ′ 1 ∈ R 1024 × 285 , W 2 , W ′ 2 ∈ R 256 × 1024 , W 3 ∈ R 256 × 512 , and W 4 ∈ R 1 × 256 , and are randomly initialized. Note that we use separate input layers ( W 1 , W ′ 1 ) and first hidden layers ( W 2 , W ′ 2 ) for the shared and disjoint features, then concatenate the hidden representations. This tweak improves performance of the zero-shot baseline.

For ItemDA, we use the same ensemble of five three-layer MLPs as in Section 5. For the ListDA discriminator, an ensemble of five transformers is used, where each transformer is a stack of three T5 encoder blocks, with 4 attention heads ( num_heads ), size-32 key, query and value projections per attention head ( d_kv ), and size-1024 intermediate feedforward layers ( d_ff ).

Results. The results are presented in Table 5, which are ensembles of 10 separately trained models to reduce the variance from the randomness in the initialization and training process, due the small dataset size. Yahoo! LETOR is annotated with 5-level relevancy, and the scores are binarized for MAP and MRR metrics by mapping 0 (bad) and 1 (fair) to negative, and 2 (good), 3 (excellent), 4 (perfect) to positive. Thanks to the availability of labeled data on Set 2, we also include supervised results from training on Set 1 and 2 to serve as an upper bound for unsupervised domain adaptation.

ListDA again outperforms ItemDA for unsupervised domain adaptation, collaborating the discussions in the main sections of the paper. Although, here, their performance gap is smaller compared to passage reranking results, which we suspect is because the contextual (i.e., query) information for defining the list structure of the data is too weak or discarded when the numerical features are generated (e.g., the numerical features could represent the difference between query and document, hence may not retain information about the original query).

Hyperparameters. The model is trained from scratch on an NVIDIA A6000 GPU for 10,000 steps with a batch size of 32 per domain (length of training lists varies from one to 140 items). We apply a learning rate schedule on η rank that decays (exponentially) by a factor of 0.7 every 500 steps.

We tune the hyperparameters by performing grid search over: η rank ∈ {1e-5, 2e-5, 4e-5, 8e-5, 1e-4 , 2e-4, 4e-4, 8e-4, 1e-3}, η ad ∈ {0.2, 0.4, 0.8, 1, 2, 4, 8, 10} × η rank as multiples of the ranker learning rate, and λ ∈ {0.08, 0.1, 0.2, 0.4, 0.8, 1}. The tuned settings are included in Table 6.

Results are from an ensemble of five scoring models. Bold indicates the best unsupervised result. Source domain is Yahoo! LETOR (Set 1). Gain function in NDCG is the identity map. * Improves upon zero-shot with statistical significance ( p ≤ 0 . 05 ) under the two-tailed Student's t -test. ‡ Improves upon ItemDA. Significance tests are not performed on supervised results.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new domain-invariant ranking method which enforces list-wise feature invariance as opposed to item-wise feature invariance. The authors prove a novel domain adaptation bound specifically for the ranking problem. Authors show strong empirical improvement over item-wise invariance methods.\n\nSTRENGTHS:\n(1) The theorem is new and not a trivial extension of the existing domain adaptation generalization bound from Ben-David (which is for classification). In particular, some new techniques are required to analyze the ranking problem.\n\n(2) The method is novel and results are good.\n\nWEAKNESSES:\n(1) My main concern is with the empirical improvement between ItemDA and ListDA. I remain unconvinced that this improvement is coming from the list-wise invariance and not from the improvement in discriminator architecture. The authors mention that ItemDA uses a three-layer MLP, while ListDA uses a stack of three transformer blocks. is there anyway to compare the two methods using the same discriminator architecture? \n\n(2) While the novel generalization bound and algorithm are both novel. The theory doesn't explain why ListDA is better than ItemDA. Personally, I do not see this as an issue.\n\nQUESTIONS:\nI find the statement in line 141 confusing (even after looking at Appendix A.2). I would think that item-level invariance implies list-level invariance. For example, consider the case where $g$ was trained to output a domain-invariant feature representation of query-document pairs (at the item level). The ranker $h$ operates on these domain invariant features to form a list. How could the resulting list be different between the domains, if the ranker is constant and the output of $g$ is domain-invariant? Perhaps it would be helpful to elaborate on this point further in the paper.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on domain adaptation in the context of ranking problems. The authors propose a novel approach called list-level alignment to learn domain-invariant representations for ranking tasks. They demonstrate the benefits of this approach through theoretical analysis and empirical experiments. \n\nExisting domain adaptation methods for ranking, particularly invariant representation learning, have been sporadically used and lack theoretical guarantees.\n\nGiven the limitations of previous approaches, the authors aim to address the conceptual discrepancy between item-level alignment and the intrinsic list structure of ranking problems. They propose list-level alignment as a solution to learn higher-level invariant representations on rank lists. This approach not only provides a theoretical foundation for domain adaptation in ranking but also achieves better empirical performance.\n\nSTRENGTHS:\nI think this problem setting is interesting and practical. I have the following views on this paper's strengths part.\n\nFrom the perspective of problem and method, this paper addresses the challenge of training ranking models in domains with limited annotated data. It introduces domain adaptation as a transfer learning technique and highlights the need for invariant representation learning in the ranking literature. The authors propose a new approach called \\list-level alignment\\ that preserves the list structure of ranking problems and achieves higher-level invariant representations on the lists. The benefits of this approach include the establishment of the first domain adaptation generalization bound for ranking and better empirical performance on unsupervised domain adaptation tasks.\n\nExperimentally, the authors conducted experiments on unsupervised domain adaptation tasks to evaluate the performance of their proposed approach, list-level alignment. The proposed approach achieved better empirical performance on unsupervised domain adaptation tasks compared to existing methods. The approach also established the first domain adaptation generalization bound for ranking, providing theoretical guarantees for its implementation.\n\nWEAKNESSES:\nI have the following views on this paper's weakness part.\n\n- The paper acknowledges that existing domain adaptation methods for ranking lack theoretical guarantees, and it claims to have theoretical foundations for its proposed approach of list-level alignment. It's already a good contribution. However, the contribution of this paper in the theoretical aspect may be considered weak, since the contribution of this paper can be seen as an extension of existing domain adaptation theorems[1] applied to the ranking problem. Correct me if I misunderstand it.\n\n- One curious aspect here is that while the paper discusses domain adaptation (DA) in the ranking problem, the experiments are conducted on reranking. In the context of reranking, which involves reordering a list of candidate documents generated by an initial retrieval model in response to a search query. However, it does not directly address the fundamental ranking problem. This limitation of the paper should be further discussed to provide clarity.\n\n[1] Ben-David, Shai, et al. \"Analysis of representations for domain adaptation.\" Advances in neural information processing systems 19 (2006).\n\nQUESTIONS:\nIn my first point at my weakness part, if I misunderstand the theoretical contributions, the author could correct me.\n\nLIMITATIONS:\nYes, it's properly stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes to learn domain-invariant representations for ranking. In contrast to prior works that typically consider item-level alignment, they introduce the concept of list-level alignment to learn higher-level invariant representations on the lists. The domain adaptation generalization bound is explicitly established and extensive experiments on benchmark datasets reveal the effectiveness of the proposed approach. 

Post rebuttal: my major concerns have been adequately addressed, and thus I've decided to change my score from 4 to 5.

STRENGTHS:
- Domain adaptation for ranking problems is significant but receives scant attention.

- The paper in general is well-motivated and presents major challenges that need to be overcome to address DA for ranking. A new paradigm termed list-level alignment is proposed. 

- They evaluate and ablate the method in multiple benchmark datasets.

WEAKNESSES:
- The proposed alignment relies on the traditional domain adversarial training, offering no significant advancements. What are the differences between the proposed alignment and prior adversarial alignment, and why the proposed list-level alignment is preferable in the context of ranking problems? The primary distinctions between list-level alignment and other paradigms are not well-articulated.

- According to the reported results, the proposed ListDA shows marginal improvements over baseline methods in terms of MAP.

- The connection between the theory and the proposed method is not sufficiently clear and precise, leaving readers uncertain about their relationship.

QUESTIONS:
Please refer to the weakness.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach for domain adaptation on list ranking problems. In significant contrast with previous works, they adopt aligning list-level representations directly instead of item-level representations, and argue for its relevance in improving on list-level metrics pertinent to the problem of ranking. They also devise a theoretical bound for the target error in terms of source error, a joint hypothesis error and the list-level domain divergence, under suitable assumptions. Results on passage re-ranking task shows their strength in practical applications.\n\nSTRENGTHS:\n- The paper is very well written, easy to follow and extremely well presented.\n\n- The idea of extending UDA approaches to list-level re-ranking problems is interesting and highly relevant. The ideas discussed in the approach might be relevant to problems beyond ranking (like object detection in computer vision which also uses unique evaluation metrics).\n\n- The empirical results on the passage re-ranking task is strong.\n\n- The simple of idea of using a transformer discriminator to allow permutation in-variance is impressive.\n\nWEAKNESSES:\n- Few choices, like the use of transformer discriminator instead of MLP, wasn't evaluated or ablated empirically, although it makes sense intuitively. \n\n- The authors might also consider showing results on diverse ranking tasks beyond passage re-ranking, possibly in tasks like fine-grained computer vision as in [1], although I agree that this is much beyond the scope of this work.\n\n[1] Wang, Xinshao, et al. \"Ranked list loss for deep metric learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\nQUESTIONS:\nIs there a typo in L98? Since both $\\mathcal{L}_\\text{rank}$ and $D(\\mu_s,\\mu_t)$ need to be minimized according L100-102, but the negative sign in between does the opposite.\n\nLIMITATIONS:\nThe work is self-sufficient, and presents good evaluation to convey the effectiveness of the proposed method. Although, I feel that the scope of this or a follow-up work could be broadened a bit, to also incorporate problems beyond passage ranking into the evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}