SlowGuess/NeurIPS_2023_OCR huggingface dataset:
ListDA + QGen PL Method, and Signal-1M Dataset. We experiment with supplementing ListDA with QGen pseudolabels by (uniformly) combining the training objectives of ListDA and QGen PL (ListDA + QGen PL). The results are presented in Table 2c. We also apply this method on the Signal-1M (RT) dataset [53], with results in Table 3. It is observed that reranking using neural rankers transferred from MS MARCO source domain does not perform better than BM25 (i.e., without any reranking) on Signal-1M, which is also noted in prior work [57, 32]. This does not mean that neural rankers are worse than BM25, but could be that MS MARCO is not a good choice as the source domain for the target of Signal-1M, because of the arguably large domain shift between tweet retrieval and MS MARCO web search—qualitatively, it can be seen from Table 7 that the text styles and task semantics of the two domains are very different. Hence, the following discussions on Signal-1M results focus on comparing reranking results.
On Signal-1M, QGen PL improves upon the zero-shot baseline, but ListDA does not, which is likely due to the large domain shift between MS MARCO and Signal-1M that prevented ListDA from finding the correct source-target feature alignment without supervision. By incorporating QGen pseudolabels, ListDA performance improves with + QGen PL, which could have benefited from QGen q-d pairs acting as anchor points for ListDA to find the correct correspondence between source and target.
Overall, ListDA + QGen PL is the only method that consistently improves upon the zero-shot baseline on all four datasets, including Signal-1M, although it underperforms ListDA without QGen PL on the other three. Further improvements to this hybrid method may be possible with better strategies for balancing the training objectives of ListDA and QGen PL.
Case Studies. In Tables 8 and 9, we include examples where the ListDA top-1 reranked results are better than those of zero-shot and QGen PL, respectively, for qualitatively understanding the benefits and advantages of ListDA.
In zero-shot, the reranker is trained on the general domain MS MARCO dataset and applied on the target domains without adaptation. When the target domain differs from MS MARCO stylistically or consists of passages containing domain-specific words, as in the TREC-COVID and BioASQ datasets for biomedical retrieval, the zero-shot model, which has not seen the specialized language usages, may resort to keyword matching. Examples of such cases are presented in Table 8, where the top-1 passages reranked by the zero-shot model contain keywords from the query but are irrelevant.
In QGen PL, the reranker is trained by assuming the QGen synthesized query-document pairs to be relevant. However, as remarked in Section 5, because QGen is trained on MS MARCO and applied on the target domains in a zero-shot manner, the pseudolabels are not guaranteed to be valid, meaning that there could be false positives. This is observed in examples presented in Tables 7 and 9. One specific scenario where false positives hurt transfer performance is when the synthesized queries (of false positive documents) coincide with queries from the evaluation set, observed from the examples presented in Table 9 on TREC-COVID and BioASQ datasets. On the other hand, ListDA does not assume the pseudolabels in its training objective, so it does not suffer from these potential issues with QGen PL.
C.2 Hyperparameter Settings
For BM25, we use the implementation of Anserini [69], set  $k_{1} = 0.82$  and  $b = 0.68$  on MS MARCO source domain, and  $k_{1} = 0.9$  and  $b = 0.4$  on all target domains without tuning. As in [57], titles are indexed as a separate field with equal weight as the document body, when available.
For the RankT5 reranking model, it is fine-tuned from the T5 v1.1 base checkpoint on a Dragonfish TPU with 8x8 topology for 100,000 steps with a batch size of 32 per domain (each training list contains 31 items). We tune the learning rate  $\eta_{\mathrm{rank}} \in \{5\mathrm{e} - 5, 1\mathrm{e} - 4, 2\mathrm{e} - 4\}$ , and select the one with the best zero-shot performance to use on all adaptation methods, on each dataset (see Fig. 5 for zero-shot sweep results). We apply a learning rate schedule on  $\eta_{\mathrm{rank}}$  that decays (exponentially) by a factor of 0.7 every 5,000 steps. The concatenated query-document text inputs are truncated to 512 tokens.
For the domain discriminators, there are two hyperparameters: the strength of invariant feature learning  $\lambda$ , and the discriminator learning rate  $\eta_{\mathrm{ad}}$ . We tune both by sweeping  $\lambda \in \{0.01, 0.02\}$  and  $\eta_{\mathrm{ad}} \in \{10, 20, 40\} \times \eta_{\mathrm{rank}}$  as multiples of the reranker learning rate. The tuned hyperparameter settings of  $\eta_{\mathrm{rank}}$ ,  $\eta_{\mathrm{ad}}$  and  $\lambda$  used in our experiments are included in Table 4.
(a) With softmax cross-entropy ranking loss.
(b) With pairwise logistic ranking loss.
Figure 5: Zero-shot performance under different hyperparameter settings for  $\eta_{\mathrm{rank}}$
(a) Fix  $\eta_{\mathrm{ad}} = 0.004$  and sweep  $\lambda$ .
(b) Fix  $\lambda = 0.02$  and sweep  $\eta_{\mathrm{ad}}$  (as multiples of  $\eta_{\mathrm{rank}}$ ).
Figure 6: ListDA performance under different hyperparameter settings for  $\lambda$  and  $\eta_{\mathrm{ad}}$ . Grey horizontal line is zero-shot performance with the best  $\eta_{\mathrm{rank}}$  setting.
The sweep results for ListDA are plotted in Fig. 6. It is observed that a balanced choice of  $\lambda$  is needed to elicit the best performance from ListDA, and the same choice works fairly well across datasets. But for  $\eta_{\mathrm{ad}}$ , each dataset prefers different settings, probably due to their distinct domain characteristics.
In terms of running time, the adaptation methods of ListDA, ItemDA, and QGen PL all have double the training time compared to zero-shot learning due to data loading. This is because in addition to source domain data, the adaptation methods require target domain (unlabeled) data. Among ListDA, ItemDA, and QGen PL, the training times are roughly the same, because the overhead of the domain discriminators is not significant.
C.3 Training List Construction
Recall from the setup of ListDA on ranking problems that, the inputs are defined over lists and the invariant representations are learned at the list level. In other words, the ranking loss and the adversarial loss are to be computed on lists of ranking scores and feature vectors for each query (containing both relevant and irrelevant ones).
In our reranking setup, each list would be the top- $r$  passages retrieved by BM25 on a query, where we set  $r = 1000$  in our evaluations. However, there are two complications. The first is that due to memory constraints, it is not feasible to feed all 1000 q-d pairs from each list through the RankT5 model in one pass during training (or time-consuming if via gradient accumulation). The second is that the source domain MS MARCO dataset only labels one relevant passage for each query, meaning that out of the 1000 passages retrieved by BM25 for each query, we may only know that (at most) one of them is relevant; the labels for the remaining 999 passages are unknown.
Negative Sampling. To address both issues, we truncate the list to length-31 for training and perform random sampling to gather irrelevant q-d pairs from the top 1000 BM25 retrieved passages.
Table 4: Hyperparameter settings of RankT5 model and domain discriminators for passage reranking experiments.
(a) With softmax cross-entropy ranking loss.
Target domain	Method	ηrank	ηad	λ
Robust04	Zero-shot	-	-
QGen PL	-	-
ItemDA (MLP)	1e-4	1e-3	0.01
ItemDA (transformer)	2e-3	0.01
ListDA	4e-3	0.01
ListDA + QGen PL	1e-3	0.02
TREC-COVID	Zero-shot	-	-
QGen PL	-	-
ItemDA (MLP)	2e-4	8e-3	0.01
ItemDA (transformer)	2e-3	0.01
ListDA	4e-3	0.01
ListDA + QGen PL	4e-3	0.02
BioASQ	Zero-shot	-	-
QGen PL	-	-
ItemDA (MLP)	2e-4	4e-3	0.01
ItemDA (transformer)	2e-3	0.01
ListDA	8e-3	0.01
ListDA + QGen PL	4e-3	0.02
Signal-1M	Zero-shot	-	-
QGen PL	5e-5	-	-
ListDA	2e-3	0.01
ListDA + QGen PL	1e-3	0.02
(b) With pairwise logistic ranking loss.
Target domain	Method	ηrank	ηad	λ
Robust04	Zero-shot	1e-4	-	-
QGen PL	-	-
ItemDA (MLP)	2e-3	0.01
ListDA	2e-3	0.01
On the MS MARCO source domain, given a query  $q$  and the top 1000 passages retrieved by BM25,  $d_{1},\dots ,d_{1000}$ , we yield a training list by bundling the one passage  $d^{*}$  labeled as relevant with 30 randomly sampled passages  $d_{N_1},\dots ,d_{N_{30}}$  to be treated as irrelevant, i.e.,  $x = ([q,d^{*}],[q,d_{N_{1}}],\dots ,[q,d_{N_{30}}])$  and  $y = (1,0,\dots ,0)$ .
On the target domains, we follow the same procedure but with QGen synthesized queries. Given a QGen query  $q$  and the top 1000 retrieved passages, the training list consists of the pseudolabeled passage  $\hat{d}$  (i.e., the passage on which the query  $q$  was synthesized) and 30 randomly sampled irrelevant passages, i.e.,  $x = ([q,\hat{d}],[q,d_{N_1}],\dots ,[q,d_{N_{30}}])$  and  $y = (1,0,\dots ,0)$ . Note that the pseudolabels  $y$  on the target domain are used by QGen PL for training but discarded by ListDA.
Reducing False Negatives on MS MARCO. One potential problem arising from sampling is that the constructed lists may contain false negatives (i.e., relevant documents that are incorrectly treated as irrelevant). In fact, false negatives are prevalent in the MS MARCO dataset (e.g., due to duplicate passages). While these false negatives are mostly harmless to source domain supervised training because the effects are canceled out when there are positively labeled q-d pairs to which the false negative q-d pairs are similar, false negatives can affect invariant representation learning.
One way that it may negatively affect invariant representation learning is that the duplicates in the lists (which have identical feature vectors) will cause ListDA to collapse distinct passages on the target domain to the same feature vector for achieving alignment, which is an artifact that will cause information loss in the target domain feature representations.
To lower the chance of sampling false negatives on MS MARCO, we rerank BM25-retrieved passages using a ranker pre-trained on MS MARCO, and sample negatives from passages ranked at 300 or lower, as the duplicates and (unlabeled) relevant passages are concentrated at the top [47]. We apply this sampling method only when constructing source domain lists for feature alignment (namely, in
ListDA and ItemDA), and it only affects the computation of adversarial loss. This sampling method is not applicable to the target domains because we do not have reliable pre-trained rankers for them. It is also not used for constructing the lists for source domain supervised training (i.e., the computation of ranking loss), because it excludes "hard negatives" from the training lists and results in a weaker reranker, as observed in our preliminary experiments.
D Experiments on Yahoo! LETOR
We evaluate ListDA on the Yahoo! Learning to Rank Challenge v2.0 [11], and compare it to zero-shot and ItemDA (QGen PL is not applicable to tabular data). It is a web search ranking dataset with two subsets, "Set 1" and "Set 2", whose data originate from the US and a country in Asia, respectively; the training set sizes are 19,944 and 1,266, and the test set sizes are 6,983 and 3,798 (of which there are 47 and 17 lists of length-1). The dataset is tabular, and each item is represented by a 700-d vector with values in the range of  $[0,1]$ . Among the 700 features, 415 are defined on both sets (shared), and the other 285 are defined on either Set 1 or 2 (disjoint); we hence write each item  $x \coloneqq [x_{\mathrm{shared}}, x_{\mathrm{disjoint}}]$  as a concatenation of the shared features and the disjoint ones,  $x_{\mathrm{shared}} \in \mathbb{R}^{415}$ ,  $x_{\mathrm{disjoint}} \in \mathbb{R}^{285}$ .
We consider unsupervised domain adaptation from Set 1 to Set 2 (i.e., we discard the labels in Set 2). Our implementation uses PyTorch and the Hugging Face Transformers library [66].<sup>7</sup>
Models. Our models have the same setup as that of the passage reranking experiments in Section 5, except that the RankT5 text ranking model is replaced by a three-hidden-layer MLP following [75], and we treat the list of 256-d outputs at the last hidden layer as feature representations:
$$
\begin{array}{l} g (x) _ {i} =: v _ {i} = \operatorname {R e L U} \left(W _ {3} \left[ \begin{array}{c} \operatorname {R e L U} (W _ {2} \operatorname {R e L U} (W _ {1} x _ {i, \text {s h a r e d}} + b _ {1}) + b _ {2}) \\ \operatorname {R e L U} (W _ {2} ^ {\prime} \operatorname {R e L U} (W _ {1} ^ {\prime} x _ {i, \text {d i s j o i n t}} + b _ {1} ^ {\prime}) + b _ {2} ^ {\prime}) \end{array} \right] + b _ {3}\right), \\ h (x) _ {i} =: s _ {i} = W _ {4} v _ {i} + b _ {4}, \\ \end{array}
$$
where  $W_{1}\in \mathbb{R}^{1024\times 415}$ ,  $W_{1}^{\prime}\in \mathbb{R}^{1024\times 285}$ ,  $W_{2},W_{2}^{\prime}\in \mathbb{R}^{256\times 1024}$ ,  $W_{3}\in \mathbb{R}^{256\times 512}$ , and  $W_{4}\in \mathbb{R}^{1\times 256}$ , and are randomly initialized. Note that we use separate input layers  $(W_{1},W_{1}^{\prime})$  and first hidden layers  $(W_{2},W_{2}^{\prime})$  for the shared and disjoint features, then concatenate the hidden representations. This tweak improves performance of the zero-shot baseline.
For ItemDA, we use the same ensemble of five three-layer MLPs as in Section 5. For the ListDA discriminator, an ensemble of five transformers is used, where each transformer is a stack of three T5 encoder blocks, with 4 attention heads (num_heads), size-32 key, query and value projections per attention head (d_kv), and size-1024 intermediate feedforward layers (d_ff).
Results. The results are presented in Table 5, which are ensembles of 10 separately trained models to reduce the variance from the randomness in the initialization and training process, due the small dataset size. Yahoo! LETOR is annotated with 5-level relevancy, and the scores are binarized for MAP and MRR metrics by mapping 0 (bad) and 1 (fair) to negative, and 2 (good), 3 (excellent), 4 (perfect) to positive. Thanks to the availability of labeled data on Set 2, we also include supervised results from training on Set 1 and 2 to serve as an upper bound for unsupervised domain adaptation.
ListDA again outperforms ItemDA for unsupervised domain adaptation, collaborating the discussions in the main sections of the paper. Although, here, their performance gap is smaller compared to passage reranking results, which we suspect is because the contextual (i.e., query) information for defining the list structure of the data is too weak or discarded when the numerical features are generated (e.g., the numerical features could represent the difference between query and document, hence may not retain information about the original query).
Hyperparameters. The model is trained from scratch on an NVIDIA A6000 GPU for 10,000 steps with a batch size of 32 per domain (length of training lists varies from one to 140 items). We apply a learning rate schedule on  $\eta_{\mathrm{rank}}$  that decays (exponentially) by a factor of 0.7 every 500 steps.
We tune the hyperparameters by performing grid search over:  $\eta_{\mathrm{rank}} \in \{1\mathrm{e} - 5, 2\mathrm{e} - 5, 4\mathrm{e} - 5, 8\mathrm{e} - 5, 1\mathrm{e} - 4, 2\mathrm{e} - 4, 4\mathrm{e} - 4, 8\mathrm{e} - 4, 1\mathrm{e} - 3\}$ ,  $\eta_{\mathrm{ad}} \in \{0.2, 0.4, 0.8, 1, 2, 4, 8, 10\} \times \eta_{\mathrm{rank}}$  as multiples of the ranker learning rate, and  $\lambda \in \{0.08, 0.1, 0.2, 0.4, 0.8, 1\}$ . The tuned settings are included in Table 6.
Our code for this set of experiments is available in the supplementary material, and the data can be requested at https://webscope.sandbox.yahoo.com/catalog.php?datatype=c.
Table 5: Ranking performance of MLP ranker on Yahoo! LETOR (Set 2).
Target domain	Method	MAP	MRR@10	NDCG@5	NDCG@10	NDCG@20
Yahoo! LETOR (Set 2)	Zero-shot	0.5464	0.6796	0.7466	0.7778	0.8308
Supervised	0.5714	0.7045	0.7760	0.8012	0.8479
ItemDA	0.5604*	0.6984*	0.7563*	0.7822	0.8350*
ListDA	0.5633*	0.7026*	0.7599*	0.7845*	0.8355*
Results are from an ensemble of five scoring models. Bold indicates the best unsupervised result. Source domain is Yahoo! LETOR (Set 1). Gain function in NDCG is the identity map. *Improves upon zero-shot with statistical significance  $(p \leq 0.05)$  under the two-tailed Student's  $t$ -test. ‡Improves upon ItemDA. Significance tests are not performed on supervised results.
Table 6: Hyperparameter settings of MLP ranker and domain discriminators for Yahoo! LETOR experiments.
Target domain	Method	ηrank	ηad	λ
Yahoo! LETOR (Set 2)	Zero-shot	8e-4	-	-
Supervised	4e-5	-	-
ItemDA	8e-4	1.6e-3	0.4
ListDA	8e-4	1.6e-3	0.8