SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhen Qin, Rolf Jagerman, Rama Pasumarthi, Honglei Zhuang, He Zhang, Aijun Bai, Kai Hui, Le Yan, Xuanhui Wang*
Google  
Mountain View, CA, US 94043
The distillation of ranking models has become an important topic in both academia and industry. In recent years, several advanced methods have been proposed to tackle this problem, often leveraging ranking information from teacher rankers that is absent in traditional classification settings. To date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide range of tasks and datasets make it difficult to assess or invigorate advances in this field. This paper first examines representative prior arts on ranking distillation, and raises three questions to be answered around methodology and reproducibility. To that end, we propose a systematic and unified benchmark, Ranking Distillation Suite (RD-Suite), which is a suite of tasks with 4 large real-world datasets, encompassing two major modalities (textual and numeric) and two applications (standard distillation and distillation transfer). RD-Suite consists of benchmark results that challenge some of the common wisdom in the field, and the release of datasets with teacher scores and evaluation scripts for future research. RD-Suite paves the way towards better understanding of ranking distillation, facilities more research in this direction, and presents new challenges.
Ranking models have become the interface between users and many applications, such as search engines and recommender systems [43, 11, 17]. In recent years, due to the popularity of large neural ranking models [22, 18, 45], as well as the interest in model serving in practical scenarios, such as on mobile devices [37], the intersection of learning to rank (LTR) [19] and knowledge distillation [10, 12] has drawn much attention in both academia and industry [32, 35, 39, 40, 44].
Examining the evaluation and experimental setup of many of these papers, we found that there is no unified consensus on what makes an acceptable test bed for benchmarking ranking distillation methods. There is also a large diversity in the types of tasks and datasets adopted, as well as the configurations of teacher and student rankers. These make comparison of different methods as well as an assessment of their relative strengths and weaknesses difficult.
In this paper, we propose a new benchmark, Ranking Distillation Suite (RD-Suite), for the purpose of benchmarking ranking distillation methods. We design a benchmark suite comprised of two major modalities in the ranking literature, sensible teacher and student model architectures, and both standard distillation and a novel distillation transfer setting. The datasets and evaluation scripts are publicly available $^2$  and the experimental configurations are made clear to encourage fair comparisons for future research on the important topic.
\*Correspond to Zhen Qin <zhenqin@google.com>. 2https://github.com/tensorflow/ranking/tree/master/tensorflow_ranking/datasets/ rd-suite
While the focus of this benchmark is an empirical comparison, we are also fundamentally interested in understanding the open questions and limitations of existing methods on ranking distillation. By examining the general formulation and prior art, we raise three questions around methodology and reproducibility, and provide discussions that challenge some of the common wisdom in the field, shine light on the empirical results, rationalize our design choices, and open up research challenges to tackle. We believe such a side-by-side performance benchmark will be valuable to the community, providing deeper insight on the practical effectiveness of different methods. The contributions and resulted artifacts of this work include:
- A systematic ranking distillation benchmark covering a variety of datasets, modalities, and methods, based on design choices around generality, simplicity, and reproducibility.
- The raise and analysis of key issues that challenge prior art and pave the way towards better understanding of ranking distillation.
- The release of easily accessible datasets and evaluation scripts to facilitate future research.
We describe the general formulation of ranking distillation and representative existing arts.
2.1 The General Formulation
For regular LTR, the training data can be represented as a set  $\Psi = \{(\mathbf{x},\mathbf{y})\in \chi^n\times \mathcal{R}^n)\}$ , where  $\mathbf{x}$  is a list of  $n$  items  $x_{i}\in \chi$  and  $\mathbf{y}$  is a list of  $n$  relevance labels  $y_{i}\in \mathcal{R}$  for  $1\leq i\leq n$ . We use  $\chi$  as the universe of all items. The objective is to learn a function that produces an ordering of items in  $\mathbf{x}$  so that the utility of the ordered list is maximized. Most LTR algorithms formulate the problem as learning a ranking function to score and sort the items in a list. As such, the goal of LTR boils down to finding a parameterized ranking function  $f(\cdot ;\theta):\chi^n\to \mathcal{R}^n$ , where  $\theta$  denotes the set of trainable parameters, to minimize the empirical loss:
$$
\mathcal {L} (f (\cdot ; \theta)) = \frac {1}{| \Psi |} \sum_ {(\mathbf {x}, \mathbf {y}) \in \Psi} l _ {r e l} (\mathbf {y}, f (\mathbf {x}; \theta)), \tag {1}
$$
where  $l_{rel}(\cdot)$  is the loss function on a single list with relevance labels.
In tabular LTR [19, 23, 30], each  $x_{i}$  corresponds to a query-item pair represented as a numeric feature vector, and  $\theta$  is usually a linear, tree, or multilayer perception (MLP) model. For text ranking [22, 45], each  $x_{i}$  represents query and document text strings and transformer-based models, such as BERT [8] and T5 [31], are the norm for  $\theta$ .
For ranking distillation, in addition to the original training data, it is assumed that there is a teacher ranker  $f(\cdot ;\theta^t)$  producing teacher scores (or distillation labels)  $\mathbf{y}^t = g(f(\cdot ;\theta^t))$ , where  $g(\cdot)$  is an optional (ranking preserving) transform function on teacher scores. The goal of ranking distillation is to train a student model  $f(\cdot ;\theta^{s})$ , where  $\theta^s$  usually has significantly smaller capacity than  $\theta^t$ , with the following loss:
$$
\mathcal {L} \left(f (\cdot ; \theta^ {s})\right) = \frac {1}{| \Psi |} \sum_ {(\mathbf {x}, \mathbf {y}) \in \Psi} \left(\alpha \times l _ {r e l} (\mathbf {y}, f (\mathbf {x}; \theta^ {s})) + (1 - \alpha) \times l _ {d i s t i l l} \left(\mathbf {y} ^ {t}, f (\mathbf {x}; \theta^ {s})\right)\right), \tag {2}
$$
where  $\alpha \in [0,1]$  is a weighting factor and  $l_{distill}$  is the additional distillation loss.
2.2 Existing Art
Regular LTR research has proposed a rich set of  $l_{rel}(\cdot)$ , including pointwise, pairwise, and listwise losses [19]. Here we describe two recent most representative ranking distillation work (see more discussions in Sec. 7) that focus on  $l_{distill}$ .
RD [35] treats top  $K$  ranked items by the teacher ranker as positive examples (no negatives are considered), and apply a pointwise sigmoid cross-entropy loss:
$$
l _ {\mathrm {R D}} \left(\mathbf {y} ^ {t}, f (\mathbf {x}; \theta^ {s})\right) = - \sum_ {i = 1} ^ {K} \log f \left(x _ {\pi (i)}; \theta^ {s}\right), \tag {3}
$$
where  $\pi (i)$  is the  $i$ -th document in the ordering  $\pi$  induced by teacher ranker.
RankDistil [32] can be treated as a listwise generalization of RD, trying to preserve ordering of the top  $K$  items induced by the teacher ranker:
$$
l _ {\text {R a n k D i s t i l}} \left(\mathbf {y} ^ {t}, f (\mathbf {x}; \theta^ {s})\right) = E _ {\pi \sim P _ {\mathbf {y} ^ {t}}} \left[ - \log P _ {s} \left(\pi , f (\mathbf {x}; \theta^ {s})\right) \right], \tag {4}
$$
$$
P _ {s} \left(\pi , f (\mathbf {x}; \theta^ {s})\right) = \frac {1}{(L - K) !} \prod_ {j = 1} ^ {K} \frac {\exp \left(f \left(x _ {\pi (j)} ; \theta^ {s}\right)\right)}{\sum_ {l = j} ^ {L} \exp \left(f \left(x _ {\pi (l)} ; \theta^ {s}\right)\right)} \tag {5}
$$
is the Plackett-Luce probability model,  $L$  is the length of the ranking list, and  $\pi(j)$  is the  $j$ -th document from a sampled ordering  $\pi$ . Essentially, it samples several length-  $K$  permutations from teacher ranker score distribution  $(P_{\mathbf{y}^t})$  and tries to maximize the log likelihood of the induced permutations from student scores of the corresponding documents.
3 Three Questions on Ranking Distillation
By examining the general formulation and prior art, we raise three questions about the current literature on ranking distillation.
3.1 Q1: Where is the knowledge in ranking distillation?
An implicit assumption of RD, RankDistil, and related methods is, the knowledge for distillation [12] is in the (pointwise or listwise) top  $K$  orderings.
There are three concerns of this assumption: First, they largely ignore the teacher score values<sup>3</sup>. In contrast, in classification distillation, the logit values of all classes are considered [12]. Importantly, ordering can be derived from score values, but not the other way. Is it possible that some knowledge is lost by only considering the ordering? Second, the top  $K$  is a hyperparameter. Besides more tuning efforts, it is counter-intuitive that it should be the same for all lists in a dataset, due to the potentially huge variance in queries and their corresponding documents in practice. Third, documents outside of top  $K$  may be useful and contribute to the knowledge. The fact that prior work mainly compared under this assumption (e.g., RankDistil mainly compared with RD) makes the answer to the questions unclear.
3.2 Q2: How to handle teacher ranker scores?
As we argued to potentially leverage teacher score values above, the teacher scores need closer examination. In general, the teacher ranker scores may not be constrained – the ranks of candidate documents are invariant to any transformation that preserves the order of the scores. An example is the scale invariance of many advanced loss functions:
- The pairwise logistic loss in RankNet [4]:
$$
l _ {\text {P a i r L o g}} (\mathbf {y}, \mathbf {s}) = - \sum_ {i \neq j} \mathbb {I} _ {y _ {i} > y _ {j}} \ln \frac {\exp \left(s _ {i} - s _ {j}\right)}{1 + \exp \left(s _ {i} - s _ {j}\right)}. \tag {6}
$$
- The listwise Softmax loss [5, 3]:
$$
l _ {\mathrm {S o f t m a x}} (\mathbf {y}, \mathbf {s}) = - \sum_ {i} y _ {i} \ln \frac {\exp (s _ {i})}{\sum_ {j} \exp (s _ {j})} = - \sum_ {i} y _ {i} \ln \frac {1}{\sum_ {j} \exp (s _ {j} - s _ {i})},
$$
where we use  $\mathbf{s} = f(\mathbf{x};\theta)$  for conciseness. In these popular ranking losses, scores always appear in the paired form of  $s_i - s_j$  or  $s_j - s_i$ . So when a constant is added to each score, the losses stay invariant. Many other ranking losses, such as Approximate NDCG loss [27, 2], also have the same translation-invariant property.
${}^{3}$  Strictly speaking,ordering depends on the values. In this work we differentiate between absolute values and ordering.
This is in contrast to classification problems, where the logits have clear probabilistic meanings over all classes. The potentially unconstrained teacher scores need careful treatment. For example, many loss functions (such as cross-entropy based ones) are ill-defined with negative labels, potentially leading to devastating downstream distillation performance.
3.3 Q3: How to ensure reproducible and fair evaluations?
We note several complexities to ensure reproducible and fair evaluation for ranking distillation research. First, for datasets with large corpus, a retrieval stage is performed before ranking, which changes the data distribution for the ranking stage. Second, it is clear that the teacher model has significant effect on the downstream distillation task. For example, as we discussed, teacher ranker scores have large freedom and are difficult to reproduce. Third, researchers should be careful about what to ablate to focus on progress on distillation methods. In Eq. 2, we can see that (ranking) distillation is multi-objective. Though classification problems almost universally use the softmax cross-entropy loss for  $l_{\mathrm{rel}}$ , ranking problems have more flexibility for this term. In [32], the two loss terms always change together - it is unclear if the performance differences are from a better relevance loss or distillation method. Last but not least, there is no consensus on how to calculate ranking metrics. For example, queries without any relevant documents may get perfect score [25], 0 [13], or ignored [23]. Different metric definition makes cross-examination among papers difficult.
4 RD-Suite
We describe RD-Suite, starting with a set of desiderata motivated by [36], followed by the actions we take, and how they link to the desiderata and questions raised in the previous section.
4.1 Desiderata
For creating the RD-Suite, we established a set of desiderata:
D1. Generality: Given the long history of learning to rank research on tabular datasets, as well as the more recent interest in text ranking, RD-Suite includes tasks in both modalities. Also, we focus on distillation objectives in this paper, instead of model architecture specific distillation techniques (see Sec. 7), though the latter should benefit from better objective functions and easily compare against the benchmark with appropriate ablations.
D2. Simplicity: The tasks should have a simple setup. All factors that make comparisons difficult should be removed. This encourages simple models instead of cumbersome pipelined approaches. For instance, we consider pretraining to be out of scope of this benchmark. The student rankers are either simple to implement or initiated from publicly available checkpoints.
D3. Challenging: The tasks should be difficult enough for current models to ensure that there is room for improvement to encourage future research in this direction. We leverage state-of-the-art teacher rankers that still have good advantage over existing methods. We also introduce a novel distillation transfer task with encouraging results but considerable headroom.
D4. Non-resource intensive and accessible: The benchmarks should be deliberately designed to be lightweight so as to be accessible to researchers without industry-grade computing resources. We consider linear students for tabular ranking and regular public BERT models for text ranking. We handle the more expensive teacher ranker training and inference, and directly expose the distillation dataset to the community.
D5. Fairness. We note that it is non-trivial and almost impossible to conduct a perfectly fair evaluation of all models. The large search space motivates us to follow a set of fixed hyperparameters for most models (discussed in Sec. 4.6 with few exceptions). The best performance and relative order of the models may change if we aggressively tune hyperparameters for all models. Hence, the results provided in this paper are not meant to be a final authoritative document on which method is the best. Instead, we provide a starting point for future research and strive to be as fair as possible with the tuning process clearly laid out.
RD-Suite has four tasks, differing in terms of data modality, teacher model domain, and availability of relevance label. They are described in Table 1. Specifically, T1 (Text Ranking Distillation) and T4 (Tabular Ranking Distillation) are standard ranking distillation tasks on two common modalities in the LTR literature. T2 (Distillation Transfer) and T3 (Distillation Transfer Zeroshot) are motivated by the popularity of domain adaptation research where knowledge is transferred from source to target domains [31, 18]. We believe the tasks are comprehensive and reflect real-world applications: T1 and T4 are standard ranking distillation tasks, T2 is applicable when target domain data is not sufficient to tune the teacher model, or the teacher model is not accessible for tuning. T3 is applicable when there are no labels for the target domain, which is the zeroshot learning scenario [38]. Note that domain adaptation is not applicable to tabular LTR since different datasets are in different feature spaces.
Table 1: The tasks of RD-Suite. In-domain means teacher was trained on the same dataset as the student and out-domain otherwise.
Tasks	Modality	Teacher Domain	Relevance Label
T1	Text	In-domain	Available
T2	Text	Out-domain	Available
T3	Text	Out-domain	Not Available
T4	Tabular	In-domain	Available
The tasks are designed to cover wide application scenarios (D1) while maintaining simplicity (D2) - for example, T1, T2, and T3 can use the same implementation by changing teacher label and  $\alpha$  (T3 simply has  $\alpha = 0$ ). The novel ranking distillation transfer setting is quite challenging (D3) and not well studied in the literature.
4.3 Teacher and Student Configurations
We make the teacher and student ranker configurations clear. For text datasets, the teacher rankers are from RankT5 [45], which use encoder-decoder T5 [31] and listwise softmax ranking losses. For tabular datasets, the teacher rankers are from [13], which use MLP and a novel LambdaLoss that will be discussed below. To our knowledge, they are arguably the state-of-the-art in their respective tasks and we get the teacher models from the authors. As for student rankers, we use the publicly available BERT-base [8] checkpoint for text datasets and linear model for tabular datasets.
The advanced teacher rankers provide reasonable headroom (D3) and the student configurations follow D2 and D5 - they are easily accessible or easy to implement, and can be trained with accessible computing resources.
4.4 Datasets
All datasets in RD-Suite are popular public datasets. For text ranking, we use MSMARCO [1] and NQ [16], two of the most popular datasets for text ranking. Both datasets have large document corpus and require a retrieval stage before ranking. We leverage recent neural retrievers [20] to retrieve the top 50 candidates for each query.
For tabular ranking, we use Web30K [26] and Istella [7], two of the most popular datasets for tabular ranking. Retrieval is not needed for these datasets.
RD-Suite will release the teacher scores and retrieved document ids (on MSMARCO and NQ) for each query, to hide the potentially expensive teacher model inference and retrieval from ranking distillation research (D4) while guaranteeing fairness (D5, Q3).
4.5 The Ranking Loss Family for Distillation
As we discussed in Q1 (Where is the knowledge in ranking distillation?), there are several concerns around existing ranking distillation methods, namely they do not consider score values, has a top  $K$  hyperparameter, and largely ignore documents outside of the top  $K$ .
Interestingly, we find that many ranking losses naturally have different behaviors around the concerns, but are not well studied in the distillation context. Therefore, we include the "ranking loss family"
for distillation in RD-Suite to study their effects and gain better understanding of assumptions of existing methods. We include six ranking losses that are some of the most representative methods in the literature, including the pointwise mean squared error (MSE), pairwise logistic loss (PairLog in Eq. 6) and mean squared error (PairMSE):
$$
l _ {\text {P a i r M S E}} (\mathbf {y}, \mathbf {s}) = \sum_ {i \neq j} \left(\left(s _ {i} - s _ {j}\right) - \left(y _ {i} - y _ {j}\right)\right) ^ {2}, \tag {7}
$$
as well as the listwise Softmax (Eq. 7), Gumbel Approximate NDCG loss (GumbelNDCG) [2]:
$$
l _ {\text {G u m b e l N D C G}} (\mathbf {y}, \mathbf {s}) = - \frac {1}{\widetilde {\mathrm {D C G}}} \sum_ {i} \frac {2 ^ {y _ {i}} - 1}{\log_ {2} (1 + r _ {i})}, \tag {8}
$$
where  $\widetilde{DCG}$  is the ideal DCG metric,  $r_i$  is the approximate rank of document  $i$  given  $\mathbf{s}$  plus Gumbel noise, and LambdaLoss [13]:
$$
l _ {\text {L a m b d a L o s s}} (\mathbf {y}, \mathbf {s}) = - \sum_ {i \neq j} \mathbb {I} _ {y _ {i} > y _ {j}} \Delta_ {i j} \ln \frac {\exp \left(s _ {i} - s _ {j}\right)}{1 + \exp \left(s _ {i} - s _ {j}\right)}, \tag {9}
$$
where  $\Delta_{ij}$  is the LambdaWeight as defined in Eq.11 of [13].
4.6 Model Tuning
We fix  $l_{\mathrm{rel}}$  for all methods on each modality to focus on the distillation component (Q3). To study Q2, we find that the Softmax transformation is naturally order preserving and ensures positive labels, motivated by the classification setting [12] $^4$  and the RankDistil work [32]:
$$
g \left(f (\mathbf {x}; \theta^ {t})\right) = \frac {\exp \left(f (\mathbf {x} ; \theta^ {t}) / T\right)}{\sum_ {i} \exp \left(f \left(x _ {i} ; \theta^ {t}\right) / T\right)}, \tag {10}
$$
where  $T$  is the temperature. However, we note that some losses do not require positive labels (such as MSE), so we have the option to not use it (see more discussion in Sec. 5.5). We list the hyperparameter search space in Tbl. 2.
Table 2: Hyperparameter search space.
Data	Text	Tabular
l rel	Softmax	LambdaLoss
Optimizer	AdamW	Adagrad
Learning Rate	1e-4, 1e-5, 1e-6	1e-1, 1, 10
Batch Size (of Lists)	32	128
Training Steps	100K	200K
Sequence Length	128	n/a
Head Weight α	0, 0.25, 0.5, 0.75, 1
Label Softmax Transform	on, off
Temperature T	0.1, 1.0, 2.0, 5.0, 10.0
AdamW is the default optimizer for BERT. The key principle of parameter search space is, all methods share the same tuning budget (D5). The only exception is RD and RankDistill: since they have an additional hyperparameter  $K$ , we sweep  $K \in \{1, 5, 10\}$  and thus they have advantage in terms of tuning budgets.
All methods are implemented in the same open-source framework TF-Ranking [24]. All methods in the ranking loss family have already been implemented in TF-Ranking. We implemented RD due to its simplicity. We got the RankDistil implementation from the authors and had several communications to make sure of its correctness. Using open-source implementations encourage simplicity and fairness (D2, D5).
4Softmax transformation is over the list of classes for each item in classification and over the list of items in ranking.
Table 3: Ranking distillation results on the MSMARCO and NQ datasets.  $\hat{\uparrow}$  means significantly better result, performed against "Relevance Only" at the  $p < 0.01$  level using a two-tailed  $t$ -test. Best model is in boldface and second best is underlined for each metric (except for the teacher).
Models	MSMARCO	NQ
MRR@10	MRR	NDCG@1	NDCG@5	NDCG	MRR@10	MRR	NDCG@1	NDCG@5	NDCG
Teacher	43.63	44.46	29.91	46.84	54.22	60.08	60.36	46.95	63.89	66.98
Relevance Only	40.03	41.03	27.15	42.85	51.29	52.67	53.50	43.47	55.07	60.98
RD	40.25	41.25	27.51	42.92	51.45	57.09↑	57.64↑	46.72↑	59.80↑	64.50↑
RankDistil	41.29↑	42.21↑	28.27↑	44.04↑	52.28↑	55.04↑	55.80↑	45.71↑	57.53↑	62.86↑
MSE	42.06↑	42.96↑	28.54↑	45.12↑	52.96↑	58.49	58.95↑	47.51↑	61.43↑	65.64↑
PairLog	41.95↑	42.87↑	28.21	45.00↑	52.86↑	58.53↑	59.00↑	47.95↑	61.38↑	65.66↑
PairMSE	42.27↑	43.20↑	28.78↑	45.19↑	53.11↑	58.34↑	58.79↑	47.05↑	61.44↑	65.55↑
GumbelNDCG	41.85↑	42.76↑	28.32↑	44.81↑	52.75↑	57.90↑	58.39↑	47.26↑	60.70↑	65.14↑
Softmax	42.37↑	43.27↑	28.87↑	45.33↑	53.17↑	59.08↑	59.58↑	48.84↑	61.87↑	66.08↑
LambdaLoss	42.30↑	43.21↑	28.75↑	45.38↑	53.14↑	58.36↑	58.85↑	47.87↑	61.13↑	65.51↑
We report results and observations on the four tasks in RD-Suite. We use some of the most popular ranking metrics, i.e., NDCG and MRR, on the relevance labels. The numbers are timed by 100 which is common in the literature. The graded labels in Web30K and Istella are binarized at 3 when computing MRR that requires binary labels. We first discuss the results on each task specifically and summarize more findings on all tasks in Sec. 5.4. All evaluation scripts are made available to facilitate cross paper comparisons (Q3).
5.1 Results on Text Ranking Distillation (T1)