SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tbl. 3 shows the ranking performance on MSMARCO and NQ datasets. We have the following observations: (1) The effectiveness of existing methods (RD and RankDistil) is verified. RankDistil significantly outperforms the Relevance Only baseline in both datasets while RD is effective on NQ. (2) However, the performance of RD and RankDistil are considerably worse compared with the ranking loss family. (3) The listwise Softmax loss tends to be the most robust method. It is encouraging to see some student models can even outperform the teacher on NQ NDCG@1.
5.2 Results on Distillation Transfer (T2 and T3)
Tbl. 4 shows the ranking performance on the NQ dataset when transferring knowledge from the teacher rankers trained on MSMARCO (the same MSMARCO teacher in T1). For NQ-Zeroshot, since the relevance labels are not available, the Relevance Only baseline is not applicable. RD is also not applicable since it has to use relevance labels (all labels are simply positive in the distillation objective).
We have the following observations: (1) Results on the distillation transfer task are interesting and encouraging: even if the teacher ranker does not perform well, all distillation methods can get significant help from it and outperform the Relevance Only baseline. Note that the "poor teacher helps distillation" effect has drawn some attention in the classification setting [42] but is not well understood for ranking problems. (2) Results on NQ-Zeroshot are expected - as no relevance labels are available for the target domain, the teacher model becomes the headroom. There is still considerate performance variance among different methods, showcasing their capabilities in a "distillation only" setting. (3) Existing methods (RD and RankDistil) are still not competitive. (4) Listwise approaches such as Softmax and GumbelNDCG tend to be effective.
5.3 Results on Tabular Ranking Distillation (T4)
Tbl. 5 shows the ranking performance on the Web30K and Istella datasets. Despite data modality and model configurations, one major difference is tabular LTR datasets have dense relevance labels - multiple documents with non-zero relevance, while for MSMARCO and NQ there is usually one labelled positive document for each query.
Table 4: Distillation transfer results on the NQ dataset, using Teacher model fine-tuned on MS-MARCO. NQ-Zeroshot means relevance label on NQ is not available.  $\uparrow$  means significantly better result, performed against "Relevance Only" at the  $p < 0.01$  level using a two-tailed  $t$ -test. Best model is in boldface and second best is underlined for each metric (except for the teacher).
Models	NQ	NQ-Zeroshot
MRR@10	MRR	NDCG@1	NDCG@5	NDCG	MRR@10	MRR	NDCG@1	NDCG@5	NDCG
Teacher	45.27	46.02	31.08	48.94	55.49	45.27	46.02	31.08	48.94	55.49
Relevance Only	52.67	53.50	43.47	55.07	60.98	n/a	n/a	n/a	n/a	n/a
RD	54.66\( \uparrow \)	55.25\( \uparrow \)	44.25	57.34\( \uparrow \)	62.57\( \uparrow \)	n/a	n/a	n/a	n/a	n/a
RankDistil	54.17\( \uparrow \)	54.95\( \uparrow \)	44.19	56.88\( \uparrow \)	62.28\( \uparrow \)	40.01	41.10	27.67	42.95	51.25
MSE	56.03\( \uparrow \)	56.64\( \uparrow \)	46.12\( \uparrow \)	58.66\( \uparrow \)	63.64\( \uparrow \)	43.20	44.09	29.49	46.76	53.84
PairLog	56.23\( \uparrow \)	56.88\( \uparrow \)	46.58\( \uparrow \)	58.76\( \uparrow \)	63.79\( \uparrow \)	43.37	44.25	29.72	46.84	53.94
PairMSE	56.25\( \uparrow \)	56.89\( \uparrow \)	46.58\( \uparrow \)	58.91\( \uparrow \)	63.84\( \uparrow \)	43.45	44.35	29.82	47.05	54.05
GumbelNDCG	56.03\( \uparrow \)	56.70\( \uparrow \)	46.43\( \uparrow \)	58.72\( \uparrow \)	63.62\( \uparrow \)	44.00	44.87	30.85	47.38	54.39
Softmax	56.72\( \uparrow \)	57.33\( \uparrow \)	47.09\( \uparrow \)	59.30\( \uparrow \)	64.16\( \uparrow \)	43.80	44.66	30.13	47.28	54.27
LambdaLoss	56.41\( \uparrow \)	57.06\( \uparrow \)	46.98\( \uparrow \)	58.80\( \uparrow \)	63.93\( \uparrow \)	43.78	44.61	30.22	47.26	54.23
We have the following observations: (1) There are generally fewer effective distillation methods on these tabular datasets than the text datasets, especially on Web30K. We hypothesize it is because of the dense relevance labels, so the benefits from teacher scores are less clear. (2) Listwise ranking losses such as Softmax and LambdaLoss are the most competitive.
Table 5: Ranking distillation results on the Web30K and Istella datasets.  $\dagger$  means significantly better result, performed against "Relevance Only" at the  $p < 0.01$  level using a two-tailed  $t$ -test. Best model is in boldface and second best is underlined for each metric (except for the teacher).
Models	Web30K	MRR@10	MRR	Istella
MRR@10	MRR	NDCG@1	NDCG@5	NDCG	NDCG@1	NDCG@5	NDCG
Teacher	34.43	35.05	48.39	47.33	71.62	84.54	84.59	71.69	67.92	81.69
Relevance Only	27.81	28.54	41.43	41.07	68.37	76.97	77.12	61.09	57.18	74.23
RD	27.44	28.19	41.17	40.92	68.32	77.01	77.16	61.36	57.35↑	74.32↑
RankDistil	27.82	28.58	41.47	41.03	68.33	77.15	77.30	61.61↑	57.41↑	74.36↑
MSE	27.44	28.19	41.37	40.92	68.31	77.31↑	77.46↑	61.89↑	57.52↑	74.41↑
PairLog	27.51	28.25	41.10	40.98	68.28	77.04	77.19	61.34	57.37↑	74.32↑
PairMSE	26.97	27.72	40.70	40.90	68.24	77.17	77.32	61.66↑	57.51↑	74.37↑
GumbelNDCG	28.31↑	29.05↑	41.79	41.29	68.32	77.44↑	77.63↑	62.96↑	57.53↑	74.06
Softmax	29.54↑	30.25↑	42.08	41.11	68.22	77.49↑	77.64↑	62.47↑	57.69↑	74.40↑
LambdaLoss	29.31↑	30.02↑	42.03	41.13	68.30	77.59↑	77.75↑	62.56↑	57.75↑	74.41↑
5.4 Overall Results
To get an overview of distillation methods, in Tbl. 6, we calculate their performance ranks on the 6 configurations, based on the NDCG@5 metric.
Table 6: The performance ranks of different distillation methods on the 6 task configurations. Smaller rank indicates better performance.
Model	Best Rank	Worst Rank	Mean Rank
Softmax	1	3	1.8
LambdaLoss	1	5	2.7
GumbelNDCG	1	6	3.7
PairMSE	1	8	3.8
MSE	3	6	4.8
PairLog	4	7	5.0
RankDistil	4	8	6.7
RD	6	8	7.2
There are several trends: first, listwise ranking losses (Softmax, LambdaLoss, GumbelNDCG) that consider the entire list of documents and teacher score values are most competitive. PairMSE and MSE being next shows the importance of teacher score values over PairLog. Existing methods (RD and RankDistil) are not competitive despite they are effective distillation methods in general (outperforming no distillation baseline in most cases.)
5.5 The Role of Softmax Transformation
To better understand Q2 (How to handle teacher ranker scores?), in experiments we have a switch on whether to apply softmax transformation and report best performing configurations in the results. Here we peek into if softmax label transformation is enabled for each method.
We examine the MSMARCO and NQ ranking distillation experiments (T1) and observe the same behavior: softmax label transformation was selected by GumbelNDCG, Softmax, and LambdaLoss, but was not selected for MSE and PairMSE. Note that PairLog and RD are indifferent to such order preserving transformation since they do not look at teacher score values, and Rankdistill has to use softmax transformation to sample distributions as done in the original paper.
Some observations are intuitive: cross-entropy based losses such as Softmax are ill-behaved with negative labels; GumbelNDCG and LambdaLoss consist of DCG computation that usually assumes positive labels. The behaviors of MSE and PairMSE are interesting - it indicates methods depending heavily on score values can be less effective with certain transformations. In fact, the performance gap is quite significant: the MRR@10 of MSE on NQ is 54.87 vs 58.49 with softmax transformation turned on and off, a difference between being highly competitive (3rd best result for that task) and not so. We provide more discussions in Appendix. Therefore, researchers should be cautious about label transformations on teacher ranker scores that have much more freedom that those in classification problems, and we provide more discussions in Sec. 6.
5.6 The Learning Dynamics
We draw the ranking performance on validation data during model training to gain insight in the learning dynamics in Fig. 1. We can see that when compared against relevance labels that we care most, the Relevance Only baseline saturates fast, while distillation denoted as Softmax (using Softmax as the distill loss) performance keeps increasing with the help of distillation. Distillation can also get good ranking performance against teacher scores, but it is interesting that it does not have to keep increasing performance on teacher scores to increase performance on relevance labels. Relevance Only model's behavior on teacher scores is also interesting: the performance can even get worse during training (note the Relevance Only model does not have access to teacher scores). Our understanding is this indicates some overfitting issues that can not be caught by relevance labels.
Figure 1: The performance of Relevance Only and Softmax on relevance label and teacher scores for the NQ ranking distillation task. NDCG is used since teacher scores are real-valued.
Relevance Only Softmax
6 Summary and Discussions
We provide a brief summary that answers the three questions we raised in Sec. 3. We then discuss topics that could be important for ranking distillation, but are not comprehensively covered in the current benchmark due to scope of this work.
Summary. For Q1: Where is the knowledge in ranking distillation, we performed comprehensive benchmark and found that the knowledge can exist in the score values from teacher rankers, challenging common assumptions made in the literature. We hope this can inspire future work in this direction. For Q2: How to handle teacher ranker scores, we showed the freedom of practical teacher rankers that is not well studied, and the effect of label transformations that were either ignored or implicitly considered in the literature. This problem also needs more future work considering the
importance of teacher score values in Q1. For Q3, we discussed drawbacks of existing efforts, and strove to provide a standardized criterion, by providing retrieved documents, teacher scores, student architectures, and the evaluation scripts.
The role of teacher ranker. We assume teacher rankers are given, which is the common setting in the literature, motivating us to release the teacher scores. However, it is still meaningful to investigate different teacher rankers and their effect in the distillation process. Some teacher ranker may not be state-of-the-art itself but may produce very competitive student ranker (as hinted in the distillation transfer experiments). Recently, there are studies around this in the classification setting [21].
The role of teacher label transformation. We formalized the importance of properly handling teacher ranker labels. We studied Softmax transformation motivated by traditional knowledge distillation and some existing work, but also argued that such transformation may not always be useful. An open question is if there exist better teacher label transformations.
What tasks need more attention? RD-Suite consists of a diverse set of tasks. However, we encourage researchers to focus more on the text ranking tasks (T1, T2, T3) for the following reasons: (1) The setting on tabular datasets can be useful from a research perspective but less from a practical perspective - MLP models may not be expensive to serve, and the community has yet to find ways to train large effective models on tabular datasets [29]. Other models such as Gradient Boosted Decision Trees (GBDTs) are also highly effective alternatives [30]. (2) Tabular datasets have dense relevance labels that may make distillation less useful. Also, having human annotated dense relevance labels is costly and not practical. (3) We can not study interesting topics such as distillation transfer since each tabular dataset has different manually designed numeric features.
On the other side, we believe the text ranking tasks are more realistic by using giant teacher models, allowing knowledge transfer, and benefiting more from distillation given sparse relevance labels. For future work, we are interested in extending RD-Suite to Large Language Model based rankers [28].
Ranking distillation has drawn much attention in the community from different perspectives. The most relevant existing work are RD [35] and RankDistil [32] that we discussed in the paper, focusing on general distillation objectives, that can be applied to any model architectures and ranking applications.
Some work focuses on architecture specific distillation or model compression. For example, [9] focused on BERT-based models and perform distillation on language model tasks (i.e., predictions on all tokens in the vocabulary). Both teacher and student rankers are BERT models, which is not very general. [41] distilled intermediate representations in BERT models, and simply assumed the teacher ranker was trained using pointwise loss and applied pointwise logistic loss for distillation. There is also a body of work on BERT specific distillation in general, not limited to ranking [33].
There is a suite of work on distillation for recommender systems that rank items for users, potentially to be served on mobile devices. These work either apply RD or RankDistil as a core component, or take a route specific to the recommendation problem. [14] was a parallel work as RankDistil and performed order preserving distillation. [15] studied student to teacher distillation while the distillation component followed RD. [34, 37] concerned about distillation of sequential recommender systems and also assumed simple teacher configurations and distillation methods (e.g., pointwise losses for both teacher and distillation). [6] mitigated popularity bias for distillation that is specific to recommender systems.
We proposed Ranking Distillation Suite (RD-Suite), a new benchmark for evaluating progress on ranking distillation research. Our new benchmark is challenging and probes at model capabilities in dealing with diverse data types and both regular ranking distillation and distillation transfer tasks. For the first time, we conduct an extensive side-by-side comparison of teachers, baseline without distillation, and eight distillation methods. The experimental results challenge common wisdom of prior art, show promises for certain methods, and leaves room for improvements. The benchmark also includes datasets with teacher labels and evaluation scripts to facilitate future benchmarking, research, and model development.
[35] Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2289-2298, 2018.
[36] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:2011.04006, 2020.
[37] Qinyong Wang, Hongzhi Yin, Tong Chen, Zi Huang, Hao Wang, Yanchang Zhao, and Nguyen Quoc Viet Hung. Next point-of-interest recommendation on resource-constrained mobile devices. In Proceedings of the Web conference 2020, pages 906-916, 2020.
[38] Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-37, 2019.
[39] Chen Xu, Quan Li, Junfeng Ge, Jinyang Gao, Xiaoyong Yang, Changhua Pei, Fei Sun, Jian Wu, Hanxiao Sun, and Wenwu Ou. Privileged features distillation at taobao recommendations. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2590–2598, 2020.
[40] Chenxiao Yang, Junwei Pan, Xiaofeng Gao, Tingyu Jiang, Dapeng Liu, and Guihai Chen. Cross-task knowledge distillation in multi-task recommendation. arXiv preprint arXiv:2202.09852, 2022.
[41] Shaowei Yao, Jiwei Tan, Xi Chen, Juhao Zhang, Xiaoyi Zeng, and Keping Yang. Reprbert: Distilling bert to an efficient representation-based relevance model for e-commerce. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4363-4371, 2022.
[42] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903-3911, 2020.
[43] Zhe Zhao, Lichan Hong, Li Wei, Jilin Chen, Aniruddh Nath, Shawn Andrews, Aditee Kumthekar, Maheswaran Sathiamoorthy, Xinyang Yi, and Ed Chi. Recommending what video to watch next: A multitask ranking system. In Proceedings of the 13th ACM Conference on Recommender Systems, page 43-51, 2019.
[44] Honglei Zhuang, Zhen Qin, Shuguang Han, Xuanhui Wang, Michael Bendersky, and Marc Najork. Ensemble distillation for bert-based ranking models. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, pages 131-136, 2021.
[45] Honglei Zhuang, Zhen Qin, Rolf Jagerman, Kai Hui, Ji Ma, Jing Lu, Jianmo Ni, Xuanhui Wang, and Michael Bendersky. Rankt5: Fine-tuning t5 for text ranking with ranking losses. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023.