SlowGuess/NAACL_2022_OCR huggingface dataset:
Giambattista Amati, Claudio Carpineto, and Giovanni Romano. 2004. Query difficulty, robustness, and selective application of query expansion. In European conference on information retrieval, pages 127-137. Springer.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the trec 2020 deep learning track.
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In International Conference on Machine Learning, pages 1607-1616. PMLR.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Sebastian Hofstätter, Sophia Althammer, Michael Schroder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv preprint arXiv:2010.02666.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data.
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39-48.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A lite BERT for self-supervised learning of language representations. CoRR, abs/1909.11942.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. 2019. When Does Label Smoothing Help? Curran Associates Inc., Red Hook, NY, USA.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.
2016. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019. Multi-stage document ranking with bert.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826.
Xiao Wang, Craig Macdonald, Nicola Tonellotto, and Iadh Ounis. 2021. Pseudo-relevance feedback for multiple representation dense retrieval. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '21, page 297-306, New York, NY, USA. Association for Computing Machinery.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
Andrew Yates, Rodrigo Nogueira, and Jimmy Lin. 2021. Pretrained transformers for text ranking: Bert and beyond. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, pages 1154-1156.
A.1 Analysis on  $\mathcal{R}_q^*$  approximated by different teachers
Given the top-100 passages ranked by the pretrained ColBERT, i.e.,  $\Pi_q^{l\leq 100}$ , the goal of our teacher is to obtain better relevance boundary through collective knowledge, by which unlabeled positives can be discriminated from true negatives. To analyze the reliability of relevance boundary from the teacher, we evaluate how well the teacher classifies positives among the top-100 passages, using TREC 2019 that provides complete relevance labels.
Figure 1 shows trade-offs between precision and recall when applying different thresholds on  $\phi_q$ . We set different cut-offs on graded relevance for deciding positive passages. For example, in the left most figure with boundary 1, we treat passages with labeled relevance 1,2,3 as positives and 0 as negatives. As baselines, we compare our collective bi-encoder teacher with the pretrained ColBERT, and a cross-encoder teacher, monoBERT (Nogueira
4https://github.com/castorini/pyggle/
Performance Metric	\( f_p \)	\( f_c \)	\( f_e \)	β
1	3	5	12	24	5	10	0.5	1.0
NDCG@10	0.651	0.703	0.702	0.717	0.703	0.700	0.703	0.710	0.703
Recall@1k	0.843	0.842	0.849	0.846	0.842	0.844	0.842	0.848	0.842
Table 4: Ranking performance of our collective bi-encoder teacher under different configurations.
Figure 1: Precision-recall curves on TREC 2019 with different thresholds on predicted/labeled relevance scores. From the left to the right, we gradually increase the threshold of the labeled relevance that decides which passages are positive passages.
et al., 2019). Better precision-recall curves should bow towards the top right corner.
For the cut-offs of 1 and 2, we observed little differences between retrievers. For example, when retrievers are tasked to classify non-relevant passages with relevance label 0 and the others (the left most figure), all retrievers show  $100\%$  recall with near  $80\%$  precision. In contrast, for the cut-off of 3 (the rightmost figure), our teacher shows much better trade-offs compared to the others. For example, collective bi-encoder teacher shows near  $50\%$  precision to achieve  $10\%$  recall, while the others show  $20\%$  precision to achieve the same recall. The noticeable drop of precision for both ColBERT and cross-encoder in low  $(< 0.5)$  recall regimes indicate that these models have difficulty distinguishing moderate relevance (1, 2) from perfect relevance (3), that is, they are not well calibrated to distinguish between those two groups. On the other hand, our teacher's refinement of  $q$  through collective knowledge is effective in calibrating relevance, to finely reflect the distinction between perfectly relevant passages and the others.
A.2 Analysis on  $(f_p, f_c, f_e, \beta)$
Exploring ranking performance of students under all different configurations of  $(f_p, f_c, f_e, \beta)$  is expensive. Instead, the optimal configuration can be decided by ranking performance of the teacher. Here, we explore  $f_p \in \{1, 3, 5\}$ ,  $f_c \in \{12, 24\}$ ,  $f_e \in \{5, 10\}$ , and  $\beta \in \{0.5, 1.0\}$ . Meanwhile, ac
cording to reported results by (Wang et al., 2021),  $(f_{p} = 3,f_{c} = 24,f_{e} = 10,\beta \in \{0.5,1.0\})$  shows strong performance. Thus, instead of testing all different configurations, we set  $(f_p = 3,f_c = 24,f_e = 10,\beta = 1.0)$  as default values, and change one variable at a time. When evaluating teachers using TREC-DL 2019 dataset, We found similar results to (Wang et al., 2021) (Table 4).