SlowGuess/NAACL_2022_OCR huggingface dataset:
Compared with BM25+BERT-base, the further pretraining method BM25+BERT+FP improves MRR@10 by  $0.5\%$  on the MS MARCO and NDCG@20 by  $0.8\%$  on the Robust04 datasets for better capturing the domain-specific language patterns. By Jointly training the BERT-base model with SJTF framework during the fine-tuning stage, BERT+SJTF achieves a  $1.3\%$  improvement in terms of MRR@10 on the MS MARCO and  $1.1\%$  improvement of NDCG@20 on the Robust04. The experimental results suggest that learning the semantic dependencies of queries and relevant documents is more effective than further learning the contextual semantic information of documents. BERT+PACRR+SJTF outperforms the original approach by  $0.5\%$  on the MS MARCO dataset and  $1.3\%$  on the Robust04 dataset, which indicates that the performance of traditional reranking methods would benefit from an improved contextual representation. The results show that the models finetuned with the joint training framework yields a
Table 2: Performance comparison of all methods on the MS MARCO Passage Ranking and Robust04. '-' means that the PARADE method is not applicable because PARADE aggregates information from multiple overlapping passages. Significant paired t-test  $(p < 0.05)$  improvements over the original models are marked with  $\dagger$ .
MS MARCO Passage Ranking	TREC Robust04
Method	MRR@10(Dev)	MRR@10(Eval)	P@20	NDCG@20
BM25+BERT	0.345	0.342	0.4042	0.4541
BM25+BERT+FP	0.355	0.347	0.4024	0.4624
BM25+BERT+SJTF	0.362†	0.355	0.4291†	0.4657†
BERT+PACRR	0.328	0.326	0.3969	0.4599
BERT+PACRR+SJTF	0.333†	0.331	0.4122†	0.4735†
PARADE	-	-	0.4604	0.5399
PARADE+SJTF	-	-	0.4701†	0.5455
CEDR-KNRM	0.337	0.331	0.4729	0.5388
CEDR-KNRM+SJTF	0.347†	0.337	0.4851†	0.5565†
better interaction-based representation based on the semantic relationships between the query and candidate passages, which leads to a significant performance improvement without the model structure change.
The improvement of PARADE by splitting a long document into overlapping passages via sliding windows is less, increased by  $0.55\%$  on Robust04 datasets, suggesting that the semantic blocks in long documents are not all tightly semantically associated with queries and using these to predict masked query tokens only slightly enhances representation learning. In comparison with the results of PARADE, CEDR has been significantly improved by applying the SJTF framework, which indicates that the document-level semantic relations captured by the SJTF framework are beneficial for representation learning. The overall experimental results verify the effectiveness and generality of the training framework of encouraging the models to adopt the positive passage as the semantically consistent context for a given query.
Inspired by the observations from previous works (Gu et al., 2020; Liu et al., 2019; Gururangan et al., 2020) that different masking strategies enable a model to learn various perspective knowledge during pre-training stage, we design three different self-supervised methods in terms of focusing on understanding the content of query or passages and using the query or passage as the context for masking prediction task.
Similar to the traditional MLM approach, we randomly mask a token of a positive document and
Table 3: The performance of BERT-base and CEDR-KNRM models by integrating different masking strategies on Robust04 dataset.
Method	NDCG@20	P@20
BM25+BERT	0.4541	0.4042
+ MPP	0.4539	0.4026
+ MQPALL	0.4567	0.3979
+ MQP	0.4657	0.4291
CEDR-KNRM	0.5388	0.4729
+ MPP	0.5433	0.4738
+ MQPALL	0.5464	0.4784
+ MQP	0.5565	0.4851
use a corresponding query and the remaining documents as contexts to predict the masked token, and this strategy is denoted as MPP. The aim of MPP is to encourage the model to learn task-specific language patterns and understand that the corresponding query and the positive documents are semantically consistent. However, it is possible to reconstruct the masked token based on the content of document itself without demanding the semantic information of the query. In addition, since MPP is a simple task for pre-trained models, it cannot effectively improve the ranking performance of models.
Considering that the documents retrieved by BM25 are relevant or partially relevant to the query at the word level, the second strategy is to treat each retrieved document (regardless of positive and negative documents) as the context of the masked query for the MQP task, which does not utilize the supervised signal and is denoted as  $\mathrm{MQP}_{ALL}$ . In contrast to the first strategy,  $\mathrm{MQP}_{ALL}$  allows the
models to focus on understanding the content of query and capture the semantic relationships between queries and documents. Table 3 illustrates the slight improvement achieved by  $\mathrm{MQP}_{ALL}$  compared to the original model and the first strategy.
The self-supervised method MQP proposed in this paper is based on the assumption that although the documents retrieved by BM25 have word-level relevance to the query, the documents containing the query terms are not always relevant. Therefore, we use supervised signals to consider only positive documents as the context of masked queries, since the semantic information of negative documents is equal to noise for the queries. As shown in table 3, after filtering out negative documents by supervised signals, BERT-base and CEDR-KNRM achieve a significant improvement on NDCG@20 and P@20 metrics. Experimental results illustrate that the reasonable use of label informations to construct a self-supervised approach enables the models to obtain more accurate knowledges of the downstream task, which helps the models to identify correct documents for each question from a large collection.
4.7 Hyper-parameter Study
As a hyper-parameter described in Equation (6), the setting of  $\alpha$  value controls the influence of the self-supervised approach on the representation learning of models. Figure 3 shows the result of different hyper-parameters setting on the Robust04 dataset. In general, a larger value  $\alpha$  allows an excessive effect of the self-supervised method, which reduces the influence of the reranking task in optimizing models while achieving sub-optimal performance. Therefore, with  $\alpha$  value greater than 0.05, the ranking performance of all three methods show a decreasing trend. For the CEDR method, a stable performance improvement is obtained using the proposed training framework, indicating that considering the positive documents as the contextual features of the query can significantly enhance the representation learning of the CEDR model. By gradually raising the  $\alpha$  value from 0.01 to 0.05, the performance of both BERT-base and BERT+PACRR also gradually achieves the highest NDCG@20, which suggests that encouraging these models to focus on query understanding can lead to better reranking performance. From Figure 3, it can be seen that the  $\alpha$  value of 0.05 is more robust for most of the models.
Figure 3: Performance of various rerankers with different configurations of  $\alpha$  on the Robust04 dataset using the metric nDCG@20.
This paper proposes a self-supervised joint training framework SJTF to improve the representation learning for document ranking. By integrating with this training framework, models are able to better understand the content of queries and capture the semantic relation between queries and positive documents, which guides the models to identify the relevant documents among a large number of candidate documents. Experimental results show that our proposed framework enhances the representation learning of reranking models and achieves better performance compared with baselines in the document reranking task.
We thank all anonymous reviewers for their insightful comments to import this paper. This work is supported by Natural Science Foundation of Guangdong Province (No. 2021A1515011339) and Guangzhou Key Laboratory of Big Data and Intelligent Education (No. 201905010009).
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018,
Marina Del Rey, CA, USA, February 5-9, 2018, pages 126-134. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Hengxin Fun, Sunil Gandhi, and Sujith Ravi. 2021. Efficient retrieval optimized multi-task learning. arXiv preprint arXiv:2104.10129.
Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train no evil: Selective masking for task-guided pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966-6974, Online. Association for Computational Linguistics.
Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016, pages 55-64. ACM.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. 2019. Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 15637-15648.
Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2018. Co-pacrr: A context-aware neural IR model for ad-hoc retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 279-287. ACM.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and
Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020. Parade: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.09093.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: contextualized embeddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1101-1104. ACM.
Jean Maillard, Vladimir Karpukhin, Fabio Petroni, Wen-tau Yih, Barlas Oguz, Veselin Stoyanov, and Gargi Ghosh. 2021. Multi-task retrieval for knowledge-intensive tasks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1098-1111, Online. Association for Computational Linguistics.
Bhaskar Mitra and Nick Craswell. 2019. An updated duet model for passage re-ranking. CoRR, abs/1903.07666.
Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018. Did the model understand the question? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1896-1906, Melbourne, Australia. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPs.
Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. CoRR, abs/1901.04085.
Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. 2019a. Multi-stage document ranking with BERT. CoRR, abs/1910.14424.
Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019b. Document expansion by query prediction. CoRR, abs/1904.08375.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835-5847, Online. Association for Computational Linguistics.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-389.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to fine-tune bert for text classification? In China National Conference on Chinese Computational Linguistics, pages 194-206. Springer.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Ellen M Voorhees. Overview of the trec 2004 robust retrieval track.
Shuohang Wang, Yuwei Fang, Siqi Sun, Zhe Gan, Yu Cheng, Jingjing Liu, and Jing Jiang. 2020. Cross-thought for sentence encoder pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 412-421, Online. Association for Computational Linguistics.
Ming Yan, Chenliang Li, Bin Bi, Wei Wang, and Songfang Huang. 2021. A unified pretraining framework for passage ranking and expansion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4555-4563.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Self-supervised contrastive cross-modality representation learning for spoken question answering. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 28-39.
Xiaoming Zhang, Mingming Meng, Xiaoling Sun, and Yu Bai. 2019. Factqa: question answering over domain knowledge graph based on two-level query expansion. Data Technologies and Applications.