SlowGuess/NAACL_2022_OCR huggingface dataset:
on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations (ICLR).
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia. Association for Computational Linguistics.
Dan Iter, Kelvin Guu, Larry Lansing, and Dan Jurafsky. 2020. Pretraining with contrastive sentence objectives improves discourse performance of language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4859-4870, Online. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, et al. 2021. Quality: Question answering with long input texts, yes! arXiv preprint arXiv:2112.08608.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big Bird: Transformers for longer sequences. Advances in Neural Information Processing Systems (NeurIPS).
A Question-Evidence Similarity Demonstration
In this section, we further interpret the outcome of our contrastive loss.
We apply PCA over the relevant normalized token representations of the validation data of HotpotQA (i.e., the question and answers representations in Fig. 1), and depicted them in 2. The projected representations of the correct and wrong answers are equally distributed at the beginning of the training (left figure). After several epochs when the model converged (right figure), the answer representations' manifold got closer to the questions' representations (in terms of radial distance). Each beam in the figure corresponds to a different question type (there are 3 in HotpotQA). The correct evidence representations (green dots) are the closest among the whole answer representations, confirming that our additional contrastive loss term generalizes and maximizes the question-evidence similarity.
B Datasets and Finetuning Details
In this section, we provide details, regrading finetuning and hyper-parameter configuration, over the benchmarks we used during our experiments.
B.1 QAsper
Since some of the questions included in QAsper are not answerable, we apply our contrastive loss only over examples that are answerable and contain at least one evidence sentence.
We train all models using the Adam optimizer (Kingma and Ba, 2014) and a triangular learning rate scheduler (Howard and Ruder, 2018) with  $10\%$  warmup. To determine number of epochs, peak learning rate, and batch size, we performed manual hyperparameter search on a subset of the training data. We searched over  $\{1,3,5\}$  epochs with learning rates  $\{1e^{-5},3e^{-5},5e^{-5},9e^{-5}\}$ , and found that smaller batch sizes generally work better than larger ones. Our final configuration was 10 epochs, peak learning rate of  $5e^{-5}$ , and batch size of 2, which we used for all reported experimental settings. When handling full text, we use gradient checkpointing (Chen et al., 2016) to reduce memory consumption. We run our experiments on a single RTX 8000 GPU, and each experiment takes 30-60 minutes per epoch.
B.2 HotpotQA
We used the HotpotQA-distractor dataset (Yang et al., 2018). Each example in the dataset is includes a question and 10 paragraphs from different documents, extracted from Wikipedia. Two gold paragraphs include the relevant information for properly answering the question, mixed and shuffled with eight distractor paragraphs (for the full dataset statistics, see Yang et al. (2018)). There are two goals for this task: detecting the supporting facts, i.e., evidence sentences, as well as extraction of the correct answer span.
For preparing the data for training and evaluation, we follow the same finetuning scheme of the CDLM (Caciularu et al., 2021) and the Longformer (Beltagy et al., 2020); For each example, we concatenate the question and all the 10 paragraphs in one long context. We particularly use the following input format with special tokens and our document separators: "[CLS] [q] question [/q]  $\langle \mathrm{doc - s}\rangle \langle \mathrm{t}\rangle$  title1  $\langle /t\rangle \langle s\rangle$  sent1,1  $\langle /s\rangle$ $\langle s\rangle$  sent1,2  $\langle /s\rangle \langle /doc - s\rangle$  ...  $\langle t\rangle$ $\langle \mathrm{doc - s}\rangle$  title2  $\langle /t\rangle$  sent2,1  $\langle /s\rangle \langle s\rangle$  sent2,2  $\langle /s\rangle \langle s\rangle$  ...” where  $[q]$ ,  $[\mathbf{\mu} / q]$ ,  $\langle t\rangle$ ,  $\langle /t\rangle$ ,  $\langle s\rangle$ ,  $[p]$  are special tokens representing, question start and end, paragraph title start and end, and sentence start and end, respectively. The new special tokens were added to the models vocabulary and randomly initialized before task finetuning. We use global attention to question tokens, paragraph title start tokens as well as sentence tokens. The model's structure is taken from Beltagy et al. (2020).
As in Beltagy et al. (2020); Caciularu et al. (2021), we finetune our models for 5 epochs, using a batch size of 32, learning rate of  $1e^{-4}$ , 100 warmup steps. Finetuning on our models took  $\sim 6$  hours per epoch, using four 48GB RTX8000 GPUs for finetuning our models. For generating the CDLM-large results, we pretrained our version using the code from https://github.com/aviclu/CDLM/tree/main/pretraining.
C Contrastive Loss Details
In this section, we provide the details for reproducing our contrastive term, which is relevant for both QAsper and HotpotQA.
We searched over  $d \times \{d, \frac{d}{2}, \frac{d}{4}, \frac{d}{8}\}$  to determine the linear projections' dimensions, where  $d$  is the model's hidden layer representation dimension (it
Figure 2: PCA plots of the learned question and answer token embeddings on the HotpotQA validation set, comparing early training epochs results (left) and results after model convergence (right). The wrong evidence representations correspond to both wrong evidences, or correct evidences using the wrong question type projections (our soft negatives).
depends on the size of the model). In order to determine the temperature hyperparameter  $\tau$ , we searched over  $\{0.2, 0.4, 0.6, 0.8, 1.0\}$  per question type (if applicable). We also applied dropout with a rate of  $p = 0.1$  over the linear projections, which consistently improved the results over all the benchmarks. Finally, we searched for the best performing  $\lambda$  hyperparameter over the values of  $\{0.2, 0.4, 0.6, 0.8, 1.0\}$ .