DocSplit: Simple Contrastive Pretraining for Large Document Embeddings
Authors: Yujie Wang
Mike Izbicki
Venue: EMNLP 2023 Findings (2023)
Keywords: Natural Language Processing; Machine Learning;Text Embeddings

Abstract: Existing model pretraining methods only consider local information. For example, in the popular token masking strategy, the words closer to the masked token are more important for prediction than words far away. This results in pretrained models that generate high-quality sentence embeddings, but low-quality embeddings for large documents. We propose a new pretraining method called DocSplit which forces models to consider the entire global context of a large document. Our method uses a contrastive loss where the positive examples are randomly sampled sections of the input document, and negative examples are randomly sampled sections of unrelated documents. Like previous pretraining methods, DocSplit is fully unsupervised, easy to implement, and can be used to pretrain any model architecture. Our experiments show that DocSplit outperforms other pretraining methods for document classification, few shot learning, and information retrieval tasks.

DOCSPLIT: Simple Contrastive Pretraining for Large Document Embeddings
Yujie Wang Claremont Graduate University yujie. wang@cgu.cc...
Mike Izbicki Claremont McKenna College
Michael www..cccieeaaaaeeonnttoooo edu
Abstract
original training document (x)
Deep leerning is par of bonader family of mastine leamnin methods based antficial neural neoworks with represemation learning. Amtiicaal communicaton
Existing model pretraining methods only con- sider local information. For example, in the popular token masking strategy, the words closer to the masked token are more important for prediction than words away. This results in pretrained models that generate high-quality sentence embeddings, but low-quality embed- dings large documents. We propose a new pretraining method called DOCSPLIT which forces models to consider the entire global con- text of a large document. Our method uses contrastive loss where the positive examples are randomly sampled sections of the input docu- ment, and negative examples randomly sam- pled sections of unrelated documents. Like pre- vious pretraining methods, DOCSPLIT is fully unsupervised, easy to implement, and can be used to pretrain any model architecture. Our experiments show that DoCSPLIT outperforms other pretraining methods for document clas- sification, few shot learning, and document re- trieval tasks.
netwoeks unspiedd infermation hiologieal systems, Artificial
and ibuted
differences boological braiss, Specifically. artificll netwirks he ststic symbolicc wheress hislogical hnin mool living organiimms is dynamic and atalog.
document summary (ag))
document summary (m2
Deep leannnng is of brooder fimily of mushine lesrning mathous based antificial neural networks reprosetatiom laaming networks vanous differences from Nillgical beains.
Antificial neura aetwocks were in- spinal by informsiion pricessing distributed communicaiion nodes bielugical systems. Spucificalll, neural
sostic symboolic, wheneas ological brain of most living organ-
juus is dysaaii and analog.
Figure DOCSPLIT pretraining, an input document (x) is split into two new document summaries (xf and x2 by randomly assigning each sentence to one of the new summaries. These new summaries are then used positive instances for contrastive learning, which forces the model to represent these summaries with similar embeddings.
illustration of the method, and Section 2 describes the implementation details and intuition.
1 Introduction
Section 3 describes how DOCSPI.IT improves prior work. We begin by demonstrating that prior work on contrastive pretraining has focused only on sentence-level representations. Then we show that work on model architectures designed for large documents has ignored the problem of better pre- training methods designed to work with these chitectures.
Generating high-quality text embeddings for doc- uments is long-standing open problem. Most previous studies focus on either learning sentence- level representations (Hill et al., 2016; Logeswaran and Lee, 2018; Gao et al., 2021) where training data usually contain short text designing spe- cific model structures for larger-range dependen- cies (Beltagy et al., 2020; Zaheer et al., 2020), but effective and efficient document representation learning methods are less explored.
In Section 4, we describe two pretrained models based on the BERT architecture (for fair compari- son to other contrastive losses) and the LongFormer architecture (for optimal performance on large doc- uments). We evaluate these models standard large document classification, few shot learning. and document retrieval tasks. We find that models pretrained using DOCSPL.T significantly outper- form models pretrained with all other published methods.
This paper presents DOCSPLIT, the first unsu- pervised pretraining method designed specifically for large documents. DOCSPLIT is simple to use, and it be applied to any model architecture improve document representations. DoCSPLIT uses contrastive learning, and our key contribution is a new method for generating positive samples for contrastive learning. Figure provides graphical

---

2 Method
3 Related Work
There are two categories of related work: mod- els trained using contrastive learning, and models designed for large documents.
We first review contrastive learning high level. then we describe our DOCSPLIT method.
2.1 Contrastive Learning
3.1 Contrastive Pretraining
Contrastive Learning learns effective representa- tions by pulling semantically close neighbors to- gether and pushing apart non-neighbors in the latent space (Hadsell al., 2006). Each data point x is converted into a contrastive instance {z] 2. 21 .i.....Ty!! that includes two positive examples and negative examples. Intuitively, the positive instances should be semantically sim- ilar to each other (they will be pushed together) and the negative instances should be semantically dissimilar to the positive instances (they will be pushed away). Following recent influential work (Chen et al., 2020; Gao et al., 2021; Li et al., 2022), we use cross entropy as our loss function. As the mathematical details are technical and not relevant to our discussion, we do not reproduce them here.
Contrastive learning has shown remarkable recent success for developing sentence embeddings. The simplest method is SimCSE (Gao al., 2021), which uses dropout generate correlated positive samples. The contrastive tension method (Carlsson et al., 2021) is similarly generic, but has a much more complicated implementation involving multi- ple models trained jointly. Because both of these pretraining strategies are generic, they can be used with any type of input including documents; but they do not take explicit advantage of document structure. The INSTRUCTOR model (Su al., 2022) is the current state-of-ther model for most downstream tasks. The contrastive objective for this model requires a specially constructed corpus of manual-human annotations, and this corpus is limited only to sentence-level annotations instead of document-level annotations. We show that our model significantly improves INSTRUCTOR on document level tasks, and it is not clear how to tend the INSTRUCTOR model to document-level tasks because human annotation for documents is significantly more expensive than for sentences.
There are many model training methods that use contrastive learning, and what differentiates these methods is how they convert the data point æ into the contrastive instance. It is standard practice to use in-batch instances for the negative pairs, and all that remains is designing method for con- structing the positive pairs (at.2.))
The Contriever model (Izacard et al., 2021) uses contrastive objective most similar to our own. They use the document cropping and inverse cloze tasks for pretraining. In document cropping. doc- ument is divided in half and the two halves are used as the positive samples; in inverse cloze, a contiguous substring of the document used as one positive sample and other strings are used as the negative sample. The DoCSPLIT pretraining method can be seen a generalization of these methods.
2.2 Document Splitting (DoCSPLIT)
Our main contribution is the DOCSPI.IT method to construct the positive pairs needed for contrastive learning. The idea is simple describe and im- plement. Given an input document, first split the document into sentences. Then each sentence is randomly assigned to either the ot or a:t docu- ment.
This splitting procedure results in two docu- ments that can be thought as "summaries" of the original document. These summaries will have similar semantic content, and the contrastive loss will ensure that these two documents have similar embeddings. Notice that because there is random- ness in the document splitting procedure, multiple passes over the same data points will result in dif- ferent contrastive instance pairs being generated. This provides built-in overfitting resistance through data augmentation when multiple epochs of the training data are used.
3.2 Large Document Architectures
All of the models discussed in Section 3.1 above are based off of the BERT architecture (Devlin et al., 2019). This architecture uses an attention mechanism that requires (nn)) memory and run- time, where is the size of the attention window. The maximum size of document that a model can understand limited by this window size, and so compute for these models scales quadratically with the length of the documents.

---

A growing body of research focuses on develop- ing new architectures with reduced computational requirements that enable processing larger docu- ments. The LongFormer (Beltagy et al., 2020) and BigBird (Zaheer et al., 2020) models pioneered this line of research, and both models reduce the run- time of the attention mechanism O(n). A variety of other architectures have subsequently been pro- posed (e.g. ?????) ? provide survey of this large body of work. Importantly, all of this research fo- cuses only on improving the computational aspects of model architecture, and none of these models use a training objective designed specifically for large documents. Because the DoCSPLIT pretrain- ing method is model agnostic, we can easily apply it to any of these newly proposed model architec- tures. For computational reasons, we limit our experimental comparisons in Section below to the LongFormer and BigBird models since these are the two most influential model architectures de- signed for large documents. We find a large perfor- mance improvement when these models are trained with DOCSPLIT, and expect this performance im- provement would extend to similar models as well.
To pretrain both models, we follow the standard pretraining procedure for contrastive losses estab- lished by Gao et al. (2021) and Li et al. (2022). We simultaneously optimize both the masked lan- guage model (MLM) loss (with weight= 0.1) and the contrastive loss (with temperature T = 0.05). We use English Wikipedia articles our pretrain- ing dataset. These articles are long, and so we expect that pretraining procedure designed for large documents will improve performance. The total number of training instances is 6,218,225. We use AdamW (Kingma and Ba, 2014) with a learn- ing rate of 5e-5. DOCSPLIT uses a batch size of 36. And due to the larger memory requirements of the LongFormer architecture, DocSPL.T uses batchsize of 12. For both models, we know that performance improvements downstream tasks must be due the pretraining procedure and not the dataset because all baseline models include En- glish language wikipedia in their training set.
4.2 Experiment 1: Text Classification
We fine tune DoCSPLIT DoCSPLIThong and all baseline models on five standard document datasets. The datasets are summarized in the ta- ble below:
4 Experiments
We perform careful ablation study to isolate the effects of the DOCSPI.IT pretraining method on downstream task performance. First, we pretrain separate models for each group of baseline models described in Section above. Then, we perform downstream experiments on standard classification, shot learning, and document retrieval tasks. all cases, our pretrained models significantly outperform prior work.
Dataset
Num Does
Classes
Words Doc
Mean
Max
FakeNews
8,558,957
15
467
33,936
arXiv
2,162,833
38
138
925
20News
18,846
20
258
11,554
NYT
13,081
5
650
5,503
BBCNews
2,225
5
133
445
Table 1 shows the accuracy and Fl score of every model on these datasets. Notice that DoCSPLITbert out performs all BERT- based models discussed in Section 3.1, and DOCSPLIT jong outperforms all models on every dataset. There are no results for INSTRUCTOR on the 20News dataset because INSTRUCTOR was pretrained on 20News, and the authors state that evaluating INSTRUCTOR on datasets it was pre- trained is incorrect due to data contamination.
4.1 Pretraining Details
We pretrain two models two different architec- tures. All prior work using contrastive learning discussed in Section 3.1 above evaluates their pre- training methods on the BERT architecture. To fairly compare against these methods, we pretrain our DoCSPLITbet model also on this architecture. Ultimately, however, we are interested in large document performance, and so we expect that a model architecture designed specifically for large documents will improve performance. We there- fore also pretrain the DOCSPLITions model on the LongFormer architecture. This second model will be used to evaluate against other models designed specifically for large documents.
CCiattions for the datasets are: Fake News Corpus ht //github. ctctcuub.coo/// com/ wwwpsseeeeeeeeeeee wtwrae/2//Eaaeeeaaoopprsr arXiv articles datasct https: http::/wwwww.ggggkk kaggle. wtaetss/oooeeellnieeeeeeeeeee
com/ 20NewsGroups (Lang. 1995); New York Times An- notated Corpus (NYT) (Sandhaus, 2008); and BBC-
News http::/mlmg aed, Le/ datastss /bbc. html.

---

Table 1: In the text classification Experiment models pretrained with DOCSPLIT outperform baseline models in all cases. Larger numbers are better.
DocSolt
Bigerd
Longformo
Accurcy
RSTRUCTIR
40
30
BENT
CT BERT
SancS:
20
SmCCE_Jong
CTOCKT
Number of Samples per Class
Accuracy
Figure 2: DOCSPLITbert outperforms all other BERT- based models in a few-shot classification task on the 20News dataset.
Figure In the standard LRA document retrieval task, DOCSPI.TTert outperforms all other BERT-hased mad- els, and DoCSPLITiong outperforms all other models.
4.3 Experiment 2: Few-shot Learning
Conclusion
Next we evaluate how DOCSPLIT pretraining per- forms on classification tasks with small number of training examples. We follow the standard proce- dure of artificially limiting the number of training examples used during training, and evaluating on the same test set. Figure 2 shows the classification accuracy the 20News dataset of the training set. We that DOCSPLITbert out- performs other BERT-based models accross all sample sizes.
DOCSPLIT is the first unsupervised pretraining method designed explieitly for large documents. DOCSPLIT can be used to pretrain any model ar- chitecture, and we provide code to do so at
https://bliiddd_oorrre
vary the size
The LongFFormer-based DoCSPLITTogg pro- vides SOTA performance on document classifica- tion, few-shot learning, and retrieval tasks.
Limitations
The results on other datasets and for LongForrer-basee based models are similar. A full set of results other datasets is available in the Appendix.
We identify three limitations with our work. Limitation 1: We evaluate DocSPLIT using only three downstream tasks (classification, few- shot learning, and document retrieval), but pre- trained models are useful in many other tasks well. For example, the LongFormer (Beltagy et al.. 2020) is evaluated also question answering and coreference resolution. We did run these ex- periments only due lack of computational re- sources, and we hope that future work will evalu- ate the performance of DOCSPLIT pretraining on other downstream tasks as well. We note, however, that our DoCSPLIT experiments in some ways are more extensive than other papers' experiments. For example, the LongFormer paper does not include any document retrieval experiments, and we do. Also, the Contriever model (Izacard et al., 2021) was specifically designed for pretraining models for the downstream retrieval experiment task and only evaluates on this downstream task. We beat
4.4 Experiment 3: Document Retrieval
We follow the document retrieval experiment of the Long Range Arena (LRA) benchmark (Tay al., 2020), which uses the ACL Anthology Net- work (AAN) dataset (Radev et al., 2009). The goal of the task to predict whether two papers have a citation link given only their embeddings, which is a common setup used in long-form doc- ument matching (Jiang et al., 2019; Yang et al., 2020). This reduces to binary classification task and accuracy the commonly reported per- formance measure. Figure 3 shows the results. DoCSPLITher outperforms all other BERT-based models, and DOCSPLIT) outperforms all other models.

---

the Contriever model on this task and provide the classification and few-shot experiments in addition. Much of the prior work on contrastive learning evaluates on a large number of tasks as well. For example, INSTRUCTOR (Su et al., 2022) is eval- uated on 70 different downstream tasks. None of these tasks, however, are applicable in our setting because these tasks include datasets with only small text and not large documents.
abs/2004.05150.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evan- gelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In ICLR.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709.
Limitation 2: Our models are pretrained only on English wikipedia, but other models are pre- trained on significantly larger datasets. For exam- ple, the LongFormer model was pretrained on En- glish wikipedia and on the Books (?), Real News (?), and Stories (?) datasets. We pretrained on this smaller dataset only due to our limited computa- tional resources. Results in scaling laws (?) suggest that pretraining on more data would significantly improve our models' performance, and so our re- sults are reported results are likely underestimating the positive effects of DOCSPLIT pretraining. Limitation 3: Our experiments do not evalu- ate how well DOCSPLIT will work on extremely large (e.g. book length) documents. Existing large document model architectures like LongFormer (Beltagy et al., 2020) and BigBird (Zaheer al., 2020) are still efficient enough to train models on these extremely large documents. More work also needs to be done on generating suitable eval- uation tasks in this extreme setting to understand DoCSPLIT performance.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. ArXiv, (bs/2104.08821..
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning invariant mapping. 2006 IEEE Computer Society Confer- ence on Computer Vision and Pattern Recognition (CVPR'0R) 06), 2:1735-1742.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In NAACL.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Se- bastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning. Trans. Mach. Learn. Res., 2022.
Jyun- Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Mare Najork. 2019. Semantic text matching for long-form documents. The World Wide Web Conference.
Ethics Statement
Learning embeddings is a standard problem in natural language processing. Our approach uses standard training datasets and training procedures. There therefore direct ethical concerns with this research.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR. abs/1412.6998
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In ICML.
Our total compute is relatively small and so we have small environmental impact. Pre- training Doc 4 days on an Nvidia 2080 GPU with 1IGB of RAM, and pretraining DoCSPLIThnng took 4 days with an Nvidia Quadro RTX 8000 with 48GB of RAM. Other models use larger GPU clusters that require considerably more energy. The finetuning procedures in less than day on the same systems.
Jiacheng Li, Jingbo Shang, and Julian McAuley. 2022. UCTopie: Unsupervised contrastive learning for phrase representations and topic mining. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pu- pers), pages 61 1159961166 Dublin, Ireland. Association for Computational Linguistics.
Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representa- tions. ArXiv, abs/1803.00899
References
Dragomir R. Radev, Pradeep Muthukrishman, Vahed
Qazvinian, and Amjad Abu-Jbara. 2009. The acl anthology network corpus. Language Resources and Evaluation, 47:919-944
Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv,

---

Evan Sandhaus. 2008. The new york times annotated corpus. Linguistic Data Consortium, Philadelphia, 6(12):e26752.
Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu. Mari Ostendorf, Wen tau Yih, Noah Smith, Luke Zettlemoyer, and Tao Yu. 2022. One embedder, any task: dnstruction-finetunee text em- beddings. ArXiv, abs/2212.09741.
Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2020. Long range arena: benchmark for efficient transformers. ArXiv, abs/201 .04006.
Liu Yang, Mingyang Zhang, Cheng Li, Michael Ben- dersky, and Mare Najork. 2020. Beyond 512 to- kens: Siamese multi-depth transformer-based hier- archical encoder for document matching. ArXiv, 12s92004.12297..
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. ArXiv, 0bs/2007.14062.

---

Datasets
FakeNews
20News
arXiv
NYT
BBCNews
Metries
Acc
FI
Acc
FI
Acc
FI
Acc
FI
Acc
FI
Few-shot Text Classification
BERT
23.96
23.73
19.94
18.71
24.08
10.14
$1.85
43,90
54.22
52.73
CT-BERT
23.71
23.06
24.11
23.53
27.02
13.53
47.23
36.83
59.56
58.95
SimCSE
25.04
22.68
42.63
41.42
32.61
17.19
86.51
78.41
83.56
83.75
SimCSEnong
26.39
23.26
48.65
47.81
23.42
12.66
85.36
75.90
84.44
83.96
Contriever
24.17
22.35
54.27
54.87
33.19
17.23
84.73
77.62
77.71
79.19
INSTRUCTTR
26.11
23.97
33.51
17.12
32.38
27.60
52.67
48.78
DOCSPLITbert
27.79
24.65
55.79
55.43
35.79
18.52
90.52
83.71
86,86
86.31
LongFormer
26.56
25.12
44.42
42.41
25.04
13.36
73.06
54.87
84.89
BigBird
25.36
23.28
39.14
39.06
23.62
10.18
86.66
78.96
79.11
76.63
DOCSPLITIous
29.17
27.13
51.18
50.96
34.33
18.80
89.78
82.88
86.78
86.66
Table Few-shot classification five datasets.
A
Appendix
A.1
Training Details
For text classification, the learning rate for fine- tuning is 3e-4; the batch size is 8; the maximum sequence length is 512 tokens. We fine-tune the last MLP layer these five datasets and evaluate the classification performance with accuracy and macro-Fil scores. For few-shot text classification, we sample 10 data instances per class for the Fak- eNewsCorpus dataset and the arXiv dataset and data instances per class for the other three datasets. Other settings are the same as the standard text clas- sification. Since there is randomness in sampling, we repeat every experiment 10 times and take the average value of metrics.
A.2 Few-shot Learning
Table 2 shows the results of few-shot text classification on these five datasets. We can see that, under the same model struc- ture, DOCSPLIT (i.c., DoCSPLITbert and DOCSPLITHon)) achieves 12.0% and 24.3% macro-F1 improvements compared to SimCSE and Longformer respectively. Surprisingly, INSTRUCTOR achieves low performance on the NYT and BBCNews under few-shot settings.
These improvements higher than standard text classification.


Reviews:
[{"review_id": "heX3PyPOPt", "reviewer": "Reviewer_yVuF", "summary": "", "questions": "1. Have you tried to train the baseline models (only MLM loss) using your data and hyperparameters? Using only English wiki may not always be a disadvantage if the downstream tasks match the domain, and the hyperparameters in pre-training also affects the results significantly as shown by the roberta paper. ", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Have you tried to train the baseline models (only MLM loss) using your data and hyperparameters?  Using only English wiki may not always be a disadvantage if the downstream tasks match the domain, and the hyperparameters in pre-training also affects the results significantly as shown by the roberta paper.\n\nYou have a valid concern about datasets and hyperparameters effecting performance on downstream tasks.  We did not perform the experiment you suggest, however, because the SimCSE paper already performed this experiment and our experimental design is modeled on theirs.\n\nIn particular, the SimCSE paper trained models on only English-language wikipedia, including retraining the BERT model with the MLM loss on this dataset.  They then evaluated on 7 semantic text similarity (STS) downstream tasks and found that the MLM-loss BERT model performed significantly worse than than the standard BERT checkpoint for these tasks.  This suggests that the reduced dataset size of using only Wikipedia has a negative impact on the BERT model's performance for downstream tasks, and that the standard checkpoint should be used instead for best performance.\n\nWe also note that follow-on papers with similar experimental setups (e.g. CT-BERT and Contriever) compare only against the BERT checkpoint and not against a retrained-from-scratch BERT.\n\nBased on this explanation, we hope that you will consider increasing your soundness evaluation of our paper.  We did not include a discussion like this in the original submission due to space limitations, but we will try to include this discussion in the camera-ready copy.\n\n> In 3.2, the attention is presented as O(n^2) time complexity, which is only true for the vanilla implementation but not for attention mechanism itself. (see https://arxiv.org/abs/2112.05682)\n\nWe believe this is incorrect.  In particular, the paper you linked provides algorithms for improving the *memory* complexity, but not the *time* complexity.  For example, the paper abstract states that \"the time complexity is still O(n^2)\".  We believe that the only known mechanisms for improving the time complexity of attention remain approximations such as the LongFormer, BigBird and other models cited in Section 3.2.\n\nThis is still a valuable citation, however, that we are grateful for, and we will add it to our discussion in Section 3.2.  One of the main advantages of our DocSplit model is that it is entirely model agnostic, and so will work with the memory efficient algorithms proposed by this paper, and that is something that we would like to highlight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is simple and model-agnostic.\n2. The method works for both short and long documents.", "reasons_to_reject": "1. The work is incremental, and lacks novelties.\n", "questions_to_authors": "1. Have you tried to train the baseline models (only MLM loss) using your data and hyperparameters? Using only English wiki may not always be a disadvantage if the downstream tasks match the domain, and the hyperparameters in pre-training also affects the results significantly as shown by the roberta paper. ", "paper_topic_and_main_contributions": "This paper proposes an unsupervised pre-training objective to improve document embeddings for text classification and document retrieval. A contrastive loss is designed to distinguish document sections of the same documents from those of different documents, which is optimized together with the masked language model loss. The method is shown to be effective in both short and long document settings against several baselines. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In 3.2, the attention is presented as O(n^2) time complexity, which is only true for the vanilla implementation but not for attention mechanism itself. (see https://arxiv.org/abs/2112.05682)"}, "extra_text": {}}, {"review_id": "9pj5KQl3MB", "reviewer": "Reviewer_7Pji", "summary": "", "questions": "The random sampling method for positive instances generation is quite interesting. Is there exists the situation where the extracted summary may contain sentences that are not very relevant to the topic of the summary? I feel a bit confused about how do the authors make sure that the model is capable of generating high-quality positive instances with random sampling. Do the authors consider extracting sentences according to their contributions to the topic of the documents, e.g., attention weights?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "These are great questions that we unfortunately weren't able to discuss in detail due to the short 4-page limitation, but we'll try to answer for you here.\n\n> Is there exists the situation where the extracted summary may contain sentences that are not very relevant to the topic of the summary? I feel a bit confused about how do the authors make sure that the model is capable of generating high-quality positive instances with random sampling.\n\nRandom sampling is a standard component in most methods that use contrastive learning.  For example, our baseline models (CT-BERT, SimCSE, Contriever, and INSTRUCTOR) all use random sampling to some degree in their loss functions.\n\nIntuitively, our use of randomness is similar to the randomness in stochastic gradient descent.  We can't \"make sure\" that all of the samples are high quality because of the randomness, but the goal is that the low quality samples are relatively few and will and so the training will still converge to a good solution.  In Figure 1, we provide one example random split of a document, but if you try any of the other possible splits you would see a similarly reasonable result.\n\n> Do the authors consider extracting sentences according to their contributions to the topic of the documents, e.g., attention weights?\n\nThis is an interesting idea that we've not thought about too much.  Our initial reaction is that it is unlikely to work well for several reasons:\n\n1. Random splitting is much faster than another model would be.  Using another model to select the division would result in approximately a 2x slowdown in training speed, which would mean that the training dataset would be about half the size for a given computational budget.  Since larger datasets have been shown to have such an important effect on performance for modern NLP models, the alternative model would have to be quite good indeend to justify using half the amount of data.\n\n2. There's no \"obvious\" way to use another model to perform the splitting.  It's reasonable to try to exploit the attention weights, but since these are at a token level rather than the sentence level, we would need to somehow aggregate these weights.  The obvious aggregation choices are ave/max pooling, but this results in a vector output, and there's no obvious way for converting that vector into a binary splitting decision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on an interesting research topic and has clear research meaning. The scientific innovation is appropriate. The experiments are reasonable and convincing.", "reasons_to_reject": "N/A", "questions_to_authors": "The random sampling method for positive instances generation is quite interesting. Is there exists the situation where the extracted summary may contain sentences that are not very relevant to the topic of the summary? I feel a bit confused about how do the authors make sure that the model is capable of generating high-quality positive instances with random sampling. Do the authors consider extracting sentences according to their contributions to the topic of the documents, e.g., attention weights?", "paper_topic_and_main_contributions": "This paper proposes a new contrastive pretraining method called DOCSPLIT which forces models to learn representation of documents with positive examples that are randomly sampled from the input document, and negative examples are randomly from unrelated documents. The proposed model achieves SOTA performance on document classification, few-shot learning, and retrieval tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1u8SN0exdc", "reviewer": "Reviewer_c7xp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This is more of a question than a weakness. The explanations in Line 097-099 is somewhat weak. Would this bias some sentences (e.g., the first K sentences in a news article) that are designed to be summaries?\n\nYou raise a good point that some sentences can be more important than others, especially in news articles.  In practice, however, this turns out not to be a problem for our method.  For example, in Experiment 1 we have 4 datasets that are comprised of news articles (FakeNews, 20News, NYT, and BBCNews).  For each of these datasets, using the DocSplit method results in SOTA accuracy and F1 scores.\n\nWe can think of several theoretical explanations for why this might be the case:\n\n1. The random split ensures that \"important\" sentences are likely to end up in both positive instances, especially as the number of \"important\" sentences increases.\n\n2. Existing research on large document embeddings suggests that the first sentences of a document are not really that much more important than other sentences.  If they were, then the standard BERT model with a small context window would perform just as well as models with a much larger context window (like LongFormer or BigBird).  But the wide range of research on these models summarized in Section 3.2 suggests that this is not the case.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-motivated problems and clear writing.\n- Well-executed experiments.\n- Simple methods that are very practical. Also, this is not necessarily a reason for acceptance, I appreciate that the authors chose not to include (seemingly unnecessary) technical details for clarity (in Lines 079-081).\n- Somewhat thoughtful limitations section.", "reasons_to_reject": "- This is more of a question than a weakness. The explanations in Line 097-099 is somewhat weak. Would this bias some sentences (e.g., the first K sentences in a news article) that are designed to be summaries?", "questions_to_authors": "", "paper_topic_and_main_contributions": "Existing model pretraining methods do not work well in generating embeddings for large documents because they consider local information. The authors propose a novel method -- DocSplit -- which forces the model to consider more global information through contrastive methods. Experimental results demonstrate the efficacy of the proposed methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple idea that improves learning of document embeddings. The idea is to apply standard a standard contrastive learning technique (operating at the below-document section level) that instills 'global' (i.e., document-level) understanding which in turn then enhances learning document embeddings. Contrastive learning operates, as usual, on the sets of positive and negative pairs, where the positive instances are sections sampled from the same document and negative instances are samples randomly taken from unrelated documents. Given that this is a short paper, it does a good coverage of tasks related to learning good document embeddings and it shows some improvements over standard document embedding models (e.g., Longformer, BigBird, Contriever). As mentioned (and also clearly stated by the authors), the idea is very simple, but it yields consistent gains in the three evaluation tasks. 

I agree with one of the reviewers that the paper should have ideally investigated the idea of contrastive learning also with non-random (e.g., topically more similar) negative samples which resembles the idea of using hard negatives in standard contrastive learning frameworks. The paper also does not disclose which exact contrastive objective/loss is used (as there are plenty of different options on how to implement the basic contrastive idea, and it is also possible to combine different losses).

However, my main and strong concern is that, although the gains over the chosen baselines are salient, the paper does not provide a good comparison to all the relevant baselines and does not cover those recent methods and baselines in future work. The lack of these comparisons reduces the potential impact and it might negatively affect the novelty of the proposed method (again, noting its sheer simplicity). For instance, a body of work aims to improve (long) document embeddings by running specific contrastive learning techniques and the authors should ideally acknowledge and compare against the following papers at least:
- https://arxiv.org/abs/2305.16031
- https://aclanthology.org/2022.emnlp-main.802
- https://arxiv.org/abs/2103.14542
- https://aclanthology.org/2021.emnlp-main.109
- https://aclanthology.org/2021.findings-emnlp.327.pdf
Decision: Accept-Findings