Re3: A Holistic Framework and Dataset for Modeling Collaborative Document Revision (2024)

Abstract: Collaborative review and revision of textual documents is the core of knowledge work and a promising target for empirical analysis and NLP assistance. Yet, a holistic framework that would allow modeling complex relationships between document revisions, reviews and author responses is lacking. To address this gap, we introduce Re3, a framework for joint analysis of collaborative document revision. We instantiate this framework in the scholarly domain, and present Re3-Sci, a large corpus of aligned scientific paper revisions manually labeled according to their action and intent, and supplemented with the respective peer reviews and human-written edit summaries. We use the new data to provide first empirical insights into collaborative document revision in the academic domain, and to assess the capabilities of state-of-the-art LLMs at automating edit analysis and facilitating text-based collaboration. We make our annotation environment and protocols, the resulting data and experimental code publicly available.1

heading: 1 Introduction
text: Textual documents are a key medium of information exchange in the modern world. These documents often result from a collaboration of multiple individuals. The typical process of collaborative text production involves iterations of drafting, getting feedback (reviews), executing revisions, and providing responses that outline the implemented changes, serving as a vital element in facilitating effective communication (Cheng et al., 2020; Kuznetsov et al., 2022). Despite the importance of collaborative text revision and its high potential for NLP applications, we are missing a framework that formally describes this review-revisionresponse procedure grounded in real-world data. While prior work in NLP has studied relationships between original and revised documents (Du et al.,
1https://github.com/UKPLab/re3
2022; Jiang et al., 2022), reviews and original documents (Dycke et al., 2023), reviews and revisions (Kuznetsov et al., 2022; D’Arcy et al., 2023), and reviews and responses (Gao et al., 2019; Cheng et al., 2020) – no prior frameworks allow jointly modeling all three components of text-based collaboration. Yet, such joint modeling is important as it provides deeper insights into the processes involved in text work, and opens new opportunities for NLP applications. Important tasks that involve reviews, revisions and responses such as edit summarization thus remain underexplored.
Comprehensive analysis of document-level revisions poses additional challenges. Contrary to sentence-level analysis, hierarchically structured documents (Ruan et al., 2022) bring distinct levels of granularity into editing. Individuals execute revisions at various granularity levels, with a range of actions and a spectrum of intents, reflecting the what, how, and why of the revisions (Figure 1 and §3.2). Realistic modeling of document revision
4635
in text-based collaboration thus requires datasets and annotations that encompass the entire document context, incorporating all edits made across various levels of granularity, and providing qualitative labels for both action and intent. We further term this kind of analysis as full-scope modeling of document revision. Prior research in NLP has primarily studied sentence-level edits while neglecting the broader document context (Daxenberger and Gurevych, 2012; Yang et al., 2017), variations in granularity (Du et al., 2022; Kashefi et al., 2022), and the underlying intent behind the edits (Spangher et al., 2022; Jiang et al., 2022). There is thus a gap in both methodologies and datasets for creating and analyzing full-scope annotations of document revisions, limiting our grasp of the intricate nature of the editing process.
To close this gap and enable a comprehensive study of text-based collaboration in NLP, we introduce Re3: the first holistic framework for modeling review, revision and response in collaborative writing (§3). We instantiate our framework in the scholarly domain and create Re3-Sci, the first large-scale human-annotated dataset that comprises 11.6k full-scope revision annotations for over 300 revised documents with substantial Inter-Annotator Agreement (IAA), as well as cross-document connections between reviews, revisions and responses (§4). Our framework and dataset, for the first time, enable large-scale empirical investigation of collaborative document revision, including edit localization and clustering within documents, edit mechanisms and motivations inferred through action and intent labels, and the impact of review requests (§5). Manually analyzing the complex relationships between reviews, revisions and responses is costly, and constitutes a promising NLP automation target. Facilitated by our data, we present a first exploration of the capability of large language models (LLMs) to address novel revision assistance tasks, such as review request extraction, revision alignment, edit intent classification and document edit summarization (§6). Our work thus makes four key contributions:
• A holistic framework for studying document revisions and associated interactions in collaborative writing, including label taxonomy and robust annotation methodology; • A high-quality large-scale dataset that instantiates the framework in the domain of academic writing and peer review;
• An in-depth analysis of human editing behavior in the scholarly domain; • Extensive experiments in automation with LLMs on four NLP tasks: review request extraction, revision alignment, edit intent classification and document edit summarization.
Our work paves the path towards comprehensive study of NLP for text-based collaboration in the scholarly domain and beyond.

heading: 2 Related Work
text: length edits full-scope align intent IteraTeR (2022) 197 7* no 4k* 4k* ArgRewrite (2022) 582 19 no 3.2k 3.2k arXivEdits (2022) 3,916 17 no 13k 1k Re3-Sci (ours) 5,033 37 yes 11.6k 11.6k
(a) Comparison of human-annotated document revision datasets. Presented are document length (words), average sentence edits per document, presence of full-scope revision annotations, and data size, i.e., count of aligned and labeled sentence edits. * refers to subsentence edits as only such annotations are available. Re3-Sci is the first large-scale corpus with full-scope annotations of edit alignments, actions, and intents across multiple granularity levels in the entire document.
full-scope review-revision revisionresponse reviewresponse
F1000RD (2022) no yes no no NLPeer (2023) no yes no no ARIES (2023) no yes no no Re3-Sci (ours) yes yes yes yes
(b) Comparison of review-revision-response datasets. Presented are presence of full-scope revision annotations, and interactions between the documents. Our work is the first to cover the entire review-revision-response procedure with full-scope annotations.
Table 1: Related datasets comparison.
Document revision datasets. Research on text revision originates in studies on Wikipedia (Daxenberger and Gurevych, 2012; Yang et al., 2017; Faruqui et al., 2018) and academic writing (Tan and Lee, 2014; Xue and Hwa, 2014), which offer partial sentence-based annotations, neglecting the document context. Recent works have expanded the analysis to news articles (Spangher et al., 2022), student essays (Zhang et al., 2016; Kashefi et al., 2022), and scientific papers (Du et al., 2022; Jiang et al., 2022). However, some focus mainly on revision alignment yet overlook the underlying intents (Spangher et al., 2022; Jiang et al., 2022). Others restrict to specific sections or short texts, limiting analysis to sentence level (Du et al., 2022; Zhang et al., 2016; Kashefi et al., 2022). In this work, we introduce Re3-Sci, the first large-scale corpus with
full-scope annotations of edit alignments, actions, and intents across multiple granularity levels in the entire document (Table 1a). Collaborative revision in peer review. Scholarly peer review is an essential example of collaborative text work in the academic domain. Open peer review provides an excellent opportunity to study the review-revision-response procedure. Prior works in NLP for peer review investigate argument miningdriven review analysis (Hua et al., 2019; Fromm et al., 2020) and the interplay between reviews and argumentative rebuttals (Gao et al., 2019; Cheng et al., 2020; Bao et al., 2021; Kennard et al., 2022), among others. Only a few studies and datasets investigate revision requests in peer reviews and their connection to the original texts (Dycke et al., 2023), or to the actual revisions (Kuznetsov et al., 2022; D’Arcy et al., 2023). However, these do not provide full-scope annotations with qualitative labels and neglect self-initiated revisions not prompted by reviewers. Our work is the first to cover the entire review-revision-response procedure with full-scope annotations in the context of scholarly publishing and peer review (Table 1b).

heading: 3 The Re3 Framework
text: The Re3 framework builds upon the recently introduced intertextual model by Kuznetsov et al. (2022). In particular, we represent documents as graphs that preserve document structure, allowing us to work on different levels of granularity, and treat cross-document relations as edges between the corresponding document graphs.

heading: 3.1 Model and Terminology
text: As shown in Figure 2, we conceptualize the reviewrevision-response procedure as a set of interactions among four document types - the original document Dt, the revised document Dt+1, the review C and
the response A - along with the diverse types of connections between their text elements. Depending on the granularity g, text elements can vary from subsentence-level words and phrases to sentences, paragraphs, or sections. Text elements of granularity g of the old and new documents are noted as xt,gj ∈ Dt and x t+1,g i ∈ Dt+1. Comparing two document versions, edit alignment links elements from the new and old versions. For analytical clarity, aligned elements maintain the same granularity in our study, noted as eij = e(x t+1,g i , x t,g j ). Each edit alignment eij is associated with an edit label, ELij = (g,EAij , EIij), which specifies the granularity, action, and intent of the edit, explaining how and why the edit is made to a text element of g (§3.2). When a new text element xt+1,gi is added or an old one xt,gj is deleted, the corresponding old or new element is null, noted as e(xt+1,gi , null) or e(null, xt,gj ).
Given that the reviews and responses are typically brief without rich structure, we focus on the sentence level in those documents. Reviews include requests ck ∈ C that may prompt edits. An addressed request ck is linked to relevant edit eij as ec(eij , ck). A single request can lead to multiple edits, while self-initiated edits may not connect to any review request. Similarly, an author’s response includes sentences an ∈ A summarizing realized revisions. Each an connects to its respective edit eij via the relation ea(eij , an).

heading: 3.2 Revision Dimensions and Label Taxonomy
text: We analyze revisions along three qualitative dimensions – granularity, action, and intent – and present our proposed label taxonomy in Figure 3. Granularity specifies the scope of the text elements subject to revision, which is crucial since the perception of revisions varies with granularity. For instance, extending a sentence may appear as adding text elements at subsentence level or as modifying an existing text element at sentence level (further exemplified in Table 13 in §A). In this work, we include section, paragraph, sentence, and subsen-
tence granularities. Action specifies how revisions are made, including basic methods like addition, deletion, and modification, as well as complex operations like merges, splits, and fusions (elaborated in Table 14 in §A). Intent categorizes the underlying purpose into surface language improvements for grammar or clarity and substantial semantic changes affecting claims or factual content, with detailed definitions and examples in Table 15 in §A. The three dimensions collectively characterize the nature, purpose, and significance of the revisions. For instance, factual content updates may entail sentence expansion with additional details or the incorporation of an entirely new sentence. When significant elaboration is necessary, new paragraphs or sections may be introduced. The taxonomy has been refined through feedback from two linguists and proved sufficient and crucial in the annotation study with six annotators (§4.3 and §B.6). The hierarchical structure of the taxonomy promotes easy expansion and adaptation across various domains by incorporating fine-grained labels.

heading: 4 Dataset Construction: Re3-Sci

heading: 4.1 Data Collection and Pre-processing
text: Scientific publishing, a prominent open source of collaborative document revision and review, offers ample data for our research objectives. We instantiate our framework based on the data from the F1000RD dataset (Kuznetsov et al., 2022) and the ARR-22 subset of the NLPeer corpus (Dycke et al., 2023), which include revisions of scientific papers along with their corresponding reviews, covering a range of fields including NLP, science policy, public health, and computational biology. Both datasets contain structured documents organized into section and paragraph levels, which we further refine to sentence level (§B.1). A total of 314 document pairs and related reviews are randomly selected for human annotation: 150 from NLPeer and 164 from F1000RD.

heading: 4.2 Pre-alignment
text: Identifying revision pairs from two lengthy documents is challenging, especially complicated by the expansive scope for comparison and the presence of recurring content (Jiang et al., 2020). To address this, we employ a lightweight sentence alignment algorithm that systematically excludes identical pairs and identifies alignment candidates from the remaining sentences, considering both
form and semantics similarity, as well as the document’s context and structure (§B.2). Annotators are given the alignments and tasked with validating and correcting any alignment errors. Based on these corrections, the proposed algorithm achieves an accuracy of 0.95. The validated alignments are subsequently used for edit action and intent labeling.

heading: 4.3 Annotation Process
text: To perform the human annotation, we develop a cross-document annotation environment using INCEpTION (Klie et al., 2018), as detailed in §B.3. A pilot study with 20 document pairs is initiated to refine the label taxonomy, optimize the prealignment algorithm, improve the annotation tool, and develop comprehensive guidelines, with the assistance of three in-house annotators skilled in computer science or linguistics.
For annotation, six master’s students with C1level English proficiency are recruited (§B.5). We employ an iterative data quality management process to ensure the quality of the annotations. Initially, a 15-hour training session is spread over three days, involving a joint review of guidelines, live demonstrations, and practice annotations on a validation set of five document pairs. Given the initial suboptimal IAA in intent labeling, highlighting its complexity, we conduct further discussions on disagreements and common mistakes, followed by a final re-annotation of the validation set. This method ensures consistent comprehension of the guidelines and familiarity with the annotation process prior to actual annotation. Documents are divided into three data packages for iterative quality assessment, with intermediary meetings by the coordinator to address annotators’ individual questions. The primary tasks, sentence-level revision alignment and edit labeling, are carried out by three annotators per sample. We release all three annotations with a gold label aggregated through majority voting. After annotation, we conducted an annotator survey to gather insights for future annotation studies.
We achieve a substantial (Landis and Koch, 1977) IAA of 0.78 Krippendorff’s α for the labeling task and a perfect IAA of 1 Krippendorff’s α for the alignment task. Table 5 in §B.4 shows progressive IAA improvement following iterative quality management between data packages, highlighting the method’s effectiveness. As a qualitative
assessment, the annotator survey (§B.6) confirms the adequacy of guidelines, label taxonomy, and annotation tool, as well as the effectiveness of iterative training. The annotators also highlight the effectiveness of the cross-document annotation environment, especially in aligning revision pairs, which potentially contributes to the perfect IAA in alignment. Further insights are provided in §B.6.

heading: 4.4 Statistics
text: The Re3-Sci dataset comprises 314 document revision pairs. 11,648 sentence-level edits comprising sentence revision pairs, additions, and deletions are identified and annotated with respective edit action and intent labels. Based on the sentencelevel annotations, 5,064 paragraph-level, and 2,008 section-level edits are identified. We also extract 2,676 subsentence-level edits from 1,453 sentence revision pairs, employing a constituency tree-based method similar to Jiang et al. (2022). These extractions and alignments are verified by a linguistic expert and labeled by three annotators. Furthermore, we randomly select 42 documents and extract 560 review sentences that may prompt changes. The review sentences are aligned with the corresponding revisions when possible, resulting in 413 linkages. Annotators summarize the document revisions in brief responses and then align a total of 784 summary sentences back to the related edits, resulting in 1,364 connections. See §B.5 for more details.

heading: 5 Dataset Analysis
text: The framework and dataset allow us to answer new questions about human behavior in collaborative document revision in scholarly publishing. RQ1: How and why do humans edit, and what are the relationships between edit actions and intents? Figure 4 reveals that authors predominantly modify existing content and add new mate-
rial, with deletions being infrequent and partition changes even less common. It also suggests that the enhancement of fact or evidence is the primary focus of revisions, highlighting its importance in improving scientific quality. Moreover, Figure 4 illustrates that additions and deletions of sentences typically pertain to improving factual content or claims, but are never intended for superficial language enhancement. On the other hand, grammar and clarity improvements are usually realized by modifying existing sentences. This suggests that, from a modeling view, the edit action and intent labels may influence the prediction of each other.
RQ2: How are edits distributed in documents? Figure 5 indicates that the initial and final parts of papers experience significantly more revisions. In terms of edit actions, the beginning of the document typically sees more modifications, while the end is characterized by a higher frequency of additions and deletions. Regarding edit intents, language enhancements for grammar or clarity are more common in the early parts, whereas changes affecting semantic content, such as facts or claims tend to occur more in the later parts. These suggest that the document position may be a valuable predictor for identifying edit actions and intents. RQ3: How significant are the differences between document versions? To gauge the magnitude of change, we introduce the Edit Ratio metric,
determined by the ratio of sentence edits to the sentence count in the original document. While the edit ratio reflects the extent of differences, the significance of document revisions is highlighted by the Semantic Edit Ratio, which is calculated by the ratio of semantic edits labeled with Fact/Evidence or Claim. The average document edit ratio stands at 18.45%. Figure 6a and Figure 10 in §C show that the majority of documents experience moderate revisions with an edit ratio of 5-25%, while a small proportion has an edit ratio exceeding 50%, and only a few documents appear to have been extensively rewritten. The average semantic edit ratio stands at 11.18%, with most documents showing 0- 20% of their content undergoing significant change. Notably, documents with a high edit ratio often do not correspond to a high semantic edit ratio, suggesting that documents with extensive revisions typically exhibit language quality issues.
RQ4: How are edits clustered by paragraphs and sections? We use Crest Factor (CF), a concept borrowed from signal processing (Parker, 2017), to assess the concentration of edits. Using a vector of sentence edit counts in each paragraph or section, CF quantifies the peak amplitude of this distribution. A CF value of 1 signifies an even distribution. The average paragraph CF is 3.79, indicating a substantial concentration of edits within a limited number of paragraphs. This trend of high edit concentration in a few paragraphs is further illustrated in Figure 6b and Figure 11 in §C. When examined at the section level, the average CF is 2.54, indicating a moderate tendency towards clustering. RQ5: Are reviewers’ requests acted upon? How are these realized in revision? Annotators categorize the relevant review requests into three types: explicit edit suggestions (28%), implicit edit suggestions (32.1%), and general weakness comments (39.8%). The first provides specific document locations and clear revision instructions; the second
delivers guidance without locations; the last highlights general issues without specific suggestions. Figure 7 shows that more than half of explicit suggestions are implemented, with 18.47% actualized through multiple sentence edits. Implicit suggestions and general weakness comments are realized to a lesser extent. This implies that reviewers’ explicit suggestions are more likely to be acted upon.

heading: 6 Automation with LLMs
text: The Re3-Sci dataset facilitates a variety of NLP tasks, such as (1) edit intent classification, (2) document edit summarization, (3) revision alignment and (4) review request extraction. Tasks (1) and (3) demonstrate the joint modeling of old and new documents (revision), with the latter providing the basis for NLP-assisted edit analysis and the former enabling in-depth analysis and being challenging even for human annotators (§B.6). Task (2) represents a novel task to jointly model the revisionresponse process, a capability uniquely enabled by our new dataset. Re3-Sci is distinguished by its full-scope annotations, which cover all edits with action and intent labels, and include information on the document’s context, structure, and granularity (§4). These features enable detailed descriptions of edits and precise localization of their positions within the document (§6.2). Task (4) focuses on the review-revision relationship, aiming to identify review sentences that need attention during the revision phase. The dataset’s utility extends beyond these four tasks, including edit generation from review requests or desired actions/intents, and anchoring edits to review requests, which we leave for future work. §D.4 provides computational details.

heading: 6.1 Edit Intent Classification
text: Task formulation and data. Formulated as a classification task given a sentence edit e(xt+1,gi , x t,g j ), the objective is to predict the intent label EIij . For
additions or deletions, only one sentence is used. We split the documents into 20% for training and 80% for testing.2 The test set contains 5,045 revision pairs and 3,891 additions or deletions. Models and methods. We evaluate Llama2-70B (Touvron et al., 2023) with multiple ICL demonstration selection methods and analyze CoT prompt formatting. The three dynamic ICL methods select the most similar demonstrations from the training set for each test sample using RoBERTa embeddings (Liu et al., 2019): cat uses cosine similarity of concatenated sentence embeddings, diff leverages the difference between sentence embeddings, and loc utilizes concatenated embeddings of the associated section titles. The static def method uses a default set of manually selected examples for each intent across all tests. For CoT, we instruct the model to predict the intent label (L) with rationale (R) in CoT style, evaluating how their order impacts results. A prompt example is provided in Table 11 in §D.1. Preliminary experiments indicate inadequate performance in jointly modeling revision pairs and single-sentence instances, leading us to separate experiments for each scenario. Results and discussion. Table 3 shows the results on revision pairs. In addition to the random baseline, other baselines use the majority label of the top n selected training examples from the three proposed methods. Using the same examples for ICL, the diff method notably excels over others (block 1). Interestingly, Llama2 doesn’t rely solely on the majority label of selected examples. Comparing block 1 with the majority baselines reveals a significant improvement and reduced disparities between the methods. Using five default examples outperforms cat and loc, and is on par with diff (block 2). Accuracy further increases when the gold label is accompanied by a rationale in CoT style (i.e., L,R: label followed by rationale). This straightforward but effective prompting method achieves performance comparable to more advanced methods, as detailed in the subsequent blocks of Table 3. However, reversing the order of the label and the rationale (i.e., R,L) notably decreases performance. Combining default examples with rationale and dynamic diff -selected examples further enhances accuracy (block 3). Altering the order of dynamic and static default examples enhances results when using diff, though this is not consistent across all
2We use training data for ICL example selection only and the rest for testing to get more reliable performance estimates.

heading: Baselines
text: selection methods (blocks 3, 4). Omitting rationale from the default examples leads to a significant and consistent performance decline, highlighting the importance of CoT demonstrations (blocks 5, 6).
The best configuration involves three dynamic diff -selected examples and the default examples with CoT rationale. This also yields the best performance for additions and deletions, as shown in Table 8 in §D.1. With this setup, joint evaluation on the full test set results in an accuracy of 0.7 and a macro-average F1 score of 0.69, significantly outperforming the baselines as shown in Table 10 in §D.1. The pronounced potential for advancement highlights the task’s complexity for LLMs. This requires precise detection of changes and advanced reasoning capabilities to understand intents.
Figure 8 displays an error analysis comparing human annotations with Llama2 predictions. Both humans and Llama2 are prone to misclassify claim and fact changes, which may stem from subjective statements being phrased in a fact-like manner and the common occurrence of intertwining both aspects within a single sentence. Llama2 demonstrates a propensity to over-predict clarity changes,
often misinterpreting fact/evidence (7.84%), grammar (3.54%), and claim (3.05%) changes.

heading: 6.2 Document Edit Summarization
text: Task formulation and data. Our full-scope annotations of document revisions enable a novel task, document edit summarization, which constitutes the foundational basis for generating author responses. Specifically, the task is formulated as a text generation task, given a complete list of sentence edits e(xt+1,gi , x t,g j ) within a document, associated action and intent labels ELi,j , as well as associated section titles that provide structural information about the edits. The output is a coherent textual summary of the document edits, as exemplified in Table 12 in §D.2. We conduct experiments on 42 documents with human-written edit summaries. These documents contain an average of 33 sentence edits, resulting in a median input length of 3,916 tokens. Models and methods. As almost half of the inputs exceed Llama2’s constraints and preliminary trials yield unsatisfactory results, we opt for GPT-4 to handle this challenging task in a zero-shot manner. We performed a human evaluation of the generated summaries, systematically comparing them with human-authored summaries across five dimensions: factuality, comprehensiveness, specificity, compactness, and organization. Each summary sentence an is linked to its respective sentence edits, creating ea
linkages. And an is annotated if it does not refer to actual edits and labels. or it is hard to connect with specific edits. For instance, vague summaries such as "there are grammar corrections" pose challenges in establishing precise associations. Factuality is quantified by the percentage of summary sentences that accurately refer to actual edits. Comprehensiveness denotes the extent of edits encapsulated within the summary. Specificity reflects the proportion of concrete summary sentences. Compactness is gauged by the average number of edits incorporated into a single summary sentence. And organization refers to the logical arrangement of the summary content. Results and discussion. Table 4 provides a comparative analysis between human-authored and LLM-generated summaries. Humans typically produce marginally lengthier text with more sentences, ensuring impeccable factuality alongside elevated comprehensiveness and specificity. Conversely, GPT-4 fails to address 21% of document edits, exhibiting factuality concerns in 4% of summary sentences, and lacking specificity in 10% of cases. Additionally, GPT-4 summaries demonstrate a slightly higher level of compactness, averaging 2.36 edits condensed into a single sentence. While humans typically organize summaries by sections, reflecting conventional sequential reading patterns. GPT4 also exhibits a structured logical arrangement but often organizes summaries by action labels, usually beginning with additions and deletions.

heading: 6.3 Revision Alignment
text: Task formulation and data. The task is conceptualized as a binary classification problem, where the goal is to determine if a given pair of sentences xt+1,gi ∈ Dt+1, x t,g j ∈ Dt constitutes a revision pair e(xt+1,gi , x t,g j ). Along with the 6,353 revision pairs from the Re3-Sci dataset, an equivalent number of negative samples are created, resulting in a total of 12,706 samples for experimental purposes. 80% are used for testing and 20% for training. To preserve the task’s complexity, negative samples are composed by pairing revised sentences within
the same document that do not link to each other but likely address similar topics. This simulates the intricate nature of revisions in lengthy documents as detailed in §B.2. Models and methods. For this task, we employ the Llama2-70B model and apply the same ICL and CoT methods used for the edit intent classification task, as detailed in §6.1. Results and discussion. Mirroring the same findings observed in the edit intent classification tasks, Table 9 in §D.3 shows that using static default examples with CoT reasoning throughout the experiments yields favorable performance (block 2), highlighting its efficacy as a straightforward yet effective prompting strategy. Using this strategy, we achieve an accuracy of 0.97 on the full test set.
It is worth noting that our proposed prealignment algorithm (§B.2) achieves a strong accuracy of 0.95, with a recall of 0.99 for non-alignment and a precision of 0.99 for alignment. However, the precision for non-alignment (0.89) and the recall for alignment (0.92) are relatively low. This discrepancy can be attributed to the utilization of high similarity thresholds and stringent aligning rules in the algorithm. In contrast, when automated with Llama2, we achieve a precision of 0.99 for non-alignment and a recall of 0.99 for alignment, which constitutes a perfect enhancement to the pre-alignment algorithm. For revision alignment, we thus propose a two-stage approach that combines the lightweight pre-alignment algorithm with Llama2 In-Context learning. The lightweight algorithm efficiently identifies candidates and accurately extracts revision pairs with minimal computational cost. Subsequently, we apply the proposed prompting strategy with Llama2 selectively to the non-aligned candidates, thereby identifying missing revision pairs without significantly increasing computational overhead.

heading: 6.4 Review Request Extraction
text: Task formulation and data. The task is framed as a binary classification problem, aiming to ascertain whether a particular review sentence ck ∈ C could instigate revisions and necessitate further processing in the revision workflow. The experimental data comprises 1,000 samples, including 560 review requests (including explicit and implicit edit suggestions, and general weakness comments) from the Re3-Sci dataset, plus 440 negative samples extracted from the same review documents. Of
these, 80% are for testing and 20% for training. Models and methods. For this task, we utilize Llama2-70B with the same ICL and CoT methods previously applied, as elaborated in §6.1. Results and discussion. Employing the straightforward def method with CoT reasoning, which involves two static default demonstrations, yields an accuracy of 0.80 on the full test set. This approach achieves a high precision of 0.95 for negative samples and a remarkable recall of 0.98 for positive samples. Nevertheless, the precision for positive samples is relatively low at 0.74, highlighting the method’s inherent challenges. Future research could expand this task into a four-label classification, differentiating various types of review requests. This approach could further elucidate the methods’ capabilities and limitations.

heading: 7 Conclusion
text: We have introduced the Re3 framework and the Re3-Sci dataset, for empirical analysis and development of NLP assistance for text-based collaboration. Through annotation study and data analysis, we have demonstrated the utility of the framework and revealed novel insights into human behavior in collaborative document revision and peer review, including relationships between specific edit actions and intents, focused localization patterns, clustering tendencies within paragraphs, and the acceptance rates for review requests. Our automation experiments have assessed the ICL and CoT capabilities of state-of-the-art LLMs on four tasks for collaborative revision assistance. In the classification tasks with Llama2-70B, we noted that using default static ICL demonstrations with CoT rationale produces satisfactory results, demonstrating the efficacy of this straightforward yet effective prompting strategy. In the document edit summarization task, GPT-4 demonstrated the ability to generate coherent summaries but faced challenges related to factuality and comprehensiveness.
Our work paves the path towards systematic fullscope study of text-based collaboration in NLP and beyond. The framework, taxonomy, annotation methods and tools are applicable to diverse domains. The dataset offers a robust foundation for multifaceted research for collaborative revision assistance. Future work may encompass tasks like identifying text segments necessitating revision and generating revisions guided by review requests or specified actions and intents.

heading: Limitations
text: This study has several limitations that should be considered when interpreting our results and the implications we draw from them. From the data and modeling perspective, the study’s exclusive focus on English-language scientific publications is due to the restricted availability of openly licensed source data. Studying the transferability of our findings to new languages, domains, application settings and editorial workflows is an exciting avenue for future research, which can be supported by our openly available annotation environment and protocols. Our study used human-generated edit summaries instead of author responses or summaries of changes written by the authors themselves due to the lack of data. As peer-reviewing data collection becomes increasingly popular in the NLP community, we expect new datasets to enable such studies in the future.
From a task perspective, it is important to highlight that the implementations and results presented in this study serve as illustrations of the proposed tasks. Their primary purpose is to ascertain the technical feasibility and lay the groundwork for the development of future NLP systems for collaborative writing and revision assistance. Consequently, the provided implementations have inherent limitations. For instance, our approach selectively utilizes state-of-the-art LLMs without conducting comprehensive comparisons with other LLMs or smaller fine-tuning-based models. A systematic exploration of NLP approaches for the proposed tasks lies beyond our scope and is left for the future.

heading: Ethics Statement
text: The analysis of text-based collaboration and the corresponding NLP assistance applications have a potential to make knowledge work more efficient across many areas of human activity. We believe that the applications and analysis proposed in this paper deliver equitable benefits to every stakeholder involved in the procedure – authors, co-authors, reviewers, and researchers who want to study their collaborative text work.
The human annotators employed in our study were fairly compensated with a standard salary for student assistants in the country of residence. They were informed and consented to the publication of their annotations as part of this study. The annotation process does not entail the gathering or handling of their personal or sensitive information.
For privacy protection, both author metadata and annotator identities have been omitted from the data release.
Both subsets of the source data are licensed under CC-BY-NC 4.0, ensuring that the construction and use of our dataset comply with licensing terms. Our annotated Re3-Sci dataset is available under a CC-BY-NC 4.0 license.

heading: Acknowledgements
text: This study is part of the InterText initiative3 at the UKP Lab. This work has been funded by the German Research Foundation (DFG) as part of the PEER project (grant GU 798/28-1) and cofunded by the European Union (ERC, InterText, 101054961). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
We extend our sincere gratitude to Dr.-Ing. Richard Eckart de Castilho for his invaluable assistance in creating the cross-document annotation environment within INCEpTION. We also express appreciation to our research assistants and annotators: Gabriel Thiem, Sooyeong Kim, Xingyu Ma, Manisha Thapaliya, Valentina Prishchepova, Malihe Mousarrezaei Kaffash, and ABM Rafid Anwar for their dedicated efforts, active involvement, and valuable feedback throughout the entire process.

heading: A Label Taxonomy and Examples
text: Table 13 presents examples of revisions analyzed according to the three dimensions: granularity, action, and intent, illustrating their importance and indispensability. Table 14 and Table 15 offer detailed definitions and examples for the edit action labels and the edit intent labels, respectively.

heading: B Annotation

heading: B.1 Sentence Segmentation
text: Both F1000RD and NLPeer datasets contain structured documents as intertextual graphs (ITG), a comprehensive document representation format that maintains document structure, cross-document links, and granularity details (Kuznetsov et al., 2022). In those ITGs, paragraphs are the most refined text elements. For our study, we opt to commence with more granular units of sentences. This creates a solid baseline for subsequent expansion to broader units, or to microscopic subdivisions.
We augment the original ITG documents with sentence nodes, employing an assembled sentence segmentation methodology using spaCy4 and ScispaCy5 (Neumann et al., 2019). In our preliminary testing, we discovered that neither spaCy nor ScispaCy sentence splitters are infallible for segmentation, with neither consistently outperforming the other. They can erroneously segment text based on punctuation, such as dots, which are critical for accurate revision alignment. For instance, a dot within numerical values in a sentence could trigger an incorrect segmentation and result in two sentence units. If this dot is omitted in the new version, the sentence is correctly extracted, leading
4Version 3.2.4 5We use the implementation provided at:
https://github.com/allenai/scispacy/blob/main/ scispacy/custom_sentence_segmenter.py
Algorithm 1 Sentence pre-alignment algorithm Input : xt+1,gi ∈ Dt+1, x t,g j ∈ Dt, g = S Output : alignS ∈ 1k×l to 0k×l Ensure : 0 < t0, t1 < 100, t0 < t1
for i← 1 to k do for j ← 1 to l do
for m ∈M do simS[m, i, j]← m(xt+1,gi , x t,g j )
end for end for
end for for i← 1 to k do
for m ∈M do
jmax = argmax i,m simS[m, i, j]
if simS[m, i, jmax] > t1 and all(simS[m, i, jmax] > t0,m ∈M) then
Ci ← Ci + jmax end if
end for if len(f(Ci)) == 1 then
jalign = f(Ci)[0] alignS[i, jalign] = 1
else if len(f(Ci)) > 1 then
jalign = argmin i
d(i, j), j ∈ f(Ci)
alignS[i, jalign] = 1 end if
end for
to significant challenges and errors in aligning the two sentences as a revision pair. We employ an assembly of the two sentence splitters, opting for fewer segmentations yielding a smaller number of longer sentences, which mitigates most incorrect splits. Additionally, special nodes such as article titles, section titles, and list elements are not split.
The segmentations are verified and corrected by a linguistics expert, demonstrating that this integrated approach significantly enhances accuracy compared to using either splitter individually.

heading: B.2 Sentence Revision Pre-alignment
text: Identifying revision pairs from two lengthy documents is challenging, especially complicated by the expansive scope for comparison and the presence of recurring content. In lengthy documents, it’s crucial to align similar sentences, but it’s even more vital to avoid aligning non-relevant sentences with overlapping content. To address this, we design a lightweight algorithm to automatically preannotate sentence revision pairs, additions and deletions, which achieves a decent accuracy of 0.95.
The algorithm is detailed in Algorithm 1 which follows these steps:
1. For sentence-level alignment (g = S), after removing identical paragraph pairs, followed by the removal of identical sentence pairs from the remaining text, there remain k sentences in the new document Dt+1 and l sentences in the old document Dt. 2. For each potential pair, a set of similarity measures m ∈ M is computed, including Levenshtein distance (Levenshtein, 1965) and fuzzy string matching6, as well as semantic similarity measured by SBERT (Reimers and Gurevych, 2019). 3. For each remaining sentence xt+1,gi ∈ Dt+1, using each measure m, the algorithm identifies the most similar candidate xt,gj . If the similarity score exceeds threshold t1 and all other similarity scores between xt+1,gi and x t,g j sur-
pass t0, xt,gj is included in the candidate list Ci, ensuring pairs similar in both form and meaning are found. 4. The function f determines the most frequent element in the resulting candidate list Ci. In cases of multiple equally frequent elements due to repeated content, the alignment is assigned to the candidate closest in location, determined by
di,j =| pi #P t+1 − pj #P t | (1)
where pi is the linear index of the paragraph containing xt+1,gi , and #P
t+1 is the total number of paragraphs in Dt+1. Similarly, pj is the linear index of the paragraph containing xt,gj , and #P
t is the total number of paragraphs in Dt. For example, a sentence from the conclusion is more likely aligned with one from the final parts rather than the introduction. 5. If the candidate list is empty after step 3, the sentence Xt+1,gi stays unaligned, indicating its addition. Finally, if a sentence in Dt
ends up unaligned to any in Dt+1, it is preannotated as a deletion.
The similarity thresholds t0, and t1 are optimized in a pilot study on 20 document pairs, where the ideal configuration was determined to be t0=40,
6We use fuzzywuzzy 0.18.0 at: https://github.com/ seatgeek/fuzzywuzzy
and t1=85, with a similarity of 100 indicating a perfect match.

heading: B.3 Cross-document Annotation Interface
text: For the human annotation process, we utilized the INCEpTION platform (Klie et al., 2018). We developed a cross-document environment7 that offers the complete context of two documents, facilitating full-document revision analysis and various crossdocument annotation tasks. Figure 9 illustrates the annotation interface.
We posit that presenting only two isolated sentences without full document context is insufficient for thorough long document revision annotation. In long papers, crucial content often recurs in sections like the abstract, introduction, and conclusion, making document structure and context essential for accurate revision alignment. Context also plays a significant role in analyzing revision intent. For instance, if the authors change the name of their proposed method, annotators might perceive it as a different method and label it as a semantic change when only given two sentences. However, with the full document context, annotators can recognize a consistent name change throughout the paper, understanding that the referred method remains the same, thus categorizing it as a change for clarity.
B.4 Iterative IAA Assessment
Table 5 demonstrates that the IAA has progressively improved after implementing iterative quality management between data packages, thereby evidencing the efficacy of the employed strategy.
7https://github.com/inception-project/ inception/tree/main/inception/ inception-io-intertext
Tasks ↓ / +Data packages→ Val.set +DP1 +DP2 +DP3

heading: B.5 Annotators and Tasks
text: For the development of the annotation environment, taxonomy, and guidelines, we recruited three annotators: one with expertise in computer science and two specializing in linguistics. The components were iteratively refined based on the annotators’ feedback, and further validated in a pilot study with 20 documents, ensuring the robustness and applicability of our methodologies in a practical setting.
For the subsequent annotation, six master’s students possessing C1-level English proficiency were recruited, including two in-house annotators who contributed to the prior development. Among these annotators, four specialize in linguistics, one has a background in computer science and one in engineering. For the sentence-level edit alignment and labeling tasks, each sample is annotated by three annotators, including one in-house annotator among them. A perfect IAA of 1 Krippendorff’s α for alignment and a substantial IAA of 0.78 α for labeling was achieved (§B.4). For tasks at the subsentence level, a linguistics expert verified and annotated the edit spans and alignments, which were subsequently labeled for action and intent by
three annotators, resulting in an IAA of 0.76 α. In the tasks of extracting and labeling review requests (refer to RQ5 in §5), eight documents were annotated by two annotators, achieving an IAA of 0.75 α. Given the substantial IAA, the remaining documents were evenly distributed and annotated individually by each annotator. Likewise, the task of associating review requests with their corresponding revisions was undertaken by two annotators on eight documents, resulting in an IAA of 0.68 α. Subsequent documents were distributed and each was handled by a single annotator. Each of the six annotators participated in generating edit summaries, with one annotator assigned per document. They then linked the summary sentences to
the corresponding edits. The human evaluation of the LLM-generated document edit summaries was conducted by the first author of this study, an NLP researcher with expertise in both linguistics and computer science. This researcher also assessed the factuality and specificity of the human-authored summaries (§6.2).

heading: B.6 Annotator Survey
text: Following annotation, annotators complete a survey evaluating the effectiveness and efficiency of the guideline, training process, label taxonomy, crossdocument annotation environment, and the annotation tool’s usability. They are presented with various statements and asked to rate their agreement or disagreement on a 7-point scale, where 1 signifies ’Strongly Disagree’ and 7 ’Strongly Agree’.
Table 6 indicates that annotators find the guidelines, label taxonomy, and annotation interface adequate, with the iterative training process being particularly effective. They highlight the value of discussions on individual queries and common mistakes as the most beneficial aspect of the training. Regarding the taxonomy, annotators report that the existing taxonomy adequately encompasses all observed revisions, and they did not feel the need for additional labels during the annotation process. Additionally, they recognize the importance of the cross-document annotation environment, especially in aligning revision pairs and labeling edit actions. This is also evident in their assessments of the subtasks’ difficulty, with the greatest challenges being change detection and intent identification. Moreover, they perceive the annotation tool as highly usable, as indicated by the UMUX (Finstad, 2010)
survey and the average system UMUX score.8

heading: C Dataset analysis
text: Table 7 demonstrates the proportions of each edit action or intent label. Figure 10 illustrates the distribution of both the document edit ratio and the semantic document edit ratio. Figure 11 displays the distribution of the Crest Factor (CF) for the documents, measured at the paragraph and section levels.
8The average system UMUX score is calculated according to: https://blucado.com/understanding-the-umux-a-guide-tothe-short-but-accurate-questionnaire/

heading: D Automation with LLMs

heading: D.1 Edit intent classification
text: Table 11 shows an example prompt with system instruction, demonstration and task instruction used for experiments. Table 8 presents the performance of Llama2-70B in identifying edit intents for additions and deletions. Echoing the findings in Table 3 on revision pairs, we observe that default examples with CoT reasoning yield strong results. These outcomes are further enhanced when three demonstrations selected via the diff method are included. Table 10 presents the results of the joint evaluation conducted on all 8,937 test samples. The challenge in identifying edit intents is particularly evident in revision pairs, highlighted by the low precision in Clarity, low recall in Fact/Evidence, and the difficulties associated with low-sourced Claim and Other classes.

heading: D.2 Document Edit Summarization
text: Table 12 provides examples of human-written and GPT-4 generated document edit summaries.

heading: D.3 Revision Alignment
text: Table 9 presents the experimental results in revision alignment.

heading: D.4 Computational details
text: In our classification tasks with Llama2-70B, covering revision alignment, edit intent classification, and review request extraction, we employed two RTX™ A6000 GPUs, each equipped with 48GB of memory. The batch size for inference was established at four. For the document edit summarization task using GPT-4, we processed 282,964 input tokens and produced 36,341 output tokens in total, resulting in a total expense of 3.92 US dollars.

heading: Future Work
text: Studying the transferability of our findings to new languages, domains, application settings and editorial workflows is an exciting avenue for future research, which can be supported by our openly available annotation environment and protocols. Our study used human-generated edit summaries instead of author responses or summaries of changes written by the authors themselves due to the lack of data. As peer-reviewing data collection becomes increasingly popular in the NLP community, we expect new datasets to enable such studies in the future. From a task perspective, it is important to highlight that the implementations and results presented in this study serve as illustrations of the proposed tasks. Their primary purpose is to ascertain the technical feasibility and lay the groundwork for the development of future NLP systems for collaborative writing and revision assistance. Consequently, the provided implementations have inherent limitations. For instance, our approach selectively utilizes state-of-the-art LLMs without conducting comprehensive comparisons with other LLMs or smaller fine-tuning-based models. A systematic exploration of NLP approaches for the proposed tasks lies beyond our scope and is left for the future