Dual Cache for Long Document Neural Coreference Resolution
Year: 2023

Abstract:
Recent works show the effectiveness of cachebased neural coreference resolution models on long documents. These models incrementally process a long document from left to right and extract relations between mentions and entities in a cache, resulting in much lower memory and computation cost compared to computing all mentions in parallel. However, they do not handle cache misses when high-quality entities are purged from the cache, which causes wrong assignments and leads to prediction errors. We propose a new hybrid cache that integrates two eviction policies to capture global and local entities separately, and effectively reduces the aggregated cache misses up to half as before, while improving F1 score of coreference by 0.7 ∼ 5.7pt. As such, the hybrid policy can accelerate existing cache-based models and offer a new long document coreference resolution solution. Results show that our method outperforms existing methods on four benchmarks while saving up to 83% of inference time against non-cache-based models. Further, we achieve a new state-of-the-art on a long document coreference benchmark, LitBank.

heading: 1 Introduction
text: Coreference Resolution (CR) is fundamental in document-level Natural Language Processing. Its goal is to identify mentions belonging to the same entity. These mentions often scatter throughout the document, but they can be linked together through coreference resolution. CR is a building block for common sense understanding (Levesque et al., 2012; Sakaguchi et al., 2021; Balahur et al., 2011; Liu et al., 2021a), reading comprehension (Dasigi et al., 2019; Storks et al., 2019), information extraction (Yao et al., 2019; Ji and Grishman, 2008; Lu and Ng, 2018), and text summarization (Liu et al., 2021b; Azzam et al., 1999; Xu et al., 2020).
Conventional CR models enumerate every pair of mentions in a document in parallel, so the com-
putation and memory cost is quadratic to the number of mentions in a document (Xia et al., 2020), where mention is a pronoun, a noun phrase, or a text span that can be referred. This quadratic overhead poses a challenge for long-doc CR. Recent work (Xia et al., 2020; Toshniwal et al., 2020, 2021) proposes cache-based CR models which scan mentions in a document from left to right, storing resolved entities in a cache and determining whether to assign a new mention to an entity in the cache or push it to the cache as a new entity. When the cache is full, it will evict entities according to a certain eviction policy, such as LRU (Least Recently Used). We denote the cache with LRU eviction policy as a local cache (L-cache) because it keeps the more recent entities. Since the small cache size bounds the number of potential coreference relations that the model probes for each mention, it reduces the computation and memory cost. Specifically, the cost reduces to O(|C||D|) from O(|D|2), where the cache size |C| ≪ |D|, |C| and |D| are cache size and document length, respectively.
However, multiple topics are common when doc-
15272
ument is long and the focus of narrative may switch (Figure 1). This phenomenon impacts the performance of LRU policy due to the repeated alternation of topics, causing an entity to be mentioned after a long span of text, increasing its chance of eviction from the cache. When processing a mention that requires the evicted entity, it results in a cache miss. Empirically, we find 11.1% mentions encounter cache misses when adopting LRU policy on real data (see Section 5.1).
Our analysis of cache miss reveals that the pattern of entity usages follows the Pareto principle (Koch, 2011) (or 80-20 rule). In other words, a few high frequency entities are used globally (i.e., key concepts for each topic) and account for most of the cache misses. In LitBank (Bamman et al., 2019), 6% of the entities account for 97% of the cache miss. Another example that fits this observation is that the 20 main characters are mentioned 1,722 times in Animal Farm (Orwell, 2021), a book of 30,000 words (see Section 5.5).
Inspired by this insight, we propose a dual cache to address different patterns of word usage. One cache stays unchanged as before, i.e., an L-cache using LRU policy. Another cache is devoted to global and high frequent entities evicted from Lcache, called G-cache. Intuitively, G-cache adopts the classical cache policy LFU (Least Frequently Used). We denote the proposed hybrid cache as Dual cache. The idea is to promote a division of labor in order to make more effective use of cache: L-cache deals with local, clustered mentions, typically in one topic, whereas G-cache targets global
entities within a longer range, typically across topics. A diagram of a model with Dual cache is shown in Figure 2.
We conduct extensive experiments on coreference resolution benchmarks, including OntoNotes (Pradhan et al., 2012), LongtoNotes (Shridhar et al., 2022), LitBank (Bamman et al., 2019), and WikiCoref (Ghaddar and Langlais, 2016). Results show that a CR model with our Dual cache outperforms the prior work (Toshniwal et al., 2020; Xia et al., 2020) which use unbounded cache while saving 56% of the inference time. The model achieves a new state-of-the-art on LitBank (Bamman et al., 2019).
The results shown in Section 4 demonstrate the effectiveness of the Dual cache on long-doc CR benchmarks. To further explore the capability of our Dual cache on long documents, we annotate a book with 30,000 words, our method outperforms the baseline by offering 10 points improvement of F1 while saving 70% of the inference time. The data and code are available https://github.com/ QipengGuo/dual-cache-coref.

heading: 2 Related Work
text: Conventional CR methods (Joshi et al., 2019; Xu and Choi, 2020; Jiang and Cohn, 2021) are designed for short documents with length less than 1,000 words. They typically encode the input document and propose candidate mentions with a pretrained model. Their model enumerates all pairs of mentions in parallel to identify coreference relations, resulting in memory and computation cost
quadratic to document length. These models also require a post-processing to gather connected mentions as entities and need to compute a large tensor of O(n × n × d), where n is the number of mentions, and d is the dimension of hidden states in the relation classifier. This is often impractical for long documents with thousands of mentions. Another direction is to reformulate CR, Wu et al. (2020) treats CR as a question answering problem and achieves higher performance on some benchmarks but costs more computation since the model needs to do question answering repeatedly.
Cache-based Neural Coreference Resolution Since directly applying conventional CR on long documents suffers from heavy computation and memory cost, recent works (Xia et al., 2020; Toshniwal et al., 2020, 2021) proposed cache-based CR models. Probing entities in the cache instead of the whole document largely saves the computation and memory cost but also brings errors when encountering a cache miss. Thus, the eviction policy plays an important role. Xia et al. (2020) follow the LRU principle, and Toshniwal et al. (2020) discuss more eviction policies, such as a variant of LRU and a learnable scorer to rank elements. However, these works lack the ability to capture the topic switching phenomena described in Figure 1.
Datasets for Long document Coreference Existing CR datasets are typically on short documents, such as the most commonly used corpus, OntoNotes (Pradhan et al., 2012), whose average document length is 487 words. However, as Bamman et al. (2019) have shown, text in scientific papers and literature, which can be much longer than OntoNotes articles, exhibit different usages of coreference relations. Note that the original document collected for OntoNotes is much longer, but they split documents to lower the annotation cost, and a recent work LongtoNotes(Shridhar et al., 2022) attempts to recover the long document. Besides, recent work proposes benchmarks for long documents, such as LitBank (Bamman et al., 2019) for literature articles, ACL-Coref (Schäfer et al., 2012) for scholarly papers, and Wiki-Coref (Ghaddar and Langlais, 2016) for documents in Wikipedia. These corpora not only increase document length against traditional coreference benchmarks but also largely increase the spread of entities (the distance between the first and the last mention of an entity , which is defined in Tosh-
niwal et al. (2020)), which requires the model to memorize longer history. Text books, story books, and professional books capture much of the human knowledge, but book-scale CR has not caught attention, and we made the first attempt to address this issue. We annotate a 30,000-word book, Animal Farm, taking the 20 characters as entities, and annotate the 1,722 mentions of them throughout the book.

heading: 3 Method
text: A common workflow of neural CR models is to detect candidate mentions, encode them into vector representations, and identify coreference relations between each mention and past entities by feeding their representations to an MLP classifier. We denote candidate mentions as {m1,m2, · · · ,mn}, and an entity is defined as a set of mentions, ek = {mk1,mk2, · · · }. We use a special operator H(·) to represent vector representations, for example, H(m1) is the representation of the mention m1. The key component of our method is a dual cache which contains an L-cache CL = {e1, · · · , eNL} and a G-cache CG = {e1, · · · , eNG}, where e is an entity in the cache, NL and NG are the sizes of the two caches respectively. Since we focus on cache design, we adopt Longformer (Beltagy et al., 2020) as the document encoder and mention detection model (an MLP scorer) from Toshniwal et al. (2021) which are described in Toshniwal et al. (2020).

heading: 3.1 Dual Cache
text: When an entity is brought to use, it tends to be reused in a local context intensively, exhibiting clustering behavior that an LRU can effectively capture. As mentioned above, topic switching may cause some entities to be intensively used in certain paragraphs scattered in the document but completely not used in others. Thus, these entities are sensitive to be purged by an LRU cache due to a long absence, but their defining characteristic is its high frequency, easily captured by an LFU policy. As such, neither policy can handle both patterns, and the integration is an intuitive way to cover both cases.
We adopt a divide-and-conquer approach to use two caches and adopt different eviction policies. Lcache captures the locally active terms and follows the LRU policy. G-cache focuses on the global terms and follows the LFU policy. When a new
mention comes, we first ask the model to classify whether it is a new entity or is is assigned to an existing entity within the cache. Next, we will check the frequency of this new or updated entity, and we will move it to the G-cache if its frequency is higher than any entity in the G-cache and trigger an eviction of the G-cache when it is full. If the entity does not enter the G-cache, we will move it to the L-cache and trigger an eviction of the Lcache when it is full. L-cache will evict the least recently used entity and G-cache will evict the least frequently used entity.
At the beginning of processing a document, both L-cache and G-cache are empty, and we will fill the G-cache first since the frequency of an empty slot equals zero and the frequency of a newly entered entity is at least one.

heading: 3.2 Coreference Resolution with a Dual Cache
text: We follow Toshniwal et al. (2021) to design an incremental neural coreference model, which scans mentions one by one and computes pair-wise scores for entities in the cache plus a placeholder vector indicating new entity (noted as N ). As its name implies, if this placeholder achieves the highest prediction score, the model will create a new entity and push it to the cache. Otherwise, the model assigns the input mention to the entity with the highest score and updates its representation. Note that the cache structure is transparent to the coreference resolution model, and the model will access all entities in the Dual cache.
The score between a mention m, an entity e in the Dual cache C = {CL, CG}, and a placeholder vector N can be computed as,
s(m,N ) = fc(H(m),N ), (1) s(m, e) = fc(H(m),H(e)), e ∈ C (2)
where H(m), H(e) are representations of a mention and an entity output by a pre-trained language model, respectively. fc means an MLP classifier.
For the dataset that requires the model to distinguish singletons, we record the s(m,N ) as a measure. For an entity only contains one mention after processing the whole document, if the recorded score exceeds the threshold (set to 0 in this work), the model determines it as a singleton. Otherwise, it will not be considered as entity and dropped from the prediction results.
When the model assigns a new mention to an existing entity, the entity representation gets updated
by a gated sum aggregator.
Update(e,m) = αH(m) + (1− α)H(e), (3) where α = fg(H(e),H(m)), (4)
where α is the aggregation coefficient and fg is an MLP module with sigmoid function. For a new created entity, its representation is set with the first mention’s representation.

heading: 3.3 Learning and Inference
text: There are three learnable components in our method, a mention-level representation extractor, which consists of a pre-trained language model and an MLP classifier and is the same as Toshniwal et al. (2021), a pair-wise scoring module eq. (2) to classify coreference relations, and an update function eq. (3) used to update clusters’ representation when assigning new mentions to an entity. Since we only introduce a new cache policy, we can reuse other cache-based models’ parameters like Xia et al. (2020) and Toshniwal et al. (2021) to perform coreference resolution without further training.
We can also retrain or finetune the model with the Dual cache for long documents, which has two potential advantages: 1) the Dual cache has a higher cache hit ratio so that it contains more ground-truth entities, which means the model can see more positive relations during training; 2) it avoids feature space shift since we get the representation of entities by merging mentions sequentially. Different cache polices lead to different merging orders of mentions so that affect the entities’ representation.
We follow Toshniwal et al. (2021) to adopt Cross Entropy as the loss function and take the groundtruth of coreference relations and mention boundaries as labels.

heading: 4 Experiments
text: We conduct experiments on four CR benchmarks and compare with prior cache-based CR methods.

heading: 4.1 Datasets
text: The four datasets for our experiments are: OntoNotes and its extension LongtoNotes for longer documents; WikiCoref with only testing documents; LitBank which provides an official split of ten-fold cross-validation and we follow prior work (Toshniwal et al., 2021) to report the result of the first fold in this work. The statistics of the datasets are listed in Table 1.

heading: 4.2 Baselines
text: Xia et al. (2020) proposed a simple cache-based CR model built on SpanBERT (Joshi et al., 2020) with an L-cache. The model architecture is inherited from a strong baseline of conventional CR model (Joshi et al., 2019) and is finetuned based on their released model. They freeze SpanBERT during finetuning and only report results on OntoNotes instead of benchmarks with longer documents.
Toshniwal et al. (2021) 1 provided more results of cache-based CR models for long documents, including results on OntoNotes, LitBank, and WikiCoref. They replace the backbone encoder with Longformer (Beltagy et al., 2020) for better performance and update all parameters during finetuning. We follow their training scripts and hyperparameters 2 to train our model.
Thirukovalluru et al. (2021) largely accelerates the model by changing span-level CR to token-level CR, which considers the relations between token pairs first, and then maps them as relations between mentions and entities. However, looking at the tokens in two mentions is not enough to identify a coreference relation, especially for long mentions. Their code 3 was not released when writing this paper. Thus, we only compare the F1 score with them. They report the result on a book of 2M tokens. Since their predictions are not released, we plan to compare with them on book-scale document in the future.

heading: 4.3 Setup
text: We initialize our model with the parameters released by Toshniwal et al. (2021), including a document encoder, a mention proposer, and a CR model. We also use a unified mention detection strategy
1We do not compare with their previous work (Toshniwal et al., 2020) since the experiment setting is slight difference, and the compared one is an upgrade version in terms of both performance and methodology.
2github.com/shtoshni/fast-coref 3github.com/raghavlite/Scalable-Coreference
and keep the top 0.4|D| candidate mentions as Toshniwal et al. (2021) did for a fair comparison except the one on Animal Farm, where |D| is the document length. We focus on CR model and report its computation cost and inference time, as the document encoder and mention proposer enable plug-and-play. All experiments are conducted on NVIDIA T4 GPU (16GB). We report MAC (Multiply Accumulate operations) to measure the computation cost. 4

heading: 4.4 Main Results
text: Table 3 compares our method and prior method on three benchmarks. We denote the size of the Dual cache as “A+B", meaning there are A slots in the L-cache and B slots in the G-cache. We report the F1 score and inference time as measurements of performance and efficiency, respectively. We list more metrics like MUC (Vilain et al., 1995), BCUBED (Bagga and Baldwin, 1998), and CEAFE (Luo, 2005) in the Appendix C.
Besides the overall performance, we compare with the strongest baseline (Toshniwal et al., 2021) under the setting of different cache sizes. Results demonstrate that our method consistently outperforms previous approaches, albeit with a slightly higher time consumption for a given cache size. Furthermore, our method achieves comparable performance in less inference time. In particular, our Dual cache with 50 slots outperforms the previous SOTA approach that utilizes an unbounded cache on LitBank, while only requiring 48% of the inference time.
Additionally, we provide results on LongtoNotes, a recent extension of OntoNotes that combines pas-
4The script for calculating MAC is adapted from https: //github.com/Lyken17/pytorch-OpCounter/
sages into longer documents. Table 2 exhibits the same trend, where our Dual cache outperforms the baselines on both the complete test set and a subset of long documents. Notably, the performance gain is more significant in the long document subset, highlighting the effectiveness of the Dual cache structure in processing such documents.
To better illustrate the effectiveness and efficiency, we report more results on WikiCoref in Figure 3. In this setting, we adopt a CR model trained with unbounded memory, so it does not have a preference of the cache structure. The curves show that the Dual cache always has the highest performance/cost ratio among different methods and outperforms both the L-cache and the G-cache either using a fixed cache size or consuming a fixed amount of time, demonstrating the benefit of integrating the L-cache and G-cache.

heading: 5 Analysis

heading: 5.1 Cache Miss Ratio
text: By comparing the performance gain when using different cache sizes in Table 3 and Table 2, we find that the improvement of the Dual cache is more significant for a small cache, such as the performance gain for 50 slots is 5.7pt F1 and the gain for 200 slots is 1.2pt F1. The reason is that the cache miss ratio decreases rapidly as the cache gets larger, so the absolute improvement of our method is also getting smaller.
In this section, we quantitatively discuss how the Dual cache reduces the cache miss ratio. We adopt an off-the-shelf mention detection model (Toshniwal et al., 2021) to detect mentions and use ground truth to replace the relation classifier to get rid of the model difference, except the cache structure.
Table 4 gives the cache miss ratio for different caches on three benchmarks. Results show that the Dual cache reduces the chance of cache miss significantly, especially in the case of 10 slots and 30 slots. A cache miss will lead to an inevitable prediction error since the groundtruth does not lie in the solution space, which means the model only considers assigning the mention to clusters within the cache, but the groundtruth cluster is not in the cache. Figure 5 illustrates the impact of cache miss ratio, indicating a lower cache miss ratio and suggesting a higher performance.
In addition, we provide case studies of four entities in Figure 4, which shows the heatmap of an entity’s occurrences. The color reveals the density of occurrences. A light color means the entity is frequently used there, and a dark color means no usage. We highlight regions with long periods of an entity’s absence by red boxes. The region length can reach 500 words, meaning the L-cache will evict it unless the cache has more than 500 slots. The second case (“Katharine Hilbery") contains multiple relatively long periods (>100 words) of absence. If the cache size is less than 100, it will encounter the cache miss 5 times and cause errors.

heading: 5.2 Impact of Entity Spread
text: As we have shown in Figure 4, an entity may spread in a long range in a document, and this type of en-
tities is challenging for cache mechanisms. One basic measurement is the Entity Spread, which describes how an entity scatters throughout a document. A large spread means the entity spans a large text area, which is likely to be missed by an L-cache if the L-cache is not large enough and the entity is absent for a while.
Figure 6 reports models’ results on different entity spread scales. We show that the Dual cache helps the model resolve entities with a large spread, resulting in nearly a 30 F1 gain. The reason is that an entity with a large spread is often frequently used, which can be captured by G-cache easily. We can also see that L-cache is good at entities with small spreads while G-cache is good at entities with large spreads, and the results also demonstrate that neither of them could solve the coreference alone. The dual cache achieves better results by taking advantage of both L-cache and G-cache.

heading: 5.3 Impact of Cache Size
text: Cache size is an essential factor for a cache design, and there are two caches in our Dual Cache structure, so we study the influence of total cache size and the allocation of L-cache and G-cache and discuss them in this section.
We test our method with the cache size varying from 10 to 100. We report the results of the models without finetuning to avoid the potential noise of introducing new parameters and training processes. Figure 5 shows results on three benchmarks, and we can see the Dual cache outperforms both the L-cache and the G-cache, especially for a small cache.
Since the Dual cache consists of two caches, it is
valuable to discuss how their allocation affects the performance. We provide a heatmap in Figure 7, where the X-axis is the size of G-cache and the Y-axis is the size of L-cache. We enumerate all their combinations (200 settings) when varying the size from 1 to 20. Results show the G-cache is more important when the total size is less than 20 (the lower triangle). The allocation of two caches becomes less matter when the total size exceeds 20 (the upper triangle).

heading: 5.4 Priority of two caches
text: The positions of the L-cache and the G-cache are not equivalent in our Dual cache. What we have described in this work is actually a G-first Dual cache, which means the priority of the G-cache is higher than the L-cache. Particularly, it means we will push an entity to the G-cache when both the G-cache and L-cache has empty slots. We also tried the L-first version and report it in Table 5. The results show that the difference between G-first and L-first is around 0.1%, which is negligible.

heading: 5.5 Results on Book-Scale CR
text: The proposed Dual cache structure is for long-doc CR. However, the maximum length of documents in the four benchmarks is fewer than 10,000 words. To test our approach on a more challenging scenario, we annotate a book of approximately 30,000 words, Animal Farm. We choose the 20 characters in this book as entities and annotate all their 1,722 mentions across the book. Appendix A describes
the details of the book annotation. We compare our model and the baseline trained on LitBank since its documents are also from books. We tokenize the book into 37,000 tokens and split them into 11 segments so that we can feed them into the document encoder. We set the cache size to 500 for the baseline and 250+250 for the Dual cache for a fair comparison.
Our model can parse the entire book in 10 minutes when selecting 0.4|D| top-scored mentions, but the baseline can not finish the inference process in an hour. Thus, we reduce the number of mentions to 0.3|D| without changing other settings. We report F1 score, inference time, and inference memory in Table 6. The results reveal that the model with a Dual cache significantly outperforms the baseline in both efficiency and effectiveness on a book-scale document.

heading: 6 Conclusion
text: We proposed a new cache structure, Dual cache, to tackle long document coreference resolution. It contains two caches governed by different eviction polices. The local cache follows LRU policy to deal with local, clustered mentions, whereas the global cache follows LFU policy to target global entities within a long span. Empirical results show that the Dual cache lowers the cache miss up to half as before and improves the F1 score consistently on four benchmarks by offering an average gain of 5.7 F1 score for small caches while taking the same inference time. We also achieve a new stateof-the-art, 79.5 F1 score, on LitBank.
Limitations
This work is motivated by the intuition that a mention is more likely to refer to an entity that occurs shortly earlier and refers to a high frequency entity but has not recently used. For the latter pattern, we show examples of topic switching to explain why these phenomena happened, but we have not found a rigorous linguistic explanation to support this finding. We provide empirical results on four benchmarks plus a book. However, the scarcity of long-doc CR benchmarks hinders us from verifying on a larger scale.
Our major contribution is a new cache design, but we also find the cache design becomes less matter when using a huge cache. NVIDIA A100 has 80G memory, which means it can handle a document of 100,000 words with a conventional CR
model. As the GPU becomes larger and cheaper, the importance of studying cache design is weaker.
Ethics Statement
We comply with the ACL Ethics Policy. Coreference resolution is fundamental in NLP, which often serves as a component of other NLP applications. Since it does not directly interact with users, there are no additional ethics concerns. All the data used in this paper is public. We confirm that the scientific artifacts used in this paper comply with their license and intended use. We list the licenses in Table 7.

heading: A Annotation of Animal Farm
text: The text of Animal Farm is accessed from the site of Project Gutenberg Australia.5 We describe the annotation process as follows.
Preprocessing We append the chapter titles in front of the texts of the chapters as the first sentences of the chapters, and concatenate the texts of the chapters into a single document. Before the annotation, we adopt a character list containing the 20 characters as the 20 entities,6 and first extract the their mentions with string matching.
Annotation Tool We developed a web-based annotation tool with a server developed with Flask7 as shown in Figure 8. The entities are listed in the top of the web page, shown in the blue boxes with the texts from the character list. The text of the book are splitted into multiple segments so we can view each segment in one page, and the annotator can switch the segment by the “Previous” and “Next” buttons. In each page, the segment of text is splitted into multiple paragraphs to avoid getting lost in the long text. When we click one entity in the blue box, e.g. “Mr. Jones”, it means we are now annotating this entity now, and the mentions of this entity in the text gets gray colored. We annotate the mentions by clicking the words in the text. Once the word is clicked, it gets gray colored (e.g. “his”, “he”, “himeself”). Consecutive words are treated as one span of mention. When finished annotation for the current page, we click the “Save” button to save the annotation to server.
Annotation Standard The mentions of the entities include pronouns (e.g. “he”, “she”, “I” “they”, etc), possessive pronouns (e.g. “his”, “her”, “my”, “their”, etc) as the case in Ontonotes 5.0, and other noun phrases (e.g. “Jones”, “Major”, “the three dogs”, etc).
The annotation is done by a human expert for 5 hours and there are totally 1,722 mentions annotated.

heading: B Data and Model license
text: Following the instruction of ACL, we list the science artifacts used in this work in Table 7.
5The book is available at https://gutenberg.net.au/ ebooks01/0100011h.html
6We adapt the character list from https://www. litcharts.com/lit/animal-farm/characters
7https://flask.palletsprojects.com/en/2.1.x/

heading: C More Metrics
text: Besides the F1 score that we reported in the main text, we give results that are evaluated by commonly used metrics for the comparison with others, including MUC (Vilain et al., 1995), B-CUBED (Bagga and Baldwin, 1998), and CEAFE (Luo, 2005). For better visualization, we split results into three tables, Table 8, Table 9, and Table 10. There metrics show the same trend of F1 score that are shown in the main text.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations section
3 A2. Did you discuss any potential risks of your work? Ethics Statement section
3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes, we did.
3 A4. Have you used AI writing assistants when working on this paper? We used ChatGPT for polishing the camera ready section, mainly for the introduction part.
B 3 Did you use or create scientific artifacts? Section 4.1, 4.2
3 B1. Did you cite the creators of artifacts you used? Section 4.1, 4.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Ethics Statement section and Appendix
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement section and Appendix
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1, 4.2
C 3 Did you run computational experiments? Section 4.4, 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4.3, 4.4, we report the computation cost (MACs) and computing devices. The number of parameters can be founded in cited papers .
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.4, Appendix
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5.5, Appendix
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Appendix
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. The data is annotated by authors as a test set.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Since their predictions are not released, we plan to compare with them on book-scale document in the future